Hangfire - Background job creation failed. See inner exception for details - azure-sql-database

Question

I create a hangfire jobs as `BackgroundJob.Enqueue(x => x.Recalculate(itemID)); to queue a job. When I try to save consecutively few times I get following error from Hangfire.
Here is the stack trace,
at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString, Boolean isInternal, Boolean forDescribeParameterEncryption)
at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task, Boolean asyncWrite, Boolean inRetry, SqlDataReader ds, Boolean describeParameterEncryptionRequest)
at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, TaskCompletionSource`1 completion, Int32 timeout, Task& task, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry)
at System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(TaskCompletionSource`1 completion, String methodName, Boolean sendToPipe, Int32 timeout, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry)
at System.Data.SqlClient.SqlCommand.ExecuteNonQuery()
at Dapper.SqlMapper.ExecuteCommand(IDbConnection cnn, CommandDefinition& command, Action`2 paramReader)
at Dapper.SqlMapper.ExecuteImpl(IDbConnection cnn, CommandDefinition& command)
at Dapper.SqlMapper.Execute(IDbConnection cnn, String sql, Object param, IDbTransaction transaction, Nullable`1 commandTimeout, Nullable`1 commandType)
at Hangfire.SqlServer.SqlServerWriteOnlyTransaction.<>c__DisplayClass7_0.<SetJobState>b__0(SqlConnection x)
at Hangfire.SqlServer.SqlServerWriteOnlyTransaction.<Commit>b__4_0(SqlConnection connection)
at Hangfire.SqlServer.SqlServerStorage.<>c__DisplayClass17_0.<UseTransaction>b__0(SqlConnection connection)
at Hangfire.SqlServer.SqlServerStorage.UseConnection[T](Func`2 func)
at Hangfire.SqlServer.SqlServerStorage.UseTransaction[T](Func`2 func, Nullable`1 isolationLevel)
at Hangfire.SqlServer.SqlServerStorage.UseTransaction(Action`1 action)
at Hangfire.Client.CoreBackgroundJobFactory.Create(CreateContext context)
at Hangfire.Client.BackgroundJobFactory.<>c__DisplayClass7_0.<CreateWithFilters>b__0()
at Hangfire.Client.BackgroundJobFactory.InvokeClientFilter(IClientFilter filter, CreatingContext preContext, Func`1 continuation)
at Hangfire.Client.BackgroundJobFactory.Create(CreateContext context)
at Hangfire.BackgroundJobClient.Create(Job job, IState state)
Exception message: Background job creation failed. See inner exception for details.(BackgroundJobClientException)
Inner Ex: Execution Timeout Expired. The timeout period elapsed prior to completion of the operation or the server is not responding.
Has anyone came across this issue? This works fine in locally. But issue happens in Azure Deployment. I am using a v12 database. Appreciate any help. I noticed that all my hangfire job creation takes like 20 seconds on average to queue the background job. When I try to create the job again and again it times out. Time out seems to be 30 seconds.
I tried to increase the connection timeout but what ever the value I put in connection string, it seems it times out after 30 seconds. But increasing the time out won't solve the underlying issue anyways.

Answer 1

I figured out that DTU usage is at 100% most of the time, which was unacceptable as there was nothing much happening in the Test environment that I was testing this. Then I looked in to performance data in Azure database which gave me worst performing queries, which was really handy. Handfire delete jobs query was causing this performance issue. It was an issue with the Hangfire version we were using. Solution was to upgrade the Hangfire to a latest version.