Is there any way of configuring hangfire to run single threaded? I'd like the jobs to be processed sequentially, rather than concurrently.
Something like:
app.UseHangfire(config =>
{
config.RunSingleThreaded();
config.UseServer();
});
Either this or the ability to "chain" jobs together so they happen in sequence.
Something like:
BackgroundJob
.Enqueue(() => taskContainer.PublishBatch(batchId, accountingPeriodId, currentUser, filePath))
.WithDependentJobId(23); // does not run until this job has finished...
Should have read the docs obviously...
http://docs.hangfire.io/en/latest/background-processing/configuring-degree-of-parallelism.html
To configure single thread use the BackgroundJobServerOptions type, and specify workerCount:
var server = new BackgroundJobServer(new BackgroundJobServerOptions
{
WorkerCount = 1
});
Also, it appears job chaining is a feature of Hangfire Pro version.
Related
Currently have Elastic Apm setup with: app.UseAllElasticApm(Configuration); which is working correctly. I've just been trying to find a way to record exactly how many SQL Queries are run via Entity Framework for each transaction.
Ideally when viewing the Apm data in Kibana the metadata tab could just include an EntityFramework.ExecutedSqlQueriesCount.
Currently on .Net Core 2.2.3
One thing you can use is the Filter API for this.
With that you have access to all transactions and spans before they are sent to the APM Server.
You can't run through all the spans on a given transaction, so you need some tweaking - for this I use a Dictionary in my sample.
var numberOfSqlQueries = new Dictionary<string, int>();
Elastic.Apm.Agent.AddFilter((ITransaction transaction) =>
{
if (numberOfSqlQueries.ContainsKey(transaction.Id))
{
// We make an assumption here: we assume that all SQL requests on a given transaction end before the transaction ends
// this in practice means that you don't do any "fire and forget" type of query. If you do, you need to make sure
// that the numberOfSqlQueries does not leak.
transaction.Labels["NumberOfSqlQueries"] = numberOfSqlQueries[transaction.Id].ToString();
numberOfSqlQueries.Remove(transaction.Id);
}
return transaction;
});
Elastic.Apm.Agent.AddFilter((ISpan span) =>
{
// you can't relly filter whether if it's done by EF Core, or another database library
// but you have all sorts of other info like db instance, also span.subtype and span.action could be helpful to filter properly
if (span.Context.Db != null && span.Context.Db.Instance == "MyDbInstance")
{
if (numberOfSqlQueries.ContainsKey(span.TransactionId))
numberOfSqlQueries[span.TransactionId]++;
else
numberOfSqlQueries[span.TransactionId] = 1;
}
return span;
});
Couple of thing here:
I assume you don't do "fire and forget" type of queries, if you do, you need to handle those extra
The counting isn't really specific to EF Core queries, but you have info like db name, database type (mssql, etc.) - hopefully based on that you'll be able filter the queries you want.
With transaction.Labels["NumberOfSqlQueries"] we add a label to the given transction, and you'll be able to see this data on the transaction in Kibana.
In django application I need to call an external rabbitmq, running on a windows server and using some application there, where the django app runs on a linux server.
I'm currently able to add a task to the queue by using the celery send_task:
app.send_task('tasks', kwargs=self.get_input(), queue=Queue('queue_async', durable=False))
My settings looks like:
CELERY_BROKER_URL = CELERY_CONFIG['broker_url']
BROKER_TRANSPORT_OPTIONS = {"max_retries": 3, "interval_start": 0, "interval_step": 0.2, "interval_max": 0.5}
CELERY_ACCEPT_CONTENT = ['application/json']
CELERY_TASK_SERIALIZER = 'json'
CELERY_RESULT_SERIALIZER = 'json'
CELERY_TASK_DEFAULT_QUEUE = 'celery'
CELERY_TASK_RESULT_EXPIRES = 3600
CELERY_RESULT_BACKEND = 'rpc://'
CELERY_CREATE_MISSING_QUEUES = True
What I'm not sure about is how I can get and parse the response, since the send_task only returns a key?
If you want to store results of your task , you could use this parameter result_backend or CELERY_RESULT_BACKEND depending on the version of celery you're using.
Complete list of Configuration options can be found here (search for result_backend on this page) => https://docs.celeryproject.org/en/stable/userguide/configuration.html
Many options are available to store results - SQL DBs , NoSQL DBs, Elasticsearch, Memcache, Redis, etc,etc . Choose as per your project stack.
Thanks that helped for the understanding. So since I want to further process the answer I use rpc as already defined in the config I had in the example.
What I found usefull was this example, because most python celery examples assume that the consumer is the same application, that describes the interaction to a Java app Celery-Java since it gives a good example on how to request from python side.
Therefore my implementation is now:
result = app.signature('tasks', kwargs=self.get_input(), queue=Queue('queue_async', durable=False)).delay().get()
which waits and parses the result.
Has anyone used multiple instances of Hangfire (in different applications) with same SQL DB for configuration. So instead of creating new SQL DB for each hangfire instance i would like to share same DB with multiple instances.
As per the hangfire documentation here it is supported since v1.5 However forum discussion here and here shows we still have issues running multiple instances with same db
Update 1
So based on suggestions and documentation i configired hangfire to use queue
public void Configure(IApplicationBuilder app, IHostingEnvironment env,
ILoggerFactory loggerFactory)
{
app.UseHangfireServer(new BackgroundJobServerOptions()
{
Queues = new string[] { "instance1" }
});
}
Method to invoke
[Queue("instance1")]
public async Task Start(int requestID)
{
}
This is how i Enqueue job
_backGroundJobClient.Enqueue<IPrepareService>(x => x.Start(request.ID));
however when i check [JobQueue] table the new job has queue name default and because of that hangfire will never pickup that job because it picks up jobs for queues.
I think is a bug
Update 2
Found one more thing. I am using instance of IBackgroundJobClient. The instance is automatically get injected by .Net Core's inbuilt container.
So if i use instance to enqueue the job then hangfire creates new job with default queue name
_backGroundJobClient.Enqueue<IPrepareService>(x => x.Start(request.ID));
However if i use static method, then hangfire creates new job with configured queue name instance1
BackgroundJob.Enqueue<IPrepareService>(x => x.Start(prepareRequest.ID));
How do i configure hangfire in .Net Core so the instance of IBackgroundJobClient will use configure queue name ?
This is possible by simply setting the SQL server options with a different schema name for each instance.
Instance 1:
configuration.UseSqlServerStorage(
configuration.GetConnectionString("Hangfire"),
new SqlServerStorageOptions { SchemaName = "PrefixOne" }
);
Instance 2:
configuration.UseSqlServerStorage(
configuration.GetConnectionString("Hangfire"),
new SqlServerStorageOptions { SchemaName = "PrefixTwo" }
);
Both instances use same connection string and will create two instances of all the required tables with the prefix specified in the settings.
Queues are used for having separate queues in the same Hangfire instance. If you want to use different queues you'll need to specify the queues you want the IBackgroundJobClient to listen to and then specify that queue when creating jobs. This doesn't sound like what you're trying to accomplish.
Hangfire (v1.3 +) has a 'clever' feature where it picks up your application's existing logging setup and uses it.
Starting from Hangfire 1.3.0, you are not required to do anything, if your application already uses one of the following libraries through the reflection (so that Hangfire itself does not depend on any of them).
Because I don't want hangfire logging mixed in with my application logs I would like to filter them out into a separate log file.
Serilog has filters to do this, but it needs something to filter on.
Does Hangfire include any useful context that I can specify when filtering?
I think the filter you can use will look something like:
Log.Logger = new LoggerConfiguration()
.WriteTo.ColoredConsole()
.Filter.ByIncludingOnly(Matching.FromSource("Hangfire"))
.CreateLogger();
See also this post.
I couldn't get the Serilog Matching.FromSource(...) to work, the Hangfire events don't appear to have that property. I have the following solution:
var logFile = "...";
var hangfireFile = "...";
var hangfireEvents = Matching.WithProperty<string>("Name", x => x.StartsWith("Hangfire"));
Log.Logger = new LoggerConfiguration()
.MinimumLevel.Verbose()
.WriteTo.Logger(lc =>
lc.Filter.ByExcluding(hangfireEvents)
.WriteTo.RollingFile(new CompactJsonFormatter(new SafeJsonFormatter()), logFile))
.WriteTo.Logger(lc =>
lc.Filter.ByIncludingOnly(hangfireEvents)
.WriteTo.RollingFile(new CompactJsonFormatter(new SafeJsonFormatter()), hangfireFile))
.CreateLogger();
I've implemented a jobs queue a few days ago and I've been experiencing problems with duplication, I'm currently working with Redis and followed the Laravel's official tutorial.
In my case, whenever someone goes to the homepage, a job is sent to the queue, lets take this example:
HomeController's index() :
public function index()
{
if(/*condition*/){
//UpdateServer being the job
$this->dispatch(new UpdateServer());
}
}
Since this task takes about 10 seconds to complete, if there's n requests to my homepage while the task is being processed, there will be n more of the same job in queue, resulting in unexpected results in my Database.
So my question is, is there any way to know if a certain job is already in queue?
I know it's an old question but I find myself coming back here again and again from Google so I wanted to give it an answer. I wanted an easy way to view the jobs in the queue inside my Laravel application on a dashboard and used the following code.
$thejobs = array();
// Get the number of jobs on the queue
$numJobs = Redis::connection()->llen('queues:default');
// Here we select details for up to 1000 jobs
$jobs = Redis::connection()->lrange('queues:default', 0, 1000);
// I wanted to clean up the data a bit
// you could use var_dump to see what it looks like before this
// var_dump($jobs);
foreach ($jobs as $job) {
// Each job here is in json format so decode to object form
$tmpdata = json_decode($job);
$data = $tmpdata->data;
// I wanted to just get the command so I stripped away App\Jobs at the start
$command = $this->get_string_between($data->command, '"App\Jobs\\', '"');
$id = $tmpdata->id;
// Could be good to see the number of attempts
$attempts = $tmpdata->attempts;
$thejobs[] = array($command, $id, $attempts);
}
// Now you can use the data and compare it or check if your job is already in queue
I don't recommend doing this, especially on page load such as the index page like the op has done. Most likely you need to rethink the way you are doing things if you need to have this code to check if a job is running.
The answer is specific to queues running Redis.
I know it's very old question, but I'm answering it for future Google users.
Since Laravel 8 there is the "Unique Jobs" feature - https://laravel.com/docs/8.x/queues#unique-jobs.
For anyone wondering why
Queue::size('queueName');
is not the same size as
Redis::llen('queues:queueName');
is because Laravel uses 3 records to count the size of a queue, so if you want the true number of jobs in the queue you must do:
Redis::lrange('queues:queueName', 0, -1);
Redis::zrange('queues:queueName:delayed', 0, -1);
Redis::zrange('queues:queueName:reserved', 0, -1);
Now you can evaluate if your desired input is in one of those queues and act according.
You can do it in jobs handle function and skip work if another same job is scheduled
public function handle()
{
$queue = \DB::table(config('queue.connections.database.table'))->orderBy('id')->get();
foreach ($queue as $job){
$payload = json_decode($job->payload,true);
if($payload['displayName'] == self::class && $job->attempts == 0){
// same job in queue, skip
return ;
}
}
// do the work
}