we are triggering a scheduled job in a configured interval and it is getting fired. This is working in a single hangfire server environment.
In some of our environment, we will be having more than one servers. So, at this scenario, all the servers will trigger that job at that particular interval. Can we restrict this job to be triggered from only one server?
string cronExpressionCleanupJob = "0 0 0/{2} ? * *";
RecurringJob.AddOrUpdate<CleanUpJobTriggerJob>(nameof(CleanUpJobTriggerJob),
job => job.ExecuteJob(null, null), cronExpressionCleanupJob, TimeZoneInfo.Local);
var hangFireJobId = BackgroundJob.Enqueue<CleanUpJobTriggerJob>(x => x.ExecuteJob(null, null));
According to the documentation, you should not worry as long as you specify the same identifier for your recurring job.
The call to AddOrUpdate method will create a new recurring job or
update existing job with the same identifier.
Related
For security or simplicity our Hangfire dashboard UI is not present on production. So we can't just go on and click "Retry". But I'd like to retry some failed jobs manually using SQL. In the DB there are stateid and statename fields. Is it possible to refresh a job by changing its stateid?
I got this idea from the Enqueue function (in SqlServerJobQueue.cs on GitHub) for SQL Server (I'm using postgresql).
insert into hangfire.jobqueue (jobid, queue)
select id, 'default' from hangfire.job where statename = 'Failed';
But it doesn't seem to do anything.
To reschedule scheduled jobs I am currently deleting the previous job and then scheduling a new one within a transaction so if the schedule fails it rolls back the delete:
using (var transaction = new TransactionScope(TransactionScopeOption.Required))
{
BackgroundJob.Delete(jobId);
BackgroundJob.Schedule(...);
}
This involves keeping storing the original arguments somewhere so I can reschedule with the original args.
I was wondering whether there was any ramifications to doing the following instead:
new BackgroundJobClient().ChangeState(jobId, new ScheduledState(timespan), ScheduledState.StateName);
This way there is no need to keep the original job arguments around and also is transactional.
There is a BQ table which has multiple data load/update/delete jobs scheduled in. Since this is automated jobs many of it are failing due to concurrent update issue.
I need to know if we have a provision in BigQuery to check if the table is already locked by DML operation and can we serialize the queries so that no job fails
You could use the job ID generated by client code to keep track of the job status and only begin the next query when that job is done. This and this describe that process.
Alternatively you could try exponential backoff to retry the query a certain amount of times to prevent automatic failure of the query due to locked tables.
I have a table in my database with defined jobs. One of jobs attribute is status, which can be [waiting, in_progress, done]. To process jobs i have defined master-worker relation between two servers, they work in the following way:
master looks for the first record in database with status 'waiting' and triggers a worker server to process the job.
Triggered worker sets status on the job to 'in_progress' in one transaction and starts executing the job (in the meantime server is looking for next job with status 'waiting' and triggers another worker)
when the job is done worker sets status on the job to 'done'
This is the perfect case scenario, however it might happen that during job execution one of the workers dies. In this case job needs to be restarted, however master has no way of verifying if job was done other than checking its status in database ('done'). Therefore if worker dies, in database the job has still status 'in_progress' and server has no idea that worker died and that job needs to be restarted. (we can't get any feedback from worker, we can not ping it and we can not get information what job is he currently working on)
My idea to solve this problem would be:
After worker changed job's status to 'in_progress' (transaction committed) he would open a new transaction with lock on the particular job.
Master, while looking for jobs to start would look for both 'waiting' and 'in_progress' which are not locked
If worker dies, transaction would break, releasing the lock from job record in database and master would reprocess it.
now i'm looking for a way to verify it this would indeed work. Possibly with SQL sctipt in SQL developer (two instances), i would like this to work in the following way:
Instance 1:
open transaction and create a row lock on row_1
sleep for 5 min
release lock
Instance 2:
Open transaction and look for row matching criteria (row_1 and row_2 match criteria)
return selected row
then i would kill instance 1 to simulate worker death and run instance 2 again.
Please advise if my approach is correct
P.S
Also could you point me to some good explanation how can i create script form instance 1?
What happens if oracle database starts job(from 'Scheduler') before last job(this same) finishes? Does oracle add it to a stack or finally stops?
Oracle is smart enough to know not to start a new job instance before the previous job is finished.
From the Oracle docs:
http://docs.oracle.com/cd/B19306_01/server.102/b14231/scheduse.htm
Setting the Repeat Interval
...
Immediately after a job is started, the repeat_interval is evaluated to determine the next scheduled execution time of the job. It is possible that the next scheduled execution time arrives while the job is still running. A new instance of the job, however, will not be started until the current one completes.