I have a table in my database with defined jobs. One of jobs attribute is status, which can be [waiting, in_progress, done]. To process jobs i have defined master-worker relation between two servers, they work in the following way:
master looks for the first record in database with status 'waiting' and triggers a worker server to process the job.
Triggered worker sets status on the job to 'in_progress' in one transaction and starts executing the job (in the meantime server is looking for next job with status 'waiting' and triggers another worker)
when the job is done worker sets status on the job to 'done'
This is the perfect case scenario, however it might happen that during job execution one of the workers dies. In this case job needs to be restarted, however master has no way of verifying if job was done other than checking its status in database ('done'). Therefore if worker dies, in database the job has still status 'in_progress' and server has no idea that worker died and that job needs to be restarted. (we can't get any feedback from worker, we can not ping it and we can not get information what job is he currently working on)
My idea to solve this problem would be:
After worker changed job's status to 'in_progress' (transaction committed) he would open a new transaction with lock on the particular job.
Master, while looking for jobs to start would look for both 'waiting' and 'in_progress' which are not locked
If worker dies, transaction would break, releasing the lock from job record in database and master would reprocess it.
now i'm looking for a way to verify it this would indeed work. Possibly with SQL sctipt in SQL developer (two instances), i would like this to work in the following way:
Instance 1:
open transaction and create a row lock on row_1
sleep for 5 min
release lock
Instance 2:
Open transaction and look for row matching criteria (row_1 and row_2 match criteria)
return selected row
then i would kill instance 1 to simulate worker death and run instance 2 again.
Please advise if my approach is correct
P.S
Also could you point me to some good explanation how can i create script form instance 1?
Related
I have a SQL Agent job that runs on the 7th of each month on an hourly basis. I want to stop it from running further once it is successful.
For example, if the job is successful at 8:00 A.M, I dont want it to run any more untill the 7th of next month. Any scripts would be helpful for this.
I am working on trying to establish this rule through the use of MSDB sys.jobs and an idea that I have is to update the Enabled flag to a 0 once the run is complete. Once the 7th of next month hits, another job on the SQL Agent could go in an update the flag back to a 1 so it can be run. i
I suggest you create a new first job-step and, using the answer provided in the question sql-agent-job-last-run-status, you can check if the last execution succeeded.
You can then cancel the job from executing by using exec msdb.dbo.sp_stop_job
I use this method in an Availability group cluster (to check if SQL Agent is running on the primary for jobs the require it) and abort the job if not. It works well.
How about, create a table with a row for last successful run, populate it on a successful run.
Then first thing in the job, check that table, and if the last successful run is since the 7th, terminate.
Otherwise run the rest of the job.
The solution that I finally deployed was to add a step to the job that would disable it once all preceding steps succeeded. I added another job to the Agent that would go and enable this job on the 1st of each month so that they are ready to be executed. Just throwing this out so that someone else can benefit.
I want to know that whether the restartable sequence jobs in datastage also rerun the job activities that were aborted but not due to the sequence failure?
In the Job Properties of a Sequence, in the 'General' Tab, you can activate "Add checkpoints so sequence is restartable on failure."
If a sequence is made restartable, the subsequent jobs are each getting a checkpoint.
When running the sequence the first time, it records which jobs have sucessfully finished. If the sequence is restarted, no matter for what reason it aborted before, the sequence will just skip the jobs that have been sucessful in the previous run.
If you need to have a job executed each time, even if it was already successful the last time, you can disable checkpinting for that job in the "Job" tab of it's Job Activity stage by checkiing the checkbox "Do not checkpoint run."
I think Justus's answer is right, but with a slight deviation:-
I found out that if a parallel job has aborted anywhere in a sequence, then it's rerun will be attempted by the controller (if the job activity is set to reset if required then run). (If we resume the sequence from Aborted/Restartable) or (Stopped/Restartable) state.
For example:-
let's say the job flow is as below:
a -> b -> c -> d -> e
if sequence is run and
job 'a' is finished,
job 'b' is aborted,
job 'c' is aborted, and the sequence stops due to user intervention,
Then if checkpointing is enabled earlier, our sequence, when restarted, will ignore job 'a' (already checkpointed) and rerun job 'b' and then job 'c' and the flow will resume.
There is a BQ table which has multiple data load/update/delete jobs scheduled in. Since this is automated jobs many of it are failing due to concurrent update issue.
I need to know if we have a provision in BigQuery to check if the table is already locked by DML operation and can we serialize the queries so that no job fails
You could use the job ID generated by client code to keep track of the job status and only begin the next query when that job is done. This and this describe that process.
Alternatively you could try exponential backoff to retry the query a certain amount of times to prevent automatic failure of the query due to locked tables.
In my application which is implemented using SoftwareAG Webmethods, I'm using JMS to process more than 2 millions of records in parallel, so basically each JMS thread will have a batch of records (lets's x1000) to process and then insert into database table (let's call it table A) and after each thread inserts each batch they will send result message on JMS which I will aggregate later to update the process status.
The problem I'm facing now is that the thread will process its batch, insert and put the result message on JMS queue but the insert transactions will get queued in the mssql database but it doesn't wait in the application itself. it considers it as done and continues with the next line of logic.
Therefore the process on each thread is completed and the main process is marked as completed while there are a lot of records still waiting to get inserted into the database yet.
so my question is that is there any trigger in mssql that can be used for when the queued transactions on a table are finished?
I suggest you instead of INSERT batches, use batches that will create a jobs with two steps. First step is insertion of data and second insert data about batch complete in some results table. After that you can check table with results.
I'm going to make up some sql here. What I want is something like the following:
select ... for update priority 2; // Session 2
So when I run in another session
select ... for update priority 1; // Session 1
It immediately returns, and throws an error in session 2 (and hence does a rollback), and locks the row in session 1.
Then, whilst session 1 holds the lock, running the following in session 2.
select ... for update priority 2; // Session 2
Will wait until session 1 releases the lock.
How could I implement such a scheme, as the priority x is just something I've made up. I only need something that can do two priority levels.
Also, I'm happy to hide all my logic in PL/SQL procedures, I don't need this to work for generic SQL statements.
I'm using Oracle 10g if that makes any difference.
I'm not aware of a way to interrupt an atomic process in Oracle like you're suggesting. I think the only thing you could do would be to programmaticaly break down your larger processes into smaller ones and poll some type of sentinel table. So instead of doing a single update for 1 million rows perhaps you could write a proc that would update 1k, check a jobs table (or something similar) to see if there's a higher priority process running, and if a higher priority process is running, to pause its own execution through a wait loop. This is the only thing I can think that would keep your session alive during this process.
If you truly want to abort the progress of your currently running, lower priority thread and losing your session is acceptable, then I would suggest a jobs table again that registered the SQL that was being run and the session ID that it is run on. If you run a higher priority statement it should again check the jobs table and then issue a kill command to the low priority session (http://www.oracle-base.com/articles/misc/KillingOracleSessions.php) along with inserting a record into the jobs table to note the fact that it was killed. When a higher-priority process finishes it could check the jobs table to see if it was responsible for killing anything and if so, reissue it.
That's what resource manager was implemented for.