We are not able to see the progress while Salesforce job(bulk-api) is being processing. Now We're exporting 300.000 tasks and the job is there for 4 days, however we can not see any progress on it. Is there a way we can see the progress? We need to know when it's going to be finished.
A job by itself doesn't do any work. It is batches that are queued to jobs that actually carry on data modifications. An open job will stay open until closed or until timed out (after 1 week if I remember correctly). The open status therefore does not signify progress, it only means you can queue more batches to this job.
As you can see on your second screenshot, no batches were queued to this job. Check the code that actually queues the batches, the API probably returns some kind of error there.
Related
I've an application that sends redid jobs output to front-end via socket. There are some special type nodes those are special & needs to wait to finish their jobs.So I'm using infinite waiting to finish those jobs to complete. Something like below.
While True:
If job.status() == "finished" :
Break
I want to if there's any way to free the resource & re run the jobs from their previous state.
Tried a solution to stay awake all the time. I want a solution where the resources are not bound to one jobs , the system can utilize their resources to other jobs
Well what you can do is,
To return if the job is special. And save the states of jobs in Redis Environment.
If u have a back end in your application, you can always check if the special jobs are finished running.
I want to know if there is any way to have 2 "manual jobs", in the same stage, and if one is triggered, the second is canceled.
Basically, what I want to do is to have 2 "manual jobs", one for continue my pipeline, and if it is triggered, the pipeline continue and the second "manual job" is canceled.
Or if the second manual job is the triggered, the first manual job is canceled and the pipeline stop.
I tried many thing but it doesn't seem to work and I didn't find topic about this kind of problematic.
I believe you can use Directed Acyclic Graph to make sure job 2 and job 3 do not start until manual job 1 finishes successfully, but as far as I know there is no way to easily cancel one job from another.
You could try using Jobs API, but I'm not sure how to get ID of manual job 1 from manual job 2 and vice versa.
Cancelling the entire pipeline would be easy. All you need for that is to use predefined variable CI_PIPELINE_ID (or CI_PIPELINE_IID - I'm not sure which would be the right one) and Pipeline API.
Edit: I suppose, knowing the Pipeline ID, you could get all the jobs for the pipeline with Jobs API and then parse the JSON into a map of Job names to IDs and finally use this map to cancel all jobs you want canceled.
Whenever I execute a query in Bigquery, I can see that in the Explanation tab the Waiting time is always average. Is it possible to execute without wait time or to reduce the wait time.
This image shows the query explanation (Bigquery wait time is average here)
Why Big Query always has waiting time?
You might have more work than can be immediately scheduled.
is it possible to execute without waiting time?
Purchase more BigQuery Slots.
Contact your sales representative for more information or support.
Otherwise, just wait. If your job isn’t time-critical, you can schedule it and wait for it to be executed as resources permit.
Bigquery has a complex scheduling system. Besides, usually there are couple stages for a simple query, and for each stage scheduler needs to find the best shard(node) which could execute the computation.
It takes some time for the scheduler to send the job to shards. But usually it should not be significantly long.
I am trying out RabbitMQ with springboot. I have a main process and within that process I am creating many number of small tasks that can be processed from other workers. From the main process perspective, I like to know when all of these tasks are completed so that it can move to next step. I did not find a easy way to query rabbitmq if the tasks are complete.
One solution I can think of is to store these tasks in a database and when each message is completed, update the database with COMPLETE status. Once all jobs are in COMPLETE status, the main process can know the jobs are COMPLETE and it can move to next step o fits process.
Another solution I can think of is that the main process maintain the list of jobs that is being sent to other workers. Once each worker completes it's job, it can send a message to the main process indicating the job is complete. Then the Main process can mark the job is complete and remove the item from the list.Once the list is empty, the main process will know the jobs are complete and it can move to next step of it's work.
I am looking to learn best practice on how other people have dealt this kind of situation. I appreciate for any suggestion.
Thank you!
There is no way to query RabbitMQ for this information.
The best way to approach this is with the use of a process manager.
The basic idea is to have your individual steps send a message back to a central process that keeps track of which steps are done. When that main process receives notice that all of the steps are done, it lets the system move on to the next thing.
The details of this approach are fairly complex, but I do have a blog post that covers the core of a process manager from a JavaScript/NodeJS perspective.
You should be able to find something like a "process manager" or "saga" as they are sometimes called, within your language and RabbitMQ framework of choice. If not, you should be able to write one for your process without too much trouble, as described in my blog post.
I use hangfire to run recurring jobs.
My job get current data from the database, perform an action and leave a trace on records processed. If a job didn't run this minute - I have no need to run it twice the next minute.
Somehow I got my recurring jobs (1 minute cycle) queued by their thousands and never executed. When I restarted my IIS it tried to execute them all at once and clog the DB.
Besides than fixing the problem of no execution, is there a way to stop them from queuing up?
If you want to disable retry of failed job simply decorate your method with an AutomaticRetryAttribute and set Attempts to 0
See https://github.com/HangfireIO/Hangfire/blob/master/src/Hangfire.Core/AutomaticRetryAttribute.cs for more details