I am using Celery to execute sending mail task.
Sending one email corresponds to one task.
The SMTP server I am using takes only 24 send mail request per second. If it exceeds it limit error is thrown.
How can I control my worker to send only 24 (or x) task per second.
Celery lets you control that via the rate limits
Related
I have a pipeline in which ingest data from an API and load them into an azure database. The pipeline is called by a trigger. The load time is normally 6 to 7 hours. But sometimes for some reason, the pipeline runs more than 24 hours and on the next day again is executed by the trigger. So I want to stop the pipeline, if pipeline it runs more than 24 hours. Appreciate any help.
In Azure Pipeline, set Timeout for agent job would achieve your demand. Each job has a timeout. If the job has not completed in the specified time, the server will cancel the job. It will attempt to signal the agent to stop, and it will mark the job as canceled: https://learn.microsoft.com/en-us/azure/devops/pipelines/process/runs?view=azure-devops#timeouts-and-disconnects
Set 1440 minutes for 24 hours.
So I'm running some code which takes about 2 hours to run on the cluster. I configured the batch file with
# Set maximum wallclock time limit for this job
#Time Format = days-hours:minutes:seconds
#SBATCH --time=0-02:15:00
Just to give some overhead if the job slows for whatever reason. I checked the directory that the generated files are stored in and the simulation completes successfully every time. Despite this, slurm keeps the job running until it hits the max time. The .out file keeps saying
slurmstepd: *** JOB CANCELLED AT 2022-03-05T10:38:26 DUE TO TIME LIMIT ***
Any ideas why it doesn't show as complete instead?
In my opinion, this error is not related to Slurm rather about your application. Your application is somehow not sending the exit signal to the slurm.
You can use sstat -j jobid to see the status of the job, may be after 2 hours to see how the cpu consumption etc going and figure out what happens in your application (where it hangs after completion or so).
I have already enabled "Service Broker" in my database. But recently I noticed that my database size is increasing day by day unexpectedly. Finally I've found out that my database size increase unexpectedly because of queue message of internal table.
The C drive is almost full, that's why I could not delete queue message from internal table. When I execute query for delete query then log file size increase accordingly, that's why I want to delete queue message without storing log.
Thanks.
I am performing a batch operation using batch scope in mule 4. I am using a Sfdc connector inside my batch step to query the Ids. The batch is happening in sets of say 200 and total 1000 inputs. I need to get all the 1000 Ids from Sfdc query outside my batch scope. I tried few ways to access the payload coming out of the batch step but failed to get the payload outside scope. I see that the payload is accessible only inside the process stage. Are there any ways to achieve this?
I found the solution. Its so easy to use:)
Set "-n mynode#%h" when start celery
and set the destination option like add_consumer
(queue=myqueue, destination=['mynode#hostname']).
Thanks.
=========================================
I use flask, celery, rabbitmq.
I want to divide server A and B by purpose of messages with same rabbitmq.
0. workers are always running on two servers(A, B) with this options.
celery -A myapp.mycelery worker -E --logfile=celery.log --pidfile=celery.pid
1. Make queue dynamically with CELERY_CREATE_MISSING_QUEUES=True option.
2. add consumer to run of newly maked queue.
mycelery.control.add_consumer(new_queue_name, reply=False)
3. Server A, B have 8 workers each. (total 16 workers)
4. add messages.
my_task.apply_async((params), queue=new_queue_name)
5. 16 workers process my_task messages concurrently very well.
But I want to divide server A and B by purpose of messages with same rabbitmq.
For example 1)
my_task_A.apply_async() -> run on server A only.
my_task_B.apply_async() -> run on server B only.
For example 2)
my_task.apply_async(queue=new_queue_name_startswith_A) -> run on server A only.
my_task.apply_async(queue=new_queue_name_startswith_B) -> run on server B only.
The queue what I make dynamically is direct exchange.
(One message must be in one queue)
And I can't increase number of rabbitmq servers.
Please let me know good way to do. or what term I need to study?