I have 2 types of tasks, one that are generated by user and the other one that are created in huge batches. Tasks are going to separate queues ("short" and "long")
When there are tasks in one queue (ie. that huge batch), Celery is fetching only those tasks completely ignoring another queue until the whole batch is done.
Example:
send 100 slow tasks to "long" queue
send 100 small tasks to "short" queue
send 100 slow tasks to "long" queue
send 100 small tasks to "short" queue
Celery behaviour:
process 100 tasks form "long" queue
process 100 tasks form "short" queue
process 100 tasks form "long" queue
process 100 tasks form "short" queue
That is happening even when I set rate_limit for slow tasks, that goes to "long" queue. All I get is slow tasks are blocking system for longer :/
Is there a way to ensure that Celery is fetching tasks from all queues? (I'm using Celery 2.5.1 with RabbitMQ)
You can launch separate celery worker for each queue:
$ celeryd -Q short
$ celeryd -Q long
In that case each type of task will be processed independently by separate worker.
Related
With celery, I have created listeners to Redis for getting all write events to Redis. Based on the events, I will trigger celery tasks to migrate data from Redis to DB.
I'm using the eventlet pool along with concurrency of 1000. Also, I'm having 5 celery queues for processing my data.
celery -A proj worker -l info -P eventlet -c 1000 -Q event_queue,vap_queue,client_queue,group_queue,ap_queue
Here, I'm facing the problem like, the listener is able to receive all the write events from Redis and workers are able to receive tasks from the listener. But, celery workers are delaying while processing huge number of data. (For example, I will be receiving 800 tasks per 10 seconds for each queue)
I have tried by increasing concurrency to higher values, changing the pool from eventlet to gevent and prefetch multiplier to 1. Still, My workers are delaying to complete a task.
Can anyone help to solve this? I'm new to celery actually :)
Some times concurrency is not the main factor in speeding up the task consumption.
When these tasks are processed.
Infact too much concurrency can lead to many context switches and slow down things, monitor your server CPU and memory to check if they are not getting overwhelmed by the tasks and find an optimum number.
For CPU bound task I will say prefer more worker than concurrent threads and for I/O bound tasks you can have concurrent threads
I have two machines on my network running Celery workers that process tasks from a common queue (the messaging back-end is RabbitMQ).
One machine is much more powerful and processes the tasks faster (which is important). If there is only one task in the queue, I always want it to run on this machine. If the queue is full, I want the less powerful machine to start accepting tasks as well.
Is there a recommended, elegant way to do this? Or do I have to set up two queues ("fast" and "slow") and implement some kind of router that sends tasks to the "slow" queue only when the "fast" queue is full?
i am new to celery and redis.
I started up my redis server by using redis-server.
Celery was run using this parameter
celery -A proj worker
There are no other configurations. However, i realised that when i have a long running job in celery, it does not process another task that is in the queue until the long running task is completed. My understanding is that since i have 8 cores on my CPU, i should be able to process 8 tasks concurrently since the default parameter for -c is the number of cores?
Am i missing something here ?
Your problem is classical, everybode met this who had long-running tasks.
The root cause is that celery tries to optimize your execution flow reserving some tasks for each worker. But if one of these tasks is long-running the others get locked. It is known as 'prefetch count'. This is because by default celery set up for short tasks.
Another related setting is a 'late ack'. By default worker takes a task from the queue and immediately sends an 'acknowledge' signal, then broker removes this task from the queue. But this means that more messages will be prefetched for this worker. 'late ack' enabled tells worker to send acknowledge only after the task is completed.
This is just in two words. You may read more about prefetch and late ack.
As for the solution - just use these settings (celery 4.x):
task_acks_late = True
worker_prefetch_multiplier = 1
or for previous versions (2.x - 3.x):
CELERY_ACKS_LATE = True
CELERYD_PREFETCH_MULTIPLIER = 1
Also, starting the worker with parameter -Ofair does the same.
I'm new to Erlang. As far as I know:
A VM has one or more (SMP mode) schedulers.
A scheduler has exactly one message run queue
Also:
A process has an 'internal' mail box, for incomming messages
Since a private mailbox seems to be local to a process, how does it relate to the 'global' scheduler run queue? Are they the same, e.g. is a mail box a small subset of the scheduler run queue? If not, how do they relate to each other?
Thanks
A process has a message queue. A scheduler has process run queues, not message run queues. Schedulers run processes, and a process receives messages from its message queue.
The scenario (I've simplified things):
Many end users can start jobs (heavy jobs, like rendering a big PDF for example), from a front end web application (producer).
The jobs are sent to a single durable RabbitMQ queue.
Many worker applications (consumers) processes those jobs and write the results back in a datastore.
This fairly standard pattern is working fine.
The problem: if a user starts 10 jobs in the same minute, and only 10 worker applications are up at that time of day, this end user is effectively taking over all the compute time for himself.
The question: How can I make sure only one job per end user is processed at any time ? (Bonus: some end users (admins for example) must not be throttled)
Also, I do not want the front end application to block end users from starting concurrent jobs. I just want the end users to wait for their concurrent jobs to finish one at a time.
The solution?: Should I dynamically create one auto-delete exclusive queue per end users ? If yes, how can I tell the worker applications to start consuming this queue ? How to ensure one (and only one) worker will consume from this queue ?
You would need to build something yourself to implement this as Dimos says. Here is an alternative implementation which requires an extra queue and some persistent storage.
As well as the existing queue for jobs, create a "processable job queue". Only jobs that satisfy your business rules are added to this queue.
Create a consumer (named "Limiter") for the job queue. The Limiter also needs persistent storage (e.g. Redis or a relational database) to record which jobs are currently processing. The limiter reads from the job queue and writes to the processable job queue.
When a worker application finishes processing a job, it adds a "job finished" event to the job queue.
------------ ------------ -----------
| Producer | -> () job queue ) -> | Limiter |
------------ ------------ -----------
^ |
| V
| ------------------------
| () processable job queue )
job finished | ------------------------
| |
| V
| ------------------------
\-----| Job Processors (x10) |
------------------------
The logic for the limiter is as follows:
When a job message is received, check the persistent storage to see if a job is already running for the current user:
If not, record the job in the storage as running and add the job message to the processable job queue.
If an existing job is running, record the job in the storage as a pending job.
If the job is for an admin user, always add it to the processable job queue.
When a "job finished" message is received, remove that job from the "running jobs" list in the persistent storage. Then check the storage for a pending job for that user:
If a job is found, change the status of that job from pending to running and add it to the processable job queue.
Otherwise, do nothing.
Only one instance of the limiter process can run at a time. This could be achieved either by only starting a single instance of the limiter process, or by using locking mechanisms in the persistent storage.
It's fairly heavyweight, but you can always inspect the persistent storage if you need to see what's going on.
Such a feature is not provided natively by rabbitMQ.
However, you could implement it in the following way. You will have to use polling though, which is not so efficient (compared to subscribing/publishing). You will also have to leverage Zookeeper for the coordination between the different workers.
You will create 2 queues: 1 high-priority queue (for the admin jobs) and 1 low-priority queue (for the normal user jobs). The 10 workers will be retrieving messages from both queues. Each worker will execute an infinite loop (with intervals of sleep ideally, when queues are empty), where it will attempt to retrieve a message from each queue interchangeably :
For the high-priority queue, the worker just retrieves a message, processes it and acknowledges to the queue.
For the low-priority queue, the worker attempts to hold a lock in Zookeeper (by writing to a specific file-znode), and if successful, then reads a message, processes it and acknowledges. If the zookeeper write was unsuccessful, someone else holds the lock, so this worker skips this step and repeats the loop.