On a celery service on CENTOS which runs a single task at a time, the termination of a task is simple:
revoke(id, terminate=True, signal='SIGINT')
However while the interrupt signal is being processed, the running task gets revoked. Then a new task - from the queue - starts on the node. This is troublesome. Two task are running at the same time on the node. The signal handling could take up to a minute.
The question is how a signal could be sent to a running task, without actually terminating the task in celery?
Or let's say is there any way to send a signal to a running task?
The assumption is user should be able to send a signal from a remote node. In other words user does not have access to list the running processes of the node.
Any other solution is welcome.
I don't understand your goal.
Are you trying to kill the worker? if so, I guess you are talking about t "Warm shutdown", so you can send the SIGTEERM to the worker's process. The running task will get a chance to finish but no new task will be added.
If you're just interested in revoking a specific task and keep using the same worker, can you share your celery configuration and the worker command? are you sure you're running with concurrency 1 ?
Related
I am running Celery with RabbitMQ backend.
Somehow I have ended up with what appears to be a zombie Celery worker. I see the worker in Flower, and in commands like celery inspect scheduled. But it references a PID that doesn't exist. There is no worker process. It is a big problem because Celery will delegate tasks to this worker, and they never get executed.
I believe what happened is the docker container within which this is running got shut down uncleanly. But now, even if I restart the docker container, this zombie worker always comes back. Always has the same name: celery#0357c65d991b.
The Celery docs say that to kill a worker you must send its process TERM. But I can't do that because there is no process. It's a zombie.
RabbitMQ must have a dangling reference to this worker. They only thing I could find in the RabbitMQ management interface is a queue named celery#0357c65d991b.celery.pidbox. I deleted this queue, but it simply reappeared a few seconds later.
Can anyone give me a pointer on where to look to get rid of this thing?
I'm using beeQueue for video transcoding job scheduling and processing
For now everything is fine and but I'm now facing challenge of working with distributed environment like auto scaling the amazon the instances for adding more workers to process more jobs which are pending in the queue, We scale well but need to implement a system which is fail safe, I mean in case a instance on which workers were processing the job has gone shutdown and we don't get job status or events, In that case the job which were running on that instance is gone into blackhole and can't be recovered and processed again.
What I did :
I'm looking up for ready made solution who works fail safe in distributed env.
Thanks
Is there a way for a celery job to be retried if the server where the worker is running dies? I don't just mean the sub-process that execute the job, but the entire server becomes unavailable.
I tried with RabbitMQ and Redis as brokers. In both cases, if a job is currently being processed, it is entirely forgotten. When a worker restarts, it doesn't even try to reprocess the job, and looking at Rabbit or Redis, their queues are empty. The result backend is also empty.
It looks like the worker grabs the message and assume it will put it back if the subprocess fails, but if the worker dies also, it can't put it back.
(yes, I work in an environment where this happens more than once a year, and I don't want to lose tasks)
In theory, set task_acks_late=True should do the trick. (doc)
With a Redis broker, the task will be redelivered after visibility_timeout, which defaults to one hour. (doc)
With RabbitMQ, the task is redelivered as soon as Rabbit noticed that the worker died.
So I have a celery setup using RabbitMQ as the broker and amqp as the results backend.
Sometimes, I will have tasks that go long because I misunderestimated the needed timeout, and as intended, Celery will kill the worker running the task.
The problem is that because this is a celery problem and not a task problem, my error handling that's supposed to email me from the task will not run, and I will receive no message about the failure.
Is there a way to have Celery do some error notification on it's own when it kills a task due to Celery-related errors? Like an on_timeout() function that I can create in the task? I really don't want to have the calling process do the error handling, because the timeout is already a couple hours and the process runs for about 30 seconds.
Looks like this question is from a while ago and you've probably resolved the issue, but in case not, have you checked out the CELERY_SEND_TASK_ERROR_EMAILS config setting?
Can you tell me what is happening when in celery you tell the task to retry? Will it retry in the same worker thread or it will be returned to broker which may send it elsewhere?
What will happen with tasks for retry if worker or dispatcher suddenly stop? If tasks can be lost is there some approach to avoid this? May be save each task in database and retry them if no result is received for some time?
Or may be dispatcher have it's own persistent storage? What about then if worker thread crash receiving the task or while executing it?
Can you tell me what is happening when
in celery you tell the task to retry?
Will it retry in the same worker
thread or it will be returned to
broker which may send it elsewhere?
Yes the task return to the broker (ex. Rabbit MQ) with a different estimated execution time
What will happen with tasks for retry
if worker or dispatcher suddenly stop?
If tasks can be lost is there some
approach to avoid this? May be save
each task in database and retry them
if no result is received for some
time?
Or may be dispatcher have it's own
persistent storage? What about then if
worker thread crash receiving the task
or while executing it?
Here a complete answer Retry Lost or Failed Tasks (Celery, Django and RabbitMQ)