I'm adding a backend for Celery results, and I'm having an issue where I send tasks, and some are accepted while others aren't.
Tasks that are and aren't executed both show this log output:
[2014-06-09 15:50:59,091: INFO/MainProcess] Received task: tasks.multithread_device_listing[e3ae6d12-ad4b-4114-9383-5802c91541f2]
Ones that ARE executed then show this output:
[2014-06-09 15:50:59,093: DEBUG/MainProcess] Task accepted: tasks.multithread_device_listing[e3ae6d12-ad4b-4114-9383-5802c91541f2] pid:2810
While tasks that AREN'T executed never arrive at the above line.
How I send tasks:
from celery import group
from time import sleep
signatures = []
signature = some_method_with_task_decorator.subtask()
signatures.append(signature)
signature = some_other_method_with_task_decorator.subtask()
signatures.append(signature)
job = group(signatures)
result = job.apply_async()
while not result.ready():
sleep(60)
My celery config from having it report it is:
software -> celery:3.1.11 (Cipater) kombu:3.0.18 py:2.7.5
billiard:3.3.0.17 py-amqp:1.4.5
platform -> system:Darwin arch:64bit imp:CPython
loader -> celery.loaders.app.AppLoader
settings -> transport:amqp results:amqp://username:pass#localhost:5672/automated_reports
CELERY_QUEUES:
(<unbound Queue automated_reports -> <unbound Exchange default(direct)> -> automated_reports>,)
CELERY_DEFAULT_ROUTING_KEY: '********'
CELERY_INCLUDE:
('celery.app.builtins',
'automated_reports.queue.tasks',
'automated_reports.queue.subtasks')
CELERY_IMPORTS:
('automated_reports.queue.tasks', 'automated_reports.queue.subtasks')
CELERY_RESULT_PERSISTENT: True
CELERY_ROUTES: {
'automated_reports.queue.tasks.run_device_info_report': { 'queue': 'automated_reports'},
'uploader.queue.subtasks.multithread_device_listing': { 'queue': 'automated_reports'},
'uploader.queue.subtasks.multithread_individual_device': { 'queue': 'automated_reports'},
'uploader.queue.tasks.multithread_device_listing': { 'queue': 'automated_reports'},
'uploader.queue.tasks.multithread_individual_device': { 'queue': 'automated_reports'}}
CELERY_DEFAULT_QUEUE: 'automated_reports'
BROKER_URL: 'amqp://username:********#localhost:5672/automated_reports'
CELERY_RESULT_BACKEND: 'amqp://username:pass#localhost:5672/automated_reports'
My startup command is:
~/Documents/Development/automated_reports/bin/celery worker --loglevel=DEBUG --autoreload -A automated_reports.queue.tasks -Q automated_reports -B --schedule=~/Documents/Development/automated_reports/log/celerybeat --autoscale=10,3
Also, when I stop celery, it pulls tasks out of my queue that were never accepted. Then when I restart, it accepts them and executes them.
Any help with this behavior is much appreciated. I'm certain it has something to do with my backend configuration, but I'm having difficulty isolating the issue or its correction. Thanks!
I found the answer to this.
I noticed that the 'inqueue' seemed to be properly receiving tasks in some cases, but not others. When I searched the Celery docs, I found this note: http://celery.readthedocs.org/en/latest/whatsnew-3.1.html?highlight=inqueue#caveats
I was executing the subtasks from within a long-running task, so this sounded very much like the behavior I was seeing. Also, I'm on the version mentioned, whereas on previous versions I hadn't had this problem with the same config.
I added the -Ofair parameter to starting the worker, and it immediately resolved the issue.
Related
I have implemented Celery with RabbitMQ as Broker. I rely on Celery v4.4.7 since I have read that v5.0+ doesn't support RabbitMQ anymore. RabbitMQ is a MUST in my case.
Everything has been containerized then deployed as pods within Kubernetes 1.19. I am able to execute long running tasks and everything apparently looks fine at first glance. However, I have few concerns which require your expertise.
I have declared inbound and outbound queues but Celery created his owns and I do not see any message within those queues (inbound or outbound) :
inbound_queue = "_IN"
outbound_queue = "_OUT"
app = Celery()
app.conf.update(
broker_url = 'pyamqp://%s//' % path,
broker_heartbeat = None,
broker_connection_timeout = int(timeout)
result_backend = 'rpc://',
result_persistent = True,
task_queues = (
Queue(algorithm_queue, Exchange(inbound_queue), routing_key='default', auto_delete=False),
Queue(result_queue, Exchange(outbound_queue), routing_key='default', auto_delete=False),
),
task_default_queue = inbound_queue,
task_default_exchange = inbound_exchange,
task_default_exchange_type = 'direct',
task_default_routing_key = 'default',
)
#app.task(bind=True,
name='osmq.tasks.add',
queue=inbound_queue,
reply_to = outbound_queue,
autoretry_for=(Exception,),
retry_kwargs={'max_retries': 5, 'countdown': 2})
def execute(self, data):
<method_implementation>
I have implemented callbacks to get results back via REST APIs. However, randomly, it can return or not some results when the status is successfull. This is probably related to message persistency. In details, when I implement flower API to get info, status is successfull and the result is partially displayed (shortened json messages) - when I call AsyncResult, for the same status, result is either None or the right one. I do not understand the mechanism between rabbitmq queues and kombu which seems to cache the resulting message. I must guarantee to retrieve results everytime the task has been successfully executed.
def callback(uuid):
task = app.AsyncResult(uuid)
Specifically, it was that Celery 5.0+ did not support amqp:// as a result back end anymore. However, as your example, rpc:// is supported.
The relevant snippet is here: https://docs.celeryproject.org/en/stable/getting-started/backends-and-brokers/index.html#rabbitmq
We tend to always ignore_results=True in our implementation, so I can't give any practical tips of how to use rpc://, other than to infer that any response is put on an application-specific queue, instead of being able to put on a specified queue (or even different broker / rabbitmq instance) via amqp://.
Consider a nextflow workflow that uses the Google Life Sciences API and uses preemptible machines. The config might look like this:
google {
project = "cool-name"
region = "cool-region"
lifeSciences {
bootDiskSize = "200 GB"
preemptible = True
}
}
Let's say you have only a single process and this process has the directive maxRetries = 5. After five retries (=after the 6th time), the process will be considered failed.
Is it somehow possible to specify in nextflow that after a certain number of retries were unsuccessful, that nextflow should request a non-preemptible machine instead and continue retrying a couple more times?
I am using node-celery github link to implement celery celery first step along with rabbitmq.
As in celery we define task and then we push them.
My task which I have defined inside tasks.py is as below:
import os
import logging
from celery import Celery
import requests
backend = os.getenv('CELERY_BACKEND_URL', 'amqp')
celery = Celery('tasks', backend=backend)
celery.conf.update(
CELERY_RESULT_SERIALIZER='json',
CELERY_ENABLE_UTC=True
)
#celery.task
def getOrders():
requests.get('locahost:4000/getOrders')
My file which I run to trigger tasks:
eta.js:
var celery = require('../celery'),
client = celery.createClient({
CELERY_BROKER_URL: 'amqp://guest:guest#localhost:5672//'
});
client.on('error', function(err) {
console.log(err);
});
client.on('connect', function() {
client.call('tasks.getOrders', {
eta: new Date(Date.now() + 15 * 1000) // an hour later
});
});
I start celery worker by using below command:
celery worker -A tasks -l info
Now when I run eta.js and my task 'getOrders' defined inside tasks.py gets triggers and it further hits the url I have requested and that URL does its work.
To run eta.js I run:
node eta.js
What I want is that my task 'getOrders' keeps running after every x seconds.
I have read about setting periodic tasks in celery periodic tasks celery but it is in python and I need this in node-celery.
I want to use celery beat in node-celery if it is possible or should I just avoid using node celery? and use celery python setup, I know python but couldn't find any link to setup celery in python.
Anyone having good knowledge about celery can help me or guide me to follow some tutorial.
thanks!
You can add the scheduled task to Redis directly so Redbeat will read this new task and it will execute it and reschedule it according to the crontab setup.
To do this:
Add the redbead redis url in your celery config
Create a periodic task
const task = {
name : "getOrdersTask",
task : "tasks.getOrders",
schedule: {
"__type__": "crontab",
minute : "*/5",
hour : "*",
day_of_week :
day_of_month : "*/7"
month_of_year : "[1-12]"
},
args : [],
enabled : true,
}
Store the task in redis with:
redisClient.hset("redbeat:getOrdersTask", "definition", JSON.stringify(task));
Insert the task into the tasks list (the 0 is to force Redbeat to execute it immediately so it will be rescheduled in the proper time according to the period config you selected, for this case every 5 minutes)
redisClient.zadd("redbeat::schedule", 0, "getOrdersTask")
I am using celery with a rabbitmq backend. It is producing thousands of queues with 0 or 1 items in them in rabbitmq like this:
$ sudo rabbitmqctl list_queues
Listing queues ...
c2e9b4beefc7468ea7c9005009a57e1d 1
1162a89dd72840b19fbe9151c63a4eaa 0
07638a97896744a190f8131c3ba063de 0
b34f8d6d7402408c92c77ff93cdd7cf8 1
f388839917ff4afa9338ef81c28aad75 0
8b898d0c7c7e4be4aa8007b38ccc00ea 1
3fb4be51aaaa4ac097af535301084b01 1
This seems to be inefficient, but further I have observed that these queues persist long after processing is finished.
I have found the task that appears to be doing this:
#celery.task(ignore_result=True)
def write_pages(page_generator):
g = group(render_page.s(page) for page in page_generator)
res = g.apply_async()
for rendered_page in res:
print rendered_page # TODO: print to file
It seems that because these tasks are being called in a group, they are being thrown into the queue but never being released. However, I am clearly consuming the results (as I can view them being printed when I iterate through res. So, I do not understand why those tasks are persisting in the queue.
Additionally, I am wondering if the large number queues that are being created is some indication that I am doing something wrong.
Thanks for any help with this!
Celery with the AMQP backend will store task tombstones (results) in an AMQP queue named with the task ID that produced the result. These queues will persist even after the results are drained.
A couple recommendations:
Apply ignore_result=True to every task you can. Don't depend on results from other tasks.
Switch to a different backend (perhaps Redis -- it's more efficient anyway): http://docs.celeryproject.org/en/latest/userguide/tasks.html
Use CELERY_TASK_RESULT_EXPIRES (or on 4.1 CELERY_RESULT_EXPIRES) to have a periodic cleanup task remove old data from rabbitmq.
http://docs.celeryproject.org/en/master/userguide/configuration.html#std:setting-result_expires
First of all please don't consider this question as a duplicate of this question
I have a setup an environment which uses celery and redis as broker and result_backend. My question is how can I make sure that when the celery workers crash, all the scheduled tasks are re-tried, when the celery worker is back up.
I have seen advice on using CELERY_ACKS_LATE = True , so that the broker will re-drive the tasks until it get an ACK, but in my case its not working. Whenever I schedule a task its immediately goes to the worker which persists it until the scheduled time of execution. Let me give some example:
I am scheduling a task like this: res=test_task.apply_async(countdown=600) , but immediately in celery worker logs i can see something like : Got task from broker: test_task[a137c44e-b08e-4569-8677-f84070873fc0] eta:[2013-01-...] . Now when I kill the celery worker, these scheduled tasks are lost. My settings:
BROKER_URL = "redis://localhost:6379/0"
CELERY_ALWAYS_EAGER = False
CELERY_RESULT_BACKEND = "redis://localhost:6379/0"
CELERY_ACKS_LATE = True
Apparently this is how celery behaves.
When worker is abruptly killed (but dispatching process isn't), the message will be considered as 'failed' even though you have acks_late=True
Motivation (to my understanding) is that if consumer was killed by OS due to out-of-mem, there is no point in redelivering the same task.
You may see the exact issue here: https://github.com/celery/celery/issues/1628
I actually disagree with this behaviour. IMO it would make more sense not to acknowledge.
I've had the issue, where I was using some open-source C libraries that went totaly amok and crashed my worker ungraceful without throwing an exception. For any reason whatsoever, one can simply wrap the content of a task in a child process and check its status in the parent.
n = os.fork()
if n > 0: //inside the parent process
status = os.wait() //wait until child terminates
print("Signal number that killed the child process:", status[1])
if status[1] > 0: // if the signal was something other then graceful
// here one can do whatever they want, like restart or throw an Exception.
self.retry(exc=SomeException(), countdown=2 ** self.request.retries)
else: // here comes the actual task content with its respected return
return myResult // Make sure there are not returns in child and parent at the same time.