how can i get result of periodic task scheduling - redis

hey guys i am new to celery. i am working on periodic task scheduling. I have configured my celeryconfig.py as follow:
from datetime import timedelta
BROKER_URL = 'redis://localhost:6379/0'
CELERY_RESULT_BACKEND = "redis"
CELERY_REDIS_HOST = "localhost"
CELERY_REDIS_PORT = 6379
CELERY_REDIS_DB = 0
CELERY_IMPORTS=("mytasks")
CELERYBEAT_SCHEDULE={'runs-every-60-seconds' :
{
'task': 'mytasks.add',
'schedule': timedelta(seconds=60),
'args':(16,16)
},
}
and mytask.pyas follow:
from celery import Celery
celery = Celery("tasks",
broker='redis://localhost:6379/0',
backend='redis')
#celery.task
def add(x,y):
return x+y
#celery.task
def mul(x,y):
return x*y
when i am running
celery beat -s celerybeat-schedule then i am getting
Configuration ->
. broker -> redis://localhost:6379/0
. loader -> celery.loaders.default.Loader
. scheduler -> celery.beat.PersistentScheduler
. db -> celerybeat-schedule
. logfile -> [stderr]#INFO
. maxinterval -> now (0s)
[2012-08-28 12:27:17,825: INFO/MainProcess] Celerybeat: Starting...
[2012-08-28 12:28:00,041: INFO/MainProcess] Scheduler: Sending due task mytasks.add
[2012-08-28 12:29:00,057: INFO/MainProcess] Scheduler: Sending due task mytasks.add
[2012-08-28 12:30:00,064: INFO/MainProcess] Scheduler: Sending due task mytasks.add
[2012-08-28 12:31:00,097: INFO/MainProcess] Scheduler: Sending due task mytasks.add
now i am not getting that i have passed arguments (16,16) then how i can get the answer of this function add(x,y)

I'm not sure I quite understand what you have asked, but from what I can tell, your issue may be one of the following:
1) Are you running celeryd (the worker daemon)? If not, did you start a celery worker in a terminal? Celery beat is a task scheduler. It is not a worker. Celerybeat only schedules the tasks (i.e. places them in a queue for a worker to eventually consume).
2) How did you plan on viewing the results? Are they being saved somewhere? Since you have set your results backend to redis, the results are at least temporarily stored in the redis results backend

Related

Celery tasks from different applications in different log files

I'm looking for configure Celery on my FreeBSD server and I get some issues according to log files.
My configuration:
FreeBSD server
2 Django applications : app1 and app2
Celery is daemonized and Redis
Each application has his own Celery task
My Celery config file:
I have in /etc/default/celeryd_app1 :
# Names of nodes to start
CELERYD_NODES="worker"
# Absolute or relative path to the 'celery' command:
CELERY_BIN="/usr/local/www/app1/venv/bin/celery"
# App instance to use
CELERY_APP="main"
# Where to chdir at start.
CELERYD_CHDIR="/usr/local/www/app1/src/"
# Extra command-line arguments to the worker
CELERYD_OPTS="--time-limit=300 --concurrency=8"
# Set logging level to DEBUG
#CELERYD_LOG_LEVEL="DEBUG"
# %n will be replaced with the first part of the nodename.
CELERYD_LOG_FILE="/var/log/celery/app1/%n%I.log"
CELERYD_PID_FILE="/var/run/celery/app1/%n.pid"
# Workers should run as an unprivileged user.
CELERYD_USER="celery"
CELERYD_GROUP="celery"
# If enabled pid and log directories will be created if missing,
# and owned by the userid/group configured.
CELERY_CREATE_DIRS=1
I have exactly the same file for celeryd_app2
Django settings file with Celery settings:
CELERY_BROKER_URL = 'redis://localhost:6379'
CELERY_ACCEPT_CONTENT = ['application/json']
CELERY_RESULT_BACKEND = 'redis://localhost:6379'
CELERY_TASK_SERIALIZER = 'json'
CELERY_RESULT_SERIALIZER = 'json'
CELERY_IGNORE_RESULT = False
CELERY_TASK_TRACK_STARTED = True
# Add a one-minute timeout to all Celery tasks.
CELERYD_TASK_SOFT_TIME_LIMIT = 60
Both settings have the same redis' port.
My issue:
When I execute a celery task for app1, I find logs from this task in app2 log file with an issue like this :
Received unregistered task of type 'app1.task.my_task_for_app1'
...
KeyError: 'app1.task.my_task_for_app1'
There is an issue in my Celery config file ? I have to set different redis port ? If yes, How I can do that ?
Thank you very much
I guess the problem lies in the fact that you are using the same Redis database for both applications:
CELERY_BROKER_URL = 'redis://localhost:6379'
Take a look into the guide for using Redis as a broker. Just change the database for each application, e.g.
CELERY_BROKER_URL = 'redis://localhost:6379/0'
and
CELERY_BROKER_URL = 'redis://localhost:6379/1'

send_task works only with a specific user

setup: Celery 4.1, RabbitMQ 3.6.1 (As broker), Redis (As backend, not relevant here).
Having two rabbit users:
admin_user with permissions of .* .* .*.
remote_user with permissions of ack ack ack.
admin_user can trigger tasks and is used by celery workers to handle tasks.
remote_user can only trigger one type of task - ack and is enqueued in a dedicated ack queue which later on being consumed by ack worker (by admin_user).
The remote_user sends the task by the following code:
from celery import Celery
app = Celery('remote', broker='amqp://remote_user:remote_pass#<machine_ip>:5672/vhost')
app.send_task('ack', args=('a1', 'a2'), queue='ack', route_name='ack')
This works perfectly in Celery 3.1. After upgrade to Celery 4.1 it doesn't send the task anymore. The call returns an AsyncResult but I don't see the message in Celery flower (or via rabbit management ui), or in the logs.
Trying to set permissions to remote_user .* .* .* as in the admin_user - doesn't help.
Trying to add administrator tag - doesn't help.
Changing the user of the broker to
'amqp://admin_user:admin_pass#<machine_ip>:5672/vhost' DOES work :
from celery import Celery
app = Celery('remote', broker='amqp://admin_user:admin_pass#<machine_ip>:5672/vhost')
app.send_task('ack', args=('a1', 'a2'), queue='ack', route_name='ack')
But I don't want to give a remote machine the admin_user permissions.
Any idea what I can do?
Solved,
API changed I guess, but to stay with the current permissions of RabbitMQ I had to use the following route:
old_celery_config.py: (celery 3.1)
CELERY_ROUTES = {
'ack_task': {
'queue': 'geo_ack'
}
}
celery_config.py: (celery 4.1)
CELERY_ROUTES = {
'ack_task': {
'exchange': 'ack',
'exchange_type': 'direct',
'routing_key': 'ack'
}
}
run_task.py:
from celery import Celery
app = Celery('remote', broker='amqp://remote_user:remote_pass#<machine_ip>:5672/vhost')
app.config_from_object('celery_config')
app.send_task('ack_task', args=('a1', 'a2'))

IO thread error : 1595 (Relay log write failure: could not queue event from master)

Slave status :
Last_IO_Errno: 1595
Last_IO_Error: Relay log write failure: could not queue event from master
Last_SQL_Errno: 0
from error log :
[ERROR] Slave I/O for channel 'db12': Unexpected master's heartbeat data: heartbeat is not compatible with local info; the event's data: log_file_name toku10-bin.000063<D1> log_pos 97223067, Error_code: 1623
[ERROR] Slave I/O for channel 'db12': Relay log write failure: could not queue event from master, Error_code: 1595
I tried to restarting the slave_io thread for many times, still its same.
we need to keep on start io_thread whenever it stopped manually, hope its bug from percona
I have simply written shell and scheduled the same for every 10mins to check if io_thread is not running , start slave io_thread for channel 'db12';. It's working as of now

Celery workers stalled on boot

We boot up a cluster of 250 worker nodes in AWS at night to handle some long-running distributed tasks.
The worker nodes are running celery with the following command:
celery -A celery_worker worker --concurrency=1 -l info -n background_cluster.i-1b1a0dbb --without-heartbeat --without-gossip --without-mingle -- celeryd.prefetch_multiplier=1
We are using rabbitmq as our broker, and there is only 1 rabbitmq node.
About 60% of our nodes claim to be listening, but will not pick up any tasks.
Their logs look like this:
-------------- celery#background_cluster.i-1b1a0dbb v3.1.18 (Cipater)
---- **** -----
--- * *** * -- Linux-3.2.0-25-virtual-x86_64-with-Ubuntu-14.04-trusty
-- * - **** ---
- ** ---------- [config]
- ** ---------- .> app: celery_worker:0x7f10c2235cd0
- ** ---------- .> transport: amqp://guest:**#localhost:5672//
- ** ---------- .> results: disabled
- *** --- * --- .> concurrency: 1 (prefork)
-- ******* ----
--- ***** ----- [queues]
-------------- .> background_cluster exchange=root(direct) key=background_cluster
[tasks]
. more.celery_worker.background_cluster
[2015-10-10 00:20:17,110: WARNING/MainProcess] celery#background_cluster.i-1b1a0dbb
[2015-10-10 00:20:17,110: WARNING/MainProcess] consuming from
[2015-10-10 00:20:17,110: WARNING/MainProcess] {'background_cluster': <unbound Queue background_cluster -> <unbound Exchange root(direct)> -> background_cluster>}
[2015-10-10 00:20:17,123: INFO/MainProcess] Connected to amqp://our_server:**#10.0.11.136:5672/our_server
[2015-10-10 00:20:17,144: WARNING/MainProcess] celery#background_cluster.i-1b1a0dbb ready.
However, rabbitmq shows that there are messages waiting in the queue.
If I login to any of the worker nodes and issue this command:
celery -A celery_worker inspect active
...then every (previously stalled) worker node immediately grabs a task and starts cranking.
Any ideas as to why?
Might it be related to these switches?
--without-heartbeat --without-gossip --without-mingle
It turns out that this was a bug in celery where using --without-gossip kept events from draining. Celery's implementation of gossip is pretty new, and it apparently implicitly takes care of draining events, but when you turn it off things get a little wonky.
The details to the issue are outlined in this github issue: https://github.com/celery/celery/issues/1847
Master currently has the fix in this PR: https://github.com/celery/celery/pull/2823
So you can solve this one of three ways:
Use gossip (remove --without-gossip)
Patch your version of celery with https://github.com/celery/celery/pull/2823.patch
Use a cron job to run a celery inspect active regularly

How to get detailed log/info about rabbitmq connection action?

I have a python program connecting to a rabbitmq server. When this program starts, it connects well. But when rabbitmq server restarts, my program can not reconnect to it, and leaving error just "Socket closed"(produced by kombu), which is meaningless.
I want to know the detailed info about the connection failure. On the server side, there is nothing useful in the rabbitmq log file either, it just said "connection failed" with no reason given.
I tried the trace plugin(https://www.rabbitmq.com/firehose.html), and found there was no trace info published to amq.rabbitmq.trace exchange when the connection failure happended. I enabled the plugin with:
rabbitmq-plugins enable rabbitmq_tracing
systemctl restart rabbitmq-server
rabbitmqctl trace_on
and then i wrote a client to get message from amq.rabbitmq.trace exchange:
#!/bin/env python
from kombu.connection import BrokerConnection
from kombu.messaging import Exchange, Queue, Consumer, Producer
def on_message(self, body, message):
print("RECEIVED MESSAGE: %r" % (body, ))
message.ack()
def main():
conn = BrokerConnection('amqp://admin:pass#localhost:5672//')
channel = conn.channel()
queue = Queue('debug', channel=channel,durable=False)
queue.bind_to(exchange='amq.rabbitmq.trace', routing_key='publish.amq.rabbitmq.trace')
consumer = Consumer(channel, queue)
consumer.register_callback(on_message)
consumer.consume()
while True:
conn.drain_events()
if __name__ == '__main__':
main()
I also tried to get some debug log from rabbitmq server. I reconfigured rabbitmq.config according to https://www.rabbitmq.com/configure.html, and set
log_levels to
{log_levels, [{connection, info}]}
but as a result rabbitmq server failed to start. It seems like the official doc is not for me, my rabbitmq server version is 3.3.5. However
{log_levels, [connection,debug,info,error]}
or
{log_levels, [connection,debug]}
works, but with this there is no DEBUG info showing in the logs, which i don't know whether it is because the log_levels configuration is not effective or there is just no DEBUG log got printed all the time.
I know that this answer comes massively late, but for future purveyors, this worked for me:
[
{rabbit,
[
{log_levels, [{connection, debug}, {channel, debug}]}
]
}
].
Basically, you just need to wrap the parameters you want to set in whichever module/plugin they belong to.