I'm running a Celery using Kombu Consumer, reading messages from a rabbitmq. After I finished coding and ran it well, except for having messages get lost and the consumer doesn't read them.
For example, if I send 5 messages to the queue, Kombu manages to get 5, but only processes some of them, the others I get a warning of "Received and deleted unknown message. Wrong destination?!?" and the message leaves the rabbitmq queue, even without using message.ack().
this is my code. I run with celery -A tasks worker -Q parse -l DEBUG --autoscale=2,1
import time
import celery
from celery import Celery, bootsteps
from kombu import Consumer, Queue, Exchange
from setuptools import setup
app = Celery("bob", broker='amqp://guest:guest#localhost:5672//')
parse_queue = Queue("parse", Exchange("parse"), "parse")
#app.task
def do_something():
print("I'm doing something")
class MyConsumerStep(bootsteps.ConsumerStep):
def get_consumers(self, channel):
return [
Consumer(
channel,
queues=[parse_queue],
callbacks=[
self.handle_parse
],
accept=["json"]
)
]
def handle_parse(self, body, message):
print('Received message: {0!r}'.format(body))
do_something.delay()
app.steps["consumer"].add(MyConsumerStep)
When I send a message through rabbitmq in queue parsing celery shows me this log:
[2023-02-15 11:53:27,634: INFO/MainProcess] Connected to amqp://guest:**#127.0.0.1:5672//
[2023-02-15 11:53:27,691: INFO/MainProcess] mingle: searching for neighbors
[2023-02-15 11:53:28,205: INFO/SpawnPoolWorker-1] child process 93064 calling self.run()
[2023-02-15 11:53:28,782: INFO/MainProcess] mingle: all alone
[2023-02-15 11:53:28,931: INFO/MainProcess] celery#parse_worker ready.
[2023-02-15 11:53:32,217: INFO/MainProcess] Events of group {task} enabled by remote.
[2023-02-15 11:53:56,288: WARNING/MainProcess] Received message: '{"teste": "teste"}'
[2023-02-15 12:30:23,125: WARNING/MainProcess] Received and deleted unknown message. Wrong destination?!?
The full contents of the message body was: body: '{"teste": "teste"}' (18b)
{content_type:None content_encoding:None
delivery_info:{'consumer_tag': 'None5', 'delivery_tag': 1, 'redelivered': False, 'exchange': '', 'routing_key': 'parse'} headers={}}
[2023-02-15 12:30:23,280: WARNING/MainProcess] Received message: '{"teste": "teste"}'
[2023-02-15 12:30:23,441: WARNING/MainProcess] Received and deleted unknown message. Wrong destination?!?
The full contents of the message body was: body: '{"teste": "teste"}' (18b)
{content_type:None content_encoding:None
delivery_info:{'consumer_tag': 'None5', 'delivery_tag': 2, 'redelivered': False, 'exchange': '', 'routing_key': 'parse'} headers={}}
[2023-02-15 12:30:23,599: WARNING/MainProcess] Received message: '{"teste": "teste"}'
[2023-02-15 12:30:23,763: WARNING/MainProcess] Received and deleted unknown message. Wrong destination?!?
The full contents of the message body was: body: '{"teste": "teste"}' (18b)
{content_type:None content_encoding:None
delivery_info:{'consumer_tag': 'None5', 'delivery_tag': 3, 'redelivered': False, 'exchange': '', 'routing_key': 'parse'} headers={}}
[2023-02-15 12:30:23,913: WARNING/MainProcess] Received message: '{"teste": "teste"}'
Looking through the flower, it doesn't display any worker information. I don't know if Kombu has any problem with flower integration, but I see that read messages are processed normally, even without information in flower.
What is going on?
Why are messages lost?
Why doesn't flower show any information?
Is there another way to process in parallel without using celery's --autoscale or --concurrency command?
Related
I am using amqplib to transfer messages in my node.js server.
Here is my code to listen on queue:
channel.consume(queue, handler1, { noAck: true })
Now, I want to update the consumer to listen the same queue
Like this:
channel.consume(queue, handler2, { noAck: true })
I try to unbindQueue or deleteQueue but don't know why the error Unhandled rejection IllegalOperationError: Channel closing is thrown
I want to know the cause of Rabbitmq crash which randomly occur. can you let me know what kind of causes could be considered?
Also my team should manually restart the rabbitmq when crash happens, so I want to know if there is a way to restart rabbitmq server automatically.
Here is the error report when rabbitmq crash occur:
=WARNING REPORT==== 6-Dec-2017::07:56:43 ===
closing AMQP connection <0.4387.0> (000000:23070 -> 00000:5672, vhost: '/', user: '00000'):
client unexpectedly closed TCP connection
Also this is part of sasl.gsd fild :
=SUPERVISOR REPORT==== 7-Dec-2017::10:03:15 ===
Supervisor: {local,sockjs_session_sup}
Context: child_terminated
Reason: {function_clause,
[{gen_server,cast,
[{},sockjs_closed],
[{file,"gen_server.erl"},{line,218}]},
{rabbit_ws_sockjs,service_stomp,3,
[{file,"src/rabbit_ws_sockjs.erl"},{line,150}]},
{sockjs_session,emit,2,
[{file,"src/sockjs_session.erl"},{line,173}]},
{sockjs_session,terminate,2,
[{file,"src/sockjs_session.erl"},{line,311}]},
{gen_server,try_terminate,3,
[{file,"gen_server.erl"},{line,629}]},
{gen_server,terminate,7,
[{file,"gen_server.erl"},{line,795}]},
{proc_lib,init_p_do_apply,3,
[{file,"proc_lib.erl"},{line,247}]}]}
Offender: [{pid,<0.20883.1160>},
{id,undefined},
{mfargs,
{sockjs_session,start_link,
["pd4tvvi0",
{service,"/stomp",
#Fun<rabbit_ws_sockjs.1.47892404>,{},
"//cdn.jsdelivr.net/sockjs/1.0.3/sockjs.min.js",
false,true,5000,25000,131072,
#Fun<rabbit_ws_sockjs.0.47892404>,undefined},
[{peername,{{172,31,6,213},9910}},
{sockname,{{172,31,5,49},15674}},
{path,"/stomp/744/pd4tvvi0/htmlfile"},
{headers,[]},
{socket,#Port<0.12491352>}]]}},
{restart_type,transient},
{shutdown,5000},
{child_type,worker}]
=CRASH REPORT==== 7-Dec-2017::10:03:20 ===
crasher:
initial call: sockjs_session:init/1
pid: <0.25851.1160>
registered_name: []
exception exit: {function_clause,
[{gen_server,cast,
[{},sockjs_closed],
[{file,"gen_server.erl"},{line,218}]},
{rabbit_ws_sockjs,service_stomp,3,
[{file,"src/rabbit_ws_sockjs.erl"},{line,150}]},
{sockjs_session,emit,2,
[{file,"src/sockjs_session.erl"},{line,173}]},
{sockjs_session,terminate,2,
[{file,"src/sockjs_session.erl"},{line,311}]},
{gen_server,try_terminate,3,
[{file,"gen_server.erl"},{line,629}]},
{gen_server,terminate,7,
[{file,"gen_server.erl"},{line,795}]},
{proc_lib,init_p_do_apply,3,
[{file,"proc_lib.erl"},{line,247}]}]}
in function gen_server:terminate/7 (gen_server.erl, line 800)
ancestors: [sockjs_session_sup,<0.177.0>]
messages: []
links: [<0.178.0>]
dictionary: []
trap_exit: true
status: running
heap_size: 987
stack_size: 27
reductions: 175
neighbours:
Please check out the error report I posted above and let me know the causes of rabbitmq crash and the way to automatically restart rabbitmq server.
Thanks!!
Recently , I am doing an experiment on a GIT project to understanding the big data processing framework.
1、GIT project:https://github.com/esperdyne/celery-message-processing
we have the following components:
1、AMPQ broker(RabbitMQ): it works as a message buffer, which works as a mail-box to exchange messages for different user!
2、worker: it works as the service-server to provide service for various service client.
3、Queue("celery":it works as a multi-processing container which is used to handle the various worker instances at the same time.
the key configuration can be seen as bellow:
We use the object proj/celery.py to define the app, the definition can be seen as below:
app = Celery('proj',
broker='amqp://',
backend='redis://localhost',
include=['proj.tasks'])
enter code here
when we start the app:
1、 when we start the application, we have seen the message which is produced from the rabbitmq, yet the celery could not handle the message.
Parse.log looks like this:[2017-02-04 14:28:06,909: WARNING/MainProcess] Received and deleted unknown message. Wrong destination?!?
we have the following question:
4.2.1 AMQP mechanism
We can see that the AMQP works as the message buffer, then there will be a message sender and a message fetcher:
In the above diagram , who is the message sender and who is the message fetcher.
4.2.2 Message definition
In our application , we can not find the code to define the Message to send ,or to receive form the AMQP.
4.2.3 Message monitor
How can we monitor the Message send and receive in the AMQP.
Hope a teacher will guide us to solve the problem , and give us some detailed
introduction on the celery broker mechenism!
note : the error log can be seen here
[2017-02-04 14:28:06,909: WARNING/MainProcess] Received and deleted unknown message. Wrong destination?!?
The full contents of the message body was: body: [[u'maildir/allen- p/inbox/1.'], {}, {u'errbacks': None, u'callbacks': None, u'chord': None, u'chain': [{u'chord_size': None, u'task': u'celery.group', u'args': [], u'immutable': False, u'subtask_type': u'group', u'kwargs': {u'tasks': [{u'chord_size': None, u'task': u'proj.tasks.deploy_db', u'args': [], u'options': {u'reply_to': u'3d9de118-f9d0-3bee-9972-b6a4d4482446', u'task_id': u'3cafda16-3e7c-44db-b05e-1327ef97ffc3'}, u'subtask_type': None, u'kwargs': {}, u'immutable': False}, {u'chord_size': None, u'task': u'proj.tasks.deploy_es', u'args': [], u'options': {u'reply_to': u'3d9de118-f9d0-3bee-9972-b6a4d4482446', u'task_id': u'1f4c728b-680d-4dde-98b9-b153d5282780'}, u'subtask_type': None, u'kwargs': {}, u'immutable': False}]}, u'options': {u'parent_id': None, u'task_id': u'f21c911e-f2ac-462e-9662-2efbd27bcf91', u'root_id': None}}]}] (801b)
{content_type:'application/json' content_encoding:'utf-8'
delivery_info:{'consumer_tag': 'None4', 'redelivered': False, 'routing_key': 'parse', 'delivery_tag': 623422L, 'exchange': ''} headers={'\xe5\xca.\xdb\x00\x00\x00\x00\x00': None, 'P&5\x07\x00': None, 'T\nKB\x00\x00\x00': 'fc8f0bed-665f-4699-89dd-a56fc247ea8b', 'N\xfd\x17=\x00\x00': 'gen17347#centos1', '\xcfb\xddR': 'py', '9*\xa8': None, '\xb7/b\x84\x00\x00\x00': 0, '\xe0\x0b\xfa\x89\x00\x00\x00': None, '\xdfR\xc4x\x00\x00\x00\x00\x00': [None, None], 'T3\x1d ': 'proj.tasks.parse', '\xae\xbf': 'fc8f0bed-665f-4699-89dd-a56fc247ea8b', '\x11s\x1f\xd8\x00\x00\x00\x00': "('maildir/allen-p/inbox/1.',)", 'UL\xa1\xfc\x00\x00\x00\x00\x00\x00': '{}'}}
[2017-02-04 15:47:22,463: INFO/MainProcess] Connected to amqp://guest:**#localhost:5672//
[2017-02-04 15:47:22,473: INFO/MainProcess] mingle: searching for neighbors
[2017-02-04 15:47:23,503: INFO/MainProcess] mingle: sync with 2 nodes
[2017-02-04 15:47:23,504: INFO/MainProcess] mingle: sync complete
[2017-02-04 15:47:23,530: INFO/MainProcess] parse#centos1 ready.
[2017-02-04 15:47:24,890: INFO/MainProcess] sync with es_deploy#centos1
[2017-02-04 15:47:51,017: WARNING/MainProcess] Received and deleted unknown message. Wrong destination?!?
The full contents of the message body was: body: [[u'maildir/allen-p/inbox/1.'], {}, {u'errbacks': None, u'callbacks': None, u'chord': None, u'chain': [{u'chord_size': None, u'task': u'celery.group', u'args': [], u'immutable': False, u'subtask_type': u'group', u'kwargs': {u'tasks': [{u'chord_size': None, u'task': u'proj.tasks.deploy_db', u'args': [], u'options': {u'reply_to': u'bd66dd5c-516d-3b51-ab40-c8337a33b18e', u'task_id': u'765e5bbe-198f-405c-b10c-023d35e03981'}, u'subtask_type': None, u'kwargs': {}, u'immutable': False}, {u'chord_size': None, u'task': u'proj.tasks.deploy_es', u'args': [], u'options': {u'reply_to': u'bd66dd5c-516d-3b51-ab40-c8337a33b18e', u'task_id': u'7dacb897-d023-40b5-9874-e00b75107bbd'}, u'subtask_type': None, u'kwargs': {}, u'immutable': False}]}, u'options': {u'parent_id': None, u'task_id': u'f0d41289-33e2-4c8c-8d84-9d1d4c5a9c80', u'root_id': None}}]}] (801b)
{content_type:'application/json' content_encoding:'utf-8'
delivery_info:{'consumer_tag': 'None4', 'redelivered': False, 'routing_key': 'parse', 'delivery_tag': 3L, 'exchange': ''} headers={'\xe5\xca.\xdb\x00\x00\x00\x00\x00': None, 'P&5\x07\x00': None, 'T\nKB\x00\x00\x00': '4d7754ed-0e36-4731-ae99-a84f42b8eba1', 'N\xfd\x17=\x00\x00': 'gen19722#centos1', '\xcfb\xddR': 'py', '9*\xa8': None, '\xb7/b\x84\x00\x00\x00': 0, '\xe0\x0b\xfa\x89\x00\x00\x00': None, '\xdfR\xc4x\x00\x00\x00\x00\x00': [None, None], 'T3\x1d ': 'proj.tasks.parse', '\xae\xbf': '4d7754ed-0e36-4731-ae99-a84f42b8eba1', '\x11s\x1f\xd8\x00\x00\x00\x00': "('maildir/allen-p/inbox/1.',)", 'UL\xa1\xfc\x00\x00\x00\x00\x00\x00': '{}'}}
enter code here
It would be helpful to give the versions of celery and librabbitmq you are using. Since I had a very similar problem, I'll guess that you are using celery 4.0.2 and librabbitmq 1.6.1.
In such case, this is a known compatibility issue, you can refer to https://github.com/celery/celery/issues/3675 and https://github.com/celery/librabbitmq/issues/93.
The first link gives you recommendation to solve your problem namely:
uninstall librabbitmq
pip uninstall librabbitmq
(you may have to call this command many times)
change the occurrences of amqp to pyamqp in your borker urls. (Though not in your config file if your are using one. Doing that did not work for me).
To answer more precisely your other questions: you are right saying that there is a sender and a fetcher.
The sender role is assumed by the app created when you call Celery(...). One of its role is to act as a registry of tasks, and if you look at its implementation in app/base.py, you'll see that it implements a method send_task which is directly called by the method apply_async of the Task class. This method's role is to send a marshalled version of your task through the wire up to the broker so it can be fetched by a worker. The application protocol used to transmit the message is amqp, for which an implementation is librabbitmq.
On the other side of the wire, there is another instance, launched by the worker which does the fetching work. In celery's parlance, it is called a Consumer. You can find its implementation in worker/consumer/consumer.py. You will see that it implements a create_task_handler which in turns defines the on_task_received functions that raises the error you are seeing. It is the function called when a new task is fetched from the worker and next in line to by processed.
The solution suggested therefore consists in changing the implementation of the amqp protocol so that a TypeError is not raised in on_task_received (which it seems to me would be caused by an encoding issue).
I hope it answers all your questions and gives you a clearer view of how celery works. I should end by saying that to my knowledge a "conventional" use of Celery would never require you to tamper with those kind of internals, and that you can achieve 99% of what you may want by implementing custom task classes and custom backends for example.
Just so that the answer is located here as well. In the thread Anis refers to 23doors mentions that Celery 4's new default protocol does not play nice with librabbitmq:
Apparently librabbitmq issue is related to new default protocol in celery 4.x.
He also mentions that to resolve this issue you can make use of the older protocol Celery offers by setting (if you're using Django):
CELERY_TASK_PROTOCOL = 1
Otherwise you can set the following in your celeryconf.py file
app.conf.task_protocol = 1
All credit to 23doors :)
In my application, after I get a message from rabbitmq, I push the message to a client, and wait for its ACK message, if the client doesn't reply with a ACK after some time, I requeue the message in rabbitmq with basic.reject with requeue being true.
This works fine for the first requeue operation, but after I requeue the same message for the second time, the channel is closed abruptly. From the server log, I get this error:
{amqp_error,precondition_failed,"unknown delivery tag 2",'basic.reject'}
I gather this is because the message has been removed from the queue. Why is this happening?
I had similar error
Error: Channel closed by server: 406 (PRECONDITION-FAILED) with message "PRECONDITION_FAILED - unknown delivery tag 1"
at Channel.C.accept (C:\Owlet\shopify_email_server\node_modules\amqplib\lib\channel.js:422:17)
at Connection.mainAccept [as accept] (C:\Owlet\shopify_email_server\node_modules\amqplib\lib\connection.js:64:33)
at Socket.go (C:\Owlet\shopify_email_server\node_modules\amqplib\lib\connection.js:478:48)
at Socket.emit (node:events:390:28)
at Socket.emit (node:domain:475:12)
at emitReadable_ (node:internal/streams/readable:578:12)
at processTicksAndRejections (node:internal/process/task_queues:82:21) {
code: 406,
classId: 60,
methodId: 80
My solution was to assert channel with enabled option durable
channel.assertQueue(queueName, { durable: true });
Using Kombu with RabbitMQ to implement a classic publish/subscribe design pattern. I have created a producer that creates a topic:
from kombu import Connection, Exchange, Queue
media_exchange = Exchange('media', 'topic', durable=False)
video_queue = Queue('video', exchange=media_exchange, routing_key='video')
with Connection('amqp://guest:guest#localhost//') as conn:
producer = conn.Producer(serializer='json')
producer.publish('Hello World!',
exchange=media_exchange, routing_key='video',
declare=[video_queue])
I then created a consumer to consume from the publisher:
from kombu import Connection, Exchange, Queue
media_exchange = Exchange('media', type='topic', durable=False)
video_queue = Queue('video', exchange=media_exchange, routing_key='video')
def process_media(body, message):
print(body)
#message.ack()
with Connection('amqp://guest:guest#localhost//') as conn:
with conn.Consumer(video_queue, callbacks=[process_media]) as consumer:
# Process messages and handle events on all channels
while True:
conn.drain_events()
In then launch two consumers, each one in a separate terminal; both wait for a message:
terminal 1: python consumer.py
terminal 2: python consumer.py
When I run the producer, only one consumer receives the message.
The producer publishes in an exchange, not in a queue. The queues are defined by the consumers. When using different queue name for each consumer then all will get the message.
When using many consumers for the same queue then it is load balancing, that's why only one of your consumers gets the message.
To clarify, the messages in the queue are 'consumed' i.e. the first consumer consumes it, and the message is no more in the queue, that's why the second consumer isn't getting anything.
To have 2 separate consumers for same message - use 2 separate queues i.e.
video_queue1 and video_queue2, declared and bound to the exchange media_exchange, using same key video.
producer.py
from kombu import Connection, Exchange, Queue
media_exchange = Exchange('media', 'topic', durable=False)
video_queue1 = Queue('video1', exchange=media_exchange, routing_key='video')
video_queue2 = Queue('video2', exchange=media_exchange, routing_key='video')
with Connection('amqp://guest:guest#localhost//') as conn:
producer = conn.Producer(serializer='json')
producer.publish('Hello World!',
exchange=media_exchange, routing_key='video',
declare=[video_queue1, video_queue2])
consumer1.py
from kombu import Connection, Exchange, Queue
media_exchange = Exchange('media', type='topic', durable=False)
video_queue = Queue('video1', exchange=media_exchange, routing_key='video')
def process_media(body, message):
print(body)
#message.ack()
with Connection('amqp://guest:guest#localhost//') as conn:
with conn.Consumer(video_queue, callbacks=[process_media]) as consumer:
# Process messages and handle events on all channels
while True:
conn.drain_events()
consumer2.py
from kombu import Connection, Exchange, Queue
media_exchange = Exchange('media', type='topic', durable=False)
video_queue = Queue('video2', exchange=media_exchange, routing_key='video')
def process_media(body, message):
print(body)
#message.ack()
with Connection('amqp://guest:guest#localhost//') as conn:
with conn.Consumer(video_queue, callbacks=[process_media]) as consumer:
# Process messages and handle events on all channels
while True:
conn.drain_events()