In my application, after I get a message from rabbitmq, I push the message to a client, and wait for its ACK message, if the client doesn't reply with a ACK after some time, I requeue the message in rabbitmq with basic.reject with requeue being true.
This works fine for the first requeue operation, but after I requeue the same message for the second time, the channel is closed abruptly. From the server log, I get this error:
{amqp_error,precondition_failed,"unknown delivery tag 2",'basic.reject'}
I gather this is because the message has been removed from the queue. Why is this happening?
I had similar error
Error: Channel closed by server: 406 (PRECONDITION-FAILED) with message "PRECONDITION_FAILED - unknown delivery tag 1"
at Channel.C.accept (C:\Owlet\shopify_email_server\node_modules\amqplib\lib\channel.js:422:17)
at Connection.mainAccept [as accept] (C:\Owlet\shopify_email_server\node_modules\amqplib\lib\connection.js:64:33)
at Socket.go (C:\Owlet\shopify_email_server\node_modules\amqplib\lib\connection.js:478:48)
at Socket.emit (node:events:390:28)
at Socket.emit (node:domain:475:12)
at emitReadable_ (node:internal/streams/readable:578:12)
at processTicksAndRejections (node:internal/process/task_queues:82:21) {
code: 406,
classId: 60,
methodId: 80
My solution was to assert channel with enabled option durable
channel.assertQueue(queueName, { durable: true });
Related
I'm running a Celery using Kombu Consumer, reading messages from a rabbitmq. After I finished coding and ran it well, except for having messages get lost and the consumer doesn't read them.
For example, if I send 5 messages to the queue, Kombu manages to get 5, but only processes some of them, the others I get a warning of "Received and deleted unknown message. Wrong destination?!?" and the message leaves the rabbitmq queue, even without using message.ack().
this is my code. I run with celery -A tasks worker -Q parse -l DEBUG --autoscale=2,1
import time
import celery
from celery import Celery, bootsteps
from kombu import Consumer, Queue, Exchange
from setuptools import setup
app = Celery("bob", broker='amqp://guest:guest#localhost:5672//')
parse_queue = Queue("parse", Exchange("parse"), "parse")
#app.task
def do_something():
print("I'm doing something")
class MyConsumerStep(bootsteps.ConsumerStep):
def get_consumers(self, channel):
return [
Consumer(
channel,
queues=[parse_queue],
callbacks=[
self.handle_parse
],
accept=["json"]
)
]
def handle_parse(self, body, message):
print('Received message: {0!r}'.format(body))
do_something.delay()
app.steps["consumer"].add(MyConsumerStep)
When I send a message through rabbitmq in queue parsing celery shows me this log:
[2023-02-15 11:53:27,634: INFO/MainProcess] Connected to amqp://guest:**#127.0.0.1:5672//
[2023-02-15 11:53:27,691: INFO/MainProcess] mingle: searching for neighbors
[2023-02-15 11:53:28,205: INFO/SpawnPoolWorker-1] child process 93064 calling self.run()
[2023-02-15 11:53:28,782: INFO/MainProcess] mingle: all alone
[2023-02-15 11:53:28,931: INFO/MainProcess] celery#parse_worker ready.
[2023-02-15 11:53:32,217: INFO/MainProcess] Events of group {task} enabled by remote.
[2023-02-15 11:53:56,288: WARNING/MainProcess] Received message: '{"teste": "teste"}'
[2023-02-15 12:30:23,125: WARNING/MainProcess] Received and deleted unknown message. Wrong destination?!?
The full contents of the message body was: body: '{"teste": "teste"}' (18b)
{content_type:None content_encoding:None
delivery_info:{'consumer_tag': 'None5', 'delivery_tag': 1, 'redelivered': False, 'exchange': '', 'routing_key': 'parse'} headers={}}
[2023-02-15 12:30:23,280: WARNING/MainProcess] Received message: '{"teste": "teste"}'
[2023-02-15 12:30:23,441: WARNING/MainProcess] Received and deleted unknown message. Wrong destination?!?
The full contents of the message body was: body: '{"teste": "teste"}' (18b)
{content_type:None content_encoding:None
delivery_info:{'consumer_tag': 'None5', 'delivery_tag': 2, 'redelivered': False, 'exchange': '', 'routing_key': 'parse'} headers={}}
[2023-02-15 12:30:23,599: WARNING/MainProcess] Received message: '{"teste": "teste"}'
[2023-02-15 12:30:23,763: WARNING/MainProcess] Received and deleted unknown message. Wrong destination?!?
The full contents of the message body was: body: '{"teste": "teste"}' (18b)
{content_type:None content_encoding:None
delivery_info:{'consumer_tag': 'None5', 'delivery_tag': 3, 'redelivered': False, 'exchange': '', 'routing_key': 'parse'} headers={}}
[2023-02-15 12:30:23,913: WARNING/MainProcess] Received message: '{"teste": "teste"}'
Looking through the flower, it doesn't display any worker information. I don't know if Kombu has any problem with flower integration, but I see that read messages are processed normally, even without information in flower.
What is going on?
Why are messages lost?
Why doesn't flower show any information?
Is there another way to process in parallel without using celery's --autoscale or --concurrency command?
I am using amqplib to transfer messages in my node.js server.
Here is my code to listen on queue:
channel.consume(queue, handler1, { noAck: true })
Now, I want to update the consumer to listen the same queue
Like this:
channel.consume(queue, handler2, { noAck: true })
I try to unbindQueue or deleteQueue but don't know why the error Unhandled rejection IllegalOperationError: Channel closing is thrown
I want to know the cause of Rabbitmq crash which randomly occur. can you let me know what kind of causes could be considered?
Also my team should manually restart the rabbitmq when crash happens, so I want to know if there is a way to restart rabbitmq server automatically.
Here is the error report when rabbitmq crash occur:
=WARNING REPORT==== 6-Dec-2017::07:56:43 ===
closing AMQP connection <0.4387.0> (000000:23070 -> 00000:5672, vhost: '/', user: '00000'):
client unexpectedly closed TCP connection
Also this is part of sasl.gsd fild :
=SUPERVISOR REPORT==== 7-Dec-2017::10:03:15 ===
Supervisor: {local,sockjs_session_sup}
Context: child_terminated
Reason: {function_clause,
[{gen_server,cast,
[{},sockjs_closed],
[{file,"gen_server.erl"},{line,218}]},
{rabbit_ws_sockjs,service_stomp,3,
[{file,"src/rabbit_ws_sockjs.erl"},{line,150}]},
{sockjs_session,emit,2,
[{file,"src/sockjs_session.erl"},{line,173}]},
{sockjs_session,terminate,2,
[{file,"src/sockjs_session.erl"},{line,311}]},
{gen_server,try_terminate,3,
[{file,"gen_server.erl"},{line,629}]},
{gen_server,terminate,7,
[{file,"gen_server.erl"},{line,795}]},
{proc_lib,init_p_do_apply,3,
[{file,"proc_lib.erl"},{line,247}]}]}
Offender: [{pid,<0.20883.1160>},
{id,undefined},
{mfargs,
{sockjs_session,start_link,
["pd4tvvi0",
{service,"/stomp",
#Fun<rabbit_ws_sockjs.1.47892404>,{},
"//cdn.jsdelivr.net/sockjs/1.0.3/sockjs.min.js",
false,true,5000,25000,131072,
#Fun<rabbit_ws_sockjs.0.47892404>,undefined},
[{peername,{{172,31,6,213},9910}},
{sockname,{{172,31,5,49},15674}},
{path,"/stomp/744/pd4tvvi0/htmlfile"},
{headers,[]},
{socket,#Port<0.12491352>}]]}},
{restart_type,transient},
{shutdown,5000},
{child_type,worker}]
=CRASH REPORT==== 7-Dec-2017::10:03:20 ===
crasher:
initial call: sockjs_session:init/1
pid: <0.25851.1160>
registered_name: []
exception exit: {function_clause,
[{gen_server,cast,
[{},sockjs_closed],
[{file,"gen_server.erl"},{line,218}]},
{rabbit_ws_sockjs,service_stomp,3,
[{file,"src/rabbit_ws_sockjs.erl"},{line,150}]},
{sockjs_session,emit,2,
[{file,"src/sockjs_session.erl"},{line,173}]},
{sockjs_session,terminate,2,
[{file,"src/sockjs_session.erl"},{line,311}]},
{gen_server,try_terminate,3,
[{file,"gen_server.erl"},{line,629}]},
{gen_server,terminate,7,
[{file,"gen_server.erl"},{line,795}]},
{proc_lib,init_p_do_apply,3,
[{file,"proc_lib.erl"},{line,247}]}]}
in function gen_server:terminate/7 (gen_server.erl, line 800)
ancestors: [sockjs_session_sup,<0.177.0>]
messages: []
links: [<0.178.0>]
dictionary: []
trap_exit: true
status: running
heap_size: 987
stack_size: 27
reductions: 175
neighbours:
Please check out the error report I posted above and let me know the causes of rabbitmq crash and the way to automatically restart rabbitmq server.
Thanks!!
I'm following the following tutorial to the letter:
https://www.rabbitmq.com/tutorials/tutorial-two-java.html.
I start the RabbitMQ server as such:
docker pull rabbitmq
docker run -d --hostname my-rabbit-host --name my-rabbit -p 5672:5672 rabbitmq:3
From the tutorial:
Using this code we can be sure that even if you kill a worker using
CTRL+C while it was processing a message, nothing will be lost. Soon
after the worker dies all unacknowledged messages will be redelivered.
I spawn two consumers, and when I CTRL+C one of them, the other running one does not receive the messages that were originally destined to the former consumer. How do I get the messages to be redelivered after CTRL+C'ing out of one of the consumers?
Edit: I'm now installing RabbitMQ via 'brew', but I'm still seeing the same issue.
brew update
brew install rabbitmq
/usr/local/sbin/rabbitmq-server &
There is no need to put sleep or anything like that in the consumer code. On the link you provided, search for paragraph starting with Manual message acknowledgments and look at the code there. The key thing is not to acknowledge the message. If you have autoACK flag set to true, then you can call anything you want, the message is acknowledged as soon as it is received. So simply don't set that flag, and also for testing you could comment out the line channel.basicAck(envelope.getDeliveryTag(), false); in order not to do manual ACK either. So when the consumer exits, the message will still be in the queue.
Strange, RabbitMQ works for me out of the box.
Step 1, I started RabbitMQ:
$ docker run -d --hostname my-rabbit-host --name my-rabbit -p 5672:5672 rabbitmq:3
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
e0c3257b8b49 rabbitmq:3 "docker-entrypoint.s…" 18 minutes ago Up 14 minutes 4369/tcp, 5671/tcp, 25672/tcp, 0.0.0.0:5672->5672/tcp my-rabbit
Step 2, I published a message (By the way, I tried with Node.js. See Appendix below for source code):
$ node src/producer.js
Publisher: TODO 1st
Step 3, I started two consumers one after another (my consumer is designed, for the testing purpose, not to acknowledge, so RabbitMQ will never dequeue a message).
Consumer 1 will receive the message, while consumer 2 won't.
Consumer 1:
$ node src/consumer.js
Consumer: TODO 1st
Consumer 2:
$ node src/consumer.js
Step 4, when I stop consumer 1 by 'Ctrl + c', Consumer 2 will immediately receive the message from RabbitMQ:
Consumer 2:
$ node src/consumer.js
Consumer: TODO 1st
Conclusion: Basically, when setting up a consumer, we need to tell RabbitMQ not to dequeue the message until its acknowledgement has been received from the consumer. As a result, if consumer 1 is stopped for any reason before it has a chance to acknowledge the message, RabbitMQ will redeliver the message to consumer 2.
Appendix
src/producer.js
var q = 'tasks'
function bail (err) {
console.error(err)
process.exit(1)
}
// Publisher
function publisher (conn) {
conn.createChannel(onOpen)
function onOpen (err, ch) {
if (err != null) bail(err)
ch.assertQueue(q)
const msg = 'TODO 1st'
ch.sendToQueue(q, Buffer.from(msg), { persistent: true })
console.log('Publisher: ', msg)
}
}
require('amqplib/callback_api')
.connect('amqp://guest:guest#localhost', function (err, conn) {
if (err != null) bail(err)
publisher(conn)
})
src/consumer.js
var q = 'tasks'
function bail (err) {
console.error(err)
process.exit(1)
}
// Consumer
function consumer (conn) {
conn.createChannel(onOpen)
function onOpen (err, ch) {
if (err != null) bail(err)
ch.assertQueue(q)
ch.consume(q, function (msg) {
if (msg !== null) {
console.log('Consumer: ', msg.content.toString())
// Commented out the line below, so RabbitMQ never dequeues a message
// ch.ack(msg)
}
}, { noAck: false })
}
}
require('amqplib/callback_api')
.connect('amqp://guest:guest#localhost', function (err, conn) {
if (err != null) bail(err)
consumer(conn)
})
Well, I've started a simple project with freediameter library. what I need to achieve in my project is to create a client diameter application that could send some CCR request to a diameter server.
for this matter, I tried to create a new extension for freediameter daemon.
So, the first thing I did was to setup the peer diameter server in config file:
ConnectPeer = "vm-pc.my.domain" { No_TLS; ConnectTo = "192.168.56.2"; Port=3868; };
and then initialize a request message in ta_entry function, in message body i specified the Destination-Host and Destination-Realm but still I receive this error:
01/10/17,01:55:24.980611 ERROR Routing error: 'No remaining suitable candidate to route the message to' for the following message:
01/10/17,01:55:24.980620 ERROR 'Credit-Control-Request'
01/10/17,01:55:24.980628 ERROR Version: 0x01
01/10/17,01:55:24.980635 ERROR Length: 20
01/10/17,01:55:24.980642 ERROR Flags: 0xC0 (RP--)
01/10/17,01:55:24.980649 ERROR Command Code: 272
01/10/17,01:55:24.980656 ERROR ApplicationId: 4
01/10/17,01:55:24.980664 ERROR Hop-by-Hop Identifier: 0x00000000
01/10/17,01:55:24.980671 ERROR End-to-End Identifier: 0xDCA05EF4
01/10/17,01:55:24.980678 ERROR {internal data}: src:(nil)(0) rwb:0x0 rt:0 cb:0x80523df30,0x0(0x803397da0) qry:0x0 asso:0 sess:0x0
01/10/17,01:55:24.981562 ERROR AVP: 'Session-Id'(263) l=8 f=-M val="hadi-pc.my.domain;1484000714;6"
01/10/17,01:55:24.981569 ERROR AVP: 'Origin-Host'(264) l=8 f=-M val="hadi-pc.my.domain"
01/10/17,01:55:24.981577 ERROR AVP: 'Origin-Realm'(296) l=8 f=-M val="my.domain"
01/10/17,01:55:24.981584 ERROR AVP: 'Destination-Host'(293) l=8 f=-M val="vm-pc.my.domain"
01/10/17,01:55:24.981591 ERROR AVP: 'Destination-Realm'(283) l=8 f=-M val="my.domain"
01/10/17,01:55:24.981599 ERROR AVP: 'Auth-Application-Id'(258) l=12 f=-M val=4 (0x4)
01/10/17,01:55:24.981606 ERROR AVP: 'CC-Request-Type'(416) l=12 f=-M val='EVENT_REQUEST' (4 (0x4))
01/10/17,01:55:24.981613 ERROR AVP: 'CC-Request-Number'(415) l=12 f=-M val=2 (0x2)
Can anybody help me on this ?
ps: as I see in the logs the CER/CEA are normal.
What Origin-Realm was signaled in the Capability-Exchange-Answer message when the connection was established?
The Diameter request routing process (as described in https://www.rfc-editor.org/rfc/rfc6733#section-6.1) relies on the realm, so if the Origin-Realm returned by the peer is not "my.domain", freeDiameter will not route to this peer, even if the Origin-Host matches.