I have sent around 500 messages to an exchange in RabbitMQ.Only 8 of them are processed by the application while 492 of them are not acknowledged.
There are 40 concurrent listeners to the exchange which am posting to.
The logs are as follows:
[taskExecutor-16] DEBUG o.s.a.r.l.BlockingQueueConsumer - Retrieving delivery for Consumer: tags=[{amq.ctag-cydNATSoJE9VYDXDxdS3og=Q102}], channel=Cached Rabbit Channel: AMQChannel(amqp://guest#127.0.0.1:5672/,30), acknowledgeMode=AUTO local queue size=0
02:24:49.130 [taskExecutor-19] DEBUG o.s.a.r.l.BlockingQueueConsumer - Retrieving delivery for Consumer: tags=[{amq.ctag-_tAOgctllmGebYliKjCwPw=Q102}], channel=Cached Rabbit Channel: AMQChannel(amqp://guest#127.0.0.1:5672/,25), acknowledgeMode=AUTO local queue size=0
02:24:49.130 [taskExecutor-6] DEBUG o.s.a.r.l.BlockingQueueConsumer - Retrieving delivery for Consumer: tags=[{amq.ctag-MV9tAhL5QbhtTSOJACziHQ=Q102}], channel=Cached Rabbit Channel: AMQChannel(amqp://guest#127.0.0.1:5672/,39), acknowledgeMode=AUTO local queue size=0
02:24:49.130 [taskExecutor-30] DEBUG o.s.a.r.l.BlockingQueueConsumer - Retrieving delivery for Consumer: tags=[{amq.ctag-SG1nB33uxABsAry7jsc6sw=Q102}], channel=Cached Rabbit Channel: AMQChannel(amqp://guest#127.0.0.1:5672/,24), acknowledgeMode=AUTO local queue size=0
02:24:49.130 [taskExecutor-12] DEBUG o.s.a.r.l.BlockingQueueConsumer - Retrieving delivery for Consumer: tags=[{amq.ctag-MOY_QKujE_JiiNLn9gn7AQ=Q102}], channel=Cached Rabbit Channel: AMQChannel(amqp://guest#127.0.0.1:5672/,34), acknowledgeMode=AUTO local queue size=0
02:24:49.130 [taskExecutor-35] DEBUG o.s.a.r.l.BlockingQueueConsumer - Retrieving delivery for Consumer: tags=[{amq.ctag-BcKYuEPnscByD-h-3NpW3Q=Q102}], channel=Cached Rabbit Channel: AMQChannel(amqp://guest#127.0.0.1:5672/,12), acknowledgeMode=AUTO local queue size=0
02:24:49.130 [taskExecutor-27] DEBUG o.s.a.r.l.BlockingQueueConsumer - Retrieving delivery for Consumer: tags=[{amq.ctag-O2d1WsSE-qHKhhZPu-plSg=Q102}], channel=Cached Rabbit Channel: AMQChannel(amqp://guest#127.0.0.1:5672/,23), acknowledgeMode=AUTO local queue size=0
02:24:49.130 [taskExecutor-18] DEBUG o.s.a.r.l.BlockingQueueConsumer - Retrieving delivery for Consumer: tags=[{amq.ctag-vO9zIyE_t2zBdEElKW4R6w=Q102}], channel=Cached Rabbit Channel: AMQChannel(amqp://guest#127.0.0.1:5672/,28), acknowledgeMode=AUTO local queue size=0
What are the settings to be done to avoid blocking queue issue?
Related
RabbitMQ 3.10.1
rabbitmq-diagnostics status
...
Config files
* /etc/rabbitmq/rabbitmq.config
...
rabbitmq.config:
[
{rabbit,
[
{heartbeat, 90}
]
}
].
RabbitMQ Management show 5s heartbeat
And log:
2022-05-13 19:56:43.235925+03:00 [error] <0.5979.0> closing AMQP connection <0.5979.0> (xxx.xxx.xxx.xxx:3555 -> xxx.xxx.xxx.xxx:5672):
2022-05-13 19:56:43.235925+03:00 [error] <0.5979.0> missed heartbeats from client, timeout: 5s
How to fix this?
Set the heartbeat to 90s in the client. Most clients are able to set the heartbeat (from the client). RabbitMQ will respect the heartbeat suggested by the client. More about that here: https://www.rabbitmq.com/heartbeats.html#heartbeats-timeout
I'm using RPC Pattern for processing my objects with RabbitMQ.
You suspect,I have an object, and I want to have that process finishes and After that send ack to RPC Client.
Ack as default has a timeout about 3 Minutes.
My process Take long time.
How can I change this timeout for ack of each objects or what I must be do for handling these like processess?
Modern versions of RabbitMQ have a delivery acknowledgement timeout:
In modern RabbitMQ versions, a timeout is enforced on consumer delivery acknowledgement. This helps detect buggy (stuck) consumers that never acknowledge deliveries. Such consumers can affect node's on disk data compaction and potentially drive nodes out of disk space.
If a consumer does not ack its delivery for more than the timeout value (30 minutes by default), its channel will be closed with a PRECONDITION_FAILED channel exception. The error will be logged by the node that the consumer was connected to.
Error message will be:
Channel error on connection <####> :
operation none caused a channel exception precondition_failed: consumer ack timed out on channel 1
Timeout by default is 30 minutes (1,800,000ms)note 1 and is configured by the consumer_timeout parameter in rabbitmq.conf.
note 1: Timeout was 15 minutes (900,000ms) before RabbitMQ 3.8.17.
if you run rabbitmq in docker, you can describe volume with file rabbitmq.conf, then create this file inside volume and set consumer_timeout
for example:
docker compose
version: "2.4"
services:
rabbitmq:
image: rabbitmq:3.9.13-management-alpine
network_mode: host
container_name: 'you name'
ports:
- 5672:5672
- 15672:15672 ----- if you use gui for rabbit
volumes:
- /etc/rabbitmq/rabbitmq.conf:/etc/rabbitmq/rabbitmq.conf
And you need create file
rabbitmq.conf
on you server by this way
/etc/rabbitmq/
documentation with params: https://github.com/rabbitmq/rabbitmq-server/blob/v3.8.x/deps/rabbit/docs/rabbitmq.conf.example
I have working srv0 with Application normally connected to rabbitmq
I've cloned srv0 to srv1 just ip chanded via customization(for test purposes)
Now Client application on srv1 can't connect to rabbit(Catalina.out):
Caused by: com.rabbitmq.client.ShutdownSignalException: channel error; protocol method: #method<channel.close>(reply-code=404, reply-text=NOT_FOUND - no queue 'wrs-checklist-delete' in vhost '/', class-id=50, method-id=10)
/var/log/rabbitmq/*.log answers
=ERROR REPORT==== 24-Nov-2016::17:54:19 ===
Channel error on connection <0.344.0> (127.0.0.1:16899 -> 127.0.0.1:5672, vhost: '/', user: 'guest'), channel 1:
{amqp_error,not_found,"no queue 'wrs-checklist-delete' in vhost '/'",
'queue.declare'}
checked:
$HOSTNAME return host of srv1
Where should I start to realize the problem?
I did't have queues, ./rabbitmqadmin list queues returned ~connect error
I've stopped rabbit-server and deleted /var/lib/rabbitmq/mnesia
Then started rabbit-server and it recreated /var/lib/rabbitmq/mnesia
after that created queues:
./rabbitmqadmin declare exchange --vhost=/ name=queuename -create type=direct
./rabbitmqadmin declare queue --vhost=/ name=
./rabbitmqadmin --vhost=/ declare binding source="queuename" destination_type="queue" destination="queuename"
and thats OK now
I've been struggling to increase the inactivity timeout on Activemq (changing settings in activemq.xml don't seem to work). I see the heartbeats being exchanged in the browser console (SockJS client) but after a while I get Connection closed by broker. Below are my configurations:
activemq.xml:
<transportConnectors>
<transportConnector name="openwire" uri="tcp://0.0.0.0:61616?maximumConnections=1000&wireFormat.maxInactivityDuration=3600000&wireFormat.maxFrameSize=104857600"/>
<transportConnector name="stomp" uri="stomp://0.0.0.0:61613?trace=true&maximumConnections=1000&wireFormat.maxInactivityDuration=3600000&wireFormat.maxFrameSize=104857600"/>
</transportConnectors>
ActiveMQ log snippet:
2016-02-18 19:14:28,185 | INFO | ActiveMQ.Advisory.Topic Inactive for longer than 30000 ms - removing ... | org.apache.activemq.broker.region.Topic | ActiveMQ Broker[localhost] Scheduler
2016-02-18 19:16:18,839 | WARN | Transport Connection to: tcp://127.0.0.1:53894 failed: org.apache.activemq.transport.InactivityIOException: Channel was inactive for too (>10000) long: tcp://127.0.0.1:53894 | org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ InactivityMonitor Worker
Spring messaging TRACE logs:
TRACE 02/18 19:16:18,839 [reactor-tcp-io-6] - Received heart-beat in session o05rr4u5
DEBUG 02/18 19:16:18,886 [reactor-tcp-io-6] - TCP connection to broker closed in session o05rr4u5
DEBUG 02/18 19:16:18,886 [reactor-tcp-io-6] - Cleaning up connection state for session o05rr4u5
TRACE 02/18 19:16:18,886 [clientOutboundChannel-44] - Encoding STOMP ERROR, headers={message=[Connection to broker closed.]}
DEBUG 02/18 19:16:18,890 [clientInboundChannel-53] - Ignoring DISCONNECT in session o05rr4u5. Connection already cleaned up.
Appreciate if someone could please guide me on this.
Thanks,
I'm having difficulty configuring my connection to CloudAMQP in my deployed grails application. I can run the application locally against a locally installed RabbitMQ instance but can't figure out how to correctly define my application to run on CloudBees using the CloudAMQP service.
In my Config.groovy, I'm defining my connection info and a queue:
rabbitmq {
connectionfactory {
username = 'USERNAME'
password = 'PASSWORD'
hostname = 'lemur.cloudamqp.com'
}
queues = {
testQueue autoDelete: false, durable: false, exclusive: false
}
}
When the application starts and tries to connect, I see the following log messages:
2013-08-23 21:29:59,195 [main] DEBUG listener.SimpleMessageListenerContainer - Starting Rabbit listener container.
2013-08-23 21:29:59,205 [SimpleAsyncTaskExecutor-1] DEBUG listener.BlockingQueueConsumer - Starting consumer Consumer: tag=[null], channel=null, acknowledgeMode=AUTO local queue size=0
2013-08-23 21:30:08,405 [SimpleAsyncTaskExecutor-1] WARN listener.SimpleMessageListenerContainer - Consumer raised exception, processing can restart if the connection factory supports it
org.springframework.amqp.AmqpIOException: java.io.IOException
at org.springframework.amqp.rabbit.connection.RabbitUtils.convertRabbitAccessException(RabbitUtils.java:112)
at org.springframework.amqp.rabbit.connection.AbstractConnectionFactory.createBareConnection(AbstractConnectionFactory.java:163)
at org.springframework.amqp.rabbit.connection.CachingConnectionFactory.createConnection(CachingConnectionFactory.java:228)
at org.springframework.amqp.rabbit.connection.ConnectionFactoryUtils$1.createConnection(ConnectionFactoryUtils.java:119)
at org.springframework.amqp.rabbit.connection.ConnectionFactoryUtils.doGetTransactionalResourceHolder(ConnectionFactoryUtils.java:163)
at org.springframework.amqp.rabbit.connection.ConnectionFactoryUtils.getTransactionalResourceHolder(ConnectionFactoryUtils.java:109)
at org.springframework.amqp.rabbit.listener.BlockingQueueConsumer.start(BlockingQueueConsumer.java:199)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$AsyncMessageProcessingConsumer.run(SimpleMessageListenerContainer.java:524)
at java.lang.Thread.run(Unknown Source)
Caused by: java.io.IOException
at com.rabbitmq.client.impl.AMQChannel.wrap(AMQChannel.java:106)
at com.rabbitmq.client.impl.AMQChannel.wrap(AMQChannel.java:102)
at com.rabbitmq.client.impl.AMQChannel.exnWrappingRpc(AMQChannel.java:124)
at com.rabbitmq.client.impl.AMQConnection.start(AMQConnection.java:381)
at com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:516)
at com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:545)
Caused by: com.rabbitmq.client.ShutdownSignalException: connection error; reason: java.net.SocketException: Connection reset
at com.rabbitmq.utility.ValueOrException.getValue(ValueOrException.java:67)
at com.rabbitmq.utility.BlockingValueOrException.uninterruptibleGetValue(BlockingValueOrException.java:33)
at com.rabbitmq.client.impl.AMQChannel$BlockingRpcContinuation.getReply(AMQChannel.java:343)
at com.rabbitmq.client.impl.AMQChannel.privateRpc(AMQChannel.java:216)
at com.rabbitmq.client.impl.AMQChannel.exnWrappingRpc(AMQChannel.java:118)
... 3 more
Caused by: java.net.SocketException: Connection reset
at com.rabbitmq.client.impl.Frame.readFrom(Frame.java:95)
at com.rabbitmq.client.impl.SocketFrameHandler.readFrame(SocketFrameHandler.java:131)
at com.rabbitmq.client.impl.AMQConnection$MainLoop.run(AMQConnection.java:508)
2013-08-23 21:30:08,406 [SimpleAsyncTaskExecutor-1] INFO listener.SimpleMessageListenerContainer - Restarting Consumer: tag=[null], channel=null, acknowledgeMode=AUTO local queue size=0
2013-08-23 21:30:08,406 [SimpleAsyncTaskExecutor-1] DEBUG listener.BlockingQueueConsumer - Closing Rabbit Channel: null
2013-08-23 21:30:08,407 [SimpleAsyncTaskExecutor-2] DEBUG listener.BlockingQueueConsumer - Starting consumer Consumer: tag=[null], channel=null, acknowledgeMode=AUTO local queue size=0
Aug 23, 2013 9:30:11 PM org.apache.catalina.core.ApplicationContext log
INFO: Initializing Spring FrameworkServlet 'grails'
Aug 23, 2013 9:30:11 PM org.apache.coyote.http11.Http11Protocol init
INFO: Initializing Coyote HTTP/1.1 on http-8634
Aug 23, 2013 9:30:11 PM org.apache.coyote.http11.Http11Protocol start
INFO: Starting Coyote HTTP/1.1 on http-8634
According to https://developer.cloudbees.com/bin/view/RUN/CloudAMQP
when you bind your CloudAMQP service to your app - some config params are provided in the pattern of CLOUDAMQP_URL_ - this is the type of thing you would need to put in your config files so they can be wired in when the app launches.
Make sure to specify the virtualHost for CloudAMQP connections. That worked for me.