Very rare issue which we observed in Production.
Exchange type got changed at runtime from x-modulus-hash to x-consistent-hash which is causing client to shutdown. What could be the reason? How to prevent this error?
Exchange with type
2022-09-19 15:54:48.158 ERROR [AMQP Connection x.x.x.x:5671] [org.springframework.amqp.rabbit.connection.CachingConnectionFactory] - <Channel shutdown: channel error; protocol method: #method<channel.close>(reply-code=406, reply-text=PRECONDITION_FAILED - inequivalent arg 'type' for exchange 'message-exchange.shovel' in vhost '/': received ''x-modulus-hash'' but current is ''x-consistent-hash'', class-id=40, method-id=10)>
Related
I have an application that is pushing data into RabbitMQ and then some other apps are subscribing to the different exchanges.
But recently, I keep having errors like this after a few hours:
2020-07-09 12:45:12.670 [error] <0.23578.1> Error on AMQP connection <0.23578.1> (172.18.0.5:48230 ->
172.18.0.3:5672, vhost: '/', user: 'guest', state: running), channel 6:
operation basic.publish caused a connection exception unexpected_frame:
"expected content header for class 60, got non content"
2020-07-09 12:45:12.674 [info] <0.23578.1> closing AMQP connection <0.23578.1> (172.18.0.5:48230 ->
172.18.0.3:5672, vhost: '/'
On the client side, I get messages like this:
"Already closed: The AMQP operation was interrupted: AMQP close-reason, initiated by Peer,
code=505, text='UNEXPECTED_FRAME - expected content body, got non content body frame instead',
classId=60, methodId=40"
This is on a docker container.
What could this error be about?
You are sharing a channel for concurrent publishing, use below code
lock (ch) { ch.BasicPublish(); }
I have a Spring Boot application that uses Camel Rabbit to consume messages from a queue. I use an URI to declare the queue with a dead letter exchange, but I'm not supplying the option deadLetterRoutingKey as I want the messages going to DLX to keep the original routing key. When the application starts it throws the following error:
Caused by: com.rabbitmq.client.ShutdownSignalException: channel error;
protocol method: #method<channel.close>(reply-code=406, reply-text=PRECONDITION_FAILED - invalid arg 'x-dead-letter-routing-key' for queue 'entry.paid.erp' in vhost '/': {unacceptable_type,void}, class-id=50, method-id=10)
Is it possible to configure Camel to have this behavior?
Some additional information:
Camel version: 2.19.1
Spring Boot version: 1.5.4.RELEASE
Example of URI I'm using:
rabbitmq://server:port/my-exchange
?connectionFactory=#connectionFactory
&exchangeType=topic
&queue=my-queue
&autoAck=true
&durable=true
&autoDelete=false
&exclusive=false
&automaticRecoveryEnabled=true
&concurrentConsumers=15
&deadLetterExchange=dlx-exchange
&deadLetterExchangeType=fanout
&deadLetterQueue=dlx-queue
When I set a value for deadLetterRoutingKey the application starts with no errors.
Thanks!
I have working srv0 with Application normally connected to rabbitmq
I've cloned srv0 to srv1 just ip chanded via customization(for test purposes)
Now Client application on srv1 can't connect to rabbit(Catalina.out):
Caused by: com.rabbitmq.client.ShutdownSignalException: channel error; protocol method: #method<channel.close>(reply-code=404, reply-text=NOT_FOUND - no queue 'wrs-checklist-delete' in vhost '/', class-id=50, method-id=10)
/var/log/rabbitmq/*.log answers
=ERROR REPORT==== 24-Nov-2016::17:54:19 ===
Channel error on connection <0.344.0> (127.0.0.1:16899 -> 127.0.0.1:5672, vhost: '/', user: 'guest'), channel 1:
{amqp_error,not_found,"no queue 'wrs-checklist-delete' in vhost '/'",
'queue.declare'}
checked:
$HOSTNAME return host of srv1
Where should I start to realize the problem?
I did't have queues, ./rabbitmqadmin list queues returned ~connect error
I've stopped rabbit-server and deleted /var/lib/rabbitmq/mnesia
Then started rabbit-server and it recreated /var/lib/rabbitmq/mnesia
after that created queues:
./rabbitmqadmin declare exchange --vhost=/ name=queuename -create type=direct
./rabbitmqadmin declare queue --vhost=/ name=
./rabbitmqadmin --vhost=/ declare binding source="queuename" destination_type="queue" destination="queuename"
and thats OK now
We are trying to implement a request/response scenario where the messages will be deleted server(consumer) is down. We start with no exchanges / queues in the rabbit mq installation.
There is a server which creates its own exchange / queue and we want this to be auto-delete=true.
In case the server is up before the client, the exchange is created with the correct configuration. But when the client is up we get this error:
RabbitMQ.Client.Exceptions.OperationInterruptedException: The AMQP operation was interrupted: AMQP close-reason, initiated by Peer, code=406, text="PRECONDITION_FAILED - inequivalent arg 'auto_delete' for exchange 'simple_request' in vhost '****': received 'false' but current is 'true'", classId=40, methodId=10, cause=
In case the client is up first, and tries to send a message an exchange is created with the queue name that we have defined but it is not auto-delete=true which results to error:
RabbitMQ receive transport failed: The AMQP operation was interrupted: AMQP close-reason, initiated by Peer, code=406, text="PRECONDITION_FAILED - inequivalent arg 'auto_delete' for exchange 'simple_request' in vhost '****': received 'true' but current is 'false'", classId=40, methodId=10, cause=RabbitMQ receive transport failed: The supervisor is stopping, no additional scopes can be created
when the server is eventually started.
How do we implement auto-delete queues in a request response scenario?
You can update the URI in your client for the service queue to include query string parameters so that the queue is created properly.
rabbitmq://host/vhost/queue?autodelete=true&durable=false
Note I included durable=false but that's only if you're using a non-durable queue and I wanted to be complete.
In deploying a partitioned stream into our XD container, the stream failed and we can find into XD container log level an Amq issue. The cause seems to come from creation of XD source DLQ : ('x-dead-letter-exchange' is not properly set by Spring XD during queue creation process ?)
Caused by: com.rabbitmq.client.ShutdownSignalException: channel error; protocol method: #method<channel.close>(reply-code=406, reply-text=PRECONDITION_FAILED - inequivalent arg 'x-dead-letter-exchange' for queue 'xdbus.partitioned.0-0' in vhost '/': received none but current is the value 'xdbus.DLX' of type 'longstr', class-id=50, method-id=10)
at com.rabbitmq.client.impl.ChannelN.asyncShutdown(ChannelN.java:478)
at com.rabbitmq.client.impl.ChannelN.processAsync(ChannelN.java:315)
at com.rabbitmq.client.impl.AMQChannel.handleCompleteInboundCommand(AMQChannel.java:144)
at com.rabbitmq.client.impl.AMQChannel.handleFrame(AMQChannel.java:91)
at com.rabbitmq.client.impl.AMQConnection$MainLoop.run(AMQConnection.java:552)
...
Partitioned stream feature implementation
This feature is described in Spring XD guide.
The stream definition is simpler than the guide, we have defined a stream with two elements: a sink and a source to check behavior;
time | log
When we are deploying the stream, we set this kind of partitioning configuration:
module.time.producer.partitionKeyExpression=payload.charAt(payload.length()-1),module.log.count=3
So, the result expected is:
the deployment of 1 instance for time source and 3 instances for log sink;
at runtime, key expression will be used to route message through log instances.
Our installation setup
Zookeeper is installed in cluster. (3 nodes)
RabbitMq is used as message bus. (2 nodes)
Spring XD is installed in cluster. (3 nodes)
NB: at XD level, the bus is configure to create DLQ fore each deployed XD 'module'.
Has anyone had similar problems?
Any idea will be well welcomed.
Thanks
here is the full stack:
2016-01-15T15:13:28+0000 1.3.0.RELEASE INFO DeploymentSupervisor-0 zk.ZKStreamDeploymentHandler - Deployment status for stream 'partitioned': DeploymentStatus{state=failed,error(s)=org.springframework.amqp.AmqpIOException: java.io.IOException
at org.springframework.amqp.rabbit.support.RabbitExceptionTranslator.convertRabbitAccessException(RabbitExceptionTranslator.java:67)
at org.springframework.amqp.rabbit.connection.RabbitAccessor.convertRabbitAccessException(RabbitAccessor.java:110)
at org.springframework.amqp.rabbit.core.RabbitTemplate.doExecute(RabbitTemplate.java:1304)
at org.springframework.amqp.rabbit.core.RabbitTemplate.execute(RabbitTemplate.java:1271)
at org.springframework.amqp.rabbit.core.RabbitTemplate.execute(RabbitTemplate.java:1247)
at org.springframework.amqp.rabbit.core.RabbitAdmin.declareQueue(RabbitAdmin.java:166)
at org.springframework.xd.dirt.integration.rabbit.RabbitMessageBus.buildOutboundEndpoint(RabbitMessageBus.java:631)
at org.springframework.xd.dirt.integration.rabbit.RabbitMessageBus.bindProducer(RabbitMessageBus.java:611)
at org.springframework.xd.dirt.plugins.AbstractMessageBusBinderPlugin.bindMessageProducer(AbstractMessageBusBinderPlugin.java:288)
at org.springframework.xd.dirt.plugins.AbstractMessageBusBinderPlugin.bindConsumerAndProducers(AbstractMessageBusBinderPlugin.java:143)
at org.springframework.xd.dirt.plugins.stream.StreamPlugin.postProcessModule(StreamPlugin.java:73)
at org.springframework.xd.dirt.module.ModuleDeployer.postProcessModule(ModuleDeployer.java:238)
at org.springframework.xd.dirt.module.ModuleDeployer.doDeploy(ModuleDeployer.java:218)
at org.springframework.xd.dirt.module.ModuleDeployer.deploy(ModuleDeployer.java:200)
at org.springframework.xd.dirt.server.container.DeploymentListener.deployModule(DeploymentListener.java:365)
at org.springframework.xd.dirt.server.container.DeploymentListener.deployStreamModule(DeploymentListener.java:334)
at org.springframework.xd.dirt.server.container.DeploymentListener.onChildAdded(DeploymentListener.java:181)
at org.springframework.xd.dirt.server.container.DeploymentListener.childEvent(DeploymentListener.java:149)
at org.apache.curator.framework.recipes.cache.PathChildrenCache$5.apply(PathChildrenCache.java:509)
at org.apache.curator.framework.recipes.cache.PathChildrenCache$5.apply(PathChildrenCache.java:503)
at org.apache.curator.framework.listen.ListenerContainer$1.run(ListenerContainer.java:92)
at com.google.common.util.concurrent.MoreExecutors$DirectExecutorService.execute(MoreExecutors.java:299)
at org.apache.curator.framework.listen.ListenerContainer.forEach(ListenerContainer.java:83)
at org.apache.curator.framework.recipes.cache.PathChildrenCache.callListeners(PathChildrenCache.java:500)
at org.apache.curator.framework.recipes.cache.EventOperation.invoke(EventOperation.java:35)
at org.apache.curator.framework.recipes.cache.PathChildrenCache$10.run(PathChildrenCache.java:762)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException
at com.rabbitmq.client.impl.AMQChannel.wrap(AMQChannel.java:106)
at com.rabbitmq.client.impl.AMQChannel.wrap(AMQChannel.java:102)
at com.rabbitmq.client.impl.AMQChannel.exnWrappingRpc(AMQChannel.java:124)
at com.rabbitmq.client.impl.ChannelN.queueDeclare(ChannelN.java:837)
at com.rabbitmq.client.impl.ChannelN.queueDeclare(ChannelN.java:61)
at sun.reflect.GeneratedMethodAccessor136.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.springframework.amqp.rabbit.connection.CachingConnectionFactory$CachedChannelInvocationHandler.invoke(CachingConnectionFactory.java:704)
at com.sun.proxy.$Proxy93.queueDeclare(Unknown Source)
at org.springframework.amqp.rabbit.core.RabbitAdmin.declareQueues(RabbitAdmin.java:499)
at org.springframework.amqp.rabbit.core.RabbitAdmin.access$200(RabbitAdmin.java:57)
at org.springframework.amqp.rabbit.core.RabbitAdmin$3.doInRabbit(RabbitAdmin.java:169)
at org.springframework.amqp.rabbit.core.RabbitAdmin$3.doInRabbit(RabbitAdmin.java:166)
at org.springframework.amqp.rabbit.core.RabbitTemplate.doExecute(RabbitTemplate.java:1298)
... 30 more
Caused by: com.rabbitmq.client.ShutdownSignalException: channel error; protocol method: #method<channel.close>(reply-code=406, reply-text=PRECONDITION_FAILED - inequivalent arg 'x-dead-letter-exchange' for queue 'xdbus.partitioned.0-0' in vhost '/': received none but current is the value 'xdbus.DLX' of type 'longstr', class-id=50, method-id=10)
at com.rabbitmq.utility.ValueOrException.getValue(ValueOrException.java:67)
at com.rabbitmq.utility.BlockingValueOrException.uninterruptibleGetValue(BlockingValueOrException.java:33)
at com.rabbitmq.client.impl.AMQChannel$BlockingRpcContinuation.getReply(AMQChannel.java:348)
at com.rabbitmq.client.impl.AMQChannel.privateRpc(AMQChannel.java:221)
at com.rabbitmq.client.impl.AMQChannel.exnWrappingRpc(AMQChannel.java:118)
... 42 more
Caused by: com.rabbitmq.client.ShutdownSignalException: channel error; protocol method: #method<channel.close>(reply-code=406, reply-text=PRECONDITION_FAILED - inequivalent arg 'x-dead-letter-exchange' for queue 'xdbus.partitioned.0-0' in vhost '/': received none but current is the value 'xdbus.DLX' of type 'longstr', class-id=50, method-id=10)
at com.rabbitmq.client.impl.ChannelN.asyncShutdown(ChannelN.java:478)
at com.rabbitmq.client.impl.ChannelN.processAsync(ChannelN.java:315)
at com.rabbitmq.client.impl.AMQChannel.handleCompleteInboundCommand(AMQChannel.java:144)
at com.rabbitmq.client.impl.AMQChannel.handleFrame(AMQChannel.java:91)
at com.rabbitmq.client.impl.AMQConnection$MainLoop.run(AMQConnection.java:552)
... 1 more
}
It's a bug.
When binding the producer with partitioned streams we don't check to see if the queue already exists (as is done with non-partitioned streams).
I opened a JIRA Issue.
A work around is to disable AutoBindDLQ on the consumer side and manually configure dead-lettering with a policy.