I have the following app https://github.com/rengthp/nestjs-rabbitmq-microservice
But I get on Producer :
[Nest] 6156 - 04/03/2021, 13:28:53 [ClientProxy] Disconnected from RMQ. Trying to reconnect. +25469ms
[Nest] 6156 - 04/03/2021, 13:28:53 [ClientProxy] Object:
{
"err": {
"code": 406,
"classId": 50,
"methodId": 10
}
}
+3ms
and on Consumer:
111ms
[Nest] 1180 - 04/03/2021, 13:22:31 [Server] Disconnected from RMQ. Trying to reconnect. +1058ms
[Nest] 1180 - 04/03/2021, 13:22:37 [Server] Disconnected from RMQ. Trying to reconnect. +6006ms
[Nest] 1180 - 04/03/2021, 13:22:43 [Server] Disconnected from RMQ. Trying to reconnect. +6017ms
[Nest] 1180 - 04/03/2021, 13:22:49 [Server] Disconnected from RMQ. Trying to reconnect. +6026ms
What could be wrong? the server is working...
This error came up because you re-declare an existing queue with different parameters. The solution is simple. The option queue for both producers and consumers must be the same and all option parameters must be the same.
try to left time to api gateway to connect rabbit,
in your controller, use onApplicationBootstrap function
async onApplicationBootstrap() { await this._clientProxyUser.connect();}
you can read more here lifecycle events nestjs
For me changing the type of the RabbitMQ QUEUE I was using from transient to durable fixed it
Related
RabbitMQ 3.10.1
rabbitmq-diagnostics status
...
Config files
* /etc/rabbitmq/rabbitmq.config
...
rabbitmq.config:
[
{rabbit,
[
{heartbeat, 90}
]
}
].
RabbitMQ Management show 5s heartbeat
And log:
2022-05-13 19:56:43.235925+03:00 [error] <0.5979.0> closing AMQP connection <0.5979.0> (xxx.xxx.xxx.xxx:3555 -> xxx.xxx.xxx.xxx:5672):
2022-05-13 19:56:43.235925+03:00 [error] <0.5979.0> missed heartbeats from client, timeout: 5s
How to fix this?
Set the heartbeat to 90s in the client. Most clients are able to set the heartbeat (from the client). RabbitMQ will respect the heartbeat suggested by the client. More about that here: https://www.rabbitmq.com/heartbeats.html#heartbeats-timeout
I have an application that is pushing data into RabbitMQ and then some other apps are subscribing to the different exchanges.
But recently, I keep having errors like this after a few hours:
2020-07-09 12:45:12.670 [error] <0.23578.1> Error on AMQP connection <0.23578.1> (172.18.0.5:48230 ->
172.18.0.3:5672, vhost: '/', user: 'guest', state: running), channel 6:
operation basic.publish caused a connection exception unexpected_frame:
"expected content header for class 60, got non content"
2020-07-09 12:45:12.674 [info] <0.23578.1> closing AMQP connection <0.23578.1> (172.18.0.5:48230 ->
172.18.0.3:5672, vhost: '/'
On the client side, I get messages like this:
"Already closed: The AMQP operation was interrupted: AMQP close-reason, initiated by Peer,
code=505, text='UNEXPECTED_FRAME - expected content body, got non content body frame instead',
classId=60, methodId=40"
This is on a docker container.
What could this error be about?
You are sharing a channel for concurrent publishing, use below code
lock (ch) { ch.BasicPublish(); }
**
geting rabbitmq connection error as follows.
**
2019-07-11 13:14:51.147.AMQP Connection 127.0.0.1:5672> ERROR - TID[] UID[] MID[] CID[] - Channel shutdown: connection error; protocol method: #method(reply-code=541, reply-text=INTERNAL_ERROR, class-id=0, method-id=0)
2019-07-11 13:14:51.831.bulkNotificationContainer-100> WARN - TID[] UID[] MID[] CID[] - Consumer raised exception, processing can restart if the connection factory supports it
com.rabbitmq.client.ShutdownSignalException: connection error; protocol method: #method(reply-code=541, reply-text=INTERNAL_ERROR, class-id=0, method-id=0)
at com.rabbitmq.client.impl.AMQConnection.startShutdown(AMQConnection.java:742) ~[amqp-client-3.6.5.jar!/:na]
at com.rabbitmq.client.impl.AMQConnection.shutdown(AMQConnection.java:732) ~[amqp-client-3.6.5.jar!/:na]
at com.rabbitmq.client.impl.AMQConnection.handleConnectionClose(AMQConnection.java:671) ~[amqp-client-3.6.5.jar!/:na]
at com.rabbitmq.client.impl.AMQConnection.processControlCommand(AMQConnection.java:625) ~[amqp-client-3.6.5.jar!/:na]
at com.rabbitmq.client.impl.AMQConnection$1.processAsync(AMQConnection.java:102) ~[amqp-client-3.6.5.jar!/:na]
at com.rabbitmq.client.impl.AMQChannel.handleCompleteInboundCommand(AMQChannel.java:143) ~[amqp-client-3.6.5.jar!/:na]
at com.rabbitmq.client.impl.AMQChannel.handleFrame(AMQChannel.java:90) ~[amqp-client-3.6.5.jar!/:na]
at com.rabbitmq.client.impl.AMQConnection$MainLoop.run(AMQConnection.java:549) ~[amqp-client-3.6.5.jar!/:na]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_73]
My spring-boot service using rabbitmq to send push notification asynchronously. So push payload publish/received via rabbitmq then send on FCM. However, this connection is working for last one year without any problem. But this morning rabbitmq server is restarted as follows
rabbitmq stop
kill process bean ( related erlang )
rabbitmq start
after that we restart spring-boot service successfully, and health-api shows rabbit-up status.
but having error when trying to send push.
application properties configuration.
spring.rabbitmq.host=127.0.0.1 spring.rabbitmq.port=5672
spring.rabbitmq.username=rabbitadmin spring.rabbitmq.password=admin
custom configuration
rabbitmq.listeners.retry-policy = UNIFORM_RANDOM_DELAY
rabbitmq.listeners.max-interval=15000
rabbitmq.listener.push-router.concurrent-consumers=2
rabbitmq.listener.push-router.max-concurrent-consumers=10
rabbitmq.binding.push-notification.queue.name=pushqueue
rabbitmq.binding.push-notification.exchange.name=pushexchange
rabbitmq.binding.push-notification.binding.routing-key=pushroute-binding
I want to publish/receive data to rabbitmq channel.
This issue is resolved after reinstalling the packages of rabbitmq.
Still dont know why this happened..
I am try to replicate the SASL_PLAIN or SASL_SSL authentication described at: http://docs.confluent.io/3.0.0/kafka/sasl.html#sasl-configuration-for-kafka-brokers
In config/server.properties, I added the following 4 lines:
listeners=SASL_SSL://localhost:9092
security.inter.broker.protocol=SASL_SSL
sasl.mechanism.inter.broker.protocol=PLAIN
sasl.enabled.mechanisms=PLAIN
In config/producer.properties, I added the following two lines:
security.protocol=SASL_SSL
sasl.mechanism=PLAIN
Then I set the following environment variable in the server terminal:
KAFKA_OPTS=/home/kafka/kafka_server_jaas.conf
This file has the following content:
KafkaServer {
org.apache.kafka.common.security.plain.PlainLoginModule required
username="admin"
password="admin-secret"
user_admin="admin-secret"
user_alice="alice-secret";
};
And in the producer terminal I define the following env variable:
KAFKA_OPTS=/home/kafka/kafka_client_jaas.conf
And this file has the following content:
KafkaClient {
org.apache.kafka.common.security.plain.PlainLoginModule required
username="alice"
password="alice-dsecret";
};
I start the server with the following command:
./bin/kafka-server-start.sh config/server.properties
And the producer with following command:
bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test
Both start without problems. But, as soon as I type something on the producer console, I get the following message that keeps scrolling:
WARN Bootstrap broker localhost:9092 disconnected (org.apache.kafka.clients.NetworkClient)
Bootstrap broker localhost:9092 disconnected (org.apache.kafka.clients.NetworkClient)
WARN Bootstrap broker localhost:9092 disconnected (org.apache.kafka.clients.NetworkClient)
WARN Bootstrap broker localhost:9092 disconnected (org.apache.kafka.clients.NetworkClient)
WARN Bootstrap broker localhost:9092 disconnected (org.apache.kafka.clients.NetworkClient)
WARN Bootstrap broker localhost:9092 disconnected (org.apache.kafka.clients.NetworkClient)
WARN Bootstrap broker localhost:9092 disconnected (org.apache.kafka.clients.NetworkClient)
WARN Bootstrap broker localhost:9092 disconnected (org.apache.kafka.clients.NetworkClient)
If I remove the security configuration from the server and the producer configuration, everything works as expected. I am using Kafka 0.10.0.1.
UPDATE:
I did some more investigations, turning log levels to DEBUG on server reveals something weird. As soon as I specify the listeners field in server.properties, the server goes in a weird state. It establishes connection to itsself that it cannot authenticate. The protocol in this case was SASL_PLAINTEXT.
The logs as here:
2016-09-15 21:43:02 DEBUG SaslClientAuthenticator:204 - Set SASL client state to RECEIVE_HANDSHAKE_RESPONSE
2016-09-15 21:43:02 DEBUG NetworkClient:476 - Completed connection to node 0
2016-09-15 21:43:02 DEBUG Acceptor:52 - Accepted connection from /127.0.0.1 on /127.0.0.1:9092. sendBufferSize [actual|requested]: [102400|102400] recvBufferSize [actual|requested]: [102400|102400]
2016-09-15 21:43:02 DEBUG Processor:52 - Processor 2 listening to new connection from /127.0.0.1:42815
2016-09-15 21:43:02 DEBUG SaslServerAuthenticator:269 - Set SASL server state to HANDSHAKE_REQUEST
2016-09-15 21:43:02 DEBUG SaslServerAuthenticator:310 - Handle Kafka request SASL_HANDSHAKE
2016-09-15 21:43:02 DEBUG SaslServerAuthenticator:354 - Using SASL mechanism 'PLAIN' provided by client
2016-09-15 21:43:02 DEBUG SaslServerAuthenticator:269 - Set SASL server state to AUTHENTICATE
2016-09-15 21:43:02 DEBUG SaslClientAuthenticator:204 - Set SASL client state to INITIAL
2016-09-15 21:43:02 DEBUG SaslClientAuthenticator:204 - Set SASL client state to INTERMEDIATE
2016-09-15 21:43:02 DEBUG SaslServerAuthenticator:269 - Set SASL server state to FAILED
2016-09-15 21:43:02 DEBUG Selector:345 - Connection with /127.0.0.1 disconnected
java.io.IOException: javax.security.sasl.SaslException: Authentication failed: Invalid JAAS configuration [Caused by javax.security.sasl.SaslException: Authentication failed: Invalid username or password]
at org.apache.kafka.common.security.authenticator.SaslServerAuthenticator.authenticate(SaslServerAuthenticator.java:243)
at org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:64)
at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:318)
at org.apache.kafka.common.network.Selector.poll(Selector.java:283)
at kafka.network.Processor.poll(SocketServer.scala:472)
There is absolutely no other client or server running. This is one server talking to himself.
Any thoughts?
Help came from the Kafka forum. See http://mail-archives.apache.org/mod_mbox/kafka-users/201609.mbox/%3CCAHX2Snk11vg7DXNVUr9oE97ikFSQUoT3kBLAxYymEDj7E14XrQ%40mail.gmail.com%3E
I had the credentials wrong. They were:
KafkaServer {
org.apache.kafka.common.security.plain.PlainLoginModule required
username="admin"
password="admin-secret"
user_admin="alice-secret"
user_alice="alice-secret";
};
Instead of:
KafkaServer {
org.apache.kafka.common.security.plain.PlainLoginModule required
username="admin"
password="admin-secret"
user_admin="admin-secret"
user_alice="alice-secret";
};
Also, the console consumer needs to be called in a certain. First the flag --new-consumer should be provided. Second, bootstrap server should be specified. Leading to this:
bin/kafka-console-consumer.sh --new-consumer --zookeeper localhost:2181 --topic test --from-beginning --consumer.config=config/consumer.properties --bootstrap-server=localhost:9092
I'm having difficulty configuring my connection to CloudAMQP in my deployed grails application. I can run the application locally against a locally installed RabbitMQ instance but can't figure out how to correctly define my application to run on CloudBees using the CloudAMQP service.
In my Config.groovy, I'm defining my connection info and a queue:
rabbitmq {
connectionfactory {
username = 'USERNAME'
password = 'PASSWORD'
hostname = 'lemur.cloudamqp.com'
}
queues = {
testQueue autoDelete: false, durable: false, exclusive: false
}
}
When the application starts and tries to connect, I see the following log messages:
2013-08-23 21:29:59,195 [main] DEBUG listener.SimpleMessageListenerContainer - Starting Rabbit listener container.
2013-08-23 21:29:59,205 [SimpleAsyncTaskExecutor-1] DEBUG listener.BlockingQueueConsumer - Starting consumer Consumer: tag=[null], channel=null, acknowledgeMode=AUTO local queue size=0
2013-08-23 21:30:08,405 [SimpleAsyncTaskExecutor-1] WARN listener.SimpleMessageListenerContainer - Consumer raised exception, processing can restart if the connection factory supports it
org.springframework.amqp.AmqpIOException: java.io.IOException
at org.springframework.amqp.rabbit.connection.RabbitUtils.convertRabbitAccessException(RabbitUtils.java:112)
at org.springframework.amqp.rabbit.connection.AbstractConnectionFactory.createBareConnection(AbstractConnectionFactory.java:163)
at org.springframework.amqp.rabbit.connection.CachingConnectionFactory.createConnection(CachingConnectionFactory.java:228)
at org.springframework.amqp.rabbit.connection.ConnectionFactoryUtils$1.createConnection(ConnectionFactoryUtils.java:119)
at org.springframework.amqp.rabbit.connection.ConnectionFactoryUtils.doGetTransactionalResourceHolder(ConnectionFactoryUtils.java:163)
at org.springframework.amqp.rabbit.connection.ConnectionFactoryUtils.getTransactionalResourceHolder(ConnectionFactoryUtils.java:109)
at org.springframework.amqp.rabbit.listener.BlockingQueueConsumer.start(BlockingQueueConsumer.java:199)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$AsyncMessageProcessingConsumer.run(SimpleMessageListenerContainer.java:524)
at java.lang.Thread.run(Unknown Source)
Caused by: java.io.IOException
at com.rabbitmq.client.impl.AMQChannel.wrap(AMQChannel.java:106)
at com.rabbitmq.client.impl.AMQChannel.wrap(AMQChannel.java:102)
at com.rabbitmq.client.impl.AMQChannel.exnWrappingRpc(AMQChannel.java:124)
at com.rabbitmq.client.impl.AMQConnection.start(AMQConnection.java:381)
at com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:516)
at com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:545)
Caused by: com.rabbitmq.client.ShutdownSignalException: connection error; reason: java.net.SocketException: Connection reset
at com.rabbitmq.utility.ValueOrException.getValue(ValueOrException.java:67)
at com.rabbitmq.utility.BlockingValueOrException.uninterruptibleGetValue(BlockingValueOrException.java:33)
at com.rabbitmq.client.impl.AMQChannel$BlockingRpcContinuation.getReply(AMQChannel.java:343)
at com.rabbitmq.client.impl.AMQChannel.privateRpc(AMQChannel.java:216)
at com.rabbitmq.client.impl.AMQChannel.exnWrappingRpc(AMQChannel.java:118)
... 3 more
Caused by: java.net.SocketException: Connection reset
at com.rabbitmq.client.impl.Frame.readFrom(Frame.java:95)
at com.rabbitmq.client.impl.SocketFrameHandler.readFrame(SocketFrameHandler.java:131)
at com.rabbitmq.client.impl.AMQConnection$MainLoop.run(AMQConnection.java:508)
2013-08-23 21:30:08,406 [SimpleAsyncTaskExecutor-1] INFO listener.SimpleMessageListenerContainer - Restarting Consumer: tag=[null], channel=null, acknowledgeMode=AUTO local queue size=0
2013-08-23 21:30:08,406 [SimpleAsyncTaskExecutor-1] DEBUG listener.BlockingQueueConsumer - Closing Rabbit Channel: null
2013-08-23 21:30:08,407 [SimpleAsyncTaskExecutor-2] DEBUG listener.BlockingQueueConsumer - Starting consumer Consumer: tag=[null], channel=null, acknowledgeMode=AUTO local queue size=0
Aug 23, 2013 9:30:11 PM org.apache.catalina.core.ApplicationContext log
INFO: Initializing Spring FrameworkServlet 'grails'
Aug 23, 2013 9:30:11 PM org.apache.coyote.http11.Http11Protocol init
INFO: Initializing Coyote HTTP/1.1 on http-8634
Aug 23, 2013 9:30:11 PM org.apache.coyote.http11.Http11Protocol start
INFO: Starting Coyote HTTP/1.1 on http-8634
According to https://developer.cloudbees.com/bin/view/RUN/CloudAMQP
when you bind your CloudAMQP service to your app - some config params are provided in the pattern of CLOUDAMQP_URL_ - this is the type of thing you would need to put in your config files so they can be wired in when the app launches.
Make sure to specify the virtualHost for CloudAMQP connections. That worked for me.