Why SpringBoot RabbitMQ client auto shutdown connection to rabbitmq server - rabbitmq

**
geting rabbitmq connection error as follows.
**
2019-07-11 13:14:51.147.AMQP Connection 127.0.0.1:5672> ERROR - TID[] UID[] MID[] CID[] - Channel shutdown: connection error; protocol method: #method(reply-code=541, reply-text=INTERNAL_ERROR, class-id=0, method-id=0)
2019-07-11 13:14:51.831.bulkNotificationContainer-100> WARN - TID[] UID[] MID[] CID[] - Consumer raised exception, processing can restart if the connection factory supports it
com.rabbitmq.client.ShutdownSignalException: connection error; protocol method: #method(reply-code=541, reply-text=INTERNAL_ERROR, class-id=0, method-id=0)
at com.rabbitmq.client.impl.AMQConnection.startShutdown(AMQConnection.java:742) ~[amqp-client-3.6.5.jar!/:na]
at com.rabbitmq.client.impl.AMQConnection.shutdown(AMQConnection.java:732) ~[amqp-client-3.6.5.jar!/:na]
at com.rabbitmq.client.impl.AMQConnection.handleConnectionClose(AMQConnection.java:671) ~[amqp-client-3.6.5.jar!/:na]
at com.rabbitmq.client.impl.AMQConnection.processControlCommand(AMQConnection.java:625) ~[amqp-client-3.6.5.jar!/:na]
at com.rabbitmq.client.impl.AMQConnection$1.processAsync(AMQConnection.java:102) ~[amqp-client-3.6.5.jar!/:na]
at com.rabbitmq.client.impl.AMQChannel.handleCompleteInboundCommand(AMQChannel.java:143) ~[amqp-client-3.6.5.jar!/:na]
at com.rabbitmq.client.impl.AMQChannel.handleFrame(AMQChannel.java:90) ~[amqp-client-3.6.5.jar!/:na]
at com.rabbitmq.client.impl.AMQConnection$MainLoop.run(AMQConnection.java:549) ~[amqp-client-3.6.5.jar!/:na]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_73]
My spring-boot service using rabbitmq to send push notification asynchronously. So push payload publish/received via rabbitmq then send on FCM. However, this connection is working for last one year without any problem. But this morning rabbitmq server is restarted as follows
rabbitmq stop
kill process bean ( related erlang )
rabbitmq start
after that we restart spring-boot service successfully, and health-api shows rabbit-up status.
but having error when trying to send push.
application properties configuration.
spring.rabbitmq.host=127.0.0.1 spring.rabbitmq.port=5672
spring.rabbitmq.username=rabbitadmin spring.rabbitmq.password=admin
custom configuration
rabbitmq.listeners.retry-policy = UNIFORM_RANDOM_DELAY
rabbitmq.listeners.max-interval=15000
rabbitmq.listener.push-router.concurrent-consumers=2
rabbitmq.listener.push-router.max-concurrent-consumers=10
rabbitmq.binding.push-notification.queue.name=pushqueue
rabbitmq.binding.push-notification.exchange.name=pushexchange
rabbitmq.binding.push-notification.binding.routing-key=pushroute-binding
I want to publish/receive data to rabbitmq channel.

This issue is resolved after reinstalling the packages of rabbitmq.
Still dont know why this happened..

Related

Kafka-s3-connect killed instantly after start

I want to connect aws-Kafka with s3 using confluence connector on my ec2 server. I try to configure everything like in tutorials. When I run connect-standalone or connect-distributed, at first everything goes well, I don't get any errors in the logs but after information about connection starting, my connector died instantly without any information. Has anybody got same problem?
config/connect-standalone.properties
bootstrap.servers=msk-connection-string
plugin.path=/home/ubuntu/connectors/confluentinc-kafka-connect-s3
key.converter=org.apache.kafka.connect.converters.ByteArrayConverter
value.converter=org.apache.kafka.connect.converters.ByteArrayConverter
offset.storage.file.filename=/tmp/connect.offsets
connector.properties
connector.class=io.confluent.connect.s3.S3SinkConnector
format.class=io.confluent.connect.s3.format.bytearray.ByteArrayFormat
flush.size=1
topics=SomeTopic
s3.bucket.name=bucket-name-here
s3.region=us-west-2
s3.part.size=5242880
aws.access.key.id=****
aws.secret.access.key=****
behavior.on.null.values=ignore
storage.class=io.confluent.connect.s3.storage.S3Storage
topics.dir=../topics
store.url=http://bucket-name.s3-website-Region.amazonaws.com
key.converter=org.apache.kafka.connect.converters.ByteArrayConverter
value.converter=org.apache.kafka.connect.converters.ByteArrayConverter
logs:
[2021-08-20 06:32:35,954] INFO Kafka version: 2.7.0 (org.apache.kafka.common.utils.AppInfoParser:119)
[2021-08-20 06:32:35,954] INFO Kafka commitId: 448719dc99a19793 (org.apache.kafka.common.utils.AppInfoParser:120)
[2021-08-20 06:32:35,954] INFO Kafka startTimeMs: 1629441155953 (org.apache.kafka.common.utils.AppInfoParser:121)
Killed
Please help!
MSK requires TLS connection
When adding few lines with ssl configuration to config/connect-standalone.properties
producer.security.protocol=SSL
consumer.security.protocol=SSL
security.protocol=SSL
ssl.protocol=TLS
ssl.truststore.location=/your/path/to/truststore/kafka.client.truststore.jks
It starts working properly!

ActiveMQ DB persistent does not reconnect when HA cluster DB failover occurs

Whenever HA failover occurs activeMQ broker got stuck with the messages that has and could not restart by itself.
Messages are processed successfully when we restart the activeMQ.
The bean is created to stop and start the connectors in case if IOExceptions.
bean id="ioExceptionHandler" class="org.apache.activemq.util.DefaultIOExceptionHandler"
property name="ignoreSQLExceptions"value=false property
property name="stopStartConnectors" value=true property
bean
We are getting connection closed as exceptions when this failover occurs as below
Initiating stop/restart of transports on BrokerService[localhost] due to IO exception, java.io.IOException: The connection is closed. | org.apache.activemq.util.DefaultIOExceptionHandler | ActiveMQ Transport: tcp:///hostname:52272#8501
java.io.IOException: The connection is closed.
Later it is trying to restart the transport connectors as a result of this config, but not able to continue further.
INFO | waiting for broker persistence adapter checkpoint to succeed
before restarting transports |
org.apache.activemq.util.DefaultIOExceptionHandler |
IOExceptionHandler: restart transports.
Please let us know if any configuration required for broker to restart and process the messages it has.

ActiveMQ vs AMQP .NET Lite versions

ActiveMQ, as per line 151 in AmqpWireFormat, refuses connections with a AMQP protocol version other than 1.0.0.
AMQP .NET Lite, as per line 411 in Connection, sends 0.1.0.0. At the same time it claims to do Full control of AMQP 1.0 protocol behavior.
Obviously ActiveMQ refuses connections from AMQP .NET Lite with "Connection attempt from non AMQP v1.0 client. AMQP,0,1,0,0".
The client does:
Address address = new Address("amqp://localhost:5672");
Connection connection = new Connection(address);
Session session = new Session(connection);
And ActiveMQ logs:
2017-03-28 09:11:28,403 | DEBUG | Transport Connection to: tcp://0:0:0:0:0:0:0:1:54352 failed: org.apache.activemq.transport.amqp.AmqpProtocolException: Connection from client using unsupported AMQP attempted | org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ Transport: tcp:///0:0:0:0:0:0:0:1:54352#5672
org.apache.activemq.transport.amqp.AmqpProtocolException: Connection from client using unsupported AMQP attempted
at org.apache.activemq.transport.amqp.protocol.AmqpConnection.onAMQPData(AmqpConnection.java:339)[activemq-amqp-5.14.4.jar:5.14.4]
at org.apache.activemq.transport.amqp.AmqpProtocolDiscriminator.onAMQPData(AmqpProtocolDiscriminator.java:96)[activemq-amqp-5.14.4.jar:5.14.4]
at org.apache.activemq.transport.amqp.AmqpTransportFilter.onCommand(AmqpTransportFilter.java:107)[activemq-amqp-5.14.4.jar:5.14.4]
at org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)[activemq-client-5.14.4.jar:5.14.4]
at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:233)[activemq-client-5.14.4.jar:5.14.4]
at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:215)[activemq-client-5.14.4.jar:5.14.4]
at java.lang.Thread.run(Thread.java:745)[:1.8.0_112]
On the command line it says:
WARN | Connection attempt from non AMQP v1.0 client. AMQP,0,1,0,0
ActiveMQ then aborts the connection.
The ActiveMQ configuration includes:
<transportConnectors>
<transportConnector name="amqp" uri="amqp://0.0.0.0:5672" />
</transportConnectors>
This all looks to me like ActiveMQ expects version 1.0.0.0 and amqpnetlite sends 0.1.0.0.
Where to, from here?
By default ActiveMQ requires authentication. try
string queueName = "q1";
Connection connection = new Connection(new Address("amqp://user:pwd#localhost:5672"));
Session session = new Session(connection);
SenderLink sender = new SenderLink(session, "sender", queueName);
Message message = new Message("Hello World");
sender.Send(message);
connection.Close();
The user name and password should be in users.properties config file. Also it is recommended to use async API to avoid blocking I/O calls.
Both ActiveMQ and AMQP .NET Lite support the ISO/IEC Standard 1.0 version of AMQP. The protocol header "AMQP0100" indicates version 1.0 of the protocol. AMQP .NET Lite is known to work with ActiveMQ so I suspect you have some other configuration issue.
My assumption is that you haven't turned off the broker's enforcement of requiring the client to connect via a SASL handshake either via SASL anonymous or SASL plain depending on whether you've configured a real authentication plugin or not. I would bet that if you configured the broker by adding the options wireFormat.allowNonSaslConnections=true that it would probably work.
From the information you've provided the .NET client is not using SASL so the broker is rejecting it. The SASL header would be "AMQP3100"

activemq failover using multiple instances in master slave mode on same linux machine

I have setup ActiveMQ mulitple instances to achieve failover in master slave mode in windows.
While setting up the same i just created 3 instances under bin folder without changing any port and started all 3 instances one by one. First instance became master and remaining were in slave mode until I stopped master instance.
Now I am trying to achieve the same in Linux environment. First instance starts successfully but when I start second instance in a different window it throws below error:
ERROR | Failed to start Apache ActiveMQ ([instance2, ID:132vm6-57227-1478597606120-0:1], java.io.IOException: Transport Connector could not be registered in JMX: java.io.IOException: Failed to bind to server socket: tcp://0.0.0.0:61616?maximumConnections=1000&wireFormat.maxFrameSize=104857600 due to: java.net.BindException: Address already in use)
INFO | Apache ActiveMQ 5.14.0 (instance2, ID:132vm6-57227-1478597606120-0:1) is shutting down
INFO | Connector openwire stopped
INFO | Connector amqp stopped
INFO | Connector stomp stopped
INFO | Connector mqtt stopped
INFO | Connector ws stopped
INFO | PListStore:[/opt/apache-activemq-5.14.0/bin/instance2/data/instance2/tmp_storage] stopped
INFO | Stopping async queue tasks
INFO | Stopping async topic tasks
INFO | Stopped KahaDB
INFO | Apache ActiveMQ 5.14.0 (instance2, ID:132vm6-57227-1478597606120-0:1) uptime 0.585 seconds
INFO | Apache ActiveMQ 5.14.0 (instance2, ID:132vm6-57227-1478597606120-0:1) is shutdown
INFO | Closing org.apache.activemq.xbean.XBeanBrokerFactory$1#4233871a: startup date [Tue Nov 08 15:03:24 IST 2016]; root of context hierarchy
WARN | Exception thrown from LifecycleProcessor on context close
java.lang.IllegalStateException: LifecycleProcessor not initialized - call 'refresh' before invoking lifecycle methods via the context: org.apache.activemq.xbean.XBeanBrokerFactory$1#4233871a: startup date [Tue Nov 08 15:03:24 IST 2016]; root of context hierarchy
at org.springframework.context.support.AbstractApplicationContext.getLifecycleProcessor(AbstractApplicationContext.java:357)[spring-context-4.1.9.RELEASE.jar:4.1.9.RELEASE]
at org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:884)[spring-context-4.1.9.RELEASE.jar:4.1.9.RELEASE]
at org.springframework.context.support.AbstractApplicationContext.close(AbstractApplicationContext.java:843)[spring-context-4.1.9.RELEASE.jar:4.1.9.RELEASE]
at org.apache.activemq.hooks.SpringContextHook.run(SpringContextHook.java:30)[activemq-spring-5.14.0.jar:5.14.0]
at org.apache.activemq.broker.BrokerService.stop(BrokerService.java:875)[activemq-broker-5.14.0.jar:5.14.0]
at org.apache.activemq.xbean.XBeanBrokerService.stop(XBeanBrokerService.java:122)[activemq-spring-5.14.0.jar:5.14.0]
at org.apache.activemq.broker.BrokerService.start(BrokerService.java:629)[activemq-broker-5.14.0.jar:5.14.0]
at org.apache.activemq.xbean.XBeanBrokerService.afterPropertiesSet(XBeanBrokerService.java:73)[activemq-spring-5.14.0.jar:5.14.0]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)[:1.7.0_65]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)[:1.7.0_65]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)[:1.7.0_65]
at java.lang.reflect.Method.invoke(Method.java:606)[:1.7.0_65]
I am using ActiveMQ 5.14 version.
If anybody has encountered a similar issue, kindly provide your inputs.
To get multiple instances of ActiveMQ running on the same machine, you need to change the ports that they try to open. There are (at least) 3 ports that need to be changed:
The transportConnector ports that accept messaging traffic. These are defined in theactivemq.xml file. Typically you only need the openwire one - this is 61616 by default; I usually change this in the other ActiveMQ instances to 61626, 61636 etc. You can usually comment out the others if you don't intend to use them.
The Jetty HTTP port. This is defined in the jetty.xml file. The default is 8161, set the next ones to 8162, 8163 etc.
The JMX port. This one's a bit tricky, as you need to stick a piece of config into the activemq.xml to explicitly define it as follows:
<managementContext>
<managementContext createConnector="true" connectorPort="1099"/>
</managementContext>
You can then change this to 1199, 1299 on the other instances. Hope this helps.

ActiveMQ Master/Slave on Weblogic - vm transport issue

I am trying to configure ActiveMQ master/slave setup on a single WebLogic machine. The problem is when I start Managed Server1 it successfully connects to vm transport and everything works perfectly, but when I start Managed Server2 I am receiving the following errors in broker logs
INFO 2016-September-27 10:08:00,227 ActiveMQEndpointWorker:124 - Connection attempt already in progress, ignoring connection exception
INFO 2016-September-27 10:08:01,161 TransportConnector:260 - Connector vm://localhost started
INFO 2016-September-27 10:08:30,228 TransportConnector:291 - Connector vm://localhost stopped
INFO 2016-September-27 10:08:30,229 TransportConnector:260 - Connector vm://localhost started
WARN 2016-September-27 10:08:30,228 ActiveMQManagedConnection:385 - Connection failed: javax.jms.JMSException: peer (vm://localhost#61) stopped.
WARN 2016-September-27 10:08:30,231 TransportConnection:823 - Failed to add Connection ID:ndl-wls-300.mydomain.com-52251-1474966937425-65:1 due to java.lang.NullPointerException
ERROR 2016-September-27 10:08:30,233 ActiveMQEndpointWorker:183 - Failed to connect to broker [vm://localhost?create=false]: java.lang.NullPointerException
javax.jms.JMSException: java.lang.NullPointerException
Please help, I am stuck with this.
I still don't see the reason for the slave within the same VM. I suggest you reach out to an ActiveMQ expert consultant to validate your architecture.
However, I think I can help you move a little bit closer to this issue:
There is a fundamental miss understanding here.. the vm url is broken down like this:
vm://${brokerName}?option=value,etc
The first time you create vm://localhost?create=true.. you have created a broker
The second time you reference vm://localhost?create=false.. you have created a client connection to the first broker.
To get two brokers, you'd need two different vm://${brokerName}?create=true