Performance using Spring RabbitMQ is slow - rabbitmq

I am trying to send a 51MB of data over 13 threads to another machine in a gap of 30 seconds. It means, at every 30 seconds, 51*13 = 663Mb of data is sent in a burst to the target machine.
Steps Followed:
I have used shovels between the sender and the receiver rabbitmq server to transmit the messages.
Both sender and receiver have their own rabbitMq server running.
I am using RabbitTemplate.send(routingKey,51mbMessage) to send the message.
I am using OnMessageListener on the target machine to receive the messages.
OnMessageListener at Receiver side:
<rabbit:listener-container connection-factory="connectionFactory" prefetch="10" >
<rabbit:listener ref="onMessageReceiverforMyTestQueue" queue- names="MyTestQueue" />
</rabbit:listener-container>
<bean id = "onMessageReceiverforLoadTestQueue" class="MyPackage.MyMessageListener">
</bean>
ConnectionFactory at the receiver side:
class="org.springframework.amqp.rabbit.connection.CachingConnectionFactory">
<constructor-arg value="***.**.**.**"/>
<property name="username" value="abcd"/>
<property name="password" value="abcd"/>
<property name="channelCacheSize" value="50"/>
Observations Made:
The messages are taking lot of time to reach the target machine.I am getting delay of about 10-15 minutes.
Can you please help me in overcoming this delay. I would really appreciate any help offered here. Thanks in advance.

Related

How to put the message got from jdbc to rabbitmq?

In the spring integration reference on JDBC Support
it has the following code to get the records out of a table and convert them into message and sent them to a channel. But my system is rabbitmq so I want the message from jdbc stored in rabbitmq and consumed by rabbitmq client. How to do that using spring integration and spring rabbitmq?
<int-jdbc:inbound-channel-adapter query="select * from item where status=2"
channel="target" data-source="dataSource"
update="update item set status=10 where id in (:id)" />
I know there are AMQP Backed Message Channels, and defined as below.
<int-amqp:channel id="p2pChannel"/>
But how to put them together? So that the message created from a table will be stored in a rabbitmq queue?
update: I used the following code:
<bean id="dataSource" class="org.springframework.jdbc.datasource.DriverManagerDataSource">
<property name="driverClassName" value="org.postgresql.Driver"/>
<property name="url" value="jdbc:postgresql://localhost/example"/>
<property name="username" value="postgres"/>
<property name="password" value="oracle"/>
</bean>
<int-amqp:channel id="myMessageChannel_test"/>
<int-jdbc:inbound-channel-adapter query="select * from my_files where message_created=0"
channel="myMessageChannel_test" data-source="dataSource"
update="update my_files set message_created=1 where file_id in (:file_id)">
<int:poller fixed-rate="1000">
<int:transactional/>
</int:poller>
</int-jdbc:inbound-channel-adapter>
But it complains:
at java.lang.Thread.run(Thread.java:745)
Caused by: org.springframework.messaging.MessageDeliveryException: Dispatcher has no subscribers for amqp-channel 'myMessageChannel_test'.; nested exception is org.springframework.integration.MessageDispatchingException: Dispatcher has no subscribers
at org.springframework.integration.amqp.channel.AbstractSubscribableAmqpChannel$DispatchingMessageListener.onMessage(AbstractSubscribableAmqpChannel.java:197)
at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:737)
... 10 common frames omitted
Caused by: org.springframework.integration.MessageDispatchingException: Dispatcher has no subscribers
at org.springframework.integration.dispatcher.UnicastingDispatcher.doDispatch(UnicastingDispatcher.java:107)
at org.springframework.integration.dispatcher.UnicastingDispatcher.dispatch(UnicastingDispatcher.java:97)
at org.springframework.integration.amqp.channel.AbstractSubscribableAmqpChannel$DispatchingMessageListener.onMessage(AbstractSubscribableAmqpChannel.java:181)
... 11 common frames omitted
2014-12-30 19:23:18.985 WARN 29203 --- [cTaskExecutor-1] s.a.r.l.ConditionalRejectingErrorHandler : Execution of Rabbit message listener failed.
org.springframework.amqp.rabbit.listener.exception.ListenerExecutionFailedException: Listener threw exception
at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.wrapToListenerExecutionFailedExceptionIfNeeded(AbstractMessageListenerContainer.java:802)
at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:740)
at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:628)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.access$001(SimpleMessageListenerContainer.java:82)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$1.invokeListener(SimpleMessageListenerContainer.java:167)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.invokeListener(SimpleMessageListenerContainer.java:1196)
at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.executeListener(AbstractMessageListenerContainer.java:600)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.doReceiveAndExecute(SimpleMessageListenerContainer.java:960)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.receiveAndExecute(SimpleMessageListenerContainer.java:944)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.access$700(SimpleMessageListenerContainer.java:82)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$AsyncMessageProcessingConsumer.run(SimpleMessageListenerContainer.java:1058)
So your channel "myMessageChannel_test" has messages from database but that needs some config like below to be able to receive/handle/subscribe to messages.
There are many ways to do it - one way as below
<rabbit:connection-factory id="rabbitConnFactory" addresses="${rabbit.addresses}"
virtual-host="${rabbit.host}" username="${rabbit.username}" password="${rabbit.password}"/>
<rabbit:admin id="admin" connection-factory="rabbitConnFactory" />
<rabbit:template id="amqpTemplate" connection-factory="rabbitConnFactory" routing-key="some.key" exchange="some.exchange" queue="some.queue" />
<int-amqp:outbound-channel-adapter routing-key="some.key" amqp-template="amqpTemplate"
channel="myMessageChannel_test" />
You need to do respective rabbit configs as well

How to monitor PooledConnectionFactory (via JMX?)

I have a client app that is consuming from a queue in an activemq cluster. The app is running in tomcat 7 and uses camel (v2.10.3) and spring 3.1.2. I use a PooledConnectionFactory to connect.
Everything works for a while (sometimes days), but then all of the connections go away in the pool (the activemq broker web console shows no consumers. I figured it was the idletimeout issue, but adding the suggested config didn't help. I also upgraded to activemq-pool-5.10.0.jar, but also no luck.
SO, I'm trying to find out what is going on and was hoping to use JMX, but I can not find any related mbeans (via jconsole) that the pool registers. Is there a way to monitor/control the pool via JMX (or another/better way)?
My config fyi:
<bean id="jmsConnectionFactory" class="org.apache.activemq.ActiveMWSslConnectionFactory">
<property name="brokerURL" value="failover://ssl://...."/>
</bean>
<bean id="pooledConnectionFactory" class="org.apache.activemq.pool.PooledConnectionFactory" init-method="start" destroy-method="stop">
<property name="connectionFactory" ref="jmsConnectionFactory"/>
<property name="idleTimeout" value="0"/>
</bean>
As simple as it sounds, I don't see any other option other than to turn on TRACE level logging for that class. Check out the logs of this question.

ActiveMQ LeaseDatabaseLocker warning for lease duration

I have a slightly strange warning from ActiveMQ 5.9.0 with JDBC Oracle backed persistence...
WARN [org.apache.activemq.store.jdbc.LeaseDatabaseLocker] LockableService
keep alive period: 2000, which renews the lease, is less than
lockAcquireSleepInterval: 1000, the lease duration.
These values will allow the lease to expire.
My question is why is LockableService reporting that 2000 < 1000? I think it should say "LockableService keep alive period: 2000, which renews the lease, is greater than lockAcquireSleepInterval: 1000, the lease duration. These values will allow the lease to expire.". What do you think, maybe I'm reading this wrong...
I do see a problem with my current settings (I have a Master and Slave, I shutdown the Master and the Slave takes over, but I startup the Master again and it doesn't become a Slave)... So I obviously need to tweak my settings, here is the current relevant configuration...
<bean id="jdbcPersistenceAdapter" class="org.apache.activemq.store.jdbc.JDBCPersistenceAdapter">
<property name="brokerName" value="messageCentreBroker" />
<property name="createTablesOnStartup" value="true" />
<property name="dataSource" ref="activeMqDataSource" />
<property name="lockKeepAlivePeriod" value="2000" />
<property name="locker" ref="leaseDatabaseLocker" />
</bean>
<bean id="leaseDatabaseLocker" class="org.apache.activemq.store.jdbc.LeaseDatabaseLocker">
<property name="lockAcquireSleepInterval" value="1000" />
</bean>
So I guess my lockAcquireSleepInterval should be greater than 2000? I'll try this, but interested to hear thoughts on the WARN message too, it seems wrong?
I think it's just a typo. Nothing more.

Properly Shutting Down ActiveMQ and Spring DefaultMessageListenerContainer

Our system will not shutdown when a "Stop" command is issued from the Tomcat Manager. I have determined that it is related to ActiveMQ/Spring. I have even figured out how to get it to shutdown, however my solution is a hack (at least I hope this isn't the "correct" way to do it). I would like to know the proper way to shutdown ActiveMQ so that I can remove my hack.
I inherited this component and I have no information about why certain architectural decisions were made, after a lot of digging I think I understand his thoughts, but I could be missing something. In other words, the real problem could be in the way that we are trying to use ActiveMQ/Spring.
We run in ServletContainer (Tomcat 6/7) and use ActiveMQ 5.9.1 and Spring 3.0.0 Multiple instances of our application can run in a "group", with each instance running on it's own server. ActiveMQ is used to facilitate communication between the multiple instances. Each instance has it's own embedded broker and it's own set of queues. Every queue on every instance has exactly 1 org.springframework.jms.listener.DefaultMessageListenerContainer listening to it, so 5 queues = 5 DefaultMessageListenerContainers for example.
Our system shut down properly until we fixed a bug by adding queuePrefetch="0" to the ConnectionFactory. At first I assumed that this change was incorrect in some way, but now that I understand the situation, I am confident that we should not be using the prefetch functionality.
I have created a test application to replicate the issue. Note that the information below makes no mention of message producers. That is because I can replicate the issue without ever sending/processing a single message. Simply creating the Broker, ConnectionFactory, Queues and Listeners during boot, is enough to keep the system from stopping properly.
Here is my sample configuration from my Spring XML. I will be happy to provide my entire project if someone wants it:
<amq:broker persistent="false" id="mybroker">
<amq:transportConnectors>
<amq:transportConnector uri="tcp://0.0.0.0:61616"/>
</amq:transportConnectors>
</amq:broker>
<amq:connectionFactory id="ConnectionFactory" brokerURL="vm://localhost?broker.persistent=false" >
<amq:prefetchPolicy>
<amq:prefetchPolicy queuePrefetch="0"/>
</amq:prefetchPolicy>
</amq:connectionFactory>
<amq:queue id="lookup.mdb.queue.cat" physicalName="DogQueue"/>
<amq:queue id="lookup.mdb.queue.dog" physicalName="CatQueue"/>
<amq:queue id="lookup.mdb.queue.fish" physicalName="FishQueue"/>
<bean id="messageListener" class="org.springframework.jms.listener.DefaultMessageListenerContainer" abstract="true">
<property name="connectionFactory" ref="ConnectionFactory"/>
</bean>
<bean parent="messageListener" id="cat">
<property name="destination" ref="lookup.mdb.queue.dog"/>
<property name="messageListener">
<bean class="com.acteksoft.common.remote.jms.WorkerMessageListener"/>
</property>
<property name="concurrentConsumers" value="200"/>
<property name="maxConcurrentConsumers" value="200"/>
</bean>
<bean parent="messageListener" id="dog">
<property name="destination" ref="lookup.mdb.queue.cat"/>
<property name="messageListener">
<bean class="com.acteksoft.common.remote.jms.WorkerMessageListener"/>
</property>
<property name="concurrentConsumers" value="200"/>
<property name="maxConcurrentConsumers" value="200"/>
</bean>
<bean parent="messageListener" id="fish">
<property name="destination" ref="lookup.mdb.queue.fish"/>
<property name="messageListener">
<bean class="com.acteksoft.common.remote.jms.WorkerMessageListener"/>
</property>
<property name="concurrentConsumers" value="200"/>
<property name="maxConcurrentConsumers" value="200"/>
</bean>
My hack involves using a ServletContextListener to manually stop the objects. The hacky part is that I have to create additional threads to stop the DefaultMessageListenerContainers. Perhaps I'm stopping the objects in the wrong order, but I've tried everything that I can imagine. If I attempt to stop the objects in the main thread, then they hang indefinitely.
Thank you in advance!
UPDATE
I have tried the following based on boday's recommendation but it didn't work. I have also tried to specify the amq:transportConnector uri as tcp://0.0.0.0:61616?transport.daemon=true
<amq:broker persistent="false" id="mybroker" brokerName="localhost">
<amq:transportConnectors>
<amq:transportConnector uri="tcp://0.0.0.0:61616?daemon=true"/>
</amq:transportConnectors>
</amq:broker>
<amq:connectionFactory id="connectionFactory" brokerURL="vm://localhost" >
<amq:prefetchPolicy>
<amq:prefetchPolicy queuePrefetch="0"/>
</amq:prefetchPolicy>
</amq:connectionFactory>
At one point I tried to add similar properties to the brokerUrl parameter in the amq:connectionFactory element and the shutdown worked properly, however after further testing I learned that the properties were resulting in an exception to be thrown from VMTransportFactory. This resulted in improper initialization and the basic message functionality didn't work.
In case anyone else is wondering, as far as I can see it's not possible to have a daemon ListenerContainer using ActiveMQ.
When the ActiveMQConnection is started, it creates a ThreadPoolExecutor with non-daemon thread. This is seemingly to avoid issues when failing over the connection from one broker to another.
https://issues.apache.org/jira/browse/AMQ-796
executor = new ThreadPoolExecutor(1, 1, 5, TimeUnit.SECONDS, new LinkedBlockingQueue<Runnable>(), new ThreadFactory() {
#Override
public Thread newThread(Runnable r) {
Thread thread = new Thread(r, "ActiveMQ Connection Executor: " + transport);
//Don't make these daemon threads - see https://issues.apache.org/jira/browse/AMQ-796
//thread.setDaemon(true);
return thread;
}
});
try setting daemon=true on your TCP transport, this allows the process to run as a deamon thread which won't block the shutdown of your container
see http://activemq.apache.org/tcp-transport-reference.html

Restarting activemq route in camel when receiving a "Connection.start()" message

I have a an activemq camel route that stops receiving messages at some point and requires a restart. I am not sure how to programmatically detect and fix this situation.
My route looks like so:
from("activemq:queue:Consumer.app.VirtualTopic.msg?messageConverter=#convertMsg")
It's all configured in Spring like so:
<!-- Configure the Message Bus Factory -->
<bean id="jmsFactory" class="com.local.messaging.activemq.SpringSslContextConnectionFactory">
<property name="brokerURL" value="${jms.broker.url}" />
<property name="sslContext" ref="sslContext" />
</bean>
<!-- Connect the Message Bus Factory to Camel. The 'activemq' bean
name is necessary for Camel to pick it up automatically -->
<bean id="activemq" class="org.apache.activemq.camel.component.ActiveMQComponent" depends-on="jmsFactory">
<property name="usePooledConnection" value="true" />
<property name="connectionFactory">
<bean class="org.apache.activemq.pool.PooledConnectionFactory">
<property name="maxConnections" value="20" />
<property name="maximumActive" value="10" />
<property name="connectionFactory" ref="jmsFactory" />
</bean>
</property>
</bean>
Finally, the broker URL is configured like so:
jms.broker.url=failover://(ssl://amq1:61616,ssl://amq1:61616)
This starts up just fine and works like a champ most of the time. Every so often, though, I see this message in the logs:
Received a message on a connection which is not yet started. Have you forgotten to call Connection.start()? Connection: ActiveMQConnection {<details>}
I suspect strongly that this happens after the message bus has restarted, but as I have no direct access to the message bus, I don't know that for certain. I don't know that that matters.
The keys for me are:
How do I programmatically detect this situation? There doesn't appear to be any exception thrown or the like and the only way I've seen this is by parsing through log files.
After detecting it, how do I fix it? Do I need to start() and stop() the route or is there a cleaner way?
Finally, I did see some suggestions that this case should be handled by activemq, using the failover scheme. As shown above, I am using failover and this still happens.