IBM provides a method to control performance when using asynchronous delivery:
setMsgBatchSize
public void setMsgBatchSize(int size)
throws javax.jms.JMSException
Sets the message batch size.
Parameters:
size - the maximum number of messages to be taken at once when using asynchronous delivery.
Throws:
javax.jms.JMSException
How can we achieve this on ActiveMQ? From the official documentation of ActiveMQ they have provided property named:
cf.setUseAsyncSend(true);
I'm not sure if ActiveMQ was setting msg batch size by default, and when we set useAsyncSend to true we can achieve results as IBM did.
Thank you in advance and willing to listen to any discussion.
ActiveMQ does not have an equivalent option. When a message is sent it is immediately put to the wire and if the async send option is enabled (default is true) then for messages that are not persistent or are within a transaction there is no wait for remote acknowledgement of the send, your code can immediately send the next message.
Related
I have written a program which requires multiple queues interaction - means consumer of one queue writes message to another queue and same program has consumer to take action on that queue.
Problem: How to handle network time-out issues with queue while sending messages asynchronously using spring rabbit ampq library?or RabbitTemplate.send() function must throw an exception if there are network issues.
Currently, I have implemented RabbitTemplate.send() that returns immediately and working fine. But, If network is down, send function returns immediately, doesn't throw any exception and client code assumes success. As a result, i have in-consistent state in DB that message is successfully processed. Please note that call to send function is wrapped inside transactional block and goal is if queue writing fails, DB commit must also rollback. I am exploring following solutions but no success:
Can we configure rabbitTemplate to throw run-time exception if any network connectivity issue so that client call is notified? Please suggest how to do this.
Shall we use synchronous SendAndReceive function call but it leads to delay in processing? Another problem, observed with this function, my consumer code gets notification while sendAndReceive function is still blocked for writing message to queue. Please advise if we can delay notification to queue unless sendAndReceive function is returned. But call to SendAndReceive() was throwing an amqp exception if network was down which we were able to capture, but it has cost associated related to performance.
My application is multi-threaded, if multiple threads are sending message using sendAndReceive(), how spring-amqp library manages queue communication? Does it internally creates channel per request? If messages are delivered via same channel, it would impact performance a lot for multi-threaded application.
Can some-one share sample code for using SendAndReceive function with best-practices?
Do we have any function in spring-amqp library to check health of RabbitMQ server before submitting send function call? I explored rabbitTemplate.isRunning() but not getting proper result. If any specific configuration required, please suggest.
Any other solution to consider for guaranteed message delivery or handle network time-out issues to throw runtime exceptions to client..
As per Gary comment below, I have set: rabbitTemplate.setChannelTransacted(true); and it makes call sync. Next part of problem is that if I have transaction block on outer block, call to RabbitTemplate.send() returns immediately. I expect transaction block of outer function must wait for inner function to return, otherwise, ii don't get expected result as my DB changes are persisted though we enabled setChannelTransacted to true. I tried various Transaction propagation level but no success. Please advise if I am doing anything wrong and review transactional propagation settings as below
#Transactional
public void notifyQueueAndDB(DBRequest dbRequest) {
logger.info("Updating Request in DB");
dbService.updateRequest(dbRequest));
//Below is call to RabbitMQ library
mqService.sendmessage(dbRequest); //If sendMessage fails because of network outage, I want DB commit also to be rolled-back.
}
MQService defined in another library of project, snippet below.
#Transactional( propagation = Propagation.NESTED)
private void sendMessage(......) {
....
rabbitTemplate.send(this.queueExchange, queueName, amqpMessage);
}catch (Exception exception) {
throw exception
}
Enable transactions so that the send is synchronous.
or
Use Publisher confirms and wait for the confirmation to be received.
Either one will be quite a bit slower.
I have gone through rabbitmq documentation,
https://www.rabbitmq.com/confirms.html#publisher-confirms
Using standard AMQP 0-9-1, the only way to guarantee that a message
isn't lost is by using transactions -- make the channel transactional
then for each message or set of messages publish, commit. In this
case, transactions are unnecessarily heavyweight and decrease
throughput by a factor of 250. To remedy this, a confirmation
mechanism was introduced. It mimics the consumer acknowledgements
mechanism already present in the protocol.
To enable confirms, a client sends the confirm.select method.
Depending on whether no-wait was set or not, the broker may respond
with a confirm.select-ok. Once the confirm.select method is used on a
channel, it is said to be in confirm mode. A transactional channel
cannot be put into confirm mode and once a channel is in confirm mode,
it cannot be made transactional.
Currently I am using RabbitTemplate.convertAndSend of spring-rabbit library to send message.
I am using transactional channel to publish messages to rabbitmq, As per the document its slower and I can can improve the throughput by using publisher-confirm.
But I am not much clear about it.
If I want to enable confirm then what are changes required and how do I handle exception?
What will be my retrial mechanism?
Does this publisher confirm work in asynchronous way?
And does transaction work in synchronously?
Any suggestion is highly appreciated.
Using publisher confirms will not improve performance significantly over transactions if you wait for the confirm for each individual send. They help significantly if you send many messages and wait for the confirms later.
Transactions are synchronous. Confirms are completely asynchronous.
See Confirms and Returns.
When you enable confirms, you provide a callback to the template which will be called when the confirm is received. You add correlation data to the send, which is provided in the callback so you can determine which send this confirm is for. Furthermore, the correlation data (in recent versions) provides a Future<?> which you can wait on to receive the confirm in a synchronous manner.
That's where you would handle any exception(s).
I hope that helps.
There is a confirms and returns sample Spring Boot application in the samples repo but it was created before the future was added to the CorrelationData. That will be fixed soon.
The correlation data can contain the original message, enabling retry.
Is it possible for a Mule message to expire (i.e. the container will discard the message) after a configured amount of time (like the JMS TTL property)?
If there is please can you point me to the documentation or example?
Can we use the attribute queueTimeout (see http://www.mulesoft.org/documentation/display/current/VM+Transport+Reference) to achieve this?
Cheers
No, the queueTimeout attribute does not control the TTL for messages on the queue. It is used when performing blocking operations on the queue (like dispatching a message or polling for a message).
This feature is not built into the VM transport. You might be able to accomplish the same idea by setting a message property with a timestamp before publishing it to the VM queue, and then filtering on the message age in the comsuming flow.
I need to implement a logic on Retry. Inbound endpoint pushes the messages to Rest (Outbound). If the REST is unavailable, I need to retry for 1 time and put it in the queue. But the second upcoming messages should not do any retry, it has to directly put the messages in to queue until the REST service is available.
Once the service is available, I need to pushes all the messages from QUEUE to REST Service (in ordering) via batch job.
Questions:
How do I know the service is unavailable for my second message? If I use until Successful, for every message it do retry and put in queue. Plm is 2nd message shouldn't do retry.
For batch, I thought of using poll, but how to tell to poll, when the service becomes available to begin the batch process. (bcz,Poll is more of with configuring timings to run batch)?
Other ticky confuses me is - Here ordering has to be preserved. once the service is available. Queue messages ( i,e Batch) has to move first to REST Services then with real time. I doubt whether Is it applicable.
It will be very helpful for the quick response to implement the logic.
Using Mule: 3.5.1
I could try something like below: using flow controls
process a message; if exception or bad response code, set a variable/property like serviceAvailable=false.
subsequent message processing will first check the property serviceAvailable to process the messages. if property is false, en-queue the messages to a DB table with status=new/unprocessed
create a flow/scheduler to process the messages from DB sequentially, but it will not check the property serviceAvailable and call the rest service.
If service throws exception it will not store the messages in db again but if processes successfully change the property serviceAvailable=true and de-queue the messages or change the status. Add another property and set it to true if there are more messages in db table like moreDBMsg=true.
New messages should not be processed/consumed until moreDBMsg=false
once moreDBMsg=false and serviceAvailable=true start processing the messages from queue.
For the timeout I would still look at the response code and catch time-outs to determine if the call was successful or requires a retry. Practically you normally do multi threading anyway, so you have multiple calls in parallel anyway. Or simply one call starts before the other ends.
That is just quite normal.
But you can simply retry calls in a queue that time out. And after x amounts of time-outs you "skip" or defer the retry.
But all of this has been done using actual Mule flow components like either:
MEL http://www.mulesoft.org/documentation/display/current/Mule+Expression+Language+Reference
Or flow controls: http://www.mulesoft.org/documentation/display/current/Choice+Flow+Control+Reference
Or for example you reference a Spring Bean and do it in native Java code.
One possibility for the queue would be to persist it in a database. Mule has database connector that has a "poll" feature, see: http://www.mulesoft.org/documentation/display/current/JDBC+Transport+Reference#JDBCTransportReference-PollingTransport
Using ActiveMQ v 5.8
I am using javax.jms.MessageProducer.send() to send messages from my producer to ActiveMQ.
I want to know whether this sending is synchronous or asynchronous? And what will be the behavior if I make "useAsyncSend" flag to true ?
Thanks,
Anuj
ActiveMQ sends message in async mode by default in several cases. It is only in cases where the JMS specification required the use of sync sending that we default to sync sending. The cases that we are forced to send in sync mode are when persistent messages are being sent outside of a transaction.
If you are not using transactions and are sending persistent messages, then each send is synch and blocks until the broker has sent back an acknowledgement to the producer that the message has been safely persisted to disk. This ack provides that guarantee that the message will not be lost but it also costs a huge latency penalty since the client is blocked.
See the documentation on this at the ActiveMQ s.
yes, by default a send() is synchronous (for persistent queue/topic, async otherwise) and will block until an ACK has been received...
with useAsyncSend=true will not block...
per http://activemq.apache.org/connection-configuration-uri.html
Async Sends adds a massive performance boost; but means that the send() method will return immediately whether the message has been sent or not which could lead to message loss.