While studying and using Mule, I couldn't figure out if there's a difference between a queued-asynchronous flow queue and a VM queue.
My question is, are they the same queues (just with different names along the documentation) or different ones?
In a concrete example:
<flow name="fooFlow" processingStrategy="queued-asynchronous">
<vm:inbound-endpoint path="foo" exchange-pattern="one-way"/>
<component class="com.foo.FooComponent"/>
</flow>
Does the VM inbound-endpoint receives messages from one queue, and the flow has another queue to receive the messages from the inbound-endpoint? Or are they the same SEDA queue?
These are two very different concepts, one is based on the way a flow is processed and the other is a lightweight queuing mechanism. VM is a transport and it has persistent queuing capabilities as well as transactions.
Please see (the last link to understand the flow execution model):
http://www.mulesoft.org/documentation/display/current/Flow+Processing+Strategies
http://www.mulesoft.org/documentation/display/current/VM+Transport+Reference
http://www.mulesoft.org/documentation/display/current/Tuning+Performance
To add some details on the specific example you showed.
You do not need to specify the processing-strategy explicitly, Mule chooses the default processing strategy based on the exchange-pattern of the inbound endpoint. So as you have a non transactions one-way endpoint the processing strategy is queued-asynchronous.
Does the VM inbound-endpoint receives messages from one queue, and the
flow has another queue to receive the messages from the
inbound-endpoint? Or are they the same SEDA queue?
To recieve messages Mule will use the thread pool dedicated to the VM connector (receiver threads are tied to the transport). Once the message has been received it will be processed using a thread from the flow's thread pool. (Would be great if I could be validated or corrected :)
(Most of the information is from the links posted in the earlier answer)
Related
could you please explain Queued-Asynchronous Flow Processing Strategy with an example
i found many documents with the explanation but not getting any example that shows how it to process and flow creation for this
i found one link where it explains synchrounus and non blocking processing strategies
https://www.ricston.com/blog/synchronous-non-blocking-processing-strategies/
The Queued Asynchronous Flow Processing Strategy works by having a thread pool for the message source of the flow (for example a JMS inbound transport), a thread pool for flow execution, and a queue of Mule events between the two threads. So when a JMS message arrives to start the flow, it is handled by a thread of the transport, then it queues in the internal queue as a Mule message, to be picked up by one of the flow threads to execute the rest of the flow. The source threads are freed to keep listening for new messages, that is the 'asynchronous' part. They don't wait for the flow to be processed.
I have JMS in my mule flow where producer reads records from cache, put in queue and consumer consumes messages and do further processing. Following is the flow for understanding.
Service 1 (Read data from file) -> Service 2 (put each line in cache)
-> JMS Service 3 (Producer Read data from cache line by line and put in queue) and Consumer read from queue -> Service 4
In above flow, from JMS component, flow becomes asynch hence as soon as producer puts all records in queue response goes back to client saying process completed but it is possible that consumer still going to consume messages.
I want to hold process from producer to send back response until consumer consumes all the messages.
Any idea on this how to achieve?
Since the async takes the copy of the exact thread and process independently, it may be possible that the producer putting the message in the queue as fast as before the consumer actually able to consume it.
One way I can think to hold the process of putting the message into the queue is by putting a sleep() before it.
You can use a Groovy component and use sleep() in it to hold the flow or slow down the process.
for example, if you put the following:
<scripting:component doc:name="Groovy">
<scripting:script engine="Groovy"><![CDATA[
sleep(10000);
return message.payload;]]>
</scripting:script>
</scripting:component>
before the putting the message into the queue, the process will slow down a bit and will hold the flow for 10000 ms till on the other side the consumer actually consume it.
Polling for completion status as described above may work OK but there's still a risk of some transactions not being completed after wait time, or waiting long after all messages have been processed.
Depending on the end goal of this exercise, you could perhaps leverage Mule batch, which already implements the splitting of the inbound request into individual messages, processing the messages in one or multiple consumer threads, keeping track of the chunks processed and remaining, and reporting the results / executing final steps once all data is processed.
If you can't use batch and need to reassemble the processed messages into a single list or map, you may be able to get the Collection Aggregator do the job of tracking the messages by correlation ID and setting the timeouts.
The crude DIY way to implement it is to build some sort of dispatcher logic for the JMS publishing component. It will submit all messages to JMS then wait for each consumer / worker thread to respond back (via a separate JMS queue) with completion message with the same correlation ID. The dispatcher will then track all submitted / processed messages in the in-memory or persistent storage and respond back once the last message in the batch has been acknowledged, or by pre-defined timeout. Which is very close to what Mule batch already does.
Cheers!
Dima
You can use exchange pattern value as request-response so that flow will wait for response from JMS.
I am using reliable delivery in mule flow. It is very simple case that takes message from JMS queue (ActiveMQ based), invokes several actions depending on it's content and, if everything is fine - delivers it into another JMS queue.
A flow is synchronized, both JMS queues are transactional (first BEGINS, second JOINS transaction), redelivery is used and DLQ for undelivered messages. Literally: I expect that all messages are properly either processed or delivered to DLQ.
For processing orchestration I am using Scatter/Gather flow control which works quite fine until I call external HTTP service using HTTP connector. When I use default threading profile it happens, that some messages are lost (like 3 of 5000 messages). They just disappear. No trace even in DLQ.
On the other hand, when I use custom profile (not utilizing thread) - all messages are getting processed without any problems.
What I have noticed is the fact, default threading profile utilizes 'ScatterGatherWorkManager', while custom uses 'ActiveMQ Session Task' threads.
So my question is: what is the possible cause of loosing these messages?
I am using Mule Server 3.6.1 CE Runtime.
by default scatter gather is setup for no failed routes you can define your own aggregation strategy to handle lost message
custom-aggregation-strategy
https://docs.mulesoft.com/mule-user-guide/v/3.6/scatter-gather
I have more-or-less implemented the Reliability Pattern in my Mule application using persistent VM queues CloudHub, as documented here. While everything works fine, it has left me with a number of questions about actually ensuring reliable delivery of my messages. To illustrate the points below, assume I have http-request component within my "application logic flow" (see the diagram on the link above) that is throwing an exception because the endpoint is down, and I want to ensure that the in flight message will eventually get delivered to the endpoint:
As detailed on the link above, I have observed that when the exception is thrown within my "application logic flow", and I have made the flow transactional, the message is put back on the VM queue. However all that happens is the message then repeatedly taken off the queue, processed by the flow, and the exception is thrown again - ad infinitum. There appears to be no way of configuring any sort of retry delay or maximum number of retries on VM queues as is possible, for example, with ActiveMQ. The best work around I have come up with is to surround the http-request message processor with the until-successful scope, but I'd rather have these sorts of things apply to my whole flow (without having to wrap the whole flow in until-successful). Is this sort of thing possible using only VM queues and CloudHub?
I have configured my until-successful to place the message on another VM queue which I want to use as a dead-letter-queue. Again, this works fine, and I can login to CloudHub and see the messages populated on my DLQ - but then it appears to offer no way of moving messages from this queue back into the flow when the endpoint comes back up. All it seems you can do in CloudHub is clear your queue. Again, is this possible using VM queues and CloudHub only (i.e. no other queueing tool)?
VM queues are very basic, whether you use them in CloudHub or not.
VM queues have no capacity for delaying redelivery (like exponential back-offs). Use JMS queues if you need such features.
You need to create a flow for processing the DLQ, for example one that regularly consumes the queue via the requester module and re-injects the messages into the main queue. Again, with JMS, you would have better control.
Alternatively to JMS, you could consider hosted queues like CloudAMQP, Iron.io or AWS SQS. You would lose transaction support on the inbound endpoint but would gain better control on the (re)delivery behaviour.
I'm working for a company where we're considering Mule ESB. We would need to set up Mule in a clustered configuration to get what Mule coins a Mule High Availability (HA) Cluster.
Now, we need to persist incoming messages to a queue in case of power outage or disk failure. As far as I understand, we can either go with the default Mule Object Store which "persists" messages to a shared memory grid. However, my first thought here is that this can't be any good if we get a power outage which takes the entire cluster out of action.
Our other option is to use a separate queue product such as RabbitMQ or ActiveMQ. However, do these integrate alright with a HA cluster? Are there any mechanism in these products which ensures that the same message won't be picked up by two machines at the same time?
Consider this scenario (based on the observer pattern):
Mule receives a message, puts it on a queue and responds with an OK
to the client which delivered the message.
Mule picks up a message from the queue, and attempts to deliver it to a subscriber.
The subscriber accepts the message, and Mule removes it from the queue.
What happens if another Mule instance in the HA cluster attempts to pick up the message between 2 and 3 above? Is there a mechanism where Mule can indicate that a message is picked up from the queue to be "attempted delivered" but then, if the delivery fails, update the message on the queue as "not delivered" if delivery fails?
Both RabbitMQ and ActiveMQ will give you the once-and-only-once functionality I think you are looking for.
Both platforms ensure that each message in a queue is received by only one subscriber.
In ActiveMQ, to return a message to a queue in the event of a failure, you can use explicit message acknowledgement or JMS transactions. Here's a quick overview.
In RabbitMQ, you do it using acknowledgements.
Also, you might want to consider reliability for your message broker. Both ActiveMQ and RabbitMQ offer highly available broker configuration options.