I have mule app with single mainFlow. Flow receives jms-message from MQ by quartz:inbound-endpoint, processes it and puts in another queue. Messages are independed. Is it possible to run multiple instances of my flow for concurrent message processing?
enter image description here
Yes. As long as your flow is neither transactional or two-way (request-response), you can set-up various asynchronous processing strategies.
You can add it either visually in the Studio in the flow's properties or by manually adding the attribute processingStrategy="...yourStrategy" to the flow's XML declaration.
For an in-depth guide to what everything means and does, see https://docs.mulesoft.com/mule-user-guide/v/3.8/flow-processing-strategies
Related
we are trying to audit all incoming/outgoing messages, header information in our mule flow.
For same we have tried to use 'wire-tap' which we dint found so useful also its working on mule 3.6.1 but giving error in 3.7.
Any idea/suggestion for auditing?
Ok let me add some more details:
What we are trying to do is- Whatever message comes or flows via flow components we want to copy it in some sub flow (say in queue) without interrupting the main flow so that we can check the message.
you can make it work in several ways
1) Wire tap is one of the choice. You can route your messages asynchronously to sub flow and sub flow will do the auditing work. But I don't know why you didn't found wiretap useful. Can you explain in more.
2) All the messages your receive from the main flow, those you can post to JMS queue. So another flow will read from there and do the auditing work. Use of this multiple projects can use the same piece of code and post JMS queue for auditing.
This can be done in many different ways and you kind of mentioned them in your question such as logger component and interceptor.
All the headers are available as message properties so if you log the entire message they are shown. Simply put an logger component after the inbound endpoint and one before the outbound endpoint and this is easily done.
If you need some log entries transformation you could always put that in a wire tap so you don't interfere with the functionality of your flow.
I am using reliable delivery in mule flow. It is very simple case that takes message from JMS queue (ActiveMQ based), invokes several actions depending on it's content and, if everything is fine - delivers it into another JMS queue.
A flow is synchronized, both JMS queues are transactional (first BEGINS, second JOINS transaction), redelivery is used and DLQ for undelivered messages. Literally: I expect that all messages are properly either processed or delivered to DLQ.
For processing orchestration I am using Scatter/Gather flow control which works quite fine until I call external HTTP service using HTTP connector. When I use default threading profile it happens, that some messages are lost (like 3 of 5000 messages). They just disappear. No trace even in DLQ.
On the other hand, when I use custom profile (not utilizing thread) - all messages are getting processed without any problems.
What I have noticed is the fact, default threading profile utilizes 'ScatterGatherWorkManager', while custom uses 'ActiveMQ Session Task' threads.
So my question is: what is the possible cause of loosing these messages?
I am using Mule Server 3.6.1 CE Runtime.
by default scatter gather is setup for no failed routes you can define your own aggregation strategy to handle lost message
custom-aggregation-strategy
https://docs.mulesoft.com/mule-user-guide/v/3.6/scatter-gather
I have more-or-less implemented the Reliability Pattern in my Mule application using persistent VM queues CloudHub, as documented here. While everything works fine, it has left me with a number of questions about actually ensuring reliable delivery of my messages. To illustrate the points below, assume I have http-request component within my "application logic flow" (see the diagram on the link above) that is throwing an exception because the endpoint is down, and I want to ensure that the in flight message will eventually get delivered to the endpoint:
As detailed on the link above, I have observed that when the exception is thrown within my "application logic flow", and I have made the flow transactional, the message is put back on the VM queue. However all that happens is the message then repeatedly taken off the queue, processed by the flow, and the exception is thrown again - ad infinitum. There appears to be no way of configuring any sort of retry delay or maximum number of retries on VM queues as is possible, for example, with ActiveMQ. The best work around I have come up with is to surround the http-request message processor with the until-successful scope, but I'd rather have these sorts of things apply to my whole flow (without having to wrap the whole flow in until-successful). Is this sort of thing possible using only VM queues and CloudHub?
I have configured my until-successful to place the message on another VM queue which I want to use as a dead-letter-queue. Again, this works fine, and I can login to CloudHub and see the messages populated on my DLQ - but then it appears to offer no way of moving messages from this queue back into the flow when the endpoint comes back up. All it seems you can do in CloudHub is clear your queue. Again, is this possible using VM queues and CloudHub only (i.e. no other queueing tool)?
VM queues are very basic, whether you use them in CloudHub or not.
VM queues have no capacity for delaying redelivery (like exponential back-offs). Use JMS queues if you need such features.
You need to create a flow for processing the DLQ, for example one that regularly consumes the queue via the requester module and re-injects the messages into the main queue. Again, with JMS, you would have better control.
Alternatively to JMS, you could consider hosted queues like CloudAMQP, Iron.io or AWS SQS. You would lose transaction support on the inbound endpoint but would gain better control on the (re)delivery behaviour.
I have a requirement where I need to make sure only one message is being processed at a time by a mule flow.Flow is triggered by a quartz scheduler which reads one file from FTP server every time
My proposed solution is to keep a global variable "FLOW_STATUS" which will be set to "RUNNING" when a message is received and would be reset to "STOPPED" once the processing of message is done.
Any messages fed to the flow will check for this variable and abort if "FLOW_STATUS" is "RUNNING".
This setup seems to be working , but I was wondering if there is a better way to do it.
Is there any best practices around this or any inbuilt mule helper functions to achieve the same instead of relying on global variables
It seems like a more simple solution would be to set the maxActiveThreads for the flow to 1. In Mule, each message processed gets it's own thread. So setting the maxActiveThreads to 1 would effectively make your flow singled threaded. Other pending requests will wait in the receiver threads. You will need to make sure your receiver thread pool is large enough to accommodate all of the potential waiting threads. That may mean throttling back your quartz scheduler to allow time process the files so the receiver thread pool doesn't fill up. For more information on the thread pools and how to tune performance, here is a good link: http://www.mulesoft.org/documentation/display/current/Tuning+Performance
I need to implement a logic on Retry. Inbound endpoint pushes the messages to Rest (Outbound). If the REST is unavailable, I need to retry for 1 time and put it in the queue. But the second upcoming messages should not do any retry, it has to directly put the messages in to queue until the REST service is available.
Once the service is available, I need to pushes all the messages from QUEUE to REST Service (in ordering) via batch job.
Questions:
How do I know the service is unavailable for my second message? If I use until Successful, for every message it do retry and put in queue. Plm is 2nd message shouldn't do retry.
For batch, I thought of using poll, but how to tell to poll, when the service becomes available to begin the batch process. (bcz,Poll is more of with configuring timings to run batch)?
Other ticky confuses me is - Here ordering has to be preserved. once the service is available. Queue messages ( i,e Batch) has to move first to REST Services then with real time. I doubt whether Is it applicable.
It will be very helpful for the quick response to implement the logic.
Using Mule: 3.5.1
I could try something like below: using flow controls
process a message; if exception or bad response code, set a variable/property like serviceAvailable=false.
subsequent message processing will first check the property serviceAvailable to process the messages. if property is false, en-queue the messages to a DB table with status=new/unprocessed
create a flow/scheduler to process the messages from DB sequentially, but it will not check the property serviceAvailable and call the rest service.
If service throws exception it will not store the messages in db again but if processes successfully change the property serviceAvailable=true and de-queue the messages or change the status. Add another property and set it to true if there are more messages in db table like moreDBMsg=true.
New messages should not be processed/consumed until moreDBMsg=false
once moreDBMsg=false and serviceAvailable=true start processing the messages from queue.
For the timeout I would still look at the response code and catch time-outs to determine if the call was successful or requires a retry. Practically you normally do multi threading anyway, so you have multiple calls in parallel anyway. Or simply one call starts before the other ends.
That is just quite normal.
But you can simply retry calls in a queue that time out. And after x amounts of time-outs you "skip" or defer the retry.
But all of this has been done using actual Mule flow components like either:
MEL http://www.mulesoft.org/documentation/display/current/Mule+Expression+Language+Reference
Or flow controls: http://www.mulesoft.org/documentation/display/current/Choice+Flow+Control+Reference
Or for example you reference a Spring Bean and do it in native Java code.
One possibility for the queue would be to persist it in a database. Mule has database connector that has a "poll" feature, see: http://www.mulesoft.org/documentation/display/current/JDBC+Transport+Reference#JDBCTransportReference-PollingTransport