I have flow where as inbound endpoint I have VM Queue. Now I want to run process as:
VM inbound endpoint gets a message and starts a long running processing flow
on VM inbound endpoint it then comes another messages (eg 10 messages) that would start another long running process but VM will keep the message in the queue until the first has completed
Every message on VM queue has timeout to removed from queue after this time
How can I do this in MuleESB ?
If it is a asynchronous flow you can use a processing strategy to limit the number of threads running a specific flow.
<queued-asynchronous-processing-strategy name="allowOneThread" maxThreads="1"/>
<flow name="OnlyOneAtTheTime" processingStrategy="allowOneThread">
<vm:inbound-endpoint path="requestQueue" exchange-pattern="one-way" />
<logger level="ERROR" message="Before sleep : #[payload]"/>
<!-- Simulate long running processor -->
<component class="Sleep" />
<logger level="ERROR" message="After sleep : #[payload]"/>
<vm:outbound-endpoint path="responseQueue"/>
</flow>
See the Mule documentation on processing strategies.
Related
mule doc says they have removed transport barrier so there is no need to have session vars in mule 4. when I am using a VM connector to publish and consume using VM vars are getting wiped off. I did cross check with mule 3.9 session vars will be persisted after reaching other flow irrespective of configured VM is one-way to req-res.
<vm:config name="vm">
<vm:queues>
<vm:queue queueName="publish" />
</vm:queues>
</vm:config>
<flow name="persistentVM">
<vm:listener queueName="publish" config-ref="vm"
numberOfConsumers="1">
<vm:response>
<vm:content><![CDATA[
]]></vm:content>
</vm:response>
</vm:listener>
<logger level="INFO" doc:name="Logger"
doc:id="1d624aa2-0aa4-4c5d-a258-0a8135a792ff" message="#[vars.test]" />
</flow>
<flow name="publish">
<http:listener doc:name="Listener"
doc:id="68f9601d-d31e-4de6-a448-98a43724de42" config-ref="HTTP_Listener_config"
path="/error" />
<set-variable value="#['My Message']" doc:name="Set Variable"
doc:id="861cc914-b1ad-4068-8753-a0fd1915bed4" variableName="test" />
<vm:publish doc:name="Publish" doc:id="153209cb-49f2-4f80-8305-56dcd046aa3c" config-ref="vm" queueName="publish"/>
`
Any comments on this is much helpful.
For Mule 4 session vars are removed and vars will not propagate across transport barriers. If you want the var in the other flow, you will either have to make it part of the payload and publish it using vm:content, persist it using the ObjectStore or use flows and flow-ref instead of the vm connector.
Do any one knows what is the exact difference between MULE_AUTO and MANUAL in ACK mode of AMQP connector ( Mule). Because I have observed that
If ACK Mode: AMQP_AUTO -> Once the message is picked up, it is being deleted automatically
If ACK Mode:MULE_AUTO -> Even after message picked up and delivered successfully, Message still present in Queue.
If Ack Mode: MANUAL -> Even after message picked up and delivered successfully, Message still present in Queue.( same case like MULE_AUTO)
In MULE_AUTO and MANUAL , both the cases if we supposed to delete the message externally. What would be the exact difference. Do I missing anything.
<amqp:connector name="AMQP_Test" validateConnections="true"
host="***" ackMode="MULE_AUTO"
username="123" password="123!" />
Could any one please help me out on its understanding.
Edited 2nd time:
I'm keeping ACK as MULE_AUTO. Message is not being deleted even after success resppnse from consumed service response. Not sure where I'm wrong
Please find the completed config.xml.
<amqp:connector name="AMQP_Test" validateConnections="true" host="****" username="123" password="123!" ackMode="MULE_AUTO" doc:name="AMQP Connector"/>
<flow name="testrabbitmqFlow1" doc:name="testrabbitmqFlow1" >
<amqp:inbound-endpoint queueName="amqp.test.queue" exchangeDurable="true" queueDurable="true" responseTimeout="1000000" connector-ref="AMQP_Test" doc:name="AMQP">
<amqp:transaction recoverStrategy="REQUEUE" action="ALWAYS_BEGIN"/>
</amqp:inbound-endpoint>
<byte-array-to-string-transformer doc:name="Byte Array to String"/>
<http:outbound-endpoint exchange-pattern="request-response" host="localhost" port="8085" path="test" method="POST" responseTimeout="1000000" doc:name="HTTP"/>
<logger message="value in queue .. #[payload]....#[message.inboundProperties['http.status']]" level="INFO" doc:name="Logger"/>
</flow>
Could you please let me know where exactly I'm missing?
With MANUAL, you have to manually ack (or reject) messages, as discussed here: https://github.com/mulesoft/mule-transport-amqp/blob/master/GUIDE.md#manual-message-acknowledgement-and-rejection
In essence, this is done with:
<amqp:acknowledge-message />
and:
<amqp:reject-message requeue="true" />
With MULE_AUTO, Mule should acknowledge the messages automatically when the flow is done processing. If it doesn't do it, it must be a bug then.
I have 2 separate Amazon SQS queues; Queue and ResponseQueue.
SQS configurations:
<sqs:config name="Amazon_SQS_Consumer" accessKey="XXX" secretKey="XXX" queueName="Queue" doc:name="Amazon SQS">
<sqs:connection-pooling-profile maxActive="10" maxIdle="10" exhaustedAction="WHEN_EXHAUSTED_GROW" maxWait="12000" minEvictionMillis="60000" evictionCheckIntervalMillis="30000" initialisationPolicy="INITIALISE_ONE"/>
<reconnect count="5" frequency="1000"/>
</sqs:config>
<sqs:config name="Amazon_SQS_Response" accessKey="XXX" secretKey="XXX" queueName="ResponseQueue" doc:name="Amazon SQS">
<sqs:connection-pooling-profile maxActive="100" maxIdle="10" exhaustedAction="WHEN_EXHAUSTED_GROW" maxWait="12000" minEvictionMillis="60000" evictionCheckIntervalMillis="30000" initialisationPolicy="INITIALISE_ONE"/>
<reconnect count="5" frequency="1000"/>
</sqs:config>
I have no problem receiving messages from the first queue (Queue) via:
<flow name="consumer" doc:name="consumer">
<sqs:receive-messages config-ref="Amazon_SQS_Consumer" preserveMessages="true" doc:name="Amazon SQS (Streaming)" visibilityTimeout="300" />
<logger level="INFO" message="#[payload]" />
</flow>
I need to also receive messages from the second queue (ResponseQueue):
<flow name="response" doc:name="response">
<sqs:receive-messages config-ref="Amazon_SQS_Response" preserveMessages="true" doc:name="Amazon SQS (Streaming)" visibilityTimeout="300" />
<logger level="INFO" message="#[payload]" />
</flow>
However, whenever the second sqs:receive-messages is added, I get the following error:
Exception in thread "Receiving Thread" java.lang.LinkageError: loader (instance of org/mule/module/launcher/plugin/MulePluginsClassLoader): attempted duplicate class definition for name: "com/amazonaws/services/sqs/QueueUrlHandler"
Is it possible to read messages from 2 different queues in the same project?
I'm using 3.4.0 CE Mule Server Runtime and 2.4.4 Amazon SQS Connector. I need to stay at these versions. If I switch to 3.5.0 EE Mule Server Runtime, there is no problem in having multiple sqs:receive-messages; it works just as expected. However, it leads to another issue.
Are you using the same credentials in both sqs:config elements? If yes, then you only need one config element and then specify the queue name on the sqs:receive-messages elements.
<sqs:receive-messages queueName="Queue"
preserveMessages="true"
visibilityTimeout="300" />
Refer to the user guide: http://mulesoft.github.io/sqs-connector/2.5.0/mule/sqs-config.html#receive-messages
I have a jms connector, i am receiving message from a queue processing the message in a flow, calling db to get the data based on some ids in the message and writing response output to files, i am using dynamic outbound endpoints to decide output location.
<jms:connector name="tibco" numberOfConsumers="20" ..... >
.....
</jms:connector>
<flow name="realtime" doc:name="ServiceId-8">
<jms:inbound-endpoint queue="${some.queue}" connector-ref="tibco" doc:name="JMS">
<jms:transaction action="ALWAYS_BEGIN"/>
</jms:inbound-endpoint>
<processor ref="proc1"></processor>
<processor ref="proc2"></processor>
<component doc:name="Java">
<spring-object bean="comp1"/>
</component>
<processor ref="proc3"></processor>
<collection-splitter doc:name="Collection Splitter"/>
<processor ref="endpointprocessor"></processor>
<foreach collection="#[message.payload.consumerEndpoints]" counterVariableName="endpoints" doc:name="Foreach">
<when expression="#[consumerEndpoint.getOutputType().equals('txt') and consumerEndpoint.getChannel().equals('file')]">
<processor-chain>
<file:outbound-endpoint path="#[consumerEndpoint.getPath()]" outputPattern="#[consumerEndpoint.getClientId()]-#[attributes['eventId']]%#[consumerEndpoint.getTicSeedCount()]-#[attributes['dateTime']].tic" responseTimeout="10000" doc:name="File"/>
</processor-chain>
</when>
<when expression="#[consumerEndpoint.getOutputType().equals('txt') and consumerEndpoint.getChannel().equals('ftp')]">
<processor-chain>
<ftp:outbound-endpoint path="#[consumerEndpoint.getPath()]" outputPattern="#[consumerEndpoint.getClientId()]-#[attributes['eventId']]%#[consumerEndpoint.getTicSeedCount()]-#[attributes['dateTime']].tic" host="#[consumerEndpoint.getHost()]" port="#[consumerEndpoint.getPort()]" user="#[consumerEndpoint.getChannelUser()]" password="#[consumerEndpoint.getChannelPass()]" responseTimeout="10000" doc:name="FTP"/>
</processor-chain>
</when>
</choice>
</foreach>
<rollback-exception-strategy doc:name="Rollback Exception Strategy">
<processor ref="catchExceptionCustomHandling"></processor>
</rollback-exception-strategy>
</flow>
Above is not complete flow. i pasted the important parts to understand.
Question 1. As i have not defined any thread strategy at any level, and connector has numberOfConsumers="20", if i drop 20 messages in queue how many threads will start.
prefetch size in the jms queue is set to 20.
Question 2: Do i need to configure threading strategy at receiver end and/or at flow level.
some time when the load is very high(let say 15k msgs in queue in a minute) i see message processing gets slow and thread dump shows some thing like below:
"TIBCO EMS Session Dispatcher (7905958)" prio=10 tid=0x00002aaadd4cf000 nid=0x3714 waiting for monitor entry [0x000000004af1e000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.mule.endpoint.DynamicOutboundEndpoint.createStaticEndpoint(DynamicOutboundEndpoint.java:153)
- waiting to lock <0x00002aaab711c0e0> (a org.mule.endpoint.DynamicOutboundEndpoint)
Any help and pointers will be appreciated.
Thanks-
Message processing is getting slow because of dynamic endpoint, I see thread congestion when dynamic outbound endpoint is created and used. I was using mule 3.3.x and after looking at mule 3.4.x code i realized that dynamic outbound endpoint creation is handled more appropriately. upgraded to 3.4 and the issue is almost gone.
How to get the number of messages in Mule ESB vm inbound endpoint Queue from the code?
Here is the configuration:
<vm:connector name="queue.vm.connector">
<vm:queue-profile maxOutstandingMessages="${oms.process.flow.max.queue.size}"/>
</vm:connector>
...
<flow name="Flow1"...
<vm:inbound-endpoint exchange-pattern="one-way" path="inbound_parcel.create.queue" responseTimeout="10000" mimeType="text/plain" connector-ref="queue.vm.connector" doc:name="Inbound Parcel Create Queue"/>
Thank you.
That can be achieved using the following custom code:
VMConnector vmConnector = (VMConnector) muleContext.getRegistry().lookupConnector("queue.vm.connector");
vmConnector.getQueueManager().getQueueSession().getQueue("inbound_parcel.create.queue").size();
If the queue if persistent the same result can be achieved by monitoring the folder where message are stored:
${MULE_HOME}/.mule/${app.name}/queuestore/inbound_parcel.create.queue