I have 2 separate Amazon SQS queues; Queue and ResponseQueue.
SQS configurations:
<sqs:config name="Amazon_SQS_Consumer" accessKey="XXX" secretKey="XXX" queueName="Queue" doc:name="Amazon SQS">
<sqs:connection-pooling-profile maxActive="10" maxIdle="10" exhaustedAction="WHEN_EXHAUSTED_GROW" maxWait="12000" minEvictionMillis="60000" evictionCheckIntervalMillis="30000" initialisationPolicy="INITIALISE_ONE"/>
<reconnect count="5" frequency="1000"/>
</sqs:config>
<sqs:config name="Amazon_SQS_Response" accessKey="XXX" secretKey="XXX" queueName="ResponseQueue" doc:name="Amazon SQS">
<sqs:connection-pooling-profile maxActive="100" maxIdle="10" exhaustedAction="WHEN_EXHAUSTED_GROW" maxWait="12000" minEvictionMillis="60000" evictionCheckIntervalMillis="30000" initialisationPolicy="INITIALISE_ONE"/>
<reconnect count="5" frequency="1000"/>
</sqs:config>
I have no problem receiving messages from the first queue (Queue) via:
<flow name="consumer" doc:name="consumer">
<sqs:receive-messages config-ref="Amazon_SQS_Consumer" preserveMessages="true" doc:name="Amazon SQS (Streaming)" visibilityTimeout="300" />
<logger level="INFO" message="#[payload]" />
</flow>
I need to also receive messages from the second queue (ResponseQueue):
<flow name="response" doc:name="response">
<sqs:receive-messages config-ref="Amazon_SQS_Response" preserveMessages="true" doc:name="Amazon SQS (Streaming)" visibilityTimeout="300" />
<logger level="INFO" message="#[payload]" />
</flow>
However, whenever the second sqs:receive-messages is added, I get the following error:
Exception in thread "Receiving Thread" java.lang.LinkageError: loader (instance of org/mule/module/launcher/plugin/MulePluginsClassLoader): attempted duplicate class definition for name: "com/amazonaws/services/sqs/QueueUrlHandler"
Is it possible to read messages from 2 different queues in the same project?
I'm using 3.4.0 CE Mule Server Runtime and 2.4.4 Amazon SQS Connector. I need to stay at these versions. If I switch to 3.5.0 EE Mule Server Runtime, there is no problem in having multiple sqs:receive-messages; it works just as expected. However, it leads to another issue.
Are you using the same credentials in both sqs:config elements? If yes, then you only need one config element and then specify the queue name on the sqs:receive-messages elements.
<sqs:receive-messages queueName="Queue"
preserveMessages="true"
visibilityTimeout="300" />
Refer to the user guide: http://mulesoft.github.io/sqs-connector/2.5.0/mule/sqs-config.html#receive-messages
Related
mule doc says they have removed transport barrier so there is no need to have session vars in mule 4. when I am using a VM connector to publish and consume using VM vars are getting wiped off. I did cross check with mule 3.9 session vars will be persisted after reaching other flow irrespective of configured VM is one-way to req-res.
<vm:config name="vm">
<vm:queues>
<vm:queue queueName="publish" />
</vm:queues>
</vm:config>
<flow name="persistentVM">
<vm:listener queueName="publish" config-ref="vm"
numberOfConsumers="1">
<vm:response>
<vm:content><![CDATA[
]]></vm:content>
</vm:response>
</vm:listener>
<logger level="INFO" doc:name="Logger"
doc:id="1d624aa2-0aa4-4c5d-a258-0a8135a792ff" message="#[vars.test]" />
</flow>
<flow name="publish">
<http:listener doc:name="Listener"
doc:id="68f9601d-d31e-4de6-a448-98a43724de42" config-ref="HTTP_Listener_config"
path="/error" />
<set-variable value="#['My Message']" doc:name="Set Variable"
doc:id="861cc914-b1ad-4068-8753-a0fd1915bed4" variableName="test" />
<vm:publish doc:name="Publish" doc:id="153209cb-49f2-4f80-8305-56dcd046aa3c" config-ref="vm" queueName="publish"/>
`
Any comments on this is much helpful.
For Mule 4 session vars are removed and vars will not propagate across transport barriers. If you want the var in the other flow, you will either have to make it part of the payload and publish it using vm:content, persist it using the ObjectStore or use flows and flow-ref instead of the vm connector.
I am using Mule ESB to design a process whereby one can post a message to a topic. Subscribers will listen to the topic and receive messages. Each subscriber will act on the messages differently. The goal here is to have the ability to post a test message to the topic from HTTP for testing subscribers.
Here is how I have the JMS connection configured:
<!-- JMS Topic connector -->
<jms:activemq-connector name="jmsTopicConnection" specification="1.1" brokerURL="tcp://localhost:61616" validateConnections="true" doc:name="Active MQ2" durable="true" numberOfConcurrentTransactedReceivers="2"/>
This is the flow:
<flow name="auditJMSServiceFlow">
<http:listener config-ref="HTTP" path="/Audit/Activity" responseStreamingMode="ALWAYS" doc:name="HTTP"/>
<set-variable variableName="#['id']" value="#[message.inboundProperties['id']]" doc:name="set dynamic id"/>
<set-payload value="===TOPIC===" doc:name="Set Payload" />
<request-reply storePrefix="mainFlow">
<jms:inbound-endpoint topic="Audit.Activity" connector-ref="jmsTopicConnection" doc:name="JMS Topic Audit.Activity" exchange-pattern="request-response" durableName="audit_activity">
<jms:transaction action="ALWAYS_BEGIN" />
<!-- Not required to explicitly have this element. Mule will put this in implicitly. -->
<!-- <jms:jmsmessage-to-object-transformer displayName="JmsMsg to Object"/> -->
</jms:inbound-endpoint>
</request-reply>
<json:object-to-json-transformer doc:name="transform JMS message to JSON"/>
<json:validate-schema schemaLocation="resource://AuditMsgSchema.json" doc:name="Validate Json Schema"/>
<component class="com.baml.panther.audit.service.impl.AuditServiceImpl" doc:name="Java"/>
<default-exception-strategy>
<commit-transaction exception-pattern="com.foo.ExpectedExceptionType"/>
<jms:outbound-endpoint queue="dead.letter" connector-ref="jmsConnection">
<jms:transaction action="JOIN_IF_POSSIBLE" />
</jms:outbound-endpoint>
</default-exception-strategy>
<logger message="=== #[message.payload] received #[org.mule.util.DateUtiles.getTimeStamp('dd-MM-yyyy_HH-mm-ss.SSS')]" level="INFO" doc:name="Logger"/>
When I am running through the test I get the following error:
Any suggestions would be greatly appreciated.
Russ
For the error: Your request-reply scope is missing an outbound endpoint. You only have the inbound-endpoint (jms:inbound-endpoint). You need to provide the outbound-endpoint as well.
<request-reply storePrefix="mainFlow">
<jms:inbound-endpoint topic="Audit.Activity" connector-ref="jmsTopicConnection" doc:name="JMS Topic Audit.Activity" exchange-pattern="request-response" durableName="audit_activity">
<jms:transaction action="ALWAYS_BEGIN" />
<!-- Not required to explicitly have this element. Mule will put this in implicitly. -->
<!-- <jms:jmsmessage-to-object-transformer displayName="JmsMsg to Object"/> -->
</jms:inbound-endpoint>
</request-reply>
Not sure what your aim there but if you put just a jms:outbound-enpoint (instead of the whole request-reply block), you can send a message to the JMS topic.
The problem is that you cannot put a message source as the first message processor in a request-reply. The request reply allows you a kind of synchronous call for async protocols like JMS.
If you want to send a message to the message broker at the point where you put the request-reply just put a JMS outbound-endpoint.
If what you want to do is consume a message from the JMS topic you have to put a JMS inbound endpoint as the first message processor in a flow.
Do any one knows what is the exact difference between MULE_AUTO and MANUAL in ACK mode of AMQP connector ( Mule). Because I have observed that
If ACK Mode: AMQP_AUTO -> Once the message is picked up, it is being deleted automatically
If ACK Mode:MULE_AUTO -> Even after message picked up and delivered successfully, Message still present in Queue.
If Ack Mode: MANUAL -> Even after message picked up and delivered successfully, Message still present in Queue.( same case like MULE_AUTO)
In MULE_AUTO and MANUAL , both the cases if we supposed to delete the message externally. What would be the exact difference. Do I missing anything.
<amqp:connector name="AMQP_Test" validateConnections="true"
host="***" ackMode="MULE_AUTO"
username="123" password="123!" />
Could any one please help me out on its understanding.
Edited 2nd time:
I'm keeping ACK as MULE_AUTO. Message is not being deleted even after success resppnse from consumed service response. Not sure where I'm wrong
Please find the completed config.xml.
<amqp:connector name="AMQP_Test" validateConnections="true" host="****" username="123" password="123!" ackMode="MULE_AUTO" doc:name="AMQP Connector"/>
<flow name="testrabbitmqFlow1" doc:name="testrabbitmqFlow1" >
<amqp:inbound-endpoint queueName="amqp.test.queue" exchangeDurable="true" queueDurable="true" responseTimeout="1000000" connector-ref="AMQP_Test" doc:name="AMQP">
<amqp:transaction recoverStrategy="REQUEUE" action="ALWAYS_BEGIN"/>
</amqp:inbound-endpoint>
<byte-array-to-string-transformer doc:name="Byte Array to String"/>
<http:outbound-endpoint exchange-pattern="request-response" host="localhost" port="8085" path="test" method="POST" responseTimeout="1000000" doc:name="HTTP"/>
<logger message="value in queue .. #[payload]....#[message.inboundProperties['http.status']]" level="INFO" doc:name="Logger"/>
</flow>
Could you please let me know where exactly I'm missing?
With MANUAL, you have to manually ack (or reject) messages, as discussed here: https://github.com/mulesoft/mule-transport-amqp/blob/master/GUIDE.md#manual-message-acknowledgement-and-rejection
In essence, this is done with:
<amqp:acknowledge-message />
and:
<amqp:reject-message requeue="true" />
With MULE_AUTO, Mule should acknowledge the messages automatically when the flow is done processing. If it doesn't do it, it must be a bug then.
I have a jms connector, i am receiving message from a queue processing the message in a flow, calling db to get the data based on some ids in the message and writing response output to files, i am using dynamic outbound endpoints to decide output location.
<jms:connector name="tibco" numberOfConsumers="20" ..... >
.....
</jms:connector>
<flow name="realtime" doc:name="ServiceId-8">
<jms:inbound-endpoint queue="${some.queue}" connector-ref="tibco" doc:name="JMS">
<jms:transaction action="ALWAYS_BEGIN"/>
</jms:inbound-endpoint>
<processor ref="proc1"></processor>
<processor ref="proc2"></processor>
<component doc:name="Java">
<spring-object bean="comp1"/>
</component>
<processor ref="proc3"></processor>
<collection-splitter doc:name="Collection Splitter"/>
<processor ref="endpointprocessor"></processor>
<foreach collection="#[message.payload.consumerEndpoints]" counterVariableName="endpoints" doc:name="Foreach">
<when expression="#[consumerEndpoint.getOutputType().equals('txt') and consumerEndpoint.getChannel().equals('file')]">
<processor-chain>
<file:outbound-endpoint path="#[consumerEndpoint.getPath()]" outputPattern="#[consumerEndpoint.getClientId()]-#[attributes['eventId']]%#[consumerEndpoint.getTicSeedCount()]-#[attributes['dateTime']].tic" responseTimeout="10000" doc:name="File"/>
</processor-chain>
</when>
<when expression="#[consumerEndpoint.getOutputType().equals('txt') and consumerEndpoint.getChannel().equals('ftp')]">
<processor-chain>
<ftp:outbound-endpoint path="#[consumerEndpoint.getPath()]" outputPattern="#[consumerEndpoint.getClientId()]-#[attributes['eventId']]%#[consumerEndpoint.getTicSeedCount()]-#[attributes['dateTime']].tic" host="#[consumerEndpoint.getHost()]" port="#[consumerEndpoint.getPort()]" user="#[consumerEndpoint.getChannelUser()]" password="#[consumerEndpoint.getChannelPass()]" responseTimeout="10000" doc:name="FTP"/>
</processor-chain>
</when>
</choice>
</foreach>
<rollback-exception-strategy doc:name="Rollback Exception Strategy">
<processor ref="catchExceptionCustomHandling"></processor>
</rollback-exception-strategy>
</flow>
Above is not complete flow. i pasted the important parts to understand.
Question 1. As i have not defined any thread strategy at any level, and connector has numberOfConsumers="20", if i drop 20 messages in queue how many threads will start.
prefetch size in the jms queue is set to 20.
Question 2: Do i need to configure threading strategy at receiver end and/or at flow level.
some time when the load is very high(let say 15k msgs in queue in a minute) i see message processing gets slow and thread dump shows some thing like below:
"TIBCO EMS Session Dispatcher (7905958)" prio=10 tid=0x00002aaadd4cf000 nid=0x3714 waiting for monitor entry [0x000000004af1e000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.mule.endpoint.DynamicOutboundEndpoint.createStaticEndpoint(DynamicOutboundEndpoint.java:153)
- waiting to lock <0x00002aaab711c0e0> (a org.mule.endpoint.DynamicOutboundEndpoint)
Any help and pointers will be appreciated.
Thanks-
Message processing is getting slow because of dynamic endpoint, I see thread congestion when dynamic outbound endpoint is created and used. I was using mule 3.3.x and after looking at mule 3.4.x code i realized that dynamic outbound endpoint creation is handled more appropriately. upgraded to 3.4 and the issue is almost gone.
I have flow where as inbound endpoint I have VM Queue. Now I want to run process as:
VM inbound endpoint gets a message and starts a long running processing flow
on VM inbound endpoint it then comes another messages (eg 10 messages) that would start another long running process but VM will keep the message in the queue until the first has completed
Every message on VM queue has timeout to removed from queue after this time
How can I do this in MuleESB ?
If it is a asynchronous flow you can use a processing strategy to limit the number of threads running a specific flow.
<queued-asynchronous-processing-strategy name="allowOneThread" maxThreads="1"/>
<flow name="OnlyOneAtTheTime" processingStrategy="allowOneThread">
<vm:inbound-endpoint path="requestQueue" exchange-pattern="one-way" />
<logger level="ERROR" message="Before sleep : #[payload]"/>
<!-- Simulate long running processor -->
<component class="Sleep" />
<logger level="ERROR" message="After sleep : #[payload]"/>
<vm:outbound-endpoint path="responseQueue"/>
</flow>
See the Mule documentation on processing strategies.