I face a complex case.
What I'm doing is as following steps:
1) <from uri="jetty:http://0.0.0.0:30100/jetty/test"/>
2) <to uri="hazelcast-client:master-test-series" />
3) <to uri="bean:modelSeriesWrapperTest" />
4)
<split parallelProcessing="true" streaming="true">
<simple>${body}</simple> <to uri="direct:dw.model.test"/>
</split>
5) From another route
<from uri="direct:dw.model.test"/>
<aggregate strategyRef="myAggregatorStrategy"
completionTimeout="1000">
<correlationExpression>
<constant>true</constant>
</correlationExpression>
<marshal ref="modelSeriesVariantColourGson" />
<camel:to uri="file:src/data/catask/output?fileName=output.xml"/>
</aggregate>
The problem is that the jetty response is empty. I use TCP trace to track the request and response, the Content-Length is 0. But the output.xml file has correct JSON format content.
Even I cross the <camel:to uri="file:src/data/catask/output?fileName=output.xml"/>. The jetty response is still empty.
I try the InOut pattern, it doesn't work as well.
It seems jetty return directly, not waiting split done. I try to set In and Out body, it doesn't work either. I Google every case that I can image. There is no helpful case.
Could you please help me? Thank you very much.
If you want the jetty response to include whatever information from your aggregator, then you must use the splitter only approach as documented at:
http://camel.apache.org/composed-message-processor.html
The splitter has built-in aggregation, and that ensures when the splitter is done, it aggregates also, and then you can use that as the jetty response.
When you use <aggregate> then it becomes a separate exchange. To understand this more then read more about the aggregate eip, and other SO, and in various Camel books etc.
Related
I'm using Apache Camel 2.13.1 to poll a database table which will have upwards of 300k rows in it. I'm looking to use the Idempotent Consumer EIP to filter rows that have already been processed.
I'm wondering though, whether the implementation is really scalable or not. My camel context is:-
<camelContext xmlns="http://camel.apache.org/schema/spring">
<route id="main">
<from
uri="sql:select * from transactions?dataSource=myDataSource&consumer.delay=10000&consumer.useIterator=true" />
<transacted ref="PROPAGATION_REQUIRED" />
<enrich uri="direct:invokeIdempotentTransactions" />
<!-- Any processors here will be executed on all messages -->
</route>
<route id="idempotentTransactions">
<from uri="direct:invokeIdempotentTransactions" />
<idempotentConsumer
messageIdRepositoryRef="jdbcIdempotentRepository">
<ognl>#{request.body.ID}</ognl>
<!-- Anything here will only be executed for non-duplicates -->
<log message="non-duplicate" />
<to uri="stream:out" />
</idempotentConsumer>
</route>
</camelContext>
It would seem that the full 300k rows are going to be processed every 10 seconds (via consumer.delay parameter) which seems very inefficient. I would expect some sort of feedback loop as part of the pattern so that the query that feeds the filter could take advantage of the set of rows already processed.
However, the messageid column in the CAMEL_MESSAGEPROCESSED table has the pattern of
{1908988=null}
where 1908988 is the request.body.ID I've set the EIP to key on so this doesn't make it easy to incorporate into my query.
Is there a better way of using the CAMEL_MESSAGEPROCESSED table as a feedback loop into my select statement so that the SQL server is performing most of the load?
Update:
So, I've since found out that it was my ognl code that was causing the odd message id column value. Changing it to
<el>${in.body.ID}</el>
has fixed it. So, now that I have a usable messageId column, I can now change my 'from' SQL query to
select * from transactions tr where tr.ID IN (select cmp.messageid from CAMEL_MESSAGEPROCESSED cmp where cmp.processor = 'transactionProcessor')
but I still think I'm corrupting the Idempotent Consumer EIP.
Does anyone else do this? Any reason not to?
Yes, it is. But you need to use scalable storage for holding sets of already processed messages. You can use either Hazelcast - http://camel.apache.org/hazelcast-idempotent-repository-tutorial.html or Infinispan - http://java.dzone.com/articles/clustered-idempotent-consumer - depending on which solution is already in your stack. Of course, JDBC repository would work, but only if it meets performance criteria selected.
I want to set a key/value pair using camel-redis. I try:
spring-redis://localhost:6379?command=SET&CamelRedis.key=testkey&CamelRedis.value=100
but no joy. I get the error:
There are 2 parameters that couldn't be set on the endpoint. Check the uri if the parameters are spelt correctly and that they are properties of the endpoint. Unknown parameters=[{key=testkey, value=100}]
Although there are plenty of examples about how to subscribe etc. I do not find a single example on how to set a key/value pair. How would I do that?
CamelRedis.Key and CamelRedis.Value (beware they are case sensitive) are message headers not URI parameters
<route>
<from uri="direct:intput"/>
<setHeader headerName="CamelRedis.Key"><constant>testkey</constant></setHeader>
<setHeader headerName="CamelRedis.Value"><constant>100</constant></setHeader>
<to uri="spring-redis://localhost:6379?command=SET"/>
</route>
I needed to split a message into 3 different payloads and transform and send to 3 routers. So the payload initially will have a header a body or detail and a footer. These 3 different payloads need to be extracted and send to 3 different routers. What would be the most efficient way to do it.
It depends on your body/payload type. If your payload is XML, you can easily split it using xpath and route it using content based routing similar to:
<splitter expression="#[xpath('//nodes/node)']" />
<choice>
<when expression="#[xpath('//node/id').text ='myid']">
<!-- Route somewhere -->
</when>
<otherwise>
<!-- Route somewhere else -->
</otherwise>
</choice>
The expression splitter above can take any MEL expression to split up your payload. There are many other splitters, for example if your payload is already a java Collection, you can simply use the collection-splitter.
Other splitter info can be found here: http://www.mulesoft.org/documentation-3.2/display/32X/Message+Splitting+and+Aggregation
Also there are other routers that can help you with fork and join patterns if you need to process messages asynchronously as well. Here's a good post on that: http://java.dzone.com/articles/aggregation-mule-%E2%80%93-%E2%80%9Cfork-and
In Mule 3.3.1, during async processing, when any of my external services are down, I would like to place the message on a queue (retryQueue) with a particular "next retry" timestamp. The flow that processes messages from this retryQueue selects messages based on "next retry" time as in if "next retry" time is past current time, select the message for processing. Similar to what has been mentioned in following link.
Retry JMS queue implementation to deliver failed messages after certain interval of time
Could you please provide sample code to achieve this?
I tried:
<on-redelivery-attempts-exceeded>
<message-properties-transformer scope="outbound">
<add-message-property key="putOnQueueTime" value="#[function:datestamp:yyyy-MM-dd hh:mm:ssZ]" />
</message-properties-transformer>
<jms:outbound-endpoint ref="retryQueue"/>
</on-redelivery-attempts-exceeded>
and on the receiving flow
<jms:inbound-endpoint ref="retryQueue">
<!-- I have no idea how to do the selector....
I tried....<jms:selector expression="#[header:INBOUND:putOnQueueTime > ((function:now) - 30)]"/>, but obviously it doesn't work. Gives me an invalid message selector. -->
</jms:inbound-endpoint>.
Another note: If I set the outbound property using
<add-message-property key="putOnQueueTime" value="#[function:now]"/>,
it doesn't get carried over as part of header. That's why I changed it to:
<add-message-property key="putOnQueueTime" value="#[function:datestamp:yyyy-MM-dd hh:mm:ssZ]" />
The expression in:
<jms:selector expression="#[header:INBOUND:putOnQueueTime > ((function:now) - 30)]"/>
should evaluate to a valid JMS selector, which is not the case here. Try with:
<jms:selector expression="putOnQueueTime > #[XXX]"/>
replacing XXX with an expression that creates the time you want.
We were trying to achieve this in one of the projects I'm working on, and tried what was being suggested in the other answer here, and it did not work, with various variantions. The problem is that the jms:selector doesn't support MEL, since it's relies on ActiveMQ classes.
We registered a support-ticket to Mulesoft, and their reply was that this is not supported.
What we ended up doing was this:
Create a simple Component, which does a Thread.sleep(numberOfMillis), where the number of millis is defined in a property.
In the flow that was supposed to delay processing, we added this component as the first step after reading the message from the inbound endpoint.
Not the best solution ever made, but it works..
I am trying to find a way to pass in "destAddr=" in smpp route below, a value that comes from the above sql query in order to import the senders number in the sms destination address but after much search, I can't find a way. How can I save the value I need from the query and then use it dynamically in the smpp option? Any suggestions would be much appreciated.
<from uri="sql:{{sql.selectRunRecList}}" />
<to uri="bean:smppBean?method=smsConstruct" />
<to uri="sql:{{sql.markSms}}"/>
<to uri="bean:smppBean?method=smsPrintText" />
<to uri="file:C:/workspace/SMPP/outbox" />
<to uri="smpp://smppclient#localhost:2775?password=password&destAddr= " />
See this FAQ how to use dynamic values when sending to an endpoint in Camel
http://camel.apache.org/how-do-i-use-dynamic-uri-in-to.html