What is the cleanest way to listen to JMS from inside a Spring-batch step? - activemq

Spring batch documentation recommends using the JmsItemReader, which is a wrapper around the JMSTemplate. However, I have discovered that the JMSTemplate has some issues - see http://activemq.apache.org/jmstemplate-gotchas.html .
This post came to my attention only because the queue was appearing to disappear before I could actually read the data of of it. The opportunity to miss messages seems like a fairly significant issue to me.

For consumers atleast try using DefaultMessageListenerContainer coupled with a SingleConnectionFactory or any such connection factory , it not need a scheduler to wake them up.there are log of examples explaining this , this one is really good in explaining stuff
http://bsnyderblog.blogspot.com/2010/05/tuning-jms-message-consumption-in.html

Here is the solution I ended up with. Since the query was about the "cleanest" way to listen to JMS from within a spring-batch step, I'm going to leave the question open for a while longer just in case there's a better way.
If someone can figure out why the code isn't formatting correctly, please let me know how to fix it.
1. In the a job listener, implement queue setup and teardown inside the beforeJob and afterJob events, respectively:
public void beforeJob(JobExecution jobExecution) {
try {
jobParameters = jobExecution.getJobParameters();
readerConnection = connectionFactory.createConnection();
readerConnection.start();
} catch (JMSException ex) {
// handle the exception as appropriate
}
}
public void afterJob(JobExecution jobExecution) {
try {
readerConnection.close();
} catch (JMSException e) {
// handle the exception as appropriate
}
}
2. In the reader, implement the StepListener and beforeStep / afterStep methods.
public void beforeStep(StepExecution stepExecution) {
this.stepExecution = stepExecution;
this.setJobExecution(stepExecution.getJobExecution());
try {
this.connection = jmsJobExecutionListener.getReaderConnection();
this.jmsSession = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);
this.messageConsumer = jmsSession.createConsumer(jmsJobExecutionListener.getQueue());
}
catch (JMSException ex)
{
// handle the exception as appropriate
}
}
public ExitStatus afterStep(StepExecution stepExecution) {
try {
messageConsumer.close();
jmsSession.close();
} catch (JMSException e) {
// handle the exception as appropriate
}
return stepExecution.getExitStatus();
}
3. Implement the read() method:
public TreeModel<SelectedDataElementNode> read() throws Exception,
UnexpectedInputException, ParseException,
NonTransientResourceException {
Object result = null;
logger.debug("Attempting to receive message on connection: ", connection.toString());
ObjectMessage msg = (ObjectMessage) messageConsumer.receive();
logger.debug("Received: {}", msg.toString());
result = msg.getObject();
return result;
}
4. Add the listeners to the Spring Batch context as appropriate:
<batch:job id="doStuff">
<batch:listeners>
<batch:listener ref="jmsJobExecutionListener" />
</batch:listeners>
... snip ...
<batch:step id="step0003-do-stuff">
<batch:tasklet transaction-manager="jtaTransactionManager"
start-limit="100">
<batch:chunk reader="selectedDataJmsReader" writer="someWriter"
commit-interval="1" />
</batch:tasklet>
<batch:listeners>
<batch:listener ref="selectedDataJmsReader" />
</batch:listeners>
</batch:step>
</batch:job>

Related

Why does SignalR recommend using finally to propagate errors in streams?

The SignalR docs on streaming state:
Wrap logic in a try ... catch statement. Complete the Channel in a finally block. If you want to flow an error, capture it inside the catch block and write it in the finally block.
They then proceed to give an example that goes through these convolutions for no apparent gain. Why is this? What difference does it make whether one captures an exception and completes the channel from the finally block versus completing then and there in the catch block?
Possibly to centralize the writer completion logic, even if takes just a single invocation - and you may want to insert additional related logic there (such as logging), if needed.
Exception localException = null;
try
{
// ...
}
catch (Exception ex)
{
localException = ex;
}
finally
{
writer.Complete(localException);
}
versus:
var completed = false;
try
{
// ...
}
catch (Exception ex)
{
writer.Complete(ex);
completed = true;
}
finally
{
if (!completed)
{
writer.Complete(null);
}
}

RabbitMQ Camel Consumer - Consume a single message

I have a scenario where I want to "pull" messages of a RabbitMQ queue/topic and process them one at a time.
Specifically if there are already messages sitting on the queue when the consumer starts up.
I have tried the following with no success (meaning, each of these options reads the queue until it is either empty or until another thread closes the context).
1.Stopping route immediately it is first processed
final CamelContext context = new DefaultCamelContext();
try {
context.addRoutes(new RouteBuilder() {
#Override
public void configure() throws Exception {
RouteDefinition route = from("rabbitmq:harley?queue=IN&declare=false&autoDelete=false&hostname=localhost&portNumber=5672");
route.process(new Processor() {
Thread stopThread;
#Override
public void process(final Exchange exchange) throws Exception {
String name = exchange.getIn().getHeader(Exchange.FILE_NAME_ONLY, String.class);
String body = exchange.getIn().getBody(String.class);
// Doo some stuff
routeComplete[0] = true;
if (stopThread == null) {
stopThread = new Thread() {
#Override
public void run() {
try {
((DefaultCamelContext)exchange.getContext()).stopRoute("RabbitRoute");
} catch (Exception e) {}
}
};
}
stopThread.start();
}
});
}
});
context.start();
while(!routeComplete[0].booleanValue())
Thread.sleep(100);
context.stop();
}
Similar to 1 but using a latch rather than a while loop and sleep.
Using a PollingConsumer
final CamelContext context = new DefaultCamelContext();
context.start();
Endpoint re = context.getEndpoint(srcRoute);
re.start();
try {
PollingConsumer consumer = re.createPollingConsumer();
consumer.start();
Exchange exchange = consumer.receive();
String bb = exchange.getIn().getBody(String.class);
consumer.stop();
} catch(Exception e){
String mm = e.getMessage();
}
Using a ConsumerTemplate() - code similar to above.
I have also tried enabling preFetch and setting the max number of exchanges to 1.
None of these appear to work, if there are 3 messages on the queue, all are read before I am able to stop the route.
If I were to use the standard RabbitMQ Java API I would use a basicGet() call which lets me read a single message, but for other reasons I would prefer to use a Camel consumer.
Has anyone successfully been able to process a single message on a queue that holds multiple messages using a Camel RabbitMQ Consumer?
Thanks.
This is not the primary intention of the component as its for continued received. But I have created a ticket to look into supporting a basicGet (single receive). There is a new spring based rabbitmq component coming in 3.8 onwards so its going to be implemeneted there (first): https://issues.apache.org/jira/browse/CAMEL-16048

Camel Route Losing Message on restart in camel rabbitmq

I am using camel-rabbitmq.
Here is my route defination
camelContext.addRoutes(new RouteBuilder() {
#Override
public void configure() throws Exception {
from("rabbitmq:TEST?queue=TEST&concurrentConsumers=5")
.routeId("jms")
.autoStartup(false)
.throttle(10)
.asyncDelayed()
.log("Consuming message ${body} to ${header.deliveryAddress}")
.process(new Processor() {
#Override
public void process(Exchange exchange) throws Exception {
System.out.println(atomicLong.decrementAndGet());
}
})
;
}
});
When I push 500 messages to this queue , when stop and start route all message on channel will be lost ,wonder where they are going.
If I configure same route with &autoAck=false it is working properly but losing performance. Why camel not offering same behavior with and without autoAck.
I managed my problem doing following change in rabbitmqconsumer of camel-rabbitmq
public void handleCancelOk(String consumerTag) {
// no work to do
log.info("Received cancelOk signal on the rabbitMQ channel");
**downLatch.countDown();**
}
#Override
protected void doStop() throws Exception {
if (channel == null) {
return;
}
this.requeueChannel=openChannel(consumer.getConnection());
if (tag != null && isChannelOpen()) {
channel.basicCancel(tag);
}
stopping=true;
downLatch.await();
try {
lock.acquire();
if (isChannelOpen()) {
channel.close();
}
} catch (TimeoutException e) {
log.error("Timeout occured");
throw e;
} catch (InterruptedException e1) {
log.error("Thread Interrupted!");
} finally {
lock.release();
}
}
By doing this camel route will for message to consumed and avoided message loss.
You need to check rabbitmq consumer prefetch count
consumer prefetch
I think By default consumer picks all the messages in queue to its memory buffers.
If you set the prefetch count to 1, consumer will acknowledge messages one by one.
All the other unacknowledged will be present in the queue in ready state. Waiting to be picked up, after the consumer completes it task on the previous message picked.

How to verify exception thrown using StepVerifier in project reactor

def expectError() {
StepVerifier.create(readDB())
.expectError(RuntimeException.class)
.verify();
}
private Mono<String> readDB() {
// try {
return Mono.just(externalService.get())
.onErrorResume(throwable -> Mono.error(throwable));
// } catch (Exception e) {
// return Mono.error(e);
// }
}
unable to make it work if externalService.get throws Exception instead of return Mono.error. Is is always recommended to transform to Mono/Flow using try catch or is there any better way to verify such thrown exception?
Most of the time, if the user-provided code that throws an exception is provided as a lambda, exceptions can be translated to onError. But here you're directly throwing in the main thread, so that cannot happen

Do WCF support Asynchronously operations' invoke within TransactionScope?

I am trying out the WCF Transaction implementation and I come up with the idea that whether asynchronous transaction is supported by WCF 4.0.
for example,
I have several service operations with client\service transaction enabled, in the client side, I use a TransactionScope and within the transaction, I create Tasks to asynchronously call those operations.
In this situation, I am assuming that the transaction is going to work correctly, is that right?
I doubt that very much. It appears that you if you are starting an ascync operation you are no longer participating on the original transaction.
I wrote a little LINQPad test
void Main()
{
using (var scope = new TransactionScope(TransactionScopeOption.Required))
{
try
{
Transaction.Current.Dump("created");
Task.Factory.StartNew(Test);
scope.Complete();
}
catch (Exception e)
{
Console.WriteLine(e);
}
Thread.Sleep(1000);
}
Console.WriteLine("closed");
Thread.Sleep(5000);
}
public void Test()
{
using (var scope = new TransactionScope(TransactionScopeOption.Required))
{
Transaction.Current.Dump("test start"); // null
Thread.Sleep(5000);
Console.WriteLine("done");
Transaction.Current.Dump("test end"); // null
}
}
You'll need to set both the OperationContext and Transaction.Current in the created Task.
More specifically, in the service you'll need to do like this:
public Task ServiceMethod() {
OperationContext context = OperationContext.Current;
Transaction transaction = Transaction.Current;
return Task.Factory.StartNew(() => {
OperationContext.Current = context;
Transaction.Current = transaction;
// your code, doing awesome stuff
}
}
This gets repetitive as you might suspect, so I'd recommend writing a helper for it.