I am developing system where messages to be processed are pushed in ActiveMQ. I have stringent requirement that consumer must process messages in incoming sequence. If a message processing fails in consumer, it need to rollback/recover and keep on re-trying infinitely. Only when a message processing is successful, consumer need to commit and proceed to next message.
How to prevent rolled-back message auto-forwarding to DLQ and what's proper way of configuring re-delivery policy for such requirement?
when set RedeliveryPolicy re-trying infinitely like below the messages will never be sent to DLQ.
policy.setMaximumRedeliveries(RedeliveryPolicy.NO_MAXIMUM_REDELIVERIES);
with ActiveMQSession.INDIVIDUAL_ACKNOWLEDGE you acknowledge messages one by one.
http://activemq.apache.org/redelivery-policy.html
import javax.jms.Connection;
import javax.jms.JMSException;
import javax.jms.Message;
import javax.jms.MessageListener;
import org.apache.activemq.ActiveMQConnectionFactory;
import org.apache.activemq.ActiveMQMessageConsumer;
import org.apache.activemq.ActiveMQSession;
import org.apache.activemq.RedeliveryPolicy;
public class SimpleConsumerIndividualAcknowledge {
public static void main(String[] args) throws JMSException {
Connection conn = null;
try {
ActiveMQConnectionFactory cf = new ActiveMQConnectionFactory("tcp://localhost:61616");
RedeliveryPolicy policy = new RedeliveryPolicy();
policy.setMaximumRedeliveries(RedeliveryPolicy.NO_MAXIMUM_REDELIVERIES);
cf.setRedeliveryPolicy(policy);
conn = cf.createConnection();
ActiveMQSession session = (ActiveMQSession) conn.createSession(false,
ActiveMQSession.INDIVIDUAL_ACKNOWLEDGE);
ActiveMQMessageConsumer consumer = (ActiveMQMessageConsumer) session
.createConsumer(session.createQueue("test"));
consumer.setMessageListener(new MessageListener() {
#Override
public void onMessage(Message message) {
try {
//do your stuff
message.acknowledge();
} catch (Exception e) {
throw new RuntimeException(e);//ActiveMQMessageConsumer.rollback() is called automatically
}
}
});
conn.start();
} catch (Exception e) {
e.printStackTrace();
} finally {
if (conn != null) {
try {
conn.close();
} catch (Exception e) {
}
}
}
}
}
if you want to manually stop and restart consumer take a look here activemq-redelivery-does-not-work
Related
I am using camel-rabbitmq.
Here is my route defination
camelContext.addRoutes(new RouteBuilder() {
#Override
public void configure() throws Exception {
from("rabbitmq:TEST?queue=TEST&concurrentConsumers=5")
.routeId("jms")
.autoStartup(false)
.throttle(10)
.asyncDelayed()
.log("Consuming message ${body} to ${header.deliveryAddress}")
.process(new Processor() {
#Override
public void process(Exchange exchange) throws Exception {
System.out.println(atomicLong.decrementAndGet());
}
})
;
}
});
When I push 500 messages to this queue , when stop and start route all message on channel will be lost ,wonder where they are going.
If I configure same route with &autoAck=false it is working properly but losing performance. Why camel not offering same behavior with and without autoAck.
I managed my problem doing following change in rabbitmqconsumer of camel-rabbitmq
public void handleCancelOk(String consumerTag) {
// no work to do
log.info("Received cancelOk signal on the rabbitMQ channel");
**downLatch.countDown();**
}
#Override
protected void doStop() throws Exception {
if (channel == null) {
return;
}
this.requeueChannel=openChannel(consumer.getConnection());
if (tag != null && isChannelOpen()) {
channel.basicCancel(tag);
}
stopping=true;
downLatch.await();
try {
lock.acquire();
if (isChannelOpen()) {
channel.close();
}
} catch (TimeoutException e) {
log.error("Timeout occured");
throw e;
} catch (InterruptedException e1) {
log.error("Thread Interrupted!");
} finally {
lock.release();
}
}
By doing this camel route will for message to consumed and avoided message loss.
You need to check rabbitmq consumer prefetch count
consumer prefetch
I think By default consumer picks all the messages in queue to its memory buffers.
If you set the prefetch count to 1, consumer will acknowledge messages one by one.
All the other unacknowledged will be present in the queue in ready state. Waiting to be picked up, after the consumer completes it task on the previous message picked.
I'd like to create a consumer which stacks arriving messages, and waits:
until n messages have arrived.
t seconds are elapsed.
to process the whole stack of messages.
Pre-fetching is not what I'm looking for. What I really need is to process messages together.
class MyListener(stomp.ConnectionListener):
def on_message(self, headers, body):
print ("Just received ONE message\n"
"I should wait for n-1 others\n"
"or t seconds before processing")
here an example
import java.util.LinkedList;
import java.util.List;
import javax.jms.Connection;
import javax.jms.JMSException;
import javax.jms.Session;
import javax.jms.TextMessage;
import org.apache.activemq.ActiveMQConnectionFactory;
import org.apache.activemq.ActiveMQMessageConsumer;
public class SimpleConsumerClientAcknowledge {
public static void main(String[] args) throws JMSException {
List<TextMessage> messages = new LinkedList<>();
Connection conn = null;
try {
ActiveMQConnectionFactory cf = new ActiveMQConnectionFactory(
"tcp://localhost:61617?jms.prefetchPolicy.all=200");
conn = cf.createConnection("admin", "admin");
Session session = conn.createSession(false, Session.CLIENT_ACKNOWLEDGE);
ActiveMQMessageConsumer consumer = (ActiveMQMessageConsumer) session
.createConsumer(session.createQueue("Q"));
conn.start();
TextMessage msg = null;
// MAX_MESSAGES have to be < prefetchSize / 2 -->
// jms.prefetchPolicy.all=200
// Once the broker has dispatched a prefetch limit number of
// messages to a consumer it will not dispatch any more messages to
// that consumer until the consumer has acknowledged at least 50% of
// the prefetched messages
int MAX_MESSAGES = 100;
long MAX_WAIT = 60000;
long millis = System.currentTimeMillis();
while ((msg = (TextMessage) consumer.receive(5000)) != null) {
if (msg != null) {
messages.add(msg);
}
if (messages.size() == MAX_MESSAGES || (System.currentTimeMillis() - millis >= MAX_WAIT)) {
millis = System.currentTimeMillis();
treatMessages(messages);
// because session is created with
// Session.CLIENT_ACKNOWLEDGE as an acknowledgeMode consumer
// need to acknowledge manually received messages
consumer.acknowledge();
}
}
} catch (Exception e) {
e.printStackTrace();
} finally {
if (conn != null) {
try {
conn.close();
} catch (Exception e) {
}
}
}
}
private static void treatMessages(List<TextMessage> messages) {
// TODO Auto-generated method stub
messages.clear();
}
}
I am trying to create a mule flow with a TCP inbound endpoint which is a TCP server that listens to a port. When a successful client connection is identified, before receiving any request from the client, I need to write a message into the socket (which lets the client know that I am listening), only after which the client sends me further requests. This is how I do it with a sample java program :
import java.net.*;
import java.io.*;
public class TCPServer
{
public static void main(String[] args) throws IOException
{
ServerSocket serverSocket = null;
try {
serverSocket = new ServerSocket(4445);
}
catch (IOException e)
{
System.err.println("Could not listen on port: 4445.");
System.exit(1);
}
Socket clientSocket = null;
System.out.println ("Waiting for connection.....");
try {
clientSocket = serverSocket.accept();
}
catch (IOException e)
{
System.err.println("Accept failed.");
System.exit(1);
}
System.out.println ("Connection successful");
System.out.println ("Sending output message - .....");
//Sending a message to the client to indicate that the server is active
PrintStream pingStream = new PrintStream(clientSocket.getOutputStream());
pingStream.print("Server listening");
pingStream.flush();
//Now start listening for messages
System.out.println ("Waiting for incoming message - .....");
PrintWriter out = new PrintWriter(clientSocket.getOutputStream(),true);
BufferedReader in = new BufferedReader(new InputStreamReader(clientSocket.getInputStream()));
String inputLine;
while ((inputLine = in.readLine()) != null)
{
System.out.println ("Server: " + inputLine);
out.println(inputLine);
if (inputLine.equals("Bye."))
break;
}
out.close();
in.close();
clientSocket.close();
serverSocket.close();
}
}
I have tried to use Mule's TCP inbound endpoint as a server, but I am not able to see how I can identify a successful connection from the client, inorder to trigger the outbound message. The flow gets triggered only when a message is sent across from the client. Is there a way I can extend the functionality of the Mule TCP connector and have a listener which could do the above requirement?
Based on the answer provided, this is how I implemented this -
public class TCPMuleOut extends TcpMessageReceiver {
boolean InitConnection = false;
Socket clientSocket = null;
public TCPMuleOut(Connector connector, FlowConstruct flowConstruct,
InboundEndpoint endpoint) throws CreateException {
super(connector, flowConstruct, endpoint);
}
protected Work createWork(Socket socket) throws IOException {
return new MyTcpWorker(socket, this);
}
protected class MyTcpWorker extends TcpMessageReceiver.TcpWorker {
public MyTcpWorker(Socket socket, AbstractMessageReceiver receiver)
throws IOException {
super(socket, receiver);
// TODO Auto-generated constructor stub
}
#Override
protected Object getNextMessage(Object resource) throws Exception {
if (InitConnection == false) {
clientSocket = this.socket;
logger.debug("Sending logon message");
PrintStream pingStream = new PrintStream(
clientSocket.getOutputStream());
pingStream.print("Log on message");
pingStream.flush();
InitConnection = true;
}
long keepAliveTimeout = ((TcpConnector) connector)
.getKeepAliveTimeout();
Object readMsg = null;
try {
// Create a monitor if expiry was set
if (keepAliveTimeout > 0) {
((TcpConnector) connector).getKeepAliveMonitor()
.addExpirable(keepAliveTimeout,
TimeUnit.MILLISECONDS, this);
}
readMsg = protocol.read(dataIn);
// There was some action so we can clear the monitor
((TcpConnector) connector).getKeepAliveMonitor()
.removeExpirable(this);
if (dataIn.isStreaming()) {
}
return readMsg;
} catch (SocketTimeoutException e) {
((TcpConnector) connector).getKeepAliveMonitor()
.removeExpirable(this);
System.out.println("Socket timeout");
} finally {
if (readMsg == null) {
// Protocols can return a null object, which means we're
// done
// reading messages for now and can mark the stream for
// closing later.
// Also, exceptions can be thrown, in which case we're done
// reading.
dataIn.close();
InitConnection = false;
logger.debug("Client closed");
}
}
return null;
}
}
}
And the TCP connector is as below:
<tcp:connector name="TCP" doc:name="TCP connector"
clientSoTimeout="100000" receiveBacklog="0" receiveBufferSize="0"
sendBufferSize="0" serverSoTimeout="100000" socketSoLinger="0"
validateConnections="true" keepAlive="true">
<receiver-threading-profile
maxThreadsActive="5" maxThreadsIdle="5" />
<reconnect-forever />
<service-overrides messageReceiver="TCPMuleOut" />
<tcp:direct-protocol payloadOnly="true" />
</tcp:connector>
What you're trying to do is a little difficult to accomplish but not impossible. The messages are received by the org.mule.transport.tcp.TcpMessageReceiver class, and this class always consumes the data in the input stream to create the message that injects in the flow.
However, you could extend that receiver and instruct the TCP module to use yours by adding a service-overrides tag in your flow's tcp connector (documented here) and replacing the messageReceiver element.
In your extended receiver you should change the TcpWorker.getNextMessage method in order to send the ack message before read from the input stream.
HTH, Marcos.
I am using ActiveMQ in my app. My question is how to delete messages that ı consumed successfully from kahadb. Because if it is not deleted, my db.data file is growing up constantly.
Here is my consumer;
ActiveMQConnectionFactory connectionFactory = new ActiveMQConnectionFactory("tcp://localhost:8182");
Connection connection = connectionFactory.createConnection();
connection.start();
Session session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);
Destination destination = session.createQueue("TEST.FOO");
MessageConsumer consumer = session.createConsumer(destination);
MessageListener listner = new MessageListener() {
int count = 0;
public void onMessage(Message message) {
if (message instanceof ObjectMessage) {
ObjectMessage objectMessage = (ObjectMessage) message;
ResponseDuration responseDuration = null;
try {
responseDuration = (ResponseDuration) objectMessage.getObject();
System.out.println("Received Time : " + new Date() + "Received: " + responseDuration.toString());
} catch (JMSException e1) {
// TODO Auto-generated catch block
e1.printStackTrace();
}
try {
ResponseDurationOperations.insertResponseDurations(responseDuration);
count++;
System.out.println("Count = " + count);
} catch (SQLException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
};
consumer.setMessageListener(listner);
Seems that ActiveMQ has a different meaning for what it means with persistance than you (and me as well).
Persistence is defined not to persist for ever but just to make you safe from message loss when you restart the server. See this
One option for you could be to switch off the persistence. See here.
For example by this way:
ActiveMQConnectionFactory("vm://localhost?broker.persistent=false");
Hi all please give some basic about ActiveMQ with JMS for novice. And configuration steps also.
We are going to create a console based application using multithreading. So create an java project for console application.
Now follow these steps..........
Add javax.jms.jar, activemq-all-5.3.0.jar, log4j-1.2.15.jar to your project library.
(You can download all of above jar files from http://www.jarfinder.com/ .
create a file naming jndi.properties and paste these following texts .. ( Deatils for jndi.properties just Google it)
# START SNIPPET: jndi
java.naming.factory.initial = org.apache.activemq.jndi.ActiveMQInitialContextFactory
# use the following property to configure the default connector
java.naming.provider.url = tcp://localhost:61616
# use the following property to specify the JNDI name the connection factory
# should appear as.
#connectionFactoryNames = connectionFactory, queueConnectionFactory, topicConnectionFactry
connectionFactoryNames = connectionFactory, queueConnectionFactory, topicConnectionFactry
# register some queues in JNDI using the form
# queue.[jndiName] = [physicalName]
queue.MyQueue = example.MyQueue
# register some topics in JNDI using the form
# topic.[jndiName] = [physicalName]
topic.MyTopic = example.MyTopic
# END SNIPPET: jndi
Add JMSConsumer.java
import javax.jms.*;
import javax.naming.Context;
import javax.naming.InitialContext;
import javax.naming.NamingException;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
public class JMSConsumer implements Runnable{
private static final Log LOG = LogFactory.getLog(JMSConsumer.class);
public void run() {
Context jndiContext = null;
ConnectionFactory connectionFactory = null;
Connection connection = null;
Session session = null;
MessageConsumer consumer = null;
Destination destination = null;
String sourceName = null;
final int numMsgs;
sourceName= "MyQueue";
numMsgs = 1;
LOG.info("Source name is " + sourceName);
/*
* Create a JNDI API InitialContext object
*/
try {
jndiContext = new InitialContext();
} catch (NamingException e) {
LOG.info("Could not create JNDI API context: " + e.toString());
System.exit(1);
}
/*
* Look up connection factory and destination.
*/
try {
connectionFactory = (ConnectionFactory)jndiContext.lookup("queueConnectionFactory");
destination = (Destination)jndiContext.lookup(sourceName);
} catch (NamingException e) {
LOG.info("JNDI API lookup failed: " + e);
System.exit(1);
}
try {
connection = connectionFactory.createConnection();
session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);
consumer = session.createConsumer(destination);
connection.start();
try {
Thread.sleep(2000);
} catch (InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
MessageListener listener = new MyQueueMessageListener();
consumer.setMessageListener(listener );
//Let the thread run for some time so that the Consumer has suffcient time to consume the message
try {
Thread.sleep(5000);
} catch (InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
} catch (JMSException e) {
LOG.info("Exception occurred: " + e);
} finally {
if (connection != null) {
try {
connection.close();
} catch (JMSException e) {
}
}
}
}
}
Add JMSProducer.java
import javax.jms.*;
import javax.naming.Context;
import javax.naming.InitialContext;
import javax.naming.NamingException;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
public class JMSProducer implements Runnable{
private static final Log LOG = LogFactory.getLog(JMSProducer.class);
public JMSProducer() {
}
//Run method implemented to run this as a thread.
public void run(){
Context jndiContext = null;
ConnectionFactory connectionFactory = null;
Connection connection = null;
Session session = null;
Destination destination = null;
MessageProducer producer = null;
String destinationName = null;
final int numMsgs;
destinationName = "MyQueue";
numMsgs = 5;
LOG.info("Destination name is " + destinationName);
/*
* Create a JNDI API InitialContext object
*/
try {
jndiContext = new InitialContext();
} catch (NamingException e) {
LOG.info("Could not create JNDI API context: " + e.toString());
System.exit(1);
}
/*
* Look up connection factory and destination.
*/
try {
connectionFactory = (ConnectionFactory)jndiContext.lookup("queueConnectionFactory");
destination = (Destination)jndiContext.lookup(destinationName);
} catch (NamingException e) {
LOG.info("JNDI API lookup failed: " + e);
System.exit(1);
}
/*
* Create connection. Create session from connection; false means
* session is not transacted.create producer, set the text message, set the co-relation id and send the message.
*/
try {
connection = connectionFactory.createConnection();
session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);
producer = session.createProducer(destination);
TextMessage message = session.createTextMessage();
for (int i = 0; i
Add MyQueueMessageListener.java
import java.io.*;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import javax.jms.*;
public class MyQueueMessageListener implements MessageListener {
private static final Log LOG = LogFactory.getLog(MyQueueMessageListener.class);
/**
*
*/
public MyQueueMessageListener() {
// TODO Auto-generated constructor stub
}
/** (non-Javadoc)
* #see javax.jms.MessageListener#onMessage(javax.jms.Message)
* This is called on receving of a text message.
*/
public void onMessage(Message arg0) {
LOG.info("onMessage() called!");
if(arg0 instanceof TextMessage){
try {
//Print it out
System.out.println("Recieved message in listener: " + ((TextMessage)arg0).getText());
System.out.println("Co-Rel Id: " + ((TextMessage)arg0).getJMSCorrelationID());
try {
//Log it to a file
BufferedWriter outFile = new BufferedWriter(new FileWriter("MyQueueConsumer.txt"));
outFile.write("Recieved message in listener: " + ((TextMessage)arg0).getText());
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
} catch (JMSException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}else{
System.out.println("~~~~Listener : Error in message format~~~~");
}
}
}
Add SimpleApp.java
public class SimpleApp {
//Run the producer first, then the consumer
public static void main(String[] args) throws Exception {
runInNewthread(new JMSProducer());
runInNewthread(new JMSConsumer());
}
public static void runInNewthread(Runnable runnable) {
Thread brokerThread = new Thread(runnable);
brokerThread.setDaemon(false);
brokerThread.start();
}
}
Now run SimpleApp.java class.
All da best. Happy coding.
Here it is a simple junit test for ActiveMQ and Apache Camel. This two technologies works very good together.
If you want more details about the code, you can find a post in my blog:
http://ignaciosuay.com/unit-testing-active-mq/
public class ActiveMQTest extends CamelTestSupport {
#Override
protected CamelContext createCamelContext() throws Exception {
CamelContext camelContext = super.createCamelContext();
ConnectionFactory connectionFactory = new ActiveMQConnectionFactory("vm://localhost?broker.persistent=false");
camelContext.addComponent("activemq", jmsComponentClientAcknowledge(connectionFactory));
return camelContext;
}
#Override
protected RouteBuilder createRouteBuilder() throws Exception {
return new RouteBuilder() {
#Override
public void configure() throws Exception {
from("mina:tcp://localhost:6666?textline=true&sync=false")
.to("activemq:processHL7");
from("activemq:processHL7")
.to("mock:end");
}
};
}
#Test
public void testSendHL7Message() throws Exception {
MockEndpoint mock = getMockEndpoint("mock:end");
String m = "MSH|^~\\&|hl7Integration|hl7Integration|||||ADT^A01|||2.5|\r" +
"EVN|A01|20130617154644\r" +
"PID|1|465 306 5961||407623|Wood^Patrick^^^MR||19700101|1|\r" +
"PV1|1||Location||||||||||||||||261938_6_201306171546|||||||||||||||||||||||||20130617134644|";
mock.expectedBodiesReceived(m);
template.sendBody("mina:tcp://localhost:6666?textline=true&sync=false", m);
mock.assertIsSatisfied();
}