I'd like to create a consumer which stacks arriving messages, and waits:
until n messages have arrived.
t seconds are elapsed.
to process the whole stack of messages.
Pre-fetching is not what I'm looking for. What I really need is to process messages together.
class MyListener(stomp.ConnectionListener):
def on_message(self, headers, body):
print ("Just received ONE message\n"
"I should wait for n-1 others\n"
"or t seconds before processing")
here an example
import java.util.LinkedList;
import java.util.List;
import javax.jms.Connection;
import javax.jms.JMSException;
import javax.jms.Session;
import javax.jms.TextMessage;
import org.apache.activemq.ActiveMQConnectionFactory;
import org.apache.activemq.ActiveMQMessageConsumer;
public class SimpleConsumerClientAcknowledge {
public static void main(String[] args) throws JMSException {
List<TextMessage> messages = new LinkedList<>();
Connection conn = null;
try {
ActiveMQConnectionFactory cf = new ActiveMQConnectionFactory(
"tcp://localhost:61617?jms.prefetchPolicy.all=200");
conn = cf.createConnection("admin", "admin");
Session session = conn.createSession(false, Session.CLIENT_ACKNOWLEDGE);
ActiveMQMessageConsumer consumer = (ActiveMQMessageConsumer) session
.createConsumer(session.createQueue("Q"));
conn.start();
TextMessage msg = null;
// MAX_MESSAGES have to be < prefetchSize / 2 -->
// jms.prefetchPolicy.all=200
// Once the broker has dispatched a prefetch limit number of
// messages to a consumer it will not dispatch any more messages to
// that consumer until the consumer has acknowledged at least 50% of
// the prefetched messages
int MAX_MESSAGES = 100;
long MAX_WAIT = 60000;
long millis = System.currentTimeMillis();
while ((msg = (TextMessage) consumer.receive(5000)) != null) {
if (msg != null) {
messages.add(msg);
}
if (messages.size() == MAX_MESSAGES || (System.currentTimeMillis() - millis >= MAX_WAIT)) {
millis = System.currentTimeMillis();
treatMessages(messages);
// because session is created with
// Session.CLIENT_ACKNOWLEDGE as an acknowledgeMode consumer
// need to acknowledge manually received messages
consumer.acknowledge();
}
}
} catch (Exception e) {
e.printStackTrace();
} finally {
if (conn != null) {
try {
conn.close();
} catch (Exception e) {
}
}
}
}
private static void treatMessages(List<TextMessage> messages) {
// TODO Auto-generated method stub
messages.clear();
}
}
Related
Hi I have downloaded pcap4j and winpcap and all jar (jna, pcap4j-core-1.8.2, slf4j-api-1.7.25, slf4j-simple-1.7.25) dependency manually. Added to the project and all compile well.
BUT:
when I began to sniff packet.getHeader() and packet.getPayload() returns null!
if I run manually rpcapd.exe then it works...
why?
package sniffer;
import java.io.IOException;
import org.pcap4j.core.BpfProgram.BpfCompileMode;
import org.pcap4j.core.NotOpenException;
import org.pcap4j.core.PacketListener;
import org.pcap4j.core.PcapHandle;
import org.pcap4j.core.PcapNativeException;
import org.pcap4j.core.PcapNetworkInterface;
import org.pcap4j.core.PcapNetworkInterface.PromiscuousMode;
import org.pcap4j.packet.Packet;
import org.pcap4j.util.NifSelector;
public class App {
static PcapNetworkInterface getNetworkDevice() {
PcapNetworkInterface device = null;
try {
device = new NifSelector().selectNetworkInterface();
} catch (IOException e) {
e.printStackTrace();
}
return device;
}
public static void main(String[] args) throws PcapNativeException, NotOpenException {
// The code we had before
PcapNetworkInterface device = getNetworkDevice();
System.out.println("You chose: " + device);
// New code below here
if (device == null) {
System.out.println("No device chosen.");
System.exit(1);
}
// Open the device and get a handle
int snapshotLength = 65536; // in bytes
int readTimeout = 50; // in milliseconds
final PcapHandle handle;
handle = device.openLive(snapshotLength, PromiscuousMode.PROMISCUOUS, readTimeout);
String filter = "tcp port 80";
handle.setFilter(filter, BpfCompileMode.OPTIMIZE);
// Create a listener that defines what to do with the received packets
PacketListener listener = new PacketListener() {
#Override
public void gotPacket(Packet packet) {
// Override the default gotPacket() function and process packet
System.out.println(handle.getTimestamp());
System.out.println(packet);
System.out.println(packet.getHeader());///////////////<<<<<<<<<<<------------
}
};
// Tell the handle to loop using the listener we created
try {
int maxPackets = 50;
handle.loop(maxPackets, listener);
} catch (InterruptedException e) {
e.printStackTrace();
}
// Cleanup when complete
handle.close();
}
}
You need to add a packet factory (e.g. pcap4j-packetfactory-static.jar) to your classpath, or Pcap4J creates UnknownPacket instances, getPayload() and getHeader() of which return null, for all packets.
Here is my code:
package pushnotiruntest;
import com.rabbitmq.client.BuiltinExchangeType;
import com.rabbitmq.client.Channel;
import com.rabbitmq.client.Connection;
import com.rabbitmq.client.ConnectionFactory;
import java.io.IOException;
import java.util.concurrent.TimeoutException;
import java.util.logging.Level;
import java.util.logging.Logger;
public class Send extends Thread {
String name = "";
String app_type = "";
private static final String EXCHANGE_NAME = "topic_exchange";
public void run()
{
ConnectionFactory connFac = new ConnectionFactory();
connFac.setHost("localhost");
try {
Connection conn = connFac.newConnection();
Channel channel = conn.createChannel();
channel.exchangeDeclare(EXCHANGE_NAME, BuiltinExchangeType.TOPIC);
for(int j=1; j<=20000; j++)
{
//randomWait();
String routingKey = j+"."+"update"+"."+app_type;
String msg = name;
channel.basicPublish(EXCHANGE_NAME, routingKey, null, msg.getBytes("UTF-8"));
System.out.println("Sent " + routingKey + " : " + msg + "");
}
channel.close();
conn.close();
} catch (IOException ex) {
Logger.getLogger(Send.class.getName()).log(Level.SEVERE, null, ex);
System.out.println("Exception1 :--"+ex);
} catch (TimeoutException ex) {
Logger.getLogger(Send.class.getName()).log(Level.SEVERE, null, ex);
System.out.println("Exception 2:--"+ex);
}
}
/*void randomWait()
{
try {
Thread.currentThread().sleep((long)(3*Math.random()));
} catch (InterruptedException x) {
System.out.println("Interrupted!");
}
}*/
public static void main(String[] args) {
// TODO code application logic here
Send test1 = new Send();
test1.name = "Hello ANDROID";
test1.app_type = "ANDROID";
Send test2 = new Send();
test2.name = "Hello IOS";
test2.app_type = "IOS";
Send test3 = new Send();
test3.name = "Hello WINDOWS";
test3.app_type = "WINDOWS";
test1.start();
test2.start();
test3.start();
}
}
//javac -cp amqp-client-4.0.2.jar Send.java Recv.java
//java -cp .;amqp-client-4.0.2.jar;slf4j-api-1.7.21.jar;slf4j-simple-1.7.22.jar Recv
//java -cp .;amqp-client-4.0.2.jar;slf4j-api-1.7.21.jar;slf4j-simple-1.7.22.jar
Send
I am writing a code in Java (message broker used is RabbitMQ) where I want to store messages send by the producers in a single queue with different routing key.
And fetch that messages with respect to there pattern for different consumers
that matches the routing key pattern. (I am using Topic exchange for the pattern matching).
If you need two consumers, you have to use two queues.
The binding if from exchange to queue(s) you cannot decide the routing key during the subscription.
You can bind more routing key to the same queue, but you can't consume it filtering by the key.
I think you need something like:
channel.exchangeDeclare(EXCHANGE_NAME, BuiltinExchangeType.TOPIC);
channel.queueDeclare(QUEUE_NAME_1, true, false, false, null);
channel.queueDeclare(QUEUE_NAME_2, true, false, false, null);
channel.queueBind(QUEUE_NAME_1, EXCHANGE_NAME, "my.rk.1");
channel.queueBind(QUEUE_NAME_2, EXCHANGE_NAME, "my.rk.2");
channel_consumer_1.basicConsume(QUEUE_NAME_1, false, new DefaultConsumer(channel_consumer) {...}
....
channel_consumer_2.basicConsume(QUEUE_NAME_2, false, new DefaultConsumer(channel_consumer) {...}
I am developing system where messages to be processed are pushed in ActiveMQ. I have stringent requirement that consumer must process messages in incoming sequence. If a message processing fails in consumer, it need to rollback/recover and keep on re-trying infinitely. Only when a message processing is successful, consumer need to commit and proceed to next message.
How to prevent rolled-back message auto-forwarding to DLQ and what's proper way of configuring re-delivery policy for such requirement?
when set RedeliveryPolicy re-trying infinitely like below the messages will never be sent to DLQ.
policy.setMaximumRedeliveries(RedeliveryPolicy.NO_MAXIMUM_REDELIVERIES);
with ActiveMQSession.INDIVIDUAL_ACKNOWLEDGE you acknowledge messages one by one.
http://activemq.apache.org/redelivery-policy.html
import javax.jms.Connection;
import javax.jms.JMSException;
import javax.jms.Message;
import javax.jms.MessageListener;
import org.apache.activemq.ActiveMQConnectionFactory;
import org.apache.activemq.ActiveMQMessageConsumer;
import org.apache.activemq.ActiveMQSession;
import org.apache.activemq.RedeliveryPolicy;
public class SimpleConsumerIndividualAcknowledge {
public static void main(String[] args) throws JMSException {
Connection conn = null;
try {
ActiveMQConnectionFactory cf = new ActiveMQConnectionFactory("tcp://localhost:61616");
RedeliveryPolicy policy = new RedeliveryPolicy();
policy.setMaximumRedeliveries(RedeliveryPolicy.NO_MAXIMUM_REDELIVERIES);
cf.setRedeliveryPolicy(policy);
conn = cf.createConnection();
ActiveMQSession session = (ActiveMQSession) conn.createSession(false,
ActiveMQSession.INDIVIDUAL_ACKNOWLEDGE);
ActiveMQMessageConsumer consumer = (ActiveMQMessageConsumer) session
.createConsumer(session.createQueue("test"));
consumer.setMessageListener(new MessageListener() {
#Override
public void onMessage(Message message) {
try {
//do your stuff
message.acknowledge();
} catch (Exception e) {
throw new RuntimeException(e);//ActiveMQMessageConsumer.rollback() is called automatically
}
}
});
conn.start();
} catch (Exception e) {
e.printStackTrace();
} finally {
if (conn != null) {
try {
conn.close();
} catch (Exception e) {
}
}
}
}
}
if you want to manually stop and restart consumer take a look here activemq-redelivery-does-not-work
I have a 3 node EC2 redis cluster setup and I am trying to add records to redis (using sadd) with pipeline mode.
I get the following error after adding about 70/82 and 81 keys in 3 nodes:
Exception in thread "main" redis.clients.jedis.exceptions.JedisMovedDataException: MOVED 1539 172.31.59.103:6379
at redis.clients.jedis.Protocol.processError(Protocol.java:93)
at redis.clients.jedis.Protocol.process(Protocol.java:122)
at redis.clients.jedis.Protocol.read(Protocol.java:191)
at redis.clients.jedis.Connection.getOne(Connection.java:258)
at redis.clients.jedis.ShardedJedisPipeline.sync(ShardedJedisPipeline.java:44)
at org.hu.e63.MovieLens21MPipeline.push(MovieLens21MPipeline.java:47)
at org.hu.e63.MovieLens21MPipeline.main(MovieLens21MPipeline.java:53
I have looked at this thread, my code pretty much looks like that:
Input file is from here(ratings.csv): http://files.grouplens.org/datasets/movielens/ml-latest-small.zip
Here's the code:
import java.io.File;
import java.util.ArrayList;
import java.util.List;
import java.util.Scanner;
import redis.clients.jedis.JedisShardInfo;
import redis.clients.jedis.ShardedJedis;
import redis.clients.jedis.ShardedJedisPipeline;
public class MovieLens21MPipeline {
ShardedJedis jedis;
public MovieLens21MPipeline() {
JedisShardInfo si = new JedisShardInfo("172.31.59.103", 6379, 5000);
List<JedisShardInfo> list = new ArrayList<JedisShardInfo>();
list.add(si);
list.add(new JedisShardInfo("172.31.59.104", 6379, 5000));
list.add(new JedisShardInfo("172.31.59.105", 6379, 5000));
jedis = new ShardedJedis(list);
}
public void push() {
ShardedJedisPipeline pipeline = jedis.pipelined();
Scanner s;
try {
s = new Scanner(new File("input/ratings.csv"));
StringBuilder key = new StringBuilder();
String s1 = s.nextLine(); // Skip first line
while (s.hasNextLine()) {
s1 = s.nextLine();
String[] spl = s1.split(",");
// key="u:"+spl[0]+":m";
key.append("u:").append(spl[0]).append(":m");
pipeline.sadd(key.toString(), spl[1]);
key.setLength(0);
}
} catch (Exception e) {
// TODO Auto-generated catch block
System.out.println("bye" + e.getMessage());
}
pipeline.sync();
}
public static void main(String[] args) {
MovieLens21MPipeline obj = new MovieLens21MPipeline();
long startTime = System.currentTimeMillis();
obj.push();
long endTime = System.currentTimeMillis();
double d = 0.0;
d = (double) (endTime - startTime);
System.out.println("Throughput: " + d);
}
}
Any help appreciated.
MOVED indicates that you're using Redis Cluster. ShardedJedis is not for Redis Cluster, so you should use JedisCluster instead.
Please note that JedisCluster doesn't have pipeline mode, so you may want to send your operation one by one.
Hope this helps.
Just like Lim said, you are using Redis in cluster mode. Use JedisCluster can solve this problem. In addition, as a supplement, why JedisCluster works?
We can known that from JedisCluster source code.
private T runWithRetries(byte[] key, int attempts, boolean tryRandomNode, boolean asking) {
if (attempts <= 0) {
throw new JedisClusterMaxRedirectionsException("Too many Cluster redirections?");
}
Jedis connection = null;
try {
// omit ... get connection
return execute(connection);
} catch (JedisNoReachableClusterNodeException jnrcne) {
throw jnrcne;
} catch (JedisConnectionException jce) {
// release current connection before recursion
// omit
return runWithRetries(key, attempts - 1, tryRandomNode, asking);
} catch (JedisRedirectionException jre) {
// if MOVED redirection occurred,
if (jre instanceof JedisMovedDataException) {
// here is the key point
// it rebuilds cluster's slot cache
// recommended by Redis cluster specification
// Jedis or ShardedJedis won't run this code
this.connectionHandler.renewSlotCache(connection);
}
// omit ...
return runWithRetries(key, attempts - 1, false, asking);
} finally {
releaseConnection(connection);
}
}
Hi all please give some basic about ActiveMQ with JMS for novice. And configuration steps also.
We are going to create a console based application using multithreading. So create an java project for console application.
Now follow these steps..........
Add javax.jms.jar, activemq-all-5.3.0.jar, log4j-1.2.15.jar to your project library.
(You can download all of above jar files from http://www.jarfinder.com/ .
create a file naming jndi.properties and paste these following texts .. ( Deatils for jndi.properties just Google it)
# START SNIPPET: jndi
java.naming.factory.initial = org.apache.activemq.jndi.ActiveMQInitialContextFactory
# use the following property to configure the default connector
java.naming.provider.url = tcp://localhost:61616
# use the following property to specify the JNDI name the connection factory
# should appear as.
#connectionFactoryNames = connectionFactory, queueConnectionFactory, topicConnectionFactry
connectionFactoryNames = connectionFactory, queueConnectionFactory, topicConnectionFactry
# register some queues in JNDI using the form
# queue.[jndiName] = [physicalName]
queue.MyQueue = example.MyQueue
# register some topics in JNDI using the form
# topic.[jndiName] = [physicalName]
topic.MyTopic = example.MyTopic
# END SNIPPET: jndi
Add JMSConsumer.java
import javax.jms.*;
import javax.naming.Context;
import javax.naming.InitialContext;
import javax.naming.NamingException;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
public class JMSConsumer implements Runnable{
private static final Log LOG = LogFactory.getLog(JMSConsumer.class);
public void run() {
Context jndiContext = null;
ConnectionFactory connectionFactory = null;
Connection connection = null;
Session session = null;
MessageConsumer consumer = null;
Destination destination = null;
String sourceName = null;
final int numMsgs;
sourceName= "MyQueue";
numMsgs = 1;
LOG.info("Source name is " + sourceName);
/*
* Create a JNDI API InitialContext object
*/
try {
jndiContext = new InitialContext();
} catch (NamingException e) {
LOG.info("Could not create JNDI API context: " + e.toString());
System.exit(1);
}
/*
* Look up connection factory and destination.
*/
try {
connectionFactory = (ConnectionFactory)jndiContext.lookup("queueConnectionFactory");
destination = (Destination)jndiContext.lookup(sourceName);
} catch (NamingException e) {
LOG.info("JNDI API lookup failed: " + e);
System.exit(1);
}
try {
connection = connectionFactory.createConnection();
session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);
consumer = session.createConsumer(destination);
connection.start();
try {
Thread.sleep(2000);
} catch (InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
MessageListener listener = new MyQueueMessageListener();
consumer.setMessageListener(listener );
//Let the thread run for some time so that the Consumer has suffcient time to consume the message
try {
Thread.sleep(5000);
} catch (InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
} catch (JMSException e) {
LOG.info("Exception occurred: " + e);
} finally {
if (connection != null) {
try {
connection.close();
} catch (JMSException e) {
}
}
}
}
}
Add JMSProducer.java
import javax.jms.*;
import javax.naming.Context;
import javax.naming.InitialContext;
import javax.naming.NamingException;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
public class JMSProducer implements Runnable{
private static final Log LOG = LogFactory.getLog(JMSProducer.class);
public JMSProducer() {
}
//Run method implemented to run this as a thread.
public void run(){
Context jndiContext = null;
ConnectionFactory connectionFactory = null;
Connection connection = null;
Session session = null;
Destination destination = null;
MessageProducer producer = null;
String destinationName = null;
final int numMsgs;
destinationName = "MyQueue";
numMsgs = 5;
LOG.info("Destination name is " + destinationName);
/*
* Create a JNDI API InitialContext object
*/
try {
jndiContext = new InitialContext();
} catch (NamingException e) {
LOG.info("Could not create JNDI API context: " + e.toString());
System.exit(1);
}
/*
* Look up connection factory and destination.
*/
try {
connectionFactory = (ConnectionFactory)jndiContext.lookup("queueConnectionFactory");
destination = (Destination)jndiContext.lookup(destinationName);
} catch (NamingException e) {
LOG.info("JNDI API lookup failed: " + e);
System.exit(1);
}
/*
* Create connection. Create session from connection; false means
* session is not transacted.create producer, set the text message, set the co-relation id and send the message.
*/
try {
connection = connectionFactory.createConnection();
session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);
producer = session.createProducer(destination);
TextMessage message = session.createTextMessage();
for (int i = 0; i
Add MyQueueMessageListener.java
import java.io.*;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import javax.jms.*;
public class MyQueueMessageListener implements MessageListener {
private static final Log LOG = LogFactory.getLog(MyQueueMessageListener.class);
/**
*
*/
public MyQueueMessageListener() {
// TODO Auto-generated constructor stub
}
/** (non-Javadoc)
* #see javax.jms.MessageListener#onMessage(javax.jms.Message)
* This is called on receving of a text message.
*/
public void onMessage(Message arg0) {
LOG.info("onMessage() called!");
if(arg0 instanceof TextMessage){
try {
//Print it out
System.out.println("Recieved message in listener: " + ((TextMessage)arg0).getText());
System.out.println("Co-Rel Id: " + ((TextMessage)arg0).getJMSCorrelationID());
try {
//Log it to a file
BufferedWriter outFile = new BufferedWriter(new FileWriter("MyQueueConsumer.txt"));
outFile.write("Recieved message in listener: " + ((TextMessage)arg0).getText());
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
} catch (JMSException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}else{
System.out.println("~~~~Listener : Error in message format~~~~");
}
}
}
Add SimpleApp.java
public class SimpleApp {
//Run the producer first, then the consumer
public static void main(String[] args) throws Exception {
runInNewthread(new JMSProducer());
runInNewthread(new JMSConsumer());
}
public static void runInNewthread(Runnable runnable) {
Thread brokerThread = new Thread(runnable);
brokerThread.setDaemon(false);
brokerThread.start();
}
}
Now run SimpleApp.java class.
All da best. Happy coding.
Here it is a simple junit test for ActiveMQ and Apache Camel. This two technologies works very good together.
If you want more details about the code, you can find a post in my blog:
http://ignaciosuay.com/unit-testing-active-mq/
public class ActiveMQTest extends CamelTestSupport {
#Override
protected CamelContext createCamelContext() throws Exception {
CamelContext camelContext = super.createCamelContext();
ConnectionFactory connectionFactory = new ActiveMQConnectionFactory("vm://localhost?broker.persistent=false");
camelContext.addComponent("activemq", jmsComponentClientAcknowledge(connectionFactory));
return camelContext;
}
#Override
protected RouteBuilder createRouteBuilder() throws Exception {
return new RouteBuilder() {
#Override
public void configure() throws Exception {
from("mina:tcp://localhost:6666?textline=true&sync=false")
.to("activemq:processHL7");
from("activemq:processHL7")
.to("mock:end");
}
};
}
#Test
public void testSendHL7Message() throws Exception {
MockEndpoint mock = getMockEndpoint("mock:end");
String m = "MSH|^~\\&|hl7Integration|hl7Integration|||||ADT^A01|||2.5|\r" +
"EVN|A01|20130617154644\r" +
"PID|1|465 306 5961||407623|Wood^Patrick^^^MR||19700101|1|\r" +
"PV1|1||Location||||||||||||||||261938_6_201306171546|||||||||||||||||||||||||20130617134644|";
mock.expectedBodiesReceived(m);
template.sendBody("mina:tcp://localhost:6666?textline=true&sync=false", m);
mock.assertIsSatisfied();
}