I am trying to consume JMS messages (IBM Websphere MQ) using Apache Flume and storing the data to HDFS. While reading the message, i am only able to see the body of the message and not the header content of the message.
Is it possible to read the jms message with the header property using Apache Flume?
My configuration:
# Source definition
u.sources.s1.type=jms
u.sources.s1.initialContextFactory=ABC
u.sources.s1.connectionFactory=<my connection factory>
u.sources.s1.providerURL=ABC
u.sources.s1.destinationName=r1
u.sources.s1.destinationType=QUEUE
# Channel definition
u.channels.c1.type=file
u.channels.c1.capacity=10000000
u.channels.c1.checkpointDir=/checkpointdir
u.channels.c1.transactionCapacity=10000
u.channels.c1.dataDirs=/datadir
# Sink definition
u.sinks.r1.type=hdfs
u.sinks.r1.channel=c1
u.sinks.r1.hdfs.path=/message/%Y%m%d
u.sinks.r1.hdfs.filePrefix=event_
u.sinks.r1.hdfs.fileSuffix=.xml
u.sinks.r1.hdfs.fileType = DataStream
u.sinks.r1.hdfs.writeFormat=Text
u.sinks.r1.hdfs.useLocalTimeStamp=TRUE
There are quite a few types of JMS messages as in "Table 30–2 JMS Message Types" here.
The Flume DefaultJMSMessageConverter uses TextMessage as in here and is given below for your reference:
...
else if(message instanceof TextMessage) {
TextMessage textMessage = (TextMessage)message;
event.setBody(textMessage.getText().getBytes(charset));
}
...
TextMessage offers only body of the message.
IMHO, you have two options:
If at all possible, send the message-header, header-value pair in the body itself and use the "DefaultJMSMessageConverter" as is.
Build your own "flume-jms-source.jar" by writing a custom JMSMessageConverter and type-cast the "message" to javax.jms.Message, get the JMS headers, set them in SimpleEvent.
Hope this gives some direction.
Related
I am trying to put two custom properties in a STOMP message header when publishing a topic message so that a subscriber can filter messages. Here are two frames that I send to ActiveMQ 5.14 to connect and publish:
CONNECT
login: myUserName
passcode: myPassword
Note: Actual string is CONNECT\nlogin: myUserName\npasscode: myPassword.
and
SEND
destination:/topic/myTopic
myTopicMessage
Note: Actual string is SEND\ndestination:/topic/myTopic\n\nmyTopicMessage.
How am I supposed to add the following two pairs of properties to above strings?
package_code = ''
whse_code = 'MyWarehouse'
BTW, I am using lua to implement this.
You can add the properties to your SEND frame with the same syntax used by destination, e.g.:
SEND
destination:/topic/myTopic
package_code:MyPackageCode
whse_code:MyWarehouse
myTopicMessage^#
If package_code (or any other header) is blank simply don't set it.
A few other details are worth noting:
Be sure to follow the body of the message with the NULL octet as noted in the "STOMP Frames" section of the STOMP 1.2 spec. The example above uses ^# (i.e. control-# in ASCII) to represent the NULL octet.
SEND frames should include a content-length header and a content-type header if a body is present as noted in the "SEND" section of the STOMP 1.2 spec.
Troubleshooting:
You can enable STOMP protocol tracing with the following steps:
ActiveMQ 5.x: Set trace=true on the STOMP transportConnector, e.g.: <transportConnector name="stomp" uri="stomp://localhost:61613?trace=true"/>. Then set the org.apache.activemq.transport.stomp.StompIO logger to TRACE in conf/log4j.properties
ActiveMQ Artemis: Set the logger org.apache.activemq.artemis.core.protocol.stomp.StompConnection to DEBUG in etc/logging.properties.
I'm new in to RabbitMQ and I've faced a problem. I'm trying to get messages from queue by API method. I've made that by now I want to get messages from queue by header or property if it is possible. I read the documentation about HTTP API. I have not found such an API for filtering messages by some headers or properties.
I use that kind of API to get messages from queue:
/api/queues/vhost/name/get
and in the body:
{"count":20,"ackmode":"ack_requeue_true","encoding":"auto"}
I was thinking, maybe it is possible to somehow pass some filter in the body so it could filter and return the message what I want.
This is how my message looks like :
I have tried to pass in the body type = "myType" or header = "myHeader"
I've made that by now I want to get messages from queue by header or
property if it is possible.
RabbitMQ only delivers messages in order from a queue. There is no way to filter once a message is in a queue.
You can filter messages as they are published to an exchange, however. Use a headers exchange and bind queues based on header values. Then, each queue will contain the messages you expect and you can then consume from them.
The RabbitMQ tutorials have a section that use a "headers exchange". Use that as a guide.
Finally, only use the HTTP API for testing. It is a very inefficient way to retrieve messages.
NOTE: the RabbitMQ team monitors the rabbitmq-users mailing list and only sometimes answers questions on StackOverflow.
A bit late to the party, but I think you can achieve the same like this
ConnectionFactory factory = new ConnectionFactory();
factory.setHost(hostname);
Connection conn = factory.newConnection();
Channel channel = connection.createChannel();
channel.queueBind(queueName, exchangeName, "");
DeliverCallback deliverCallback = (consumerTag, delivery) -> {
Map<String, Object> headers = delivery.getProperties().getHeaders();
String message = new String(delivery.getBody(), "UTF-8");
System.out.println(" [x] Received '" + message + "', with header : " + headers.get("TestHeader") );
};
channel.basicConsume(queue, true, deliverCallback, consumerTag -> { });
I am creating utiltiy which send mail using my outlook account, to do so I am creating object of javax.mail.Message and send it, if message sending is failed due to SendingFailedException, I want to add those messages into jms queue, and at the other end listener will run at every 10 min interval to consume these messages from the queue and try to resend those messages.
I have gone through some of the stackoverflow topics related to same, they instructed to change message into xml or in JSON, I just want to know how to deal with it, if that would be the way to implement this.
Thanks in advance
Using the MimeMessage.writeTo method you can turn the message into a byte stream. Collect it in a ByteArrayOutputStream and then include the bytes in the JMS message. At the other end, you can reconstitute the message using the MimeMessage constructor that takes an InputStream.
For example:
ByteArrayOutputStream bos = new ByteArrayOutputStream();
msg.writeTo(bos);
byte[] data = bos.toByteArray();
// put the data in a JMS message
// in the receiver, extract the byte array from the message
byte[] data = ...
MimeMessage msg = new MimeMessage(session, data);
Sorry, I can't help you with the JMS part.
I am new to activeMQ, I have issues pushing messages to a queue defined by activeMQ from a message producer residing on another server.
I have a few queues in the application created on activeMQ using camel routes. I am trying to perform remote JNDI lookup on these queues from an application on another server. I have used the snippets from activemq documentation from http://activemq.apache.org/jndi-support.html page.
I could get connected to the activeMQ, but I couldn't look up the queues defined using camel routes.
The queue consumer is created through the camel route defined below.
from("jms:queue:APP.IF.JMS.OUTBOUND")
.... // This route does some processing.
But I don't see this queue in the lookup as performed below -
String destination = "APP.IF.JMS.OUTBOUND";
ConnectionFactory cf = null;
Destination dest = null;
Context ctx = null;
Properties params = new Properties();
readProperty(params, Context.INITIAL_CONTEXT_FACTORY, "org.apache.activemq.jndi.ActiveMQInitialContextFactory", false);
readProperty(params, Context.PROVIDER_URL, "tcp://localhost:61616", false);
readProperty(params, "queue.AS.IF.JMS.REQUEST",
"AS.IF.JMS.REQUEST", false);
ctx = new InitialContext(params);
cf = (ConnectionFactory) ctx.lookup("ConnectionFactory");
System.out.println(ctx.getEnvironment());
dest = (Destination) ctx.lookup(destination);
..............
The last line fails when lookup is done on this queue. I do see this on the console. Am I missing some configuration to expose this queue on JNDI?
Appreciate your response.
Change the word localhost in your PROVIDER_URL to your machineName.
Since it is remote operation, it needs your machine name.
I was currently using NMS to develop application based ActiveMQ(5.6).
We have several consumers(exe) trying to recieving massgaes from the same queue(not topic). While all the messages just all go to one consumer though I have make the consumer to sleep for seconds after recieving a message. By the way, we don't want the consumers recieving the same messages other consumers have recieved.
It is mentioned in the official website that we should set Prefetch Limit to decide how many messages can be streamed to a consumer at any point in time. And it can both be configured and coded.
One way I tried is to code using PrefetchPolicy class binding the ConnectionFactory class like bellow.
PrefetchPolicy poli = new PrefetchPolicy();
poli.QueuePrefetch = 0;
ConnectionFactory fac = new ConnectionFactory("activemq:tcp://Localhost:61616?jms.prefetchPolicy.queuePrefetch=1");
fac.PrefetchPolicy = poli;
using (IConnection con = fac.CreateConnection())
{
using (ISession se = con.CreateSession())
{
IDestination destination = SessionUtil.GetDestination(se, queue, DestinationType.Queue);
using (IMessageConsumer consumer = se.CreateConsumer(queue1))
{
con.Start();
while (true)
{
ITextMessage message = consumer.Receive() as ITextMessage;
Thread.Sleep(2000);
if (message != null)
{
Task.Factory.StartNew(() => extractAndSend(message.Text)); //do something
}
else
{
Console.WriteLine("No message received~");
}
}
}
}
}
But no matter what prefetch value I set the behavior of the consumers stay the same as before.
And I've tried the second way tying to get the result, namely configure the server conf file. I change the activemq.xml of the server like bellow.
" producerFlowControl="true" memoryLimit="5mb" />
" producerFlowControl="true" memoryLimit="5mb">
But though I've set the dispatchpolicy the messages still go to one consumer.
I want to know that:
Whether this behavior can be achieved by just configuring the server xml file to enable all the consumers recieve messages from one queue? If so, how to configure this and what is wrong with my configuration? If not, how can I use codes to achieve the goal?
Thanks.
Take a look at "Message Groups" feature.
I had the same problem. Only one consumer processed all messages. I found in my code I used group header during send:
request.Properties["NMSXGroupID"] = "cheese";
According to official docs:
Standard JMS header JMSXGroupID is used to define which message group
the message belongs to. The Message Group feature then ensures that
all messages for the same message group will be sent to the same JMS
consumer - while that consumer stays alive. As soon as the consumer
dies another will be chosen.
See full details at http://activemq.apache.org/message-groups.html