Selectively consume messages based on message body attributes in RabbitMQ - rabbitmq

Let's say I have a situation where I need to wait for up to 1 minute for some action to be performed.
If it is expired, then try different action.
My current solution proposal is based on RabbitMQ features.
I would create following resources:
#Bean
DirectExchange exchangeDirect() {
return new DirectExchange("exchange.direct");
}
#Bean
Queue bufferQueue() {
Map<String, Object> args = new HashMap<>();
args.put("x-message-ttl", amqpProperties.getTimeToLive().toMillis());
args.put("x-dead-letter-exchange", "exchange.direct");
args.put("x-dead-letter-routing-key", "timedOutQueue");
return new Queue("buffer.queue", true, false, false, args);
}
#Bean
Queue timedOutQueue() {
return new Queue("timed.out.queue", true);
}
#Bean
Binding bufferQueueToExchangeDirect() {
return bind(bufferQueue())
.to(exchangeDirect())
.with("buffer.queue");
}
#Bean
Binding timedOutQueueToExchangeDirect() {
return bind(timedOutQueue())
.to(exchangeDirect())
.with("timed.out.queue");
}
When I add action to bufferQueue and I don't receive any delivery update within 1 minute, this request is then moved to timedOutQueue thanks to bufferQueue's TTL.
I can attach application rabbit listener to timedOutQueue and use different action.
When I add action to bufferQueue and I receive confirmation that action was successfully performed, I'd like to remove given action event from bufferQueue.
I couldn't find such feature in RabbitMQ, i.e. being able to receive selectively.
I also found some articles saying that selective consuming is antipattern.
Is it possible to selectively consume messages from RabbitMQ queue?
What is proper way to implement this pattern in RabbitMQ?

There is no concept of message selection in RabbitMQ.
The "proper" way for an application that wants to selectively receive messages is to use multiple queues/routing keys with a consumer on each specific queue he expresses interest in.
However, there is no way to "remove" a message from the middle of a queue; only the head.
When I add action to bufferQueue and I receive confirmation that action was successfully performed, I'd like to remove given action event from bufferQueue.
That makes no sense to me; when the message timed out in bufferQueue due to TTL, and was moved to timedOutQueue, it no longer exists in bufferQueue so there is nothing to remove.
There is also no mechanism to ...
and I don't receive any delivery update within 1 minute,
... because each message in a queue is independent.
It doesn't sound like your application is suitable for a message broker at all.

Related

Servicestack.Redis Pub/Sub limitations with other nested Redis commands

I am having a great experience with ServiceStack & Redis, but I'm confused by ThreadPool and Pub/Sub within a thread, and an apparent limitation for accessing Redis within a message callback. The actual error I get states that I can only call "Subscribe" or "Publish" within the "current context". This happens when I try to do another Redis action from the message callback.
I have a process that must run continuously. In my case I can't just service a request one time, but must keep a thread alive all the time doing calculations (and controlling these threads from a REST API route is ideal). Data must come in to the process on a regular basis, and data must be published. The process must also store and retrieve data from Redis. I am using routes and services to take data in and store it in Redis, so this must take place async from the "calculation" process. I thought pub/sub would be the answer to glue the pieces together, but so far that does not seem possible.
Here is how my code is currently structured (the code with the above error). This is the callback for the route that starts the long term "calculation" thread:
public object Get(SystemCmd request)
{
object ctx = new object();
TradingSystemCmd SystemCmd = new TradingSystemCmd(request, ctx);
ThreadPool.QueueUserWorkItem(x =>
{
SystemCmd.signalEngine();
});
return (retVal); // retVal defined elsewhere
}
Here is the SystemCmd.signalEngine():
public void signalEngine(){
using (var subscription = Redis.CreateSubscription())
{
subscription.OnSubscribe = channel =>
{
};
subscription.OnUnSubscribe = channel =>
{
};
subscription.OnMessage = (channel, msg) =>
{
TC_CalcBar(channel, redisTrade);
};
subscription.SubscribeToChannels(dmx_key); //blocking
}
}
The "TC_CalcBar" call does processing on data as it becomes available. Within this call is a call to Redis for a regular database accesses (and the error). What I could do would be to remove the Subscription and use another method to block on data being available in Redis. But the current approach seemed quite nice until it failed to work. :-)
I also don't know if the ThreadPool has anything to do with the error, or not.
As per Redis documentation:
Once the client enters the subscribed state it is not supposed to
issue any other commands, except for additional SUBSCRIBE, PSUBSCRIBE,
UNSUBSCRIBE and PUNSUBSCRIBE commands.
Source : http://redis.io/commands/subscribe

RabbitMQ dead letter exchange never getting messages

I'm trying to setup my first RabbitMQ dead letter exchange, here are the steps I'm using through the web admin interface:
Create new DIRECT exchange with the name "dead.letter.test"
Create new queue "dead.letter.queue"
Bind "dead.letter.queue" to "dead.letter.test"
Create new queue "test1" with the dead letter exchange set to "dead.letter.test"
Send a message into "test1"
Nack (with requeue = false) the message in "test1"
I am expecting that these steps should put a record into the "dead.letter.queue" through the "dead.letter.test" exchange. This is not happening.
I can manually put a message into the "dead.letter.test" exchange and it shows up in "dead.letter.queue" so I know that is fine.
When I look at the admin UI it shows that the DLX parameter is setup on the queue "test1".
Where am I going wrong?
Gentilissimo Signore was kind enough to answer my question on Twitter. The problem is that if your dead letter exchange is setup as DIRECT you must specify a dead letter routing key. If you just want all your NACKed message to go into a dead letter bucket for later investigation (as I do) then your dead letter exchange should be setup as a FANOUT.
Here are the updated steps that work:
Create new FANOUT exchange with the name "dead.letter.test"
Create new queue "dead.letter.queue"
Bind "dead.letter.queue" to "dead.letter.test"
Create new queue "test1" with the dead letter exchange set to "dead.letter.test"
Send a message into "test1"
Nack (with requeue = false) the message in "test1"
Dead Letter Exchange without routing key and with direct exchange
Follow the steps these will work for sure:-
1. Create a new queue named 'dead_queue'.
2. Create an exchange named 'dead_exchange' and type of exchange should be 'direct'.
3. Bind 'dead_queue' and 'dead_exchange' without routing key.
4. Create a new queue named 'test_queue' and set its 'x-dead-letter-exchange' name as 'dead_exchange'
5. Create an exchange named 'test_exchange' and type of exchange should be 'direct'
6. Bind 'test_exchange' and 'test_queue' without routing key.
And at last we will check it. For this publish something on 'test_exchange' with argument 'expiration' set to 10000. After this when a message is publish on 'test_exchange' it will go to 'test_queue' and when a message is expired with in a queue it will look for DLX Parameter(Dead Letter Exchange name) there that message find the name 'dead_exchange' then that message will reach 'dead_exchange' deliver it to 'dead queue' ..
If still you have any problem regarding this and if i miss understood your problem... write your problem i will surely look over it... Thanks..
Note: Must publish the message on 'test_exchange' because that test_queue and test_exchange binding is without routing key and it will work fine but If you publish message on 'test_queue' default exchange and routing key will be used.Then after expiration of message queue tries to deliver that dead message to dead_exchange with some default routing key and message will not go to that queue.
If you want to use custom routing key on dead letter exchange you have to set x-dead-letter-routing-key when declaring working queue (in your case it is test1), otherwise default routing key will be used. In your case RabbitMQ broker detects cycling and simply drop rejected messages.
What you need is to have x-dead-letter-exchange=dead.letter.test and x-dead-letter-routing-key=dead.letter.queue arguments set on test1 queue.
If you want all your queues to have same dead letter exchange it is easier to set a general policy:
sudo rabbitmqctl -p /my/vhost/path set_policy DLX ".*" '{"dead-letter-exchange":"MyExchange.DEAD"}' --apply-to queues
Don't need to create FANOUT exchange if it is not compulsory.
You can create DIRECT exchange using the same routing key which you have used already for other exchange. And also don't need to create a new queue for the new exchange. You can use existing queues with new exchange. You just need to bind that new exchange with the queue.
Here is my receive.js file:
var amqp = require("amqplib/callback_api");
var crontab = require('node-crontab');
amqp.connect("amqp://localhost", function (err, conn) {
conn.createChannel(function (err, ch) {
var ex = 'direct_logs';
var ex2 = 'dead-letter-test';
var severity = 'enterprise-1-key';
//assert "direct" exchange
ch.assertExchange(ex, 'direct', { durable: true });
//assert "dead-letter-test" exchange
ch.assertExchange(ex2, 'direct', { durable: true });
//if acknowledgement is nack() then message will be stored in second exchange i.e. ex2="dead-letter-test"
ch.assertQueue('enterprise-11', { exclusive: false, deadLetterExchange: ex2 }, function (err, q) {
var n = 0;
console.log(' [*] Waiting for logs. To exit press CTRL+C');
console.log(q);
//Binding queue with "direct_logs" exchange
ch.bindQueue(q.queue, ex, severity);
//Binding the same queue with "dead-letter-test"
ch.bindQueue(q.queue, ex2, severity);
ch.consume(q.queue, function (msg) {
// consume messages via "dead-letter-exchange" exchange at every second.
if (msg.fields.exchange === ex2) {
crontab.scheduleJob("* * * * * *", function () {
console.log("Received by latest exchange %s", msg.fields.routingKey, msg.content.toString());
});
} else {
console.log("Received %s", msg.fields.routingKey, msg.content.toString());
}
if (n < 1) {
// this will executes first time only. Here I'm sending nack() so message will be stored in "deadLetterExchange"
ch.nack(msg, false, false);
n += 1;
} else {
ch.ack(msg)
n = 0
}
}, { noAck: false });
});
});
});
Create new DIRECT exchange with the name "dead.letter.test"
Correct
Create new queue "dead.letter.queue"
Correct
Bind "dead.letter.queue" to "dead.letter.test"
Correct
Create new queue "test1" with the dead letter exchange set to "dead.letter.test"
I am assuming you are creating test1 queue and binding it to dead.letter.test exchange
Send a message into "test1"
If you want your message to be received by dead.letter.queue you will have to provide routing key while sending message and clients consuming dead.letter.queue should also use same routing key
If you are publishing without routing key then only clients who are subscribed to test1 will receive the message.
If you publish message to direct.letter.test exchange then all the queue will receive the message. It will work like a fanout exchange
So, if you want dead.letter.queue to receive message you will have to publish message in that queue or you will have to use same routing key while publishing and subscribing and publish message to exchange
In my case the problem was because the queue had
ackMode="MANUAL"
But I never set it (because a runtime exception), use Defaul ACK instead.
For those who use Spring-AMQP
In my case the problem was different. I wanted a dead-letter-exchange to be of type direct. And i set both x-dead-letter-exchange and x-dead-letter-routing-key for the queue. Plus i had spring.rabbitmq.listener.simple.default-requeue-rejected=false in the application.properties.
Seems everything fine, but while debugging i noticed that my SimpleRabbitListenerContainerFactory has defaultRequeueRejected as null. So the reason was that when you declare SimpleRabbitListenerContainerFactory in your #Configuration, you create a new "non-default" bean. The default one is created for you behind the scene out of your properties. But your SimpleRabbitListenerContainerFactory in the #Config, these properties are not read, you must read it yourself and set in java code.
It happened to me, because i just copy-pasted the config from the Spring-AMQP docs when wanted to configure the concurrency. But you should do everything in one place, either in properties, like
spring.rabbitmq.listener.simple.default-requeue-rejected=false
spring.rabbitmq.listener.simple.concurrency=5
spring.rabbitmq.listener.simple.max-concurrency=10
or completely in java, like
#Bean
public SimpleRabbitListenerContainerFactory rabbitListenerContainerFactory(ConnectionFactory connectionFactory) {
SimpleRabbitListenerContainerFactory factory = new SimpleRabbitListenerContainerFactory();
factory.setConnectionFactory(connectionFactory);
factory.setConcurrentConsumers(5);
factory.setMaxConcurrentConsumers(10);
factory.setDefaultRequeueRejected(false);
return factory;
}
These 2 above are the same.
I would expect that when i use the second (java) option is still picks up properties from the application.properties and then i customize then in java, but it doesn't work like this.
And yes, "copy-paste" is evil :)

ActiveMQ: multi-consumers connected to one queue but only one consumer recieve all the messages

I was currently using NMS to develop application based ActiveMQ(5.6).
We have several consumers(exe) trying to recieving massgaes from the same queue(not topic). While all the messages just all go to one consumer though I have make the consumer to sleep for seconds after recieving a message. By the way, we don't want the consumers recieving the same messages other consumers have recieved.
It is mentioned in the official website that we should set Prefetch Limit to decide how many messages can be streamed to a consumer at any point in time. And it can both be configured and coded.
One way I tried is to code using PrefetchPolicy class binding the ConnectionFactory class like bellow.
PrefetchPolicy poli = new PrefetchPolicy();
poli.QueuePrefetch = 0;
ConnectionFactory fac = new ConnectionFactory("activemq:tcp://Localhost:61616?jms.prefetchPolicy.queuePrefetch=1");
fac.PrefetchPolicy = poli;
using (IConnection con = fac.CreateConnection())
{
using (ISession se = con.CreateSession())
{
IDestination destination = SessionUtil.GetDestination(se, queue, DestinationType.Queue);
using (IMessageConsumer consumer = se.CreateConsumer(queue1))
{
con.Start();
while (true)
{
ITextMessage message = consumer.Receive() as ITextMessage;
Thread.Sleep(2000);
if (message != null)
{
Task.Factory.StartNew(() => extractAndSend(message.Text)); //do something
}
else
{
Console.WriteLine("No message received~");
}
}
}
}
}
But no matter what prefetch value I set the behavior of the consumers stay the same as before.
And I've tried the second way tying to get the result, namely configure the server conf file. I change the activemq.xml of the server like bellow.
" producerFlowControl="true" memoryLimit="5mb" />
" producerFlowControl="true" memoryLimit="5mb">
But though I've set the dispatchpolicy the messages still go to one consumer.
I want to know that:
Whether this behavior can be achieved by just configuring the server xml file to enable all the consumers recieve messages from one queue? If so, how to configure this and what is wrong with my configuration? If not, how can I use codes to achieve the goal?
Thanks.
Take a look at "Message Groups" feature.
I had the same problem. Only one consumer processed all messages. I found in my code I used group header during send:
request.Properties["NMSXGroupID"] = "cheese";
According to official docs:
Standard JMS header JMSXGroupID is used to define which message group
the message belongs to. The Message Group feature then ensures that
all messages for the same message group will be sent to the same JMS
consumer - while that consumer stays alive. As soon as the consumer
dies another will be chosen.
See full details at http://activemq.apache.org/message-groups.html

How can I delete/remove an ActiveMQ subscriber using NMS API

I need to remove/delete my topic subscriber. I found this http://activemq.apache.org/manage-durable-subscribers.html
However, it's not good enough for us. We want to control the timing of removing a subscriber, and no matter there are any message or not. Besides, our program is written by C#. So the best solution for us is NMS API.
Thanks.
Here are the code,
Apache.NMS.ActiveMQ.ConnectionFactory factory = new Apache.NMS.ActiveMQ.ConnectionFactory(m_brokerURI);
m_connection = factory.CreateConnection(username, password);
Apache.NMS.ActiveMQ.Connection con = (Apache.NMS.ActiveMQ.Connection)m_connection;
ISession session = m_connection.CreateSession(AcknowledgementMode.AutoAcknowledge);
try
{
session.DeleteDurableConsumer(strQueueName);
}
catch (Exception ex)
{
// log the error message
}
Update
Our scenario is quite simple.
A client built a queue and subscribed a consumer on a topic.
the client side closed the connection.
delete the consumer on the server side(as the example code in the last update)
Here is the snapshot of activemq broker via jconsole:
jconsole snapshot
We would like to remove the subscriber “7B0FD84D-6A2A-4921-967F-92B215E22751” by following method,
But always got this error "javax.jms.InvalidDestinationException : No durable subscription exists for: 7B0FD84D-6A2A-4921-967F-92B215E22751"
strSubscriberName = “7B0FD84D-6A2A-4921-967F-92B215E22751”
session.DeleteDurableConsumer(strSubscriberName);
To delete a durable subscription from the NMS API you use the DeleteDurableConsumer method defined in ISession. You must call this method from a Connection that uses the same client Id as was used when the subscription was created and you pass the name of the subscription that is to be removed. The method will fail if there is an active subscriber though so be prepared for that exception.
In the sample code you don't set a Client Id on the connection. When working with durable subscriptions you must, must, MUST always use the same client Id and subscription name. So in you same you will get this error until you set the client Id to the same value as the connection that created the subscription in the first place.

While handling an NServiceBus message, is it possible to peek at the input queue?

I have a Windows service using NServiceBus to handle incoming messages.
While processing a message, I would like to check to see if there are any other remaining messages on the queue to process.
What is the best way to approach this?
For this specific scenario I'd say that a saga could be appropriate where it is created by the first message received, opens a timeout (for let's say one minute), collects all messages during that period of time, then Bus.SendLocal's a message containing all rows, for which another handler creates the spreadsheet and uploads.
Since, NServiceBus is using MSMQ, you can use the methods from System.Messaging.
Included is a modified method, I'm currently working on, to do a kind of batch processing.
using System.Messaging;
public int PeekAtQueue()
{
const string QUEUE_NAME = "private$\\you_precious_queuname";
if (!MessageQueue.Exists(".\\" + QUEUE_NAME))
return 0;
var messageQueues = MessageQueue.GetPrivateQueuesByMachine(Environment.MachineName);
var queue = messageQueues.Single(x => x.QueueName == QUEUE_NAME);
return queue.GetAllMessages().Count();
}
Modified here itself in the editor. Hope it still compiles :)
Found a similar discussion here, by the way:
http://jopinblog.wordpress.com/2008/03/12/counting-messages-in-an-msmq-messagequeue-from-c/