Does anyone know what is the easiest way to check if a consumer is active for a particular queue? I could hit the api on localhost manually, but I'd love a way to do this with Pika.
For example, I have a queue xyz I want to know how many consumers are listening to that queue.
You can do a queue.declare with passive=true for the same queue you want to get the consumer count from. This will return a response (queue.declare-ok) which includes the consumer count.
From the protocol reference documentation:
bit passive
If set, the server will reply with Declare-Ok if the queue already
exists with the same name, and raise an error if not. The client can
use this to check whether a queue exists without modifying the server
state.
Related
I'm trying to setup RabbitMQ in a model where there is only one producer and one consumer, and where messages sent by the producer are delivered to the consumer only if the consumer is connected, but dropped if the consumer is not present.
Basically I want the queue to drop all the messages it receives when no consumer is connected to it.
An additional constraint is that the queue must be declared on the RabbitMQ server side, and must not be explicitly created by the consumer or the producer.
Is that possible?
I've looked at a few things, but I can't seem to make it work:
durable vs non-durable does not work, because it is only useful when the broker restarts. I need the same effect but on a connection.
setting auto_delete to true on the queue means that my client can never connect to this queue again.
x-message-ttl and max-length make it possible to lose message even when there is a consumer connected.
I've looked at topic exchanges, but as far as I can tell, these only affect the routing of messages between the exchange and the queue based on the message content, and can't take into account whether or not a queue has connected consumers.
The effect that I'm looking for would be something like auto_delete on disconnect, and auto_create on connect. Is there a mechanism in rabbitmq that lets me do that?
After a bit more research, I discovered that one of the assumptions in my question regarding x-message-ttl was wrong. I overlooked a single sentence from the RabbitMQ documentation:
Setting the TTL to 0 causes messages to be expired upon reaching a queue unless they can be delivered to a consumer immediately
https://www.rabbitmq.com/ttl.html
It turns out that the simplest solution is to set x-message-ttl to 0 on my queue.
You can not doing it directly, but there is a mechanism not dificult to implement.
You have to enable the Event Exchange Plugin. This is a exchange at which your server app can connect and will receive internal events of RabbitMQ. You would be interested in the consumer.created and consumer.deleted events.
When these events are received you can trigger an action (create or delete the queue you need). More information here: https://www.rabbitmq.com/event-exchange.html
Hope this helps.
If your consumer is allowed to dynamically bind / unbind a queue during start/stop on the broker it should be possible by that way (e.g. queue is pre setup and the consumer binds the queue during startup to an exchange it wants to receive messages from)
I'm trying to use RabbitMQ in a more unconventional way (though at this point i can pick any other message queue implementation if needed)
I have one queue (I can have more if needed) that where customers are fetching N messages asynchronous. After they do their work I send the results from the client to the db.
I have two problems: first I don't want that they will work on the same message, second I want to grantee that I wont lose messages in case that my customer will close the browser or just stop working.
I looked at the documentation and saw the TTL which was perfect for me if I could alter that message that got timeout isn't going to be deleted but to move to another queue. can't find a way to alter this.
Moreover I looked at the confirmation option which in the first glance looked what I wanted,that mechanism is working like this: when the consumer gets a message he send confirmation to queue, I thought I can delay this confirm and send it when the work is done on the client side.
my problem was that I can't program the queue that if any message didn't get confirm then return it to the queue (or to another).
I also find how to do a scheduled message but it didn't help either because I don't want that the message will be inserted to the queue in five min,I want that when a customer will receive a message it will be locked in the queue for 5 min until confirm to delete is set otherwise return it to the queue.
Can I do temporary queue that enables my mechanism?
If someone can help with one of the problems or suggest another architecture or option to do it in another MQ it would be great.
Resources:
confirmation:
http://www.rabbitmq.com/blog/2011/02/10/introducing-publisher-confirms/
post about locks but his problem was a batcher component:
Locks and batch fetch messages with RabbitMq
TTL:
https://www.rabbitmq.com/ttl.html
Schedule a message:
https://www.rabbitmq.com/blog/2015/04/16/scheduling-messages-with-rabbitmq/
my problem was that I can't program the queue that if any message
didnt get confirm then return it to the queue (or to another).
RabbitMQ does this anyhow, so all you have to do is switch off the auto-ack flag, you figured this out
I thought I can delay this confirm and send it when the work is done
on the client side.
so just send the ACK once you've finished with processing the message.
All the unacknowledged messages remain in the queue and are re-delivered to next consumer (or the same one when it's up again, depending on your setup)
I have a clients that uses API. The API sends messeges to rabbitmq. Rabbitmq to workers.
I ought to reply to clients if somethings went wrong - message wasn't routed to a certain queue and wasn't obtained for performing at this time ( full confirmation )
A task who is started after 5-10 seconds does not make sense.
Appropriately, I must use mandatory and immediate flags.
I can't increase counts of workers, I can't run workers on another servers. It's a demand.
So, as I could find the immediate flag hadn't been supporting since rabbitmq v.3.0x
The developers of rabbitmq suggests to use TTL=0 for a queue instead but then I will not be able to check status of message.
Whether any opportunity to change that behavior? Please, share your experience how you solved problems like this.
Thank you.
I'm not sure, but after reading your original question in Russian, it might be that using both publisher and consumer confirms may be what you want. See last three paragraphs in this answer.
As you want to get message result for published message from your worker, it looks like RPC pattern is what you want. See RabbitMQ RPC tuttorial. Pick a programming language section there you most comfortable with, overall concept is the same. You may also find Direct reply-to useful.
It's not the same as immediate flag functionality, but in case all your publishers operate with immediate scenario, it might be that AMQP protocol is not the best choice for such kind of task. Immediate mean "deliver this message right now or burn in hell" and it might be a situation when you publish more than you can process. In such cases RPC + response timeout may be a good choice on application side (e.g. socket timeout). But it doesn't work well for non-idempotent RPC calls while message still be processed, so you may want to use per-queue or per-message TTL (or set queue length limit). In case message will be dead-lettered, you may get it there (in case you need that for some reason).
TL;DR
As to "something" can go wrong, it can go so on different levels which we for simplicity define as:
before RabbitMQ, like sending application failure and network problems;
inside RabbitMQ, say, missed destination queue, message timeout, queue length limit, some hard and unexpected internal error;
after RabbitMQ, in most cases - messages processing application error or some third-party services like data persistence or caching layer outage.
Some errors like network outage or hardware error are a bit epic and are not a subject of this q/a.
Typical scenario for guaranteed message delivery is to use publisher confirms or transactions (which are slower). After you got a confirm it mean that RabbitMQ got your message and if it has route - placed in a queue. If not it is dropped OR if mandatory flag set returned with basic.return method.
For consumers it's similar - after basic.consumer/basic.get, client ack'ed message it considered received and removed from queue.
So when you use confirms on both ends, you are protected from message loss (we'll not run into a situation that there might be some bug in RabbitMQ itself).
Bogdan, thank you for your reply.
Seems, I expressed my thought enough clearly.
Scheme may looks like this. Each component of system must do what it must do :)
The an idea is make every component more simple.
How to task is performed.
Clients goes to HTTP-API with requests and must obtain a respones like this:
Positive - it have put to queue
Negative - response with error and a reason
When I was talking about confirmation I meant that I must to know that a message is delivered ( there are no free workers - rabbitmq can remove a message ), a client must be notified.
A sent message couldn't be delivered to certain queue, a client must be notified.
How to a message is handled.
Messages is sent for performing.
Status of perfoming is written into HeartBeat
Status.
Clients obtain status from HeartBeat by itself and then decide that
it's have to do.
I'm not sure, that RPC may be useful for us i.e. RPC means that clients must to wait response from server. Tasks may works a long time. Excess bound between clients and servers, additional logic on client-side.
Limited size of queue maybe not useful too.
Possible situation when a size of queue maybe greater than counts of workers. ( problem in configuration or defined settings ).
Then an idea with 5-10 seconds doesn't make sense.
TTL doesn't usefull because of:
Setting the TTL to 0 causes messages to be expired upon reaching a
queue unless they can be delivered to a consumer immediately. Thus
this provides an alternative to basic.publish's immediate flag, which
the RabbitMQ server does not support. Unlike that flag, no
basic.returns are issued, and if a dead letter exchange is set then
messages will be dead-lettered.
direct reply-to :
The RPC server will then see a reply-to property with a generated
name. It should publish to the default exchange ("") with the routing
key set to this value (i.e. just as if it were sending to a reply
queue as usual). The message will then be sent straight to the client
consumer.
Then I will not be able to route messages.
So, I'm sorry. I may flounder in terms i.e. I'm new in AMQP and rabbitmq.
I've configured a rabbitmq fanout exchange called "ex_foo" for a RPC workload. When clients connect to the server, they create their own non-durable RPC receive queue and connect to it with a BasicConsumer. The apps listen for messages/commands and respond to the queue defined in the reply_to part of the request.
One of the simple messages/commands I'm sending out the the fanout exchange (and thus, every application/client connected to it) is a type of ping request message, and my problem is that I don't know how many ping responses I will get (or should expect), because I don't know how many clients are connected to the fanout exchange at any one time. All clients connected to the fanout exchange should reply.
If gets delivered to 10 queues on the fanout exchange (ie: 10 clients are connected), how do I know how many responses to expect? In order to know that, would I have to know how many times it was delivered? Is there anything more sophisticated and a sleep timer? Simply, my admin tool can't just wait indefinitely and needs to quit after it has recveived all pings (or a time-out has elapsed).
What you are looking for is something like a Scatter-Gather (http://www.eaipatterns.com/BroadcastAggregate.html) pattern, isn’t it?
You don’t know the consumers bound to the fan-out, so you can:
implement an keep-alive from the consumer(s) using for example an queue where the producer is bound.
Each consumer sends a keep-alive each one second, if you don’t receive a message you can considerer the consumer off-line.
Use an in-memory database where the consumer are registered (always with a keep-alive).
Use the HTTP API to know the consumers list bound to the fan-out, in this way:
http://rabbitmqip/vhost/yourfanout/bindings/source and the result is like this:
[{"source":"yourfanout","vhost":"/","destination":"amq.gen-xOpYc8m10Qy1s4KCNFCgFw","destination_type":"queue","routing_key":"","arguments":{},"properties_key":"~"},{"source":" yourfanout","vhost":"/","destination":"myqueue","destination_type":"queue","routing_key":"","arguments":{},"properties_key":"~"}]
Once count the consumers you know the replies count.
Call the API before send a request.
NOTE the last-one can works only if you use a temporary queue bound to the consumers.
I found this resource that could help you (http://geekswithblogs.net/michaelstephenson/archive/2012/08/06/150373.aspx)
I don't know exactly your final scope, but with a keep-alive you can wait max one second before decide if the consumer is alive.
Is is possible to configure a RabbitMQ exchange or a queue in such a way that at most one message with a given routing key is pending at any time? If new message arrives, the old one would be dropped and the new one enqueued.
If such option is not available, what would be the best way to implement this at the application level? I.e. when application receives a message how can it check if there any more pending messages?
You need to install Last Value Cache and enable it. Your exchange will be type "x-lvc", which inherits from the direct exchange type.
each time you connect to MQ, create a queue and bind to this exchange. It will deliver the most recent message to the queue. It is perfect for making sure you get only the most uptodate message. All other messages sent to this exchange are discarded unless there is a queue connected. So once connected you will continue to receive updates.
here are installation instructions:
https://github.com/simonmacmullen/rabbitmq-lvc-plugin
here is a similar question:
RabbitMQ messaging - initializing consumer