Is there a way to get the value of the "Consumer utilisation" (as seen in the rabbitmq overview in the queues) via the RabbitMQ client? Can I ask for and react on this value via API?
Regards.
I found a solution in the meantime. And I think the question is important, as it is needed for bulk and monitoring purpose. But as the question was even downvoted, I will not post an answer here. It seems it's only important to me.
Following will return the metric for all consumers:
rabbitmqctl list_queues name messages messages_ready state consumer_utilisation --formatter json
Or via REST api for a given queue:
curl http://guest:guest#127.0.0.1:15672/api/queues/%2F/my-queue
Which gives out consumer_utilisation as one of the returned properties. Note it returns null if the value is too small. For an accurate read, use rabbitmqctl.
Related
I am looking to replace an in-house key-value store and dispatch system and I keep hearing that RabbitMQ may be a solution.
I understand that sends and receives messages using queues, and that these events are triggered by producers creating messages, and consumers receiving them.
But what happens if a consumer is created after a message was sent? Can the consumer ask the queue what its last message was? If not, do I need to include some sort of database to store these messages? Or am I looking for some other technology?
A use case is that I want a GUI to get/set parameters that are used by other apps on a local network. On initialization, the GUI needs to know what the last values were.
In an attempt to answer my own question, it may be that RabbitMQ is not what I am looking for. I may want to instead use Kafka which stores its latest key:value pair in a table. Or I may want to use Redis. What do you think?
Thank you for your assistance.
I think I found a satisfactory answer to my question. I'm looking to create a request-reply model, which RabbitMQ is quite capable of handling. Upon opening the GUI, it sends a request to some other process for some variable, stored either in memory or in a database. That process responds with the requested data. Easy enough.
When my web app subscribes to a Redis channel (mostly on Application_Start), it should automatically load the current channel content, but not wait for the next publish within this channel.
I couldn't find any way to achieve this - but as this "problem" appears to be so common and trivial I guess there must be an easy solution for this?
In the web app I'm using StackExchange.Redis (in case that's relevant). Who can help? Thx in advance!
The answer is no, there is no option to do this using Redis pub/sub functionality, Redis don't actually store messages which being published to the channel, so you can't retrieve them when you connect to channel.
Take a look at RabbitMQ with their persistent queues and message acknowledgements, which they have out of the box.
As there's obviously no comfortable option available in Redis, I'm now publishing the channel message also as a regular key-value. So the clients take it from key-value store before subscribing to the channel.
I was just reading about Enterprise Service Bus and trying to figure out how to implement it. However, the more I read about it, my conclusion was that it is just a glorified message queue.
I read about it here: What is an ESB and what is it good for?
We use RabbitMQ in our architecture quite a lot and what I was having hard time understanding was that there any many similarities between both concepts:
Both are basically post and forget
You can post a message of any format in both queues
My question is that what is it that an ESB does and RabbitMQ is not able to do?
I have not used RabbitMQ so I wont be able to comment on it. I have used ESB and currently using it.
ESB: It provides you multiple ways of subscribing to your message.Its mostly useful in Publisher-Subscriber model in which topics and subscription is used. You can publish your message payload in topics(similar to queues). Unlike a queue,topic provides us with capability to have more than one subscription for a single topic. This subscription can be divided based on your business need and you can define some kind of filter expression on those topic (also called channel)and with the specified filter a proper subscriber will pull the message from bus. Also one single message can be subscribed by multiple subscriber at a time. If no filtering is used on topics then it means all subscriber for that topic will pull the message from the channel.
This is asynchronous mechanism as you mentioned, post and forget. There is a retry mechanism in ESB where you can try subscribing the message for some number of times I think its 10 times(max) after which its sent in dead queue.
So if your requirement is to connect multiple enterprise system with loosely couple architecture then ESB is a good option.
I hope this was helpful to know about ESB
Can anyone give me examples of how in production a correlation id can be used?
I have read it is used in request/response type messages but I don't understand where I would use it?
One example (which maybe wrong) I can think off is in a publish subscribe scenario where I could have 5 subscribers and if I get 5 replies with the same correlation id then I could say all my subscribers have received it. Not sure if this would the be correct usage of it.
Or if I send a simple message, the I can use the correlation to guarantee that the client received it.
Any other examples?
A web application that is providing HTTP API for outsiders for performing a processing task and you want to give the results for the caller as a response to the HTTP request they made.
A request comes in, message describing the task is pushed to queue by the frontend server. After that the frontend server blocks to wait for response message with the same correlation id. A pool of worker machines are listening on queue and one of them picks up the task, performs it and returns the result as message. Once a message with right correlation id comes in, frontend server continues to return the response to the caller.
In the context of CQRS and EventSourcing a command message correlation id will most likely get stored togehter with the corresponding events from the domain. This information can later be used to form an audit trail.
Streaming engines like Apache Flink use correlation ids, much like you said, to guarantee exactness of processing.
I am currently interested in seeing what channels are subscribed to in a Redis pub/sub application I have. When a client connects to our server, we register them to a channel that looks like:
user:user_id
The reason for this is I want to be able to see who's "online". I currently blindly fire off messages to a channel without knowing if a client is online since it's not critical that they receive these types of messages.
In an effort to make my application smarter, I'd like to be able to discover if a client is online or not using the pub/sub API, and if they are offline, cache their messages to a separate redis queue which I can push to them when they get back online.
This does not have to be 100% accurate, but the more accurate it is, the better. I'm assuming a generic key does not get created when a channel gets subscribed to, so I cannot do something as trivial as:
redis-cli keys user* to find all online users.
The other strategy I've thought of is just maintaining my own Redis Set whenever a user published or removes themselves from a channel (which the client automatically handles when they hop online and close the app). That would be an additional layer of complexity that I need to manage and I'm hoping there is a more trivial approach with the data that's already available.
As of Redis 2.8 you can do:
PUBSUB CHANNELS [pattern]
The PUBSUB CHANNELS command has O(N) complexity, where N is the number of active channels.
So in your case:
redis-cli PUBSUB CHANNELS user*
would give you want you want.
There is currently no command for showing what channels "exist" by way of being subscribed to, but there is and "approved" issue and a pull request that implements this.
https://github.com/antirez/redis/issues/221
https://github.com/antirez/redis/pull/412
Due to the nature of this call, it is not something that can scale, and is thus a "DEBUG" command.
There are a few other ways to solve your problem, however.
If you have reason to believe that a channel may be subscribed to, you can send it a message and look at the result. The result is the number of subscribers that got the message. If you got 0, you know that they're not there.
Assuming that your user_ids are incremental, you might be interested in using SETBIT to set a 1 or 0 to a user's offset bit to track presence. You can then do cool things like the new BITCOUNT to see how many users are online, and GETBIT to determine if a specific user is online.
The way I have solved your problem more specifically in the past is by signaling a subscription manager that I have subscribed to a channel. The manager then "pings" the channel by sending a blank message to confirm that there is a subscriber, and occasionally pings the channel thereafter to determine if the user is still online. Not ideal, but better than using DEBUG CHANNELS in production.
From version 2.8.0 redis has a pubsub command that would help in this case:
http://redis.io/commands/pubsub
Remark: currently the state of 2.8.0 is not stable yet (RC2)
I am unaware of any specific way to query what channels are being subscribed to, and you are correct that there isn't any key created when this happens. Also, I wouldn't use the KEYS command in production anyway, as it's really a debugging command.
You have the right idea about using a set to add the user when they're online, and then query this with SISMEMBER <set> <user_id> to determine if the messages should be sent to them or added to a Redis list for processing once they do come online.
You will need to figure out when a user logs off so you can remove them from the list of online users, but I don't know enough about your system to know exactly how you would go about that.
If the connected clients have the ability to send a message back to inform the server that the message(s) were consumed, you could use this to keep track of which messages should be stored for later retrieval.
Cheers,
Mike
* PUBSUB NUMSUB [channel-1 ... channel-N]
Returns the number of subscribers (not counting clients subscribed to patterns) for the specified channels.
https://redis.io/commands/pubsub