I found a nice plugin for copying messages from one broker to another and even change from queue to exchange, as I understood.
http://www.rabbitmq.com/shovel.html
Is it possible to add a filter to the shovel?
Or is there another re-routing plugin which helps me doing that?
My original idea was to add a attribute-based and envelope based filter on Rabbit-Level which is copying messages of my interest to another exchange.
The alternative is to implement the filter on consumers end, but that looks like a waste of performance, because the original queue deals with about 20000 msg / minute.
As always any help is apreciated
Thanks in advance
Related
I am looking to replace an in-house key-value store and dispatch system and I keep hearing that RabbitMQ may be a solution.
I understand that sends and receives messages using queues, and that these events are triggered by producers creating messages, and consumers receiving them.
But what happens if a consumer is created after a message was sent? Can the consumer ask the queue what its last message was? If not, do I need to include some sort of database to store these messages? Or am I looking for some other technology?
A use case is that I want a GUI to get/set parameters that are used by other apps on a local network. On initialization, the GUI needs to know what the last values were.
In an attempt to answer my own question, it may be that RabbitMQ is not what I am looking for. I may want to instead use Kafka which stores its latest key:value pair in a table. Or I may want to use Redis. What do you think?
Thank you for your assistance.
I think I found a satisfactory answer to my question. I'm looking to create a request-reply model, which RabbitMQ is quite capable of handling. Upon opening the GUI, it sends a request to some other process for some variable, stored either in memory or in a database. That process responds with the requested data. Easy enough.
Can anyone explain in which cases I need to create multiple queues (one user -> one queue name), and when one queue name for all clients with different routing keys (one user -> one routing key) and why?
A user should not be able to read messages intended for another user.
I'm using direct exchange type.
First off I am going to assume that when you say "user" you are interchangeably referring to a consumer or producer, and they aren't the same thing so I would read up on that here in rabbitmq's simplest explanation. Walking through that tutorial will definitely help solidify your understanding of rabbit a bit more overall too, which is always good.
In any case, I would recommend doing this:
Create multiple queue's, each one linked to a single consumer. The reason for doing this instead of using a single queue with multiple is discussed here but if you don't want a bunch of programmer jargon, it pretty much says that a single queue is super slow because only one message can be consumed at a time from the queue.
Also, there is a built in "default exchange" that you can use instead of setting up another direct exchange which it sounds like you're putting effort into that you might not need to, obviously I'm not sure what you are doing but I would take that into consideration... hope this helps!
We have a queue which takes incoming data and pushes the data into another queue in another machine through a shovel.
For some reason, we found that the source queue had backed up with around 2M messages. We couldn't figure out the cause for that as it seemed that the destination queue and the consumer of that queue was working fine.
We also realized that the shovel was setup with the default pre-fetch count of 1000.
We are not able to modify the shovel to set a higher pre-fetch count, the only option is to delete the shovel and setup a new one with higher pre-fetch count.
What will happen if we delete the shovel ?
Will it delete the messages in the queues ?
Thanks
Based on your latest comment, if I understand correctly, you have:
RabbitMQ1 - exchange RMQ1EXCA -> queueA1
RabbitMQ2 - exchange RMQ2EXCB -> queueB1
A shovel has been configured, exchange [RMQ1EXCA] to exchange [RMQ2EXCB]
And you found out that the queueA1 is filled up with millions of message.
If indeed this is an accurate depiction of your setup:
it's quite normal, as queueA1 is not part of the shovel process
if you check the queues bound to RMQ1EXCA, you should see one queue with a name starting with amq.gen-......
deleting the shovel will not impact the queueA1, as it's not related to the process (but will delete the queue amq.gen-...... that is)
If the description provided doesn't match your setup, please provide additional information to clarify your situation so that I can adapt my answer accordingly
I'm trying to achieve load balancing between different types of messages. I would not know in advance what the messages coming in might be until they hit the queue. I know I can try resequencing the messages, but I was thinking that maybe if there was a way to have the various consumers round robin between either queues or between topics, this would solve my problem.
The main problem i'm trying to solve is that I have many services sending messages to one queue with many consumers feeding off one queue. I do not want one type of service monopolizing the entire worker cluster. Again I don't know in advance what the messages that are going to hit the queue are going to be.
To try to clearly repeat my question:
Is there a way to tell the consumers to round robin between either existing queues or topics?
Thank you in advance.
I found the answer to my question on another post just had to know to look there. I resolved my problem by not creating AMQ consumer but a JMS listener with a composite destination as specified in this post: jms-listener-dynamically-choose-destinations. It turns out the JMS listener automatically round robins though all the queues you assign to it.
Consumers on a Queue will already do round robin processing of the messages on the Queue. The one thing to keep in mind is consumer prefetch which can allow one consumer to grab many messages before others arrive on the Queue so you may need to adjust prefetch depending on your scenario.
Read up on the differences between Queue and Topic here.
So we will have a topic exchange that looks something like
{class}.{genus}
So we have some consumers that bind with the topic
mammal.*
(or bird.*, etc.)
Now suppose later on we want to include species information so the topic exchange now looks like this:
{class}.{genus}.{species}
Now the old consumers are broken :(
However they could have bound as
mammal.*.#
And been able to listen to whatever future information is added. However, this is something my team came up with on our own which leads me to ask:
Is this good practice?
Are there tradeoffs to this I should be aware of?
Is there an alternate way to have a producer be able to add information without breaking existing consumers, without publishing to multiple exchanges?
Typically if you have a need maximum control on queue delivery and want to do the logic in rabbit, then you should consider header exchanges.
Usually when we code up the publish we know exactly which queue it needs to go to, so whether you want to use a routing key or a boolean to do this might not make much difference depending on your application.
This brings up another design consideration to be aware of: whether you want routing logic in rabbit. Someone people prefer to just use simple routing keys and either direct or topic exchanges, focusing on flexible consumers. Its going to be hard to guess at what is best for your application obviously.
Keep in mind that your consumers will be subscribed, often statically, to the queue(s) that the exchange delivers to. Also mammal.# is the same as mammal.*.# (see: ref)