I have an exchange with a vhost, user, etc from that exchange I bind different queues. The exchange and the queues are in the same vhost.
Now I want to create a different vhost for a different queue but I cannot bind this new queue from the previous exchange as It is in different vhost.
What is the best solution for that?
Thanks
Your requirement is typically done using the federation or shovel plugins. For your requirement a federated exchange seems the more appropriate.
If I quote your question by naming vhost:
The exchange and the queues are in the same vhost A. Now I want to
create a different vhost B for a different queue but I cannot bind this
new queue from the previous exchange as It is in different vhost.
You create a federated exchange on vhost B
This exchange is linked to upstream exchange on vhost A
The new queue on vhost B is bound to this federated exchange.
Related
I'm wondering... Can I create (in my code) rules to bind an exchange to a queue (possibly) in another server or virtual host. I mean, lets consider a exchange named A, that should forward messages assign to it to a queue named B that lies on another server. Is this possible? Can you provide some examples? Thank you!
The shovel plugin is ideal for this use-case and can be configured either statically or dynamically.
See the rabbitmq docs here to learn more about shovel administration.
I created a Logstash instance that has a RabbitMQ input. I specified the key, exchange and queue in the configuration of Logstash.
When looking at the RabbitMQ web interface some time later, I realized that the exchange and queue used by Logstash weren't automatically created in RabbitMQ. Is there a way to tune Logstash to create the exchanges and queues it needs in RabbitMQ, without having to create them with the web interface?
I have a design for a RabbitMQ topology, but recently learned that RabbitMQ federation ignores messages that aren't "directly published" to the upstream exchange. This is a problem, because I am using a combination of exchange-to-exchange bindings and federation, so my setup isn't working.
Essentially, our setup is to have messages flowing into one exchange on an "inbound" server, federated to an exchange on a "routing" server, which is bound to another exchange on a routing server, which is federated to an "outgoing" server (which is where clients create queues and bind them). The reasoning behind the exchange-to-exchange binding is to force the routing to happen there, instead of allowing it to happen all the way upstream as would occur without that link. For load reasons, we can't afford for the routing to happen upstream in the "inbound" servers.
Is there a way to re-publish messages in the routing server so federation picks them up, or something to that effect? Is there something other than federation I should use in this topology?
Yes, the shovel plugin allows you to do just that. It consumes from one exchange and re-publishes to another, and the exchanges can be on the same or different nodes.
Is it possible to use federations or shovels to mirror the creation of exchanges and queues on one server to another ?
All the examples I've seen of using shovels and federations use exchanges and queues that already exist on the servers. What I want to do is create an exchange on server A and have a federation or shovel re-create it on Server B then start to send messages to it.
If this cannot be done with a federation or shovel is there anyway of achieving this without using clustering, the connection between the two servers is not consistent so clustering isn't possible.
I'm running RabbitMQ on windows.
You can use the federation plug-in.
It supports the exchange exchange and the queue federation, in order to mirror the queues and exchanges you can configure a policies ( using the management console or command line),for example with this parameters:
Name: my_policy
Pattern: ^mirr\. <---- mirror exchanges and queues with prefix “mirr.”
Definition: federation-upstream-set:all
you can apply the configuration for exchanges and queues, as:
The pattern policy supports regular expression
In this way each new or old exchange or queue that starts with the prefix “mirr.” will be mirrored to the other broker.
I think this could solve your problem.
Unfortunately in this way it is not possible to do this, because the connection is a point-to-point connection. You have to link a exchange with a remote exchange and in your topology this cant be created automatically.
I had also this problem in the past. And how i resolved the problem was over a business logic side. If there was a need for a new Exchange/Queue "on the fly", my data input gateway recognized this and created on the local and on the remote exchange the new exchange and queues with the connection, before the message was sent to RabbitMQ.
For the company I work for we would like to use RabbitMQ as our main message bus. The idea we have is that every single application uses their own vhost for internal communication and that via the shovel or federation plugin we would make it possible to share certain type of the events across multiple vhosts (maybe even multiple machines (non-clustered)).
We chose for application per vhost to separate internal communication from public events and to keep the security adjustable per application.
Based on the information published on the RabbitMQ website I don't get it when I have to choose for shovels or when I have to choose for the federation plugin.
RabbitMQ has the following explanation when to use what:
Typically you would use the shovel to link brokers across the internet when you need more control than federation provides.
What is the fine grain control in shovels which I am missing when I choose for federation?
At this moment I think I would prefer the federation plugin because I could automate the inter-vhost-communication via the REST API provided by the federation plugin.
In case of shovels I would need to change the shovel configuration and reboot the RabbitMQ instance every time we would like to share an event between vhosts. Are my thoughts correct about this?
We are currently running RMQ on Windows with clients connecting from .NET. In the near future Java/Perl/PHP clients will join.
To summarize my questions:
What is the fine grain control in shovels which I am missing when I
choose for federation?
Is it correct that the only way to change the
inter-vhost-communication when I use shovels is by changing theconfig file and rebooting the instance?
Does the setup (vhost per application) make sense or am I missing the point completely?
Shovels and queue provide different means to be forward messages from one RabbitMQ node to another.
Federated Exchange
With a federated exchange, queues can be connected to the queue on the upstream(source) node. In addition, an exchange on the downstream(destination) node will receive a copy of messages that are published to the upstream node.
Federated exchanges are a similar to exchange-to-exchange bindings, in that, they can (optionally) subscribe to a limited set of messages from an upstream exchange.
Federated Queue
(NOTE: These are new in RabbitMQ 3.2.x)
With a federated queue, consumers can be connected to the queue on both the upstream(source) and downstream(destination) nodes.
In essence the downstream queue is a consumer on the upstream queue, with the expectation that there will be additional downstream consumers that process the messages in the same manner as a consumer attached to the upstream queue.
Any messages consumed by the downstream (federated) queue will not be available for consumers on the upstream queue.
Use Case:
If consumers are being migrated from one node to another, a federated queue will allow this to happen without messages being missed, or processed twice.
Use Case: from the RabbitMQ docs
The typical use would be to have the same "logical" queue distributed
over many brokers. Each broker would declare a federated queue with
all the other federated queues upstream. (The links would form a
complete bi-directional graph on n queues.)
Shovel
Shovels on the other hand, attach an "upstream" queue to a "downstream" exchange. (I place the terms in quotes because the shovel documentation does not describe the nodes with the same semantics as the federation documentation.)
The shovel consumes the messages from the queue and sends them to the exchange on the destination node. (NOTE: While not normally discussed as part of this the pattern, there is nothing stopping a consumer from connecting to the queue on the origin node.)
To answer the specific questions:
What is the fine grain control in shovels which I am missing when I
choose for federation?
A shovel does not have to reside on an "upstream" or "downstream" node. It can be configured and operate from an independent node.
A shovel can create all of the elements of the linkage by itself: the source queue, the bindings of the queue, and the destination exchange. Thus, it is non-invasive to either the source or destination node.
Is it correct that the only way to change the
inter-vhost-communication when I use shovels is by changing theconfig
file and rebooting the instance?
This has generally been the accepted downside of the shovel.
With the following command (caveat: only tested on RabbitMQ 3.1.x, and with a very specific rabbitmq.config file that only contain ) you can reload a shovel configuration from the specified file. (in this case /etc/rabbitmq/rabbitmq.config)
rabbitmqctl eval 'application:stop(rabbitmq_shovel), {ok, [[{rabbit, _}|[{rabbitmq_shovel, [{shovels, Shovels}] }]]]} = file:consult("/etc/rabbitmq/rabbitmq.config"), application:set_env(rabbitmq_shovel, shovels, Shovels), application:start(rabbitmq_shovel).'
.
Does the setup (vhost per application) make sense or am I missing the
point completely?
This decision is going to depend on your use case. vhosts primarily provide logical (and access) separation between queues/exchanges and authorized users.
Shovel acts like a well-designed built-in consumer. It can consume messages from a source broker and queue, and publish them into a destination broker and exchange. You could write an application to do that, but shovel already got it right - if all you need is to move messages from a queue to an exchange in the same or another broker, shovel can do it for you. Just as a well-behaving app, it can declare exchanges/queues/bindings, reconnect, change the routing key etc. You can set it up on the source or on the destination broker, or even use a third broker. It's basically an AMQP client.
Federation, on the other hand is used to connect your broker to one or multiple upstream brokers, or you can even create chains of brokers, bending the topology any way you like. You can federate exchanges or queues, and e.g. distribute messages to multiple brokers without the need to bind additional queues to a topic exchange or using a fanout exchange, and shoveling messages from each queue to a downstream broker.
To recap, federation operates at a higher level, while shovel is mostly "just" a well-written client.
To reconfigure shovel, you have to restart the broker, unfortunately.
I don't think you really need a per app vhost. You can add a per-app user to the broker without separate vhosts. Not sure what you mean on "share an event between vhosts", though.