It seems that I am hitting a limitation in RMQ regarding the federation. I tried different setups and configurations but I can't receive more than 4-7k messages per seconds (depend on if I configure the federation as on-publish or no-ack).
The things that I already tried:
federation with different configs (with and without prefetch, ack, etc)
multiple exchanges in RMQ1 and create the same number of federations in RMQ2 and each federation point to one different exchange in RMQ1
federation to queues
Using the Consistent Hash Exchange (https://github.com/rabbitmq/rabbitmq-consistent-hash-exchange)
shovel (same message rate as federation)
I want to know how I can transfer more messages from one RMQ to another.
This is a limitation of RMQ itself: https://github.com/rabbitmq/discussions/issues/65
Related
I'm trying to configure federation between two RabbitMQ environments using the Federation Plugin.
I followed this article. But when I look in Federation Status page under Admin tab, then I can't see any link. All I see is ... no links ....
Can anyone show me the right direction to make federation work? I have questions like, do I have to create the policy on the upstream or the downstream server? And the same for the configuration of the Federation Upstream.
I only want queue federation, no exchange federation. In other words, I only want single consumption of a message. The article I mentioned above looks perfectly fitting for this. But unfortunately I can't see any link in Federation Status ...
Any help is appreciated.
EDIT
Downstream RMQ specs
A cluster with 3 nodes
Uses SSL
version 3.7.13 Erlang 21.3
Upstream RMQ specs
Single node, not clustered
No SSL
version 3.7.5 Erlang 20.2
Federation configuration on downstream RMQ cluster
New policy:
I added a policy with a pattern exactly matching the queue name and with definition federation-upstream-set: all:
Pattern: RmqQueue
Apply to: queues
Definition: federation-upstream-set: all
Priority: 0
When I look at the Queues tab, I can see that this policy is applied to the queue.
New Federation Upstream:
I created a new Federation Upstream from downstream (cluster) to upstream (single-node). Only name and uri is filled, other fields are left empty.
General parameters
URI amqp://<username>:<password>#hostnamesinglenode
Prefetch Count ?
Reconnect Delay
Ack Mode on-confirm
Trust User-ID ○
Federated exchange parameters
Exchange ?
Max Hops ?
Expires
Message TTL
HA Policy ?
Federated queue parameters
Queue ?
Upstream servers are the servers towards messages are published initially. Downstream servers are where the messages get forwarded to, so messages should be federated from the upstream server to downstream server. All configuration settings that you need to do, should be done on the "new" server, the server to where you want you messages to be moved (the downstream server).
Here is a link with more images.
Please note that you can move the publisher and/or the consumer in any order, after you have configured the federation. The federated queue will ONLY retrieve messages when it has run out of messages locally, when it has consumers that need messages, or when the upstream queue has "spare" messages that are not being consumed.
I have a design for a RabbitMQ topology, but recently learned that RabbitMQ federation ignores messages that aren't "directly published" to the upstream exchange. This is a problem, because I am using a combination of exchange-to-exchange bindings and federation, so my setup isn't working.
Essentially, our setup is to have messages flowing into one exchange on an "inbound" server, federated to an exchange on a "routing" server, which is bound to another exchange on a routing server, which is federated to an "outgoing" server (which is where clients create queues and bind them). The reasoning behind the exchange-to-exchange binding is to force the routing to happen there, instead of allowing it to happen all the way upstream as would occur without that link. For load reasons, we can't afford for the routing to happen upstream in the "inbound" servers.
Is there a way to re-publish messages in the routing server so federation picks them up, or something to that effect? Is there something other than federation I should use in this topology?
Yes, the shovel plugin allows you to do just that. It consumes from one exchange and re-publishes to another, and the exchanges can be on the same or different nodes.
I would like to use MassTransmit similar to NServiceBus, every publisher and subscriber has a local queue. However I want to use RabbitMQ.
So do all my desktop clients have to have RabbitMQ installed, I think so, then should I just connect the 50 desktop clients and 2 servers into a cluster?
I know the two servers must be in the same cluster. However 50 client nodes, seems a bi tmuch to put in one cluster.....Or should I shovel them or Federate them to the server cluster exchange?
The desktop machine send messages like: LockOrder, UnLock Order.
The Servers are dealing with backend hl7 messages.
Any help and advice here is much appreciated, this is all on windows machines.
Basically I am leaving NServiceBus behind, as it is now too expensive, they aiming it at large corporations with big budgets, hence Masstransmit.
However I want reliable/durable messaging, hence local queues on ALL publishers and ALL subscribers.
The desktops also use CQS to update their views.
should I just connect the 50 desktop clients and 2 servers into a cluster?
Yes, you have to connected your clients to the cluster.
However 50 client nodes, seems a bi tmuch to put in one cluster.
No, (or it depends how big are your servers) 50 clients is a small number
Or should I shovel them or Federate them to the server cluster exchange?
The desktop machine send messages like: LockOrder, UnLock Order.
I think it's better the cluster, because federation and shovel are asynchronous, it means that your LockOrder could be not replicated in time.
However I want reliable/durable messaging, hence local queues on ALL publishers and ALL subscribers
Withe RMQ you can create a persistent queue and messages, and it is not necessary if the client(s) is connected. It will get the messages when it will connect to the broker.
I hope it helps.
I have a FOSS ESB rpoject called Shuttle, if you would like to give it a spin: https://github.com/Shuttle/shuttle-esb
I haven't used NServiceBus for a while and actually started Shuttle when it went commercial. The implementation is somewhat different from NServiceBus. I don't know MassTransit at all, though. Currently process managers (sagas) have to be hand-rolled in Shuttle whereas MassTransit and NServiceBus have this incorporated. If I do get around to adding sagas I'll be adding them as a Module that can be plugged into the receiving pipeline. This way one could have various implementations and choose the flavour you like :)
Back to your issue. Shuttle has the concept of an optional outbox for queuing technologies like RabbitMQ. Shuttle does have a RabbitMQ implementation. I believe the outbox works somewhat like 'shovel' does. So the outbox would be local and sending messages would first go to the outbox. It would periodically try to send messages on to the recipients and, after a configurable number of attempts, send the message to an error queue. It can then be returned to the outbox for further attempts, or even moved directly to the recipient queue once it is up.
Documentation here: http://shuttle.github.io/shuttle-esb/
Is it possible to use federations or shovels to mirror the creation of exchanges and queues on one server to another ?
All the examples I've seen of using shovels and federations use exchanges and queues that already exist on the servers. What I want to do is create an exchange on server A and have a federation or shovel re-create it on Server B then start to send messages to it.
If this cannot be done with a federation or shovel is there anyway of achieving this without using clustering, the connection between the two servers is not consistent so clustering isn't possible.
I'm running RabbitMQ on windows.
You can use the federation plug-in.
It supports the exchange exchange and the queue federation, in order to mirror the queues and exchanges you can configure a policies ( using the management console or command line),for example with this parameters:
Name: my_policy
Pattern: ^mirr\. <---- mirror exchanges and queues with prefix “mirr.”
Definition: federation-upstream-set:all
you can apply the configuration for exchanges and queues, as:
The pattern policy supports regular expression
In this way each new or old exchange or queue that starts with the prefix “mirr.” will be mirrored to the other broker.
I think this could solve your problem.
Unfortunately in this way it is not possible to do this, because the connection is a point-to-point connection. You have to link a exchange with a remote exchange and in your topology this cant be created automatically.
I had also this problem in the past. And how i resolved the problem was over a business logic side. If there was a need for a new Exchange/Queue "on the fly", my data input gateway recognized this and created on the local and on the remote exchange the new exchange and queues with the connection, before the message was sent to RabbitMQ.
For the company I work for we would like to use RabbitMQ as our main message bus. The idea we have is that every single application uses their own vhost for internal communication and that via the shovel or federation plugin we would make it possible to share certain type of the events across multiple vhosts (maybe even multiple machines (non-clustered)).
We chose for application per vhost to separate internal communication from public events and to keep the security adjustable per application.
Based on the information published on the RabbitMQ website I don't get it when I have to choose for shovels or when I have to choose for the federation plugin.
RabbitMQ has the following explanation when to use what:
Typically you would use the shovel to link brokers across the internet when you need more control than federation provides.
What is the fine grain control in shovels which I am missing when I choose for federation?
At this moment I think I would prefer the federation plugin because I could automate the inter-vhost-communication via the REST API provided by the federation plugin.
In case of shovels I would need to change the shovel configuration and reboot the RabbitMQ instance every time we would like to share an event between vhosts. Are my thoughts correct about this?
We are currently running RMQ on Windows with clients connecting from .NET. In the near future Java/Perl/PHP clients will join.
To summarize my questions:
What is the fine grain control in shovels which I am missing when I
choose for federation?
Is it correct that the only way to change the
inter-vhost-communication when I use shovels is by changing theconfig file and rebooting the instance?
Does the setup (vhost per application) make sense or am I missing the point completely?
Shovels and queue provide different means to be forward messages from one RabbitMQ node to another.
Federated Exchange
With a federated exchange, queues can be connected to the queue on the upstream(source) node. In addition, an exchange on the downstream(destination) node will receive a copy of messages that are published to the upstream node.
Federated exchanges are a similar to exchange-to-exchange bindings, in that, they can (optionally) subscribe to a limited set of messages from an upstream exchange.
Federated Queue
(NOTE: These are new in RabbitMQ 3.2.x)
With a federated queue, consumers can be connected to the queue on both the upstream(source) and downstream(destination) nodes.
In essence the downstream queue is a consumer on the upstream queue, with the expectation that there will be additional downstream consumers that process the messages in the same manner as a consumer attached to the upstream queue.
Any messages consumed by the downstream (federated) queue will not be available for consumers on the upstream queue.
Use Case:
If consumers are being migrated from one node to another, a federated queue will allow this to happen without messages being missed, or processed twice.
Use Case: from the RabbitMQ docs
The typical use would be to have the same "logical" queue distributed
over many brokers. Each broker would declare a federated queue with
all the other federated queues upstream. (The links would form a
complete bi-directional graph on n queues.)
Shovel
Shovels on the other hand, attach an "upstream" queue to a "downstream" exchange. (I place the terms in quotes because the shovel documentation does not describe the nodes with the same semantics as the federation documentation.)
The shovel consumes the messages from the queue and sends them to the exchange on the destination node. (NOTE: While not normally discussed as part of this the pattern, there is nothing stopping a consumer from connecting to the queue on the origin node.)
To answer the specific questions:
What is the fine grain control in shovels which I am missing when I
choose for federation?
A shovel does not have to reside on an "upstream" or "downstream" node. It can be configured and operate from an independent node.
A shovel can create all of the elements of the linkage by itself: the source queue, the bindings of the queue, and the destination exchange. Thus, it is non-invasive to either the source or destination node.
Is it correct that the only way to change the
inter-vhost-communication when I use shovels is by changing theconfig
file and rebooting the instance?
This has generally been the accepted downside of the shovel.
With the following command (caveat: only tested on RabbitMQ 3.1.x, and with a very specific rabbitmq.config file that only contain ) you can reload a shovel configuration from the specified file. (in this case /etc/rabbitmq/rabbitmq.config)
rabbitmqctl eval 'application:stop(rabbitmq_shovel), {ok, [[{rabbit, _}|[{rabbitmq_shovel, [{shovels, Shovels}] }]]]} = file:consult("/etc/rabbitmq/rabbitmq.config"), application:set_env(rabbitmq_shovel, shovels, Shovels), application:start(rabbitmq_shovel).'
.
Does the setup (vhost per application) make sense or am I missing the
point completely?
This decision is going to depend on your use case. vhosts primarily provide logical (and access) separation between queues/exchanges and authorized users.
Shovel acts like a well-designed built-in consumer. It can consume messages from a source broker and queue, and publish them into a destination broker and exchange. You could write an application to do that, but shovel already got it right - if all you need is to move messages from a queue to an exchange in the same or another broker, shovel can do it for you. Just as a well-behaving app, it can declare exchanges/queues/bindings, reconnect, change the routing key etc. You can set it up on the source or on the destination broker, or even use a third broker. It's basically an AMQP client.
Federation, on the other hand is used to connect your broker to one or multiple upstream brokers, or you can even create chains of brokers, bending the topology any way you like. You can federate exchanges or queues, and e.g. distribute messages to multiple brokers without the need to bind additional queues to a topic exchange or using a fanout exchange, and shoveling messages from each queue to a downstream broker.
To recap, federation operates at a higher level, while shovel is mostly "just" a well-written client.
To reconfigure shovel, you have to restart the broker, unfortunately.
I don't think you really need a per app vhost. You can add a per-app user to the broker without separate vhosts. Not sure what you mean on "share an event between vhosts", though.