Let's say I have 3 subscribers A,B,C for one topic, and I want A,B to be treated "same subscriber", meaning they can get only one copy of each message. And C got another copy.
I find that http://activemq.apache.org/virtual-destinations.html is one way. But what if I can't change the activemq broker's config?
I wonder if there is some "id" props, can make two subscribers to be treated as one? like group id in kafka?
I presume you are using JMS from your client. ActiveMQ 5.x only support JMS 1.1 which does not allow load balanced topics. JMS 2.0 does and is implemented in ActiveMQ Artemis, but that's another product.
However, you can use Virtual Topics without config changes, but you have to change your naming of topics and queues.
Publish to topic: VirtualTopic.[TopicName] and consume from queues: Consumer.[LogicalConsumerId].VirtualTopic.[TopicName].
I.e.
Publish orders to VirtualTopic.Orders
Consume from:
Consumer.ManufacturingSystem.VirtualTopic.Orders
Consumer.CRMSystem.VirtualTopic.Orders
etc.
As the consumers consume from a queue, they can use different (or no) ClientId and still have a load balancing among nodes in each system. I.e. the CRMSystem may have two nodes and will only receive one message in total.
This naming convention can be customized if you change ActiveMQ config, but works OOTB.
Related
Here's an example:
TYPE : TOPIC
exchange.v1 -> queue.order
exchange.v2 -> queue.log
so when the apps running it's must configure the exchange first right? and in a single service only can have 1 exchange?
I have 1 service for logging and 1 service for ordering. all proses will be sent into logging service and then forward another event. in this case to queue.order
So it's possible to publish an event from a different exchange? or I miss something? please let me know :(
Exchanges are not tied to “services”, much less in a 1:1 manner.
Exchanges in RabbitMQ are message sinks. Any existing exchanges can be published to by any number of applications (“services”) with adequate permissions.
Exhanges can either be pre-deployed or created automatically by an application. Pre-deployment is usually more common. This may or may not be outside the lifecycle of a single “service”.
Exchanges (depending on type) may also route to any number of queues on the same vhost.
Now, with all of that out of the way..
It is very possible to forward a message from a queue to another exchange: read from queues (stores), publish to exchanges (sinks). This can be done in code or even from a tool like the Shovel plugin - the “correct” approach depends significantly based on semantics, just as the choice of routing.
Personally, I recommend keeping RabbitMQ processing chains to as limited a scope as allowed by the application domain.
I have a RabbitMQ with multiple consumers subscribed on a single queue. And I want the messages with same hash key can be consumed by the same consumer for each time. I know the default behavior for RabbitMQ is loop through all consumers and dispatch the message 1 by 1.
Does it have the same ability like Kafka partition?
Thanks
Rebalancer (forked from Jack Vanlightly and improved)
Create Kafka style consumer groups in other technologies. Rebalancer was born of the need for consumer groups with RabbitMQ. But Rebalancer is completely technology agnostic and will balance activity over any group of resources across a group of participating nodes.
Use cases
Create Kafka-like "consumer groups" with messaging technologies like RabbitMQ, SQS, etc.
Consume a group of resources such as file shares, FTPs, S3 buckets between the instances of a scaled out application.
Single Active-Consumer / Active-Backup
Create an application cluster that consumes a single resource in a highly available manner. The cluster leader (Coordinator) consumes the single resource and the slaves (Followers) remain idle in backup in case the leader dies.
Well not exactly but a very close one .
You need to use RabbitMQ Consistent Hash Exchange Type which is available by adding the rabbitmq-consistent-hash-exchange plugin. It adds a consistent-hash exchange type to RabbitMQ. This exchange type uses consistent hashing to distribute messages between the bound queues. It is recommended to get a basic understanding of the concept before evaluating this plugin and its alternatives.
The undelying use case
It is typical pubsub use case: Consider we have M news sources, and there are N subscribers who subscribe to the desired news sources, and who want to get news updates. However, we want these updates to land up in mongodb - essentially maintain most recent 'k' updates (and can be indexed and searched etc.). We want to design for M to scale upto million publishers, N to scale to few millions.
Subscribers' updates are finally received and stored in more than one hosts and their native mongodbs.
Modeling in rabbitmq
Rabbitmq will be used to persist the mappings (who subscribes to which news source).
I have setup a pubsub system in this way: We create publisher exchanges (each mapping to one news source) and of type 'fanout'.
For modelling subscribers, there are two options.
In the first option, have one queue for each subscriber bound to relevant publisher exchanges. And let the client process open connections to all these subscriber queues and receive the updates (and persist them to mongodb). Note that in this option, when the client is restarted, it has to manage list of all susbcribers, and open connections to all subscriber queues it is responsible for.
In the second option, we want to be able to remove overhead of having to explicitly open on each user queue upon startup. Instead, we want to listen to only one queue - representative of all subscribers who will send updates to this client host.
For achieving this, we first create one exchange for each subscriber and let it bind to the publisher exchange(s) that it follows. We let a single queue for each client, and let the subscriber exchange bind to this queue (type=direct) if the subscriber belongs to that client.
Once the client receives the update message, it should come to know which subscriber exchange it came from. Only then we can add it to mongodb for relevant subscriber. Presumably the subscriber exchange should add this information as a new header on the message.
As per rabbitmq docs, I believe there is no way to get achieve this. (Or more specifically, to get the 'delivery path' property from the delivered message, from which we can get this information).
My questions:
Is it possible to add a new header to message as it passes through exchange?
If this is not possible, then can we achieve it through custom exchange and relevant plugin? Any plugin that I can readily use for this purpose?
I am curious as to why rabbitmq is not providing delivery path property as an optional configuration?
Is there any other way I can achieve the same? (See pubsubhubbub note below)
PubSubHubBub
The use case is very similar to what pubsubhubbub protocol provides for. And there is rabbitmq plugin too called rabbithub. However, our system will be a closed system, and I believe that the webhook approach of the protocol is going to be too much of overhead compared to listening on single queue (and from performance perspective.)
The producer (RMQ Client) of the message should add all the required headers (including the originator's identity) before producing (publishing) it on RMQ. These headers are used for routing.
If, while in transit, the message (including headers) needs to be transformed (e.g. adding new headers), it needs to be sent to the transformer (another RMQ Client). This transformer will essentially become the new publisher.
The actual consumer should receive its intended messages (for which it has subscribed to) through single queue. The routing of all its subscribed messages should be arranged on the RMQ Exchange.
Managing the last 'K' updates should neither be the responsibility of the producer nor the consumer. So, it should be done in the transformer. Producers' messages should be routed to this transformer (for storage) before further re-routing to exchange(s) from where consumers consume.
I have to implement this scenario:
An external application publish message to rabbitmq.
This message has a client_id property. We can place this id to routing key or message header or some other property.
I have to implement sharding in a exchange routng logic - the message should be delivered to specific queue based on the client_id range.
Is it possible to implement in a standard exchanges?
If not what exchange should I take as the base?
How to dynamicly change client_id ranges?
Take a look at the rabbitmq plugin. It's included in the RabbitMQ distribution from v3.6.0 onwards.
Just have your producer put enough info into the routing key that causes the message to go into the right queue on the other side of the Exchange.
So for example, create two queues called 1 and 2 and bind them with routing keys matching the names. Then have your producer decide which routing key to use when producing the event message. Customers with names starting with letters a-m go to 1, n-z go to 2, you get the idea. It pushes the sharding to the producer but that might be OK for your application.
AMQP doesn't have any explicit implementation of sharding, but its architecture should help you to do that.
Spreading messages to several queues is just a rabbitmq challenge (and part of amqp specification), and with routing, way you can attach hetereogeneous consumers to handle specific messages routed via the same exchange. Therefore, producer should push a specific key to be consumed by specific queue/consumer...
You can decide to make a static sharding, perhaps you have 10 queues with one consumer per queue. You could implement a consistent hashing function such that key is CLIENT_ID % 10.
Another ways and none static solutions could be propoused, and you can try to over this architecture.
Well, lets say I'm building ActiveMQ based chat application. It's pretty simple. Having only one QUEUE.IN and one TOPIC.OUT. All messages are simply routed right away from QUEUE.IN to TOPIC.OUT. Clients are producing their chat messages to QUEUE.IN and consuming from TOPIC.OUT. That's all.
Now, I wanna cluster it. Don't need something complex. Just run few other identical nodes (A..N). Basically, client, subscribed to A node, sends message to A.QUEUE.IN. This message must then appear on all other nodes (A..N).TOPIC.OUT. This could be easily done by simple camel route that re-route all messages comes to TOPIC.OUT to other nodes, but is there some nice ActiveMQ-native way to do so? Like some queue/topic shared among several AMQ instances?
I think you can find your answer here:
http://activemq.apache.org/how-do-distributed-queues-work.html
You can forward messages to multiple endpoints in activemq using virtual destinations.
http://activemq.apache.org/virtual-destinations.html