I am thinking how to remove duplicacy for subscriber cluster in pubsub, for example:
There is a service called email, which should send welcome emails after user signing up. By using pub/sub, email service shall listen a event called "signedUp" which will be triggered each time user sign up. However, how about if I have 2 email services to balance? Without any special efforts, I think two welcome emails will be sent out. So how to solve this issue?
I prefer redis as pubsub server for simplicity, or rabbitmq if redis doesn't work out.
I don't think it is possible to do it in redis. But in rabbitmq, it can, let me explain below:
Rabbitmq has a separate stuff called 'exchange' from queue. So server publish a message to exchange, client can create queues to bind to the exchange. Therefore, instances from one services can create the same queue to bind with exchange, by doing that exchange will delivery message to the queue once and handled by only one instance once.
Account service:
channel.assertExchange(‘signedUp’, 'fanout')
channel.publish(ex, '', new Buffer(message)
Email service:
let queue = channmel.assertQueue(‘email’);
channel.bindQueue(queue, 'signedUp'); // bind this queue to exchange
ch.consume(queue, logMessage)
By given a queue name in email service, no matter how many email services started, the published message (signedUp in this case) will be handled by one and ONLY ONE email service.
Related
Requirement
A system undergoes some state change, and multiple other parts of the system has to know this(lets call them observers) so that they can perform some actions based on the current state, the actions of the observers are important, if some of the observers are not online(not listening currently due to some trouble, but will be back soon), the message should not be discarded till all the observers gets the message.
Trying to accomplish this with pub/sub model, here are my findings, (please correct if this understanding is wrong) -
The publisher creates an event on specific topic, and multiple subscribers can consume the same message. This model either provides no delivery guarantee(in redis), or delivery is guaranteed once(with messaging queues), ie. when one of the consumer acknowledges a message, the message is discarded(rabbitmq).
Example
A new Person Profile entity gets created in DB
Now,
A background verification service has to know this to trigger the verification process.
Subscriptions service has to know this to add default subscriptions to the user.
Now both the tasks are important, unrelated and can run in parallel.
Now In Queue model, if subscription service is down for some reason, a BG verification process acknowledges the message, the message will be removed from the queue, or if it is fire and forget like most of pub/sub, the delivery is anyhow not guaranteed for both the services.
One more point is both the tasks are unrelated and need not be triggered one after other.
In short, my need is to make sure all the consumers gets the same message and they should be able to acknowledge them individually, the message should be evicted only after all the consumers acknowledged it either of the above approaches doesn't do this.
Anything I am missing here ? How should I approach this problem ?
This scenario is explicitly supported by RabbitMQ's model, which separates "exchanges" from "queues":
A publisher always sends a message to an "exchange", which is just a stateless routing address; it doesn't need to know what queue(s) the message should end up in
A consumer always reads messages from a "queue", which contains its own copy of messages, regardless of where they originated
Multiple consumers can subscribe to the same queue, and each message will be delivered to exactly one consumer
Crucially, an exchange can route the same message to multiple queues, and each will receive a copy of the message
The key thing to understand here is that while we talk about consumers "subscribing" to a queue, the "subscription" part of a "pub-sub" setup is actually the routing from the exchange to the queue.
So a RabbitMQ pub-sub system might look like this:
A new Person Profile entity gets created in DB
This event is published as a message to an "events" topic exchange with a routing key of "entity.profile.created"
The exchange routes copies of the message to multiple queues:
A "verification_service" queue has been bound to this exchange to receive a copy of all messages matching "entity.profile.#"
A "subscription_setup_service" queue has been bound to this exchange to receive a copy of all messages matching "entity.profile.created"
The consuming scripts don't know anything about this routing, they just know that messages will appear in the queue for events that are relevant to them:
The verification service picks up the copy of the message on the "verification_service" queue, processes, and acknowledges it
The subscription setup service picks up the copy of the message on the "subscription_setup_service" queue, processes, and acknowledges it
If there are multiple consuming scripts looking at the same queue, they'll share the messages on that queue between them, but still completely independent of any other queue.
Here's a screenshot from this interactive visualisation tool that shows this scenario:
As you mentioned it is not something that you can control with Redis Pub/Sub data structure.
But you can do it easily with Redis Streams.
Streams will allow you to post messages using the XADD command and then control which consumers are dealing with the message and acknowledge that message has been processed.
You can look at these sample application that provides (in Java) example about:
posting and consuming messages
create multiple consumer groups
manage exceptions
Links:
Getting Started with Redis Streams and Java
Redis Streams in Action ( Project that shows how to use ADD/ACK/PENDING/CLAIM and build an error proof streaming application with Redis Streams and SpringData )
I have a system where clients pull tasks from a queue (via amqp, using RabbitMQ as a message server). When a new client gets spawned, he pulls a task, tells the main server that he is the one in charge for that specific task and starts executing the long-time task. From the server side I want to be able to cancel the task on this client so he can move on to another one in case the task already causes errors.
Pushing a cancel task on the existing task queue is no option because I have to guarantee it arrives in time or even at the same client.
When the client gets spawned, is it a good approach that the client sets up a dedicated exchange and queue only for him where the server can send commands to?
The client won't have to set up a dedicated exchange - you can use the default topic exchange or set up your own topic exchange.
Then, when the client pulls a task it also creates its own exclusive queue and binds that queue to the topic exchange for a specific "cancel task X" routing key. Then, the client subscribes to this queue. If the task must be cancelled, the "server side" can publish a message to the topic exchange with the appropriate routing key.
NOTE: the RabbitMQ team monitors the rabbitmq-users mailing list and only sometimes answers questions on StackOverflow.
I use activemq with spring boot.
I wanted send topic only to specific subscribers. I know that to achieve it I can specify selectors on JMSListener but this mean that I already sent message to subscriber. What I want to do is to dispatch messages on broker level.
Currently I have below code:
public Broker broker() throws Exception {
BrokerService broker = new BrokerService();
broker.setBrokerName(BROKER_NAME);
broker.addConnector(BROKER_URL);
policy.setTopic(USER_TOPIC);
broker.start();
return broker.getBroker();
}
Let say I have messages which are able to send two subject BOOKS and FOOD.
For subscriber A only BOOKS are available and for subscriber B BOOKS and FOOD.
Question:
Is it possible to achieve it on broker level? If so how. I know that there is something like destination policy but I don't know how to use it :(
Messages can be routed and secured to different topic or queue destinations in a variety of ways:
Separate topics per subscribers (topic://BOOKS, topic://FOOD)
Single topic, but use a message header (topic://EVENTS)
a. Message header "EVENT_TYPE='FOOD'" or "EVENT_TYPE='BOOKS'"
You can secure the destinations so only the 'books' user can subscribe to the topic://BOOKS and only the 'food' user can subscribe to the topic://FOOD
Use a Virtual Topic which is publish to a topic, subscribe to a queue and get the best of all the above. For example publish to: topic://EVENTS and then subscribe from: queue://vConsumer.BOOKS.EVENTS, queue://vConsumer.FOOD.EVENTS
Links:
ActiveMQ Authorization: http://activemq.apache.org/security.html
ActiveMQ Virtual Topics: http://activemq.apache.org/virtual-destinations.html
I'm building a basic event based message system for a couple of services.
For my user service, I'm going to use a user topic exchange which will have routing keys like user.event.created, user.event.updated and user.event.deleted.
My logs service will consume user.event.* keys so I can log all events, whereas my email service will only listen for user.event.created as I'll only send out email on creation.
Now say I created a posts service, I want the logs service to consume events from here as well. Is it ok for me to bind both exchanges to the single logs.process queue?
Is there a better way of achieving this?
As long as each of the consume threads has it's own connection, it's fine. So, one thread consumes from topic exchange, the other from direct one etc.
As for the better part, I don't know - would require some more details.
This is the scenario - There are multiple app servers. Browser can connect via websocket to any app server.
The app servers (consumers) are all listening on a particular queue. As soon as a web socket connection is received, the particular app server binds the queue with a routing key {userId} to a direct exchange.
I want a message sent to the direct exchange with the routing key {userId} to be received by only the particular app server where the binding has occured.
Is a direct exchange the right exchange to use in this case? Or should some other type of exchange be used?
I'm using spring-amqp to create dynamic bindings when a websocket comes in
// create the RabbitMq queue and bind to it
String routingKey = MessageConstants.getRoutingKeyForUserRecommendationQueue(user);
Binding userRecommendationBinding = BindingBuilder.bind(userRecommendationsQueue).
to(directExchange).with(routingKey);
amqpAdmin.declareBinding(userRecommendationBinding);
Send message to a particular consumer in a queue
this is not possible. any consumer connected to a queue has a chance of consuming any given message in the queue
I want a message sent to the direct exchange with the routing key {userId} to be received by only the particular app server where the binding has occured.
you can do this by creating exclusive / autoDelete queues for your consumer, with a binding that directs all messages for that consumer to that queue.
Is a direct exchange the right exchange to use in this case?
either a direct exchange or a topic exchange is fine. direct exchange is slightly easier to understand, but topic exchange is more flexible
Actually you go right way.
And yes: Direct Exchange with an appropriate binding should save you.
See more info in the RabbitMQ Tutorial: http://www.rabbitmq.com/tutorials/tutorial-four-java.html
Also take a look into Spring AMQP Samples on the matter: https://github.com/spring-projects/spring-amqp-samples/tree/master/rabbitmq-tutorials
UPDATE
Unfortunately that is not what is happening. The messages seem to go randomly to any consumer, and not just the consumer that created the binding.
M-m-m. That's possible, because we route only my the key, but after that the message is placed to the queue, which may have several consumers on different machines.
In this case yes: the dynamic binding doesn't help.
You should consider to create an unique new queue (auto-deleted is fine) and bind and listen exactly from that. The SimpleMessageListenerContainer supports addQueues() at runtime to start a new consumer for a new queue.
I think that should work for you.
You still shouldn't do anything on the producer side: the same exhchange and routingKey logic.