I'm building a basic event based message system for a couple of services.
For my user service, I'm going to use a user topic exchange which will have routing keys like user.event.created, user.event.updated and user.event.deleted.
My logs service will consume user.event.* keys so I can log all events, whereas my email service will only listen for user.event.created as I'll only send out email on creation.
Now say I created a posts service, I want the logs service to consume events from here as well. Is it ok for me to bind both exchanges to the single logs.process queue?
Is there a better way of achieving this?
As long as each of the consume threads has it's own connection, it's fine. So, one thread consumes from topic exchange, the other from direct one etc.
As for the better part, I don't know - would require some more details.
Related
As I have been able to verify, in MassTransit with Azure Service Bus, each type of object consumed by a "Consumer" generates a Topic for that type regardless of whether it is only consumed in a specific "receive endpoint" (queue). When sending a message of this type with the "Send()" method, the message is sent directly to the "receive endpoint" (queue) without going through the topic. If this same message is published with the "Publish()" method, it is published in the Topic, and is forwarded to the receive endpoint (queue) from the corresponding subscriber.
My application uses a CQRS pattern where the messages are divided into commands and events. Commands use the send-receive pattern and are therefore always dispatched in MassTransit with the "Send()" method. The events, however, are based on the publish-subscribe pattern, and therefore are always dispatched in MassTransit with the "Publish()" method. As a result, a large number of topics are created on the bus that are never used (one for each type of command), since the messages belonging to these topics are sent directly to the receiver's queue.
For all these reasons, the question I ask is whether it is possible to configure MassTransit so that it does not automatically create the topics of some types of messages consumed because they will only be sent using the "Send()" method? Does this make sense in MassTransit or is it not possible/recommended?
Thank you!
Regards
Edited 16/04/2021
After doing some testing, I edit this topic to clarify that the intention is to configure MassTransit so that it does not automatically create the topics of some types of messages consumed, all of them received on the same receive endpoint. That is, the intention is to configure (dynamically if possible, through the type of object) which types of messages consumed create a topic and which do not in the same receive endpoint. Let's imagine that we have a receive endpoint (a queue) associated with a service, and this service is capable of consuming both commands and events, since the commands are only dispatched through Send(), it is not necessary to create the topic for them, however the events that are dispatched via Publish(), they need their topic (and their subscribers) to exist in order to deliver the message and be consumed.
Thanks in advance
Yes, for a receive endpoint hosting a consumer that will only receive Sent messages, you can specify ConfigureConsumeTopology = false for that receive endpoint. You can do that via a ConsumerDefinition, or when configuring the receive endpoint directly.
UPDATE
It is also possible to disable topology configuration per message type using an attribute on the message contract:
[ConfigureConsumeTopology(false)]
public interface SomeCommand
{
}
This will prevent the topic/exchange from being created and bound to the receive endpoint.
While I can understand the desire to be "pure to the CQRS mantra" and only Send commands, I'd suggest you read this answer and take it into consideration before overburdening your developers with knowing every single endpoint in the system by name...
We are using a microservice architecture to implement Web-APIs in nodejs. Every service exposes HTTP endpoints so the app / website can interact with it. To synchronize the different databases, we are currently using RabbitMQ. A microservice can publish a message on a fanout exchange and every subscribed microservice receives the message.
There are two problems with this architecture.
What if we want to add a second instance of a microservice (for loadbalancing purposes etc.). If the second service would subscribe to the same fanout exchange, the messages would be consumed two times.
Either acknowledgments do not work with fanout exchanges, or I'm doing something wrong. When I publish a message on a fanout exchange without subscribers, the messages disappears immediately without being acked.
This leads me to the my question. Is RabbitMQ a good choice for microservice synchronization or should we change our architecture. Here is a short example of how I would like it to work:
The user creates a new account
The auth-mc inserts the user in its database and publishes a 'user.created' event
a1-mc, a2-mc (same mc, just loadbalanced) and b1-mc are subscribers of the exchange. Either a1 or a2, as well as b1 receive the event and insert the user in their respective database
The event is only removed from each microservices queue, after its acknowledged
This way I can be sure, that every microservice (loadbalanced or not) receives the message one time. Can such a pattern even be implemented using RabbitMQ?
EDIT: Also looking for good literature about microservices if there are any suggestions.
Let's use topic exchange instead of fanout for your purpose. Only one consumer will receive the message instead of all of them. You could route your messages based on routing_key params for different consumers. For instance, you have an exchange. You have three different queues bound to this exchange with the same routing key. Your message will be duplicated for each queue! Your consumers from different microservices could read the message separately and do what they need. The message will be not dropped until you acknowledge it, but it's a good practice to push message with TTL.
TTL can be set on queues, so they will expire after a period of time if they are not used. Is there a similar option for exchanges?
I'm trying to build a social application and each exchange represents a user. Each time someone wants to send a message to this user, he would send the message to the user exchange. If number of the users become large, like 20 million users, there would be 20 million exchanges in the system. I'm afraid that much exchanges degrade the system. Instead I want to only keep exchanges for online users.
By the way the messages are only valuable if the user is online and I don't want to store messages for later delivery.
Having a separate exchange for every user ideed would be overkill. Try a different approach.
Use a single direct exchange.
When a client comes online it creates a new exclusive, auto-delete queue and consumes from it.
The client also binds the single exchange to its queue using the name of the user as the routing key.
Producers publish messages to the single exhange with the name of the user as the routing key of the message.
This will automatically
only keep queues for online users and
discard messages for offline users.
Edit: If a user shall be able to use multiple clients, that's possible using the above approach.
Every client creates a new exclusi auto-delete queue and consumes from it as above.
It binds the single exchange to this queue as above.
Note that it is possible to have multiple bindings from an exchange using identical routing keys. Every client has its own queue and its own binding, even if the routing key on this binding is the same routing key as on another binding created by a different client.
I am thinking how to remove duplicacy for subscriber cluster in pubsub, for example:
There is a service called email, which should send welcome emails after user signing up. By using pub/sub, email service shall listen a event called "signedUp" which will be triggered each time user sign up. However, how about if I have 2 email services to balance? Without any special efforts, I think two welcome emails will be sent out. So how to solve this issue?
I prefer redis as pubsub server for simplicity, or rabbitmq if redis doesn't work out.
I don't think it is possible to do it in redis. But in rabbitmq, it can, let me explain below:
Rabbitmq has a separate stuff called 'exchange' from queue. So server publish a message to exchange, client can create queues to bind to the exchange. Therefore, instances from one services can create the same queue to bind with exchange, by doing that exchange will delivery message to the queue once and handled by only one instance once.
Account service:
channel.assertExchange(‘signedUp’, 'fanout')
channel.publish(ex, '', new Buffer(message)
Email service:
let queue = channmel.assertQueue(‘email’);
channel.bindQueue(queue, 'signedUp'); // bind this queue to exchange
ch.consume(queue, logMessage)
By given a queue name in email service, no matter how many email services started, the published message (signedUp in this case) will be handled by one and ONLY ONE email service.
I have some camel routes with mina sockets and jetty websockets. I am able to broadcast a message to all the clients connected to the websocket but how do i send a message to a specific endpoint. How do i maintain a list of all connected clients with a client id as reference so i can route to a specific client. Is that possible? Will i be able to mention a dynamic client in the to URI?
Or maybe i am thinking about this wrong and i need to create topics on active mq and have the clients subscribe to it. That would mean that i create a topic for every websocket client? and route the message to the right topic.
Am i atleast on the right track here, any examples you can point out? Google was not helpful.
The approach you take depends on how sensitive the client information is. The downside of a single topic with selectors is that anyone can subscribe to the topic without a selector and see all the information for everyone - not usually something that you want to do.
A better scheme is to use a message distribution mechanism (set of Camel routes) that act as an intermediary between the websocket clients and the system producing the messages. This mechanism is responsible for distributing messages from a single destination to client-specitic destinations. I have worked on a couple of banking web front-ends that used a similar scheme.
In order for this to work you first generate for each user a distinct token/UUID; this is presented to the user when the session is established (usually through some sort of profile query/message).
It's essential that the UUID can be worked out as a hash of the clientId rather than being stored in a DB, as it will be used all the time and you want to make sure this is worked out quickly.
The user then uses that information to connect to specific topics that use that UUID as a suffix. For example two users subscribing to an orderConfirmation topic would each subscribe to their own version of that topic:
clientA -> orderConfirmation.71jqsd87162iuhw78162wd7168
clientB -> orderConfirmation.76232hdwe7r23j92irjh291e0d
To keep track of "presence", your clients would need to periodically send a heartbeat message containing their clientId to a well-known topic that your distribution mechanism listens on. Clients should not be able to subscribe to this topic for reads (see ActiveMQ Security). The message distribution mechanism needs to keep in memory a data structure that contains the clientId and the time a heartbeat was last seen.
When a message is received by the distribution mechanism, it checks whether the clientID for which it received the message has a "live/present" session, determines the UUID for the client, and broadcasts the message on the appropriate topic.
Over time this will create a large number of topics on your broker that you don't want hanging around when the user has gone away. You can configure ActiveMQ to delete these if they have been inactive for some time.
You definitely do not want to create separate endpoint for each client.
Topic and a subscription with selector is an elegant way to resolve it.
I would say the best one.
You need single topic, which every client would subscribe to with the selector looking like where clientId in ('${myClientId}', 'EVERYONE'). Now when you want to publish a message to specific client, you set a property clientId to the id of this client. If you want to broadcast, you set it to 'EVERYONE'
I hope I understand the problem right...