Take the following sample requirement:
Service A does some work. That work is done periodically and no one asks Service A to do it. It's done automatically.
Service B needs to query data produced by Service A. Service A resides in a different server than Service B.
That is, Service B won't be able to get data if Service A doesn't provide some way of asking it for the data.
I want to require Service A data the SOA way using RabbitMQ: when Service B requires some data, it sends a message to a given exchange and its written to some queue. Then, Service A processes the message and publishes the answer to some other exchange. Finally, Service B listens the answer message and the cycle ends.
My question
I need some way to both publish and consume messages identified by the operation that requested data to Service A, and I also need that each started operation could be identified by an unique identifier.
My question is about how to publish a message and be able to receive an answer for a particular invocation of an operation.
I just want to validate that RabbitMQ routing keys are the answer to this requirement. For example, Service A sends a message with a routing key 072e6ee1-6046-4c3b-bade-9077c863637b. There's a consumer in Service B which consumes any message ignoring the routing key, but once it produces a result, it does publishing a message to an exchange with the same routing key . Therefore, Service A receives message because it's bound to the whole routing key.
Is it a possible right usage of routing keys?
This doesn't seem to be the correct way to use routing keys, if I understand you correctly. Routing keys tend to be used to define an operation, eg 'do-this' or 'do-that' - I wouldn't expect them to contain keys, unless those keys were a limited set that defined how the system operates. The Ids that you mention (they seem to be correlation Ids, so that you can match a received response to an outbound request, asynchronously) would be contained in the body of the message. So, you have the following setup:
Service B -> send message to exchange with routing key 'process-data'. The message contains the Id in its body.
Service A listens to messages on Queue A, which is is bound to the exchange with binding key 'process-data'. It then dequeue's Service B's message, whose body contains the Id
Service A performs processing
Service A -> send message to exchange with routing key 'data-processed'. The message contains the original Id in its body.
Service B listens to messages on Queue B, which is is bound to the exchange with binding key 'data-processed'. It then dequeue's Service A's message, whose body contains the Id
Finally, RabbitMQ already provides a solution to solve my scenario: Remote Procedure Calls tutorial.
At the end of the day, a message can have correlation id and a reply to fields:
Correlation id. An arbitrary value used to uniquely-identify an operation across many exchanges/queues.
Reply to. An arbitrary string to define on which exchange or queue you want to put the result of a given operation. The target exchange/queue will be the callback information that will be received by the operation.
Thus, Service A sends a message to a given exchange with a correlation id and a reply to value. It gets subscribed to reply to exchange. When Service B processes the message it publishes a message to the reply to exchange and Service A receives the whole result.
BTW, I still have some concerns about this model, because you need to receive callback messages from other operations while you wait for the result, and whenever you consume a message that's not marked with desired correlation id, you simply don't process it and you wait for the next one, and so on, until you get what you want.
Maybe it might be a good idea that callback messages would be queued with a time-to-live so if who started an operation isn't waiting for the callback anymore, a given callback message would be automatically dropped.
Another approach would be throwing the callback information as a document or record in a SQL/NoSQL database, and poll the database with an indexed query by callback identifier...
If I find some other approach using RabbitMQ I'll update my answer.
Related
As I have been able to verify, in MassTransit with Azure Service Bus, each type of object consumed by a "Consumer" generates a Topic for that type regardless of whether it is only consumed in a specific "receive endpoint" (queue). When sending a message of this type with the "Send()" method, the message is sent directly to the "receive endpoint" (queue) without going through the topic. If this same message is published with the "Publish()" method, it is published in the Topic, and is forwarded to the receive endpoint (queue) from the corresponding subscriber.
My application uses a CQRS pattern where the messages are divided into commands and events. Commands use the send-receive pattern and are therefore always dispatched in MassTransit with the "Send()" method. The events, however, are based on the publish-subscribe pattern, and therefore are always dispatched in MassTransit with the "Publish()" method. As a result, a large number of topics are created on the bus that are never used (one for each type of command), since the messages belonging to these topics are sent directly to the receiver's queue.
For all these reasons, the question I ask is whether it is possible to configure MassTransit so that it does not automatically create the topics of some types of messages consumed because they will only be sent using the "Send()" method? Does this make sense in MassTransit or is it not possible/recommended?
Thank you!
Regards
Edited 16/04/2021
After doing some testing, I edit this topic to clarify that the intention is to configure MassTransit so that it does not automatically create the topics of some types of messages consumed, all of them received on the same receive endpoint. That is, the intention is to configure (dynamically if possible, through the type of object) which types of messages consumed create a topic and which do not in the same receive endpoint. Let's imagine that we have a receive endpoint (a queue) associated with a service, and this service is capable of consuming both commands and events, since the commands are only dispatched through Send(), it is not necessary to create the topic for them, however the events that are dispatched via Publish(), they need their topic (and their subscribers) to exist in order to deliver the message and be consumed.
Thanks in advance
Yes, for a receive endpoint hosting a consumer that will only receive Sent messages, you can specify ConfigureConsumeTopology = false for that receive endpoint. You can do that via a ConsumerDefinition, or when configuring the receive endpoint directly.
UPDATE
It is also possible to disable topology configuration per message type using an attribute on the message contract:
[ConfigureConsumeTopology(false)]
public interface SomeCommand
{
}
This will prevent the topic/exchange from being created and bound to the receive endpoint.
While I can understand the desire to be "pure to the CQRS mantra" and only Send commands, I'd suggest you read this answer and take it into consideration before overburdening your developers with knowing every single endpoint in the system by name...
I wonder if the following scenario is possible:
Create an exchange of type direct
Publish a message to that exchange with routing key rk1
After that:
Create a queue which accepts messages with routing key rk1
Consume message published to exchange
It seems like if there is no queue present, the message is dropped and there is no way to receive it.
So basically I want to be able to produce messages when there are no consumers present. And consume them some time later.
It seems like if there is no queue present, the message is dropped and there is no way to receive it.
Yes, this is correct, but it's only part of the story.
A message queue is the storage location for messages published to the server. A consumer is a designated connection set to receive messages put into a queue. The exchange is simply a location to push messages. It contains the routing semantics to determine which messages wind up in the queues on the server. When a message cannot be routed to a queue and/or consumer, there are various semantics that can apply, but the default is that the message is dropped.
Options for dealing with unroutable messages:
Alternate exchange - designates a different exchange where messages can be dumped if they cannot be routed to a queue on the current exchange. This can be thought of similar to how TCP/IP works when a destination host is not reachable on the current subnet, and traffic is forwarded to the gateway. Note that a queue must be bound to the alternate exchange for the message to be dumped into. A typical case might be to have it configured as fanout exchange with one queue to trap all messages sent into the alternate exchange.
Mandatory or Immediate - return a message back to the sender if it can't be delivered. The server does not store the message.
Mandatory designates that the message must be deliverable to a queue at the time it is published. If the message is not routable, the publisher will receive a basic.return.
Immediate designates that, in addition to being deliverable, must be immediately routed to a consumer on a particular queue (e.g. it's not good enough that it be dumped in a queue for pickup later - it has to be delivered to the end consumer right now.
In every case, if there is no queue, the server cannot store the message.
The entity queue is the one that is supposed to hold the messages , so without a queue the messages will be lost.
However in case you do not create any exchange with appropriate routing key you may leverage dead lettering feature in rabbitmq.
Another solution could be to declare the queue with the binding after the exchange and before publishing the message; this way the message will be routed and stored, but you may have to add some TTLs ( https://www.rabbitmq.com/ttl.html ).
My scenario is such that I want to have one queue for the producer where the published messages will be stored. The messages will be associated with some routing key such that abcd.1234(some id).
Now from the consumer part I want that every consumer will connect to this queue and pop the messages based on the routing key pattern. Such as say consumer for abcd.123 will pop the messages with routing key abcd.123 and so on.
I tried to use topic exchange but I don't think its directly fitting into this use case as topic exchange generates one queue for each subscriber(consumer)
So, in there any way to consume (pop) message from queue based on the routing key of that queue.
Similar to queue.pop(:routing_key:xyz)
kindly help on this how to proceed?
I am new to RabbitMQ and this is confusing me. I have setup a direct exchange and different queues subscribe to different routing keys on this exchange. What I want is that if a message gets published with one routing key, it gets consumed from one subscriber only, no matter how many queues subscribe to that routing key.
Current scenario:
Exchange (type: direct)
-QueueA1 (receives message A from exchange with routing key of "TypeA")
-QueueA2 (also receive message A from exchange with routing key of "TypeA")
-QueueB (doesn't receive message A because it subscribes to key "TypeB")
Desired:
-QueueA1 (receives message A from exchange with routing key of "TypeA")
-QueueA2 (doesn't receive message A because it's already consumed by QueueA1)
-QueueB (doesn't receive message A because it subscribes to key "TypeB")
Do I need to use a different exchange? How do I achieve desired scenario?
You can achieve what you want by using a single QueueA with multiple consumers subscribed to this queue:
Direct exchange
|
|-- ["TypeA"]--> QueueA
| |-- Consumer A1
| `-- Consumer A2
|
`-- ["TypeB"]--> QueueB
In this case, a message queued on QueueA will be delivered to only one consumer. The consumer which gets the message is undefined however: they are picked in a round-robin manner.
w/ RabbitMQ, all routing keys that match within a given exchange, will have a copy of the message delivered to the specified queue.
So, in your scenario, you will always have QueueA1 and QueueA2 receive messages of TypeA. This is how routing keys work. There is no way around this, using a single exchange.
If you need QueueA1 and QueueA2 to receive different messages, then you either:
need to use different routing keys to bind the queue, or
need to use a different exchange
Regarding the suggestion from Jean-Sebastient...
This scenario will allow either Consumer A1 or Consumer A2 to handle the message in question, but does so by having them both subscribe to the same queue.
If you're trying to ensure there is one 1 consumer that receives the message, then this is what you want to do. But, if you need both queues and need to ensure only one queue receives the message, then you need to go with one of the other options I suggested.
Finally, if you are looking at guaranteeing a message is processed only once, none of these options will do that.
On the surface, it will look like this works most of the time. But there will be scenarios where something goes wrong and you'll have more than one queue or consumer process the same message.
To handle this scenario, you need to look a "idempotence" in your message handling. This is most commonly handled with a database and an ID to say it has been processed already, but there are other solutions out there.
In our scenario I'm thinking of using the pub sub technique. However I don't know which is the better option.
1 ########
A web service of ours will publish a message that something has happened when it is called externally, ExternalPersonCreatedMessage!
This message will contain a field that represents the destinations to process the message into (multiple allowed).
Various subscribers will subscribe. These subscribers will filter the message to see if any action is required by checking the destination field.
2 ########
A web service of ours will parse the incoming call and publish specific types of messages depending on the destinations supplied in the field. i.e. many Destination[n]PersonCreatedMessage messages would be created.
Subscribers will subscribe to only the specific message they care for. i.e. not having to filter any messages
QUESTIONS
Which of the above is the better option and why? And how do I stop myself from making RequestMessages. From what I've read/seen I should be trying to structure this in a way of PersonCreated, PersonDeleted i.e. SOMETHING HAS HAPPENED and NOT in the REQUEST SOMETHING TO HAPPEN form such as CreatePerson or DeletePerson
Are my thoughts correct? I've been looking for guidance on how to structure messages and making sure I don't go down a wrong path but have found no guidance out there on do's and dont's. Can any one help and guide? I want to try and get this correct from the off :)
Based on the integration scenario in the referenced article, it appears to me that you may need a Saga to complete the workflow of accept message -> operate on message -> send confirmation. In the case that the confirmation is sent immediately after the operation, you could use NSBs message handler pipeline feature which allows you to chain handlers in a specified sequence such as...
First<FilterHandler>.Then<DoWorkHandler>().AndThen<SendConfirmationHandler>();
In terms of the content filtering, you can do this although you incur some transport overhead, meaning the queue will have to accept the message and the process will always call the first handler on every message(you can short-circuit the above pipeline at any point). It may be the case that what you really want is a Distributor/Worker setup where all Workers are the same and you can handle some load.
If you truly have different endpoints with completely different logic, then I would have the Publisher process(only accepts and Publishes message) do the work of translating the inbound message to something else a Subscriber can then be interested in. If then you find that a given Published message only ever has 1 Subscriber, then you don't need to Publish at all, you need to just Bus.Send() to the correct endpoint.
The way NServiceBus handles pub-sub is more like your option two.
A publisher service has an input queue and a subscription store.
A subscriber service has an input queue
The subscriber, on start-up will send a subscription message to the input queue of the publisher
The subscription message contains the type of message subscriber is interested in and the subscribers queue address
The publisher records the subscription in the subscription store.
The publisher receives a message.
The publisher evaluates the message type against the list of subscriptions
For each match found the publisher sends the message to the queue address.
In my opinion, you should stop thinking about destinations. Messages are messages. They should not have any inherent destination information in them. The subscription mechanism defines the addressing/routing requirements for the solution.