I have several exchanges that are dumping messages into a single queue for consumption by a client app. When the messages are received, I'd like to be able to see the exchange the message was originally published to so I can add some metadata to the object. Is there anything I can look at in the message properties that can tell me where it came from?
The AMQP 0-9-1 specifications include the name of the exchange the message was published to in the basic.get_ok answer of the basic.get method.
Therefore, you should be able to retrieve such information. It just depends on the client you are using and to what degree it honors the AMQP specifications.
Related
As I have been able to verify, in MassTransit with Azure Service Bus, each type of object consumed by a "Consumer" generates a Topic for that type regardless of whether it is only consumed in a specific "receive endpoint" (queue). When sending a message of this type with the "Send()" method, the message is sent directly to the "receive endpoint" (queue) without going through the topic. If this same message is published with the "Publish()" method, it is published in the Topic, and is forwarded to the receive endpoint (queue) from the corresponding subscriber.
My application uses a CQRS pattern where the messages are divided into commands and events. Commands use the send-receive pattern and are therefore always dispatched in MassTransit with the "Send()" method. The events, however, are based on the publish-subscribe pattern, and therefore are always dispatched in MassTransit with the "Publish()" method. As a result, a large number of topics are created on the bus that are never used (one for each type of command), since the messages belonging to these topics are sent directly to the receiver's queue.
For all these reasons, the question I ask is whether it is possible to configure MassTransit so that it does not automatically create the topics of some types of messages consumed because they will only be sent using the "Send()" method? Does this make sense in MassTransit or is it not possible/recommended?
Thank you!
Regards
Edited 16/04/2021
After doing some testing, I edit this topic to clarify that the intention is to configure MassTransit so that it does not automatically create the topics of some types of messages consumed, all of them received on the same receive endpoint. That is, the intention is to configure (dynamically if possible, through the type of object) which types of messages consumed create a topic and which do not in the same receive endpoint. Let's imagine that we have a receive endpoint (a queue) associated with a service, and this service is capable of consuming both commands and events, since the commands are only dispatched through Send(), it is not necessary to create the topic for them, however the events that are dispatched via Publish(), they need their topic (and their subscribers) to exist in order to deliver the message and be consumed.
Thanks in advance
Yes, for a receive endpoint hosting a consumer that will only receive Sent messages, you can specify ConfigureConsumeTopology = false for that receive endpoint. You can do that via a ConsumerDefinition, or when configuring the receive endpoint directly.
UPDATE
It is also possible to disable topology configuration per message type using an attribute on the message contract:
[ConfigureConsumeTopology(false)]
public interface SomeCommand
{
}
This will prevent the topic/exchange from being created and bound to the receive endpoint.
While I can understand the desire to be "pure to the CQRS mantra" and only Send commands, I'd suggest you read this answer and take it into consideration before overburdening your developers with knowing every single endpoint in the system by name...
Is there any way to check a published message if no consumers are created? Now I just have a project which publishes events and I can see that they are published by checking exchanges created in RabbitMQ. But I'm not sure if there is a way to check the message content from the Rabbit MQ interface if there are no consumers. Maybe is it better to cover a publish logic with a unit test to check that a message with correct content is published?
You can specify an alternate exchange for published messages, and those messages will be delivered to the alternate exchange if no other exchanges are bound. You can then bind that an exchange to a queue to retain those messages if there are no subscribers.
Alternatively, you can specify the Mandatory flag on the published message, and the Publish call with throw an exception if there are no exchanges bound.
Alternate Exchange Unit Test
Mandatory Unit Test
Validation ("checking") of message content is a client function, and cannot be performed in the message queue itself. It either must be performed prior to publish (trust the publisher) or it must be performed immediately after dequeue but before processing (don't trust the publisher).
I've been trying to find a way to CC messages from a Qpid Exchange to another Queue for testing/monitoring purposes. I noticed that a RabbitMQ user out there was having a similar problem, and the solution seemed to be RabbitMQ's Firehose feature. Is there a similar solution in Qpid?
Here's some more details for the curious.
Let's call the Exchange "App.Ex", and through it are flowing messages for a single other intended recipient (let's call him "Bob")
I connect to App.Ex, initiate a session, start a receiver, and start fetching (using code adapted from the QPID Book's "A Simple Messaging Program in Python")
I start seeing the messages I want to see. HOWEVER, in doing so I've robbed Bob of the messages he needs!
So there's the rub, how can I get the messages CC'd to me, but in a way that Bob still receives his messages?
I have permission to modify the messaging configuration, so I can create my own Queues and Exchanges if need be. Thoughts appreciated!
A direct exchange is probably most appropriate because you can have some queues with CC like behavior and some without, and you can change it anytime on a live exchange.
You can have two queues bound to the same subject/routing key. When a message is sent to the exchange with that particular subject/routing key, both bound queues will receives copies of the same message.
Both queues bar1 and bar2 are bound to routing_key foo. When producer B posts messages to the exchange with routing_key = foo, both bar1 and bar1 receive copies of all messages.
Ask if you need commands for creating the exchange and appropriate bindings.
However there are more ways to do the same thing:
You can also achieve the similar behavior using a topic queue, with exact matches on topic name
Lastly, you can also use a fanout exchange, where any message you send to the queue, a copy is routed to all the queues bound to the exchange.
Note that all of these exchange types are from the AMQP spec, so they are not qpid specific, you could do the same thing or something very similar in any AMQP implementation.
I have a code which has both consumer and producer. I want to differentiate or find the exact exchange name through which the consumer has consumed the message. For example, I have almost 5 exchanges and I want to know through which exchange out of that 5 the consumer has received it's message. How can this be achieved?
I have done lot of homework but couldn't find a solution.
Messages are consumed from queues, not exchanges.
The way to figure out original exchange that message was published to is to use Firehose Tracer plugin (maybe even with rabbitmq-tracing
plugin alongside).
Alternatively, you may figure out original exchange by comparing queues bindings with message routing key. This usually work well in most cases, unless you have really wired publishers and routing logic.
P.S.: finally, if you have at least read access to publishers code you can figure out where each messages goes from.
The undelying use case
It is typical pubsub use case: Consider we have M news sources, and there are N subscribers who subscribe to the desired news sources, and who want to get news updates. However, we want these updates to land up in mongodb - essentially maintain most recent 'k' updates (and can be indexed and searched etc.). We want to design for M to scale upto million publishers, N to scale to few millions.
Subscribers' updates are finally received and stored in more than one hosts and their native mongodbs.
Modeling in rabbitmq
Rabbitmq will be used to persist the mappings (who subscribes to which news source).
I have setup a pubsub system in this way: We create publisher exchanges (each mapping to one news source) and of type 'fanout'.
For modelling subscribers, there are two options.
In the first option, have one queue for each subscriber bound to relevant publisher exchanges. And let the client process open connections to all these subscriber queues and receive the updates (and persist them to mongodb). Note that in this option, when the client is restarted, it has to manage list of all susbcribers, and open connections to all subscriber queues it is responsible for.
In the second option, we want to be able to remove overhead of having to explicitly open on each user queue upon startup. Instead, we want to listen to only one queue - representative of all subscribers who will send updates to this client host.
For achieving this, we first create one exchange for each subscriber and let it bind to the publisher exchange(s) that it follows. We let a single queue for each client, and let the subscriber exchange bind to this queue (type=direct) if the subscriber belongs to that client.
Once the client receives the update message, it should come to know which subscriber exchange it came from. Only then we can add it to mongodb for relevant subscriber. Presumably the subscriber exchange should add this information as a new header on the message.
As per rabbitmq docs, I believe there is no way to get achieve this. (Or more specifically, to get the 'delivery path' property from the delivered message, from which we can get this information).
My questions:
Is it possible to add a new header to message as it passes through exchange?
If this is not possible, then can we achieve it through custom exchange and relevant plugin? Any plugin that I can readily use for this purpose?
I am curious as to why rabbitmq is not providing delivery path property as an optional configuration?
Is there any other way I can achieve the same? (See pubsubhubbub note below)
PubSubHubBub
The use case is very similar to what pubsubhubbub protocol provides for. And there is rabbitmq plugin too called rabbithub. However, our system will be a closed system, and I believe that the webhook approach of the protocol is going to be too much of overhead compared to listening on single queue (and from performance perspective.)
The producer (RMQ Client) of the message should add all the required headers (including the originator's identity) before producing (publishing) it on RMQ. These headers are used for routing.
If, while in transit, the message (including headers) needs to be transformed (e.g. adding new headers), it needs to be sent to the transformer (another RMQ Client). This transformer will essentially become the new publisher.
The actual consumer should receive its intended messages (for which it has subscribed to) through single queue. The routing of all its subscribed messages should be arranged on the RMQ Exchange.
Managing the last 'K' updates should neither be the responsibility of the producer nor the consumer. So, it should be done in the transformer. Producers' messages should be routed to this transformer (for storage) before further re-routing to exchange(s) from where consumers consume.