I've got 2 services set up to receive the same message, e.g. ConsumerA : IConsumer<IMessageA> and ConsumerB : IConsumer<IMessageA>. Each service sets up a unique endpoint, e.g. queue_a and queue_b and registers its consumer. I see in RabbitMQ an exchange for IMessageA that is of type fanout and bound to queue_a and queue_b. So far, so good.
I run both services and publish a message, but only service A is gets it.
I stop service A and manually publish a message to B in RabbitMQ (service A is a web service that publishes IMessageA in response to a POST using IRequestClient<IMessageA, IMessageAResponse>, which I why I need to manually post) and now service B gets the message and consumes it as expected.
To be clear, with service A stopped, the message is routed by RabbitMQ to both queue_a and queue_b. If service A is running, the message only goes to queue_b in spite of exchange bindings existing showing that queue_b is bound to the IMessageA exchange and absolutely should be getting them. Or at least by the time I'm able to examine RabbitMQ through the management web UI, there is no evidence there was ever a message delivered to queue_b (i.e. nothing in queue_b_error or queue_b_skipped, the latter not even existing).
I've added IReceiveObserver to both service A and B but nothing is triggering ReceiveFault or ConsumeFault.
The consumer in service A is basically doing:
var result = await MethodThatReturnsIMessageAResponse(messageA);
context.Respond(result);
Why is service A interfering with delivery of messages to service B? Where do I even begin looking?
The problem was that I was "publishing" the messages using:
c.Resolve<IBus>().CreateRequestClient<IToDoMessage, IToDoMessageResponse>(new Uri(QueueAddress + QueueName),TimeSpan.FromSeconds(5));
which requires a specific enpoint (e.g. queue name). Instead, I needed to use CreatePublishRequestClient:
c.Resolve<IBus>().CreatePublishRequestClient<IToDoMessage, IToDoMessageResponse>(TimeSpan.FromSeconds(5));
which uses the bus to publish, and goes through the exchange, not to a specific queue. It doesn't help that the GitHub sample project shows the former...
Related
As I have been able to verify, in MassTransit with Azure Service Bus, each type of object consumed by a "Consumer" generates a Topic for that type regardless of whether it is only consumed in a specific "receive endpoint" (queue). When sending a message of this type with the "Send()" method, the message is sent directly to the "receive endpoint" (queue) without going through the topic. If this same message is published with the "Publish()" method, it is published in the Topic, and is forwarded to the receive endpoint (queue) from the corresponding subscriber.
My application uses a CQRS pattern where the messages are divided into commands and events. Commands use the send-receive pattern and are therefore always dispatched in MassTransit with the "Send()" method. The events, however, are based on the publish-subscribe pattern, and therefore are always dispatched in MassTransit with the "Publish()" method. As a result, a large number of topics are created on the bus that are never used (one for each type of command), since the messages belonging to these topics are sent directly to the receiver's queue.
For all these reasons, the question I ask is whether it is possible to configure MassTransit so that it does not automatically create the topics of some types of messages consumed because they will only be sent using the "Send()" method? Does this make sense in MassTransit or is it not possible/recommended?
Thank you!
Regards
Edited 16/04/2021
After doing some testing, I edit this topic to clarify that the intention is to configure MassTransit so that it does not automatically create the topics of some types of messages consumed, all of them received on the same receive endpoint. That is, the intention is to configure (dynamically if possible, through the type of object) which types of messages consumed create a topic and which do not in the same receive endpoint. Let's imagine that we have a receive endpoint (a queue) associated with a service, and this service is capable of consuming both commands and events, since the commands are only dispatched through Send(), it is not necessary to create the topic for them, however the events that are dispatched via Publish(), they need their topic (and their subscribers) to exist in order to deliver the message and be consumed.
Thanks in advance
Yes, for a receive endpoint hosting a consumer that will only receive Sent messages, you can specify ConfigureConsumeTopology = false for that receive endpoint. You can do that via a ConsumerDefinition, or when configuring the receive endpoint directly.
UPDATE
It is also possible to disable topology configuration per message type using an attribute on the message contract:
[ConfigureConsumeTopology(false)]
public interface SomeCommand
{
}
This will prevent the topic/exchange from being created and bound to the receive endpoint.
While I can understand the desire to be "pure to the CQRS mantra" and only Send commands, I'd suggest you read this answer and take it into consideration before overburdening your developers with knowing every single endpoint in the system by name...
I'm using Masstransit with RabbitMq to communicate between two web applications .net core.
In the publisher application I publish a message to an exchange 'producer'.
rabbitConfigurator.Message<TMessage>(x => x.SetEntityName("producer"));
In the same application I have a consumer-A of said messages and that works fine (the consumer has an exchange and a queue as recommended by rabbit/masstransit connected to the producer exchange)
rabbitConfigurator.ReceiveEndpoint("consumer-A", x =>
{
x.Consumer<TConsumer>(context);
x.Bind("producer");
});
In a second application I'm trying to setup another consumer-B of the same message type connected to the exchange 'producer'.
rabbitConfigurator.ReceiveEndpoint("consumer-B", x =>
{
x.Consumer<TConsumer>(context);
x.Bind("producer");
});
However messages destined for this second application are getting sent to a skipped queue. From what I can see this second consumer is correctly configured in RabbitMq though.
I can't see what I'm missing.
Edit: In the producing application I can also successfully receive messages in a second consumer. But from another web application it doesn't work - perhaps something to do with the fact that there's a differetn connection/channel ?
Messages sent to the producer exchange should be in a format that can be deserialized by MassTransit. If they aren't, you'll need to add a message deserializer which can support the message type. If it is application/json then you'll need to add the RawJsonMessageDeserializer to the receive endpoint.
If the messages are in the correct format, verify that you have a consumer on the receive endpoint that can consume that message type. Message types must match entirely, including namespace, to be consumed. More details are available in the documentation.
Requirement
A system undergoes some state change, and multiple other parts of the system has to know this(lets call them observers) so that they can perform some actions based on the current state, the actions of the observers are important, if some of the observers are not online(not listening currently due to some trouble, but will be back soon), the message should not be discarded till all the observers gets the message.
Trying to accomplish this with pub/sub model, here are my findings, (please correct if this understanding is wrong) -
The publisher creates an event on specific topic, and multiple subscribers can consume the same message. This model either provides no delivery guarantee(in redis), or delivery is guaranteed once(with messaging queues), ie. when one of the consumer acknowledges a message, the message is discarded(rabbitmq).
Example
A new Person Profile entity gets created in DB
Now,
A background verification service has to know this to trigger the verification process.
Subscriptions service has to know this to add default subscriptions to the user.
Now both the tasks are important, unrelated and can run in parallel.
Now In Queue model, if subscription service is down for some reason, a BG verification process acknowledges the message, the message will be removed from the queue, or if it is fire and forget like most of pub/sub, the delivery is anyhow not guaranteed for both the services.
One more point is both the tasks are unrelated and need not be triggered one after other.
In short, my need is to make sure all the consumers gets the same message and they should be able to acknowledge them individually, the message should be evicted only after all the consumers acknowledged it either of the above approaches doesn't do this.
Anything I am missing here ? How should I approach this problem ?
This scenario is explicitly supported by RabbitMQ's model, which separates "exchanges" from "queues":
A publisher always sends a message to an "exchange", which is just a stateless routing address; it doesn't need to know what queue(s) the message should end up in
A consumer always reads messages from a "queue", which contains its own copy of messages, regardless of where they originated
Multiple consumers can subscribe to the same queue, and each message will be delivered to exactly one consumer
Crucially, an exchange can route the same message to multiple queues, and each will receive a copy of the message
The key thing to understand here is that while we talk about consumers "subscribing" to a queue, the "subscription" part of a "pub-sub" setup is actually the routing from the exchange to the queue.
So a RabbitMQ pub-sub system might look like this:
A new Person Profile entity gets created in DB
This event is published as a message to an "events" topic exchange with a routing key of "entity.profile.created"
The exchange routes copies of the message to multiple queues:
A "verification_service" queue has been bound to this exchange to receive a copy of all messages matching "entity.profile.#"
A "subscription_setup_service" queue has been bound to this exchange to receive a copy of all messages matching "entity.profile.created"
The consuming scripts don't know anything about this routing, they just know that messages will appear in the queue for events that are relevant to them:
The verification service picks up the copy of the message on the "verification_service" queue, processes, and acknowledges it
The subscription setup service picks up the copy of the message on the "subscription_setup_service" queue, processes, and acknowledges it
If there are multiple consuming scripts looking at the same queue, they'll share the messages on that queue between them, but still completely independent of any other queue.
Here's a screenshot from this interactive visualisation tool that shows this scenario:
As you mentioned it is not something that you can control with Redis Pub/Sub data structure.
But you can do it easily with Redis Streams.
Streams will allow you to post messages using the XADD command and then control which consumers are dealing with the message and acknowledge that message has been processed.
You can look at these sample application that provides (in Java) example about:
posting and consuming messages
create multiple consumer groups
manage exceptions
Links:
Getting Started with Redis Streams and Java
Redis Streams in Action ( Project that shows how to use ADD/ACK/PENDING/CLAIM and build an error proof streaming application with Redis Streams and SpringData )
In my system I use spring-cloud-stream and RabbitMQ for sending and receiving events. I got my RabbitMQ running, service A up and service B down. Service A sends an event to service B. Then I turn up my service B and now I expect Rabbit to deliever the event - but nothing happens. Is it correct behaviour? I'm new to RabbitMQ but I though that it should guarantee that all events will eventually finds its receivers. My application is simple, based on example on github with no extra configuration. What do I miss?
If your consumers don't have a group, the queue is an anonymous, auto-delete queue. You need a group for persistence. See consumer groups.
Producers don't bind queues to the exchange, consumers do.
If you bind the producer first, before a new consumer group, messages will also be lost.
With the RabbitMQ binder, if you know the consumer groups ahead of time, you can set the ...producer.requiredGroups property and the queue(s) will be bound.
See the documentation.
requiredGroups
A comma-separated list of groups to which the producer must ensure message delivery even if they start after it has been created (e.g., by pre-creating durable queues in RabbitMQ).
Just started learning NServiceBus and trying to understand the concept.
When it talks about queues, are we talking about MSMQs on both publisher and subscriber?
So, if I have an application that generates a list of something (say, name of animals), then it dumps the list into publisher’s queue. The publisher polls the queue every minute and if there is something in the queue, it will publish to subscriber’s queue for further processing. Does this make sense?
Thanks.
The sequence of events for a publish is as follows:
The Publisher will start up(Windows Service)
A Subscriber will start up and place a message into the Publisher's input queue(MSMQ)
The Publisher will take that message, read the address of the Subscriber and place that into storage(subscription storage: memory, MSMQ, or RDBMS)
When it is time to publish and event, the Publisher will inspect the type of message and then read subscription storage to find Subscribers interested in that message
The Publisher will then send a message to each of the Subscribers found in subscription storage
The Subscriber receives the message in its input queue(MSMQ) and processes it
You can leverage other messaging platforms instead of MSMQ, but MSMQ is the default. There really is no polling done, all the endpoints are signaled when a message hits the queues.
MSMQ is a transport layer. It passes the messages around.
The application will publish something using a NServiceBus queue. If you configured it to use MSMQ, that's what it will use for its transport layer and this is what the subscribers will be looking at.
NServiceBus follows the publisher/subscriber model as you have correctly stated. However your confusion is based on the use of two queues. This is incorrect. The server (publisher) will maintain the queue which is interfaced via the MSMQ protocol and so your application would communicate directly with this possibly remotely or locally.
You would typically use a WCF service which would raise an event upon a new message being pushed onto the queue. Your application can then make use of this new message as desired. See the NServiceBus documentation for examples: http://www.nservicebus.com/ArchitecturalPrinciples.aspx