I have:
- one subscriber SUB with QUEUE0
- publisher PUB1 with QUEUE1
- publisher PUB2 with QUEUE2
- event MyEvent being published by both publishers
When:
- SUB explicitly subscribes to PUB1 with queue name QUEUE1 only
subscriberEndpointConfiguration.UnicastRouting().AddPublisher("PUB1", typeof(MyEvent));
Outcome:
- SUB also receives MyEvent from PUB2 (which has queue name QUEUE2)
Expected:
- SUB should not receive MyEvent from PUB2, because it's not subscribed to that publisher queue name
From NSB wiki:
subscribers express interest in one or more classes, and only receive messages that are of interest, without knowledge of what, if any, publishers there are
Questions:
What's the point of specifying the publisher endpoint in the AddPublisher method shown above? The Subscription table in the Azure Table Storage has an event type and a subscriber columns only, publisher endpoint is not stored.
If AddPublisher is some sort of an obsolete method, then the endpointInstance.Subscribe<MyEvent>() simply fails - it says "no publishers could be found".
Is it possible to scope/group publishers so having only one event type MyEvent the subscribers will receive that events from publishers which are created with the same queue name only?
E.g. you create PUB1 with QUEUE-A, PUB2 with queue QUEUE-A, PUB3 with QUEUE-B, and SUB with AddPublisher to QUEUE-A, so the SUB does not receive MyEvent from PUB3 (QUEUE-B).
I'm using:
NServiceBus 6.0.0-beta0004
NServiceBus.Persistence.AzureStorage 1.0.0-beta0004
NServiceBus.Azure.Transports.WindowsAzureStorageQueues 7.0.0-beta0004
Azure Storage Queues transport supports pub/sub using persistence. The need to specify the publisher endpoint is there to allow sending a subscription message by the subscriber upon startup. By default, all endpoints use the same shared storage table and that's why you experience what you describe. If you'd split subscriptions per endpoint (each endpoint to have its own storage table), you'd see that SUB would only receive an event from PUB1 if that's the only publisher it's subscribed to.
AddPublisher() is not an obsolete method. Obsoleted message would be marked as such. Saying that, the routing feature still can be modified during the beta stage we're in right now.
Scoping can be done the way Philip outlined. At the same time, I would encourage to look into why you have two different endpoints generating the same event. Usually, you'd want to have an event be unique and raised by a single endpoint (or, all of it instances), but not two or more different endpoints.
Each Publisher needs its own Persistence Table, in addition to its own queue.
When you setup each publisher's IEndpointInstance:
busConfiguration
.UsePersistence<AzureStoragePersistence, StorageType.Subscriptions>()
.TableName("NameOfPublisher")
See the documentation for more info: http://docs.particular.net/nservicebus/azure-storage-persistence/configuration
Related
Requirement
A system undergoes some state change, and multiple other parts of the system has to know this(lets call them observers) so that they can perform some actions based on the current state, the actions of the observers are important, if some of the observers are not online(not listening currently due to some trouble, but will be back soon), the message should not be discarded till all the observers gets the message.
Trying to accomplish this with pub/sub model, here are my findings, (please correct if this understanding is wrong) -
The publisher creates an event on specific topic, and multiple subscribers can consume the same message. This model either provides no delivery guarantee(in redis), or delivery is guaranteed once(with messaging queues), ie. when one of the consumer acknowledges a message, the message is discarded(rabbitmq).
Example
A new Person Profile entity gets created in DB
Now,
A background verification service has to know this to trigger the verification process.
Subscriptions service has to know this to add default subscriptions to the user.
Now both the tasks are important, unrelated and can run in parallel.
Now In Queue model, if subscription service is down for some reason, a BG verification process acknowledges the message, the message will be removed from the queue, or if it is fire and forget like most of pub/sub, the delivery is anyhow not guaranteed for both the services.
One more point is both the tasks are unrelated and need not be triggered one after other.
In short, my need is to make sure all the consumers gets the same message and they should be able to acknowledge them individually, the message should be evicted only after all the consumers acknowledged it either of the above approaches doesn't do this.
Anything I am missing here ? How should I approach this problem ?
This scenario is explicitly supported by RabbitMQ's model, which separates "exchanges" from "queues":
A publisher always sends a message to an "exchange", which is just a stateless routing address; it doesn't need to know what queue(s) the message should end up in
A consumer always reads messages from a "queue", which contains its own copy of messages, regardless of where they originated
Multiple consumers can subscribe to the same queue, and each message will be delivered to exactly one consumer
Crucially, an exchange can route the same message to multiple queues, and each will receive a copy of the message
The key thing to understand here is that while we talk about consumers "subscribing" to a queue, the "subscription" part of a "pub-sub" setup is actually the routing from the exchange to the queue.
So a RabbitMQ pub-sub system might look like this:
A new Person Profile entity gets created in DB
This event is published as a message to an "events" topic exchange with a routing key of "entity.profile.created"
The exchange routes copies of the message to multiple queues:
A "verification_service" queue has been bound to this exchange to receive a copy of all messages matching "entity.profile.#"
A "subscription_setup_service" queue has been bound to this exchange to receive a copy of all messages matching "entity.profile.created"
The consuming scripts don't know anything about this routing, they just know that messages will appear in the queue for events that are relevant to them:
The verification service picks up the copy of the message on the "verification_service" queue, processes, and acknowledges it
The subscription setup service picks up the copy of the message on the "subscription_setup_service" queue, processes, and acknowledges it
If there are multiple consuming scripts looking at the same queue, they'll share the messages on that queue between them, but still completely independent of any other queue.
Here's a screenshot from this interactive visualisation tool that shows this scenario:
As you mentioned it is not something that you can control with Redis Pub/Sub data structure.
But you can do it easily with Redis Streams.
Streams will allow you to post messages using the XADD command and then control which consumers are dealing with the message and acknowledge that message has been processed.
You can look at these sample application that provides (in Java) example about:
posting and consuming messages
create multiple consumer groups
manage exceptions
Links:
Getting Started with Redis Streams and Java
Redis Streams in Action ( Project that shows how to use ADD/ACK/PENDING/CLAIM and build an error proof streaming application with Redis Streams and SpringData )
While looking at the Pub/Sub pattern, i came across the fellowing scenario:
Assume that you have a horizontally scaled app, that has X instances. All of them subscribe to a topic where messages like "Transfer $10 from account A to account B". When someone publish a message to that topic, all subscriber will get that message?
In the case above, clearly, the message should be taken by only 1 subscriber and handled only once.
How does one handle this scenario? Do you abandon the pub/sub and starts pooling?
Let me explain few things with example before you understand that completely. I have worked on Azure service bus so i will explain in that context.
In Pub/sub you have one topic and possible multiple subscription. Lets say we have topic "Shopping-Topic". We have 2 Subscriptions called "Payment-Subscription", "Cart-Subscription". Now we publish message "Payment-processed" on the topic. It's the discretion of subscription to pick that message and reason is that subscription have to mention that which messages it want pick.
In Azure service bus we have something called rule (message label). Default rule is that subscription is listening to all the messages but we can overwrite this behavior and say i am only interested in particular message. In the above case rule added against "Payment-Subscription" to listen the message "Payment-processed" so the message is added to "Payment-Subscription" subscription for it to process. Even though "Cart-Subscription" is also subscribed to the same topic but it is ignoring this message so it's not added to its subscription. This way any intended subscription can listen to particular message not necessarily all of them.
Now we discuss individual subscription. Let's say we have message added to "Payment-Subscription". This subscription has 2 instances/processes that are ready to process the message "Payment-processed". The first process to pick the message will process the message and remove it from subscription.
In RabbitMQ Normally, active consumers connected to the same queue receive messages from it in a round-robin fashion. So this insures that a message is processed exactly once.
So in your case you should design a queue where all the messages for
"Transfer $10 from account A to account B"
Are routed to and all the consumers register themselves on this queue itself , this insures that one message will go to only one subscriber.
Another point not related to your question but is important to know is that there is another concept called "Consumer Priorities" which allows you to ensure that high priority consumers receive messages while they are active, with messages only going to lower priority consumers when the high priority consumers block.
More info can be found here
In our scenario I'm thinking of using the pub sub technique. However I don't know which is the better option.
1 ########
A web service of ours will publish a message that something has happened when it is called externally, ExternalPersonCreatedMessage!
This message will contain a field that represents the destinations to process the message into (multiple allowed).
Various subscribers will subscribe. These subscribers will filter the message to see if any action is required by checking the destination field.
2 ########
A web service of ours will parse the incoming call and publish specific types of messages depending on the destinations supplied in the field. i.e. many Destination[n]PersonCreatedMessage messages would be created.
Subscribers will subscribe to only the specific message they care for. i.e. not having to filter any messages
QUESTIONS
Which of the above is the better option and why? And how do I stop myself from making RequestMessages. From what I've read/seen I should be trying to structure this in a way of PersonCreated, PersonDeleted i.e. SOMETHING HAS HAPPENED and NOT in the REQUEST SOMETHING TO HAPPEN form such as CreatePerson or DeletePerson
Are my thoughts correct? I've been looking for guidance on how to structure messages and making sure I don't go down a wrong path but have found no guidance out there on do's and dont's. Can any one help and guide? I want to try and get this correct from the off :)
Based on the integration scenario in the referenced article, it appears to me that you may need a Saga to complete the workflow of accept message -> operate on message -> send confirmation. In the case that the confirmation is sent immediately after the operation, you could use NSBs message handler pipeline feature which allows you to chain handlers in a specified sequence such as...
First<FilterHandler>.Then<DoWorkHandler>().AndThen<SendConfirmationHandler>();
In terms of the content filtering, you can do this although you incur some transport overhead, meaning the queue will have to accept the message and the process will always call the first handler on every message(you can short-circuit the above pipeline at any point). It may be the case that what you really want is a Distributor/Worker setup where all Workers are the same and you can handle some load.
If you truly have different endpoints with completely different logic, then I would have the Publisher process(only accepts and Publishes message) do the work of translating the inbound message to something else a Subscriber can then be interested in. If then you find that a given Published message only ever has 1 Subscriber, then you don't need to Publish at all, you need to just Bus.Send() to the correct endpoint.
The way NServiceBus handles pub-sub is more like your option two.
A publisher service has an input queue and a subscription store.
A subscriber service has an input queue
The subscriber, on start-up will send a subscription message to the input queue of the publisher
The subscription message contains the type of message subscriber is interested in and the subscribers queue address
The publisher records the subscription in the subscription store.
The publisher receives a message.
The publisher evaluates the message type against the list of subscriptions
For each match found the publisher sends the message to the queue address.
In my opinion, you should stop thinking about destinations. Messages are messages. They should not have any inherent destination information in them. The subscription mechanism defines the addressing/routing requirements for the solution.
I am getting a little confused with NServiceBus. It seems like a lot of examples that I see, they always use publish() and subscribe(). What I am trying to do is that I have a publisher that polling from its queue and distributes the message to subscriber’s queue. The messages are being generated by other application and the body of message will contain a text, which will be parsed later.
Do I still need to call publish() and subsribe() to transfer the messages from publisher's queue to subscriber's queue? The way I understood was that I only need to configure the queue names in both config file and call LoadAllMessages() on subscriber side, will take above scenario. I don't even have to handle the message on the subscriber side.
Thanks.
Your Publisher will still need to call Publish. What this does is the Publisher then looks into Subscription Storage to find out who is interested in that message type. It then will send a message to each Subscriber. On the Subscriber side you need to implement message handlers to do something with those messages. This is done via implementing the IHandleMessages<T> interface in the Subscriber assembly. NSB will discover this and autowire everything up. Be aware by default, the Subscriber will subscriber to all message types. If you want to only subscribe to certain messages, use the .DoNotAutoSubscribe setting in the manual configuration.
I wonder if it’s possible using nServiceBus to subscribe to all Messages of a Type without specifying the publisher’s end point.
The Background for this, is a distributed algorithm, that uses the distributor infra structure of nServiceBus to delegate sub problems to distributed workers on the network.
After a task is finished, the worker should send a message to notifying the sender.
I could use IBus.Reply() to notify it but I have also some monitoring and logging services, which are also interested in those messages. Making the sender republish all received replied doesn’t sound right.
Can I subscribe to a message from multiple publisher in nServiceBus?
You're exactly right to use Reply - simple and works.
In order to do logging/monitoring, you can audit messages so each endpoint forwards the messages it receives.