I am investigating NServiceBus and I am unsure how (or even if) I could use it to handle this scenario:
I have multiple clients sending work requests, which the distributor farms out to workers. The work will take a long time to complete and I would like the workers to report progress back to the client that sent the original request.
I have looked at the full duplex sample and also how to add the distributor to that sample. I've got these working, but when I modify them to reply with a series of progress messages (with a delay between the messages, as per code shown below), the client receives all the progress messages at the same time.
public class RequestDataMessageHandler : IHandleMessages<RequestDataMessage>
{
public IBus Bus { get; set; }
public void Handle(RequestDataMessage message)
{
for (var i = 0; i < 10; i++)
{
var count = i;
var response = this.Bus.CreateInstance<DataResponseMessage>(m =>
{
m.DataId = message.DataId;
m.Progress = count * 10;
});
this.Bus.Reply(response);
Thread.Sleep(1000);
}
}
}
I suspect I've not understood something basic about how NServiceBus works. Could someone explain where I've gone wrong, or point me at some examples and/or documentation?
What you have constructed will always send the messages as part of the same transaction. Since there is one transaction per handler, you won't be able to communicate progress this way. You would have to have a separate endpoint for each chunk of processing that would communicate progress. We've implemented communicating progress by updating something externally that is not involved in the transaction. That could be done by sending a non-transactional message to another endpoint to update progress or something like an RPC call. From there you could have something poll that progress data store.
Have your workers use bus.Reply() to send messages back to your clients. Reply will automatically send the message to the endpoint that sent the original message
Related
I have a situation where I have an asp.net core application which registers a subscription client to a topic on startup (IHostedService), this subscription client essentially has a dictionary of callbacks that need to be fired whenever it detects a new message in a topic with an id (this id is stored on the message properties). This dictionary lives throughout the lifetime of the application, and is in memory.
Everything works fine on a single instance of the asp.net core app service on azure, as soon as I scale up to 2, I notice that sometimes the callbacks in the subscription are not firing. This makes sense, as we have two instances now, each with its own dictionary store of callbacks.
So I updated the code to check if the id of the subscription exists, if not, abandon message, if yes, get the callback and invoke it.
public async Task HandleMessage(Microsoft.Azure.ServiceBus.Message message, CancellationToken cancellationToken)
{
var queueItem = this.converter.DeserializeItem(message);
var sessionId = // get the session id from the message
if (string.IsNullOrEmpty(sessionId))
{
await this.subscriptionClient.AbandonAsync(message.SystemProperties.LockToken);
return;
}
if (!this.subscriptions.TryGetValue(sessionId, out var subscription))
{
await this.subscriptionClient.AbandonAsync(message.SystemProperties.LockToken);
return;
}
await subscription.Call(queueItem);
// subscription was found and executed. Complete message
await this.subscriptionClient.CompleteAsync(message.SystemProperties.LockToken);
}
However, the problem still occurs. My only guess is that when calling AbandonAsync, the same instance is picking up the message again?
I guess what I am really trying to ask is, if I have multiple instances of a topic subscription client all pointing to the same subscriber for the topic, is it possible for all the instances to get a copy of the message? Or is that not guaranteed.
if I have multiple instances of a topic subscription client all pointing to the same subscriber for the topic, is it possible for all the instances to get a copy of the message? Or is that not guaranteed.
No. If it's the same subscription all clients are pointing to, only one will be receiving that message.
You're running into an issue of scaling out with competing consumers. If you're scaling out, you never know what instance will pick the message. And since your state is local (in memory of each instance), this will fail from time to time. Additional downside is the cost. By fetching messages on the "wrong" instance and abandoning, you're going to pay higher cost on the messaging side.
To address this issue you either need to have a shared/centralized or change your architecture around this.
I managed to solve the issue by making use of service bus sessions. What I was trying to do with the dictionary of callbacks is basically a session manager anyway!
Service bus sessions allow me to have multiple instances of a session client all pointing to the same subscription. However, each instance will only know or care about the sessions it is currently dealing with.
i have a azure worker role with an nservicebus host 4.7.5 . This host sends events on azure servicebus transport and to a topic. Is there a way to either delay the sending of the event or setting some properties to make sure that the message appears after a delay on the topic subscription? The host sends out events after it notices a change in the primary database. There are several secondary databases into which the primary data write is replicated. The receivers are also azure worker roles that use nservicebus host and have subscription to the topics.
By the time the receivers receive the message, due to replication lag the secondaries may have out of sync data.
one option is to use primary database to read but that is a route which i dont want to take.
Would it be possible to fail-early in your subscription endpoints and let the retries take care of it? You can fine-tune the retry times/delays to make sure your secondary databases are updated before the message is retried.
You still need to find the best way to lookup your data from the database and a way to differentiate between the version in the event. You could use version numbers or last update dates in case of updates, or just lookup by an identifier in case of creation.
The endpoint reading data off the secondary database might have a event handler like this:
public class CustomerCreationHandler : IHandlesMessage<CustomerCreated>
{
public void Handle(CustomerCreated #event)
{
var customer = Database.Load(#event.CustomerId);
if(customer == null)
{
throw new CustomerNotFoundException("Customer was not found.");
}
//Your business logic goes here
}
}
You can control how many times the event handler will retry and how much delay there'll be between each attempt. In this case, the message will be retried by First-Level retries and then handed over to Second-Level retries which is configured below.
class ProvideConfiguration :
IProvideConfiguration<SecondLevelRetriesConfig>
{
public SecondLevelRetriesConfig GetConfiguration()
{
return new SecondLevelRetriesConfig
{
Enabled = true,
NumberOfRetries = 2,
TimeIncrease = TimeSpan.FromSeconds(10)
};
}
}
Alternatively, instead of just publishing the event, you can send a deferred message to the same endpoint to then publish the actual event after certain amount of time is passed.
Let's just accept for a moment that it is not a horrible idea to implement RPC over message queues (like RabbitMQ) -- sometimes it might be necessary when interfacing with legacy systems.
In case of RPC over RabbitMQ, clients send a message to the broker, broker routes the message to a worker, worker returns the result through the broker to the client. However, if a worker implements more than one remote method, then somehow the different calls need to be routed to different listeners.
What is the general practice in this case? All RPC over MQ examples show only one remote method. It would be nice and easy to just set the method name as the routing rule/queue name, but I don't know whether this is the right way to do it.
Let's just accept for a moment that it is not a horrible idea to implement RPC over message queues (like RabbitMQ)
it's not horrible at all! it's common, and recommended in many situations - not just legacy integration.
... ok, to your actual question now :)
from a very high level perspective, here is what you need to do.
Your request and response need to have two key pieces of information:
a correlation-id
a reply-to queue
These bits of information will allow you to correlate the original request and the response.
Before you send the request
have your requesting code create an exclusive queue for itself. This queue will be used to receive the replies.
create a new correlation id - typically a GUID or UUID to guarantee uniqueness.
When Sending The Request
Attach the correlation id that you generated, to the message properties. there is a correlationId property that you should use for this.
store the correlation id with the associated callback function (reply handler) for the request, somewhere inside of the code that is making the request. you will need to this when the reply comes in.
attach the name of the exclusive queue that you created, to the replyTo property of the message, as well.
with all this done, you can send the message across rabbitmq
when replying
the reply code needs to use both the correlationId and the replyTo fields from the original message. so be sure to grab those
the reply should be sent directly to the replyTo queue. don't use standard publishing through an exchange. instead, send the reply message directly to the queue using the "send to queue" feature of whatever library you're using, and send the response directly to the replyTo queue.
be sure to include the correlationId in the response, as well. this is the critical part to answer your question
when handling the reply
The code that made the original request will receive the message from the replyTo queue. it will then pull the correlationId out of the message properties.
use the correlation id to look up the callback method for the request... the code that handles the response. pass the message to this callback method, and you're pretty much done.
the implementation details
this works, from a high level perspective. when you get down into the code, the implementation details will vary depending on the language and driver / library you are using.
most of the good RabbitMQ libraries for any given language will have Request/Response built in to them. If yours doesn't, you might want to look for a different library. Unless you are writing a patterns based library on top of the AMQP protocol, you should look for a library that has common patterns implemented for you.
If you need more information on the Request/Reply pattern, including all of the details that I've provided here (and more), check out these resources:
My own RabbitMQ Patterns email course / ebook
RabbitMQ Tutorials
Enterprise Integration Patterns - be sure to buy the book for the complete description / implementation pattern. it's worth having this book
If you're working in Node.js, I recommend using the wascally library, which includes the Request/Reply feature you need. For Ruby, check out bunny. For Java or .NET, look at some of the many service bus implementations around. In .NET, I recommend NServiceBus or MassTransit.
I've found that using a new reply-to queue per request can get really inefficient, specially when running RabbitMQ on a cluster.
As suggested in the comments direct reply-to seems to be the way to go. I've documented here all the options I tried before settling to that one.
I wrote an npm package amq.rabbitmq.reply-to.js that:
Uses direct reply-to - a feature that allows RPC (request/reply) clients with a design similar to that demonstrated in tutorial 6 (https://www.rabbitmq.com/direct-reply-to.html) to avoid declaring a response queue per request.
Creates an event emitter where rpc responses will be published by correlationId
as suggested by https://github.com/squaremo/amqp.node/issues/259#issuecomment-230165144
Usage:
const rabbitmqreplyto = require('amq.rabbitmq.reply-to.js');
const serverCallbackTimesTen = (message, rpcServer) => {
const n = parseInt(message);
return Promise.resolve(`${n * 10}`);
};
let rpcServer;
let rpcClient;
Promise.resolve().then(() => {
const serverOptions = new rabbitmqreplyto.RpcServerOptions(
/* url */ undefined,
/* serverId */ undefined,
/* callback */ serverCallbackTimesTen);
return rabbitmqreplyto.RpcServer.Create(serverOptions);
}).then((rpcServerP) => {
rpcServer = rpcServerP;
return rabbitmqreplyto.RpcClient.Create();
}).then((rpcClientP) => {
rpcClient = rpcClientP;
const promises = [];
for (let i = 1; i <= 20; i++) {
promises.push(rpcClient.sendRPCMessage(`${i}`));
}
return Promise.all(promises);
}).then((replies) => {
console.log(replies);
return Promise.all([rpcServer.Close(), rpcClient.Close()]);
});
//['10',
// '20',
// '30',
// '40',
// '50',
// '60',
// '70',
// '80',
// '90',
// '100',
// '110',
// '120',
// '130',
// '140',
// '150',
// '160',
// '170',
// '180',
// '190',
// '200']
My requirement is to make the Subscriber pause processing the messages depending on whether a web service is up or not. So, when the web service is down, the messages should keep coming to the subscriber queue from Publisher and keep piling up until the web service is up again. (These messages should not go to the error queue, but stay on the Subscriber queue.)
I tried to use unsubscribe, but the publisher stops sending messages as the unsubscribe seems to clear the subscription info on RavenDB. I have also tried setting the MaxConcurrencyLevel on the Transport class, if I set the worker threads to 0, the messages coming to Subscriber go directly to the error queue. Finally, I tried Defer, which seems to put the current message in audit queue and creates a clone of the message and sends it locally to the subscriber queue when the timeout is completed. Also, since I have to keep checking the status of service and keep defering, I cannot control the order of messages as I cannot predict when the web service will be up.
What is the best way to achieve the behavior I have explained? I am using NServiceBus version 4.5.
It sounds like you want to keep trying to handle a message until it succeeds, and not shuffle it back in the queue (keep it at the top and keep trying it)?
I think your only pure-NSB option is to tinker with the MaxRetries setting, which controls First Level Retries: http://docs.particular.net/nservicebus/msmqtransportconfig. Setting MaxRetries to a very high number may do what you are looking for, but I can't imagine doing so would be a good practice.
Second Level Retries will defer the message for a configurable amount of time, but IIRC will allow other messages to be handled from the main queue.
I think your best option is to put retry logic into your own code. So the handler can try to access the service x number of times in a loop (maybe on a delay) before it throws an exception and NSB's retry features kick in.
Edit:
Your requirement seems to be something like:
"When an MyEvent comes in, I need to make a webservice call. If the webservice is down, I need to keep trying X number of times at Y intervals, at which point I will consider it a failure and handle a failure condition. Until I either succeed or fail, I will block other messages from being handled."
You have some potentially complex logic on handling a message (retry, timeout, error condition, blocking additional messages, etc.). Keep in mind the role that NSB is intended to play in your system: communication between services via messaging. While NSB does have some advanced features that allow message orchestration (e.g. sagas), it's not really intended to be used to replace Domain or Application logic.
Bottom line, you may need to write custom code to handle your specific scenario. A naive solution would be a loop with a delay in your handler, but you may need to create a more robust in-memory collection/queue that holds messages while the service is down and processes them serially when it comes back up.
The easiest way to achieve somewhat the required behavior is the following:
Define a message handler which checks whether the service is available and if not calls HandleCurrentMessageLater and a message handler which does the actual message processing. Then you specify the message handler order so that the handler which checks the service availability gets executed first.
public interface ISomeCommand {}
public class ServiceAvailabilityChecker : IHandleMessages<ISomeCommand>{
public IBus Bus { get; set; }
public void Handle(ISomeCommand message) {
try {
// check service
}
catch(SpecificException ex) {
this.Bus.HandleCurrentMessageLater();
}
}
}
public class ActualHandler : IHandleMessages<ISomeCommand>{
public void Handle(ISomeCommand message) {
}
}
public class SomeCommandHandlerOrdering : ISpecifyMessageHandlerOrdering{
public void SpecifyOrder(Order order){
order.Specify(First<ServiceAvailabilityChecker>.Then<ActualHandler>());
}
}
With that design you gain the following:
You can check the availability before the actual business code is invoked
If the service is not available the message is put back into the queue
If the service is available and your business code gets invoked but just before the ActualHandler is invoked the service becomes unavailable you get First and Second Level retries (and again the availability check in the pipeline)
I am using request reply model of NServiceBUs for one of my project.There is a self hosted service bus listening for a request message and reply back with a request message.
So in WCF message code i have coded like
// sent the message to bus.
var synchronousMessageSent =
this._bus.Send(destinationQueueName, requestMessage)
.Register(
(AsyncCallback)delegate(IAsyncResult ar)
{
// process the response from the message.
NServiceBus.CompletionResult completionResult = ar.AsyncState as NServiceBus.CompletionResult;
if (completionResult != null)
{
// set the response messages.
response = completionResult.Messages;
}
},
null);
// block the current thread.
synchronousMessageSent.AsyncWaitHandle.WaitOne(1000);
return response;
The destinaton que will sent the reply.
I am getting the resault one or tweo times afetr that the reply is not coming to the client. Am i doing anything wrong
Why are you trying to turn an a-synchronous framework into a synchronous one? There is a fundamental flaw with what you are trying to do.
You should take a long hard look at your design and appreciate the benefits of a-sync calls. The fact that you are doing
// block the current thread.
synchronousMessageSent.AsyncWaitHandle.WaitOne(1000);
Is highly concerning. What are you trying to achieve with this? Design your system based on a-synchronous messaging communication and you will have a MUCH better system. Otherwise you might as well just use some kind of blocking tcp/ip sockets.