The current advice on handling exceptions in NServiceBus is to use the built in facilities. Errored out messages go to the error message queue, and a log is written to disk.
But what if I want to send my errors to a service like AirBrake which has better functionality for grouping similar exceptions, metrics, and other good stuff? Is there a global exception handler I can tap into?
As mentioned in the original post the recommended solution is to use ServicePulse for monitoring errors. The client I currently work for is using a custom made centralized logger, and we want NServiceBus to log to this log store when messages are forwarded to the error queue.
We could have achieved this by just editing the log4net config if the exception from NServiceBus had included the original exception, currently NServiceBus just logs a generic error message with no details about what caused the failure.
NServiceBus has a class named NServiceBus.Faults.ErrorsNotifications which contains the following observables:
MessageSentToErrorQueue
MessageHasFailedAFirstLevelRetryAttempt
MessageHasBeenSentToSecondLevelRetries
You can subscribe to these observables when the endpoint starts, like in the following example which logs an error when messages are sent to the error queue:
public class GlobalErrorHandler : IWantToRunWhenBusStartsAndStops
{
private readonly ILogger _logger;
private readonly BusNotifications _busNotifications;
readonly List<IDisposable> _notificationSubscriptions = new List<IDisposable>();
public GlobalErrorHandler(ILogger logger, BusNotifications busNotifications)
{
_logger = logger;
_busNotifications = busNotifications;
}
public void Start()
{
_notificationSubscriptions.Add(_busNotifications.Errors.MessageSentToErrorQueue.Subscribe(LogWhenMessageSentToErrorQueue));
}
public void Stop()
{
foreach (var subscription in _notificationSubscriptions)
{
subscription.Dispose();
}
}
private void LogWhenMessageSentToErrorQueue(FailedMessage message)
{
var properties = new
{
MessageType = message.Headers["NServiceBus.EnclosedMessageTypes"],
MessageId = message.Headers["NServiceBus.MessageId"],
OriginatingMachine = message.Headers["NServiceBus.OriginatingMachine"],
OriginatingEndpoint = message.Headers["NServiceBus.OriginatingEndpoint"],
ExceptionType = message.Headers["NServiceBus.ExceptionInfo.ExceptionType"],
ExceptionMessage = message.Headers["NServiceBus.ExceptionInfo.Message"],
ExceptionSource = message.Headers["NServiceBus.ExceptionInfo.Source"],
TimeSent = message.Headers["NServiceBus.TimeSent"]
};
_logger.Error("Message sent to error queue. " + properties, message.Exception);
}
}
The observable is implemented by using Reactive Extensions, so you will have to install the NuGet package Rx-Core for this to work.
Could you maybe use a custom log4net configuration to do it?
http://help.airbrake.io/discussions/suggestions/157-net-plug-in-for-api-v2
There is an interface in NServiceBus called IManageMessageFailures, but I don't believe you can use it in conjucntion with the Second Level Retries so you would probably lose that if you did go for your own.
You could write the errors to SQL Server with log4net and the from there you could forward them to AirBrake using their API marking each error in the error table as sent afterwards maybe?
We switched to Serilog, it has perfect logging ans tracing support for NServiceBus. Creating a custom Serilog sink that will send your log events wherever you want is quite easy.
I would suggest that you create a custom endpoint that feeds off the error queue and uploads the data to AirBrake.
Side note: v4.0|4.1 will come with a nice Rest-Api that gives you nice management/querying capabilities for the errors. Tools like the Profiler and Ops will use this api to present the data to you in various ways.
http://particular.net/service-platform
Related
i use #restcontrolleradvice and #ExceptionHandler , but i can handle controller exception. server error like 404, 500 can't handle.
#RestControllerAdvice
public class HttpExceptionHandler {
private static final Logger logger = LoggerFactory.getLogger(HttpExceptionHandler.class);
#ExceptionHandler(value = Exception.class)
public String exceptions(Exception e) {
String code = Global.ERR_UNKNOWN;
if (e instanceof MethodNotAllowedException) {
code = Global.ERR_HTTP_METHOD;
}
return code;
}
}
If you're using Spring Boot, this is already done for you and you can customize this support as well quite easily (see Spring Boot reference docs).
If you're using plain Spring Framework, then you need to register a custom WebExceptionHandler bean to handle that (see Spring Framework reference docs). Because those errors can happen at any point during request handling (i.e. not only during the controller handling phase, but also during response encoding, within a WebFilter...), the API there is quite low level and you need to deal with raw DataBuffer instances. If you're looking for inspiration on how to achieve higher level error handling support, you can also take a look at what's done in Spring Boot.
I hate the questions that have "Not Enough Info". So I will try to give detailed information. And in this case it is code.
Server:
64 bit of https://github.com/MSOpenTech/redis/tree/2.6/bin/release
There are three classes:
DbOperationContext.cs: https://gist.github.com/glikoz/7119628
PerRequestLifeTimeManager.cs: https://gist.github.com/glikoz/7119699
RedisRepository.cs https://gist.github.com/glikoz/7119769
We are using Redis with Unity ..
In this case we are getting this strange message:
"Redis Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool. This may have occurred because all pooled connections were in use.";
We checked these:
Is the problem configuration issue
Are we using wrong RedisServer.exe
Is there any architectural problem
Any idea? Any similar story?
Thanks.
Extra Info 1
There is no rejected connection issue on server stats (I've checked it via redis-cli.exe info command)
I have continued to debug this problem, and have fixed numerous things on my platform to avoid this exception. Here is what I have done to solve the issue:
Executive summary:
People encountering this exception should check:
That the PooledRedisClientsManager (IRedisClientsManager) is registed in a singleton scope
That the RedisMqServer (IMessageService) is registered in a singleton scope
That any utilized RedisClient returned from either of the above is properly disposed of, to ensure that the pooled clients are not left stale.
The solution to my problem:
First of all, this exception is thrown by the PooledRedisClient because it has no more pooled connections available.
I'm registering all the required Redis stuff in the StructureMap IoC container (not unity as in the author's case). Thanks to this post I was reminded that the PooledRedisClientManager should be a singleton - I also decided to register the RedisMqServer as a singleton:
ObjectFactory.Configure(x =>
{
// register the message queue stuff as Singletons in this AppDomain
x.For<IRedisClientsManager>()
.Singleton()
.Use(BuildRedisClientsManager);
x.For<IMessageService>()
.Singleton()
.Use<RedisMqServer>()
.Ctor<IRedisClientsManager>().Is(i => i.GetInstance<IRedisClientsManager>())
.Ctor<int>("retryCount").Is(2)
.Ctor<TimeSpan?>().Is(TimeSpan.FromSeconds(5));
// Retrieve a new message factory from the singleton IMessageService
x.For<IMessageFactory>()
.Use(i => i.GetInstance<IMessageService>().MessageFactory);
});
My "BuildRedisClientManager" function looks like this:
private static IRedisClientsManager BuildRedisClientsManager()
{
var appSettings = new AppSettings();
var redisClients = appSettings.Get("redis-servers", "redis.local:6379").Split(',');
var redisFactory = new PooledRedisClientManager(redisClients);
redisFactory.ConnectTimeout = 5;
redisFactory.IdleTimeOutSecs = 30;
redisFactory.PoolTimeout = 3;
return redisFactory;
}
Then, when it comes to producing messages it's very important that the utilized RedisClient is properly disposed of, otherwise we run into the dreaded "Timeout Expired" (thanks to this post). I have the following helper code to send a message to the queue:
public static void PublishMessage<T>(T msg)
{
try
{
using (var producer = GetMessageProducer())
{
producer.Publish<T>(msg);
}
}
catch (Exception ex)
{
// TODO: Log or whatever... I'm not throwing to avoid showing users that we have a broken MQ
}
}
private static IMessageQueueClient GetMessageProducer()
{
var producer = ObjectFactory.GetInstance<IMessageService>() as RedisMqServer;
var client = producer.CreateMessageQueueClient();
return client;
}
I hope this helps solve your issue too.
We are implementing a saga that calls out to other services with NServiceBus. I'm not quite clear about how NServiceBus deals with exceptions inside a saga.
Inside the saga we have a handler, and that handler calls an external service that should only be called once the original message handler completes succesfully. Is it okay to do:
public void Handle(IFooMessage message)
{
var message = Bus.CreateInstance<ExternalService.IBarMessage>();
Bus.Send(message);
// something bad happens here, exception is thrown
}
or will the message be sent to ExternalService multiple times? Someone here has suggested changing it to:
// handler in the saga
public void Handle(IFooMessage message)
{
// Do something
var message = Bus.CreateInstance<ISendBarMessage>();
Bus.SendLocal(message);
// something bad happens, exception is thrown
}
// a service-level handler
public void Handle(ISendBarMessage message)
{
var message = Bus.CreateInstance<ExternalService.IBarMessage>();
Bus.Send(message);
}
I've done an experiment and from what I can tell the first method seems fine, but I can't find any documentation other than http://docs.particular.net/nservicebus/errors/ which says:
When an exception bubbles through to the NServiceBus infrastructure, it rolls back the transaction on a transactional endpoint, causing the message to be returned to the queue, and any messages that user code tried to send or publish to be undone as well.
Any help to clarify this point would be much appreciated.
As long as you're doing messaging from your saga and not doing any web service calls, then you're safe - no need to do SendLocal.
I'm getting started with NServiceBus and have a question about the Pubsub sample.
My intention was to have multiple instances of Publisher1 running and receiving the message sent from the publisher. I also hacked the Publisher to only send messages of the eventMessage type.
But if I start the publisher and three instances of Subscriber1, only one of them gets the message at a time.
why is that? Is it a config setting or something else?
This is what I've tried which returns an exception "Exception when starting endpoint, error has been logged. Reason: Cannot configure property before the component has been configured. Please call 'Configure' first.":
using NServiceBus;
namespace Subscriber1
{
public class EndpointConfig : IConfigureThisEndpoint, AsA_Server
{
}
public class OverrideInputQueue : IWantCustomInitialization
{
public void Init()
{
Configure
.Instance
.Configurer
.ConfigureProperty<NServiceBus.Config.MsmqTransportConfig>(t => t.InputQueue, "testQueue");
}
}
}
/J
NServiceBus assumes that you have one input queue per process. Make sure that each of your subscribers are configured with a unique input queue. If not all three will be polling the same queue producing the behavior you're describing.
To do this you would probably have to copy paste sub1 to 3 different folders, modfying the app.config and start them up.
Hope this helps!
You should use this-
Configure.Instance.Configurer.ConfigureProperty<NServiceBus.Unicast.Transport.Msmq.MsmqTransport>(msmq => msmq.InputQueue, "SomeQueueHere");
Make sure you use MsmqTransport and not MsmqTransportConfig as you mentioned.
The backgound: I am trying to forward the server-side ApplyChangeFailed event that is fired by a Sync Services for ADO 1.0 DBServerSyncProvider to the client. All the code examples for Sync Services conflict resolution do not use WCF, and when the client connects to the server database directly, this problem does not exist. My DBServerSyncProvider is wrapped by a head-less WCF service, however, and I cannot show the user a dialog with the offending data for review.
So, the obvious solution seemed to be to convert the HTTP WCF service that Sync Services generated to TCP, make it a duplex connection, and define a callback handler on the client that receives the SyncConflict object and sets the Action property of the event.
When I did that, I got a runtime error (before the callback was attempted):
System.InvalidOperationException: This operation would deadlock because the
reply cannot be received until the current Message completes processing. If
you want to allow out-of-order message processing, specify ConcurrencyMode of
Reentrant or Multiple on CallbackBehaviorAttribute.
So I did what the message suggested and decorated both the service and the callback behavior with the Multiple attribute. Then the runtime error went away, but the call results in a "deadlock" and never returns. What do I do to get around this? Is it not possible to have a WCF service that calls back the client before the original service call returns?
Edit: I think this could be the explanation of the issue, but I am still not sure what the correct solution should be.
After updating the ConcurrencyMode have you tried firing the callback in a seperate thread?
This answer to another question has some example code that starts another thread and passes through the callback, you might be able to modify that design for your purpose?
By starting the sync agent in a separate thread on the client, the callback works just fine:
private int kickOffSyncInSeparateThread()
{
SyncRunner syncRunner = new SyncRunner();
Thread syncThread = new Thread(
new ThreadStart(syncRunner.RunSyncInThread));
try
{
syncThread.Start();
}
catch (ThreadStateException ex)
{
Console.WriteLine(ex);
return 1;
}
catch (ThreadInterruptedException ex)
{
Console.WriteLine(ex);
return 2;
}
return 0;
}
And this is my SyncRunner:
class SyncRunner
{
public void RunSyncInThread()
{
MysyncAgent = new MySyncAgent();
syncAgent.addUserIdParameter("56623239-d855-de11-8e97-0016cfe25fa3");
Microsoft.Synchronization.Data.SyncStatistics syncStats =
syncAgent.Synchronize();
}
}