I have WCF services hosted as http endpoints in IIS. At some points in the handling of command requests to these services, I want to publish an event on NServiceBus to indicate what command was processed. Is it possible for me to do that within a TransactionScope that I create and manage in my code (so that I can include my database interactions in the same transaction)? I would want to be able to publish from the bus within that scope, such that the publication actually goes through if the scope is completed. When I run the following:
using (var scope = new TransactionScope()) {
bus.Publish(new SomethingHappened { Description = String.Format("{0} logged in at {1}", user.Name, DateTime.Now.ToLongTimeString()) }).ConfigureAwait(false).GetAwaiter().GetResult();
}
I find that the message is received by subscribers, even though I did not call scope.Complete() on the transaction scope. What, if anything, can I change so that the publish is enrolled in the scope?
It turns out that in order for the transaction scope to manage the async call, you need to open it with the special option TransactionScopeAsyncFlowOption.Enabled like so:
using (var scope = new TransactionScope(TransactionScopeAsyncFlowOption.Enabled)) {
bus.Publish(new SomethingHappened { Description = String.Format("{0} logged in at {1}", user.Name, DateTime.Now.ToLongTimeString()) }).ConfigureAwait(false).GetAwaiter().GetResult();
}
Doing that, you get the behavior I was expecting.
Related
So I have the following scenario. I have a method in my WCF, where the client will send a request, the WCF service would then perform some background processing and do call an external webservice method, and the method will respond with an acknowledgement immediately (before the background processing has been completed).
The way I have thought of doing is having my WCF method return a response after spawning a thread to do the background processing, and call the external webservice. The flow is something like this:
Caller sends request to INITIAL_CALL
WCF starts a thread that calls PROCESS
WCF returns true
PROCESS makes call to EXTERNALWS and gets response in postResponse
postReponse gets logged to the database
See example code below:
[ServiceBehavior(InstanceContextMode = InstanceContextMode.PerCall)]
public class Service : IService
public bool INITIAL_CALL()
{
new Thread(()=>
{
PROCESS();
}).Start();
return true;
}
private void PROCESS()
{
//Do some background processing and create request for call below
var processRequest = "Request goes here";
using (var client = new EXTERNALWS.ResponseTypeClient())
{
var postResponse = client.POST(processRequest);
//Log postResponse to database
}
}
Having in mind that PROCESS() may run for a long time, I just wanted to see if there is a better way of doing this with WCF and IIS? Or if there are any pitfalls that I have to consider i.e IIS app pool recycling destroying the thread.
I have found a solution for this. I ended up using Hangfire to do the background processing needed (https://www.hangfire.io/). Hangfire seems to be specifically made for this. I have implemented it following the documentation found at their homepage, in a separate ASP MVC application. I have also configured it as always running on IIS. All instructions and sample codes to setup Hangfire to do this are found here https://docs.hangfire.io/en/latest/index.html. I had to change the flow (since I am not spawning any new threads manually as previously), and also create a new table in the database so that the INITIAL_CALL in the WCF Application would queue all the long running jobs (later to be picked up and executed by Hangfire). Have in mind this is seperate from Hangfire's queue, this table will be checked by Hangfire in a predefined interval, and will check this database table that stores which function to call, its parameters, and an indicator if the job has already been picked up by Hangfire or not (to avoid the re-entrant scenario described here https://docs.hangfire.io/en/latest/best-practices.html). A little extra work, but works like a charm.
The way the flow works now is as follows:
Caller sends request to INITIAL_CALL
In INITIAL_CALL, an entry is made in a new database table (this is the job
queue that will be checked by Hangfire in a predefined interval).
INITIAL_CALL returns true
Hangfire checks this database table in a predefined interval using PROCESS_JOBS (this interval can be defined in Hangfire itself).
If there is a queued item, PROCESS_JOBS proceeds and makes the call to EXTERNALWS and gets response in postResponse. If not, it just exits and does nothing further.
postReponse gets logged to the database.
See updated example code below:
WCF Application
[ServiceBehavior(InstanceContextMode = InstanceContextMode.PerCall)]
public class Service : IService
public bool INITIAL_CALL()
{
//Add job queue entry in database table to be picked up by Hangfire
return true;
}
Hangfire Application
public void PROCESS_JOBS()
{
//Check in a predefined interval if there is a pending job in the queue.
//If there is continue with below, otherwise exit function.
//Do some background processing and create request for call below
var processRequest = "Request goes here";
using (var client = new EXTERNALWS.ResponseTypeClient())
{
var postResponse = client.POST(processRequest);
//Log postResponse to database
}
}
I hate the questions that have "Not Enough Info". So I will try to give detailed information. And in this case it is code.
Server:
64 bit of https://github.com/MSOpenTech/redis/tree/2.6/bin/release
There are three classes:
DbOperationContext.cs: https://gist.github.com/glikoz/7119628
PerRequestLifeTimeManager.cs: https://gist.github.com/glikoz/7119699
RedisRepository.cs https://gist.github.com/glikoz/7119769
We are using Redis with Unity ..
In this case we are getting this strange message:
"Redis Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool. This may have occurred because all pooled connections were in use.";
We checked these:
Is the problem configuration issue
Are we using wrong RedisServer.exe
Is there any architectural problem
Any idea? Any similar story?
Thanks.
Extra Info 1
There is no rejected connection issue on server stats (I've checked it via redis-cli.exe info command)
I have continued to debug this problem, and have fixed numerous things on my platform to avoid this exception. Here is what I have done to solve the issue:
Executive summary:
People encountering this exception should check:
That the PooledRedisClientsManager (IRedisClientsManager) is registed in a singleton scope
That the RedisMqServer (IMessageService) is registered in a singleton scope
That any utilized RedisClient returned from either of the above is properly disposed of, to ensure that the pooled clients are not left stale.
The solution to my problem:
First of all, this exception is thrown by the PooledRedisClient because it has no more pooled connections available.
I'm registering all the required Redis stuff in the StructureMap IoC container (not unity as in the author's case). Thanks to this post I was reminded that the PooledRedisClientManager should be a singleton - I also decided to register the RedisMqServer as a singleton:
ObjectFactory.Configure(x =>
{
// register the message queue stuff as Singletons in this AppDomain
x.For<IRedisClientsManager>()
.Singleton()
.Use(BuildRedisClientsManager);
x.For<IMessageService>()
.Singleton()
.Use<RedisMqServer>()
.Ctor<IRedisClientsManager>().Is(i => i.GetInstance<IRedisClientsManager>())
.Ctor<int>("retryCount").Is(2)
.Ctor<TimeSpan?>().Is(TimeSpan.FromSeconds(5));
// Retrieve a new message factory from the singleton IMessageService
x.For<IMessageFactory>()
.Use(i => i.GetInstance<IMessageService>().MessageFactory);
});
My "BuildRedisClientManager" function looks like this:
private static IRedisClientsManager BuildRedisClientsManager()
{
var appSettings = new AppSettings();
var redisClients = appSettings.Get("redis-servers", "redis.local:6379").Split(',');
var redisFactory = new PooledRedisClientManager(redisClients);
redisFactory.ConnectTimeout = 5;
redisFactory.IdleTimeOutSecs = 30;
redisFactory.PoolTimeout = 3;
return redisFactory;
}
Then, when it comes to producing messages it's very important that the utilized RedisClient is properly disposed of, otherwise we run into the dreaded "Timeout Expired" (thanks to this post). I have the following helper code to send a message to the queue:
public static void PublishMessage<T>(T msg)
{
try
{
using (var producer = GetMessageProducer())
{
producer.Publish<T>(msg);
}
}
catch (Exception ex)
{
// TODO: Log or whatever... I'm not throwing to avoid showing users that we have a broken MQ
}
}
private static IMessageQueueClient GetMessageProducer()
{
var producer = ObjectFactory.GetInstance<IMessageService>() as RedisMqServer;
var client = producer.CreateMessageQueueClient();
return client;
}
I hope this helps solve your issue too.
The current advice on handling exceptions in NServiceBus is to use the built in facilities. Errored out messages go to the error message queue, and a log is written to disk.
But what if I want to send my errors to a service like AirBrake which has better functionality for grouping similar exceptions, metrics, and other good stuff? Is there a global exception handler I can tap into?
As mentioned in the original post the recommended solution is to use ServicePulse for monitoring errors. The client I currently work for is using a custom made centralized logger, and we want NServiceBus to log to this log store when messages are forwarded to the error queue.
We could have achieved this by just editing the log4net config if the exception from NServiceBus had included the original exception, currently NServiceBus just logs a generic error message with no details about what caused the failure.
NServiceBus has a class named NServiceBus.Faults.ErrorsNotifications which contains the following observables:
MessageSentToErrorQueue
MessageHasFailedAFirstLevelRetryAttempt
MessageHasBeenSentToSecondLevelRetries
You can subscribe to these observables when the endpoint starts, like in the following example which logs an error when messages are sent to the error queue:
public class GlobalErrorHandler : IWantToRunWhenBusStartsAndStops
{
private readonly ILogger _logger;
private readonly BusNotifications _busNotifications;
readonly List<IDisposable> _notificationSubscriptions = new List<IDisposable>();
public GlobalErrorHandler(ILogger logger, BusNotifications busNotifications)
{
_logger = logger;
_busNotifications = busNotifications;
}
public void Start()
{
_notificationSubscriptions.Add(_busNotifications.Errors.MessageSentToErrorQueue.Subscribe(LogWhenMessageSentToErrorQueue));
}
public void Stop()
{
foreach (var subscription in _notificationSubscriptions)
{
subscription.Dispose();
}
}
private void LogWhenMessageSentToErrorQueue(FailedMessage message)
{
var properties = new
{
MessageType = message.Headers["NServiceBus.EnclosedMessageTypes"],
MessageId = message.Headers["NServiceBus.MessageId"],
OriginatingMachine = message.Headers["NServiceBus.OriginatingMachine"],
OriginatingEndpoint = message.Headers["NServiceBus.OriginatingEndpoint"],
ExceptionType = message.Headers["NServiceBus.ExceptionInfo.ExceptionType"],
ExceptionMessage = message.Headers["NServiceBus.ExceptionInfo.Message"],
ExceptionSource = message.Headers["NServiceBus.ExceptionInfo.Source"],
TimeSent = message.Headers["NServiceBus.TimeSent"]
};
_logger.Error("Message sent to error queue. " + properties, message.Exception);
}
}
The observable is implemented by using Reactive Extensions, so you will have to install the NuGet package Rx-Core for this to work.
Could you maybe use a custom log4net configuration to do it?
http://help.airbrake.io/discussions/suggestions/157-net-plug-in-for-api-v2
There is an interface in NServiceBus called IManageMessageFailures, but I don't believe you can use it in conjucntion with the Second Level Retries so you would probably lose that if you did go for your own.
You could write the errors to SQL Server with log4net and the from there you could forward them to AirBrake using their API marking each error in the error table as sent afterwards maybe?
We switched to Serilog, it has perfect logging ans tracing support for NServiceBus. Creating a custom Serilog sink that will send your log events wherever you want is quite easy.
I would suggest that you create a custom endpoint that feeds off the error queue and uploads the data to AirBrake.
Side note: v4.0|4.1 will come with a nice Rest-Api that gives you nice management/querying capabilities for the errors. Tools like the Profiler and Ops will use this api to present the data to you in various ways.
http://particular.net/service-platform
I have a WCF service that implements the 'Custom-Username-Password-Validator'.
The service itself checks the username+password against a local file,
and if there is no match - it throws a FaultException with a message.
.
When I use the service synchronously it works fine.
When I go to work with it ASYNC, I have a problem.
If I pass the wrong 'Username+Password' credentials - and open the client,
instead of returning immediatly from the service going into my 'Channel_Faulted()' method,
the client thread simply waits until the Timeout triggers,
and then I get a 'TimeoutException'.
try
{
client = new MyServiceClient("WSDualHttpBinding_IMyervice");
client.ClientCredentials.UserName.UserName = "username";
client.ClientCredentials.UserName.Password = "bad password";
client.ChannelFactory.Faulted += new EventHandler(ChannelFactory_Faulted);
client.Open(); // This hangs for 01:00:00 minute
// Doesn't reach here
client.DoSomethingCompleted += new EventHandler<DoSomethingEventArgs(client_DoSomethingCompleted);
client.DoSomethingAsync(param);
}
catch (Exception ex)
{
// Enters here with a 'TimeoutException' exception
}
why does the client not trigger the 'Faulted' method I have ?
Why does it wait for a response from the service even though the service through a 'FaultException' during the 'Validate' method of the 'CustomUserNameValidator' ?
Sure, the code you are using appears to be missing 3 lines after your code line:
client.ChannelFactory.Faulted += new EventHandler(ChannelFactory_Faulted);
But again, I'm taking a shot in the dark since I've not made use of this option yet.
var local = client.ChannelFactory.CreateChannel();
((IClientChannel)local).Faulted += ChannelFaulted;
local.Open();
Better yet, the open method doesn't appear to be necessary according to the sample provide here: ChannelFactory
I personally have not used the ChannelFactory.Faulted event handler however, here is a post for your consideration: creating-wcf-channelfactory
The backgound: I am trying to forward the server-side ApplyChangeFailed event that is fired by a Sync Services for ADO 1.0 DBServerSyncProvider to the client. All the code examples for Sync Services conflict resolution do not use WCF, and when the client connects to the server database directly, this problem does not exist. My DBServerSyncProvider is wrapped by a head-less WCF service, however, and I cannot show the user a dialog with the offending data for review.
So, the obvious solution seemed to be to convert the HTTP WCF service that Sync Services generated to TCP, make it a duplex connection, and define a callback handler on the client that receives the SyncConflict object and sets the Action property of the event.
When I did that, I got a runtime error (before the callback was attempted):
System.InvalidOperationException: This operation would deadlock because the
reply cannot be received until the current Message completes processing. If
you want to allow out-of-order message processing, specify ConcurrencyMode of
Reentrant or Multiple on CallbackBehaviorAttribute.
So I did what the message suggested and decorated both the service and the callback behavior with the Multiple attribute. Then the runtime error went away, but the call results in a "deadlock" and never returns. What do I do to get around this? Is it not possible to have a WCF service that calls back the client before the original service call returns?
Edit: I think this could be the explanation of the issue, but I am still not sure what the correct solution should be.
After updating the ConcurrencyMode have you tried firing the callback in a seperate thread?
This answer to another question has some example code that starts another thread and passes through the callback, you might be able to modify that design for your purpose?
By starting the sync agent in a separate thread on the client, the callback works just fine:
private int kickOffSyncInSeparateThread()
{
SyncRunner syncRunner = new SyncRunner();
Thread syncThread = new Thread(
new ThreadStart(syncRunner.RunSyncInThread));
try
{
syncThread.Start();
}
catch (ThreadStateException ex)
{
Console.WriteLine(ex);
return 1;
}
catch (ThreadInterruptedException ex)
{
Console.WriteLine(ex);
return 2;
}
return 0;
}
And this is my SyncRunner:
class SyncRunner
{
public void RunSyncInThread()
{
MysyncAgent = new MySyncAgent();
syncAgent.addUserIdParameter("56623239-d855-de11-8e97-0016cfe25fa3");
Microsoft.Synchronization.Data.SyncStatistics syncStats =
syncAgent.Synchronize();
}
}