So I have been tasked with setting up MSMQ so that if our mail server goes down (which is seems to often) the messages just end up in the Queue and will be delivered when they come back up. With that said I have to say I don't know much about this except what I have learned in the past 24 hours however I believe I know enough to take the right approach but I wanted to ask someone in the community because there is some confusion amongst my colleagues given some existing setup in our WCF application.
Currently we have some services that use msmq as the protocol for the endpoint. the endpoint looks like this
<endpoint address="net.msmq://localhost/private/Publisher"
behaviorConfiguration="BatchBehaviour"
binding="netMsmqBinding"
bindingConfiguration="MSMQNoSecurity"
contract="HumanArc.Compass.Shared.Publisher.Interfaces.Service.IPublisherSubscriber"
name="PublishSubscriber"/>
This of course lets the client make a service call and if for some reason the service wasn't up it will ensure that when the service comes back up the call will be processed. What I don't think that it will do is if you have something like the following in you service method.
try
{
smtp.Send(mail);
return true;
}
catch (System.Net.Mail.SmtpFailedRecipientException ex)
{
throw new Exception("User Credentials for sending the Email are Invalid",ex);
}
catch (System.Net.Mail.SmtpException smtpEx)
{
throw new Exception(string.Format("Application encountered a problem send a mail message to {0} ", smtpHostName),smtpEx);
}
WCF isn't going to retry and send the message again somehow, am I correct about this assumption?
What I think we should have is something that looks like the following in place of the call to smtp.send() above. (from http://www.bowu.org/it/microsoft/net/email-asp-net-mvc-msmq-2.html)
string queuePath = #".\private$\WebsiteEmails";
MessageQueue msgQ;
//if this queue doesn't exist we will create it
if(!MessageQueue.Exists(queuePath))
MessageQueue.Create(queuePath);
msgQ = new MessageQueue(queuePath);
msgQ.Formatter = new BinaryMessageFormatter();
msgQ.Send(msg);
Then somewhere in the startup of the service (I am not sure where yet) we set up an event handler that will actually call send() on the SmtpClient object. Something like this
msgQ.ReceiveCompleted += new ReceiveCompletedEventHandler(msgQ_ReceiveCompleted)
So to sum it all up my first question is which way is better? Create a service that uses net:msmq as the protocol or just change the email method to put messages in the queue and set up a handler for it? The next question, if my assumption about changing the method that calls SmtpClient.Send() is correct then where in the program should I wire up ReceiveCompleted? Out WCF service is hosted in a windows service, meaning there is actually a call to ServiceBase.Run(servicesToRun). Is there a place I could wire it up there? My experience with WCF is with much simpler IIS hosted services so I am not 100% sure.
Thanks - I realize this is a long question but I have been trying to research it and there is a lot of information and I can't seem to find a clear explanation of the benefits of doing things one way vs another.
Your approach to using msmq to address availability in a downstream dependency (in this case your smtp server) is valid. However, there are a couple of things you should understand about msmq first.
If you create a queue in msmq then by default it is non-transactional. In this mode the queue will not provide the kind of guaranteed delivery semantic you require. So create your queues as transactional.
Then you can tell WCF that your service operation will enlist in the transaction when it receives a message for processing. You do this by defining a behavior on your service operation implementation:
[OperationBehavior(TransactionScopeRequired = true, TransactionAutoComplete = true)]
public void SendEmail(Something mail)
{
....
smtp.Send(mail);
}
TransactionScopeRequired tells WCF that the service operation should enlist in the same transaction used to transmit the message from sender to receiver. TransactionAutoComplete states that the service method should commit the transaction once the operation has successfully completed. So in answer to your query above, a failure in the service operation will cause the transaction to rollback.
What happens at this point depends on your service bindings configuration.
<netMsmqBinding>
<binding name="netMsmqBinding_IMyServiceInterface"
exactlyOnce="true"
maxRetryCycles="3"
retryCycleDelay="00:01:00"
receiveErrorHandling="Move"> <-- this defines behavior after failure
...
</binding>
</netMsmqBinding>
When, for whatever reason the transaction is not committed (for example, an unhandled exception occurs), WCF will roll the message back onto the queue and retry processing once per minute up to 3 times (defined by maxRetryCycles and retryCycleDelay).
If the message still fails processing after this time then the receiveErrorHandling attribute tells WCF what to do next (The above binding specifies that the message be moved to the system poison message queue).
Note: exactlyOnce tells WCF that we require transactions, that each message will be delivered exactly once and in the order they were sent.
So your original approach is in fact correct and you just need to configure your service correctly to implement the behavior you want.
Related
My requirement is to make the Subscriber pause processing the messages depending on whether a web service is up or not. So, when the web service is down, the messages should keep coming to the subscriber queue from Publisher and keep piling up until the web service is up again. (These messages should not go to the error queue, but stay on the Subscriber queue.)
I tried to use unsubscribe, but the publisher stops sending messages as the unsubscribe seems to clear the subscription info on RavenDB. I have also tried setting the MaxConcurrencyLevel on the Transport class, if I set the worker threads to 0, the messages coming to Subscriber go directly to the error queue. Finally, I tried Defer, which seems to put the current message in audit queue and creates a clone of the message and sends it locally to the subscriber queue when the timeout is completed. Also, since I have to keep checking the status of service and keep defering, I cannot control the order of messages as I cannot predict when the web service will be up.
What is the best way to achieve the behavior I have explained? I am using NServiceBus version 4.5.
It sounds like you want to keep trying to handle a message until it succeeds, and not shuffle it back in the queue (keep it at the top and keep trying it)?
I think your only pure-NSB option is to tinker with the MaxRetries setting, which controls First Level Retries: http://docs.particular.net/nservicebus/msmqtransportconfig. Setting MaxRetries to a very high number may do what you are looking for, but I can't imagine doing so would be a good practice.
Second Level Retries will defer the message for a configurable amount of time, but IIRC will allow other messages to be handled from the main queue.
I think your best option is to put retry logic into your own code. So the handler can try to access the service x number of times in a loop (maybe on a delay) before it throws an exception and NSB's retry features kick in.
Edit:
Your requirement seems to be something like:
"When an MyEvent comes in, I need to make a webservice call. If the webservice is down, I need to keep trying X number of times at Y intervals, at which point I will consider it a failure and handle a failure condition. Until I either succeed or fail, I will block other messages from being handled."
You have some potentially complex logic on handling a message (retry, timeout, error condition, blocking additional messages, etc.). Keep in mind the role that NSB is intended to play in your system: communication between services via messaging. While NSB does have some advanced features that allow message orchestration (e.g. sagas), it's not really intended to be used to replace Domain or Application logic.
Bottom line, you may need to write custom code to handle your specific scenario. A naive solution would be a loop with a delay in your handler, but you may need to create a more robust in-memory collection/queue that holds messages while the service is down and processes them serially when it comes back up.
The easiest way to achieve somewhat the required behavior is the following:
Define a message handler which checks whether the service is available and if not calls HandleCurrentMessageLater and a message handler which does the actual message processing. Then you specify the message handler order so that the handler which checks the service availability gets executed first.
public interface ISomeCommand {}
public class ServiceAvailabilityChecker : IHandleMessages<ISomeCommand>{
public IBus Bus { get; set; }
public void Handle(ISomeCommand message) {
try {
// check service
}
catch(SpecificException ex) {
this.Bus.HandleCurrentMessageLater();
}
}
}
public class ActualHandler : IHandleMessages<ISomeCommand>{
public void Handle(ISomeCommand message) {
}
}
public class SomeCommandHandlerOrdering : ISpecifyMessageHandlerOrdering{
public void SpecifyOrder(Order order){
order.Specify(First<ServiceAvailabilityChecker>.Then<ActualHandler>());
}
}
With that design you gain the following:
You can check the availability before the actual business code is invoked
If the service is not available the message is put back into the queue
If the service is available and your business code gets invoked but just before the ActualHandler is invoked the service becomes unavailable you get First and Second Level retries (and again the availability check in the pipeline)
For some reason, all the examples I find on WCF MSMQ show void as the return type of the method. Is this just coincidence? If not, why? What are the valid return types?
There are no valid Return types for sending to to a WCF service on an MSMQ end point.
Because you are writing to a queue and not directly communicating with the service the communication channel is not available for the service to send a response. Additionally one of the major benefits of writing to the queue is that the service may not even be running at the point the message is sent, the message will get picked up once the service becomes active again.
If no exception is thrown whilst sending to the service then you know that the message has at the very least been added to the message queue.
I have a WPF client that makes calls to 2 WCF services.
One service is for querying only and one service is for commands (CQS pattern).
How should I make the calls to the commands service ?
I read somewhere that all the operations in the command service must be 'One-Way',
because they should not return any values. And that if something went wrong - the operation should throw a 'FaultException' to the client.
But if the commands are all One-Way - what do I do in the client ?
Say I have an 'AddProduct' window in the WPF client, and I enter information and press 'Save'.
I now call 'AddProduct(Product)' in the service, but :
Should it close the window ?
Should it wait for 10 seconds to see if there wasn't any FaultException ?
Should the operation not be 'One-Way' ? If so - should all operations in the command service return some type of generic 'Result' object with 'succeeded' or 'failed' ?
If section 3 is 'Yes' - should I call the service in a seperate thread and 'disable' all the controls on the window until I get a response back from the service ?
Thanks.
I would say option 3 is the way to go, but you probably do not need the generic Result object to communicate errors to the client. As you might know, exceptions are not serialized in the SOAP message so you won't get any of the usual .NET exceptions on the client side. On the other hand, you can still take advantage of SOAP Faults by catching FaultException on the client. Accordingly, if no exceptions were caught on the client, then everything went well.
For more information about fault exceptions and how you can use them to your benefit, take a look at:
Specifying and Handling Faults in Contracts and Services
I think using On-Way is fine but you have to be aware of some one-way call characteristic. If you care and can handle service exceptions then #4 is fine option.
One Way message - Once the client issues the call, WCF generates the request message but no correlated message will be ever returned to the client. Any exceptions thrown on the service side will not make it to the client.
One thing that you should have on is the reliability on your service so side so that you can insure that request has been delivered to the service.
When there is no transport session (basic or wsHttp binding) if exception occurs during the call of one-way operation client will be unaffected and it can continue sending calls on the same proxy instance.
If there is a presence of transport session - service side exception will fault the channel hence client will not be able to re-use proxy for sending more calls. This can give you an option to discover if something went wrong on the server side (but not what went wrong). Although, if service is using a FaultContracts you can still get into situation where client is unaware that something went wrong.
When service throws an exception listed in service side fault contract this will not fault the communication channel hence the client using one-way contract cannot detect communication failure.
The below text is an effort to expand and add color to this question:
How do I prevent a misbehaving client from taking down the entire service?
I have essentially this scenario: a WCF service is up and running with a client callback having a straight forward, simple oneway communication, not very different from this one:
public interface IMyClientContract
{
[OperationContract(IsOneWay = true)]
void SomethingChanged(simpleObject myObj);
}
I'm calling this method potentially thousands of times a second from the service to what will eventually be about 50 concurrently connected clients, with as low latency as possible (<15 ms would be nice). This works fine until I set a break point on one of the client apps connected to the server and then everything hangs after maybe 2-5 seconds the service hangs and none of the other clients receive any data for about 30 seconds or so until the service registers a connection fault event and disconnects the offending client. After this all the other clients continue on their merry way receiving messages.
I've done research on serviceThrottling, concurrency tweaking, setting threadpool minimum threads, WCF secret sauces and the whole 9 yards, but at the end of the day this article MSDN - WCF essentials, One-Way Calls, Callbacks and Events describes exactly the issue I'm having without really making a recommendation.
The third solution that allows the service to safely call back to the client is to have the callback contract operations configured as one-way operations. Doing so enables the service to call back even when concurrency is set to single-threaded, because there will not be any reply message to contend for the lock.
but earlier in the article it describes the issue I'm seeing, only from a client perspective
When one-way calls reach the service, they may not be dispatched all at once and may be queued up on the service side to be dispatched one at a time, all according to the service configured concurrency mode behavior and session mode. How many messages (whether one-way or request-reply) the service is willing to queue up is a product of the configured channel and the reliability mode. If the number of queued messages has exceeded the queue's capacity, then the client will block, even when issuing a one-way call
I can only assume that the reverse is true, the number of queued messages to the client has exceeded the queue capacity and the threadpool is now filled with threads attempting to call this client that are now all blocked.
What is the right way to handle this? Should I research a way to check how many messages are queued at the service communication layer per client and abort their connections after a certain limit is reached?
It almost seems that if the WCF service itself is blocking on a queue filling up then all the async / oneway / fire-and-forget strategies I could ever implement inside the service will still get blocked whenever one client's queue gets full.
Don't know much about the client callbacks, but it sounds similar to generic wcf code blocking issues. I often solve these problems by spawning a BackgroundWorker, and performing the client call in the thread. During that time, the main thread counts how long the child thread is taking. If the child has not finished in a few milliseconds, the main thread just moves on and abandons the thread (it eventually dies by itself, so no memory leak). This is basically what Mr.Graves suggests with the phrase "fire-and-forget".
Update:
I implemented a Fire-and-forget setup to call the client's callback channel and the server no longer blocks once the buffer fills to the client
MyEvent is an event with a delegate that matches one of the methods defined in the WCF client contract, when they connect I'm essentially adding the callback to the event
MyEvent += OperationContext.Current.GetCallbackChannel<IFancyClientContract>().SomethingChanged
etc... and then to send this data to all clients, I'm doing the following
//serialize using protobuff
using (var ms = new MemoryStream())
{
ProtoBuf.Serializer.Serialize(ms, new SpecialDataTransferObject(inputData));
byte[] data = ms.GetBuffer();
Parallel.ForEach(MyEvent.GetInvocationList(), p => ThreadUtil.FireAndForget(p, data));
}
in the ThreadUtil class I made essentially the following change to the code defined in the fire-and-foget article
static void InvokeWrappedDelegate(Delegate d, object[] args)
{
try
{
d.DynamicInvoke(args);
}
catch (Exception ex)
{
//THIS will eventually throw once the client's WCF callback channel has filled up and timed out, and it will throw once for every single time you ever tried sending them a payload, so do some smarter logging here!!
Console.WriteLine("Error calling client, attempting to disconnect.");
try
{
MyService.SingletonServiceController.TerminateClientChannelByHashcode(d.Target.GetHashCode());//this is an IContextChannel object, kept in a dictionary of active connections, cross referenced by hashcode just for this exact occasion
}
catch (Exception ex2)
{
Console.WriteLine("Attempt to disconnect client failed: " + ex2.ToString());
}
}
}
I don't have any good ideas how to go and kill all the pending packets the server is still waiting to see if they'll get delivered on. Once I get the first exception I should in theory be able to go and terminate all the other requests in some queue somewhere, but this setup is functional and meets the objectives.
I have an asynchronous WCF service using nettcpbinding. And I send a bunch of requests to it and get result by invoking EndDoWork() in my callback. If EndDonWork throw a exception once, all the invocation after that will throw exception said: communication object, System.ServiceModel.Channels.ServiceChannel, cannot be used for communication because it is in the Faulted state.
I think that's something close the connection because of the first exception. My question is:
1. what decide this behavior? If I use basicHttpBinding, the later invocation of EndDoWork work well. Is it related with keepAlive support?
2. Is there any property of configuration item I can set to ask service reconnect automatically?
The Faulted state of the channel indicates that it cannot be relied on any more. You did not mention what the reason was why the exception was thrown (connectivity, server stopped etc), but as far as WCF is concerned the endpoint is invalid and therefore faulted.
You should recreate the channel and connect to the service again to continue any of your operations. If you use features like Reliable connections then some of the work may be done for you, but if the channel is eventually faulted, the same rules apply.
You will also have to implement your own message queue to re-request messages that were pending when the channel faulted. You cannot rely on the channel to keep and resend the messages.
If I remember correctly, you can avoid the channel faulting if you declare the Fault in the operation contract.
For example:
[ServiceContract]
public interface IService
{
[OperationContract]
[FaultContract(typeof(MyDefinedFault))]
void Operation();
}
As you have already declared MyDefinedFault in the Operation contract if you throw that from the service, the channel is not going to fault (unless of course you are using the System.ServiceModel.Description.ServiceDebugBehavior.IncludeExceptionDetailInFaults=true that may fault the channel anyways).
Where is the MyDefinedFault class.
how to define this class.