WCF, MSMQ and independent transactions on 2 queues - wcf

I have built a WCF service that processes an MSMQ, let's call the service QueueService. The Contract looks like this:
// Each call to the service will be dispatched separately, not grouped into sessions.
[ServiceBehavior(InstanceContextMode = InstanceContextMode.PerCall)]
[AspNetCompatibilityRequirements(RequirementsMode = AspNetCompatibilityRequirementsMode.Allowed)]
public class QueueServiceContract : IQueueServiceContract
{
[OperationBehavior(TransactionScopeRequired = true)]
public void QueueItems(List<Item> items) // really should be called 'HandleQueueItems
{
// Early in the processing I do:
Transaction qTransaction = Transaction.Current;
...
// I then check if the destination database is available.
if(DB.IsAvailable)
... process the data
else
qTransaction.Rollback;
...
}
IQueueServiceContract looks like this:
// The contract will be session-less. Each post to the queue from the client will create a single message on the queue.
[ServiceContract(SessionMode = SessionMode.NotAllowed, Namespace = "MyWebService")]
public interface IQueueServiceContract
{
[OperationContract(IsOneWay = true)]
void QueueItems(List<Item> items);
}
Relevant parts of the App.config for the queue service look like this.
<services>
<service name="QueueService.QueueServiceContract">
<endpoint address="net.msmq://localhost/private/MyQueueServiceQueue" binding="netMsmqBinding" contract="QueueService.IQueueServiceContract">
<identity>
<dns value="localhost" />
</identity>
</endpoint>
...
<netMsmqBinding>
<binding exactlyOnce="true" maxRetryCycles="1000" receiveRetryCount="1"
retryCycleDelay="00:10:00" timeToLive="7.00:00:00" useActiveDirectory="false">
</binding>
</netMsmqBinding>
This all works fine. When the DB is not available, the Rollback causes the queue entry to be put in the retry subqueue that I have configured to retry every 10 mins for 7 days. Everything about it works and has been in production for 6 months or so.
Now I am adding logging to the service. The QueueService is going to queue log entries into another queue we will call: LogQueue. The requirement is that whether the qTransaction is rolled back or not, a message should be sent to the LogQueue indicating the status of the request.
In the QueueService app.config I have added:
<client>
<endpoint address="net.msmq://localhost/private/MyLogQueue"
binding="netMsmqBinding" bindingConfiguration="NetMsmqBinding_ILogContract"
contract="LogServiceReference.ILogContract" name="NetMsmqBinding_ILogContract">
<identity>
<dns value="localhost" />
</identity>
</endpoint>
</client>
...
<binding name="NetMsmqBinding_ILogContract" timeToLive="7.00:00:00">
<security mode="None" />
</binding>
In the LogService app.config, I have:
<service name="LogService.LogContract">
<endpoint address="net.msmq://localhost/private/MyLogQueue" binding="netMsmqBinding" contract="LogService.ILogContract">
<identity>
<dns value="localhost" />
</identity>
</endpoint>
...
<netMsmqBinding>
<binding exactlyOnce="true" maxRetryCycles="1000" receiveRetryCount="1" retryCycleDelay="00:10:00" timeToLive="7.00:00:00" useActiveDirectory="false">
</binding>
</netMsmqBinding>
...
Then, at the end of the QueueItems method I do the following:
LogContractClient proxy = new LogContractClient();
proxy.LogTransaction(myLoggingInformation); // This queues myLoggingInformation to the LogQueue.
This all works fine too... until... a database is not available and the transaction is rolled back.
The Rollback will happen before the call to proxy.LogTransaction and I will get:
System.ServiceModel.CommunicationException: 'An error occurred while sending to the queue: The transaction specified cannot be enlisted. (-1072824232, 0xc00e0058).Ensure that MSMQ is installed and running. If you are sending to a local queue, ensure the queue exists with the required access mode and authorization.'
If I move the proxy.LogTransaction before the qTransaction.Rollback the log entry is never put in the LogQueue.
My working theory is that WCF considers the operations on the two queues: read from QueueService queue and write to LogQueue, as a single transaction. So, if I try to write to the LogQueue after the Rollback the transaction has already ended, but if I write to the LogQueue before calling Rollback the write to the queue is also rolled back.
Is there any way I can retain the ability to rollback the queueService transaction while not simultaneously rolling back the LogService transaction?

I think you can fix this by wrapping the call to the log queue client in a transaction scope with TransactionScopeOption.Suppress set. This will force the action to happen outside of the ambient transaction.
Something like:
using (var scope = new TransactionScope (TransactionScopeOption.Suppress))
{
// Call to enqueue log message
}
Your theory makes perfect sense. Because you're using transactional queues WCF is ensuring transactional consistency by enlisting everything inside the message handler in a DTC transaction. That obviously includes the enqueue of the logging message, and is expected, if unwanted, behaviour.

Related

MSMQ receives messages only from some Clients

I have a WCF service where client applications can connect via a MSMQ:
[ServiceContract(Namespace="http://example.com", SessionMode = SessionMode.NotAllowed)]
public interface IMyService
{
[OperationContract(IsOneWay = true)]
void Update(DataSet ds);
}
and then:
string queueName = ConfigurationManager.AppSettings["QueueName"];
NetMsmqBinding binding = new NetMsmqBinding("MyBinding");
if (!MessageQueue.Exists(#".\private$\" + queueName))
{
MessageQueue.Create(#".\private$\" + queueName, binding.ExactlyOnce);
}
ServiceHost msmqHost = new ServiceHost(typeof(MyService));
msmqHost.AddServiceEndpoint(typeof(IMyService), binding, "net.msmq://localhost/private/" + queueName);
with the following configuration:
<system.serviceModel>
<bindings>
<netMsmqBinding>
<binding name="MyBinding" durable="false" exactlyOnce="false" maxReceivedMessageSize="20000000">
<security mode="None" />
<readerQuotas maxDepth="32" maxStringContentLength="543192" maxArrayLength="2147483647" maxBytesPerRead="4096" maxNameTableCharCount="8456384" />
</binding>
</netMsmqBinding>
</bindings>
<services>
<service name="MyService" behaviorConfiguration="MsMqBehavior" />
</services>
<behaviors>
<serviceBehaviors>
<behavior name="MsMqBehavior">
<serviceThrottling maxConcurrentCalls="50" maxConcurrentSessions="50" maxConcurrentInstances="50" />
</behavior>
</serviceBehaviors>
</behaviors>
</system.serviceModel>
I have the service with the same configuration already in use without problems on other installations. But now on a new installation I receive only messages from some clients (9 actually - there are 31). The messages I receive are always from the same servers. I can't find an error message anywhere (Windows Event Log (Client/Server), WCF Trace file) and also the MSMQ state says "connected" on the client machines that don't send messages. The dead letter queues are also empty.
The messages must get lost somewhere between the MSMQ Client and Server (I stopped my app and on the server queue I received only messages from the nine Clients - same behaviour if I enable journaling).
Any help would be appreciated
Update
I've used performance counters to monitor the queue. The session counter shows the correct value of 31 sessions. Also the incoming message counter shows correct values. However if I stop the app or enable journaling only a part of the messages are stored in the queue.
The problem comes through cloning the server as described in this blog entry: http://blogs.msdn.com/b/johnbreakwell/archive/2007/02/06/msmq-prefers-to-be-unique.aspx
Basically it says there that you must not clone servers with MSMQ feature turned on. If you do so you either have to re-install the MSMQ feature on your client machines or do a registry change:
1.Stop the MSMQ Service
2.Delete the QMId value completely
3.Add a SysPrep DWORD (Under HKLM\Software\Microsoft\MSMQ\Parameters) and set it to 1
4.Start the MSMQ Service

WCF dropping connections (random timeouts, socket aborts, endpoint not found) -Port sharing

I'm doing some wcf tracing on a sever with 800 clients (verbose clients) and I am seeing a lot of SocketConnection aborted after Process action 'http://tempuri.org/IConnectionRegister/ValidateUriRoute'. This is during a ** ** activity
I removed (maxed out) all possible throttling (binding, service, port sharing (both on custom binding and SMSvcHost.exe). Still a lot of random timeouts and socket aborted.
Also the 0000 activity is absolutely full (less than every second) of
"The socket connection was aborted. This could be caused by an error processing your message or a receive timeout being exceeded by the remote host, or an underlying network resource issue. Local socket timeout was '10675199.02:48:05.4775807'."
"The I/O operation has been aborted because of either a thread exit or an application request"
Any help would be very much appreciate in identifying who is causing the dropping of connections.
I can also upload a sample svc trace if anyone is interested.
Stack trace of Socket Connection Abort
System.ServiceModel.Channels.SocketConnection.Abort()
System.ServiceModel.Channels.TracingConnection.Abort()
System.ServiceModel.Channels.InitialServerConnectionReader.Abort()
System.ServiceModel.Channels.ServerSessionPreambleConnectionReader.OnValidate()
......
System.Runtime.Fx.AsyncThunk.UnhandledExceptionFrame()
System.Runtime.AsyncResult.Complete()
System.ServiceModel.Channels.TransportDuplexSessionChannel.TryReceiveAsyncResult.OnReceive()
System.Runtime.Fx.AsyncThunk.UnhandledExceptionFrame()
System.Runtime.AsyncResult.Complete()
System.ServiceModel.Channels.SynchronizedMessageSource.ReceiveAsyncResult.OnReceiveComplete()
<customBinding>
<binding name="externalRoutingcustomBinding">
<binaryMessageEncoding>
</binaryMessageEncoding>
<security
authenticationMode="UserNameOverTransport"
allowInsecureTransport="true"
enableUnsecuredResponse="true"
>
</security><tcpTransport>
<connectionPoolSettings groupName = "default" maxOutboundConnectionsPerEndpoint = "1000"/>
</tcpTransport>
</binding>
<service name="ZPRoutingWorkflow.ZpRoutingService">
<endpoint address="Transactions" binding="customBinding" bindingConfiguration="externalRoutingcustomBinding"
name="Transactions" contract="TopUpInterface.ITransactionsService">
<identity>
<dns value="appserver.test.com" />
</identity>
</endpoint>
<endpoint address="Retry" binding="customBinding" bindingConfiguration="externalRoutingcustomBinding" name="Retry" contract="IRetryInterface.IRetryService">
<identity>
<dns value="appserver.test.com" />
</identity>
</endpoint>
Config summary:
Multiple net.tcp services over same port hosted in iis. One of the services is a routing service implementing IRequestReplyRouter. Basically all messages go through the router and the router connects internally to the services and passes them their messages

Remote computer is not available when trying to pull from remote private queue

I have a remote private queue that I can write to without issue. I am trying to create a WAS-hosted WCF service to pull the data for processing. I need the service to be on a different machine than the queue; I currently have it in the same site as the writing service - same app pool but a different application.
The queue I am targetting is called PersistenceService/Log.svc and is transactional and authenticated. My service is running at http://appdev.me.com/PersistenceService/Log.svc. The below contains my endpoint and binding:
<bindings>
<msmqIntegrationBinding>
<binding exactlyOnce="true" durable="true" serializationFormat="Xml" maxReceivedMessageSize="2147483647"
closeTimeout="00:00:30" sendTimeout="00:00:30" receiveTimeout="00:00:30" timeToLive="24:00:00"
receiveRetryCount="1" maxRetryCycles="1" retryCycleDelay="00:10:00" receiveErrorHandling="Move">
<security mode="Transport" />
</binding>
</msmqIntegrationBinding>
</bindings>
<services>
<service name="Me.Logging.Persistence.PersistenceService">
<endpoint address="msmq.formatname:DIRECT=OS:DIRECT=OS:meserver\private$\persistenceservice/log.svc"
binding="msmqIntegrationBinding"
bindingNamespace="http://me.logging/services/2012/11"
contract="Me.Logging.Service.Shared.Service.Contracts.ICatchAll" />
</service>
</services>
The error that I get is the below:
An error occurred while opening the queue:The remote computer is not available. (-1072824215, 0xc00e0069). The message cannot be sent or received from the queue. Ensure that MSMQ is installed and running. Also ensure that the queue is available to open with the required access mode and authorization.
I have cleared the way for communications by shutting down McAfee so it's not blocking traffic. I have granted ANONYMOUS LOGON access to the queue temporarily - though it should be coming in as the same user through which I'm writing to the queue, and that user has full rights on the queue.
The WCF trace doesn't reveal anything other than this error. Any ideas as to where to look would be greatly appreciated.
Your endpoint looks to be incorrect. You have "DIRECT=OS:" in there twice. Try:
<endpoint address="msmq.formatname:DIRECT=OS:meserver\private$\persistenceservice/log.svc"
binding="msmqIntegrationBinding"
bindingNamespace="http://me.logging/services/2012/11"
contract="Me.Logging.Service.Shared.Service.Contracts.ICatchAll" />

WCF Service Listener to MSMQ - Pending Messages

I've created a WCF service that listens to a private MSMQ for jobs. I've throttled my service so that it will only handle one job at a time (maxConcurrentInstances = 1). The problem I have is that when two messages are submitted and I inspect the queue through my Computer Management console, it's empty. I expect there to be one pending message. When I submit three messages, I'll see one pending message in the MSMQ. From reading MSDN, it looks like the ServiceHost is holding the next job in memory until the current job is done, but I can't find a way to turn it off so that the it doesn't hold the message in memory. Does anyone know of a way to make it so that the ServiceHost won't hold the pending message in memory and leave it in the queue? Thanks!
<configuration>
<system.serviceModel>
<services>
<service
name="MyService"
behaviorConfiguration="DefaultServiceBehavior">
<host>
<baseAddresses>
<add baseAddress="http://localhost:8000/MyService/"/>
</baseAddresses>
</host>
<endpoint
address="net.msmq://localhost/private/MyService"
binding="netMsmqBinding" bindingConfiguration="MsmqBindingNoSecurity"
contract="IMyService" />
<endpoint address="mex"
binding="mexHttpBinding"
contract="IMetadataExchange" />
</service>
</services>
<behaviors>
<serviceBehaviors>
<behavior name="DefaultServiceBehavior">
<serviceMetadata httpGetEnabled="True"/>
<serviceThrottling
maxConcurrentCalls="1"
maxConcurrentSessions="1"
maxConcurrentInstances="1"
/>
</behavior>
</serviceBehaviors>
</behaviors>
<bindings>
<netMsmqBinding>
<binding name="MsmqBindingNoSecurity"
useActiveDirectory="false"
exactlyOnce="false">
<security mode="None">
<transport
msmqAuthenticationMode="None"/>
</security>
</binding>
</netMsmqBinding>
</bindings>
</system.serviceModel>
</configuration>
I have also noticed this behaviour in netMsmqBinding and as far as I know it's not addressable from the service end.
This is only an issue if you're not using transactional queues whereby a failure in your service could result in the in-memory message being dropped permanently.
If you use transactional queues even though the message has been read from the inbound queue it's actually still there on the queue (but it becomes "invisible"). If you suffer a failure on your service at this time the message will become re-queued and then processed when you come back up.
If you cannot use transactional queueing then the only way you can address this is to do so from the client, which means checking to see if a message has been transmitted before making another call. This can be done using System.Messaging or I assume you could bake this into a custom behaviour.
If you can't use a transactional queue (e.g. if you are using netMessagingBinding) you can use the ReceiveContext attribute for more fine-grained control. Its well explained by Juval Lowy here in his book:
http://books.google.co.uk/books?id=PvNrurEhmiEC&pg=PA500&lpg=PA500&dq=ReceiveContext&source=bl&ots=ChDvHuH_Jq&sig=XHhiz2ebmXuu0QNYcBYhtlN99p4&hl=en&sa=X&ei=XgYmUo3kHOOH0AW--oHoCg&ved=0CFsQ6AEwCTgK#v=onepage&q=ReceiveContext&f=false
Also see this MSDN article to see how specifically it can be used in the netMessagingBinding scenario (I know that's not directly relevant to your question but the principal still holds)
http://msdn.microsoft.com/en-us/library/hh532034(VS.103).aspx

Service too busy error in WCF

I intermittently get the following exception in my .Net WCF Service.
"The HTTP service located at http://MyServer/TestWCF/MyService.svc is too busy."
Am I missing something here?
Am using basic http binding and have enabled WCF throttling.
<basicHttpBinding>
<binding name="BasicHttpBinding_MyService" maxReceivedMessageSize="2147483647"
messageEncoding="Text" textEncoding="utf-16" sendTimeout="00:01:00" >
<readerQuotas maxStringContentLength="2147483647" maxArrayLength="163840000"
maxDepth="2147483647" maxBytesPerRead="2147483647" maxNameTableCharCount="16384" />
</binding>
.
.
.
.
<behavior name="MyWCFServices.MyServiceBehavior">
<serviceMetadata httpGetEnabled="true" />
<serviceDebug includeExceptionDetailInFaults="true" />
<serviceThrottling
maxConcurrentCalls="16"
maxConcurrentInstances="2147483647"
maxConcurrentSessions="10"/>
</behavior>
Will throttling help resolving the issue?
Also,may i know the recommended parameter values for throttling for a high traffic web site?
You could definitely try to increase the maxConcurrentSessions and maxConcurrentCalls in your service throttling behavior to the standard values of 30 or so and see if that makes the error go away. Server too busy would seem to indicate that more requests have come in than area allowed by your service throttling behavior, and they've been discarded since no service instance became available to service them within the given timeout period.
My answer would be, check if the app pool is up and well?
I've seen this error occurring when the app pool has died due to exceptions being thrown that aren't caught.
Consider for example, custom config sections - having an error in there, will cause your app to fail before it's even started. Too many of these in a short space of time will kill the app pool.
If you're service is running under your account (Identity), it's quite possible that you've recently changed your password--you'll need to reset it for its IIS application pool in Advanced Settings | Identity dialog box.
It is not just the maxConcurrentSessions, it is also how long the session lasts.
If the client does not close the connection, it will remain open until it timesout. You could then hit the maxConcurrentSessions limit with very little activity on the server.
Make sure you check the inner exception, too; during our deployments, we disable the application pool of a WCF web service, and clients start getting this error during that time:
System.ServiceModel.ServerTooBusyException: The HTTP service located at https://ourserver.x.com/path/service.svc is too busy. ---> System.Net.WebException: The remote server returned an error: (503) Server Unavailable.
So in this case an HTTP error 503 is being (mis?)interpreted as "server too busy".
The only source of this exception that I am aware of is if you are using sessions, and you manage to hit the MaxPendingChannels throttle,. Its default is something pretty low like 4. You could try setting it higher (128 for example), or if you just want to repro, set it to 1 and you should see it under load testing.
See here for more information about sessions: http://msdn.microsoft.com/en-us/library/ms733795.aspx
I just ran into this error, and it boiled down to a simple configuration problem. I had a service up on the exact same port and same interface (mock service). I ran the service with the appropriate command line switch to run the "original" service I intended. The error went away.
My Solution would be, Check the App.Config file, whether the service tag is there for this particular service.
eg:
<service name="MyServices.ServiceName">
<endpoint address="" binding="wsHttpBinding" bindingConfiguration="TestBinding" contract="MyServices.ServiceName">
<identity>
<dns value="localhost" />
</identity>
</endpoint>
<endpoint address="mex" binding="mexHttpBinding" contract="IMetadataExchange" />
<host>
<baseAddresses>
<add baseAddress="http://localhost:8732/Design_Time_Addresses/MyServices/ServiceName/" />
</baseAddresses>
</host>
</service>