I am working to speed up a large number of integration tests in our environment.
The problem I am facing right now is that during teardown between tests one WCF service using msmq binding takes about 1 minute to close down.
I the teardown process we loop over our servicehosts calling the Close() method with a very short timeout, overriding the closeTimeout value in the WCF configuration. This works well for net.tcp bindings but the one service that uses msmq still takes 1 minute to close down. The closeTimeout doesn't seem to have any effect.
The config look like this for the test service:
<netMsmqBinding>
<binding name="NoMSMQSecurity" closeTimeout="00:00:01" timeToLive="00:00:05"
receiveErrorHandling="Drop" maxRetryCycles="2" retryCycleDelay="00:00:01" receiveRetryCount="2">
<security mode="None" />
</binding>
</netMsmqBinding>
And the closing call I use is straight forward like this:
service.Close(new TimeSpan(0, 0, 0, 0, 10));
Is there another approach I can take to close down the servicehost faster?
As this is an automated test that at this point has succeded or failed I don't want to wait for any other unprocessed messages or similar.
Best regards,
Per Salmi
I found the cause of the delayed closing down of the service host using Msmq.
The reason for the long close times seems to be that the service uses another net.tcp based service which has reliableSession activated and the servicehost. The reliableSession settings had an inactivity timeout set to 5 minutes which causes it to send keep-alive infrastructure messages, they should be sent every 2.5 minutes. This keep-alive messaging interval seems to cause the msmq based service to hang around for 1-2 minutes probably waiting for some of the keep-alive messages to arrive.
When I set the inactivityTimeout down to 5 seconds the shutdown of the msmq service completes in about 2.5 seconds. This makes the automatic integration tests pass a lot faster!
Could there be some transaction that is blocking the close.
Say for example there is an open transaction, if you close without commiting the transaction, then it will wait 1 min for the transaction to timeout before it can close.
Related
I host a WCF Service on IIS and have the following binding in web.config:
<bindings>
<wsHttpBinding>
<binding name="transactionalBinding"
transactionFlow = "true"
sendTimeout = "00:00:01"
receiveTimeout = "00:00:01"
openTimeout = "00:00:01"
closeTimeout = "00:00:01">
<security mode="Transport">
<transport clientCredentialType="None" proxyCredentialType="None" realm=""/>
</security>
</binding>
</wsHttpBinding>
</bindings>
In my service method I sleep for 10 seconds. I do not get a timeout exception when calling my service method from a client.
Is there any meaning in defining timeouts in server side bindings?
I do not get a timeout exception when calling my service method from a client.
TL;DR: because WCF timeouts by default are one minute so naturally a server operation that only takes 10 seconds isn't going to timeout. The timeouts you have specified on the server would only affect transmission not execution of your method. (you aren't calling anything else)
You are specifying the timeouts in the server config. What you need to do is specify the timeouts in the client's config file, specifically SendTimeout. Essentially whatever end is making the call, needs to specify the operation timeout. Probably not relevant in your case but if your "server" in turn made another WCF call to another service, you would want your own timeout there too.
MSDN:
SendTimeout – used to initialize the OperationTimeout, which governs the whole process of sending a message, including receiving a reply message for a request/reply service operation. This timeout also applies when sending reply messages from a callback contract method.
Generally, WCF client and server configs should match one another and unless you are using Add Service Reference/Refresh Service Reference each time the server contracts and/or config change, the client won't know about it. By the way, avoid the latter because it duplicates your model and can lead to runtime errors if they are out of sync. Not to mention service contracts can get out of sync.
A passing thought
And this brings up one of the problems of WCF configuration via config files, they are subject to runtime errors impossible to find at compile time.
A better practice is to do away with config files completely and do programatic configuration via a common assembly that both your client and server use. Specify bindings in code along with your timeouts.
That way both server and client are always in sync with regards to WCF configuration.
With both client and server agreeing on timeouts would have addressed some issues.
Tell me more
WCF the Manual Way… the Right Way
I am using Visual Studio 2012 to generate a web service to be used by a winforms client. I created the client side by using "add service reference". This winforms client is a .net c# replacement of an old VB 6 app. Previously, in the VB app there were external settings for timeout values including the following:
DNS timeout
Connect timeout
Request timeout
The DNS timeout would work when the endpoint host address is a FQDN forcing a DNS lookup. The timeout value here would place a limit on the amount of time to wait for DNS resolution.
The connect timeout would place a limit on the amount of time the winforms client would wait to establish an http connection to the server. DNS lookup would have been successful.
The request timeout would place a limit on the amount of time to wait for the request to return after an http connection was successful. This would come into play if a long running query took too long after the web service call was initiated.
Is there something similar to the above in .net 4.0. I would like to be able to configure this in the app.config. I do know about the below.
<bindings>
<basicHttpBinding>
<binding name="IncreasedTimeout"
openTimeout="12:00:00"
receiveTimeout="12:00:00" closeTimeout="12:00:00"
sendTimeout="12:00:00">
</binding>
</basicHttpBinding>
Could these map to the ones I need or does it really not matter?
thanks
The OpenTimeout setting for the WCF binding is for the length of time to wait when opening the channel, so I believe this will be analogous to your old Connect timeout. This should be fast so you normally would only want to specify a few seconds to wait (30 or less), not 12 hours.
The WCF CloseTimeout is for when a Close Channel message is sent, and this is how long to wait for an acknowledgement. This may not have an equivalent in your old architecture. Again, this should be fast and should only need a few seconds.
The WCF SendTimeout (for the client config) essentially covers the time for the Client to send the message to the service, and to receive back the response (if any). This would correspond to your old Request timeout. This may need to be for several minutes if your server takes a while to process things.
The WCF SendTimeout (for the server config) is for when you want callbacks, so that the Server knows how long to wait for acknowledgement that its callback was received.
The WCF ReceiveTimeout does not apply to client-side configuration. For Server-side config the ReceiveTimeout is used by ServiceFramework layer to initialize the session-idle timeout (to be honest I don't really know what that is)
This MSDN discussion may be helpful http://social.msdn.microsoft.com/Forums/vstudio/en-US/84551e45-19a2-4d0d-bcc0-516a4041943d/explaination-of-different-timeout-types?forum=wcf
As a final note, having really big timeout values isn't a good idea unless you definitely have long running requests. This is because you can run out of available resources on your server if the client isn't closing the connections properly.
I have a WCF service with a NetTcpBinding running with about 100 clients. The clients regulary poll information from the server and after a while the service does not respond anymore.
Looking at netstat, I can see many connections that are in the CLOSE_WAIT state.
This is my binding:
<netTcpBinding>
<binding name="default" maxReceivedMessageSize="2147483647" maxBufferPoolSize="2147483647" maxConnections="10000">
<readerQuotas maxDepth="2147483647" maxStringContentLength="2147483647" maxArrayLength="2147483647" maxBytesPerRead="2147483647" maxNameTableCharCount="2147483647" />
</binding>
</netTcpBinding>
I have also tried to change the values of closeTimeout from the default of 00:01:00 to 00:00:10, but with no effect.
The machine is a Windows Server 2008 R2 64bit.
Update:
I have added a ServiceThrottlingBehavior now, but the result is still the same.
new ServiceThrottlingBehavior
{
MaxConcurrentCalls = 1000,
MaxConcurrentInstances = 1000,
MaxConcurrentSessions = 1000
};
Update2
I have set the SessionMode to NotAllowed and changed the binding to streamed.
Any ideas what I could do to improve performance or to figure out the problem?
From your description, it seems: 1. initially the clients were able to connect to your server with no problem, so this rules out configuration problem 2. After a while server stopped responding, but you didn't say how long, and how big is the request rate, and whether the server stopped responding at all, or only intermittently responding. Based on this one possibility is that something is wrong on the server side. Did you noticed anything unusual on the server side? Things to look for is:
Thread count -- was the thread pool being depicted (as some settings
may set a cap on thread pool thread)? Especially try a fresh launch
of the server and observe the thread count till it stopped
responding, any pattern there? You may have dead locks, long
blocking operations etc. which holds thread for too long.
Memory -- is there a problem with memory leaks?
Is it a self hosting service? Do you have proper code to catch ServiceHost.Faulted event (and
restart the service)? If a ServiceHost is faulted, it'll not respond
to any requests.
See what WCF performance counter tells you, especially the queue size and number of active connections. From the performance counter, you'll know whether the service is taking any request, or if your throttling
configurations are necessary at all.
The ultimate diagnostic tool: turned on service side WCF tracing? Open a trace file will
definitely tell you what happened with a request. If you see any
exception in the tracing file, you'll find your root cause.
It sounds like your clients are never disconnecting.
Are you sure your clients are correctly closing the channel? Note that you should call ChannelFactory.Close, not just Dispose.
Set the receiveTimeout to something low to verify this is the problem.
Your client closed the connection by calling close(), which sent FIN to the server socket, which ACKed the FIN and the state of which now changed to CLOSE_WAIT, and stays that way unless the server issues close() call on that socket.
Your server program needs to detect whether the client has aborted the connection, and then close() it immediately to free up the port. How? Refer to read(). Upon reading end-of-file (meaning FIN is received), zero is returned.
You can detect if client is disconnected.
Any WCF channel implements ICommunicationObject , which provides events for the channel lifetime.
You should listen to the Faulted event
The sessionId can be accessed as always from the OperationContext.Current property.
When your client open the channel (on the first operation), register to the adequate events:
OperationContext.Current.Channel.Faulted += new EventHandler(Channel_Faulted);
OperationContext.Current.Channel.Closed += new EventHandler(Channel_Faulted);
and
void Channel_Faulted(object sender, EventArgs e)
{
Logout((IContextChannel)sender);
}
protected void Logout(IContextChannel channel)
{
string sessionId = null;
if (channel != null)
{
sessionId = channel.SessionId;
}
}
if the socket is disconnected, you should get a channel Fault event. The Closed event is raised when the client shuts down gracefully, the Faulted when it's unexpected (as in case of network failure).
check out following link.. Its kind a similar . It helped me ..
TCP Socket Server Builds Up CLOSE_WAITs Occasionally Over Time Until Inoperable
Verify that the router isn't the problem, since some consumer grade routers have a cap on the number of allowed open sockets/connections.
I have been technically testing a WCF service recently and have got to the point where, my lack of understanding is not allowing me to progress forward and find a solution to a timeout problem we see.
We are load testing a WCF Service which is hosted on IIS7 on windows server 2008. The system set up to fire the messages actually fires them at an application which is biztalk. Biztalk then process the messages and sends them on to the end point of the WCF Service. The WCF Serviceis also using .net 2.0 in it's app pool (I guess this means it could actually be 3.0 or 3.5 as these were not full releases?
We fire 40 messages within a seconds and 90% of them become timed out due to the send timeout on the client (biztalk). We thought at first this was strange because we expected the server's basic http binding receive timeout to trigger first, but it turned out that was set at 10 minutes and the client send timeout was set at 1Min and 30 Secs.
What I understand:
WCF Services have config files which have inside them behaviors and http bindings. The Server end point we are sending an XML message to is using BasicHtppBindings: Timeouts:Open/Close is 1 Minute, Send and Recieve are 10 minutes. The server's timeout which we know are involved so far is: sendtimeout: 1 minute.
I understand WCF's architecture works by creating an instance of either a channel factory or service host and creates a channel stack which contains the behaviors and binding settings from the config as channels. There is a TransportAdaptor which is used to move the xml message once it has been processed through the channel stack.
I understand from IIS that http.sys handles the incoming requests. It passes requests to the workerprocess and when that is busy, it places requests onto the kernel mode queue? I understand there some machine.config settings that can be set to increase this queue/limit this queue?
I also know about how to make an app pool into a webgarden and I have read you can increase the number of threads per core, from the default of 12; this is don e via a registry setting or a later on in .net a web config change.
I just read about InstanceContextMode and how it can effect the server's service too... But I'm unsure what that is set to in this case.
We recorded some perforamance counters, .net ones and I noticed the number of current requests minus the (Queued+Disconnected) = 12. Which indicates we are using 1 core? and the number of threads of on that core is set to 12.
Can anyone help me for a clearer picture and help piece my knowledge with some extra into something that is more complete?
The WCF Behavior has a throttle setting. Here is an example (grabbed from msdn):
<service
name="Microsoft.WCF.Documentation.SampleService"
behaviorConfiguration="Throttled" />
..... .....
<behaviors>
<serviceBehaviors>
<behavior name="Throttled">
<serviceThrottling
maxConcurrentCalls="1"
maxConcurrentSessions="1"
maxConcurrentInstances="1"/>
</behavior>
</serviceBehaviors>
By default (if not specified), the service is throttled to 10 concurrent calls.
I find that a sensible production setting for high volume clients running short calls is more like 100. Of course it depends on your implementation, but the defualt definitely hurts performance on my test and production systems.
I need to create a service which can process queued requests on a configured time interval. For example go to the web and get financial data from a site the requires we limit requests to once per second. I am new to WCF and I am not sure if (1) WCF with MSMQ a proper choice for implementing this? and (2) if so what is the best mechanism for enforcing the interval? a thread wait? a timer (not sure how that would work).
There's nothing built into WCF that would allow you to handle this explicitly, so you'd still need to do all the work yourself.
While your service could certainly process requests from MSMQ, the MSMQ listeners in WCF will pick and process messages as soon as possible; you can't configure them to process messages every X seconds only (you could fake it given the right tools, but seems to me it wouldn't be all that great).
One option if your delay between processing requests isn't very short, would be to use an intermediate queue to hold pending requests. That is, whatever sends the real requests writes them to a queue nobody is directly listening to (queue A), while your WCF service listens on a differet queue (queue B). Then, have something else (could be as simple as a script run from task scheduler) that runs once every X seconds/minutes/whatever and moves just 1 message from queue A to queue B, thus triggering the actual WCF service to run.
WCF and MSMQ are a great team! Definitely worth checking out.
The part that WCF doesn't provide out of the box is the "check every x seconds". The best approach here would be to host your WCF service inside a Windows NT Service, and have a timer inside the NT Service that goes to check the MSMQ queue only once every x seconds. Shouldn't be too hard to implement, really. The beauty is: you can very easily self-host a WCF Service inside a NT Service - just a few lines of code, and you get complete control over what's happening, and when. See the MSDN docs on How to Host a WCF service in a managed application for details.
Resources:
Tom Hollander's blog post series on MSMQ, WCF, IIS: Getting them to play nice
Motley Queue: MSMQ and WCF Getting Started
SOAizing MSMQ with WCF (and why it's worth it)
Or you could just use a window service to consume the messages instead. If you are not using the WCF functionality of consuming a message as soon as it is posted, then you probably have no reason to use wcf in the first place