We have a wcf service hosted in IIS (Win 2012). We expect thousands of messages coming throughout the day, however the peak would be around 4K concurrent messages. All these are one way (async) requests and wcf service performs some processing which will take several seconds for each request.
- Is there any limitations on max concurrent requests that can be sent to wcf svc hosted on IIS? Does it depend on thread availability on server? Any settings that needs tweaking would be useful.
In another scenario, we have less number of async requests coming to wcf svc, however for each request service performs few things parallely (Parallel.ForEach). What would be maximum parallel threads available in this scenario and does it depend on any other factors? Any settings that needs tweaking would be useful.
Related
In order to improve performance* of a WCF service, one of the following can be done:
Use WCF inbuilt features (throttling etc)
Install multiple instances (separate websites in IIS) of the same service in the same machine.
I understand that these things are better tested than discussed but just wanted to get an opinion if someone has already tried both these approaches.
This service uses InstanceMode.PerSession and ConcurrencyMode.Multiple
Performance: This service handles data (MTOM encoded). There should not be any timeouts since clients will make synchronous calls to this service.
No, multiple endpoints from a single service won't help, as you describe it.
Yes, you can have a running WCF in IIS with multiple endpoints, but the same service is processing the requests whether they come into endpoints 1, 2, 3 or n. And since WCF requests are processed on their own threads, there's no benefit to adding extra end points.
Think of it this way: 10 requests come into a WCF service. Each request is processed on its own thread whether there are 10 endpoints or just 1. So there's no speed advantage gained by adding endpoints.
I've spent 2 years building industrial-scale WCF services. If you're worried about performance, the WCF service is the least of your worries. I've load tested a WCF service, sending 1000 concurrent users (each uploading multiple 157kb files) at a medium size (4 core) server; the server barely breaks a sweat while uploading 160 files/second.
If you're planning to build huge web service, the way to spread out the processing load is to have 1-n WCF web services fronted by a load balancer like F5. Then you can scale up to Amazon.Com size if you like.
Is there any tool for monitoring incoming and outgoing calls from a WCF service?
I do not mean MSMQ, I mean queued calls that the service is not able to handle cause of it is busy.
Some profile for Performance monitor perhaps?
Kind Regards
Martin
For WCF hosted on IIS you can use ASP.NET performance counters to see number of queued requests, current requests etc. for example:
ASP.NET\Requests Queued
Web Service\Current Connections
Although WCF has a lot of performance counters available, there is none indicating queues length.
If you are talking about queued messages content, there is no easy way. The only I can think of is taking a full memory dump of the process and investigating it. But this is of course very intrusive, single shot task.
We have a WCF service which we are hosting on azure. It takes some xml and processes it in memory (no external calls/db etc and it takes about 150ms) and returns some xml.
We have been load testing it and when we run it on 1,2 and 4 core machines we can max out the processors and get around a max of 40 calls per second throughput (on the 4 core machine). However when we switch to an 8 core machine or two 4 core machines we still only get around 40 calls per second.
Why might I not be able to get more throughput when I scale up the number of machines doing the processing? I would expect adding more machines would increase my throughput fairly linearly, but it doesn't. Why not?
Not sure if Azure has specific throttling, but the .NET framework has a limit on the number of outgoing connections to the same address that can be active at a time. In this MSDN article called Improving Web Services Performance it mentions that the default value for this is 2.
Configure The maxconnection Attribute
The maxconnection attribute in Machine.config limits the number of concurrent outbound calls.
Note This setting does not apply to local requests (requests that originate from ASP.NET applications on the same server as the Web service). The setting applies to outbound connections from the current computer, for example, to ASP.NET applications and Web services calling other remote Web services.
The default setting for maxconnection is two per connection group. For desktop applications that call Web services, two connections may be sufficient. For ASP.NET applications that call Web services, two is generally not enough. Change the maxconnection attribute from the default of 2 to (12 times the number of CPUs) as a starting point.
<connectionManagement>
<add address="*" maxconnection="12"/>
</connectionManagement>
Note that 12 connections per CPU is an arbitrary number, but empirical evidence has shown that it is optimal for a variety of scenarios when you also limit ASP.NET to 12 concurrent requests (see the "Threading" section later in this chapter). However, you should validate the appropriate number of connections for your situation.
These limits are in place to prevent a single users from monopolizing all the resources on a remote server (DOS attack). Since this is a service running in Azure I would guess that they have throttling on their end to prevent a user from consuming all of their incoming connections from a single IP.
My next step would be to check and see if there is a concurrent connection limit for azure web roles (this thread suggests there is and it's configurable) and to either increase it. Otherwise I would try to perform my load test from multiple sources and see if you still experience the same limits.
Even if this question is a year old, I am still searching a good answer for this question. I appreciate any information that will lead me to fully understand this issue regarding low performances of communicating web services hosted on the same machine.
I am currently developing a system with several WCF Web Services that communicate intensively.
They are running under IIS7, on the same machine, each service being in a different Application Pool, with multiple workers in the Web Garden.
During the individual evaluation of each Web Service, I can serve 10000-20000 requests per minute, quickly and without any issues for the resource consumption (processor and memory).
When I test the whole system or just a subsystem formed by two Web Services I can't serve more than 2000 requests/minute.
I also observed that communication time between Web Service is a big issue (sometimes more than 10 seconds). But when testing with only 1000 requests per minute everything goes smoothly (connection time of no more than 60 ms).
I have tested the system both with SOAPUI and JMETER, but the times were computed based on system logs, not from the testing tools.
Memory and network aren't an issue (they are used very little).
Later on, I have tested the performance of 2 communicating WCF web services, hosted on two server and on the same server. It again seems that there is a bottleneck when the services are on the same machine, lowering the number of connection with from ten thousands to thousands; again, no memory or processor limiting.
As a note, I am working with quite big data in some cases and some of the operations needed are long ones.
I used perf.mon to see what's going on, for memory, processes, webservice, aspnet, etc. but I didn't see anything that could indicate what it's going wrong.
I also tried all the performance settings and tuning options I could find on the Internet.
Does someone know what can be wrong? Why the communication between Web Services could last so long? Why the Web Service which serves as an entry point in the system can accept 10000 requests/minute when is tested alone, but when communicating with another Web Service barely accepts 2000?
It's an IIS7 problem? Could my system perform better if each Web Service will be deployed on a different server?
I want to understand better how things internally function (IIS and WCF services) to improve performances for current and future systems.
You could try to collect data from WCF performance counters : concurrent calls, instances, duration, ... In addition, WCF throttling provides some properties that you can use to limit how many instances or sessions are created at the application level. Performance of the WCF service can be improved by creating proper instance.
Finally, in load testing, there are many configuations to apply to different component : max concurrent http connection, IIS limits, having many load clients... You load test is invalidated because of this.
I have already asked a similar question here: WCF Service calling an external web service results in timeouts in heavy load environment but I've got a better idea now as to what's happening so posting a new question.
This is what is happening:
.NET client sends multiple requests at the same time to a WCF service (if it helps - I'm replicating this scneario by using Visual Studio Load Tests)
The client has got a "sendTimeout" set to 5 seconds
The WCF service receives it and start processing it. The processing involves sending a request to an external service which could take about 1 second to come back with a response
This is where I think the problem is: the client has sent many requests to the service and since the service is still busy processing the concurrent requests, some of the reqeusts from the client are timing out after 5 seconds
I have tried the following:
Changed the InstanceContextMode to PerCall
Increased the values of maxConcurrentCalls & maxConcurrentInstances
Increased the value of connectionManagement.maxconnection in machine.config
But none of that seems to be making any difference. Does anyone has any idea how can I ensure that I don't run into this timeout issue?
OK, you say WCF and that is not enough. What binding are you using and where are you hosting it? If you are using IIS, the could be different underlying problem than self-hosting.
The likely reason is the small number of ThreadPool size. You can use ThreadPool.SetMaxThreads() to change this but beware this is a sensitive value. Have a look here.
Check out the following link:
http://weblogs.asp.net/paolopia/archive/2008/03/23/wcf-configuration-default-limits-concurrency-and-scalability.aspx
I'm not sure what you're trying to achieve. Since the WCF service is doing a time consuming operation, you can't overload it and expect it to function. You can do the following (check the link about to set the following):
Increase the receiving capacity of the wcf service
Increase the send timeout of the service
Increase the send timeout of the client
Increase the receive timeout of the client
Limit the outgoing connections to the wcf service
The best and most robust option would be to configure and use MSMQ with the WCF service.