I have a WCF service that works fine in IIS 7, however once deployed to Windows Server 2003, IIS6, I'm now getting - "The thread was being aborted" error message. This happens after a few minutes of the service running.
I've tried manually changing some timeout values and turned off IIS keep alives.
Any ideas on how to fix this problem would be welcomed.
Thanks
If you're having this problem - please read! Hopefully you'll save yourself A LOT of trouble knowing this. Get coffee first!
You might come from a traditional programming background, in fields not SOA related, and now you're writing SOA services with the mindset of "traditional programmer". Here are 4 of the most important lessons I've learnt since building SOA services.
Rule number 1
Try your very best not to write services that take an extended amount of time to complete. I know this can be VERY tricky to accomplish, but it is much more reliable to have smaller operations being called many times, than 1 long service performing all the work, then returning a response. For example recently I wrote a service which processed ALL tasks. Each task was stored as an XML file in the IIS site, and each task would export data to a system for example : SharePoint. At any given times during high volumes there could be up to 30 000 tasks waiting to be processed. Over the past 2 months I have yet to get it 100% reliable, this is after diving deep into timeout settings in IIS, AppPools and WCF bindings. Every now and again I would get - "The thread was being aborted" and no reason or explanation as to why this was happening. I exhausted all online knowledge bases, no one seemed the wiser. Eventually after not being able to fix the issues or even reproduce them in a reliable way, I opted for a complete rewrite. I changed my code to instead of process ALL tasks, process just 1 task at a time.
This essentially meant calling 1 web service 30 000 times, rather than calling it once, but performance wise, it is around the same. Each call issues a response quick, and does a lot less work. This has another benefit, I can provide instant feedback on each operation to the client. In the Long call, you get a response back right at the end and ALL at once.
You can also much more easily catch and retry a service call if it does fail, because you don't have to redo the whole call for each operation again, but simply the operation that failed.
Its easier to test too, not only because of the live feedback, but also because you can test 1 inner operation, without the overhead of the loop if you wanted to.
Lastly it adds better scaling if you plan on extending your application later, because you're broken things down into more manageable units of work. So for example: Before you had 1 service which processed ALL Tasks, now you have a web service that can process 1 TASK, because of this you can more easily extend the functionality if you needed to process 10 Tasks, or tasks by selection.
Rule Number 2
Don't upgrade your existing ASMX web services to WCF 3 just because you think its a better technology. WCF 3 is over architectured and not a real pleasure to work with, or deploy. If you need to go WCF, try your best to hold out for the version that ships with .net 4 of the framework, it seems to have been revamped. Another thing you will miss is that WCF has no test forms, so you can't just fire up a web browser quick to test your services. If you're like me - "Keep it simple stupid" Then WCF 3.5 will frustrate you.
Rule Number 3
IIS6 can be dodgy, if at all possible avoid having to host your services in IIS6, if you're after reliable services. I am not saying its impossible to achieve reliability in IIS6, but it requires a LOT of work, and a great deal of testing. If you're dealing with services that are critical, try avoid using a product developed in 2001.
Rule Number 4
Don't underestimate the development and testing required to create reliable SOA services. To be honest all I can say is it is a massive undertaking.
I thought I'd mention that this error is thrown by SharePoint when calling some functions from a user account. Those functions need to be run with SPSecurity.RunWithElevatedPrivileges
This answer shows up when searching for "wcf sharepoint Thread was being aborted" so hopefully this can be useful to someone since 'thread being aborted' isn't very useful of SharePoint to throw when its a permissions issue.
Related
I am trying to understand mocking the wcf services using SOAPUI.
Quoting from smartbear's blog this can be handy in
Rapid Web Services Prototyping
Generate a complete static mock implementation from a WSDL in seconds
and add dynamic functionality using Groovy. This allows you to
implement and test clients much faster than if you had needed to wait
for the actual solution to get build.
Client testing or development
Clients can be developed against the MockService and
tested without access to the live services.
So from this and some other blogs that I went through I understand that the Primary thing (by what I read) is to keep testing moving along before the service is available (I must say that I didn't get this actually. The service has to up before you sent up some mock requests and responses). Does it implies that the service should be available at the time we setting up this mocks so that we can play with them later on when it is actually not available?
Also can we say that there won't be a difference between saving multiple test cases for a given service and mocking given the service is up and running (after all its service it is supposed to be running).
I was on a large project where 3 different systems all traded data with services. It was the same WSDL (an "industry standard", very complicated beast that didn't fit either of our systems particularly well), and we all had clients for sending data to the servers of the other systems. Each dev/test team had to develop both a client and a service, and we didn't really understand mocks.
As you would expect, we all had our clients done before any of the services were ready. And testing couldn't do anything for months. And when they could finally test (the day after the devs were able to get data to finally flow), things were a mess.
So I can't time-travel back to 2010 and save myself, but I CAN save YOU.
Here's where you still don't get it:
The service has to up before you sent up some mock requests and
responses
You don't need the service to be up, built, coded, funded, or even approved. The SoapUI Mock Service IS the service. Rather, it's a very capable stand-in. So once you have a WSDL, you can built the mock service, create some sample responses, and hit it with a client (possibly other instances of SoapUI).
So why do this? Lots of reasons.
Multiple dev teams, on different timelines.
Testing can proceed, yes.
Avoid surprises down the road (after code) Just because we agreed to use this WSDL, doesn't mean that my data is what your system expects, and vice-versa. Let's find out now!
Examples:
For the Armor field, the WSDL just says "string" but our system allows 25 chars and yours allows 45.
We need you to send UserHighScore if you change LifetimeAchievements. Otherwise it gets reset.
I thought we agreed to put UserRank in the User Atributes tag, not the Power tag?
UserRank needs Effective Date, otherwise it causes our side to delete all of the UserRank history.
That's how it used to work.
Stop sending us the same data that we just sent you. When you UPPERCASE our data that you just received, that ISN'T A CHANGE that you need to tell us about.
Ideally, the system would be developed first with mock services and SoapUI. Once the WSDL is developed, stand up the mock service, then send submit sample requests via SoapUI. Both test and dev should be involved. Look at the data being sent from the SoapUI clients and build/script responses. Spend a few days developing test cases. Audit for invalid data, return realistic responses, make sure that you return failures as well as successes, and try to consider (and document) all expected failure scenarios, including time-outs (can be scripted with sleep() function in mock service). The time-out scenario can be used to simulate load, so that you can see the impact on clients and infrastructure (we were able to tip over Layer7 gateways by sending transactions at a higher rate than the service could handle, if we kept at it for 30 minutes).
So use the mockservices as a joint workshop to hammer out the details of what your service-oriented solution will look like, THEN code it up. You'll be glad you did.
I have a situation where I have two programs (one exe and one dll loaded into the process space of another third-party exe) communicating requests with each other using a local machine wcf service (using net named pipe binding). There's a third host exe that starts hosting the service. It all works great (so far anyways... I'm still learning), but I got to thinking about what would happen if the channel faults or the service times out. What would be the best practice for checking and handling faults as well as keep the channel alive?
In my case it will be up to the user to keep the applications open or close them and we do have those users who tend to keep them open overnight, over the weekend, etc... It seems to me this could open the possibility of a fault or loss of service and I don't have a clue how to recover. Any help would be greatly appreciated.
Firstly, why would you keep the channel alive indefinitely?
Imagine you are connecting to a database from which you want to read over the course of one day. Would you create the database connection in the morning and then close it in the evening?
It is relatively cheap to construct a channel in WCF for each call, unless you know you are going to be making multiple calls within a few seconds of each other, in which case you should reuse the channel.
EDIT
This post explains how to do it. It's pretty complicated and it may be easier to just set a huge timeout value for the binding in code (as suggested at the end of the post):
Do WCF Callbacks TimeOut
EDIT
There's tons of stuff on google about this: http://bit.ly/10ZPWE2
I'm trying to nail down a performance issue under load in an application which I didn't build, but have become very familiar with the workings of.
The architecture is: mobile apps call an ASP.NET MVC 3 website to get data to display. The ASP.NET site calls a third-party SOAP API using WCF clients (basicHttpBinding), caching results as much as it can to minimize load on that third party.
The load from the mobile apps is in the order of 200+ requests per second at peak times, which translates to something in the order of 20 SOAP requests per second to the third-party, after caching.
Normally it runs fine but we get periods of cascading slowness where every request to the API starts taking 5 seconds.. then 10.. 15.. 20.. 25.. 30.. at which point they time out (we set the WCF client timeout to 30 seconds). Clearly there is a bottleneck somewhere which is causing an increasingly long queue until requests can't be serviced inside 30 seconds.
Now, the third-party API is out of my control but they swear that it should not be having any issues whatsoever with 20 requests per second. So I've been looking into the possibility of a bottleneck at my end.
I've read questions on StackOverflow about ServicePointManager.DefaultConnectionLimit and connectionManagement, but digging through the source, I think the problem is somewhat more fundamental. It seems that our WCF client object (which is a standard System.ServiceModel.ClientBase<T> auto-generated by "Add Service Reference") is being stored in the cache, and thus when multiple requests come in to the ASP.NET site simultaneously, they will share a single Client object.
From a quick experiment with a couple of console apps and spawning multiple threads to call a deliberately slow WCF service with a shared Client object, it seems to me that only one call will occur at a time when multiple threads use a single ClientBase. This would explain a bottleneck when e.g. 20 calls need to be made per second and each one takes more than 50ms to complete.
Can anyone confirm that this is indeed the case?
And if so, and if I switched to every request creating it's own WCF Client object, I would just need to alter ServicePointManager.DefaultConnectionLimit to something greater than the default (which I believe is 2?) before creating the Client objects, in order to increase my maximum number of simultaneous connections?
(sorry for the verbose question, I figured too much information was better than too little)
I am creating an application in Silverlight using Ria Services that can take quite a bit of time once the service call is initiated. I've looked for ways to increase the WCF service timeout, but the more I think it through, this is not the right aproach.
What I would rather do is call the DomainContext and return right away, then have the client poll the server to find out when the long running query is complete.
I'm looking for a pattern or example of a good way to implement something like this. One potential issue that keeps coming to mind are that web services should not keep state between service calls, but this is exactly what I would be doing.
Any thoughts?
Thanks,
-Scott
Take a look at the WCF Duplex Service. It should solve your problem.
Can you make the service call take less time? If not, why not?
Typically when I've seen queries take this long, it either means the SQL running at the end isn't efficient enough, the SQL server has poor indexes, or the client is requesting far more data than they'll actually be able to use in a short period of time.
For example, instead of requesting 500 entities right away and showing a large list/DataGrid/whatever, why not request 10-50 at a time and have a paging UI that only requests the next batch when the user nedes it?
Take a look at signalr, it can run side by side with ria and it lets you push messages back to the client from the server.
I am in the process of writing an enterprise-level application utilizing WCF and NetTCP services. I chose NetTCP initially out of curiosity, but later determined it to be the best option for me since I can have services that are called that take 5+ hours to return results due to the amount of data crunching involved.
The way I currently spawn my services is a multi-step process. I have a configuration piece (using System.Configuration) that specifies some of the default stuff (port number, server name for clients connecting in, whether to enable HTTP as well as NetTCP, etc) and that has a collection of "services" underneath it. For example, here's what a basic one looks like:
<serverConfiguration tcpListenerPortNumber="60000" httpGetEnabled="true" httpListenerPortNumber="6000" serverName="localhost" retryEnabled="true" retryInterval="5" maxRetryAttempts="3">
<services>
<add virtualDirectory="Service1" applicationName="Service1" assembly="SampleService" type="SampleService.Service1" />
</services>
</serverConfiguration>
Basically what's happening here is my Windows service kicks off and looks at everything in the <services /> collection and spawns off a thread per service to speed startup time and each thread contains an AppDomain where the service truly lives so if a service has some kind of fault it doesn't bring the system down.
The "problem" I am running into is this application is hosting approximately 20 services and it takes a good 15-20 seconds for all the services to be up and running. I did the threading and AppDomain pieces to get it down to that value (used to take over a minute as each service was opened sequentially) but it still seems to me that this could actually go a lot faster.
Anyone have any suggestions? Google Bing has a plethora of examples for hosting one service but I'm not finding much out there for real-world applications (sadly "Hello World" just isn't appealing to end users). If you're currently hosting multiple services via a Windows Service and NetTCP, how are you doing it?
I figured it out finally and it had nothing to do with WCF or my configuration pieces after all. When I was creating the AppDomain I stole the code from another project we had that was much smaller in size and found that the section creating AppDomains was using the SingleDomain option. Changing it to MultiDomain made things go to >4 seconds for total load and memory usage dropped to ~40MB from ~150MB.
Thanks for the assistance though - at least it got me reviewing the code again!
I have three suggestions:
Firstly, if a call can take 5+ hours, I would consider a queuing / call back style architecture.
Consider splitting your services into 20 windows services, where each service runs in it's own windows services. This adds complexity and increases memory usage, but an individual service may be available faster.
Lastly, check the code that is in the consructor of the service, for any unneeded code.