I Have a .NET 4.5 WCF web service that consumes messages from a local private MSMQ queue running on Windows Server 2008 R2 with AppFabric installed.
This service reads the messages of the queue and processes the files referenced in the message, i have used AppFabric to throttle the service to process 16 concurrent messages, 8 on each AppPool worker process.
The AppPool is uses a domain account that has full privileges on the network share where the files to be processed are stored.
This service has been working fine for years, except in the last week the ~90% of the files its been asked to process have failed with either a UnauthorizedAccessException.
This behavior was exhibited across all of the services on that application server, no matter which file server the service was asked to process files from. Even files that had previously processed file were now failing.
After a long a fruitless weekend searching and hacking of various different things including:
Shared Folder Permissions and Quotas
Windows Licencing (CALs etc)
Firewalls
Various software patches to the Web app
I eventually discovered the actual issue by accident, whilst redeploying the Web app i noticed something odd. When i stopped the web app via the WCF menu in IIS, the messages continued to be consumed so i stopped the stopped the app pool running the web service, but the messages continue to be consumed, I though this might be due to the large latency added to MQMQ message state by the Distributed transaction service when lots of messages are rolled back to the poison message queue, so i went to lunch. When i came back the messages were still being consumed and process explorer confirmed the apppool running my server was no longer executing.
Something was clearly up but it was uncertain weather this was the cause, a symptom or a coincidence. The clincher was when i throttled my service back to only process one message at a time to see if the access, to the share was reaching some sort of limit, I noticed that failure rate went up to ~98%. This suggested that something else was processing the messages and failing, but also reporting those failures into my reporting system in a way only my application could.
I little further investigation revealed that the default application pool used to serve the default web site, was also executing my WCF web service but failing to access the files on the file server as the identity used to run the default application pool had no privileges the failures took less time the than the successful file processes therefor the slower i made my service go the more messages were failed by the default app pool.
The Cause
Whilst i was adjusting the throttling on my web app, i inadvertently set the throttling or the default web site that was the parent to the web application, i noticed this strait away and reset them back to the default value. What i hadn't realized at the time was that this had added a <system.servicemodel> tag to the web config of the default website. The outcome of which was that my default web site started to behave like a web application and for reasons i am yet to understand, it started to execute the functionality of its child web application, it may be related to the WAS activation, all i know is that i was most certainly not the desired behavior.
The Fix
I removed the <system.servicemodel> tag and its contents from the web.conf of the default website and removed net.msmq from its list of enabled protocols and everything is back to normal.
Related
I'm using Simple Injector in my WCF service. While running it from VS2010 everything is fine. However, when I publish it to my server using IIS 7, after some time (20 min, counted) my WCF loses all registered assemblies, modules, classes in container.
I guess IIS recycles the WCF Service Application Pool and drops my container registrations.
Can anyone help me on this?
While there exists many legitimate cases of using self-hosting WCF services, however, approaching self-hosting just because of IIS recycling may be counter productive.
Hosting in IIS gives you a lot benefit during development and daily operations, and I am not going to repeat what benefits which you could easily find out in google search.
So when IIS receive the first request to your application, it will launch a worker process named "w3wp.exe" according the settings in the application pool associated with your web app. And by default IIS will shutdown in 20 minutes of idle time. Check the Advanced Settings of the application pool, you will find a lot settings for the life cycle. You won't get such flexibility and robustness through self-hosting out of the box.
So basically you could have a few options provided you decide to stay with IIS hosting.
Change the Idle Time-out to 24-hours or even a month.
Write a small program or use cUrl to ping your application every 10 minute.
Leave it as it is
If you want to keep states during operations, save them in disk, then load them during next launch triggered by a request.
I'm working on my fourth or fifth implementation of a WCF service over MSQM with IIS/WAS activation. And I was never able to make it work properly. It's always the same story: my services are activated only if the IIS web site was interacted some other way (like servicing the service metadata page at /somewhere/myService.svc). Suddenly, if the only thing happening is sending messages into the queue, my services stop to process messages, and restart as soon as I visit the .svc page...
It's a so common pattern for me, that I also came to a common solution: scheduling a job (every few minutes) that runs a powershell script that access that page. Quite simple, but not very elegant. And, further more, unnecessary in theory.
This happened over different IIS versions (7.0 and 7.5), over various Win 2008 service packs and releases and with server in AD domains or workgroups. I think I've read every bits on the web about this, especially MSDN and microsofties blog, so binding configuration, MSMQ permissions, and all the other small details you can discover here and there are set up.
So the question: does anybody was successful with WAS over MSMQ?
I have an IIS/WAS hosted service running as myself in the development environment. It is activated using net.msmq binding. It is configured to use BufferedReceive (it is a WF service), which involves locking the queue. (Sub queues prefixed lock_ are created).
After some time and after the app pool is terminated, the lock queues are automatically removed.
After moving the system to production, with the environment AFAIK identical, but with the service running as a dedicated account, the lock queues are no longer cleaned up. If I stop the app pool, the lock queue remains, with messages.
The most obvious difference is the accounts under which the service runs but I can't find what kind of permissions are affecting this. Alternatively, is there some other configuration difference I am overlooking. Thanks for help.
OK this turns out to be a bug in IIS/WAS.
I had 3 app pools , 2 of which were auto-starting, one of them consuming the msmq. All shared the same AD domain identity, which was shared with the NetMsmqListenerActivator. That netmsmqlistener service was originally stopped, but after I had started it, the symptoms appeared.
Somehow IIS/WAS was creating multiple channels, sending messages to the wrong processes. I have worked around this by creating a separate identity, used now by the mq consuming web service and the netmsmqlistener, followed by a server reboote.
This was a painful problem that only occured on the production server. The lesson learned is to make sure that the production server shares the same hotfixes and patches as the test server .
UPDATE: Hotfix for Win 2k8 R2 SP1: http://support.microsoft.com/kb/2672277
I have a WCF service running under a service user on my local system. Every time I try to debug it is giving me a message Attach Security warning.
In Visual Studio, by default (even without attaching), I get this error:
Attaching to this process can potentially harm your computer. If the
information below looks suspicious or you are unsure, do not attach to
this process
Name: C:\Windows\System32\inetsrv\w3wp.exe
What is w3wp.exe? According to a Google search, I think it is related to IIS. But what does it do? What setting should be changed so that this won't give this message everytime I try to debug on my local system?
An Internet Information Services (IIS) worker process is a windows
process (w3wp.exe) which runs Web applications, and is responsible for
handling requests sent to a Web Server for a specific application
pool.
It is the worker process for IIS. Each application pool creates at least one instance of w3wp.exe and that is what actually processes requests in your application. It is not dangerous to attach to this, that is just a standard windows message.
Chris pretty much sums up what w3wp is. In order to disable the warning, go to this registry key:
HKEY_CURRENT_USER\Software\Microsoft\VisualStudio\10.0\Debugger
And set the value DisableAttachSecurityWarning to 1.
A worker process runs as an executables file named W3wp.exe
A Worker Process is user mode code whose role is to process requests,
such as processing requests to return a static page.
The worker process is controlled by the www service.
worker processes also run application code, Such as ASP .NET
applications and XML web Services.
When Application pool receive the request, it simply pass the request
to worker process (w3wp.exe) . The worker process“w3wp.exe” looks up
the URL of the request in order to load the correct ISAPI extension.
ISAPI extensions are the IIS way to handle requests for different
resources. Once ASP.NET is installed, it installs its own ISAPI
extension (aspnet_isapi.dll)and adds the mapping into IIS.
When Worker process loads the aspnet_isapi.dll, it start an
HTTPRuntime, which is the entry point of an application. HTTPRuntime
is a class which calls the ProcessRequest method to start Processing.
For more detail refer URL
http://aspnetnova.blogspot.in/2011/12/how-iis-process-for-aspnet-requests.html
w3wp.exe is a process associated with the application pool in IIS. If you have more than one application pool, you will have more than one instance of w3wp.exe running. This process usually allocates large amounts of resources. It is important for the stable and secure running of your computer and should not be terminated.
You can get more information on w3wp.exe here
http://www.processlibrary.com/en/directory/files/w3wp/25761/
My application has 50 service endpoints (such as /mysite/myService.svc). It's hosted in IIS. Intermittently (once every two or three days) a service stops responding. It's never the same service that hangs. While a service is hung, some of the other services work fine and some other are also hung.
All clients (from different computers) get this error:
ServiceModel.CommunicationException
Message: An error occurred while receiving the HTTP response to
https://server/mysite/myservice1.svc.
This could be due to the service endpoint binding not using the HTTP
protocol. This could also be due to an HTTP request context being
aborted by the server (possibly due to the service shutting down).
See server logs for more details.
No exceptions are raised by the server when the client attempts to call the service that is hung. All I have is that error on the client side.
I have to manually recycle the application pool to fix the problem.
Do you know what could be the cause? How can I investigate this issue? I'm willing to take a memory dump of the worker process when a service is hung but I would not know what to search for in the dump.
Update (Aug 13 2009): I have almost ruled out the idea that the server runs out of connections (see comment in Shiraz Bhaiji's answer). I might have a new lead: I log all server-side exceptions in a log file. So in theory, when this occurs on the client, no exceptions are raised on the server; otherwise I'd have proof of that in my logs. But what if an error does occur on the server but is happening at a low level where exceptions are not routed to my exception handling code? I have posted this question about scenarios where low level exceptions cannot be handled. I'll keep you informed of the progress of my investigation.
Sounds like you are running out of connections.
By default WCF has a timeout and therefore holds a connection open for 10 mins.
When you recycle the app pool all connections are closed, and therefore things work again.
To fix it check your code to make sure that you close connections / dispose of proxies.
To resolve this, we set establishSecurityContext to False on the binding.
I have not come across this particular issue but would suggest to turn on tracing/message logging for the WCF service in the config for the service and/or the client app (if you have control over that). I've done this in the last few days for a service that I needed to troubleshoot.
The MSDN link here is a good starting point.
Also see the table in this post for the varying levels of trace detail you can configure. There are several levels which can go from exception only logging to full message details. It is quite quick to set this up in the app.config file.
To parse the log file output use the SvcTraceViewer.exe that comes with the Windows SDK, which if you have it installed should be located in this folder: C:\Program Files\Microsoft SDKs\Windows\v6.0\Bin