For a few weeks now I've been having a really weird problem. I have a couple of services which work just fine when self-hosted in a command line app. However in IIS+AppFabric I cannot access one of the services - I get TimeoutException and am pretty sure that the call doesn't even make it to the service (all services have an aspect to log all calls before doing anything). Note that both services are configured identically with regards to bindings and behaviors by code. I tried many things like putting them on different app pools, disabling some of the transports... And what is really strange that if both services are in one app pool - one of the services works but if I put them on separate threads - the other service times-out. It really drives me nuts...
Also I see pretty often events in the system event log: "A process serving application pool 'Authorization Management' suffered a fatal communication error with the Windows Process Activation Service. The process id was '11852'. The data field contains the error number." The error number is 0x80070218. After the event the service host initializes without problems (I can see my own info log messages) however the service is unreachable.
Does this ring a bell to anyone?
Thanks!
It turned out that I had a bug in the initialization of the services' hosts. I was trying something, and when I removed the try code, apparently I didn't delete the first line which was locking some resource.
Anyway, it is a good lesson. Nevertheless, if your services do not work, your initialization might be buggy...
Sorry about the noice.
Related
I'm working on my fourth or fifth implementation of a WCF service over MSQM with IIS/WAS activation. And I was never able to make it work properly. It's always the same story: my services are activated only if the IIS web site was interacted some other way (like servicing the service metadata page at /somewhere/myService.svc). Suddenly, if the only thing happening is sending messages into the queue, my services stop to process messages, and restart as soon as I visit the .svc page...
It's a so common pattern for me, that I also came to a common solution: scheduling a job (every few minutes) that runs a powershell script that access that page. Quite simple, but not very elegant. And, further more, unnecessary in theory.
This happened over different IIS versions (7.0 and 7.5), over various Win 2008 service packs and releases and with server in AD domains or workgroups. I think I've read every bits on the web about this, especially MSDN and microsofties blog, so binding configuration, MSMQ permissions, and all the other small details you can discover here and there are set up.
So the question: does anybody was successful with WAS over MSMQ?
We have some WCF services implemented in an IIS application, communicating over net.tcp on the default port (808), using the Microsoft Net.Tcp Port Sharing Service, throwing an error on production servers. When I instantiate a connection to the first of the services, I get back an exception:
The server at <URL> rejected the session-establishment request. All the other services respond fine.
But it runs fine on our test servers.
I initially thought there was something wrong with the particular service that was failing, but I tried rearranging the list of services into a different order, and it SEEMS to always be the first service that I hit that fails. (I say SEEMS because it think once in the early iterations of testing, I saw it happen on the second service that it hit. But I haven't been able to reproduce that.)
I've looked at application startup delays, and that doesn't seem to be the problem, because I can come back and run the test again as soon as it finishes - a delay of only a minute or two - and get the same error. Also, in the lower level environments, there is a start up delay of probably 30 seconds to a minute, but the result still comes back as expected.
I've tried accessing the services over http from INetManager, and I get intermittent failures on all the services - a particular service will return a yellow screen of death on on invocation, then come up with the expected link to the WSDL on the next one seconds later.
I'm completely at a loss to explain this behavior, or how to resolve it. I've googled the error message, and not found anything helpful. It may be a configuration issue - the production servers are newly provisioned VM's, and we may not have the config exactly right (whereas all the lower level environments have been running this and other similar apps for some time), but I have not idea what to look for. I've looked at the properties of the app pool that the app is running on and compared it to the lower level environments without finding any differences.
If somebody can point me in the right direction, you would have my undying gratitude.
Things I can find:
http://go4answers.webhost4life.com/Example/connect-busy-wcf-service-host-while-725.aspx:
MaxConcurrentSessions (default = 10) [Per-channel] The maximum number of sessions that a service can accept at one time. Only comes into play with session-based bindings (wsHttp or netTcp)"
http://blogs.infosupport.com/unable-to-generate-a-wcf-proxy-using-svcutil-but-retreiving-the-wsdl-works/
So in the end the trick is to add the additional right on the c:\windows\temp folder for your App Pool Identity [for the service to be able to generate metadata] to solve the problem.
Also, are timeouts or other limits configured and being hit? Give tracing a look and access the service using WcfTestClient and see if you can find underlying errors.
This may be a shot in the dark (I don't know much about the internals of WCF), but here goes...
I'm currently working with a legacy application at a client site and we're experiencing a persistent issue with a WCF service. The application is using the Microsoft Sync Framework 2.0 and syncing through the aforementioned service. The server-side implementation of the service has a lot of custom code in various states of "a mess."
Anyway, we're seeing an error on the client application most of the time and the pattern we're narrowing down centers around different users using the application on the same machine hitting the same service. It seems that the service and the client are getting out of sync in some way on an authentication level.
The error is discussed in an article here, and we're currently investigating the approach of switching from message layer security to transport layer security, which will hopefully solve the problem. However, we may be able to solve it in a less invasive manner if this question makes sense.
In the linked article, one of the suggestions was to forcibly terminate the connection if the specific exception is caught, try again, and if it fails again it wasn't because of this particular theory. Sounds good, and easy to implement. However, I find myself unable to say with confidence if the connection is being properly terminated.
The service operates through a custom interface, which is implemented on the server-side. The only thing that interface can do to end the connection is call EndSession() on the proxy itself, which calls EndSession() on the server which is a custom method.
So...
From a WCF service method, is there a way to properly and gracefully terminate the connection with the client in a way the client will like?
That is, in this custom EndSession() is there some last step I can take to cause the server to completely forget that this connection was open? Because it seems like when another user on the same machine tries to hit the service within the application, that's when it fails with the error in the linked article.
The idea is that, at the client side of things, code which calls EndSession() is followed by nulling out the proxy object, then a factory method is called to supply another one the next time it's needed. So I wonder if something additional needs to happen on the server side (and does by default in WCF were it not for all this custom implementation code) to terminate the connection on that end?
Like I said, a shot in the dark. But maybe in answers/discussions here I can at least further diagnose the problem, so any help is much appreciated. Thanks.
Unfortunately there are only really three ways in which a session can terminated
The client closes the proxy
The service's receiveTimeout is exceeded
before the client sends another
request
the service throws a
non-fault exception which will fault
the channel and so terminate the
session
if you don't want the client involved then you only have 2 and 3 neither of which end well for the client - they will get an exception in both situation on the next attempt to talk to the service.
You could use Duplex messaging and get the service to notify the client that its requires session termination - the client then gets an opportunity to close down the proxy gracefully but this is a cooperative strategy
I have some very odd situation at a client that wants me to maintain some of their software:
upon start/stop, the EventLog entries of a Windows Service are correctly appearing.
But during incoming WCF calls, they are not.
All EventLog creating is going through the same global static factory class.
When I put all classes in a console app, everything logs fine.
When sending those entiries to OutputDebugString, the same happens: windows Service Startup/Shutdown are coming through fine, but WCF execution not.
Where should I start digging?
These kinds of problems are hard to figure out. I would start by attaching the debugger to the WCF service and adding some breakpoints in the logging classes and see if the code actually stops. If not move the brake point to the service and see why it doesn't get into the logging code.
Given that the code works in a console app and only partly in a WCF service my first guess is that it could be a threading issue. But without more info that is just pure speculation.
My application has 50 service endpoints (such as /mysite/myService.svc). It's hosted in IIS. Intermittently (once every two or three days) a service stops responding. It's never the same service that hangs. While a service is hung, some of the other services work fine and some other are also hung.
All clients (from different computers) get this error:
ServiceModel.CommunicationException
Message: An error occurred while receiving the HTTP response to
https://server/mysite/myservice1.svc.
This could be due to the service endpoint binding not using the HTTP
protocol. This could also be due to an HTTP request context being
aborted by the server (possibly due to the service shutting down).
See server logs for more details.
No exceptions are raised by the server when the client attempts to call the service that is hung. All I have is that error on the client side.
I have to manually recycle the application pool to fix the problem.
Do you know what could be the cause? How can I investigate this issue? I'm willing to take a memory dump of the worker process when a service is hung but I would not know what to search for in the dump.
Update (Aug 13 2009): I have almost ruled out the idea that the server runs out of connections (see comment in Shiraz Bhaiji's answer). I might have a new lead: I log all server-side exceptions in a log file. So in theory, when this occurs on the client, no exceptions are raised on the server; otherwise I'd have proof of that in my logs. But what if an error does occur on the server but is happening at a low level where exceptions are not routed to my exception handling code? I have posted this question about scenarios where low level exceptions cannot be handled. I'll keep you informed of the progress of my investigation.
Sounds like you are running out of connections.
By default WCF has a timeout and therefore holds a connection open for 10 mins.
When you recycle the app pool all connections are closed, and therefore things work again.
To fix it check your code to make sure that you close connections / dispose of proxies.
To resolve this, we set establishSecurityContext to False on the binding.
I have not come across this particular issue but would suggest to turn on tracing/message logging for the WCF service in the config for the service and/or the client app (if you have control over that). I've done this in the last few days for a service that I needed to troubleshoot.
The MSDN link here is a good starting point.
Also see the table in this post for the varying levels of trace detail you can configure. There are several levels which can go from exception only logging to full message details. It is quite quick to set this up in the app.config file.
To parse the log file output use the SvcTraceViewer.exe that comes with the Windows SDK, which if you have it installed should be located in this folder: C:\Program Files\Microsoft SDKs\Windows\v6.0\Bin