IIS 7 Application Pool - Can't connect to WCF Service - wcf

Greetings,
I have to following problem. I have a WCF Service which runs under IIS7. Application connects to it and WCF Service makes some requests to DB. I have notice in activity monitor in SQL Server 2005 that after there are exactly 102 active connections, the application pool in IIS7 hangs. After this I can't connect to my WCF Service. Then only IIS7 restart helps.
For connection I am using ChannelFactory, it's closed after each request. I've also introduced code like this to be sure that Channel is closed:
catch (FaultException)
{
Factory.Abort();
return null;
}
catch (CommunicationException)
{
Factory.Abort();
return null;
}
catch (TimeoutException)
{
Factory.Abort();
return null;
}
catch (Exception ex)
{
Factory.Abort();
return null;
}
finally
{
Factory.Close();
Factory.Abort();
}
I have also the following behvavior for my service class:
[ServiceBehavior(InstanceContextMode= InstanceContextMode.Single, ConcurrencyMode=ConcurrencyMode.Multiple, AutomaticSessionShutdown=true)]
I have also the following in my service web.config file:
<serviceBehaviors>
<behavior name="Server.Service1Behavior">
<dataContractSerializer maxItemsInObjectGraph="2147483647"/>
<serviceThrottling maxConcurrentCalls="2147483647"
maxConcurrentSessions="2147483647"
maxConcurrentInstances="2147483647" />
I tried everything. Please help me because user's can't work like this. Why it's happen that after 102 connections to DB application pool hangs?
Here is the code that calls on the database
internal SqlConnection CheckIfConnectionOpen()
{
if (_Connection.State != ConnectionState.Open)
{
_Connection.Open();
}
return _Connection;
}
using (SqlCommand cmd = new SqlCommand(query, _Connection))
{
CheckIfConnectionOpen();
//some parameters for sqlcommand here and execute nonQuery or execute reader
}
Can someone please help me with this because I am still looking for a solution

On a long shot the link below explains a situation when the app pool crashes it does not restart. If you can get it to restart that should decrease the severity of the issue.
http://i.nconspicuo.us/2008/06/25/iis7-on-windows-server-2008-503-service-unavailable-error-application-pool-stops-unexpectedly/
You mentioned you have checked the Channel is closed, it might be good to confirm the DB connections are closed too.
Hope this helps!

You could set up a web farm (multiple processes for the same IIS) which would help minimise the issue, and depend on a single process less (if one dies and restarts, the others can be there to hold up the fort until it restarts.

As an aside, your code above is equivalent to:
catch (Exception ex)
{
Factory.Abort();
return null;
}
finally
{
Factory.Close();
Factory.Abort();
}
And it's just as bad. You probably want to log the exception somewhere so that you know what happened.
I'd like to see the code that calls on the database. I'd be concerned you might not be cleaning up properly.

Related

How to catch error when message have been sent from JMS

I am sending an message through my standalone application that uses EJB MDB to communicate to my other application server that is running on JBOSS server.My application server is connected to a MSSQL server. In certain scenario, connection to the database is lost on application server side and we get following error -
Connection is reset.
Later , when i try to send message i don't get any error at my standalone EJB MDB logs and the process just stops executing.I get error log on application server side logs but same logs don't get propagated to my EJB MDB error logs.
As per my understanding, when db connection is lost all the ejb bean present in jboss container get nullified too.(I could be wrong here, i am new to EJB).
I tried implementing below code in my code that use to send message -
QueueConnection qcon = null;
#PostConstruct
public void initialize() {
System.out.println("In PostConstruct");
try {
qcon = qconFactory.createQueueConnection();
} catch (Exception e) {
e.printStackTrace();
}
}
#PreDestroy
public void releaseResources() {
System.out.println("In PreDestroy");
try {
if(qcon != null)
{
qcon.close();
}
if(qcon== null){
throw new Exception(" new exception occured.");
}
} catch (Exception e) {
e.printStackTrace();
}
}
I was in a impression that Queueconnection object will be nullified, when our db connection have been lost(as we are creating bean and making connection for message). But it doesn't seem to work.
I did found a way to call back my application after sending message. I used a separate temporary queue and used setJMSReplyTo method to set the reply destination. More info could be obtained from this
link. Hope this helps others.

How can my WCF service recover from unavailable message queue?

I have a WCF service that receives messages from the Microsoft Message Queue (netMsmqBinding).
I want my service to recover if the message queue is unavailable. My code should fail to open the service, but then try again after a delay.
I have code to recognize the error when the queue is unavailable:
static bool ExceptionIsBecauseMsmqNotStarted(TypeInitializationException ex)
{
MsmqException msmqException = ex.InnerException as MsmqException;
return ((msmqException != null) && msmqException.HResult == (unchecked((int)(0xc00e000b))));
}
So this should be straightforward: I call ServiceHost.Open(), catch this exception, wait for a second or two, then repeat until my Open call is successful.
The problem is, if this exception gets thrown once, it continues to be thrown. The message queue might have become available, but my running process is in a bad state and I continue to get the TypeInitializationException until I shut down my process and restart it.
Is there a way around this problem? Can I make WCF forgive the queue and genuinely try to listen to it again?
Here is my service opening code:
public async void Start()
{
try
{
_log.Debug("Starting the data warehouse service");
while(!_cancellationTokenSource.IsCancellationRequested)
{
try
{
_serviceHost = new ServiceHost(_dataWarehouseWriter);
_serviceHost.Open();
return;
}
catch (TypeInitializationException ex)
{
_serviceHost.Abort();
if(!ExceptionIsBecauseMsmqNotStarted(ex))
{
throw;
}
}
await Task.Delay(1000, _cancellationTokenSource.Token);
}
}
catch (Exception ex)
{
_log.Error("Failed to start the service host", ex);
}
}
And here is the stack information. The first time it is thrown the stack trace of the inner exception is:
at System.ServiceModel.Channels.MsmqQueue.GetMsmqInformation(Version& version, Boolean& activeDirectoryEnabled)
at System.ServiceModel.Channels.Msmq..cctor()
And the top entries of the outer exception stack:
at System.ServiceModel.Channels.MsmqChannelListenerBase`1.get_TransportManagerTable()
at System.ServiceModel.Channels.TransportManagerContainer..ctor(TransportChannelListener listener)
Microsoft have made the source code to WCF visible, so now we can work out exactly what's going on.
The bad news: WCF is implemented in such a way that if the initial call to ServiceModel.Start() triggers a queueing error there is no way to recover.
The WCF framework includes an internal class called MsmqQueue. This class has a static constructor. The static constructor invokes GetMsmqInformation, which can throw an exception.
Reading the C# Programming Guide on static constructors:
If a static constructor throws an exception, the runtime will not invoke it a second time, and the type will remain uninitialized for the lifetime of the application domain in which your program is running.
There is a programming lesson here: Don't put exception throwing code in a static constructor!
The obvious solution lies outside of the code. When I create my hosting service, I could add a service dependency on the message queue service. However, I would rather fix this problem with code then configuration.
Another solution is to manually check that the queue is available using non-WCF code.
The method System.Messaging.MessageQueue.Exists returns false if the message queue service is unavailable. Knowing this, the following works:
private const string KNOWN_QUEUE_PATH = #".\Private$\datawarehouse";
private static string GetMessageQueuePath()
{
// We can improve this by extracting the queue path from the configuration file
return KNOWN_QUEUE_PATH;
}
public async void Start()
{
try
{
_log.Debug("Starting the data warehouse service");
string queuePath = GetMessageQueuePath();
while(!_cancellationTokenSource.IsCancellationRequested)
{
if (!(System.Messaging.MessageQueue.Exists(queuePath)))
{
_log.Warn($"Unable to find the queue {queuePath}. Will try again shortly");
await Task.Delay(60000, _cancellationTokenSource.Token);
}
else
{
_serviceHost = new ServiceHost(_dataWarehouseWriter);
_serviceHost.Open();
return;
}
}
}
catch(System.OperationCanceledException)
{
_log.Debug("The service start operation was cancelled");
}
catch (Exception ex)
{
_log.Error("Failed to start the service host", ex);
}
}

how do i get an exception out of a web service?

I have a web service that runs perfectly when i reference it from within the project solution. As soon as i upload it to the remote server, it starts blowing up. Unfortunately, the only error message I get is on the client side "faultexception was unhandled by user code". Inside of the web service, I have exceptions handled in all of the methods, so I'm pretty sure it's getting caught somewhere, but I don't know how to see it. I suspect that the problem is permissions related, but I can't see where it's happening.
I tried placing an error message into object returns, but it's still not making it out; something like this:
public bool SetDirectReports(ADUser user)
{
try
{
var adEntry = new DirectoryEntry(string.Format("LDAP://<GUID={0}>", user.Guid), "administrator", "S3cur1ty");
if (adEntry.Properties["directReports"].Count > 0)
{
user.DirectReports = new List<ADUser>();
foreach (string directReport in adEntry.Properties["directReports"]) //is being returned as full distinguished name
{
var dr = new DirectoryEntry(string.Format("LDAP://{0}", directReport), "administrator", "S3cur1ty");
user.DirectReports.Add(GetUserByGuid(dr.NativeGuid));
}
return true;
}
else
{
user.DirectReports = new List<ADUser>();
return false;
}
}
catch (Exception ex)
{
user.HasError = true;
user.ErrorMessage = "Error setting direct reports: " + ex.Message;
return false;
}
}
but its' still not catching. I was hoping for a better approach. I'm not sure if I could add something that would output the exception to the console or what. Any help would be appreciated. TIA
P.S. this isn't necessarily the method thats crashing, there's a web of them in the service.
You should dump all of your exceptions to a log file on the server side; exposing error information to the client is a potential security risk, which is why it's turned off by default.
If you really want to send exception information to the client, you can turn it on. If you are using a WCF service you should set the "includeExceptionDetailsInFaults" property on for the service behavior, as described in this MSDN article on dealing with unhandled exceptions in WCF. Once you do so, you will have a property on the FaultException called Detail that should itself be a type of Exception.
For better error handling you should also take a look at typed faults using the FaultContract and FaultException<> class; these have the benefit that they don't throw the channel into a faulted state and can be handled correctly:
try
{
// do stuff here
}
catch (Exception ex)
{
var detail = new CustomFaultDetail
{
Message = "Error setting direct reports: " + ex.Message
};
throw new FaultException<CustomFaultDetail>(detail);
}
If you are using an ASP.NET Web Service, you should set the customErrors mode to "Off" in your web.config. This will send back the entire exception detail as HTML, which the client should receive as part of the SOAP exception that it receives.
The error your are seeing ("faultexception was unhandled by user code") is happening because this is a remote exception and it is standard behavior to only display exceptions on the local computer by default. In order to make it work how you intend, you need to change the customErrors section of the web.config and set it to Off
UPDATE: I found a related question: c# exception not captured correctly by jquery ajax
(Three years later..)
Here's the solution I came up with, along with some sample WCF code, and Angular code to catch, and display the exception message:
Catching exceptions from WPF web services
Basically, you just need to wrap your WCF service in a try..catch, and when something goes wrong, set a OutgoingWebResponseContext value.
For example, in this web service, I've slipped in an Exception, which will make my catch code set the OutgoingWebResponseContext value.
It looks odd... as I then return null, but this works fine.
public List<string> GetAllCustomerNames()
{
// Get a list of unique Customer names.
//
try
{
throw new Exception("Oh heck, something went wrong !");
NorthwindDataContext dc = new NorthwindDataContext();
var results = (from cust in dc.Customers select cust.CompanyName).Distinct().OrderBy(s => s).ToList();
return results;
}
catch (Exception ex)
{
OutgoingWebResponseContext response = WebOperationContext.Current.OutgoingResponse;
response.StatusCode = System.Net.HttpStatusCode.Forbidden;
response.StatusDescription = ex.Message.Replace("\r\n", "");
return null;
}
}
What is brilliant about this try..catch is that, with minimal changes to your code, it'll add the error text to the HTTP Status, as you can see here in Google Chrome:
If you didn't have this try..catch code, you'd just get an HTTP Status Error of 400, which means "Bad Request".
So now, with our try..catch in place, I can call my web service from my Angular controller, and look out for such error messages coming back.
$http.get('http://localhost:15021/Service1.svc/getAllCustomerNames')
.then(function (data) {
// We successfully loaded the list of Customer names.
$scope.ListOfCustomerNames = data.GetAllCustomerNamesResult;
}, function (errorResponse) {
// The WCF Web Service returned an error
var HTTPErrorNumber = errorResponse.status;
var HTTPErrorStatusText = errorResponse.statusText;
alert("An error occurred whilst fetching Customer Names\r\nHTTP status code: " + HTTPErrorNumber + "\r\nError: " + HTTPErrorStatusText);
});
Cool, hey ?
Incredibly simple, generic, and easy to add to your services.
Shame some readers thought it was worth voting down. Sorry about that.
You have several options:
1) If you are using WCF, throw a FaultException on the server and catch it on the client. You could, for instance, implement a FaultContract on your service, and wrap the exception in a FaultException. Some guidance to this here.
2) You could use the Windows Server AppFabric which would give you more details to the exception within IIS. (requires some fiddling to get it working, though)
3) Why not implement some sort of server-side logging for the exceptions? Even if to a file, it would be invaluable to you to decipher what is really happening. It is not a good practice (especially for security reasons) to rely on the client to convey the inner workings of the server.

WCF FaultException - crashes App Pool when "re-throwing" from internal WCF call

I have a WCF service that's hosted in IIS, and uses a WS HTTP binding (the external service). This service ends up calling a second WCF service that's hosted in a Windows service, over Net TCP (the internal service). When the internal service throws a FaultException, the external service crashes rather than throwing it to the client. All the client sees is the connection being forcibly closed.
The internal service uses the Enterprise Library Validation Application Block to validate the incoming messages. When validation errors occur, the service throws a FaultException<ValidationFault>.
Both the internal and external service have a [FaultContract(typeof(ValidationFault)] attribute in the service contract. If I change the external service to just immediately throw a new FaultException<ValidaitonFault>, this gets back to the client fine. I can catch the exception from the internal service in the external service, but if I try to re-throw it, or even wrap it in a new exception and throw that, the whole Application Pool in IIS crashses. I can't see anything useful in the event log, so I'm not sure exactly what the problem is.
The client object the external service uses to communicate with the internal service is definitely being closed and disposed of correctly. How can I get the internal service's faults to propagate out to the client?
updated:
Below is a simplified version of the external service code. I can catch the validation fault from the internal service call. If I throw a brand new FaultException<ValidationFault>, everything is fine. If I use the caught exception, the connection to the external client is broken. The only difference I can see is when debugging the service - trying to use the caught exception results in a message box appearing when exiting the method, which says
An unhandled exception of type
'System.ServiceModel.FaultException`1'
occurred in mscorlib.dll
This doesn't appear if I throw a brand new exception. Maybe the answer is to manually copy the details of the validation fault into a new object, but this seems crazy.
public class ExternalService : IExternalService
{
public ExternalResponse DoSomething(ExternalRequest)
{
try
{
var response = new ExternalResponse();
using (var internalClient = new InternalClient())
{
response.Data = internalClient.DoSomething().Data;
}
return response;
}
catch (FaultException<ValidationFault> fEx)
{
// throw fEx; <- crashes
// throw new FaultException<ValidationFault>(
// fEx.Detail as ValidationFault); <- crashses
throw new FaultException<ValidationFault>(
new ValidationFault(new List<ValidationDetail> {
new ValidationDetail("message", "key", "tag") }),
"fault message", new FaultCode("faultCode"))); // works fine!
}
}
}
I have almost the exact design as you and hit a similar issue (not sure about a crash, though!).
If I remember correctly, even though the ValidationFault is a common class when the Fault travels over the wire the type is specific to the WCF interface. I think this is because of the namespace qualifiers on the web services (but this was a while back so I could be mistaken).
It's not terribly elegant, but what I did was to manually re-throw the exceptions:
try
{
DoStuff();
}
catch (FaultException<ValidationFault> fe)
{
HandleFault(fe);
throw;
}
...
private void HandleFault(FaultException<ValidationFault> fe)
{
throw new FaultException<ValidationFault>(fe.Detail as ValidationFault);
}
Well, it works if I do this, but there must be a better way...
This only seems to be a problem for FaultException<ValidationFault>. I can re-throw FaultException and FaultException<SomethingElse> objects with no problems.
try
{
DoStuff();
}
catch (FaultException<ValidationFault> fe)
{
throw this.HandleFault(fe);
}
...
private FaultException<ValidationFault> HandleFault(
FaultException<ValidationFault> fex)
{
var validationDetails = new List<ValidationDetail>();
foreach (ValidationDetail detail in fex.Detail.Details)
{
validationDetails.Add(detail);
}
return new FaultException<ValidationFault>(
new ValidationFault(validationDetails));
}

Closing WCF connection

We are using WCF service
on the client side we are planning to explicitly close the connection
It seems there are more then one way of closing
Sample1:
In the finally block of the WCF service consumption use
if (client.State == CommunicationState.Faulted)
{
client.Abort();
}
client.Close();
Since if the service is in fault state we will not be able to call close()
Sample2:
using(ClientProxy proxy = new ClientProxy())
{
//call your service methods
}
in sample2 i am not sure what will happen if the service is in fault state, will it throw error closing the connection?
You have all the necessary information at hand - the resulting Best Practice to use and properly close/abort all your WCF client proxies would be:
YourClientProxy clientProxy = new YourClientProxy();
try
{
.. use your service
clientProxy.Close();
}
catch(FaultException)
{
clientProxy.Abort();
}
catch(CommunicationException)
{
clientProxy.Abort();
}
catch (TimeoutException)
{
clientProxy.Abort();
}
Catching the FaultException handles all cases when the service responsded with an error condition (and thus your channel is in a faulted state), and CommunicationException will handle all other communication-related exceptions that can occur, like network connectivity dropping etc.
The approach with the using() block won't work, since if an exception happens at the end of the block, when the Dispose() method calls the Close() method on the client proxy, you have no way to catching and handling that.
The 2nd sample using the "using" block is incorrect. The using block ensures that the Dispose method is called on the proxy object. The Dispose method in turn calls the Close method which will (try to) connect to the service which will throw an exception when the communication state is faulted. So your feelings/hunch are absolutely right. It would be nice if the proxy Dispose method used the code from your first sample but it doesn't so don't use the using block :)
In Juval Lowy's Excellent Programming WCF book he recommends:
try
{
ClientProxy clientProxy = new ClientProxy();
clientProxy.SomeMethod();
clientProxy.Close();
}
catch
{
proxy.Abort();
}
Use sample 1
Here is a good article on why you should not use using:
http://msdn.microsoft.com/en-us/library/aa355056.aspx