I have two BizTalk development machines, one of which I am trying to get into a consistent state with the other. One of my tests checks the content of a SOAP fault response received from an orchestration - this is by design. The problem is that between the two machines, which as far as I know are configured identically and have the same application with the same configuration installed on them, handle faults differently as indicated by the stack trace of the caught exception within the orchestration.
The incoming fault expected is received from a Specify Later, request-response port with a SOAP 1.1 Fault operation configured. This is caught by a catch block which simply serializes the exception detail into another fault message and returns it to the caller. I can see that the fault is caught in the same way by the same catch block on both machines.
Baseline machine stack trace:
at Microsoft.BizTalk.Adapter.Wcf.Runtime.BizTalkAsyncResult.End()
at Microsoft.BizTalk.Adapter.Wcf.Runtime.BizTalkServiceInstance.EndOperation(IAsyncResult result)
at Microsoft.BizTalk.Adapter.Wcf.Runtime.BizTalkServiceInstance.Microsoft.BizTalk.Adapter.Wcf.Runtime.ITwoWayAsync.EndTwoWayMethod(IAsyncResult result)
at AsyncInvokeEndEndTwoWayMethod(Object , Object[], IAsyncResult )
at System.ServiceModel.Dispatcher.AsyncMethodInvoker.InvokeEnd(Object instance, Object[]&outputs, IAsyncResult result)
at System.ServiceModel.Dispatcher.DispatchOperationRuntime.InvokeEnd(MessageRpc&rpc)
at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage7(MessageRpc&rpc)
at System.ServiceModel.Dispatcher.MessageRpc.Process(Boolean isOperationContextSet)
Other machine's stack trace:
at Microsoft.BizTalk.Adapter.Wcf.Runtime.BizTalkServiceInstance.EndOperation(IAsyncResult result)
at AsyncInvokeEndEndTwoWayMethod(Object , Object[], IAsyncResult )
at System.ServiceModel.Dispatcher.AsyncMethodInvoker.InvokeEnd(Object instance, Object[]&outputs, IAsyncResult result)
at System.ServiceModel.Dispatcher.DispatchOperationRuntime.InvokeEnd(MessageRpc&rpc)
at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage7(MessageRpc&rpc)
at System.ServiceModel.Dispatcher.MessageRpc.Process(Boolean isOperationContextSet)
This is the only difference in behaviour that I have noticed. Why are two instances of the same orchestration handling the same fault differently?
This is due to the bitness of the hosting environment when hosting an orchestration in the out-of-process Isolated Host. When the IIS application pool is set to Enable 32-bit Applications = False, the BizTalk adapter's fault processing logic is slightly different, or expressed differently in terms of stack trace, and this causes the test to fail.
I actually set this myself and forgot about it, but the reason for doing it was that a completely unrelated application had added a global module to my IIS configuration that was compiled in 64-bit mode, which caused the application pool to repeatedly error then shut down whenever it ran in 32-bit mode. This module was UxCertAuthModule.dll and it is installed, I believe, by one of the components of Windows Azure Pack. I believe this is a bug, and removing this global module fixes 32-bit application pools, and my test.
edit:
I have raised this as a possible bug on the Azure Pack forums.
Related
I have a WCF service that is coded to throw a Custom FaultException under certain conditions. When hosted locally and on several servers this executes as excpected, Custom fault thrown by service custom fault caught by client, but on production and UAT server the Custom fault is thrown by what the client recieves is a Protocol Exception (500 error).
Is anyone aware of an IIS or sever setting that could be effecting this WCF server? This issue is driving me crazy
I am having a similar Issue,
our server is using a third party web service, when a client connects to our server from the same machine, our server can catch the Fault Exception, but if the client is connecting over the network our server can't handle the Fault Exception, and is only getting the "500 Internal Server Error".
I used a sniffer to see the incoming data, and I can see that the webservice is sending the Fault Exception in both cases, this is a third party webservice so i have no control over it.
the clients use .Net Remotting to connect to the server.
The Solution:
add RemotingConfiguration.CustomErrorsMode = CustomErrorsModes.Off;
to the Remoting server,
for some reason it is affecting the Exceptions it (the server) is receiving.
this is not the ideal solution because now we are exposing our servers Exceptions ...
First of all, sorry, i'm not fluent.
I'm trying to figure out why my WCF services stop working when we have an environment with high calls/second rate. I'm not sure that just increasing timeout will solve the issue.
We have 2 webservices:
The first is hosted on IIS 7.5, Windows Server 2008 R2 Enterprise SP1 x64, with AppFabric (and WAS)
Second, hosted on Windows Service, Windows 2003 R2 SP1 x86
Both webservices have minimum configuration: No authentication, No trasaction, Without special treating of message.. check the binding:
<netTcpBinding>
<binding transactionFlow="false">
<security mode="None">
<message clientCredentialType="None" />
<transport clientCredentialType="None"></transport>
</security>
<reliableSession enabled="false"/>
</binding>
</netTcpBinding>
We are trying to use Net.Tcp binding because of its realibility and velocity.
FACT 1 - Net.Tcp Binding is primary reason
When the load is high, the channel Net.Tcp stop working. That's it! But the BasicHttp still working like a charm.
The WindowsService: the channel net.tcp last down for some minutes (3m - 10m) before get working back (BY ITSELF, without we change anything. Goblins are working hard).
The AppFabric/IIS/WAS: the channel net.tcp keep down. Need manual restart.
The BasicHttpBinding configuration is similar to net.tcp: without any treating of the message, whitout security concerns or something like that.
FACT 2 - Without any kind of logging
We couldn't find any kind, tip, trick to figure out what's happening. I have tried Dump the memory, event logs, System.Diagnostics and nothing relevant. The most relevant tip is an Error from SMSvcHost 4.0.0.0:
An error occurred while dispatching a duplicated socket: this handle
is now leaked in the process. ID: 2272 Source:
System.ServiceModel.Activation.TcpWorkerProcess/62875109 Exception:
System.TimeoutException: This request operation sent to
http://schemas.microsoft.com/2005/12/ServiceModel/Addressing/Anonymous
did not receive a reply within the configured timeout (00:01:00). The
time allotted to this operation may have been a portion of a longer
timeout. This may be because the service is still processing the
operation or because the service was unable to send a reply message.
Please consider increasing the operation timeout (by casting the
channel/proxy to IContextChannel and setting the OperationTimeout
property) and ensure that the service is able to connect to the
client.
Server stack trace: at
System.Runtime.AsyncResult.End[TAsyncResult](IAsyncResult result)
at
System.ServiceModel.Channels.ServiceChannel.SendAsyncResult.End(SendAsyncResult
result) at
System.ServiceModel.Channels.ServiceChannel.EndCall(String action,
Object[] outs, IAsyncResult result) at
System.ServiceModel.Channels.ServiceChannelProxy.InvokeEndService(IMethodCallMessage
methodCall, ProxyOperationRuntime operation) at
System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage
message)
Exception rethrown at [0]: at
System.Runtime.AsyncResult.End[TAsyncResult](IAsyncResult result)
at
System.ServiceModel.Activation.WorkerProcess.EndDispatchSession(IAsyncResult
result) Process Name: SMSvcHost Process ID: 1532
Do you have any tip or configuration trick to help me solve this issue?
Whats the best configuration for high load scenarios?
If you generated a service reference in Visual Studio, or with the svcutil tool, make sure you always call the Close or Abort methods of your proxies. I encountered a similar problem some days ago because I forgot to call these methods.
In case you are calling the Close() and Abort() methods accordingly and still receive this error consider the following scenario:
You run a Microsoft .NET Framework 3.0-based or .NET Framework 3.5-based Windows Communication Foundation (WCF) service.
The WCF service uses the Net.Tcp Port Sharing Service (Smsvchost.exe) and is hosted on a computer that is running Internet Information Services (IIS).
One of the following conditions is true:
The CPU usage is high on the computer that is running IIS.
A throttle occurs in a service model for the WCF service.
Multiple requests are sent to the WCF service at the same time.
In this scenario, the WCF service takes longer than one minute to process a request from a client application. Additionally, an error message that assembles the following event entry is logged in the event log:
Log Name: System
Source: SMSvcHost 3.0.0.0
Date:
Event ID: 8
Task Category: Sharing Service
Level: Error
Keywords: Classic
User: LOCAL SERVICE
Computer:
Description: An error occurred while dispatching a duplicated socket: this handle is now leaked in the process.
ID: 2620
Source: System.ServiceModel.Activation.TcpWorkerProcess
Exception:
System.TimeoutException: This request operation sent to did not receive a reply within the configured timeout (00:01:00). The time allotted to this operation may have been a portion of a longer timeout. This may be because the service is still processing the operation or because the service was unable to send a reply message. Please consider increasing the operation timeout (by casting the channel/proxy to IContextChannel and setting the OperationTimeout property) and ensure that the service is able to connect to the client.
Note: You must restart IIS to recover the WCF service from this issue.
Cause:
This issue occurs because of the Smsvchost.exe process times out after one minute when it tries to transfer an incoming connection request to the W3wp.exe worker process. Additionally, this time-out is not configurable.
When the CPU has a heavy workload, or when many concurrent connection requests are incoming, the Smsvchost.exe process cannot transfer the incoming connection to the W3wp.exe worker process within one minute. Therefore, the Smsvchost.exe process times out and eventually stops responding. When this issue occurs, the Smsvchost.exe process cannot route later requests to the W3wp.exe worker process until IIS is restarted.
Solution:
Microsoft suggests applying the hot fix 2504602 that is described in Microsoft Knowledge Base (KB) article. This hot fix is available for WCF in the .NET Framework 3.0 SP2, in the .NET Framework 3.5 SP1 and the .NET Framework 4.
In addition, Microsoft claims to have solved this issue in the .Net Framework 4.5, therefore, you should upgrade to the latest version.
In case you upgrade to the .Net Framework 4.5 and the problem persists the workaround is to modify the smsvchost.exe.config file to increase timeout and pending accepts and various other parameters.
I have a WCF service run on Windows Server 2008 RC2 IIS 7 with no firewall. When I trying to call it with netTcpBinding binding, I get this exception:
System.TimeoutException: The open
operation did not complete within the
allotted timeout of 00:00:30. The time
allotted to this operation may have
been a portion of a longer timeout.
---> System.TimeoutException: The socket transfer timed out after
00:00:30. You have exceeded the
timeout set on your binding. The time
allotted to this operation may have
been a portion of a longer timeout.
---> System.Net.Sockets.SocketException: A
connection attempt failed because the
connected party did not properly
respond after a period of time, or
established connection failed because
connected host has failed to respond
at
System.Net.Sockets.Socket.Receive(Byte[]
buffer, Int32 offset, Int32 size,
SocketFlags socketFlags) at
System.ServiceModel.Channels.SocketConnection.ReadCore(Byte[]
buffer, Int32 offset, Int32 size,
TimeSpan timeout, Boolean closing)...
The method I call just returns a numeric value, nothing else, so the problem not in timeout. If I use wsHttpBinding - it works without problems. Also I added logging to method I call, so I know that it even not executed.
I made all steps to configure IIS from here. The questions are:
Anybody know what the problem may be?
How can I troubleshoot/debug this
problem?
I am as frustrated as you are with this misleading fault message. If I am not wrong you get this TimeoutException within the first second. I can assure you your issue either related to security or serialization problems. I took this issue to Microsoft's WCF product group, they seemed to be surprised but didn't hear back anything.
First suggestion is to look at NetTcpBinding security settings. Make sure both client and service have identical bare bones security settings to start with (such as not message encryption nor transport layer security). If you can make it work without security, step by step increase security settings.
Second suggestion some serialization problem may be crashing the service: An empty nullable field, overloaded methods mixing with operations, ambiguous contract names, invalid casts.To debug, setup tracing in your service's config settings. You can do this easily via WCF Service Config utility(SvcConfigEditor.exe). Run your service get the exception, stop it and open the generated trace logs with WCF Service Trace Viewer Tool. Both tools comes with .NET (not Visual Studio) and can be found in Program Files/Windows SDKs folder.
I am trying to add a service reference to axapta 2009. All is working well, its a simple web method(external webservice) that gets executed on the server tier(necessary, otherwise clr interop error)
But I've ran into the following problems :
is it possible to close the proxy one way or another? Because this option is not available in the generated service object in AX (only the webmethods and a tostring).
at a certain moment, i ran into a service with faulted state. Normally, you create the service object again, but this didnt solve anything, until i restarted the AOS. Is this normal behaviour? Is the service object cached or something like that on server side?
Thx in advance.
This is due to the fact that the WCF service is throwing Faults, probably unhandled faults.
Do you have access to the WCF Service? If so then have a look at this link: How do I prevent a WCF service from enter a faulted state?
Try to catch any exceptions within the WCF Service and log them.
Unfortunately Ax cannot catch FaultExceptions thrown by WCF so you will be limited to modify the WCF Service with an object encapsulating the return message, along with a flag if the method processed successfully or if an exception was thrown.
Yes it is normal behavior for a faulted WCF service to stay in the Faulted state. You may have to restart the IIS service or just recycle the AppPooll the WCF Service is running under.
My application has 50 service endpoints (such as /mysite/myService.svc). It's hosted in IIS. Intermittently (once every two or three days) a service stops responding. It's never the same service that hangs. While a service is hung, some of the other services work fine and some other are also hung.
All clients (from different computers) get this error:
ServiceModel.CommunicationException
Message: An error occurred while receiving the HTTP response to
https://server/mysite/myservice1.svc.
This could be due to the service endpoint binding not using the HTTP
protocol. This could also be due to an HTTP request context being
aborted by the server (possibly due to the service shutting down).
See server logs for more details.
No exceptions are raised by the server when the client attempts to call the service that is hung. All I have is that error on the client side.
I have to manually recycle the application pool to fix the problem.
Do you know what could be the cause? How can I investigate this issue? I'm willing to take a memory dump of the worker process when a service is hung but I would not know what to search for in the dump.
Update (Aug 13 2009): I have almost ruled out the idea that the server runs out of connections (see comment in Shiraz Bhaiji's answer). I might have a new lead: I log all server-side exceptions in a log file. So in theory, when this occurs on the client, no exceptions are raised on the server; otherwise I'd have proof of that in my logs. But what if an error does occur on the server but is happening at a low level where exceptions are not routed to my exception handling code? I have posted this question about scenarios where low level exceptions cannot be handled. I'll keep you informed of the progress of my investigation.
Sounds like you are running out of connections.
By default WCF has a timeout and therefore holds a connection open for 10 mins.
When you recycle the app pool all connections are closed, and therefore things work again.
To fix it check your code to make sure that you close connections / dispose of proxies.
To resolve this, we set establishSecurityContext to False on the binding.
I have not come across this particular issue but would suggest to turn on tracing/message logging for the WCF service in the config for the service and/or the client app (if you have control over that). I've done this in the last few days for a service that I needed to troubleshoot.
The MSDN link here is a good starting point.
Also see the table in this post for the varying levels of trace detail you can configure. There are several levels which can go from exception only logging to full message details. It is quite quick to set this up in the app.config file.
To parse the log file output use the SvcTraceViewer.exe that comes with the Windows SDK, which if you have it installed should be located in this folder: C:\Program Files\Microsoft SDKs\Windows\v6.0\Bin