I am trying to achieve the following - one client-side proxy instance (kept open) accessed by multiple threads using a reliable session. What I have managed so far is to have either A) a reliable session with a client-side proxy which is created and disposed per call or B) what I aim for, but without a reliable session.
When I enable reliable sessions on my binding however, the following behaviour is exhibited:
Client-side
Upon application startup everything appears to work fine until roughly 18 messages in to the WCF session. I firstly get the proxy.InnerChannel.Faulted event raised, then an exception is caught at the point where I am calling the method on the proxy. The exception is a System.TimeoutException, with message:
"The request channel timed out while waiting for a reply after
00:00:59.9062512. Increase the timeout value passed to the call to
Request or increase the SendTimeout value on the Binding. The time
allotted to this operation may have been a portion of a longer
timeout."
The inner exception has a similar message:
"The request operation did not complete within the allotted timeout of
00:01:00. The time allotted to this operation may have been a portion
of a longer timeout."
With the method at the top of the inner stack trace being:
System.ServiceModel.Channels.ReliableRequestSessionChannel.SyncRequest.WaitForReply(TimeSpan timeout)
I then call proxy.Close followed by proxy.Abort (catching and ignoring exceptions). If I utilize the default settings (i.e. have simply <reliableSession/>), then calling proxy. Close results in another System.Timeout exception (although this time the allotted timeout is 00:00:00), however if I override the defaults as specified above no exception is thrown.
Service-side
Utilizing WCF tracing I get a System.ServiceModel.CommunicationException, with message:
"The sequence has been terminated by the remote endpoint. The session
has stopped waiting for a particular reply. Because of this the
reliable session cannot continue. The reliable session was faulted."
And a stack trace ending at:
System.ServiceModel.AsyncResult.End[TAsyncResult](IAsyncResult result)
When remotely attaching to the server I get the same message, which occurs when code execution steps over the return statement of my service in the service call which causes the error.
The puzzling thing to me is that the service is stable and runs with options A) or B) as decribed at the beginning of my post, and occurs after a varying number of messages (around 18). The former fact points to there being nothing wrong with the code (indeed I have checked that no exceptions are thrown), and the latter just serves to confuse me and is why I modified the settings on the reliable session binding.
I am quite stuck on this. Can anyone suggest why the reliable session would fault in such a way?
You need to overide the default ,and set your timeout higher or lower depends on cause,it seems the timout is causing an exception just after some other program has started or stopped just a millisecond before the exception
OR most likely cause
your alloted timeouts may be added as a continous single timeout of 18 min or 18 calls ..plus other usage times are added together as one complete time out .which may be why it asking for more time.
in any case ,you have to staticly set your own settings because automatic default will always over ride any changes you made..
type in your local host http binding name and set your closetimeout at maybe 5.00 min
and maybe even change the request time as well . Requesttimeout 2.00 min
closeTimeout="00:05:00"
Related
I have a program where I start several process instances using a cron. For each process instance I have a maximum time, and if the execution time exceeds it, I have to consider it as failure and use some specific methods.
For now what I did was simply to check, once my process instance has finished, if the elapsed time exceeds or not the given maximum time.
But what if my process instance gets blocked for some reason (e.g. server not responding)? I need to catch this event and perform failure operations as soon as the process gets blocked and timeout is exceeded.
How can I catch these two conditions?
I had a look at the FlowableEngineEventType, but there isn’t a PROCESS_BLOCKED/SUSPENDED type of event. But, even if it were, how do I fire it only if a certain amount of time has passed?
I assume that this is the same question as this from the Flowable Forum.
If you are using the Flowable HTTP Task then have a look at the documentation to see how you can set the timeouts on it and how you can react on errors there. If you are firing GET requests from your own code you would need to write your own business logic that would throw some kind of BpmnError and you would then handle that in your process.
The Flowable Process instance does not have the concept of being blocked, and you have to manually to that in your modelling.
We are using KingswaySoft SSIS integration to push data (60k) into CRM 2015. There is a daily job which is set to run this package, which passes successfully sometime and fails sometime with this error. Please suggest troubleshooting/fix.
Delete Sales Transaction data of months in CRM:Error: An error occurred with the following error message:
KingswaySoft.IntegrationToolkit.DynamicsCrm.CrmServiceException: CRM service call returned an error: CRM service call returned an error: The request channel timed out while waiting for a reply after 00:02:59.9843634. Increase the timeout value passed to the call to Request or increase the SendTimeout value on the Binding. The time allotted to this operation may have been a portion of a longer timeout.(Error Type / Reason: Timeout) (SSIS Integration Toolkit for Microsoft Dynamics CRM, v7.0.0.5660 - ISServerExec, v11.0.5058.0)
KingswaySoft.IntegrationToolkit.DynamicsCrm.CrmServiceException: CRM service call returned an error: The request channel timed out while waiting for a reply after 00:02:59.9843634. Increase the timeout value passed to the call to Request or increase the SendTimeout value on the Binding. The time allotted to this operation may have been a portion of a longer timeout.(Error Type / Reason: Timeout)
System.TimeoutException: The request channel timed out while waiting for a reply after 00:02:59.9843634. Increase the timeout value passed to the call to Request or increase the SendTimeout value on the Binding. The time allotted to this operation may have been a portion of a longer timeout.
System.TimeoutException: The HTTP request to 'crm.abc.com.au/.../Organization.svc' has exceeded the allotted timeout of 00:03:00. The time allotted to this operation may have been a portion of a longer timeout.
System.Net.WebException(Status Reason: Timeout): The operation has timed out
Also since it is giving timeout issue, is it worth trying to increase timeout period? please guide how we can perform it.
First of all, we thank you for choosing KingswaySoft.
For the timeout error that you are getting, it indicates that a service call is taking a longer time than what you have specified in the connection manager. There are a couple of options in your case.
If this is happening to a CRM destination component, you should
consider decreasing its Batch Size setting.
If you are using Upsert action with manually specified key fields in the destination component, you want to make sure you have created db indexes for the fields that you have selected for Upsert matching.
You may also increase your CRM connection manager’s Timeout setting.
Based on the error message, it appears that you are using 180 seconds as your timeout setting, which is plenty for any service calls. Having a timeout error in this case could be an indication that you have some design issues with your CRM application. You might want to revisit your CRM plugin or workflow code if you have any.
Please excuse the Obvious Self-Q/A, but this information is widely misunderstood, and almost always incorrectly answered. So I Wanted to place this information here for people searching for a definitive answer to this problem.
Even so, there's still some information I haven't been able to nail down. I will put this towards the end of the question (skip to that if you are not interested in the preamble).
How do I correctly configure a WCF NetTcp Duplex Reliable Session?
There are many questions and answers regarding this topic, and nearly all of them suggest setting inactivityTimeout="Infinite" in your configuration. This doesn't really seem to work correctly, particularly for the case of NetTcp (It may work correctly for WSDualHttp Bindings, but I have never used those).
There are a number of other issues that are often associated with this: Including, Channel not faulting after client or server unexpectedly disconnected, Channel disconnecting after 10 minutes, Channel randomly disconnecting... Channel throwing exception when trying to open... Unable to configure Metadata on same endpoint...
Please note: There are two concepts that are important below. Infrastructure messages are internal to the way WCF communicates, and are used by the framework to keep things running smoothly. Operation messages are messages that occur because your app has done something, like send a message across the wire. Infrastructure messages are largely invisible to your app (but they still occur in the background) while operation messages are the result of an action your app has taken.
Information I have figured out, through hard won trial and error.
Infinite does not appear to be a valid configuration setting in all situations (and certainly, the visual studio validation schema doesn't know about it).
There are two special configuration converters, called InfiniteIntConverter and InfiniteTimeSpanConverter which will sometimes work to convert the value Infinite to either Int.MaxValue or TimeSpan.MaxValue, but I haven't yet figured out the situations in which this appears to be valid as sometimes it works, and sometimes it doesn't. What's more, it appears that some libraries will allow Infinite in the config, while others will not, so you can succeed in one part of a configuration, but fail in another.
You must configure BOTH inactivityTimeout and receiveTimeout, on both the client and the server. While these values do not HAVE to be the same, they probably should be as they will probably cause confusion if they are not. (technically, you can leave inactivityTimeout to its default value if you want, but you should be aware of its value, and what it does)
inactivityTimeout should NEVER be set to a large value, much less Infinite or TimeSpan.MaxValue.
inactivityTimeout has two functions (and this is not widely understood). The first function defines the maximum amount of time that can elapse on a channel without receiving any "infrastructure" or "operation" messages. The second function defines the time period in which infrastructure messages are sent (half the time specified). If no infrastructure or operation messages have been received during the timeout period, the connection is aborted.
receiveTimeout specifies the maximum amount of time that can elapse between operation messages only. This value can be set to a large value, such as TimeSpan.MaxValue (particularly if your channel runs internally over a trusted network or over a vpn). This value is what defines how long the reliable session will "stay alive" if there is no activity between client and server (other than infrastructure messages). ie, your client does not call any methods of the interface, and your server does not call back into the client.
setting a short inactivityTimeout and a large receiveTimeout keeps your reliable session "tacked up" even when there is no operational activity between your client and server. The short inactivity timeout (i like to keep the default 10 minutes or less) sends infrastructure "ping" messages to keep the TCP connection alive while the long receive timeout keeps the reliable session active. while at the same time providing a reasonable timeout in case of disconnection.
If you set inactivityTimeout to a large value, then the reliable session will not be reliable as it has no way to keep the Tcp connection alive, nor does it have any way to verify the integrity of the connection. It won't know if a user has disconnected unexpectedly until you try and send a message to that client and find out the connection is no longer there. This is why many people who use Infinite for this setting resort to creating a "Ping" method in their service, which is completely unnecessary if you've configured these settings correctly.
If you set inactivityTimeout to a value larger than receiveTimeout then it will likewise also be unreliable, as you will still be governed by the receiveTimeout for operation messages. ie. if you forget to set receiveTimeout and leave it at the default 10 minutes, then if the user is idle for 10 minutes, the connection will be aborted.
When the client or server unexpectedly disconnects (app crashes, network failure, someone trips over the power cord, etc..), the other side may not notice right away. I have attached various ChannelFaulted event handlers in various test situations, and sometimes the connection is faulted right away... other times it doesn't seem to fault at all. What i have discovered through trial and error is that the when it doesn't seem to fault, it will actually fault after the inactivityTimeout expires on that end. (so if it's set to 10 minutes, then after 10 minutes it will call the ChannelFaulted event).
I have not yet figured out why in some situations it notices the disconnection right away, and others it waits for the timer to expire. In both cases, I notice internal first chance communication exceptions thrown and handled by the framework, and there are calls to Abort the connection... but somehow the call to the event handler gets lost and it must wait for the timeout. My suspicion is this is somehow thread related.
When trying to configure Metadata to work across the NetTcp channel, I have had sporadic results. Sometimes it works, sometimes it doesn't. I've read many reports that Metadata doesn't work over NetTcp and that you have to use an Http channel for the Metadata, but I have in fact had it work on several occasions using the net.tcp:// url to generate the proxy. Then I would change something, recompile and it would no longer work. Changing things back, it wouldn't work again. So it was very confusing what magic incantation was necessary to make Metadata function over net.tcp, shared with the endpoint on the same port (obviously with a different address).
When configuring both a NetTcp and Metatdata endpoint on the same service, and specifying non-default settings for connection parameters like listenBacklog, and maxConnections, you also need to make sure the Metadata endpoint uses the same settings, which typically means you have to define a custom binding, since these settings are not available from the standard tcp mex binding. This includes setting listenBacklog and maxPendingConnections on tcpTransport, and groupName and maxOutboundConnectionsPerEndpoint on connectionPoolSettings.
The default setting for the Ordered setting of ReliableSession is True. This uses a lot more overhead than turning it off. If you don't need ordered messages, i would suggest turning it off (need to set this on both sides)
-
Configuration I still need to understand:
How do I correctly configure the shared net.tcp Metadata endpoint? (I will add an example when I get a chance) Currently, i'm specifying an http get url to bypass the problem. It's so inconsistent as to why it sometimes works and sometimes does not. I kept getting the error `The URI Prefix is not recognized' when generating the proxy in Visual Studio.
Why does WCF sometimes Fault the channel immediately upon disconnect, and sometimes waits for inactivityTimeout to expire? What controls/causes one vs the other behavior?
Say I have the following chain of execution in my WCF Service:
ServiceMethod calls and waits for Method1, then calls and waits for Method2, which calls and waits for Method3. Finally ServiceMethod calls and waits for Method4 before returning.
What happens if the service's configured timeout is hit during the execution of Method 3 (or any of those methods)? Does the thread executing ServiceMethod just get terminated immediately? With no further execution? Or does the process allow the thread to continue to the end, without returning any result?
My concern is in knowing how far processing went before the timeout was encountered. If the thread is allowed to complete, then one can know that all completed anyway (even though no result was returned). But if the thread just gets terminated immediately, one would have to design the ServiceMethod so that one can trace how far it got, and then try again from there.
The operation is allowed to run to completion on the server - it's the WCF channel that times out. In fact, some people have asked here for a way to force the server side processing to abort when a timeout occurs, and it's generally agreed that doing that cleanly would be difficult:
Why doesn’t WCF support service-side timeouts?
After the service instance has been created by the host, it will continue to run until either process is terminated, your logic completes and exits the entry point (operation) or uncaught exception is thrown. The timeout you are concerned with is between the client and the host.
Client will get an exception on the channel signalling the timeout as the channel fault. This tells the client that channel is not safe for use and it will have to be re-created.
Small comment on the call chain. You are better of encapsulate your step by step logic in a single workflow or manager which can help you with requirements for re-startability or compensation logic. Have a single entry point in your service which then can execute the workflow.
+1 on the answer for Why doesn't WCF support service-side timeouts from "500-internal server"
When I invoke a remote WCF service I get the following timeout:
The request channel timed out while waiting for a reply after 00:00:59.2810338. Increase the timeout value passed to the call to Request or increase the SendTimeout value on the Binding. The time allotted to this operation may have been a portion of a longer timeout.
Please note that I am sending a single object which is LOADED with a LOT of data.
Any ideas how to fix this issue and is this a problem on the client (ME) or the Server.
Given the size, have you tried increasing your maxBufferSize/maxReceivedMessageSize in your binding?
Chunk your data into smaller pieces if possible and try again. This is a server setting that you will need to work around or request that the service provider increase it.
Without a stack trace I'm can't be 100% sure, but I'm relatively certain this is a client side exception. If you know it's going to take more than a minute to send the data all you need to do is change the sendTimeout on your binding to be whatever amount of time you need it to be.