We have two applications (client and server) on the same machine that communicate via a WCF service using named pipes.
The issue that we're seeing is that occasionally a request from the client to the server will time out. WCF tracing is enabled on both ends, and on the server side we see the request being processed and responded to immediately as expected. However on the client side we don't see the response and the request times out after 1 minute.
What's suspicious is that almost immediately after the timeout, the client-side WCF trace shows the server's response arriving, almost as if it had been stuck somewhere and the timeout exception caused it to come un-stuck.
Attached are some screenshots from the client trace of the activity that relates to a particular message. Notice that the message gets sent out at 13:26:31.38, it times out at 13:27:31.80, and then the response gets received at 13:27:31.215.
Has anyone ever seen this behavior with WCF named pipes, where messages responses randomly seem to get stuck?
Related
I am implementing standard request/response scenario with MT and RabbitMQ. Client is Asp.net core API and consumer is a windows service.
As part of testing the exception cases if I stop the consumer and submit a request from API using request client, since there is no consumer processing, API got request timeout exception which is cool. But the message is sitting in the consumer queue and when I start the consumer, it picks the message and process the stuff( sending the message to external endpoint) and moved to a _skipped queue as there is no request client listening for this message.
Do you think it is correct behavior? First place when the api got request timeout exception, he will retry anyway so what’s the point of processing first message still?
How can I ignore those message where request clients were already finished processing with any error?
Thanks
What you are describing is very common, and I'd recommend reading up on idempotence and other distributed system failure scenarios.
Sending commands (the request, in this case) and conveying outcomes via a timeout in a message-based system can be very misleading. For instance, if you look at the ForkJoint, in the event of a request timeout, the response is actually a 202/Accepted instead of communicating an error.
The message is in the queue, it will be processed, so there is no reason to fail the controller and report an error back to the caller. So an intermediate response is used instead.
The sample is part of MassTransit Season 3 where I discuss a new idiom to deal with eventual completion/failure of commands in distributed systems. There might be some useful examples in there to help you understand the failure scenarios.
As doc sayed, to discard skipped messages so they are not moved to the _skipped queue:
cfg.ReceiveEndpoint("input-queue", ec =>
{
ec.DiscardSkippedMessages();
});
I have implemented message layer security using message inspector mechanism in a wcf service.
On the client side, in IClientMessageInspector.BeforeSendRequest I add an authentication header.
On the service side, in IDispatchMessageInspector.AfterReceiveRequest I inspect the authentication header in the message. If it is not found or as expected, I throw a SecurityException and try to log it to a database.
Here comes the interesting part. When logging to database, I try to read from this webservice again (this is web service which provides configuration info).
This is where the service stalls/deadlocks. I can see that the call to read configuration (when logging to db) is made, but I don't receive the call on the service. I keep getting a timedout exception every time.
After a little googling, I came across this post, which mentions that message inspectors are synchronous in nature. If that is so, how can I achieve what I am after?
We have an attribute called LoggingImplementationBehavior that causes a WCF Logging service to get called before and after service calls it's attached to.
This works great both locally and on the shared development server. However, on the staging server a 401 error is returned when a call is made to the Logging service from the invoker. To ensure that the issue is not in the Logging service I commented all code out so that it does nothing, but I still get this error when the stub is called.
In IIS Manager, Anonymous Authentication is enabled (and nothing else) at both the server level and at the services website level. Digest Authentication is disabled at both levels, so what could be the cause of these 401 errors?
I've seen others have this problem with large data but the data being sent here is quite small.
Any tips on how to continue debugging this problem would be greatly appreciated as I'm currently at a dead end.
The exact text of the error is:
The HTTP request is unauthorized with client authentication scheme 'Anonymous'. The authentication header received from the server was 'Digest qop="auth",algorithm=MD5-sess,nonce="+Upgraded+v171b25f8f2632897bff13b10710dac91aa1d64068a3cccf011b44f8580e932354dfd50d56778ba404d674864cf9d5216e589c616fb1a48583",charset=utf-8,realm="Digest"'.
The remote server returned an error: (401) Unauthorized.
I have a peculiar problem with client requests randomly not getting a response to a WCF service call.
The setup is a Silverlight 5 application making calls to a WCF service, using both HTTP and PollingDuplex bindings. The SL makes a number of requests upon loading (20+ in 20s), and every now and then does not receive a response to one or two of them. To be clear, the exact same request works 90% of time, if I refresh the page all requests could get a response.
The error cannot be the actual request sent. I use Fiddler to validate this as well, since I see the request being made (make sure the content is the same as previous successful requests), but there is no response to the request. Eventually the client times out the request. The WCF service is hosted in IIS7, and I have diagnostics and logging enabled on both. In the WCF server trace logs, I only see message logged when the client times the request out. It has the following exception action logged under a "Processing message" activity, at the time of the client timing out:
"The number of bytes available is inconsistent with the HTTP Content-Length header. There may have been a network error or the client may be sending invalid requests."
This is as if the body of the HTTP is not getting through to WCF (I do not know if I can log the full request received by IIS before passing to WCF handlers?). As I said, using Fiddler I can see the full message is valid (note that this behaviour also occurs when Fiddler is not sniffing the traffic, so I've ruled Fiddler out as the problem).
Typically the "Processing message" activity has a "To: Process action xxx"
Like I said this occurs with Http and PollingDuplex services, on my dev box as well as production web servers. Occurs on different endpoints as well, and I don't think it has to do with WCF throttling behaviour, since it occurs
Any information or help will be appreciated to get to the cause, whether it's additional information I can gather to help diagnose or any hints.
I am using a client server application in which client send request to server in request queue, server receive this request object from queue process it and send response in response queue which is received by client application. I want same functionality in wcf service and client so whether I need to create two different end points for both msmq and if yes than how same client will work with both endpoints.
You should not think of it as strictly a client server application.
You do have a request originator referred to as client and a request processor referred to as Server,
but when thinking WCF- client is the one sending the message, server is the one receiving. Meaning that in WCf terms, at first your client is really a classic "client" and the server is really a classic "server". But when you get to the point after the original request is processed and needs to be sent back- the roles are reversed! the server becomes a WCf client and the client becomes a WCf server.
What this means is that you processes need to expose a separate endpoint for each other. The server listens on a certain EP for incoming messages (requests), and the client listens on a certain EP for incoming messages (responses).
Hope this clarifies things a bit.