I have a WCF service that reads from a message queue. I noticed with our latest deployment that there is a message sitting in the queue that has not been read. The WCF service is up and running and it is correctly establishing a connection to the queue (or I would receive and error).
We made some changes to our code recently and we are wondering if perhaps that is related. It seems odd to us that the service would be running and yet the message not read. At a minimum, I would expect WCF to throw some sort of error if there was a malformed message in the queue.
I looked at the properties of the message and it says there have been zero moves, so I don't think it is being sent to a retry sub-queue. It is just sitting there and the service won't read it.
Is there a circumstance where WCF would ignore a message in the queue? How does WCF handle malformed messages?
Sorry if this isn't a lot of information to go on. At this point I am just trying to understand what is preventing the message from being processed.
This situation seemed just too odd. It didn't make sense that WCF would do nothing at all.
I eventually decided to simply try restarting the MSMQ service on the server. Once I did that, my service immediately picked up the messages and everything started working again.
I have no idea if this had something to do with a Windows Update or some other server change. I am glad it was this easy to fix - it will be the first thing I try in the future.
Related
I have configured MassTransit with RabbitMQ as transport. And I just use an instance of generic IRequestClient to send requests to a consumer that then should return a response.
My problem is that every other request fails with a TimeoutException. Execute it once, the next time it fails, and then it works again.
The Consumer is not even invoked when failed.
What can be the reason for this?
I have other services share a similar name in their requests and consumers. I have tried to figure out if that is the problem.
You should post the configuration code of your application using the request client and the one configuring the consumer.
If you have other consumers with the same name, it's likely they're on the same queue if you're using ConfigureEndpoints, which could be the root cause of the issue.
Since it's every-other-message that times out, that would make sense since RabbitMQ will load balance the queue across the different services with the same queue name.
I have a WCF service that is set up to use MSMQ to transmit to a service on another machine. We are trying to move the client onto a different machine, but it's not working. Enabling the MSMQ.End2End event log gives us
Message with ID {6940f8fa-3d31-4db0-ae2b-59bc98c99f2c}\25321 was sent to queue DIRECT=OS:iisapp1-vvpm\private$\TransactionalEmailService/TransactionalEmail.Service.TransactionalEmailService.svc
which makes me think that it is working correctly from our machine, but we can't find any trace of it on the target machine. The service is not being invoked, and we can't find the message in the dead-letter queue (or anywhere else we can think of to look).
Also, running the code directly from Visual Studio on my machine causes it to work.
Changing the receiving queue to the DEV machine also causes the code to work, which makes me further think it's a problem with the receiving machine. (I just have no idea what)
UPDATE 1:
I came back to it and noticed all the messages I tried to send in the transactional dead-letter queue. The error message is "the time-to-reach-queue has elapsed". Looking at the connection state, it's inactive, and sending another message won't cause it to become connected. I restart the machine, and it is "Connected" again. I try to send the message again, and look at the queue state. There are 12 messages, all of which are unacknowledged (0 are unprocessed).
So it started happening again once the the endpoint machine got restarted. I came across this article, which was the real solution:
http://blogs.msdn.com/b/johnbreakwell/archive/2007/02/06/msmq-prefers-to-be-unique.aspx
( In case the link goes away: )
It looks like there is an ID in MSMQ that is sent as part of the message. The remote host uses that ID as a key into a cache to determine who to send the ack back to. If you clone a machine, it gets the same value in the registry for that ID, which makes the other machine not know who to send it back to. So it will send it to the wrong machine, which will discard it, and the other machine will be stuck with a bunch of messages it keeps trying to send. This also explains why it just started working one day... the cache expired and the "correct" machine got put in as the endpoint.
Reinstalling MSMQ on the cloned machine fixes the issue.
I'm really not sure this may be the case here (I don't have any experience with WCF in the context of MSMQ), but one of the more common reasons of this kind of behaviour you're talking of is missing the obligatory camel casing on FormatName in your queue name when using a MessageQueue constructor (like 'FormatName:DIRECT=...'), or getting the name somehow wrong. The queue name in the message looks a bit odd with the svc ending, but that could just be a WCF thing? Hope this helps at least getting you on the right direction.
Not sure what to say here, but it works now. Reading some stuff helped point me to the status of the queue (click on Outgoing Queues under Features/Message Queueing). From there I found this KB article with a hotfix: http://support.microsoft.com/kb/976438 It didn't seem like it was applicable, but the symptoms people were having were all the same. Our guys tried to install it, but it failed and they didn't restart... but for some reason the message queues started working.
If someone comes along with some insight, I'll gladly upvote them or give them the bounty (if it's soon enough). But I'll just accept this as the answer for now.
This is usually caused by permissions on the remote queue, usual scenario is if you are using a private queue and this is accessed remotely by your wcf service.
Try using a public queue.
I have spent days reading MSDN, forums and article about this, and cannot find a solution to my problem.
As a PoC, I need to consume a queue from more than one machine since I need fault tolerance on the consumers side. Performance is not an issue since less than 100 messages a day should by exchanged.
I have coded two trivial console application , one as client, the other one as server. Using Framework 4.0 (tested also on 3.5). Messages are using transactions.
Everything runs fines on a single machine (Windows 7), even when running multiple consumers application instance.
Now I have a 2012 and a 2008 R2 virtual test servers running in the same domain (but don't want to use AD integration anyway). I am using IP address or "." in endpoint address attribute to prevent from DNS / AD resolution side effects.
Everything works fine IF the the queue is hosted by the consumer and the producer is submitting messages on the remote private queue. This is also true if I exchange the consumer / producer role of the 2012 and 2008 server.
But I have NEVER been able to make this run, using WCF, when the consumer is reading from remote queue and the producer is submitting messages localy. Submition never fails, my problem is on the consumer side.
My wish is to make this run using netMsmqBinding, but I also tried using msmqIntegrationBinding. For each test, I adapted code and configuration, then confirmed this was running ok when the consumer was consuming from the local queue.
The last test I have done is using WCF (msmqIntegrationBinding) only on the producer (local queue) and System.Messaging.MessageQueue on the consumer (remote queue) : It works fine ! => My goal is to make the same using WCF and netMsmqBinding on both sides.
In my point of view, I have proved this problem is a WCF issue, not an MSMQ one. This has nothing to do with security, authentication, firewall, transport, protocol, MSMQ version etc.
Errors info using MS Service Trace Viewer :
Using msmqIntegrationBinding when receiving the message (openning queue was ok) : An error occurred while receiving a message from the queue: The transaction specified cannot be imported. (-1072824242, 0xc00e004e). Ensure that MSMQ is installed and running. Make sure the queue is available to receive from.
Using netMsmqBinding, on opening the queue : An error occurred when converting the '172.22.1.9\private$\Test' queue path name to the format name: The queue path name specified is invalid. (-1072824300, 0xc00e0014). All operations on the queued channel failed. Ensure that the queue address is valid. MSMQ must be installed with Active Directory integration enabled and access to it is available.
If someone can help to find why my configuration cannot be handled by WCF, a much elegant and configurable way than Messaging, I would greatly appreciate !
Thank you.
You may need to post you consumer code and config to give more of an idea but it could be the construction of the queue name - e.g.
FormatName:DIRECT=TCP:192.168.0.2\SomeQueue
There are several different ways to connect to a queue and it changes when you are remote or local as well.
I have found this article in the past to help:
http://blogs.msdn.com/b/johnbreakwell/archive/2009/02/26/difference-between-path-name-and-format-name-when-accessing-msmq-queues.aspx
Also, MessageQueue Constructor on MSDN...
http://msdn.microsoft.com/en-us/library/ch1d814t.aspx
Imagine the following setup: a Silverlight client tunnels a serialized command over the network using a WCF service which in turn deserializes the command and sends it using NServiceBus to a generic host which is responsible for processing the command. The WCF service has - upon sending the command - registered a callback to be invoked. The generic host validates the command and 'returns' an error code (either 0 == success or >0 == failure).
Note: The WCF service is modelled after the built-in WCF service. The difference is that this WCF service receives a 'universal command' (not an IMessage), deserializes it into a real command (which does implement IMessage), and consequently sends the deserialized command off to the bus.
When unexpected exceptions occur, the command gets (after a certain amount of retries) queued in an error queue. At this point, the initiating WCF service sits there idle, unaware of what just happened. At some later point, the Silverlight client will time out according to the WCF client proxy configuration.
Things which are fuzzy in my head:
Does NServiceBus handle this scenario in any way? When does the timeout exception get thrown (if at all)? Or is this something exclusive to sagas?
Presuming I use [OperationContract(AsyncPattern=true)], are there any WCF related timeout settings that will kill the service operation? Or will the EndXXX method be somehow called? Or will it sit there forever, leaking, waiting for something that will never come?
Ways to proceed:
reuse existing timeout mechanisms, provided things don't leak.
build my own timeout mechanism between the wcf service and nservicebus.
notify the wcf service somehow when the command lands in the error queue.
build my own async notifcation mechanism using full blown callback message handler in the WCF service layer.
Things I've done:
run the example provided with NServiceBus.
spiked the happy case.
Any guidance on how to proceed is welcome, be it blog post, mailing list entries, ...
Some motivations for picking my current approach
I'm trying to leverage some of the scalability advantages (using distributor in a later phase) of NServiceBus.
I don't want to host a gazillion WCF services (one for each command), that's why I cooked up a bus-like WCF service.
Even though this is somewhat request/response style, I'm mostly concerned with gracefully handling a command reply not coming through.
You can develop any sort of message type you desire, IMessage is simply a marker interface. If you inspect the WSDL file that the service mex endpoint provides, there is no reference to IMessage, therefore you can define any command you like in you service. That being the case you should be able to use the provided WCF host.
I was able to reproduce the issue you describe using the built-in WCF hosting option. When an exception is thrown, the entire transaction is rolled back and this includes the Bus.Return, and therefore the service never gets a response.
I found a hack around this that I could provide, but I recommend reconsidering how you are using the service. If you are truly looking to do some expensive operations in a separate process then I would recommend in your WCF endpoint that you do a Bus.Send to a different process altogether. This would ensure to your client that the command was successfully received and that work is in progress. From there it would be up to the server to complete the command(some up front validation would help ensure its success). If the command was not completed successfully this should be made known on another channel(some background polling from the client would do).
My application has 50 service endpoints (such as /mysite/myService.svc). It's hosted in IIS. Intermittently (once every two or three days) a service stops responding. It's never the same service that hangs. While a service is hung, some of the other services work fine and some other are also hung.
All clients (from different computers) get this error:
ServiceModel.CommunicationException
Message: An error occurred while receiving the HTTP response to
https://server/mysite/myservice1.svc.
This could be due to the service endpoint binding not using the HTTP
protocol. This could also be due to an HTTP request context being
aborted by the server (possibly due to the service shutting down).
See server logs for more details.
No exceptions are raised by the server when the client attempts to call the service that is hung. All I have is that error on the client side.
I have to manually recycle the application pool to fix the problem.
Do you know what could be the cause? How can I investigate this issue? I'm willing to take a memory dump of the worker process when a service is hung but I would not know what to search for in the dump.
Update (Aug 13 2009): I have almost ruled out the idea that the server runs out of connections (see comment in Shiraz Bhaiji's answer). I might have a new lead: I log all server-side exceptions in a log file. So in theory, when this occurs on the client, no exceptions are raised on the server; otherwise I'd have proof of that in my logs. But what if an error does occur on the server but is happening at a low level where exceptions are not routed to my exception handling code? I have posted this question about scenarios where low level exceptions cannot be handled. I'll keep you informed of the progress of my investigation.
Sounds like you are running out of connections.
By default WCF has a timeout and therefore holds a connection open for 10 mins.
When you recycle the app pool all connections are closed, and therefore things work again.
To fix it check your code to make sure that you close connections / dispose of proxies.
To resolve this, we set establishSecurityContext to False on the binding.
I have not come across this particular issue but would suggest to turn on tracing/message logging for the WCF service in the config for the service and/or the client app (if you have control over that). I've done this in the last few days for a service that I needed to troubleshoot.
The MSDN link here is a good starting point.
Also see the table in this post for the varying levels of trace detail you can configure. There are several levels which can go from exception only logging to full message details. It is quite quick to set this up in the app.config file.
To parse the log file output use the SvcTraceViewer.exe that comes with the Windows SDK, which if you have it installed should be located in this folder: C:\Program Files\Microsoft SDKs\Windows\v6.0\Bin