We have an issue with a windows service which uses nServiceBus. At some random moment, the nServiceBus stops processing messages and direct them directly to Error queue, and I have to restart the service. After the restart, the messages arrived in the input message queue are handled, and everything gets back to normal. If we re-drop the messages which were went to error queue, it is processing it successfully without any issue.
We are using log4net logs to audit the message flow and storing in DB. The NServiceBus Handler stops to log in log4net. After we restart the windows service (NServiceBus) then it start to log again. We are NOT able to redproduce this issue in development environment. We are suspecting this could be a NService Bus Memory Leak issue. But we don't know how to confirm this issue and resolve the same.
We are planning to move this Windows Service (NServiceBus) to different server as a trial and error basis. Did anyone face this issue ever and resolved it? Please help us to resolve this issue as it is causing more troubles in Production environment.
NServiceBus Version that we are using : 2.0.0.1329
Message queue and windows service are in the same machine.
I believe you're running on a version of NServiceBus that is about 5 years old and is no longer supported. While I could give you the standard recommendation of upgrading to a more current release, it could very well be that some of the configuration APIs that you're using have been made obsolete so you may need to make some modifications there and/or in the app.configs.
I'm sorry to say that there probably isn't a better solution for you at this time.
In general, I'd suggest trying to track the NServiceBus releases somewhat more closely. If you're within 6-12 months of the current release, you should generally be in good shape.
Related
I am experimenting the new version of NServicsBus. I find following step by step sample on particular site.
https://docs.particular.net/samples/step-by-step/
Can any one tell me how to configure MSMQ for Transport. Here is my scenario.
Client create message
Client message should be stored in MSMQ
Server Application running on same machine which subscribe the message.
Server handler get message from MSMQ and process it further. i.e Store in DB or send to other web service.
Retry to process message if it does not worked first time
after 3 retries send message to error queue
How do i configure this sample to use MSMQ for my scenario.
Helpful information to include
Product name:NServiceBus.Core
Version: 6.3.4
Stacktrace:
Description:
Did you know that we have released a LearningTransport and LearningPersistence just for purposes like these? Have a look at it here.
Having said that, the transport swapping should be rather seamless so even if you have setup a small PoC using this transport/persistence, you can change it to MSMQ or other production-ready transports/persistence when you go live.
Again, as stated in the documentation page and as the name suggests, this is not for use in production.
I would recommend you walk through this.
https://docs.particular.net/tutorials/intro-to-nservicebus/
Will answer your questions, and future ones you have.
I have a WCF service that is set up to use MSMQ to transmit to a service on another machine. We are trying to move the client onto a different machine, but it's not working. Enabling the MSMQ.End2End event log gives us
Message with ID {6940f8fa-3d31-4db0-ae2b-59bc98c99f2c}\25321 was sent to queue DIRECT=OS:iisapp1-vvpm\private$\TransactionalEmailService/TransactionalEmail.Service.TransactionalEmailService.svc
which makes me think that it is working correctly from our machine, but we can't find any trace of it on the target machine. The service is not being invoked, and we can't find the message in the dead-letter queue (or anywhere else we can think of to look).
Also, running the code directly from Visual Studio on my machine causes it to work.
Changing the receiving queue to the DEV machine also causes the code to work, which makes me further think it's a problem with the receiving machine. (I just have no idea what)
UPDATE 1:
I came back to it and noticed all the messages I tried to send in the transactional dead-letter queue. The error message is "the time-to-reach-queue has elapsed". Looking at the connection state, it's inactive, and sending another message won't cause it to become connected. I restart the machine, and it is "Connected" again. I try to send the message again, and look at the queue state. There are 12 messages, all of which are unacknowledged (0 are unprocessed).
So it started happening again once the the endpoint machine got restarted. I came across this article, which was the real solution:
http://blogs.msdn.com/b/johnbreakwell/archive/2007/02/06/msmq-prefers-to-be-unique.aspx
( In case the link goes away: )
It looks like there is an ID in MSMQ that is sent as part of the message. The remote host uses that ID as a key into a cache to determine who to send the ack back to. If you clone a machine, it gets the same value in the registry for that ID, which makes the other machine not know who to send it back to. So it will send it to the wrong machine, which will discard it, and the other machine will be stuck with a bunch of messages it keeps trying to send. This also explains why it just started working one day... the cache expired and the "correct" machine got put in as the endpoint.
Reinstalling MSMQ on the cloned machine fixes the issue.
I'm really not sure this may be the case here (I don't have any experience with WCF in the context of MSMQ), but one of the more common reasons of this kind of behaviour you're talking of is missing the obligatory camel casing on FormatName in your queue name when using a MessageQueue constructor (like 'FormatName:DIRECT=...'), or getting the name somehow wrong. The queue name in the message looks a bit odd with the svc ending, but that could just be a WCF thing? Hope this helps at least getting you on the right direction.
Not sure what to say here, but it works now. Reading some stuff helped point me to the status of the queue (click on Outgoing Queues under Features/Message Queueing). From there I found this KB article with a hotfix: http://support.microsoft.com/kb/976438 It didn't seem like it was applicable, but the symptoms people were having were all the same. Our guys tried to install it, but it failed and they didn't restart... but for some reason the message queues started working.
If someone comes along with some insight, I'll gladly upvote them or give them the bounty (if it's soon enough). But I'll just accept this as the answer for now.
This is usually caused by permissions on the remote queue, usual scenario is if you are using a private queue and this is accessed remotely by your wcf service.
Try using a public queue.
What is the best way to combine a single instance WCF service that uses ActiveMQ and runs within IIS/AppFabric?
Our Services need to support both HTTP transports and ActiveMQ (listening and sending messages). We've elected not to use MSMQ, and will use Spring.Net.NMS. The fundamental issue I have now is that ActiveMQ needs to connect to the queue(s) at startup and remain connected, but WAS is getting in the way with it's message-activation feature. If the service is not activated until a message arrives (HTTP/MSMQ, etc) then there is no trigger to have the connection to AMQ occur.
I know I can disable the recycling behavior, and I know I can do self-hosting with a Windows Service. But I want to take advantage of the monitoring and other features in AppFabric. I've already been down the route with IServiceBehavior and will use that for other nice things. But that interface is not called until a (non-AMQ) message arrives. So it won't work for this. What I was hoping for was something along the line of how ServletContextListeners work in Java, where you get both the start up and shutdown events. But it seems no such thing exists in WAS... it is driven only by messages arriving.
I've scoured every inch of web info for 3 days and the only thing I came across was to use a static class construction (C#) trick as the trigger. That's a hack, but i can live with it. It still leaves the issue of cleanly shutting down, which I can figure out later.
Anyone have a solid solution to this?
The direct WCF support for ActiveMQ that Ladislav mentions is still being supported. There just hasn't been an official release for the module in a while. However, you can still get the latest version of it from the 1.5.x branch or trunk and compile it yourself.
1.5.x branch for use with Apache.NMS 1.5.0:
https://svn.apache.org/repos/asf/activemq/activemq-dotnet/Apache.NMS.WCF/branches/1.5.x/
Check out instructions:
http://activemq.apache.org/nms/source.html
There was direct WCF support for ActiveMQ but I guess it is not developed anymore. Your problem actually is the IIS / WAS (provides hosting for non-http protocols) hosting architecture. Services in WAS are always activated when message arrives - there is no global startup. The reason for this is that WAS hosting expects that there is separate process (windows service) running the listener all the time and this process has adapter which calls WAS and uses message level activation. I guess you don't have such process for ActiveMQ and because of that you will have trouble to use ActiveMQ endpoint hosted in WAS. Developing such listener can be challenging task (example for UDP).
Creating custom listener can be probably avoided by using IIS 7.5 / AppFabric auto start feature. There is also not very well documented way to run the code when the application starts.
I'm using NserviceBus 2.0 with pub/sub mode.
My subscribers are installed as a windows service.
However after computer restart I always get the following problem in log : "Problem in peeking a message from queue: ServiceNotAvailable".
After digging into source code I found that this is NserviceBus custom error and it occurs in MsmqTransport class. It seems like my subscriber's service is started before Msmq service. Bus this should be impossible because subscriber's service has Msmq as dependency.
After some time service is starting and working correctly. But I have several megabytes of errors in log. And sometimes service is not even starting.
Can anyone help me? I'm using Windows 7. Msmq is installed with NserviceBus utils.
You need to configure your service to be dependent on the MSMQ service. This is should be automatically taken care of if you're using the NServiceBus host.
Installing a Windows Service with dependencies
Have seen the same problem. Actually the impact was even worse since we used log4net and SmtpAppender. Took down the mailserver, ouch! Seems like this is fixed in NSB 3. It sets number of workerthreads to zero and logs "please reboot service". You can even execute own code when the error occurs. Config with lambda using OnCriticalError. We ended up patching the NSB 2 code, since we havent upgraded to NSB 3 yet. Handling MSMQExceptions, logging and stopping the process on errorcode ServiceNotAvailable like they already do when you don't have correct rights to queue. You should probably stop the service on any MSMQExceptions exept IOTimeout.
I have a WCF application with a couple thousand clients connecting to a pair of services running under IIS. What I've noticed is that some of these clients get into a hung state, and I'm trying to reproduce this.
When this problem was first noticed, I had not modified the throttling configuration and the services were set to ConcurrencyMode.Single. One thing I noticed was that an IISReset on the server caused many clients to hang. Yet pulling this same stunt on the client running against IIS on my local machine doesn't seem to cause the problem.
I caught this only once in the wild, but didn't have debugging enabled at the time. The symptom I witnessed was that the client appeared to be trying to open a connection to the web server, but did not succeed. While monitoring with Fiddler, I saw no attempt to reach the service endpoint. Obviously that makes me suspect the client proxy.
I have a very solid hunch as to what's happening -- namely I've been using "Close()" instead of "Abort()" when the service throws an exception, which I believe is causing the channels to become corrupted. But considering the effort to get a new version out there, I need to reproduce this problem by causing a client on my own machine to hang before I can start making changes to the code.
Where should I start?
Thanks in advance,
roufamatic
Have you got any logging turned on? This could help in diagnosing the problem. It can be done completely in config, so no need to build a new version. Use the Service Configuration Editor tool to set it all up. The Visual Studio 2008 Training Kit has a good tutorial on how to use logging and the log viewer.
I suppose this was too vague a question though I was mostly curious what people might suggest. As it turns out there was a nontrivial difference between my workstation and a production environment that, once resolved, allowed me to see the problem. In this case, somehow using Fiddler to watch the traffic actually prevented the error from occurring! Now to ask another question.