We are seeing very peculiar issue in our WSO2 ESB servers. 10% of the time we are seeing some of the back end service end points are getting suspended and we are seeing end point suspension messages in wso2 carbon logs. But the end pointing is working fine. Is there a way to trouble shoot end point suspension issues in WSo2 ESB 4.9.0
Related
recently we upgraded our nServiceBus version to 7. Post this upgrade we are noticing that this endpoint stops processing messages intermediately.
We checked our logs and noticed below message there
Replication destinations cleared for url http://RavendbURL/databases/XXXBusDB. Failover servers count: 0
What could the reason of this message? And if this bringing down the endpoint?
Thanks
I have a WCF service that reads from a message queue. I noticed with our latest deployment that there is a message sitting in the queue that has not been read. The WCF service is up and running and it is correctly establishing a connection to the queue (or I would receive and error).
We made some changes to our code recently and we are wondering if perhaps that is related. It seems odd to us that the service would be running and yet the message not read. At a minimum, I would expect WCF to throw some sort of error if there was a malformed message in the queue.
I looked at the properties of the message and it says there have been zero moves, so I don't think it is being sent to a retry sub-queue. It is just sitting there and the service won't read it.
Is there a circumstance where WCF would ignore a message in the queue? How does WCF handle malformed messages?
Sorry if this isn't a lot of information to go on. At this point I am just trying to understand what is preventing the message from being processed.
This situation seemed just too odd. It didn't make sense that WCF would do nothing at all.
I eventually decided to simply try restarting the MSMQ service on the server. Once I did that, my service immediately picked up the messages and everything started working again.
I have no idea if this had something to do with a Windows Update or some other server change. I am glad it was this easy to fix - it will be the first thing I try in the future.
In WSO2 ESB 4.7.0, Request Count would always increment for my auto-acknowledging HL7 proxy services every time a new message came in. After upgrading to 4.8.1, Request Count is always 0. I have verified through my downstream dependencies that messages are indeed coming through.
What happened to proxy level statistics in 4.8.1, and what do I need to do to re-enable them?
Thanks for pointing out this bug - there seem to be some changes in the carbon statistics modules between 4.1.0 and 4.2.0. I've created a JIRA to track the issue at https://wso2.org/jira/browse/ESBJAVA-2995
I'm using NserviceBus 2.0 with pub/sub mode.
My subscribers are installed as a windows service.
However after computer restart I always get the following problem in log : "Problem in peeking a message from queue: ServiceNotAvailable".
After digging into source code I found that this is NserviceBus custom error and it occurs in MsmqTransport class. It seems like my subscriber's service is started before Msmq service. Bus this should be impossible because subscriber's service has Msmq as dependency.
After some time service is starting and working correctly. But I have several megabytes of errors in log. And sometimes service is not even starting.
Can anyone help me? I'm using Windows 7. Msmq is installed with NserviceBus utils.
You need to configure your service to be dependent on the MSMQ service. This is should be automatically taken care of if you're using the NServiceBus host.
Installing a Windows Service with dependencies
Have seen the same problem. Actually the impact was even worse since we used log4net and SmtpAppender. Took down the mailserver, ouch! Seems like this is fixed in NSB 3. It sets number of workerthreads to zero and logs "please reboot service". You can even execute own code when the error occurs. Config with lambda using OnCriticalError. We ended up patching the NSB 2 code, since we havent upgraded to NSB 3 yet. Handling MSMQExceptions, logging and stopping the process on errorcode ServiceNotAvailable like they already do when you don't have correct rights to queue. You should probably stop the service on any MSMQExceptions exept IOTimeout.
This is a problem which has had me baffled for weeks now on a client's Live environment.
The WCF service is hosted on Windows Server 2003, and has both HTTP and MSMQ endpoints.
When placing the service in the test environment, the service cleanly starts and stops, and messages are passed without problems. However on the Live environment, the service starts fine, but does not exit cleanly.
When attempting to stop the service, the machine takes a long time to respond and eventually displays an error saying that the service could not be stopped. Inspecting the error on the event log, it says that it was unable to write to the MSMQ queue (access denied), however, the service is able read and remove messages from the queue. If one then refreshes the service manager, the service is in fact stopped.
The MSMQ queue is hosted on a different physical machine, and we have been unable to reproduce the error on the test environment.
We are not sure if it is related or not, but the service will also occasionally stop pulling messages from the queue. This has been solved by restarting the service. Again, we have not been able to reproduce the error.
Recently we experienced another error with the HTTP based client where upon midnight one night, the service suddenly started rejecting connections with the following exception:
The HTTP request is unauthorized with client authentication scheme 'Anonymous'. The authentication header received from the server was 'Negotiate,NTLM'. ---> System.Net.WebException: The remote server returned an error: (401) Unauthorized.
Even more curious, is that simply restarting the service seems to correct the problem.
If anyone has seen anything like this before or has any comments, it would be much appreciated!
Speaking to a colleague, apparently setting the ServiceModelEx throttling options all to "1" help with the lock ups on MSMQ based WCF services.