Nservice bus queue not found exception - nservicebus

I have a connector( which receives messages from different connectors. While receiving the message my connector gives the following message:
[Worker.11] WARN NServiceBus.Unicast.Transport.Transact
ional.TransactionalTransport [(null)] <(null)> - Failed raising 'transport messa
ge received' event for message with ID=fd970068-55ad-49c0-8abc-4133b7f7fe12\2138
47
NServiceBus.Unicast.Queuing.QueueNotFoundException ---> System.Messaging.Message
QueueException: Cannot enlist the transaction.
at System.Messaging.MessageQueue.SendInternal(Object obj, MessageQueueTransac
tion internalTransaction, MessageQueueTransactionType transactionType)
at NServiceBus.Unicast.Queuing.Msmq.MsmqMessageSender.NServiceBus.Unicast.Que
uing.ISendMessages.Send(TransportMessage message, Address address)
--- End of inner exception stack trace ---
at NServiceBus.Unicast.UnicastBus.HandleTransportMessage(IBuilder childBuilde
r, TransportMessage msg)
at NServiceBus.Unicast.UnicastBus.TransportMessageReceived(Object sender, Tra
nsportMessageReceivedEventArgs e)
at System.EventHandler`1.Invoke(Object sender, TEventArgs e)
at NServiceBus.Unicast.Transport.Transactional.TransactionalTransport.OnTrans
portMessageReceived(TransportMessage msg)
The weird thing is when i was doing the internal testing it was working all fine as soon as this has been now hosted on to amazon server it has started blowing up.
The following is how i have configured my nservice bus with
NServiceBus.Configure.With(
AllAssemblies.Except("libBL.dll").And("libCommon.dll").And("libExtra.dll")).StructureMapBuilder()
.JsonSerializer()
.UnicastBus()
.DoNotAutoSubscribe()
.TransactionTimeout(TimeSpan.FromMinutes(5));
Any help would be much appreciated. If any more information is needed please let me know
Thanks

On searching through the code found out that if a message coming in was failing then i was not rolling back the transaction in which the whole operation began and hence the nservicebus was not able to query the database as it was locked. After putting in the logic for transaction roll back got my connector to work.

Related

NServiceBus: Cannot enlist the transaction (failed to send msg to audit queue)

I have an endpoint that works as a distributor with three other worker endpoints.
The handling endpoint of the received message opens a transaction and tries to import some xml data into an sql db. If some exception is thrown during this process, the exception is caught, the transaction is rolled back and the xml data is written to an error folder.
Simplified, it looks like that:
public void Handle(doSomethingCmd message)
{
System.Data.SqlClient.BeginTransaction();
try
{
//... some xml data import
throw new Exception();
//Commit if succeded
}
catch (Exception exception)
{
System.Data.IDbTransaction.Rollback();
//...Write file to error folder
}
}
In the first place, no retry happens after the transaction rollback. But when the message is sent again, all the worker endpoints (only the workers) get an exception (Cannot enlist transaction, failed to send msg to control queue --> see stacktrace below) and nservicebus does retry the message (this leads to the case, that the file appears several times in the error folder)
It looks like distributed transaction is in an invalid state. I could just handing over the exception (re-throw the exception), so nservicebus handles the rollback for me, but in that case the file is written to the error folder several times as well (due to retry mechanism)
Failed raising finished message processing event.|NServiceBus.Unicast.Queuing.FailedToSendMessageException: Failed to send message to address: someEndpoint.distributor.control#SRVPS01 ---> System.Messaging.MessageQueueException: Cannot enlist the transaction.
at System.Messaging.MessageQueue.SendInternal(Object obj, MessageQueueTransaction internalTransaction, MessageQueueTransactionType transactionType)
at System.Messaging.MessageQueue.Send(Object obj, MessageQueueTransactionType transactionType)
at NServiceBus.Transports.Msmq.MsmqMessageSender.Send(TransportMessage message, Address address) in c:\BuildAgent\work\31f8c64a6e8a2d7c\src\NServiceBus.Core\Transports\Msmq\MsmqMessageSender.cs:line 49
--- End of inner exception stack trace ---
at NServiceBus.Transports.Msmq.MsmqMessageSender.ThrowFailedToSendException(Address address, Exception ex) in c:\BuildAgent\work\31f8c64a6e8a2d7c\src\NServiceBus.Core\Transports\Msmq\MsmqMessageSender.cs:line 88
at NServiceBus.Transports.Msmq.MsmqMessageSender.Send(TransportMessage message, Address address) in c:\BuildAgent\work\31f8c64a6e8a2d7c\src\NServiceBus.Core\Transports\Msmq\MsmqMessageSender.cs:line 75
at NServiceBus.Distributor.MSMQ.ReadyMessages.ReadyMessageSender.SendReadyMessage(String sessionId, Int32 capacityAvailable, Boolean isStarting) in c:\BuildAgent\work\c3100604bbd3ca20\src\NServiceBus.Distributor.MSMQ\ReadyMessages\ReadyMessageSender.cs:line 62
at NServiceBus.Distributor.MSMQ.ReadyMessages.ReadyMessageSender.TransportOnFinishedMessageProcessing(Object sender, FinishedMessageProcessingEventArgs e) in c:\BuildAgent\work\c3100604bbd3ca20\src\NServiceBus.Distributor.MSMQ\ReadyMessages\ReadyMessageSender.cs:line 50
at System.EventHandler1.Invoke(Object sender, TEventArgs e)
at NServiceBus.Unicast.Transport.TransportReceiver.OnFinishedMessageProcessing(TransportMessage msg) in c:\BuildAgent\work\31f8c64a6e8a2d7c\src\NServiceBus.Core\Unicast\Transport\TransportReceiver.cs:line 435
NServicebus version: 4.6.0.0
Queueing: MSMQ
The worker ends it unit of work by sending a message back to the distributor. This send will join the existing distributed transaction. The error you get is caused by this new transactional resource trying to join an already failing transaction. Something has marked the distributed transaction as rolling back.
This is normally caused by your code. Either your database operation is failing somehow or you probably have exceeded the transaction timeout limit handling the message. (Default one minute)
Check your logs to see if you are using above the transaction timeout limit to process the message on the worker.

Weird intermittent error occures while sending messages using NServiceBus

We are seeing very strange intermittent error in our logs with the NServiceBus code stack trace
NServiceBus version that we are using is 3.3.4
Here the stack trace of the error:
System.ArgumentException: An item with the same key has already been added.
at System.Collections.Generic.Dictionary`2.Insert(TKey key, TValue value, Boolean add)
at NServiceBus.MessageHeaders.MessageHeaderManager.NServiceBus.MessageMutator.IMutateOutgoingTransportMessages.MutateOutgoing(Object[] messages, TransportMessage transportMessage)
at NServiceBus.Unicast.UnicastBus.InvokeOutgoingTransportMessagesMutators(Object[] messages, TransportMessage result)
at NServiceBus.Unicast.UnicastBus.MapTransportMessageFor(Object[] rawMessages, TransportMessage result)
at NServiceBus.Unicast.UnicastBus.SendMessage(List`1 addresses, String correlationId, MessageIntentEnum messageIntent, Object[] messages)
at NServiceBus.Unicast.UnicastBus.SendMessage(Address address, String correlationId, MessageIntentEnum messageIntent, Object[] messages)
at NServiceBus.Unicast.UnicastBus.NServiceBus.IBus.Send(Object[] messages)
Any ideas what is the issue ?
Our production code from time to time fails because of this and we need to know why.
Thanks. Any help would be appreciated.

NServicebus : {WARN} failed raising transport message with ID =

Here's the scenario.
I am on NServiceBus 3.0.0.1504 (core dll version). I get above warning message when there's an exception on one of the handlers. Queues are setup as DTC. The interesting thing is, though I get this warning message in QA - it successfully retries N number of times and places the message in the error queue. However, in Production, (in few cases - I haven't seen this happening consistently), it just tries processing the message once and it stops. It didn't put the message in the error queue or neither it retired N Number of times nor it put the message back in the queue.
I don't really see the difference between QA and Prod environment, its the same code base, and we give permission to queues using build scripts.
Here's the stack trace
NServiceBus.Unicast.Transport.Transactional.TransactionalTransport - Failed raising 'transport message received' event for message with ID=1fb282b5-7a9e-41ea-834a-5f6767273324\195311762 NServiceBus.Unicast.Transport.TransportMessageHandlingFailedException: Exception of type 'NServiceBus.Unicast.Transport.TransportMessageHandlingFailedException' was thrown.
at NServiceBus.Unicast.UnicastBus.DispatchMessageToHandlersBasedOnType(IBuilder builder, IMessage toHandle, Type messageType) in d:\BuildAgent-01\work\NServiceBus.Trunk\src\unicast\NServiceBus.Unicast\UnicastBus.cs:line 899
at NServiceBus.Unicast.UnicastBus.HandleMessage(IBuilder builder, TransportMessage m) in d:\BuildAgent-01\work\NServiceBus.Trunk\src\unicast\NServiceBus.Unicast\UnicastBus.cs:line 827
at NServiceBus.Unicast.UnicastBus.HandleTransportMessage(IBuilder builder, TransportMessage msg) in d:\BuildAgent-01\work\NServiceBus.Trunk\src\unicast\NServiceBus.Unicast\UnicastBus.cs:line 1026
at NServiceBus.Unicast.UnicastBus.TransportMessageReceived(Object sender, TransportMessageReceivedEventArgs e) in d:\BuildAgent-01\work\NServiceBus.Trunk\src\unicast\NServiceBus.Unicast\UnicastBus.cs:line 975
at System.EventHandler`1.Invoke(Object sender, TEventArgs e)
at NServiceBus.Unicast.Transport.Transactional.TransactionalTransport.OnTransportMessageReceived(TransportMessage msg) in d:\BuildAgent-01\work\NServiceBus.Trunk\src\impl\unicast\transport\NServiceBus.Unicast.Transport.Transactional\TransactionalTransport.cs:line 409
I tried dropping the same message in QA to test the scenario, however, QA seems to work (with the warning exception log).
Questions
1) Why would this exception happen? Has anyone seen in their system? Why is it logged only as a WARN? I looked at the code and it seems to come from catch block of Unicast.cs where it is handling "Exception".
2) Any suggestions to resolve / dig into the issue?

The destination queue '<QueueName>#<servername>' could not be found

While testing the pub/sub model, I changed the name of the subscriber queue, while the subscription for the old queue still exists in the DB, so there is a dangling subscription in the DB.
So when publisher and subscriber started and I tried to send message from publisher, following exception happened and basically publisher stopped and no longer send any more message
2011-02-09 09:56:21,115 [6] ERROR Publisher.ServerEndpoint [(null)] <(null)> - Problem occurred when starting the endpoint.
System.Configuration.ConfigurationErrorsException: The destination queue 'StoreInputQueue#' could not be found. You may have misconfigured the destination for this kind of message (Message.EventMessage) in the MessageEndpointMappings of the UnicastBusConfig section in your configuration file.It may also be the case that the given queue just hasn't been created yet, or has been deleted. ---> System.Messaging.MessageQueueException: The queue does not exist or you do not have sufficient permissions to perform the operation.
at System.Messaging.MessageQueue.MQCacheableInfo.get_WriteHandle()
at System.Messaging.MessageQueue.StaleSafeSendMessage(MQPROPS properties, IntPtr transaction)
at System.Messaging.MessageQueue.SendInternal(Object obj, MessageQueueTransaction internalTransaction, MessageQueueTransactionType transactionType)
at System.Messaging.MessageQueue.Send(Object obj, MessageQueueTransactionType transactionType)
at NServiceBus.Unicast.Transport.Msmq.MsmqTransport.Send(TransportMessage m, String destination) in d:\BuildAgent-02\work\20b5f701adefe8f8\src\impl\unicast\NServiceBus.Unicast.Msmq\MsmqTransport.cs:line 334
--- End of inner exception stack trace ---
at NServiceBus.Unicast.Transport.Msmq.MsmqTransport.Send(TransportMessage m, String destination) in d:\BuildAgent-02\work\20b5f701adefe8f8\src\impl\unicast\NServiceBus.Unicast.Msmq\MsmqTransport.cs:line 346
at NServiceBus.Unicast.UnicastBus.SendMessage(IEnumerable`1 destinations, String correlationId, MessageIntentEnum messageIntent, IMessage[] messages) in d:\BuildAgent-02\work\20b5f701adefe8f8\src\unicast\NServiceBus.Unicast\UnicastBus.cs:line 593
at NServiceBus.Unicast.UnicastBus.Publish[T](T[] messages) in d:\BuildAgent-02\work\20b5f701adefe8f8\src\unicast\NServiceBus.Unicast\UnicastBus.cs:line 343
at Publisher.ServerEndpoint.Run() in C:\Downloads\ESB\NServiceBus\publisher\publisher\ServerEndpoint.cs:line 26
at NServiceBus.Host.Internal.ConfigManager.<>c_DisplayClass1.b_0() in d:\BuildAgent-02\work\20b5f701adefe8f8\src\host\NServiceBus.Host\Internal\ConfigurationManager.cs:line 56
Is there a timeout period after which it will try to send message to rest of subscribers, I waited quite long...
I don't think it will retry.
Pulling the rug (queue) out from under a running endpoint is not a good thing to do. In production this really should never happen.
Since you're just testing, delete the offending subscription row from the database, and restart the endpoint, and everything should be fine.

w3wp.exe crash - ThreadAbortException

we have a problem that we can't reproduce, it happens randomly on our web server (Windows Server 2008 Datacenter 64-bit, hosted on Amazon EC2) running ASP.NET 4.
It starts with a ASP.NET warning in the error log (a strange GET request with a really long URL):
Exception information:
Exception type: HttpException
Exception message: The length of the URL for this request exceeds the configured maxUrlLength value.
at System.Web.HttpRequest.ValidateInputIfRequiredByConfig()
at System.Web.HttpApplication.PipelineStepManager.ValidateHelper(HttpContext context)
Then an error:
Application ID: /LM/W3SVC/2/ROOT
Process ID: 4604
Exception: System.Threading.ThreadAbortException
Message: Thread was being aborted.
StackTrace: at System.Web.HttpRuntime.ProcessRequestNotificationPrivate(IIS7WorkerRequest wr, HttpContext context)
at System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr managedHttpContext, IntPtr nativeRequestContext, IntPtr moduleData, Int32 flags)
at System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr managedHttpContext, IntPtr nativeRequestContext, IntPtr moduleData, Int32 flags)
Then another error:
Application ID: DefaultDomain
Process ID: 4604
Exception: System.Threading.ThreadAbortException
Message: Thread was being aborted.
StackTrace:
And then the application error:
Faulting application w3wp.exe, version 7.0.6002.18005, time stamp 0x49e03238, faulting module kernel32.dll, version 6.0.6002.18005, time stamp 0x49e041d1, exception code 0xe0434352, fault offset 0x00000000000176fd, process id 0x%9, application start time 0x%10.
Windbg
I managed to get a dump for a crash with adplus, but I really don't know what to look for. I've troubleshooted some stackoverflows with Windbg before, but I don't know which approach to use for this error.
!pe on the thread with the error:
Exception object: 00000001c323d948
Exception type: System.Threading.ThreadAbortException
Message: Thread was being aborted.
InnerException:
StackTrace (generated):
StackTraceString:
HResult: 80131530
!clrstack
Child SP IP Call Site
0000000015f7f078 0000000076c176fd [GCFrame: 0000000015f7f078]
0000000015f7f258 0000000076c176fd [GCFrame: 0000000015f7f258]
k
Child-SP RetAddr Call Site
0000000015f7edf0 000007fef2826e39 kernel32!RaiseException+0x39
0000000015f7eec0 000007fef2bbbfb4 clr!RaiseTheExceptionInternalOnly+0x363
0000000015f7eff0 000007fef2bbc906 clr!RaiseTheException+0xa4
0000000015f7f020 000007fef2c3b99b clr!BStrFromString+0x66
0000000015f7f050 000007fef2c3b9a9 clr!RealCOMPlusThrow+0x3b
0000000015f7f0c0 000007fef2c93156 clr!RealCOMPlusThrow+0x9
0000000015f7f0f0 000007fef2b229f3 clr!Thread::RaiseCrossContextException+0x2a7
0000000015f7f310 000007fef2830886 clr! ?? ::FNODOBFM::string'+0xafb03
0000000015f7f4c0 000007fef27fcce3 clr!UM2MDoADCallBack+0x9e
0000000015f7f530 000007fef845ba59 clr!UMThunkStubAMD64+0x273
0000000015f7f5c0 000007fef8458f02 webengine4!W3_MGD_HANDLER::ProcessNotification+0x79
0000000015f7f5f0 000007fef27d4595 webengine4!ProcessNotificationCallback+0x43
0000000015f7f620 000007fef27d3ac8 clr!UnManagedPerAppDomainTPCount::DispatchWorkItem+0x181
0000000015f7f6d0 000007fef294658f clr!ThreadpoolMgr::NewWorkerThreadStart+0x2e5
0000000015f7f770 000007fef29447c6 clr!ThreadpoolMgr::WorkerThreadStart+0x3b
0000000015f7f810 0000000076c1be3d clr!Thread::intermediateThreadProc+0x7d
0000000015f7fbd0 0000000076d56a51 kernel32!BaseThreadInitThunk+0xd
0000000015f7fc00 00000000`00000000 ntdll!RtlUserThreadStart+0x1d
Anybody have a clue what this could be? Or can point me in the right direction for analyzing it with Windbg?
Edit:
The incoming urls usually looks like this:
foo.bar.com/wEPDwULLTE1MTk5MzIzMTFkGAMFFmN0bDAwJGRiMSRkZGxEYXRhYmFzZXMPFCsAAmRkZAU7Y3RsMDAkU2VhcmNoQ2xvdWQxJF9SaWdodENvbHVtbiRfU2VhcmNoQ2xvdWQkbHN2U2VhcmNoVGVybXMPFCsADmRkZGRkZGQ8KwAUAAIUZGRkZgL/D2QFK2N0bDAwJHN1cnZleTEkX1JpZ2h0Q29sdW1uJF9JUiR1c2VyQ29tbWVudHMPFCsAA2VnZGQeuUcvQDsShDIp1k7YjJw70Ry 9/Q1B9Sd1egrovYgkw==/
but I found in the event log that this happen for this urls like this aswell:
foo.bar.com/&
("dangerous request" in .NET 4)
You should at least load symbols and then see the correct strings displayed in call stack.
If you are not capable of handling such analysis, I think opening a support case via http://support.microsoft.com is good for you.
I have been trying to work out a similar issue.
I found this article which purports a process for determining the cause, I think it actually makes sense:
http://blogs.msdn.com/b/asiatech/archive/2012/06/21/how-to-troubleshoot-httpexception-request-timed-out-asp-net-4-0-64-bit.aspx
Caveat: I have not been able to sucessfully complete the steps described. I will be posting here asking for help.