We are seeing very strange intermittent error in our logs with the NServiceBus code stack trace
NServiceBus version that we are using is 3.3.4
Here the stack trace of the error:
System.ArgumentException: An item with the same key has already been added.
at System.Collections.Generic.Dictionary`2.Insert(TKey key, TValue value, Boolean add)
at NServiceBus.MessageHeaders.MessageHeaderManager.NServiceBus.MessageMutator.IMutateOutgoingTransportMessages.MutateOutgoing(Object[] messages, TransportMessage transportMessage)
at NServiceBus.Unicast.UnicastBus.InvokeOutgoingTransportMessagesMutators(Object[] messages, TransportMessage result)
at NServiceBus.Unicast.UnicastBus.MapTransportMessageFor(Object[] rawMessages, TransportMessage result)
at NServiceBus.Unicast.UnicastBus.SendMessage(List`1 addresses, String correlationId, MessageIntentEnum messageIntent, Object[] messages)
at NServiceBus.Unicast.UnicastBus.SendMessage(Address address, String correlationId, MessageIntentEnum messageIntent, Object[] messages)
at NServiceBus.Unicast.UnicastBus.NServiceBus.IBus.Send(Object[] messages)
Any ideas what is the issue ?
Our production code from time to time fails because of this and we need to know why.
Thanks. Any help would be appreciated.
Related
We have seen a noticeable uptick in problems related to RavenDB connections. We're using the IIS server connection method.
I've checked the index errors and I don't see anything listed.
This server is replicated with a MySQL server and there is one error related to replication in the log from today. Could the client stack trace errors actually be coming from the replicated server and not Raven?
EDIT
The client in this case is a single job that runs a few hundred times a day...succeeding many of those times but increasing in failures due to these errors.
Sql Replication failure to replication
Here are the partial stack trace errors from the client log:
[WebException: Unable to connect to the remote server]
System.Net.HttpWebRequest.GetRequestStream(TransportContext& context):367
System.Net.HttpWebRequest.GetRequestStream():3
Raven.Abstractions.Connection.HttpRequestHelper.WriteDataToRequest(HttpWebRequest req, String data, Boolean disableCompression):7
Raven.Client.Connection.ServerClient.DirectBatch(IEnumerable`1 commandDatas, String operationUrl):171
Raven.Client.Connection.ReplicationInformer.TryOperation[T](Func`2 operation, String operationUrl, Boolean avoidThrowing, T& result, Boolean& wasTimeout):35
Raven.Client.Connection.ReplicationInformer.ExecuteWithReplication[T](String method, String primaryUrl, Int32 currentRequest, Int32 currentReadStripingBase, Func`2 operation):169
Raven.Client.Connection.ServerClient.ExecuteWithReplication[T](String method, Func`2 operation):33
Raven.Client.Document.DocumentSession.SaveChanges():65
and
[WebException: Unable to connect to the remote server]
System.Net.HttpWebRequest.GetResponse():570
Raven.Client.Connection.HttpJsonRequest.ReadJsonInternal(Func`1 getResponse):45
Raven.Client.Connection.HttpJsonRequest.ReadResponseJson():206
Raven.Client.Connection.ServerClient.DirectGet(String[] ids, String operationUrl, String[] includes, String transformer, Dictionary`2 queryInputs, Boolean metadataOnly):631
Raven.Client.Connection.ServerClient+<>c__DisplayClass77.<Get>b__76(String u):51
Raven.Client.Connection.ReplicationInformer.TryOperation[T](Func`2 operation, String operationUrl, Boolean avoidThrowing, T& result, Boolean& wasTimeout):35
Raven.Client.Connection.ReplicationInformer.ExecuteWithReplication[T](String method, String primaryUrl, Int32 currentRequest, Int32 currentReadStripingBase, Func`2 operation):169
Raven.Client.Connection.ServerClient.ExecuteWithReplication[T](String method, Func`2 operation):33
Raven.Client.Document.HiLoKeyGenerator.GetDocument(IDatabaseCommands databaseCommands):41
Raven.Client.Document.HiLoKeyGenerator.GetNextRange(IDatabaseCommands databaseCommands):109
Raven.Client.Document.HiLoKeyGenerator.NextId(IDatabaseCommands commands):58
Raven.Client.Document.HiLoKeyGenerator.GenerateDocumentKey(IDatabaseCommands databaseCommands, DocumentConvention convention, Object entity):9
Raven.Client.Document.MultiTypeHiLoKeyGenerator.GenerateDocumentKey(IDatabaseCommands databaseCommands, DocumentConvention conventions, Object entity):174
Raven.Client.Document.DocumentStore+<>c__DisplayClass4.<Initialize>b__2(String dbName, IDatabaseCommands databaseCommands, Object entity):20
Raven.Client.Document.DocumentConvention.GenerateDocumentKey(String dbName, IDatabaseCommands databaseCommands, Object entity):164
Raven.Client.Document.GenerateEntityIdOnTheClient.GenerateDocumentKeyForStorage(Object entity):46
Raven.Client.Document.InMemoryDocumentSessionOperations.StoreInternal(Object entity, Etag etag, String id, Boolean forceConcurrencyCheck):79
Raven.Client.Document.InMemoryDocumentSessionOperations.Store(Object entity):23
The error says the client cannot connect to the the server. Maybe the client is offline. Or the server is. Or a firewall is in the way. Or maybe the server is there but blows up before returning a response to the client. If the issue is intermittent, and users arent complaining, it's probably just a connectivity thing that you can ignore. If users are complaining, you should look in server-side logs.
I have the following scenario.
First write something to a db
Send message on bus
I want to execute those like an atomic action, that is, running them in a transaction but can't get it to work when it is really distributed. Everything was working fint on local machine, but as soon as the service was on another machine it stopped working.
My code looks something like this:
using(var ts = new TransactionScope())
{
// Write something to db
_bus.Send(SomeMessage);
ts.Complete();
}
I have done everything I could think of and find information about, like enabling DTC on client, server and db with inbound and outbound enabled, set the port DCOM protocols to 5000-6000. The error I get is:
FailedToSendMessageException: "Failed to send message to address: myserver"
and the stacktrace is:
at NServiceBus.Transports.Msmq.MsmqMessageSender.ThrowFailedToSendException(Address address, Exception ex) in c:\BuildAgent\work\d4de8921a0aabf04\src\NServiceBus.Core\Transports\Msmq\MsmqMessageSender.cs:line 89
at NServiceBus.Transports.Msmq.MsmqMessageSender.Send(TransportMessage message, Address address) in c:\BuildAgent\work\d4de8921a0aabf04\src\NServiceBus.Core\Transports\Msmq\MsmqMessageSender.cs:line 80
at NServiceBus.Unicast.UnicastBus.SendMessage(List`1 addresses, String correlationId, MessageIntentEnum messageIntent, Object[] messages) in c:\BuildAgent\work\d4de8921a0aabf04\src\NServiceBus.Core\Unicast\UnicastBus.cs:line 658
at NServiceBus.Unicast.UnicastBus.SendMessage(Address address, String correlationId, MessageIntentEnum messageIntent, Object[] messages) in c:\BuildAgent\work\d4de8921a0aabf04\src\NServiceBus.Core\Unicast\UnicastBus.cs:line 583
at NServiceBus.Unicast.UnicastBus.NServiceBus.IBus.Send(Object[] messages) in c:\BuildAgent\work\d4de8921a0aabf04\src\NServiceBus.Core\Unicast\UnicastBus.cs:line 480
at Rapporteringsregisteret.Web.Controllers.RapporteringController.Post(OpprettRapportering opprettRapportering) in c:\dev\git\Rapporteringsregisteret\src\Rapporteringsregisteret.Web\Controllers\RapporteringController.cs:line 37
at lambda_method(Closure , Object , Object[] )
at System.Web.Http.Controllers.ReflectedHttpActionDescriptor.ActionExecutor.<>c__DisplayClass10.<GetExecutor>b__9(Object instance, Object[] methodParameters)
at System.Web.Http.Controllers.ReflectedHttpActionDescriptor.ActionExecutor.Execute(Object instance, Object[] arguments)
at System.Web.Http.Controllers.ReflectedHttpActionDescriptor.ExecuteAsync(HttpControllerContext controllerContext, IDictionary`2 arguments, CancellationToken cancellationToken)
Why isn't this working?
When running in a distributed scenario your endpoint mapping needs to explicitely define the machine name where the queue of the receiver is located. When running on the same box this is not necessary. Therefore your mapping should not contain "myserver" but "myserver#machinename". This error is not related to DTC. For future questions always include Nservicebus version number and transport layer
I have a connector( which receives messages from different connectors. While receiving the message my connector gives the following message:
[Worker.11] WARN NServiceBus.Unicast.Transport.Transact
ional.TransactionalTransport [(null)] <(null)> - Failed raising 'transport messa
ge received' event for message with ID=fd970068-55ad-49c0-8abc-4133b7f7fe12\2138
47
NServiceBus.Unicast.Queuing.QueueNotFoundException ---> System.Messaging.Message
QueueException: Cannot enlist the transaction.
at System.Messaging.MessageQueue.SendInternal(Object obj, MessageQueueTransac
tion internalTransaction, MessageQueueTransactionType transactionType)
at NServiceBus.Unicast.Queuing.Msmq.MsmqMessageSender.NServiceBus.Unicast.Que
uing.ISendMessages.Send(TransportMessage message, Address address)
--- End of inner exception stack trace ---
at NServiceBus.Unicast.UnicastBus.HandleTransportMessage(IBuilder childBuilde
r, TransportMessage msg)
at NServiceBus.Unicast.UnicastBus.TransportMessageReceived(Object sender, Tra
nsportMessageReceivedEventArgs e)
at System.EventHandler`1.Invoke(Object sender, TEventArgs e)
at NServiceBus.Unicast.Transport.Transactional.TransactionalTransport.OnTrans
portMessageReceived(TransportMessage msg)
The weird thing is when i was doing the internal testing it was working all fine as soon as this has been now hosted on to amazon server it has started blowing up.
The following is how i have configured my nservice bus with
NServiceBus.Configure.With(
AllAssemblies.Except("libBL.dll").And("libCommon.dll").And("libExtra.dll")).StructureMapBuilder()
.JsonSerializer()
.UnicastBus()
.DoNotAutoSubscribe()
.TransactionTimeout(TimeSpan.FromMinutes(5));
Any help would be much appreciated. If any more information is needed please let me know
Thanks
On searching through the code found out that if a message coming in was failing then i was not rolling back the transaction in which the whole operation began and hence the nservicebus was not able to query the database as it was locked. After putting in the logic for transaction roll back got my connector to work.
Here's the scenario.
I am on NServiceBus 3.0.0.1504 (core dll version). I get above warning message when there's an exception on one of the handlers. Queues are setup as DTC. The interesting thing is, though I get this warning message in QA - it successfully retries N number of times and places the message in the error queue. However, in Production, (in few cases - I haven't seen this happening consistently), it just tries processing the message once and it stops. It didn't put the message in the error queue or neither it retired N Number of times nor it put the message back in the queue.
I don't really see the difference between QA and Prod environment, its the same code base, and we give permission to queues using build scripts.
Here's the stack trace
NServiceBus.Unicast.Transport.Transactional.TransactionalTransport - Failed raising 'transport message received' event for message with ID=1fb282b5-7a9e-41ea-834a-5f6767273324\195311762 NServiceBus.Unicast.Transport.TransportMessageHandlingFailedException: Exception of type 'NServiceBus.Unicast.Transport.TransportMessageHandlingFailedException' was thrown.
at NServiceBus.Unicast.UnicastBus.DispatchMessageToHandlersBasedOnType(IBuilder builder, IMessage toHandle, Type messageType) in d:\BuildAgent-01\work\NServiceBus.Trunk\src\unicast\NServiceBus.Unicast\UnicastBus.cs:line 899
at NServiceBus.Unicast.UnicastBus.HandleMessage(IBuilder builder, TransportMessage m) in d:\BuildAgent-01\work\NServiceBus.Trunk\src\unicast\NServiceBus.Unicast\UnicastBus.cs:line 827
at NServiceBus.Unicast.UnicastBus.HandleTransportMessage(IBuilder builder, TransportMessage msg) in d:\BuildAgent-01\work\NServiceBus.Trunk\src\unicast\NServiceBus.Unicast\UnicastBus.cs:line 1026
at NServiceBus.Unicast.UnicastBus.TransportMessageReceived(Object sender, TransportMessageReceivedEventArgs e) in d:\BuildAgent-01\work\NServiceBus.Trunk\src\unicast\NServiceBus.Unicast\UnicastBus.cs:line 975
at System.EventHandler`1.Invoke(Object sender, TEventArgs e)
at NServiceBus.Unicast.Transport.Transactional.TransactionalTransport.OnTransportMessageReceived(TransportMessage msg) in d:\BuildAgent-01\work\NServiceBus.Trunk\src\impl\unicast\transport\NServiceBus.Unicast.Transport.Transactional\TransactionalTransport.cs:line 409
I tried dropping the same message in QA to test the scenario, however, QA seems to work (with the warning exception log).
Questions
1) Why would this exception happen? Has anyone seen in their system? Why is it logged only as a WARN? I looked at the code and it seems to come from catch block of Unicast.cs where it is handling "Exception".
2) Any suggestions to resolve / dig into the issue?
While testing the pub/sub model, I changed the name of the subscriber queue, while the subscription for the old queue still exists in the DB, so there is a dangling subscription in the DB.
So when publisher and subscriber started and I tried to send message from publisher, following exception happened and basically publisher stopped and no longer send any more message
2011-02-09 09:56:21,115 [6] ERROR Publisher.ServerEndpoint [(null)] <(null)> - Problem occurred when starting the endpoint.
System.Configuration.ConfigurationErrorsException: The destination queue 'StoreInputQueue#' could not be found. You may have misconfigured the destination for this kind of message (Message.EventMessage) in the MessageEndpointMappings of the UnicastBusConfig section in your configuration file.It may also be the case that the given queue just hasn't been created yet, or has been deleted. ---> System.Messaging.MessageQueueException: The queue does not exist or you do not have sufficient permissions to perform the operation.
at System.Messaging.MessageQueue.MQCacheableInfo.get_WriteHandle()
at System.Messaging.MessageQueue.StaleSafeSendMessage(MQPROPS properties, IntPtr transaction)
at System.Messaging.MessageQueue.SendInternal(Object obj, MessageQueueTransaction internalTransaction, MessageQueueTransactionType transactionType)
at System.Messaging.MessageQueue.Send(Object obj, MessageQueueTransactionType transactionType)
at NServiceBus.Unicast.Transport.Msmq.MsmqTransport.Send(TransportMessage m, String destination) in d:\BuildAgent-02\work\20b5f701adefe8f8\src\impl\unicast\NServiceBus.Unicast.Msmq\MsmqTransport.cs:line 334
--- End of inner exception stack trace ---
at NServiceBus.Unicast.Transport.Msmq.MsmqTransport.Send(TransportMessage m, String destination) in d:\BuildAgent-02\work\20b5f701adefe8f8\src\impl\unicast\NServiceBus.Unicast.Msmq\MsmqTransport.cs:line 346
at NServiceBus.Unicast.UnicastBus.SendMessage(IEnumerable`1 destinations, String correlationId, MessageIntentEnum messageIntent, IMessage[] messages) in d:\BuildAgent-02\work\20b5f701adefe8f8\src\unicast\NServiceBus.Unicast\UnicastBus.cs:line 593
at NServiceBus.Unicast.UnicastBus.Publish[T](T[] messages) in d:\BuildAgent-02\work\20b5f701adefe8f8\src\unicast\NServiceBus.Unicast\UnicastBus.cs:line 343
at Publisher.ServerEndpoint.Run() in C:\Downloads\ESB\NServiceBus\publisher\publisher\ServerEndpoint.cs:line 26
at NServiceBus.Host.Internal.ConfigManager.<>c_DisplayClass1.b_0() in d:\BuildAgent-02\work\20b5f701adefe8f8\src\host\NServiceBus.Host\Internal\ConfigurationManager.cs:line 56
Is there a timeout period after which it will try to send message to rest of subscribers, I waited quite long...
I don't think it will retry.
Pulling the rug (queue) out from under a running endpoint is not a good thing to do. In production this really should never happen.
Since you're just testing, delete the offending subscription row from the database, and restart the endpoint, and everything should be fine.