How do I configure distributed transactions for NServiceBus - nservicebus

I have the following scenario.
First write something to a db
Send message on bus
I want to execute those like an atomic action, that is, running them in a transaction but can't get it to work when it is really distributed. Everything was working fint on local machine, but as soon as the service was on another machine it stopped working.
My code looks something like this:
using(var ts = new TransactionScope())
{
// Write something to db
_bus.Send(SomeMessage);
ts.Complete();
}
I have done everything I could think of and find information about, like enabling DTC on client, server and db with inbound and outbound enabled, set the port DCOM protocols to 5000-6000. The error I get is:
FailedToSendMessageException: "Failed to send message to address: myserver"
and the stacktrace is:
at NServiceBus.Transports.Msmq.MsmqMessageSender.ThrowFailedToSendException(Address address, Exception ex) in c:\BuildAgent\work\d4de8921a0aabf04\src\NServiceBus.Core\Transports\Msmq\MsmqMessageSender.cs:line 89
at NServiceBus.Transports.Msmq.MsmqMessageSender.Send(TransportMessage message, Address address) in c:\BuildAgent\work\d4de8921a0aabf04\src\NServiceBus.Core\Transports\Msmq\MsmqMessageSender.cs:line 80
at NServiceBus.Unicast.UnicastBus.SendMessage(List`1 addresses, String correlationId, MessageIntentEnum messageIntent, Object[] messages) in c:\BuildAgent\work\d4de8921a0aabf04\src\NServiceBus.Core\Unicast\UnicastBus.cs:line 658
at NServiceBus.Unicast.UnicastBus.SendMessage(Address address, String correlationId, MessageIntentEnum messageIntent, Object[] messages) in c:\BuildAgent\work\d4de8921a0aabf04\src\NServiceBus.Core\Unicast\UnicastBus.cs:line 583
at NServiceBus.Unicast.UnicastBus.NServiceBus.IBus.Send(Object[] messages) in c:\BuildAgent\work\d4de8921a0aabf04\src\NServiceBus.Core\Unicast\UnicastBus.cs:line 480
at Rapporteringsregisteret.Web.Controllers.RapporteringController.Post(OpprettRapportering opprettRapportering) in c:\dev\git\Rapporteringsregisteret\src\Rapporteringsregisteret.Web\Controllers\RapporteringController.cs:line 37
at lambda_method(Closure , Object , Object[] )
at System.Web.Http.Controllers.ReflectedHttpActionDescriptor.ActionExecutor.<>c__DisplayClass10.<GetExecutor>b__9(Object instance, Object[] methodParameters)
at System.Web.Http.Controllers.ReflectedHttpActionDescriptor.ActionExecutor.Execute(Object instance, Object[] arguments)
at System.Web.Http.Controllers.ReflectedHttpActionDescriptor.ExecuteAsync(HttpControllerContext controllerContext, IDictionary`2 arguments, CancellationToken cancellationToken)
Why isn't this working?

When running in a distributed scenario your endpoint mapping needs to explicitely define the machine name where the queue of the receiver is located. When running on the same box this is not necessary. Therefore your mapping should not contain "myserver" but "myserver#machinename". This error is not related to DTC. For future questions always include Nservicebus version number and transport layer

Related

Weird intermittent error occures while sending messages using NServiceBus

We are seeing very strange intermittent error in our logs with the NServiceBus code stack trace
NServiceBus version that we are using is 3.3.4
Here the stack trace of the error:
System.ArgumentException: An item with the same key has already been added.
at System.Collections.Generic.Dictionary`2.Insert(TKey key, TValue value, Boolean add)
at NServiceBus.MessageHeaders.MessageHeaderManager.NServiceBus.MessageMutator.IMutateOutgoingTransportMessages.MutateOutgoing(Object[] messages, TransportMessage transportMessage)
at NServiceBus.Unicast.UnicastBus.InvokeOutgoingTransportMessagesMutators(Object[] messages, TransportMessage result)
at NServiceBus.Unicast.UnicastBus.MapTransportMessageFor(Object[] rawMessages, TransportMessage result)
at NServiceBus.Unicast.UnicastBus.SendMessage(List`1 addresses, String correlationId, MessageIntentEnum messageIntent, Object[] messages)
at NServiceBus.Unicast.UnicastBus.SendMessage(Address address, String correlationId, MessageIntentEnum messageIntent, Object[] messages)
at NServiceBus.Unicast.UnicastBus.NServiceBus.IBus.Send(Object[] messages)
Any ideas what is the issue ?
Our production code from time to time fails because of this and we need to know why.
Thanks. Any help would be appreciated.

RavenDB connection issues

We have seen a noticeable uptick in problems related to RavenDB connections. We're using the IIS server connection method.
I've checked the index errors and I don't see anything listed.
This server is replicated with a MySQL server and there is one error related to replication in the log from today. Could the client stack trace errors actually be coming from the replicated server and not Raven?
EDIT
The client in this case is a single job that runs a few hundred times a day...succeeding many of those times but increasing in failures due to these errors.
Sql Replication failure to replication
Here are the partial stack trace errors from the client log:
[WebException: Unable to connect to the remote server]
System.Net.HttpWebRequest.GetRequestStream(TransportContext& context):367
System.Net.HttpWebRequest.GetRequestStream():3
Raven.Abstractions.Connection.HttpRequestHelper.WriteDataToRequest(HttpWebRequest req, String data, Boolean disableCompression):7
Raven.Client.Connection.ServerClient.DirectBatch(IEnumerable`1 commandDatas, String operationUrl):171
Raven.Client.Connection.ReplicationInformer.TryOperation[T](Func`2 operation, String operationUrl, Boolean avoidThrowing, T& result, Boolean& wasTimeout):35
Raven.Client.Connection.ReplicationInformer.ExecuteWithReplication[T](String method, String primaryUrl, Int32 currentRequest, Int32 currentReadStripingBase, Func`2 operation):169
Raven.Client.Connection.ServerClient.ExecuteWithReplication[T](String method, Func`2 operation):33
Raven.Client.Document.DocumentSession.SaveChanges():65
and
[WebException: Unable to connect to the remote server]
System.Net.HttpWebRequest.GetResponse():570
Raven.Client.Connection.HttpJsonRequest.ReadJsonInternal(Func`1 getResponse):45
Raven.Client.Connection.HttpJsonRequest.ReadResponseJson():206
Raven.Client.Connection.ServerClient.DirectGet(String[] ids, String operationUrl, String[] includes, String transformer, Dictionary`2 queryInputs, Boolean metadataOnly):631
Raven.Client.Connection.ServerClient+<>c__DisplayClass77.<Get>b__76(String u):51
Raven.Client.Connection.ReplicationInformer.TryOperation[T](Func`2 operation, String operationUrl, Boolean avoidThrowing, T& result, Boolean& wasTimeout):35
Raven.Client.Connection.ReplicationInformer.ExecuteWithReplication[T](String method, String primaryUrl, Int32 currentRequest, Int32 currentReadStripingBase, Func`2 operation):169
Raven.Client.Connection.ServerClient.ExecuteWithReplication[T](String method, Func`2 operation):33
Raven.Client.Document.HiLoKeyGenerator.GetDocument(IDatabaseCommands databaseCommands):41
Raven.Client.Document.HiLoKeyGenerator.GetNextRange(IDatabaseCommands databaseCommands):109
Raven.Client.Document.HiLoKeyGenerator.NextId(IDatabaseCommands commands):58
Raven.Client.Document.HiLoKeyGenerator.GenerateDocumentKey(IDatabaseCommands databaseCommands, DocumentConvention convention, Object entity):9
Raven.Client.Document.MultiTypeHiLoKeyGenerator.GenerateDocumentKey(IDatabaseCommands databaseCommands, DocumentConvention conventions, Object entity):174
Raven.Client.Document.DocumentStore+<>c__DisplayClass4.<Initialize>b__2(String dbName, IDatabaseCommands databaseCommands, Object entity):20
Raven.Client.Document.DocumentConvention.GenerateDocumentKey(String dbName, IDatabaseCommands databaseCommands, Object entity):164
Raven.Client.Document.GenerateEntityIdOnTheClient.GenerateDocumentKeyForStorage(Object entity):46
Raven.Client.Document.InMemoryDocumentSessionOperations.StoreInternal(Object entity, Etag etag, String id, Boolean forceConcurrencyCheck):79
Raven.Client.Document.InMemoryDocumentSessionOperations.Store(Object entity):23
The error says the client cannot connect to the the server. Maybe the client is offline. Or the server is. Or a firewall is in the way. Or maybe the server is there but blows up before returning a response to the client. If the issue is intermittent, and users arent complaining, it's probably just a connectivity thing that you can ignore. If users are complaining, you should look in server-side logs.

Booksleeve (1.3.39) + Redis 2.6.16 + Sentinel: The connection has been closed (Error)

I am getting some random disconnects from Booksleeve when deployed in the production environment. I have run tests on my machine and I can not seem to reproduce the errors. I am not sure what to check so I decided to post here.
My set-up:
2 x Redis 2.6.16 (Debian Linux)
2 x Redis sentinel (running on the same machines as redis)
4 x Windows Server 2008 R2
Application connection handling
My app has one static connection that is opened using ConnectionUtils. I also have a timer (System.Timers.Timer) that runs every connection.Ping every 5'' to check whether the connection is still open. If I get an exception for the Ping I log it, close and dispose the connection and replace it with a new one using ConnectionUtils again.
I am also logging Task.UnobservedExceptions
Here is what I am getting in my logs:
A Task's exception(s) were not observed either by Waiting on the Task or accessing its Exception property. As a result, the unobserved exception was rethrown by the finalizer thread. ---> System.IO.IOException: Unable to write data to the transport connection: An established connection was aborted by the software in your host machine. ---> System.Net.Sockets.SocketException: An established connection was aborted by the software in your host machine
at System.Net.Sockets.NetworkStream.Write(Byte[] buffer, Int32 offset, Int32 size)
--- End of inner exception stack trace ---
at System.Net.Sockets.NetworkStream.Write(Byte[] buffer, Int32 offset, Int32 size)
at System.IO.BufferedStream.Write(Byte[] array, Int32 offset, Int32 count)
at BookSleeve.RedisMessage.WriteUnified(Stream stream, Byte[] value) in d:\Dev\BookSleeve\BookSleeve\RedisCommand.cs:line 271
at BookSleeve.RedisMessage.RedisMessageMulti.Write(Stream stream) in d:\Dev\BookSleeve\BookSleeve\RedisCommand.cs:line 564
at BookSleeve.RedisConnectionBase.WriteMessage(Int32& db, RedisMessage next, IList`1 queued) in d:\Dev\BookSleeve\BookSleeve\RedisConnectionBase.cs:line 1597
at BookSleeve.RedisConnectionBase.EnqueueMessage(RedisMessage message, Boolean queueJump) in d:\Dev\BookSleeve\BookSleeve\RedisConnectionBase.cs:line 1965
at BookSleeve.RedisConnectionBase.ExecuteInt64(RedisMessage message, Boolean queueJump, Object state) in d:\Dev\BookSleeve\BookSleeve\RedisConnectionBase.cs:line 1688
at System.Threading.Tasks.Task.Execute()
--- End of inner exception stack trace ---
---> (Inner Exception #0) System.IO.IOException: Unable to write data to the transport connection: An established connection was aborted by the software in your host machine. ---> System.Net.Sockets.SocketException: An established connection was aborted by the software in your host machine
at System.Net.Sockets.NetworkStream.Write(Byte[] buffer, Int32 offset, Int32 size)
--- End of inner exception stack trace ---
at System.Net.Sockets.NetworkStream.Write(Byte[] buffer, Int32 offset, Int32 size)
at System.IO.BufferedStream.Write(Byte[] array, Int32 offset, Int32 count)
at BookSleeve.RedisMessage.WriteUnified(Stream stream, Byte[] value) in d:\Dev\BookSleeve\BookSleeve\RedisCommand.cs:line 271
at BookSleeve.RedisMessage.RedisMessageMulti.Write(Stream stream) in d:\Dev\BookSleeve\BookSleeve\RedisCommand.cs:line 564
at BookSleeve.RedisConnectionBase.WriteMessage(Int32& db, RedisMessage next, IList`1 queued) in d:\Dev\BookSleeve\BookSleeve\RedisConnectionBase.cs:line 1597
at BookSleeve.RedisConnectionBase.EnqueueMessage(RedisMessage message, Boolean queueJump) in d:\Dev\BookSleeve\BookSleeve\RedisConnectionBase.cs:line 1965
at BookSleeve.RedisConnectionBase.ExecuteInt64(RedisMessage message, Boolean queueJump, Object state) in d:\Dev\BookSleeve\BookSleeve\RedisConnectionBase.cs:line 1688
at System.Threading.Tasks.Task.Execute()
And this is my connection monitor reporting on the error in the ping:
Redis connection is broken: System.InvalidOperationException: The connection has been closed (Error); no new messages can be delivered; the last command was sent 1453ms ago
at BookSleeve.RedisConnectionBase.EnqueueMessage(RedisMessage message, Boolean queueJump) in d:\Dev\BookSleeve\BookSleeve\RedisConnectionBase.cs:line 1965
at BookSleeve.RedisConnectionBase.ExecuteInt64(RedisMessage message, Boolean queueJump, Object state) in d:\Dev\BookSleeve\BookSleeve\RedisConnectionBase.cs:line 1688
at BookSleeve.RedisConnectionBase.PingImpl(Boolean queueJump, Boolean duringInit, Object state) in d:\Dev\BookSleeve\BookSleeve\RedisConnectionBase.cs:line 129
at Atcom.Data.Caching.RedisConnectionContainer.CheckConnections(Object sender, EventArgs e) in C:\Projects\Atcom.Data.Caching.Redis\Atcom.Data.Caching.Redis\RedisConnectionContainer.cs:line 66
I have tried connecting through both the Sentinels and directly to Redis but I am getting the same thing.
I have asked my network administrator to check but he says there is no problem (and I cant really argue with what I dont know oO)
I have reviewed Redis settings and documentation and as far as I can tell clients that do not use monitor etc. should not get disconnected by redis. Idle time disconnect is also disabled on redis.
Could sentinel (since its beta) be the culprit?
Any help troubleshooting the problem would be GREATLY appreciated.

Access Denied Error for a method in web service

I have a webservice that works perfectly from localhost but when I hosted it on the testing server I get an error on one of the methods. I am using wsHttp Binding. everything else works fine on the client side except for this method. This is the error that I am getting
client error:
System.ServiceModel.Security.SecurityAccessDeniedException: Access is denied.
Server stack trace:
at System.ServiceModel.Channels.ServiceChannel.ThrowIfFaultUnderstood(Message reply, MessageFault fault, String action, MessageVersion version, FaultConverter faultConverter)
at System.ServiceModel.Channels.ServiceChannel.HandleReply(ProxyOperationRuntime operation, ProxyRpc& rpc)
at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)
Server error:
namespace.Service Error: 10001 : Error occurred in methodname().
System.Security.SecurityException: The source was not found, but some or all event logs could not be searched. Inaccessible logs: Security.
at System.Diagnostics.EventLog.FindSourceRegistration(String source, String machineName, Boolean readOnly)
at System.Diagnostics.EventLog.SourceExists(String source, String machineName)
at System.Diagnostics.EventLog.VerifyAndCreateSource(String sourceName, String currentMachineName)
at System.Diagnostics.EventLog.WriteEvent(EventInstance instance, Byte[] data, Object[] values)
at System.Diagnostics.EventLog.WriteEvent(EventInstance instance, Object[] values)
at System.Diagnostics.EventLogTraceListener.TraceEvent(TraceEventCache eventCache, String source, TraceEventType severity, Int32 id, String format, Object[] args)
at System.Diagnostics.TraceSource.TraceEvent(TraceEventType eventType, Int32 id, String format, Object[] args)
at AutoWatch.Entity.WcfService.TrackingService.UpdateIncidentStatusHistory(Int64 incidentId, String status, String username, String comment, Boolean SuspectFaultyUnit) in C:\..servicename.cs:line 566
at AutoWatch.Entity.WcfService.TrackingService.GetNewIncidentMessage(String username) in C:\..servicename.cs:line 444
The Zone of the assembly that failed was:
MyComputer
I added in the error I am getting on the server.
Is it possible I am getting this error because the service cannot write to the event log?
Please help.
Ensure that account running your process where the service is hosted has access rights to the database. For example in case of IIS the account running the application pool where the service is hosted must have login to database server and it must have permissions to do all necessary operations in your database.
Edit:
The server stack trace looks quite straightforward. You have a problem with writing to Windows Event Log! It cannot find source you requested and it doesn't have permission to create it.
I got the same error when I tried to use the following statement in a Web Method of a WCF Service:
string myTypeName = typeof(ErrorHandlerBehavior).AssemblyQualifiedName;
where ErrorHandlerBehavior derives from BehaviorExtensionElement.
The line works well when I test the service on my localhost. On the web, the call of the containing method raises the Exception "Access is denied".
In this case, I guess that the cause was that my IP does not allow the creation of a BehaviourExtensionElement in a Partially Trusted environment (my service is in a shared hosting environment).
Finally, I succeded following the first mechanism described at https://learn.microsoft.com/en-us/dotnet/framework/wcf/extending/configuring-and-extending-the-runtime-with-behaviors about Service Behaviors:
"Using an attribute on the service class. When a ServiceHost is constructed, the ServiceHost implementation uses reflection to discover the set of attributes on the type of the service. If any of those attributes are implementations of IServiceBehavior, they are added to the behaviors collection on ServiceDescription. This allows those behaviors to participate in the construction of the service run time."
I just modified the derivation (no other modification required):
public class ErrorServiceBehavior : Attribute, IServiceBehavior
{ ... }
and use the class as Attribute of my service:
[ErrorServiceBehavior()]
public partial class MyService : IMyService
{...}
No other modification required. Check the original sample at How do I create a global exception handler for a WCF Services?.

The destination queue '<QueueName>#<servername>' could not be found

While testing the pub/sub model, I changed the name of the subscriber queue, while the subscription for the old queue still exists in the DB, so there is a dangling subscription in the DB.
So when publisher and subscriber started and I tried to send message from publisher, following exception happened and basically publisher stopped and no longer send any more message
2011-02-09 09:56:21,115 [6] ERROR Publisher.ServerEndpoint [(null)] <(null)> - Problem occurred when starting the endpoint.
System.Configuration.ConfigurationErrorsException: The destination queue 'StoreInputQueue#' could not be found. You may have misconfigured the destination for this kind of message (Message.EventMessage) in the MessageEndpointMappings of the UnicastBusConfig section in your configuration file.It may also be the case that the given queue just hasn't been created yet, or has been deleted. ---> System.Messaging.MessageQueueException: The queue does not exist or you do not have sufficient permissions to perform the operation.
at System.Messaging.MessageQueue.MQCacheableInfo.get_WriteHandle()
at System.Messaging.MessageQueue.StaleSafeSendMessage(MQPROPS properties, IntPtr transaction)
at System.Messaging.MessageQueue.SendInternal(Object obj, MessageQueueTransaction internalTransaction, MessageQueueTransactionType transactionType)
at System.Messaging.MessageQueue.Send(Object obj, MessageQueueTransactionType transactionType)
at NServiceBus.Unicast.Transport.Msmq.MsmqTransport.Send(TransportMessage m, String destination) in d:\BuildAgent-02\work\20b5f701adefe8f8\src\impl\unicast\NServiceBus.Unicast.Msmq\MsmqTransport.cs:line 334
--- End of inner exception stack trace ---
at NServiceBus.Unicast.Transport.Msmq.MsmqTransport.Send(TransportMessage m, String destination) in d:\BuildAgent-02\work\20b5f701adefe8f8\src\impl\unicast\NServiceBus.Unicast.Msmq\MsmqTransport.cs:line 346
at NServiceBus.Unicast.UnicastBus.SendMessage(IEnumerable`1 destinations, String correlationId, MessageIntentEnum messageIntent, IMessage[] messages) in d:\BuildAgent-02\work\20b5f701adefe8f8\src\unicast\NServiceBus.Unicast\UnicastBus.cs:line 593
at NServiceBus.Unicast.UnicastBus.Publish[T](T[] messages) in d:\BuildAgent-02\work\20b5f701adefe8f8\src\unicast\NServiceBus.Unicast\UnicastBus.cs:line 343
at Publisher.ServerEndpoint.Run() in C:\Downloads\ESB\NServiceBus\publisher\publisher\ServerEndpoint.cs:line 26
at NServiceBus.Host.Internal.ConfigManager.<>c_DisplayClass1.b_0() in d:\BuildAgent-02\work\20b5f701adefe8f8\src\host\NServiceBus.Host\Internal\ConfigurationManager.cs:line 56
Is there a timeout period after which it will try to send message to rest of subscribers, I waited quite long...
I don't think it will retry.
Pulling the rug (queue) out from under a running endpoint is not a good thing to do. In production this really should never happen.
Since you're just testing, delete the offending subscription row from the database, and restart the endpoint, and everything should be fine.