We are using NServiceBus 5.2.14 hosted inside .Net 4.6.1 windows services.
We have added NServiceBus auditing to our services via the AuditConfig:
<AuditConfig QueueName="AuditService" />
When we turned on Auditing for the windows service containing our Sagas, we noticed some new subscriptions added to our subscriptions Sql table. Like these:
SubscriberEndpoint MessageType
ServiceThatIsPublishingEventTypeA#MyMachine MyAssembly.MessageTypeA,1.0.147.0
AuditService#MyMachine MyAssembly.MessageTypeA,1.0.147.0
It seems that after enabling auditing on the Saga service, the service that is publishing an event (that the saga is subscribing to), now gets a recursive subscription to itself.
When a message is processed, a copy is successfully pushed to the audit queue.
However, the problem I am having is that I also get a copy of the message pushed into the ServiceThatIsPublishingEventTypeA.Error queue and an exception raised in my log.
The exception is:
System.InvalidOperationException: No handlers could be found for message type: MyAssembly.MessageTypeA
at NServiceBus.LoadHandlersBehavior.Invoke(IncomingContext context, Action next) in C:\BuildAgent\work\3206e2123f54fce4\src\NServiceBus.Core\Unicast\Behaviors\LoadHandlersBehavior.cs:line 29
at NServiceBus.BehaviorChain`1.InvokeNext(T context) in C:\BuildAgent\work\3206e2123f54fce4\src\NServiceBus.Core\Pipeline\BehaviorChain.cs:line 107
at NServiceBus.BehaviorChain`1.<>c__DisplayClass4_0.<InvokeNext>b__0() in C:\BuildAgent\work\3206e2123f54fce4\src\NServiceBus.Core\Pipeline\BehaviorChain.cs:line 94
This makes sense as there are no handlers for MyAssembly.MessageTypeA event within ServiceThatIsPublishingEventTypeA service (it just publishes these events), yet the auditing has added this subscription.
So my question is: Is that recursive subscription correct? what is the correct way to audit the traffic through a saga (and avoid the errors being raised)?
Thanks for your help
I'm guessing the endpoint is auto subscribing to the events it finds in assembly scanning, can you try and disable auto subscribe?
== EDIT
After looking at the code on github i created a pull request with a version that works, the main issue was that the audit component was processing subscription messages it received via the audit channel...
I also cleaned up some minor redundant code
Related
I have a WCF application which consists in some async communications with ecternal services. When we start a new expedient, a new instance is created; it process data and send an xml to a external service and waits for the response. This response requires that a person review the xml and send the response so it usually it is delayed for a long time. For this reason, the workflow go to idle and we use persistence with AppFabric.
The fact is that sometime, when we receive the response, the next exception is raised:
The execution of the InstancePersistenceCommand named {urn:schemas-microsoft-com:System.Activities.Persistence/command}LoadWorkflowByInstanceKey was interrupted by an error.
Normally this error does not occur, it can occur very sporadically. However, we are trying to update the app to include a new functionality (it does not modify the workflow) but when the application is deployed to the server, the instances that were created with the old deployment and were waiting for the response, throw this exception when they receive the response from the external service. However, the instances initiated with the new deployment process the response without problem.
I have been looking for information about this problem but I haven't found much. Anybody can help me?
SOLUTION:
Thanks a lot for your answer, it may be helpful for me in the future. In this case, the problem was that I was updating an assembly version of one of the implicated project (to upload a nuget package) and for a reason that I don’t understand, the instances created with an old version raised this exception when the service with the new version had to manipulate the mentioned instances.
If I change the assembly version to upload the nuget and then set the original version and deploy with this version, everything works ok. Anybody knows what is the reason?
Thanks a lot.
This may be because there is a program running in the background and trying to extend the lock on the instance store every 30 seconds, and it seems that whenever the connection to the SQL service fails, it marks the instance store as invalid.
You can try <workflowIdle timeToUnload="0"/>, if it doesn't work you can look at the methods provided by other links.
Windows workflow 4.0 InstancePersistenceCommand Error
Why do I get exception "The execution of the InstancePersistenceCommand named LoadWorkflowByInstanceKey was interrupted by an error"
WF4 InstancePersistenceCommand interrupted
I have a windows service programmed in vb.NET, using Topshelf as Service Host.
Once in a while the service doesn't start. On the event log, the SCM writes errors 7000 and 7009 (service did not respond in a timely fashion). I know this is a common issue, but I (think) I have tried everything with no result.
The service only relies in WMI, and has no time-consuming operations.
I read this question (Error 1053: the service did not respond to the start or control request in a timely fashion), but none of the answers worked for me.
I Tried:
Set topshelf's start timeout.
Request additional time in the first line of "OnStart" method.
Set a periodic timer wich request additional time to the SCM.
Remove TopShelf and make the service with the Visual Studio Service Template.
Move the initialization code and "OnStart" code to a new thread to return inmediately.
Build in RELEASE mode.
Set GeneratePublisherEvidence = false in the app.config file (per application).
Unchecked "Check for publisher’s certificate revocation" in the internet settings (per machine).
Deleted all Alternate Streams (in case some dll was marked as web and blocked).
Removed any "Debug code"
Increased Window's general service timeout to 120000ms.
Also:
The service doesn't try to communicate with the user's desktop in any way.
The UAC is disabled.
The Service Runs on LOCAL SYSTEM ACCOUNT.
I believe that the code of the service itself is not the problem because:
It has been on production for over two years.
Usually the service starts fine.
There is no exception logged in the Event Log.
The "On Error" options for the service dosn't get called (since the service doesn't actually fails, just doesn't respond to the SCM)
I've commented out almost everything on it, pursuing this error! ;-)
Any help is welcome since i'm completely out of ideas, and i've been strugling with this for over 15 days...
For me the 7009 error was produced by my NET core app because I was using this construct:
var builder = new ConfigurationBuilder()
.SetBasePath(Directory.GetCurrentDirectory())
.AddJsonFile("appsettings.json");
and appsettings.json file obviously couldn't be found in C:\WINDOWS\system32.. anyway, changing it to Path.Combine(AppContext.BaseDirectory, "appsettings.json") solved the issue.
More general help - for Topshelf you can add custom exception handling where I finally found some meaningfull error info, unlike event viewer:
HostFactory.Run(x => {
...
x.OnException(e =>
{
using (var fs = new StreamWriter(#"C:\log.txt"))
{
fs.WriteLine(e.ToString());
}
});
});
I've hit the 7000 and 7009 issue, which fails straight away (even though the error message says A timeout was reached (30000 milliseconds)) because of misconfiguration between TopShelf and what the service gets installed as.
The bottom line - what you pass in HostConfigurator.SetServiceName(name) needs to match exactly the SERVICE_NAME of the Windows service which gets installed.
If they don't match it'll fail straight away and you get the two event log messages.
I had this start happening to a service after Windows Creator's Edition update installed. Basically it made the whole computer slower, which is what I think triggered the problem. Even one of the Windows services had a timeout issue.
What I learned online is that the constructor for the service needs to be fast, but OnStart has more leeway with the SCM. My service had a C# wrapper and it included an InitializeComponent() that was called in the constructor. I moved that call to OnStart and the problem went away.
We are using Microsoft's Release Management to promote code changes through the environments. The installer creates the MSMQ queues if they are not already installed. The problem is that creating the queues requires elevated privileges to correctly create the queues. The release management agent account is an administrator on the local machine. The error message we receive is:
The queue does not exist or you do not have sufficient permissions to perform the operation.
Is there a way to bypass the creation of the queues on the install phase and let the creation happen on service start up (which runs by default for us as Local System)?
We are open to any ideas to help get this service correctly installed and started.
[UPDATE]
Here is the message from the logs:
2014-09-03 08:47:55.522 -04:00 [Fatal] Exception when starting endpoint.
System.InvalidOperationException: There is a problem with the input queue: FormatName:DIRECT=OS:DNABUS1\private$\Purchasing.OrderEntry.Bridging. See the enclosed exception for details. ---> System.Messaging.MessageQueueException: The queue does not exist or you do not have sufficient permissions to perform the operation.
at System.Messaging.MessageQueue.MQCacheableInfo.get_Transactional()
at System.Messaging.MessageQueue.get_Transactional()
at NServiceBus.Transports.Msmq.MsmqDequeueStrategy.QueueIsTransactional() in c:\BuildAgent\work\31f8c64a6e8a2d7c\src\NServiceBus.Core\Transports\Msmq\MsmqDequeueStrategy.cs:line 144
--- End of inner exception stack trace ---
at NServiceBus.Transports.Msmq.MsmqDequeueStrategy.QueueIsTransactional() in c:\BuildAgent\work\31f8c64a6e8a2d7c\src\NServiceBus.Core\Transports\Msmq\MsmqDequeueStrategy.cs:line 153
at NServiceBus.Transports.Msmq.MsmqDequeueStrategy.Init(Address address, TransactionSettings settings, Func2 tryProcessMessage, Action2 endProcessMessage) in c:\BuildAgent\work\31f8c64a6e8a2d7c\src\NServiceBus.Core\Transports\Msmq\MsmqDequeueStrategy.cs:line 67
at NServiceBus.Unicast.Transport.TransportReceiver.StartReceiver() in c:\BuildAgent\work\31f8c64a6e8a2d7c\src\NServiceBus.Core\Unicast\Transport\TransportReceiver.cs:line 245
at NServiceBus.Unicast.Transport.TransportReceiver.Start(Address address) in c:\BuildAgent\work\31f8c64a6e8a2d7c\src\NServiceBus.Core\Unicast\Transport\TransportReceiver.cs:line 211
at NServiceBus.Unicast.UnicastBus.Start(Action startupAction) in c:\BuildAgent\work\31f8c64a6e8a2d7c\src\NServiceBus.Core\Unicast\UnicastBus.cs:line 806
at NServiceBus.Hosting.GenericHost.Start() in c:\BuildAgent\work\31f8c64a6e8a2d7c\src\NServiceBus.Core\Hosting\GenericHost.cs:line 78
[UPDATE 2]
After disabling UAC on the target server and running "NServiceBus.Host.exe /install", the install and queue creation worked properly. Does anyone know of a way we could do the install without disabling UAC on the target server?
What we ended up doing is creating the MSMQ queues as a part of the release management template. That way, when by the time the service installer runs, the queues are already created, which allows us to keep UAC enabled.
I can't seem to get the Castle Windsor Integration working for Mass Transit over RabbitMQ. Everything was working fine until I introduced Windsor into the picture. I referenced Castle.Windsor 3.2 and MassTransit.WindsorIntegration 2.9 and configured the container for use within my application. I'm registering the MassTransit Consumers via:
Container.Register(..., Types.FromThisAssembly().BasedOn<IConsumer>());
When I debug and inspect the container after this line is ran, I can see that it successfully registered all of the consumers along with all of my other components. I then have the following code to initialize and register the service bus:
var serviceBus = ServiceBusFactory.New(sbc =>
{
sbc.UseRabbitMq();
sbc.ReceiveFrom(Config.ServiceBusEndpoint);
sbc.Subscribe(sc => sc.LoadFrom(Container));
});
Container.Register(Component.For<IServiceBus>().Instance(serviceBus));
I am using the LoadFrom(IWindsorContainer container) extension method provided by MassTransit.WindsorIntegration.
All of the examples I've found so far stop here and indicate that this is all you should have to do. Unfortunately for me my Consumers are never being called and messages are just piling up in the queue (eventually timing out and going to error queue). The fact that messages are showing up in the Consumer queue at all (+ I see a single consumer bound to the queue via the RabbitMQ Admin Tool) indicates to me that the consumers are probably being subscribed properly - so I'm not sure where the problem lies.
I added NLog logging for Windsor and MassTransit but no errors are showing up in the logs. I'm not sure how I should proceed troubleshooting at this point. Any ideas?
Also, this application is currently just a console application using Topshelf for development. Ultimately it will be installed as a Windows Service. Not sure if that is relevant or not but I thought I'd mention it just in case.
UPDATE
As a test I created a very simple Consumer with a parameter-less constructor for processing a very simple test message. This Consumer is successfully called! The "real" Consumers however have dependencies that need to be injected into them via the constructor. I was hoping the Container would resolve these but apparently it's having some sort of trouble. Weird that nothing is showing up in the logs about it. Stay tuned...
Okay I figured it out. Somewhere along the way when I was adding/removing NuGet packages I somehow managed to delete a reference to a DLL (ServiceStack.Text.dll) that one of my components needed (RedisClientsManager).
I started the debugger, let all my components get registered then popped open the Immediate Window and attempted to resolve each component one by one (by calling container.Resolve<RegisteredType>()) until I found the one that threw the exception when I attempted to resolve it.
The Exception message from Windsor at that point told me exactly what the problem was. I'm a little lost as to why this wasn't being logged or why the Exception was not raised when the container itself attempted to resolve it. Anyhow, moral of the story is make sure your components resolve.
I have several messages (pub/sub) in the error queue. When I use the ReturnToSourceQueue tool it says that all message are handled successful but when I look in the error queues the messages are still there and the event handlers are not triggered. I'm using NServiceBus 4.3.3 with sql server. I have disabled the timeoutmanager and not using second level retries. Someone has an idea ?
Found it. It was the same reason as the problem I had with second level retry. I had to disable the saga feature and then I could remove the disabling of the time out manager. After that everything works :-)