I am using NServiceBus 5. My messages are sending / receiving correctly, but I'm having trouble with Service Pulse. I have configured the auditing using the default endpoint names.
When I navigate to Service Pulse (http://localhost:9090/) I get the following error.
Can't connect to ServiceControl (http://localhost:33333/api/)
Looking at my services I see that Particular ServiceControl is not started. When I attempt to start it, it starts and immediately stops.
I have checked the logs at:
%LOCALAPPDATA%\Particular\ServiceControl\logs
and
%WINDIR%\System32\config\systemprofile\AppData\Local\Particular\ServiceControl\logs
But apart from the errors about the missing queues from yesterday (see below) - nothing. When I attempt to restart the service now I get no errors.
Anyone know what I should do to get Service Pulse working correctly?
I deleted all my private queues yesterday thinking that they would be recreated automatically. Now I realise only the endpoint ones are recreated, I have recreated some manually.
Right now along with my endpoint queues I have:
audit
auditqueue
error
error.log
particular.servicecontrol
particular.servicecontrol.errors
particular.servicecontrol.retries
particular.servicecontrol.timeouts
particular.servicecontrol.timeoutsdispatcher
--- EDIT ---
Ended up just uninstalling and reinstalling - fixed the problem.
ServiceControl --uninstall
ServiceControl --install
Try and run ServiceControl --install in an admin Console and it will create the queues (C:\Program Files (x86)\Particular Software\ServiceControl> .\ServiceControl.exe --install)
If not you need to add these queues manually or reinstall ServiceControl:
particular.servicecontrol
particular.servicecontrol.errors
particular.servicecontrol.timeouts
particular.servicecontrol.timeoutsdispatcher
Related
Erlang crashes on rabbitMQ service start, due to bad unable to create erlang.cookie path Image Attached (Log File). I need to figure out why it's adding an extra c: to the beginning of the path, and where that's defined. Any suggestions. I've uninstalled and deleted everything relating to rabbitMQ multiple times. include the registry items and environment variables.
The following piece of code has been working for some time and it has suddenly stopped returning:
await availableChangedPublishEndpoint
.Publish<IAvailableStockChanged>(
AvailableStockCounter.ConvertSkuQtyToAvailableStockChangedEvent(
newAvailable,
absMessage.Warehouse)
);
There is nothing clever in ConvertSkuQtyToAvailableStockChangedEvent - it just maps one simple class to another.
We added logs before and after this code and it's definitely just stopping at this point. Other systems are publishing fine, other messages are being sent from this application (for e.g. logs are actually sent via RabbitMQ). We have redeployed and we have upgraded to latest MassTransit version. We are seeing that the messages are being published - possibly multiple times, but this Publish method never returns.
We had a broken RabbitMQ node and a clean service restart on one node fixed it. I appreciate there might be other reasons for this behaviour, but this was our problem.
systemctl restart rabbitmq-server
Looking further into RabbitMQ we saw that some of the empty queues that were connected to this exchange were not synchronized (see below) and when we tried to synchronize them that wouldn't work.
We also couldn't delete some of these unsynchronized queues.
We believe an unexpected shutdown of one of the nodes had caused this problem - but it left most queues / exchanges completely OK.
Our web app process is restarting regularly and we are unable to determine the reason.
When looking into Application Events (using the 'Diagnostics and solve problems' blade in the Azure Portal), there exists a bunch of the following Info logs by 'IIS AspNetCore Module'
Event ID 1005:
Failed to gracefully shutdown process '14040'.
Event ID 1001:
Application 'MACHINE/WEBROOT/APPHOST/myapplication__xxxx' started process '31628' successfully and is listening on port '17663'.
There is nothing fishy with general resource usage and nothing in our application logs.
What is the best way to troubleshoot the reason behind these process restarts?
EDIT 1:
After fiddling around with web logging in the Web App's Diagnostic Logs, I now get an error logged from W3SVC-WP after each restart, but the message is nonsense:
1<br/>5<br/>50000780
EDIT 2:
Event Id 2284 refers to this:
FailedRequestTracing module failed to write buffered events to log
file for the request that matched failure definition. No logs will be
generated until this condition is corrected. The problem happened at
least %1 times in the last %2 minutes. The data is the error.
I'm not sure if this could be related to our Diagnostic Logs configuration, but seems unlikely.
EDIT 3:
As per Brando Zhang's suggestion, I've used the Web App Crash Diagnoser extension and tried monitoring 2nd Chance Unhandled Exceptions on both my application process AND on w3wp, but nothing is dumped.
From how I understand it, 1st Chance Exceptions will not crash the process, so no need to monitor these.
Very likely application is crashing due to fatal exception and causing the restarts.
On Azure App Service platform.You can use the Diagnostics as a
Service (DaaS) to troubleshoot this
It can also do an analysis and tell you the root cause most of the time.More step by step infofrmation can be found on this msdn blog .Also refer tips for using crash diagnoser
The services terminate due to MSMQ not being started at the time they're trying to start up, so the obvious solution would be to set the startup type to automatic (delayed). Rather than going with the obvious solution I'm wondering if there's something I'm doing wrong?
I'm using NServiceBus V5 on Win2012 and started to get these messages on all my installed services after I upgraded to V5 from V4. They're installed to start automatically when the server boots up. This hasn't happened before, and the services start normally if I manually start them.
I don't think you're doing anything wrong--I think it's what you suspect: your service is starting up before the Message Queuing Service is ready.
Assuming the NServicebus is running in NServiceBus.Host.exe, in order to get the service to start after MSMQ, there's a switch that needs to be used when installing the service (I used one line, broke it into multiple lines for readability):
.\NServiceBus.Host.exe /install
/servicename:"service"
/displayname:"display"
/dependsOn:"MSMQ"
I had tried "Message Queing" since that's what was in the service control manager, but I guess it doesn't take the display name. I found that MSMQ was the service name by right clicking on the service and viewing the service name field at the top.
You can set service dependencies without having to reinstall.
use the following command: sc config YOURSERVICENAME depend=MSMQ
I can't seem to get the Castle Windsor Integration working for Mass Transit over RabbitMQ. Everything was working fine until I introduced Windsor into the picture. I referenced Castle.Windsor 3.2 and MassTransit.WindsorIntegration 2.9 and configured the container for use within my application. I'm registering the MassTransit Consumers via:
Container.Register(..., Types.FromThisAssembly().BasedOn<IConsumer>());
When I debug and inspect the container after this line is ran, I can see that it successfully registered all of the consumers along with all of my other components. I then have the following code to initialize and register the service bus:
var serviceBus = ServiceBusFactory.New(sbc =>
{
sbc.UseRabbitMq();
sbc.ReceiveFrom(Config.ServiceBusEndpoint);
sbc.Subscribe(sc => sc.LoadFrom(Container));
});
Container.Register(Component.For<IServiceBus>().Instance(serviceBus));
I am using the LoadFrom(IWindsorContainer container) extension method provided by MassTransit.WindsorIntegration.
All of the examples I've found so far stop here and indicate that this is all you should have to do. Unfortunately for me my Consumers are never being called and messages are just piling up in the queue (eventually timing out and going to error queue). The fact that messages are showing up in the Consumer queue at all (+ I see a single consumer bound to the queue via the RabbitMQ Admin Tool) indicates to me that the consumers are probably being subscribed properly - so I'm not sure where the problem lies.
I added NLog logging for Windsor and MassTransit but no errors are showing up in the logs. I'm not sure how I should proceed troubleshooting at this point. Any ideas?
Also, this application is currently just a console application using Topshelf for development. Ultimately it will be installed as a Windows Service. Not sure if that is relevant or not but I thought I'd mention it just in case.
UPDATE
As a test I created a very simple Consumer with a parameter-less constructor for processing a very simple test message. This Consumer is successfully called! The "real" Consumers however have dependencies that need to be injected into them via the constructor. I was hoping the Container would resolve these but apparently it's having some sort of trouble. Weird that nothing is showing up in the logs about it. Stay tuned...
Okay I figured it out. Somewhere along the way when I was adding/removing NuGet packages I somehow managed to delete a reference to a DLL (ServiceStack.Text.dll) that one of my components needed (RedisClientsManager).
I started the debugger, let all my components get registered then popped open the Immediate Window and attempted to resolve each component one by one (by calling container.Resolve<RegisteredType>()) until I found the one that threw the exception when I attempted to resolve it.
The Exception message from Windsor at that point told me exactly what the problem was. I'm a little lost as to why this wasn't being logged or why the Exception was not raised when the container itself attempted to resolve it. Anyhow, moral of the story is make sure your components resolve.