The following piece of code has been working for some time and it has suddenly stopped returning:
await availableChangedPublishEndpoint
.Publish<IAvailableStockChanged>(
AvailableStockCounter.ConvertSkuQtyToAvailableStockChangedEvent(
newAvailable,
absMessage.Warehouse)
);
There is nothing clever in ConvertSkuQtyToAvailableStockChangedEvent - it just maps one simple class to another.
We added logs before and after this code and it's definitely just stopping at this point. Other systems are publishing fine, other messages are being sent from this application (for e.g. logs are actually sent via RabbitMQ). We have redeployed and we have upgraded to latest MassTransit version. We are seeing that the messages are being published - possibly multiple times, but this Publish method never returns.
We had a broken RabbitMQ node and a clean service restart on one node fixed it. I appreciate there might be other reasons for this behaviour, but this was our problem.
systemctl restart rabbitmq-server
Looking further into RabbitMQ we saw that some of the empty queues that were connected to this exchange were not synchronized (see below) and when we tried to synchronize them that wouldn't work.
We also couldn't delete some of these unsynchronized queues.
We believe an unexpected shutdown of one of the nodes had caused this problem - but it left most queues / exchanges completely OK.
Related
I am using NServiceBus 5. My messages are sending / receiving correctly, but I'm having trouble with Service Pulse. I have configured the auditing using the default endpoint names.
When I navigate to Service Pulse (http://localhost:9090/) I get the following error.
Can't connect to ServiceControl (http://localhost:33333/api/)
Looking at my services I see that Particular ServiceControl is not started. When I attempt to start it, it starts and immediately stops.
I have checked the logs at:
%LOCALAPPDATA%\Particular\ServiceControl\logs
and
%WINDIR%\System32\config\systemprofile\AppData\Local\Particular\ServiceControl\logs
But apart from the errors about the missing queues from yesterday (see below) - nothing. When I attempt to restart the service now I get no errors.
Anyone know what I should do to get Service Pulse working correctly?
I deleted all my private queues yesterday thinking that they would be recreated automatically. Now I realise only the endpoint ones are recreated, I have recreated some manually.
Right now along with my endpoint queues I have:
audit
auditqueue
error
error.log
particular.servicecontrol
particular.servicecontrol.errors
particular.servicecontrol.retries
particular.servicecontrol.timeouts
particular.servicecontrol.timeoutsdispatcher
--- EDIT ---
Ended up just uninstalling and reinstalling - fixed the problem.
ServiceControl --uninstall
ServiceControl --install
Try and run ServiceControl --install in an admin Console and it will create the queues (C:\Program Files (x86)\Particular Software\ServiceControl> .\ServiceControl.exe --install)
If not you need to add these queues manually or reinstall ServiceControl:
particular.servicecontrol
particular.servicecontrol.errors
particular.servicecontrol.timeouts
particular.servicecontrol.timeoutsdispatcher
The services terminate due to MSMQ not being started at the time they're trying to start up, so the obvious solution would be to set the startup type to automatic (delayed). Rather than going with the obvious solution I'm wondering if there's something I'm doing wrong?
I'm using NServiceBus V5 on Win2012 and started to get these messages on all my installed services after I upgraded to V5 from V4. They're installed to start automatically when the server boots up. This hasn't happened before, and the services start normally if I manually start them.
I don't think you're doing anything wrong--I think it's what you suspect: your service is starting up before the Message Queuing Service is ready.
Assuming the NServicebus is running in NServiceBus.Host.exe, in order to get the service to start after MSMQ, there's a switch that needs to be used when installing the service (I used one line, broke it into multiple lines for readability):
.\NServiceBus.Host.exe /install
/servicename:"service"
/displayname:"display"
/dependsOn:"MSMQ"
I had tried "Message Queing" since that's what was in the service control manager, but I guess it doesn't take the display name. I found that MSMQ was the service name by right clicking on the service and viewing the service name field at the top.
You can set service dependencies without having to reinstall.
use the following command: sc config YOURSERVICENAME depend=MSMQ
Redis server which was working fine got stopped suddenly and the error is:
BeginForkOperation: system error caught. error code=0x00000000, message=Forked
Process did not respond in a timely manner.
Not able to figure out why it is happening, and also when I am restarting my machine then
if I start the redis-server it's working fine.
Please help me in this regard.
You should try updating your Redis version, the guys from MSOpenTech fixed a lot of bugs in the last months and this one looks related, at least the error message is identical: https://github.com/MSOpenTech/redis/issues/144
So today we run into a disturbing solr issue.
After a restart of the whole cluster one of the shard stop being able to index/store documents.
We had no hint about the issue until we started indexing (querying the server looks fine).
The error is:
2014-05-19 18:36:20,707 ERROR o.a.s.u.p.DistributedUpdateProcessor [qtp406017988-19] ClusterState says we are the leader, but locally we don't think so
2014-05-19 18:36:20,709 ERROR o.a.s.c.SolrException [qtp406017988-19] org.apache.solr.common.SolrException: ClusterState says we are the leader (http://x.x.x.x:7070/solr/shard3_replica1), but locally we don't think so. Request came from null
at org.apache.solr.update.processor.DistributedUpdateProcessor.doDefensiveChecks(DistributedUpdateProcessor.java:503)
at org.apache.solr.update.processor.DistributedUpdateProcessor.setupRequest(DistributedUpdateProcessor.java:267)
at org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:550)
at org.apache.solr.handler.loader.JsonLoader$SingleThreadedJsonLoader.processUpdate(JsonLoader.java:126)
at org.apache.solr.handler.loader.JsonLoader$SingleThreadedJsonLoader.load(JsonLoader.java:101)
at org.apache.solr.handler.loader.JsonLoader.load(JsonLoader.java:65)
at org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:92)
at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74)
at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1916)
We run Solr 4.7 in Cluster mode (5 shards) on jetty.
Each shard run on a different host with one zookeeper server.
I checked the zookeeper log and I cannot see anything there.
The only difference is that in the /overseer_election/election folder I see this specific server repeated 3 times, while the other server are only mentioned twice.
45654861x41276x432-x.x.x.x:7070_solr-n_00000003xx
74030267x31685x368-x.x.x.x:7070_solr-n_00000003xx
74030267x31685x369-x.x.x.x:7070_solr-n_00000003xx
Not even sure if this is relevant. (Can it be?)
Any clue what other check can we do?
We've experienced this error under 2 conditions.
Condition 1
On a single zookeeper host there was an orphaned Zookeeper ephemeral node in
/overseer_elect/election. The session this ephemeral node was associated with no longer existed.
The orphaned ephemeral node cannot be deleted.
Caused by: https://issues.apache.org/jira/browse/ZOOKEEPER-2355
This condition will also be accompanied by a /overseer/queue directory that is clogged-up with queue items that are forever waiting to be processed.
To resolve the issue you must restart the Zookeeper node in question with the orphaned ephemeral node.
If after the restart you see Still seeing conflicting information about the leader of shard shard1 for collection <name> after 30 seconds
You will need to restart the Solr hosts as well to resolve the problem.
Condition 2
Cause: a mis-configured systemd service unit.
Make sure you have Type=forking and have PIDFile configured correctly if you are using systemd.
systemd was not tracking the PID correctly, it thought the service was dead, but it wasn't, and at some point 2 services were started. Because the 2nd service will not be able to start (as they both can't listen on the same port) it seems to just sit there in a failed state hanging, or fails to start the process but just messes up the other solr processes somehow by possibly overwriting temporary clusterstate files locally.
Solr logs reported the same error the OP posted.
Interestingly enough, another symptom was that zookeeper listed no leader for our collection in /collections/<name>/leaders/shard1/leader normally this zk node contains contents such as:
{"core":"collection-name_shard1_replica1",
"core_node_name":"core_node7",
"base_url":"http://10.10.10.21:8983/solr",
"node_name":"10.10.10.21:8983_solr"}
But the node is completely missing on the cluster with duplicate solr instances attempting to start.
This error also appeared in the Solr Logs:
HttpSolrCall null:org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /roles.json
To correct the issue, killall instances of solr (or java if you know it's safe), and restart the solr service.
We figured out!
The issue was that jetty didn't really stop so we had 2 running processes, for whatever reason this was fine for reading but not for writing.
Killing the older java process solved the issue.
I have several messages (pub/sub) in the error queue. When I use the ReturnToSourceQueue tool it says that all message are handled successful but when I look in the error queues the messages are still there and the event handlers are not triggered. I'm using NServiceBus 4.3.3 with sql server. I have disabled the timeoutmanager and not using second level retries. Someone has an idea ?
Found it. It was the same reason as the problem I had with second level retry. I had to disable the saga feature and then I could remove the disabling of the time out manager. After that everything works :-)