We're having an issue where our NServiceBus ServiceControl instance is not purging messages from it's RavenDB as per the supposed expiration policy.
We have set the following key in the ServiceConfig.exe.Config file, which should expire messages after 1 hour, but I can still see messages from yesterday using ServiceInsight and the RavenDB has increased considerably in size.
<add key="ServiceControl/HoursToKeepMessagesBeforeExpiring" value="1" />
We need to get the automatic purging of messages working before our system goes into production, so any assistance is appreciated.
As discussed offline with #starskythehutch the issue is that 1 as the HoursToKeepMessagesBeforeExpiring is not a supported value, the minimum value is 24. Setting one will case ServiceControl to revert back to its default value that is 720 causing no purge for a long time.
We are currently improving the way ServiceControl enforces the above behavior in order to allow the use to better understand what is going on.
Related
I installed archiver extension (https://github.com/ckan/ckanext-archiver/projects) in my CKAN instance and I can't make it work through my entire catalog. The 'queuing' proccess (paster --plugin=ckanext-archiver archiver update --queue=bulk-c ) stops with the next message at some point of the execution (usually when it has proccessed less than 500 datasets -347 last time-):
kombu.exceptions.InconsistencyError:`
Cannot route message for exchange 'bulk': Table empty or key no longer exists.
Probably the key ('_kombu.binding.bulk') has been removed from the Redis database.
I can't find anything related on Internet and I'm a little bit lost here. I'm running:
ckan 2.5.2
redis 2.10.1
celery 3.1.25
Thank you in advance.
similar issue here
https://github.com/celery/kombu/issues/1063
This can happen if you set Redis to LRU mode or something similar.
Please configure Redis correctly and increase the memory of your Redis
instance.
We recently had an endpoint with logic to defer message processing by simply calling Bus.HandleCurrentMessageLater() malfunction. Within ~48 hours ServiceControl's RavenDB file grew to over 100 gigs (It just kept calling defer on the same few messages).
Here are our SC retention settings:
Audit Messages will be removed after this period: 14 Days.
Error Messages that have been archived or resolved will be removed after this period: 10 Hours.
What is the difference between a single endpoint repeatedly calling Bus.HandleCurrentMessageLater() and a large scale system handling 1,000s of messages per second? Is there some configuration for auditing that isn't available when simply calling HandleCurrentMessageLater or is the expectation that systems with such throughput can handle an SC database growing over 50 gigs in less than a day
While looking for a better way to defer messages, I noticed there is quite a lot of discussion around deprecating the simple mechanisms present in NSB versions 4 and 5 with "throttling behaviors." Are custom behaviors still the "approved" method for throttling?
Would it be possible to create a CustomCheck that can enable/disable a given handler if certain criteria have not been met?
I'm getting over 10000 updates in 60 seconds in my Redis server and this triggers the background save which consumes resources.
I want to track the changed keys so that I can debug my app (which method causing this much change).
Is there a way to get updated keys?
While MONITOR is perfectly valid, it does include everything that gets sent to Redis. That means filtering read requests, pings, ...
Instead I recommend that you check the keyspace notifications documentation and configure your database the AK flags. By subscribing to the __keyspace:* pattern you'll be notified about every change to keys.
As I learned, it's only possible by using MONITOR command and figure out from output.
I am using activemq 5.5.0 version in my project.
I am facing one problem due to some reason my inflight count keep on increasing ,
Rate of increment is not high.
After some days it will become equal to prefetch size then my queue stops responding.
Could anyone can help me how to make inflight count 0 without deleting the queue ?
You can make the in-flight count go to 0 by purging the queue from the web console (there's a Purge link on each queue on the Queues page); that will delete all messages in the queue without deleting the queue itself.
That will satisfy the question as you've asked it, though I suspect it's not really what you want and what you meant was "without deleting any messages from the queue". But if I'm wrong and this is what you wanted, then great; if not, answer my questions in the comment I added and we'll go from there.
I am concerned with my NServiceBus solution.
I have a "MessageHub" that publishes some very important messages. But sometimes it loses track of its subscriptions and just discards the message because it thinks no one is listening.
I have tried turning on "NServiceBus.Integration" to store the subscriptions. But despite that, I still have issues with bad start up order where it thinks nothing is listening.
Is there a way to debug this process? Try to figure out why it is getting confused?
I don't even know a way to look at what subscriptions it "thinks" it has...
I went with NServiceBus because it is not supposed to lose data ever. Now I am losing large chucks. I know it is a config issue, but it is causing much grief.
What is probably happening in your case is that you are using MSMQ for subscription storage. Even though it's possible for subscriptions to endure for a while, using MSMQ to store things long term is always going to be volatile.
For durable subscriptions storage (which survive "forever") you should be using SQL server as your subscription storage.
Note: You can always view your current subscriptions whether you are using sql or msmq to store them. In SQL just look in the subscriptions table and for msmq look in the publisher's subscription queue.
UPDATE
Since version 3 I have been using RavenDb which is the default.
In my experiance, to get the subscriptions assigned correctly, one should first start the EventHandler projects and then when they are all idle, start the CommandHandlers (Publishers).
You can see what messages are being Subscribed to using Service Bus MQ Manager, it has a dialog listing all "messages" and their subscribers/publishers. A side project of mine, its free and open sourced.
http://blog.halan.se/page/Service-Bus-MQ-Manager.aspx