I would like to set the maxPageSize to a larger number from its default 200.
This is how I set in the activemq.xml file:
<destinationPolicy>
<policyMap>
<policyEntries>
---
<policyEntry queue=">" producerFlowControl="true" memoryLimit="1mb" maxPageSize="SOME_LARGE_NUMBER">
</policyEntry>
</policyEntries>
</policyMap>
</destinationPolicy>
This change helps me get the number of messages in a queue using QueueBrowser.getEnumeration() as it returned 200 even if the number of messages in the queue were greater than 200.
Please see: http://docs.oracle.com/javaee/1.4/api/javax/jms/QueueBrowser.html for QueueBrowser.getEnumeration().
What is the side effect of changing the maxPageSize from 200 to say 1000?
Does it affect the broker's performance anyway?
I am not seeing any documentation for this property other than "maximum number of persistent messages to page from store at a time" on this page:
http://activemq.apache.org/per-destination-policies.html
Thanks for your time!
Max page size simply indicates the number of messages that will be loaded into memory, so the impact is.. that it will consume more memory.
Reading between the lines though, the reason that you are doing this is an anti-pattern. Queue browsing as part of an application is really a misuse of messaging - a message queue works best when treated as a queue. First in, first out. Not as an array that you scan to see whether a message has arrived.
You are much better off consuming each of the messages, and either:
sorting them onto a bunch of other queues depending on their payload and then processing that second level of queues differently, or
storing the payloads into a database and selecting based on the content.
Related
ActiveMQ 5.15.4
Context: the producer I'm working with publishes to many different queues.
Things I've tried:
1.
<policyEntry queue=">" producerFlowControl="true" memoryLimit="10 mb">
</policyEntry>
This correctly limits the queue size, but it throttles the producer, as described here. This leads to other, non-problematic queues being af
2.
<policyEntry queue=">" producerFlowControl="false" memoryLimit="10 mb">
</policyEntry>
Doesn't seem to do limit queue size.
3.
I've tried to use messageEvictionStrategy and pendingMessageLimitStrategy but they don't seem to work for queues, only topics.
Am I missing some other possible strategy?
You need to use a Time To Live value on the messages to control how long they stay in the Queue, otherwise the broker will either block the producer using flow control or page the messages to disk if you disable it. Queues are not generally meant for fixed size messaging as the assumption on the Queue is that the contents are important and should not be discarded unless the sender allows for it via a TTL.
We have been using AMQ in production for quite a while some time already, and we are noticing a strange behavior on one of our queues.
The situation is as follows:
we do clickstream traffic so when we have identified a user, all his events are "grouped" by JMSXGroupID property (which is an UUID, in our case, we can have millions of these per hour) so we have some order in consuming the events for the same user in case they do burst
we use KahaDB with kinda the following config:
<mKahaDB directory="${activemq.data}/mkahadb">
<filteredPersistenceAdapters>
<filteredKahaDB perDestination="true">
<persistenceAdapter>
<kahaDB checkForCorruptJournalFiles="true" journalDiskSyncStrategy="PERIODIC" journalDiskSyncInterval="5000" preallocationStrategy="zeros" concurrentStoreAndDispatchQueues="false" />
</persistenceAdapter>
</filteredKahaDB>
</filteredPersistenceAdapters>
the broker is in a rather beefy EC2 instance, but it doesn't seem to hit any limits, neither file limits, nor IOPS, nor CPU limits
destination policy for this destination uses, very similar to a lot other destinations that use the same grouping for JMSXGroupID:
<policyEntry queue="suchDestination>" producerFlowControl="false" memoryLimit="256mb" maxPageSize="5000" maxBrowsePageSize="2000">
<messageGroupMapFactory>
<simpleMessageGroupMapFactory/>
</messageGroupMapFactory>
<deadLetterStrategy>
<individualDeadLetterStrategy queuePrefix="DLQ." useQueueForQueueMessages="true" />
</deadLetterStrategy>
consumers consume messages fairly slowly compared to other destinations (about 50-100ms per message compared to
other consumers for other destinations- about 10-30ms per message)
however, it seems we end up in a situation, where the consumers are not consuming with the speed we expect them to be doing, and seem to wait for something, while there is a huge load of messages on the remote broker for that destination. The consumers seem to also not be neither CPU, nor IO bound, nor network traffic bound.
a symptom is that if we split that queue to two queues and we attach the same number of consumers in the same number of nodes to consume it, things are somehow becoming better. Also, if there is a huge workload for that queue, if we just rename it to suchQueue2 on producers, and assign some consumers on it, these consumers are much faster (for a while) than the consumers on the "old" suchQueue.
the queue doesn't have "non-grouped messages", all messages on it have the JMSXGroupID property and are of the same type.
increasing the number of consumers or lowering it for that queue seems to have little effect
rebooting the consumer apps seems to have little effect once the queue becomes "slow to consume"
Has anybody experienced this:
in short:
Broker is waiting a considerable time for the consumers who seem to be free and not busy all the time.
Looking for some help around systemUsage and destinationPolicy configuration as I'm having some difficulty fully understanding the relationship between systemUsage, destinationPolicy, and flow control.
All our messages are persistent! producerFlowControl is on.
So we give ActiveMQ say a maximum of 512MB heap space.
Our systemUsage is set as below:
<systemUsage>
<systemUsage>
<memoryUsage>
<memoryUsage limit="200 mb"/>
</memoryUsage>
<storeUsage>
<storeUsage limit="10 gb"/>
</storeUsage>
<tempUsage>
<tempUsage limit="1000 mb"/>
</tempUsage>
</systemUsage>
</systemUsage>
Our destination policy as below:
<destinationPolicy>
<policyMap>
<policyEntries>
<policyEntry topic=">" producerFlowControl="true" memoryLimit="1mb">
<pendingSubscriberPolicy>
</pendingSubscriberPolicy>
</policyEntry>
<policyEntry queue=">" producerFlowControl="true" memoryLimit="1mb">
</policyEntry>
</policyEntries>
</policyMap>
</destinationPolicy>
Can anyone verify the if the following is correct:
This means that for each individual queue/topic the memory limit is 1MB. What exactly happens when this 1MB is hit, does the queue block for producers or does it page to disc?
The total allowed memory for all queues and topics is 200MB. Meaning that we could have 200 channels operating at their full capacity of 1MB. We currently have 16 queues and topics in total so obviously that is never reached.
Are we better to remove the individual policy entry on the memory limit and share the memory between the various channels?
If we do this at what point would they block?
Any help very much appreciated! Can paypal you some beer money!
You're touching on a number of points here, which I'll answer out of order.
memoryUsage corresponds to the amount of memory that's assigned to the in-memory store. storeUsage corresponds to how much space should be given to the KahaDB store. You either use one or the other, depending on how you want your broker to persist messages or not. tempUsage is a special case for file cursors (http://activemq.apache.org/message-cursors.html) - a mechanism to overflow the memory from the in-memory store to disk if the memory limit is exceeded (you have to configure this behaviour at the destination level if you want it).
policyEntry#memoryLimit is a sub-limit for individual destinations.
What happens when memory limits are exceeded depends on whether producer flow control (PFC) is turned on. It's on by default for queues, off for topics and asynchronous sends to queues; all this can be configured in the policyEntry (http://activemq.apache.org/per-destination-policies.html).
If you hit a "memory limit" when PFC is on, your clients will block until someone frees up space by consuming messages from the store. If it's off, the send will throw an exception (better the client fall over than the broker). "Memory limit" means either the one defined by memoryUsage across all queues, or the queue-specific limit (it's possible to hit the former before the latter).
Whether or not you want a destination-specific limit depends on your use case. I'd suggest to ignore it unless you're trying to achieve a specific outcome.
Today, I saw many errors on my ActiveMQ 5.3.2 console:
INFO | Usage Manager Memory Limit
reached. Stopping producer (ID:...) to
prevent flooding topic://mytopic. See
http://activemq.apache.org/producer-flow-control.html
for more info (blocking for: 3422ms)
I did a little bit of poking around, and determined that the subscriber had gone out to lunch:
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp6 0 130320 10.208.87.178:61613 66.31.31.216:37951 ESTABLISHED
In this situation, I don't want the producer to block; I would prefer to drop the client completely. http://activemq.apache.org/slow-consumer-handling.html explains how to limit the number of messages queued, which is a good start, but isn't really what I want. http://activemq.apache.org/slow-consumers.html alludes to being able to drop a slow consumer, but doesn't explain how one might do this.
So, this is my question: is it possible to set up ApacheMQ to drop slow consumers completely, and how do I do so?
I would turn off producerFlowControl for the topics you want or all of them like so:
<policyEntry topic=">" producerFlowControl="false">
This has the ability to cause you to run out of memory or disk space now though because the message queue could keep growing. So make sure you set up an eviction strategy and pending message limit strategy like so:
<messageEvictionStrategy>
<oldestMessageEvictionStrategy/>
</messageEvictionStrategy>
<pendingMessageLimitStrategy>
<constantPendingMessageLimitStrategy limit="10"/>
</pendingMessageLimitStrategy>
This will start throwing away messages after a limit of 10 has been reached. The oldest messages will be thrown out first.
AbortSlowConsumerStrategy which has been improved in Active MQ 5.9.0 should help in this aspect. I haven't had the opportunity to test it thoroughly for various configurations. But I think this is what you are looking for, especially ignoreIdleConsumers=true.
http://java.dzone.com/articles/coming-activemq-59-new-way
You can configure below property - "sendFailIfNoSpaceAfterTimeout" for throwing error to Producer, instead of blocking.
<systemUsage>
<systemUsage sendFailIfNoSpaceAfterTimeout="3000">
<memoryUsage>
<memoryUsage limit="256 mb"/>
</memoryUsage>
</systemUsage>
</systemUsage>
I cannot give you an answer on how to drop the client ( in fact: i am not so sure it is possible all together) but what you can do is add a policy to not block the producer:
" memoryLimit="10mb" producerFlowControl="false"/>
This way, your producer will not suffer from slow consumers.
I am using apache servicemix and apache activeMQ in my product. Here in case of HttpConnector i found a performance issue.
If i increase number of total request at a time then as the number will increase exchange queue gets stuck on any of the component. After all the requests start processing, most of the exchange gets stuck at the end component or at the dispatcher* component. Container's heap size reaches to very high and after some time it crashes and restarts automatically.
I have used transactionmanager also. flowname is also mentioned as jms.
need some solution urgently.
You might try using KahaDB instead of the "amqPersistenceAdapter". We saw a huge throughput increase just by switching to this.
here is the config we used (which is highly dependent on your application, but make sure "enableJournalDiskSyncs" is set to false)
<persistenceAdapter>
<kahaDB directory="../data/kaha"
enableJournalDiskSyncs="false"
indexWriteBatchSize="10000"
indexCacheSize="1000" />
</persistenceAdapter>
If i increase number of total request at a time then as the number will increase exchange queue gets stuck on any of the component.
ActiveMQ has a mechanism to stop producer writing messages, it is called "flow controll", seems that you producer is faster than consumer (or consumer is not stable it its speed), so first check the memoryLimit config for your AMQ (also defining it or special queue possible). Try to increase it.
<destinationPolicy>
<policyMap>
<policyEntries>
<policyEntry topic="FOO.>" producerFlowControl="false" memoryLimit="1mb">
<dispatchPolicy>
<strictOrderDispatchPolicy/>
</dispatchPolicy>
<subscriptionRecoveryPolicy>
<lastImageSubscriptionRecoveryPolicy/>
</subscriptionRecoveryPolicy>
</policyEntry>
</policyEntries>
</policyMap>
Also, you can disable that stopping of processing incoming messages with producerFlowControl="false" option. So, in case when all buffer will be used, AMQ will slow down and all messages will be stored to HD. More info Producer Flow Control and Message Cursors
Container's heap size reaches to very high and after some time it crashes and restarts automatically.
But anyway, it is just way of tunning of your app, it is not solution, because always you will have the case when some resources run out :)
You should limit the incoming requests or balance them f.e. using Pound