ActiveMQ destinationPolicy and systemUsage Configuration - activemq

Question

Looking for some help around systemUsage and destinationPolicy configuration as I'm having some difficulty fully understanding the relationship between systemUsage, destinationPolicy, and flow control.
All our messages are persistent! producerFlowControl is on.
So we give ActiveMQ say a maximum of 512MB heap space.
Our systemUsage is set as below:
<systemUsage>
<systemUsage>
<memoryUsage>
<memoryUsage limit="200 mb"/>
</memoryUsage>
<storeUsage>
<storeUsage limit="10 gb"/>
</storeUsage>
<tempUsage>
<tempUsage limit="1000 mb"/>
</tempUsage>
</systemUsage>
</systemUsage>
Our destination policy as below:
<destinationPolicy>
<policyMap>
<policyEntries>
<policyEntry topic=">" producerFlowControl="true" memoryLimit="1mb">
<pendingSubscriberPolicy>
</pendingSubscriberPolicy>
</policyEntry>
<policyEntry queue=">" producerFlowControl="true" memoryLimit="1mb">
</policyEntry>
</policyEntries>
</policyMap>
</destinationPolicy>
Can anyone verify the if the following is correct:
This means that for each individual queue/topic the memory limit is 1MB. What exactly happens when this 1MB is hit, does the queue block for producers or does it page to disc?
The total allowed memory for all queues and topics is 200MB. Meaning that we could have 200 channels operating at their full capacity of 1MB. We currently have 16 queues and topics in total so obviously that is never reached.
Are we better to remove the individual policy entry on the memory limit and share the memory between the various channels?
If we do this at what point would they block?
Any help very much appreciated! Can paypal you some beer money!

Answer 1

You're touching on a number of points here, which I'll answer out of order.
memoryUsage corresponds to the amount of memory that's assigned to the in-memory store. storeUsage corresponds to how much space should be given to the KahaDB store. You either use one or the other, depending on how you want your broker to persist messages or not. tempUsage is a special case for file cursors (http://activemq.apache.org/message-cursors.html) - a mechanism to overflow the memory from the in-memory store to disk if the memory limit is exceeded (you have to configure this behaviour at the destination level if you want it).
policyEntry#memoryLimit is a sub-limit for individual destinations.
What happens when memory limits are exceeded depends on whether producer flow control (PFC) is turned on. It's on by default for queues, off for topics and asynchronous sends to queues; all this can be configured in the policyEntry (http://activemq.apache.org/per-destination-policies.html).
If you hit a "memory limit" when PFC is on, your clients will block until someone frees up space by consuming messages from the store. If it's off, the send will throw an exception (better the client fall over than the broker). "Memory limit" means either the one defined by memoryUsage across all queues, or the queue-specific limit (it's possible to hit the former before the latter).
Whether or not you want a destination-specific limit depends on your use case. I'd suggest to ignore it unless you're trying to achieve a specific outcome.