Active MQ giving OutOfMemory error - activemq

Question

We are running activeMQ5.6 on tomcat 6.0.35 as embedded broker with message delivery option set as PERSISTENT. We are getting OutOfMemory problem at one of the consumer side. The consumer is slow as its doing time consuming job. We used to get the OOM after running for 8-10 hrs. There are ~10000 messages has to be processed nut its giving OOM after processing 3000 messages and rest 7000 messaged kept in pending state. The message size is very small ~1KB in xml format. While we have other consumer on diff queue who are very fast and there also ~10000 messages are published and the message size is quite high ~100 KB but we are not getting OOM on that queue. Though it is setup on the same broker.
Here is stacktrace of the error and out activemq.xml file
INFO [11/08/12 05:39:31]ActiveMQ Session
Task-4- Start Uploading
Nam2011_08_prototype/gdfas/mnada/usa/uf3.7z.001 to Amazon S3 bucket -
aws-s3-infotech Exception in thread "InactivityMonitor WriteCheck"
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:640)
at java.util.concurrent.ThreadPoolExecutor.addIfUnderMaximumPoolSize(ThreadPoolExecutor.java:727)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:657)
at org.apache.activemq.transport.AbstractInactivityMonitor.writeCheck(AbstractInactivityMonitor.java:142)
at org.apache.activemq.transport.AbstractInactivityMonitor$2.run(AbstractInactivityMonitor.java:111)
at org.apache.activemq.thread.SchedulerTimerTask.run(SchedulerTimerTask.java:33)
at java.util.TimerThread.mainLoop(Timer.java:512)
at java.util.TimerThread.run(Timer.java:462)
Here is snapshot from activemq.xml
<persistenceAdapter>
<kahaDB directory="${activemq.data}/kahadb"/>
</persistenceAdapter>
<transportConnectors>
<!-- <transportConnector name="openwire" uri="tcp://localhost:61616"/> -->
<transportConnector name="openwire" uri="tcp://0.0.0.0:61616"/>
<transportConnector name="stomp" uri="stomp://localhost:61613"/>
</transportConnectors>
<networkConnectors>
<!-- by default just auto discover the other brokers -->
<networkConnector name="defaultNetwork" uri="multicast://default"/>
<!--
<networkConnector name="host1 and host2" uri="static://(tcp://host1:61616,tcp://host2:61616)" failover="true"/>
-->
</networkConnectors>
<systemUsage>
<systemUsage>
<memoryUsage>
<memoryUsage limit="512 mb"/>
</memoryUsage>
<storeUsage>
<storeUsage limit="100 gb"/>
</storeUsage>
<tempUsage>
<tempUsage limit="50 gb"/>
</tempUsage>
</systemUsage>
</systemUsage>
<!-- lets define the dispatch policy -->
<destinationPolicy>
<policyMap>
<policyEntries>
<policyEntry queue="SyncServer.>" memoryLimit="512mb" optimizedDispatch="true" queuePrefetch="10">
<pendingQueuePolicy>
<fileQueueCursor/>
</pendingQueuePolicy>
</policyEntry>
</policyEntries>
</policyMap>
</destinationPolicy>

Answer 1

This has nothing to do with ActiveMQ. The error
java.lang.OutOfMemoryError: unable to create new native thread
means that the OS does not have enough free memory to allocate for the thread. The way I think of it is for every thread that Java creates the OS needs to be able to create a 'native' thread, and that takes memory. You need to free up memory on the machine, add memory, or most of the time, unintuitively, you should actually decrease your heap allocation to leave more for the OS.
A general rule of thumb is you need to leave at least the amount of memory free for the OS as you allocate to the JVM. So for example if you have a 2GB heap, you need to have at least 2GB free there after (taking into account that the OS is going to use some memory too).
If you update your answer with your JVM settings, OS, 64/32 bit? and hardware I can help you tune it.