one my server instances has started to go haywire in a multi-instance setup running adobe coldfusion 8.1 enterprise. the built in cf server monitor is throwing these alert snapshots constantly, however, this same box is running Fusion Reactor {http://www.fusion-reactor.com/fr/} and I can not figure out where they are coming from. anyone have any pointers or a good tool to decode these files?
full link: http://pastebin.com/42M2Nzpj
"CM Configuration Updater" prio=5 tid=82 WAITING
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at org.apache.felix.cm.impl.UpdateThread.run(UpdateThread.java:76)
at java.lang.Thread.run(Thread.java:662)
"AWT-Windows" prio=6 tid=299 RUNNABLE
at sun.awt.windows.WToolkit.eventLoop(Native Method)
at sun.awt.windows.WToolkit.run(WToolkit.java:293)
at java.lang.Thread.run(Thread.java:662)
"Signal Dispatcher" prio=9 tid=5 RUNNABLE
"Timer-3" prio=5 tid=67 TIMED_WAITING
at java.lang.Object.wait(Native Method)
at java.util.TimerThread.mainLoop(Timer.java:509)
at java.util.TimerThread.run(Timer.java:462)
"scheduler-4" prio=5 tid=36 TIMED_WAITING
at java.lang.Object.wait(Native Method)
at jrunx.scheduler.SchedulerService.createRunnable(SchedulerService.java:188)
at jrunx.scheduler.ThreadPool$ThreadThrottle.createRunnable(ThreadPool.java:349)
at jrunx.scheduler.WorkerThread.run(WorkerThread.java:62)
"jrpp-145" prio=5 tid=11622 RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
at jrun.servlet.jrpp.ProxyEndpoint.readFully(ProxyEndpoint.java:581)
at jrun.servlet.jrpp.ProxyEndpoint.readFully(ProxyEndpoint.java:573)
at jrun.servlet.jrpp.ProxyEndpoint.readInt(ProxyEndpoint.java:591)
at jrun.servlet.jrpp.ProxyEndpoint.readRequest(ProxyEndpoint.java:231)
at jrun.servlet.jrpp.JRunProxyService.swapRunnable(JRunProxyService.java:143)
at jrunx.scheduler.ThreadPool$ThreadThrottle.swapRunnable(ThreadPool.java:410)
at jrunx.scheduler.WorkerThread.run(WorkerThread.java:76)
I suspect you have an snapshot/alert set to collect a snapshot file when your JVM allocates more than 1000 megabytes of memory. This is probably not a great idea since under normal circumstances your Jvm might indeed allocate that much and recover with a GC. I would look at your alert/snapshot settings and adjust them.
Related
Does contention only occurs if non daemon threads (application thread ) are trying to acquire the same lock or it can also occurs if daemon threads (jvm internal threads ) causes some issue ?
So, my main question is can daemon threads can cause contention ?
The short answer is yes... WebLogic's threads are all daemon threads.
We are changing our cluster topology and facing a 100% CPU issue in the "default I/O-x" thread (org.xnio.nio.WorkerThread).
Looks like an infinity loop in the ConduitStreamSinkChannel.write(..) but I'm not sure about it.
The problem is intermittent and not simple to reproduce, and it needs a restart to stop.
Every time that it enters this state, the thread dump is the same:
"default I/O-2" #103 prio=5 os_prio=0 tid=0x00007fd0f83a4800 nid=0x6941 runnable [0x00007fd0e9be0000]
java.lang.Thread.State: RUNNABLE
at java.lang.Throwable.fillInStackTrace(Native Method)
at java.lang.Throwable.fillInStackTrace(Throwable.java:783)
- locked <0x00000007a2be6ec0> (a java.nio.channels.ClosedChannelException)
at java.lang.Throwable.(Throwable.java:250)
at java.lang.Exception.(Exception.java:54)
at java.io.IOException.(IOException.java:47)
at java.nio.channels.ClosedChannelException.(ClosedChannelException.java:52)
at org.xnio.ssl.JsseStreamConduit.write(JsseStreamConduit.java:1022)
at org.xnio.conduits.ConduitStreamSinkChannel.write(ConduitStreamSinkChannel.java:150)
at org.xnio.http.HttpUpgrade$HttpUpgradeState$StringWriteListener.handleEvent(HttpUpgrade.java:385)
at org.xnio.http.HttpUpgrade$HttpUpgradeState$StringWriteListener.handleEvent(HttpUpgrade.java:372)
at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92)
at org.xnio.conduits.WriteReadyHandler$ChannelListenerHandler.writeReady(WriteReadyHandler.java:65)
at org.xnio.ssl.JsseStreamConduit.run(JsseStreamConduit.java:393)
at org.xnio.ssl.JsseStreamConduit.writeReady(JsseStreamConduit.java:524)
at org.xnio.ssl.JsseStreamConduit$1.writeReady(JsseStreamConduit.java:287)
at org.xnio.nio.NioSocketConduit.handleReady(NioSocketConduit.java:94)
at org.xnio.nio.WorkerThread.run(WorkerThread.java:567)
The change in the topology is adding a few more servers groups and changes the reverse proxy (nginx) in order to redirect (status 301) every http connection to https.
Analyzing the thread dump, seems to be something related to https (package org.xnio.ssl.*), but everything behind the nginx is plain http.
Our application provides a Websocket endpoint (WSS through nginx) and makes a lot of EJB remote invocations to others servers (http-remoting, directly to the other server without passing through nginx).
Also, the application make/receive some REST calls (RESTEasy) to others servers in the cluster through the load balancer (nginx https, in order to avoid the 301 redirection)
What could be causing this?
thanks!
Hi am working on wso2esb and using Active MQ for message queues.
After around 3 weeks of usage ESB server was hanging and with the help of JMX monitoring of ESB i found that they are a huge number of java threads are in WAITING state.
[EsbMonitoring] ***************** java Threads Attributes *********************
[EsbMonitoring] ThreadCount :8873
[EsbMonitoring] DaemonThreadCount :104
[EsbMonitoring] PeakThreadCount :8992
[EsbMonitoring] TotalStartedThreadCount :16086123
Initially when we start the ESB server they are around 560 threads
[EsbMonitoring] ***************** java Threads Attributes *********************
[EsbMonitoring] ThreadCount :592
[EsbMonitoring] DaemonThreadCount :78
[EsbMonitoring] PeakThreadCount :592
[EsbMonitoring] TotalStartedThreadCount :1510
I took thread dump with jstack of ESB server with
"ActiveMQ Connection Executor: tcp://my-desktop/127.0.1.1:61616#60595" prio=10 tid=0x00007fdb1c890800 nid=0x259e waiting on condition [0x00007fda951cd000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x000000070bdf6c18> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"ActiveMQ Connection Executor: tcp://my-desktop/127.0.1.1:61616#60589" prio=10 tid=0x00007fdb1c8b7800 nid=0x259a waiting on condition [0x00007fda950cc000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x000000070be68c48> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"ActiveMQ Connection Executor: tcp://my-desktop/127.0.1.1:61616#60590" prio=10 tid=0x00007fdb1c8ae800 nid=0x2597 waiting on condition [0x00007fda946c2000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x000000070be22c88> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
All this connections are closed in the active MQ side but ESB is still holding these threads and troubling ESB and making consumption high and after reaching curtain point it hangs . I need to restart the ESB server to solve this issue.
Firstly why are these connection are WAITING state forever and why ESB is not removing these kind of threads , Is there any way i can kill or destroy WAITING threads with out restarting ESB.
Any help would be appreciated.
Thank You..!
Another few questions
What version of WSO server ?
What is jvm version ?
Anyway, i assume you have pretty high load. I don't think i can reproduce it easy in short time. I had something similar with postgresql jdbc driver, connections were getting stuck after some time, and all connection in pool are got blocked. It was fixed just on jdbc driver update.
According http://mvnrepository.com/artifact/org.apache.activemq/activemq-broker version 5.8.0 quite old. It was released in 2013. There are some bug reports in activemq jira about connection leak. Few thing you can try.
Get latest version of library from jboss repository. Link you will find on same page as shown on picture underneath
Just be accurate with library dependencies. Update dependencies as well. List of jar files is underneath (It is from official documentation https://docs.wso2.com/display/ESB500/Configure+with+ActiveMQ)
activemq-broker-5.8.0.jar
activemq-client-5.8.0.jar
activemq-kahadb-store-5.8.0.jar
geronimo-jms_1.1_spec-1.1.1.jar
geronimo-j2ee-management_1.1_spec-1.0.1.jar
geronimo-jta_1.0.1B_spec-1.0.1.jar
hawtbuf-1.9.jar
slf4j-api-1.6.6.jar
activeio-core-3.1.4.jar (available in /lib/optional folder)
Another way just update to next release of activemq broker. ActiveMq mature project, so it less possible that in every new release it introduces massive amount of new features and break compatibility with previous version. If release notes doesn't state anything extra ordinary, try it, test it in test environment and make decision.
I am using Spring AMQP.
My Consumer is stopped listening messages, however there is no exception in logs and I could not see any deadlock on thread dump.
I have two servers which are listening to 10 queues.
Everything was working fine when cache-mode of rabbit:connection-factory was 'CONNECTION'.
But when I changed it to 'CHANNEL' this issue started appearing frequently.
From the rabbitMQ UI console I can see there is one unacknowledged messages. (Acknowledge mode is 'AUTO').
It looks to me that client could not send acknowledgement back to the server.
I can see following in my thread dump
"" prio=10 tid=0x00007f5f3823c000 nid=0x5faa waiting on condition [0x00007f5f847e2000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007b83a5260> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at org.springframework.amqp.rabbit.listener.BlockingQueueConsumer.nextMessage(BlockingQueueConsumer.java:359)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.doReceiveAndExecute(SimpleMessageListenerContainer.java:1000)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.receiveAndExecute(SimpleMessageListenerContainer.java:989)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.access$700(SimpleMessageListenerContainer.java:82)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$AsyncMessageProcessingConsumer.run(SimpleMessageListenerContainer.java:1103)
at java.lang.Thread.run(Thread.java:745)
How can I integrate Jprofiler UI running in my laptop connect to mule server running in a remote server. I had installed jprofiler remotely in the server box and tried to run jpenable to attach to the mule server, and it's unable to. Here is the error I'm getting.
Connecting to org.mule.module.reboot.MuleContainerBootstrap start0 [9828] ...
Please select the profiling mode:
GUI mode (attach with JProfiler GUI) [1, Enter]
Offline mode (use config file to set profiling settings) [2]
1
ERROR: Attaching is not supported: Unable to open socket file: target process not responding or HotSpot VM not loaded
com.sun.tools.attach.AttachNotSupportedException: Unable to open socket file: target process not responding or HotSpot VM not loaded
at sun.tools.attach.LinuxVirtualMachine.<init>(LinuxVirtualMachine.java:100)
at sun.tools.attach.LinuxAttachProvider.attachVirtualMachine(LinuxAttachProvider.java:63)
at com.sun.tools.attach.VirtualMachine.attach(VirtualMachine.java:195)
at com.jprofiler.frontend.attach.c.b(ejt:265)
at com.jprofiler.frontend.attach.c.a(ejt:252)
at com.jprofiler.frontend.attach.c.a(ejt:222)
at com.jprofiler.frontend.EnableApplication.a(ejt:118)
at com.jprofiler.frontend.EnableApplication.g(ejt:81)
at com.jprofiler.frontend.EnableApplication.main(ejt:238)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.exe4j.runtime.LauncherEngine.launch(Unknown Source)
at com.install4j.runtime.launcher.Launcher.main(Unknown Source)
The attach mechanism in the JVM is not very reliable, there are circumstances that can cause it to fail that are outside the control of a tool vendor.
First, check if jpenable is run as the same user as the application that you want to profile. If using the same users has no effect, you have to add the VM parameter for profiling to the startup command of your application and restart it.
You can get that that parameter in JProfiler by invoking
Session->Integration Wizards->New Remote Integration
and following the steps in the wizard.
change permissions to /usr/lib/jvm(set it 777) to probe