Tomcat 7.0.25 apr jre crash - jvm
I just upgraded from tomcat 6.0.29 to 7.0.25 using native apr.
Tomcat has crashed a few times with the report below.
None of it means much to me other than that the crash seemed to happen when using zip, which is could be to do with the gzip setting to compress certain file types.
I couldn't find any other references anywhere with this issue, and when I looked at the changelog from 7.0.25 to 7.0.26 no mention was made of fixing any issues like this.
Any advice/help/solutions welcome.
Thanks.
#
# A fatal error has been detected by the Java Runtime Environment:
#
# EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x00000000739a576c, pid=2504, tid=3792
#
# JRE version: 7.0_02-b13
# Java VM: Java HotSpot(TM) 64-Bit Server VM (22.0-b10 mixed mode windows-amd64 compressed oops)
# Problematic frame:
# C [zip.dll+0x576c]
#
# Core dump written. Default location: C:\production\Tomcat-7.0.25\bin\hs_err_pid2504.mdmp
#
# If you would like to submit a bug report, please visit:
# http://bugreport.sun.com/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#
--------------- T H R E A D ---------------
Current thread (0x00000000089e9000): JavaThread "http-apr-80-exec-32" daemon [_thread_in_native, id=3792, stack(0x0000000019400000,0x0000000019500000)]
siginfo: ExceptionCode=0xc0000005, reading address 0x000000000909f000
Registers:
RAX=0x0000000009056110, RBX=0x0000000012dc1020, RCX=0x0000000000048ef0, RDX=0x0000000009d2a290
RSP=0x00000000194fe710, RBP=0x0000000000000004, RSI=0x0000000000000001, RDI=0x0000000000000000
R8 =0x0000000000000000, R9 =0x0000000000048eee, R10=0x4b0c4ed76a743daf, R11=0x00000007c260d3c0
R12=0x0000000000008eec, R13=0x0000000000000003, R14=0x0000000000000002, R15=0x0000000000000002
RIP=0x00000000739a576c, EFLAGS=0x0000000000010286
Top of Stack: (sp=0x00000000194fe710)
0x00000000194fe710: 00000000089e91d0 00000000739a4262
0x00000000194fe720: 0000000000000024 0000000069fa4915
0x00000000194fe730: 0000000000000001 00000000ffffffff
0x00000000194fe740: 000000000000029a 00000000739a49ae
0x00000000194fe750: 0000000012dc1020 0000000000000004
0x00000000194fe760: 0000000013241630 0000000000000000
0x00000000194fe770: 0000000000000200 0000000000002301
0x00000000194fe780: 00000007c260d600 0000000008c42580
0x00000000194fe790: 0000000000000000 00000000739a16df
0x00000000194fe7a0: 00000000089e91d0 00000000194fe848
0x00000000194fe7b0: 0000000013241630 0000000755fe5e48
0x00000000194fe7c0: 0000000755fe5d38 0000000755fe5d68
0x00000000194fe7d0: 00000000089e9000 0000000000002328
0x00000000194fe7e0: 0000000755fe5e48 0000000000000000
0x00000000194fe7f0: 0000000000000000 0000000000000200
0x00000000194fe800: 00000000194fe880 0000000002715bbc
Instructions: (pc=0x00000000739a576c)
0x00000000739a574c: 85 c0 0f 84 d9 03 00 00 41 3b c5 72 69 48 8b 43
0x00000000739a575c: 50 44 8b 8b 94 00 00 00 48 8b 53 68 41 8d 49 02
0x00000000739a576c: 44 0f b6 04 01 8b 43 70 8b 8b 80 00 00 00 d3 e0
0x00000000739a577c: 48 8b 4b 60 44 33 c0 8b 43 4c 44 23 43 7c 4c 23
Register to memory mapping:
RAX=0x0000000009056110 is an unknown value
RBX=0x0000000012dc1020 is an unknown value
RCX=0x0000000000048ef0 is an unknown value
RDX=0x0000000009d2a290 is an unknown value
RSP=0x00000000194fe710 is pointing into the stack for thread: 0x00000000089e9000
RBP=0x0000000000000004 is an unknown value
RSI=0x0000000000000001 is an unknown value
RDI=0x0000000000000000 is an unknown value
R8 =0x0000000000000000 is an unknown value
R9 =0x0000000000048eee is an unknown value
R10=0x4b0c4ed76a743daf is an unknown value
R11=0x00000007c260d3c0 is an unknown value
R12=0x0000000000008eec is an unknown value
R13=0x0000000000000003 is an unknown value
R14=0x0000000000000002 is an unknown value
R15=0x0000000000000002 is an unknown value
Stack: [0x0000000019400000,0x0000000019500000], sp=0x00000000194fe710, free space=1017k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C [zip.dll+0x576c] ZIP_Open+0x1690
Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
J java.util.zip.Deflater.deflateBytes(J[BIII)I
J java.util.zip.GZIPOutputStream.finish()V
j org.apache.coyote.http11.filters.FlushableGZIPOutputStream.finish()V+9
j org.apache.coyote.http11.filters.GzipOutputFilter.end()J+26
J org.apache.coyote.http11.AbstractHttp11Processor.action(Lorg/apache/coyote/ActionCode;Ljava/lang/Object;)V
J org.apache.catalina.connector.OutputBuffer.close()V
j org.apache.catalina.connector.CoyoteWriter.close()V+4
j org.apache.catalina.core.ApplicationDispatcher.doForward(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V+457
j org.apache.catalina.core.ApplicationDispatcher.forward(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V+55
j org.tuckey.web.filters.urlrewrite.RewrittenUrl.doRewrite(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;Ljavax/servlet/FilterChain;)Z+110
j org.tuckey.web.filters.urlrewrite.UrlRewriteFilter.doFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;Ljavax/servlet/FilterChain;)V+289
J org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V
J org.apache.catalina.core.StandardWrapperValve.invoke(Lorg/apache/catalina/connector/Request;Lorg/apache/catalina/connector/Response;)V
J org.apache.catalina.authenticator.AuthenticatorBase.invoke(Lorg/apache/catalina/connector/Request;Lorg/apache/catalina/connector/Response;)V
J org.apache.catalina.valves.ErrorReportValve.invoke(Lorg/apache/catalina/connector/Request;Lorg/apache/catalina/connector/Response;)V
j org.apache.catalina.valves.AccessLogValve.invoke(Lorg/apache/catalina/connector/Request;Lorg/apache/catalina/connector/Response;)V+6
j org.apache.catalina.core.StandardEngineValve.invoke(Lorg/apache/catalina/connector/Request;Lorg/apache/catalina/connector/Response;)V+71
j org.apache.catalina.connector.CoyoteAdapter.service(Lorg/apache/coyote/Request;Lorg/apache/coyote/Response;)V+196
J org.apache.coyote.http11.AbstractHttp11Processor.process(Lorg/apache/tomcat/util/net/SocketWrapper;)Lorg/apache/tomcat/util/net/AbstractEndpoint$Handler$SocketState;
J org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(Lorg/apache/tomcat/util/net/SocketWrapper;Lorg/apache/tomcat/util/net/SocketStatus;)Lorg/apache/tomcat/util/net/AbstractEndpoint$Handler$SocketState;
J java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V
j java.util.concurrent.ThreadPoolExecutor$Worker.run()V+5
j java.lang.Thread.run()V+11
v ~StubRoutines::call_stub
--------------- P R O C E S S ---------------
Java Threads: ( => current thread )
0x0000000009bc0000 JavaThread "http-apr-80-exec-90" daemon [_thread_blocked, id=3868, stack(0x000000001c680000,0x000000001c780000)]
0x0000000009bbf000 JavaThread "http-apr-80-exec-89" daemon [_thread_in_native, id=3396, stack(0x000000001c530000,0x000000001c630000)]
0x0000000009bbe800 JavaThread "http-apr-80-exec-88" daemon [_thread_blocked, id=5560, stack(0x000000001bd10000,0x000000001be10000)]
0x0000000009bbd800 JavaThread "http-apr-80-exec-87" daemon [_thread_blocked, id=5900, stack(0x000000001a2e0000,0x000000001a3e0000)]
0x0000000009bbd000 JavaThread "http-apr-80-exec-86" daemon [_thread_blocked, id=4708, stack(0x000000001c420000,0x000000001c520000)]
0x0000000009bbc000 JavaThread "http-apr-80-exec-85" daemon [_thread_in_native, id=2408, stack(0x000000001c1d0000,0x000000001c2d0000)]
0x0000000009bbb800 JavaThread "http-apr-80-exec-84" daemon [_thread_in_native, id=1864, stack(0x000000001c010000,0x000000001c110000)]
0x0000000009c22800 JavaThread "http-apr-80-exec-83" daemon [_thread_in_native, id=5280, stack(0x000000001be30000,0x000000001bf30000)]
0x0000000009c22000 JavaThread "http-apr-80-exec-82" daemon [_thread_in_native, id=5624, stack(0x000000001ba70000,0x000000001bb70000)]
0x0000000009c21800 JavaThread "http-apr-80-exec-81" daemon [_thread_in_native, id=5812, stack(0x000000001bc10000,0x000000001bd10000)]
0x0000000009c20800 JavaThread "http-apr-80-exec-80" daemon [_thread_in_native, id=4076, stack(0x000000001b820000,0x000000001b920000)]
0x0000000009c20000 JavaThread "http-apr-80-exec-79" daemon [_thread_in_native, id=260, stack(0x000000001b960000,0x000000001ba60000)]
0x0000000009c1f000 JavaThread "http-apr-80-exec-78" daemon [_thread_in_native, id=5868, stack(0x000000001b6a0000,0x000000001b7a0000)]
0x0000000009c1e800 JavaThread "http-apr-80-exec-77" daemon [_thread_in_native, id=1748, stack(0x000000001b3d0000,0x000000001b4d0000)]
0x0000000009c1d800 JavaThread "http-apr-80-exec-76" daemon [_thread_in_native, id=6000, stack(0x000000001b510000,0x000000001b610000)]
0x0000000009c1d000 JavaThread "http-apr-80-exec-75" daemon [_thread_in_native, id=4028, stack(0x000000001acf0000,0x000000001adf0000)]
0x0000000009c1c000 JavaThread "http-apr-80-exec-74" daemon [_thread_in_native, id=5540, stack(0x000000001b240000,0x000000001b340000)]
0x0000000009c1b800 JavaThread "http-apr-80-exec-73" daemon [_thread_in_native, id=5920, stack(0x000000001a0b0000,0x000000001a1b0000)]
0x000000000b4a2000 JavaThread "http-apr-80-exec-72" daemon [_thread_in_native, id=5164, stack(0x000000001b0f0000,0x000000001b1f0000)]
0x000000000b4a1000 JavaThread "http-apr-80-exec-71" daemon [_thread_in_native, id=4544, stack(0x00000000110d0000,0x00000000111d0000)]
0x000000000b4a0800 JavaThread "http-apr-80-exec-70" daemon [_thread_in_native, id=4236, stack(0x000000001afd0000,0x000000001b0d0000)]
0x000000000b49f800 JavaThread "http-apr-80-exec-69" daemon [_thread_in_native, id=5012, stack(0x000000001adf0000,0x000000001aef0000)]
0x000000000b49f000 JavaThread "http-apr-80-exec-68" daemon [_thread_in_native, id=2808, stack(0x000000001a600000,0x000000001a700000)]
0x000000000b49e000 JavaThread "http-apr-80-exec-67" daemon [_thread_in_native, id=3112, stack(0x000000001aba0000,0x000000001aca0000)]
0x000000000b49d800 JavaThread "http-apr-80-exec-66" daemon [_thread_in_native, id=2512, stack(0x000000001aa70000,0x000000001ab70000)]
0x000000000b49d000 JavaThread "http-apr-80-exec-65" daemon [_thread_in_native, id=2184, stack(0x000000001a970000,0x000000001aa70000)]
0x000000000b49c000 JavaThread "http-apr-80-exec-64" daemon [_thread_in_native, id=4912, stack(0x000000001a840000,0x000000001a940000)]
0x000000000b49b800 JavaThread "http-apr-80-exec-63" daemon [_thread_in_native, id=3704, stack(0x000000001a700000,0x000000001a800000)]
0x000000000b49a800 JavaThread "http-apr-80-exec-62" daemon [_thread_in_native, id=5352, stack(0x000000001a4f0000,0x000000001a5f0000)]
0x00000000089e7000 JavaThread "http-apr-80-exec-61" daemon [_thread_in_native, id=5744, stack(0x000000001a3f0000,0x000000001a4f0000)]
0x00000000098af800 JavaThread "http-apr-80-exec-60" daemon [_thread_in_native, id=5604, stack(0x000000001a1e0000,0x000000001a2e0000)]
0x0000000008de6800 JavaThread "http-apr-80-exec-59" daemon [_thread_in_native, id=1052, stack(0x0000000019f80000,0x000000001a080000)]
0x0000000008dea800 JavaThread "http-apr-80-exec-58" daemon [_thread_in_native, id=2332, stack(0x0000000019ce0000,0x0000000019de0000)]
0x0000000008de8000 JavaThread "http-apr-80-exec-57" daemon [_thread_in_native, id=4608, stack(0x0000000017bb0000,0x0000000017cb0000)]
0x0000000008de9000 JavaThread "http-apr-80-exec-56" daemon [_thread_in_native, id=5772, stack(0x0000000012a40000,0x0000000012b40000)]
0x0000000009498800 JavaThread "http-apr-80-exec-55" daemon [_thread_in_native, id=4008, stack(0x0000000019a90000,0x0000000019b90000)]
0x000000000949a800 JavaThread "http-apr-80-exec-54" daemon [_thread_in_native, id=5852, stack(0x0000000019e00000,0x0000000019f00000)]
0x000000000949f000 JavaThread "http-apr-80-exec-53" daemon [_thread_in_native, id=1260, stack(0x0000000019ba0000,0x0000000019ca0000)]
0x000000000949c800 JavaThread "http-apr-80-exec-52" daemon [_thread_in_native, id=4816, stack(0x0000000019980000,0x0000000019a80000)]
0x0000000009fde800 JavaThread "http-apr-80-exec-51" daemon [_thread_in_native, id=4572, stack(0x00000000190a0000,0x00000000191a0000)]
0x0000000009fd9800 JavaThread "http-apr-80-exec-50" daemon [_thread_in_native, id=6056, stack(0x0000000019560000,0x0000000019660000)]
0x0000000009fdf000 JavaThread "http-apr-80-exec-49" daemon [_thread_in_native, id=4456, stack(0x0000000018e30000,0x0000000018f30000)]
0x0000000009fda000 JavaThread "http-apr-80-exec-48" daemon [_thread_in_native, id=3568, stack(0x0000000018f30000,0x0000000019030000)]
0x0000000009fda800 JavaThread "http-apr-80-exec-47" daemon [_thread_in_native, id=5616, stack(0x0000000017ab0000,0x0000000017bb0000)]
0x000000000b4c2000 JavaThread "http-apr-80-exec-46" daemon [_thread_in_native, id=5184, stack(0x000000000f420000,0x000000000f520000)]
0x000000000b4c1000 JavaThread "http-apr-80-exec-45" daemon [_thread_in_native, id=1280, stack(0x0000000015990000,0x0000000015a90000)]
0x000000000b4c0800 JavaThread "http-apr-80-exec-44" daemon [_thread_in_native, id=5172, stack(0x000000000de80000,0x000000000df80000)]
0x000000000b4bf800 JavaThread "http-apr-80-exec-43" daemon [_thread_in_native, id=2656, stack(0x00000000154d0000,0x00000000155d0000)]
0x000000000b4bf000 JavaThread "http-apr-80-exec-42" daemon [_thread_in_native, id=5436, stack(0x0000000014260000,0x0000000014360000)]
0x000000000b4be000 JavaThread "http-apr-80-exec-41" daemon [_thread_in_native, id=3660, stack(0x0000000013f50000,0x0000000014050000)]
0x000000000b4bd800 JavaThread "http-apr-80-exec-40" daemon [_thread_in_native, id=1200, stack(0x0000000013d60000,0x0000000013e60000)]
0x000000000b4bd000 JavaThread "http-apr-80-exec-39" daemon [_thread_in_native, id=5528, stack(0x0000000012930000,0x0000000012a30000)]
0x000000000b4ba800 JavaThread "http-apr-80-exec-38" daemon [_thread_in_native, id=3808, stack(0x000000000fef0000,0x000000000fff0000)]
0x000000000949e000 JavaThread "Thread-87" daemon [_thread_blocked, id=2140, stack(0x0000000011370000,0x0000000011470000)]
0x000000000949b000 JavaThread "http-apr-443-exec-12" daemon [_thread_blocked, id=4604, stack(0x000000000e140000,0x000000000e240000)]
0x000000000949c000 JavaThread "http-apr-80-exec-37" daemon [_thread_in_native, id=1380, stack(0x000000000dfc0000,0x000000000e0c0000)]
=>0x00000000089e9000 JavaThread "http-apr-80-exec-32" daemon [_thread_in_native, id=3792, stack(0x0000000019400000,0x0000000019500000)]
0x0000000009fe0000 JavaThread "http-apr-80-exec-31" daemon [_thread_in_native, id=4196, stack(0x0000000019260000,0x0000000019360000)]
0x0000000009fdd800 JavaThread "http-apr-80-exec-28" daemon [_thread_in_native, id=3416, stack(0x0000000014140000,0x0000000014240000)]
0x0000000009fdd000 JavaThread "http-apr-80-exec-27" daemon [_thread_in_native, id=5372, stack(0x0000000013b20000,0x0000000013c20000)]
0x0000000009fdc000 JavaThread "http-apr-80-exec-26" daemon [_thread_in_native, id=756, stack(0x00000000157b0000,0x00000000158b0000)]
0x0000000009fdb800 JavaThread "http-apr-443-exec-11" daemon [_thread_blocked, id=3500, stack(0x0000000018ce0000,0x0000000018de0000)]
0x0000000009fd8800 JavaThread "http-apr-80-exec-24" daemon [_thread_in_native, id=3312, stack(0x0000000015600000,0x0000000015700000)]
0x0000000009499800 JavaThread "Java2D Disposer" daemon [_thread_blocked, id=5792, stack(0x0000000015370000,0x0000000015470000)]
0x00000000089e8800 JavaThread "http-apr-80-exec-21" daemon [_thread_in_native, id=5348, stack(0x0000000013c30000,0x0000000013d30000)]
0x0000000009499000 JavaThread "http-apr-80-exec-15" daemon [_thread_in_native, id=1016, stack(0x000000000a360000,0x000000000a460000)]
0x0000000009497800 JavaThread "http-apr-80-exec-13" daemon [_thread_in_native, id=3952, stack(0x00000000127c0000,0x00000000128c0000)]
0x00000000098b3800 JavaThread "http-apr-80-exec-12" daemon [_thread_in_native, id=4452, stack(0x0000000011630000,0x0000000011730000)]
0x00000000098b2800 JavaThread "http-apr-80-exec-11" daemon [_thread_in_native, id=5472, stack(0x0000000011530000,0x0000000011630000)]
0x00000000098b1000 JavaThread "http-apr-443-exec-10" daemon [_thread_blocked, id=3688, stack(0x000000000a460000,0x000000000a560000)]
0x00000000098b0800 JavaThread "http-apr-443-exec-9" daemon [_thread_blocked, id=4764, stack(0x000000000e810000,0x000000000e910000)]
0x00000000098af000 JavaThread "http-apr-443-exec-7" daemon [_thread_blocked, id=3676, stack(0x00000000111d0000,0x00000000112d0000)]
0x00000000098ae800 JavaThread "http-apr-443-exec-6" daemon [_thread_blocked, id=3772, stack(0x0000000010fd0000,0x00000000110d0000)]
0x00000000098ad800 JavaThread "http-apr-443-exec-5" daemon [_thread_blocked, id=3936, stack(0x0000000010e30000,0x0000000010f30000)]
0x00000000098ad000 JavaThread "http-apr-443-exec-4" daemon [_thread_blocked, id=4256, stack(0x000000000fa80000,0x000000000fb80000)]
0x00000000098ac000 JavaThread "http-apr-443-exec-3" daemon [_thread_blocked, id=3240, stack(0x0000000010cd0000,0x0000000010dd0000)]
0x00000000089e9800 JavaThread "http-apr-443-exec-2" daemon [_thread_blocked, id=3108, stack(0x0000000010b20000,0x0000000010c20000)]
0x00000000089e7800 JavaThread "http-apr-80-exec-10" daemon [_thread_in_native, id=2676, stack(0x000000000ed30000,0x000000000ee30000)]
0x00000000089e6000 JavaThread "http-apr-443-AsyncTimeout" daemon [_thread_blocked, id=3988, stack(0x00000000101e0000,0x00000000102e0000)]
0x00000000089e5800 JavaThread "http-apr-443-Acceptor-0" daemon [_thread_in_native, id=4804, stack(0x000000000fdb0000,0x000000000feb0000)]
0x00000000089e4800 JavaThread "http-apr-443-CometPoller-7" daemon [_thread_blocked, id=2556, stack(0x000000000fbc0000,0x000000000fcc0000)]
0x00000000089e4000 JavaThread "http-apr-443-CometPoller-6" daemon [_thread_blocked, id=3820, stack(0x000000000f970000,0x000000000fa70000)]
0x00000000089e3000 JavaThread "http-apr-443-CometPoller-5" daemon [_thread_blocked, id=2528, stack(0x000000000f850000,0x000000000f950000)]
0x00000000089e2800 JavaThread "http-apr-443-CometPoller-4" daemon [_thread_blocked, id=3748, stack(0x000000000f6e0000,0x000000000f7e0000)]
0x00000000089e2000 JavaThread "http-apr-443-CometPoller-3" daemon [_thread_blocked, id=1936, stack(0x000000000f520000,0x000000000f620000)]
0x00000000089e1000 JavaThread "http-apr-443-CometPoller-2" daemon [_thread_blocked, id=5028, stack(0x000000000f310000,0x000000000f410000)]
0x00000000089e0800 JavaThread "http-apr-443-CometPoller-1" daemon [_thread_blocked, id=5672, stack(0x000000000f090000,0x000000000f190000)]
0x00000000089df800 JavaThread "http-apr-443-CometPoller-0" daemon [_thread_blocked, id=3224, stack(0x000000000f1c0000,0x000000000f2c0000)]
0x00000000089df000 JavaThread "http-apr-443-Poller-7" daemon [_thread_blocked, 0x0000000008de1800 JavaThread "http-apr-80-CometPoller-5" daemon [_thread_blocked, id=5124, stack(0x000000000d5b0000,0x000000000d6b0000)]
0x0000000008de1000 JavaThread "http-apr-80-CometPoller-4" daemon [_thread_blocked, id=4944, stack(0x000000000cf30000,0x000000000d030000)]
0x0000000008de0000 JavaThread "http-apr-80-CometPoller-3" daemon [_thread_blocked, id=4924, stack(0x000000000d040000,0x000000000d140000)]
0x0000000008ddf800 JavaThread "http-apr-80-CometPoller-2" daemon [_thread_blocked, id=2076, stack(0x000000000c910000,0x000000000ca10000)]
Other Threads:
0x0000000001f0b000 VMThread [stack: 0x0000000007530000,0x0000000007630000] [id=4292]
0x00000000070d5800 WatcherThread [stack: 0x0000000007960000,0x0000000007a60000] [id=2920]
VM state:not at safepoint (normal execution)
VM Mutex/Monitor currently owned by a thread: None
Heap
PSYoungGen total 1037952K, used 70503K [0x00000007c0000000, 0x0000000800000000, 0x0000000800000000)
eden space 1026688K, 6% used [0x00000007c0000000,0x00000007c406a560,0x00000007feaa0000)
from space 11264K, 40% used [0x00000007ff500000,0x00000007ff96f770,0x0000000800000000)
to space 10624K, 0% used [0x00000007feaa0000,0x00000007feaa0000,0x00000007ff500000)
PSOldGen total 2097152K, used 430155K [0x0000000740000000, 0x00000007c0000000, 0x00000007c0000000)
object space 2097152K, 20% used [0x0000000740000000,0x000000075a412ce8,0x00000007c0000000)
PSPermGen total 88768K, used 47835K [0x0000000680000000, 0x00000006856b0000, 0x0000000740000000)
object space 88768K, 53% used [0x0000000680000000,0x0000000682eb6eb8,0x00000006856b0000)
Code Cache [0x00000000020e0000, 0x00000000029f0000, 0x00000000050e0000)
total_blobs=2605 nmethods=1994 adapters=562 free_code_cache=39996Kb largest_free_block=40849792
Dynamic libraries:
0x000000013f360000 - 0x000000013f393000 C:\Java\jdk1.7.0_02\bin\java.exe
0x0000000076d40000 - 0x0000000076eec000 C:\Windows\SYSTEM32\ntdll.dll
0x0000000076c20000 - 0x0000000076d3f000 C:\Windows\system32\kernel32.dll
0x000007fefd000000 - 0x000007fefd06b000 C:\Windows\system32\KERNELBASE.dll
0x000007fefef70000 - 0x000007feff04b000 C:\Windows\system32\ADVAPI32.dll
0x000007fefe6c0000 - 0x000007fefe75f000 C:\Windows\system32\msvcrt.dll
0x000007fefd680000 - 0x000007fefd69f000 C:\Windows\SYSTEM32\sechost.dll
0x000007fefe8d0000 - 0x000007fefe9fe000 C:\Windows\system32\RPCRT4.dll
0x0000000076b20000 - 0x0000000076c1a000 C:\Windows\system32\USER32.dll
0x000007fefef00000 - 0x000007fefef67000 C:\Windows\system32\GDI32.dll
0x000007fefd6a0000 - 0x000007fefd6ae000 C:\Windows\system32\LPK.dll
0x000007fefee30000 - 0x000007fefeefa000 C:\Windows\system32\USP10.dll
0x000007fefb740000 - 0x000007fefb934000 C:\Windows\WinSxS\amd64_microsoft.windows.common-controls_6595b64144ccf1df_6.0.7600.16661_none_fa62ad231704eab7\COMCTL32.dll
0x000007fefea00000 - 0x000007fefea71000 C:\Windows\system32\SHLWAPI.dll
0x000007fefd730000 - 0x000007fefd75e000 C:\Windows\system32\IMM32.DLL
0x000007fefe760000 - 0x000007fefe869000 C:\Windows\system32\MSCTF.dll
0x0000000073030000 - 0x0000000073101000 C:\Java\jdk1.7.0_02\jre\bin\msvcr100.dll
0x0000000069eb0000 - 0x000000006a58a000 C:\Java\jdk1.7.0_02\jre\bin\server\jvm.dll
0x000007fef97b0000 - 0x000007fef97b9000 C:\Windows\system32\WSOCK32.dll
0x000007fefe880000 - 0x000007fefe8cd000 C:\Windows\system32\WS2_32.dll
0x000007fefe870000 - 0x000007fefe878000 C:\Windows\system32\NSI.dll
0x000007fef97c0000 - 0x000007fef97fb000 C:\Windows\system32\WINMM.dll
0x0000000076f10000 - 0x0000000076f17000 C:\Windows\system32\PSAPI.DLL
0x0000000073c00000 - 0x0000000073c0f000 C:\Java\jdk1.7.0_02\jre\bin\verify.dll
0x0000000073bd0000 - 0x0000000073bf8000 C:\Java\jdk1.7.0_02\jre\bin\java.dll
0x00000000739a0000 - 0x00000000739b5000 C:\Java\jdk1.7.0_02\jre\bin\zip.dll
0x00000000739c0000 - 0x00000000739d9000 C:\java\jdk1.7.0_02\jre\bin\net.dll
0x000007fefc4d0000 - 0x000007fefc524000 C:\Windows\system32\mswsock.dll
0x000007fefc4c0000 - 0x000007fefc4c7000 C:\Windows\System32\wship6.dll
0x0000000073410000 - 0x0000000073421000 C:\java\jdk1.7.0_02\jre\bin\nio.dll
0x0000000073c10000 - 0x0000000073c1b000 C:\java\jdk1.7.0_02\jre\bin\management.dll
0x0000000010000000 - 0x0000000010125000 C:\production\Tomcat-7.0.25\bin\tcnative-1.dll
0x000007fefc530000 - 0x000007fefc547000 C:\Windows\system32\CRYPTSP.dll
0x000007fefc230000 - 0x000007fefc277000 C:\Windows\system32\rsaenh.dll
0x000007fefcb90000 - 0x000007fefcb9f000 C:\Windows\system32\CRYPTBASE.dll
0x0000000072f70000 - 0x0000000072f94000 C:\java\jdk1.7.0_02\jre\bin\sunec.dll
0x000000006de80000 - 0x000000006de8b000 C:\java\jdk1.7.0_02\jre\bin\sunmscapi.dll
0x000007fefcdf0000 - 0x000007fefcf56000 C:\Windows\system32\CRYPT32.dll
0x000007fefcd40000 - 0x000007fefcd4f000 C:\Windows\system32\MSASN1.dll
0x000007fefbed0000 - 0x000007fefbed7000 C:\Windows\System32\wshtcpip.dll
0x000007fefbfe0000 - 0x000007fefbffe000 C:\Windows\system32\USERENV.dll
0x000007fefcca0000 - 0x000007fefccaf000 C:\Windows\system32\profapi.dll
0x000007fefb620000 - 0x000007fefb635000 C:\Windows\system32\NLAapi.dll
0x000007fefc350000 - 0x000007fefc3ab000 C:\Windows\system32\DNSAPI.dll
0x000007fef9780000 - 0x000007fef978b000 C:\Windows\System32\winrnr.dll
0x000007fef9760000 - 0x000007fef9775000 C:\Windows\system32\napinsp.dll
0x000007fefa520000 - 0x000007fefa547000 C:\Windows\system32\IPHLPAPI.DLL
0x000007fefa510000 - 0x000007fefa51b000 C:\Windows\system32\WINNSI.DLL
0x000007fef9200000 - 0x000007fef9208000 C:\Windows\system32\rasadhlp.dll
0x000007fefa360000 - 0x000007fefa3b3000 C:\Windows\System32\fwpuclnt.dll
0x000000006d7f0000 - 0x000000006d820000 C:\java\jdk1.7.0_02\jre\bin\jpeg.dll
0x000000006c680000 - 0x000000006c814000 C:\java\jdk1.7.0_02\jre\bin\awt.dll
0x000007fefec90000 - 0x000007fefed67000 C:\Windows\system32\OLEAUT32.dll
0x000007fefea80000 - 0x000007fefec82000 C:\Windows\system32\ole32.dll
0x000000006d500000 - 0x000000006d5a2000 C:\java\jdk1.7.0_02\jre\bin\mlib_image.dll
0x000007fefaff0000 - 0x000007fefb115000 C:\Windows\system32\DBGHELP.DLL
VM Arguments:
jvm_args: -Xms3072m -Xmx3072m -XX:MaxPermSize=3072m -Djava.util.logging.config.file=C:\production\Tomcat-7.0.25\conf\logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Dlog4j.configuration=logger.properties -Djava.endorsed.dirs=C:\production\Tomcat-7.0.25\endorsed -Dcatalina.base=C:\production\Tomcat-7.0.25 -Dcatalina.home=C:\production\Tomcat-7.0.25 -Djava.io.tmpdir=C:\production\Tomcat-7.0.25\temp
java_command: org.apache.catalina.startup.Bootstrap start
Launcher Type: SUN_STANDARD
Environment Variables:
JAVA_HOME=C:\Java\jdk1.7.0_02
JRE_HOME=C:\Java\jdk1.7.0_02
CLASSPATH=C:\production\Tomcat-7.0.25\bin\bootstrap.jar;C:\production\Tomcat-7.0.25\bin\tomcat-juli.jar
PATH=C:\app\Administrator\product\11.1.0\db_1\bin;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\
USERNAME=***
OS=Windows_NT
PROCESSOR_IDENTIFIER=Intel64 Family 6 Model 15 Stepping 11, GenuineIntel
--------------- S Y S T E M ---------------
OS: Windows NT 6.1 , 64 bit Build 7600
CPU:total 4 (4 cores per cpu, 1 threads per core) family 6 model 15 stepping 11, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3
Memory: 4k page, physical 8387032k(487964k free), swap 16772168k(9441420k free)
vm_info: Java HotSpot(TM) 64-Bit Server VM (22.0-b10) for windows-amd64 JRE (1.7.0_02-b13), built on Nov 17 2011 13:51:15 by "java_re" with unknown MS VC++:1600
time: Fri Mar 16 15:16:36 2012
elapsed time: 11112 seconds
That is a JVM bug in the zip compression. It (or something very like it) has been seen before [1]. Disabling compression is really the only option until Oracle fix the bug. If you have a simple test case that reproduces the issue then it would be worth opening up a bug report.
[1] http://tomcat.markmail.org/thread/ar2r7qop7n76hsgi
I also ran into this with Tomcat 7.0.47 / Java 1.7 and Tomcat 7.0.55 / Java 8
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0xb65d73a4, pid=5872, tid=1834171200
#
# JRE version: Java(TM) SE Runtime Environment (7.0_51-b13) (build 1.7.0_51-b13)
# Java VM: Java HotSpot(TM) Server VM (24.51-b03 mixed mode linux-x86 )
# Problematic frame:
# C [libzip.so+0x83a4] deflate_slow+0x44
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x00007fa80506bdcd, pid=1463, tid=140358692861696
#
# JRE version: Java(TM) SE Runtime Environment (8.0_20-b26) (build 1.8.0_20-b26)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.20-b23 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# C [libzip.so+0x4dcd] deflate_slow+0x3d
It appears to be data dependent, but I didn't have the time to try to figure it out. I "solved" issue this by disabling compression in Tomcat.
I also got crashed tomcat enabling compression.
>java -version
java version "1.6.0_31"
OpenJDK Runtime Environment (IcedTea6 1.13.3) (6b31-1.13.3-1ubuntu1~0.12.04.2)
OpenJDK 64-Bit Server VM (build 23.25-b01, mixed mode)
con/server.xml
<Connector URIEncoding="UTF-8" port="80" protocol="HTTP/1.1"
connectionTimeout="20000"
redirectPort="8443"
compression="on"
compressionMinSize="2048"
noCompressionUserAgents="gozilla, traviata"
compressableMimeType="text/html,text/xml,text/plain,text/javascript,text/css,application/javascript"
/>
crash report.
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x00007fa902990c14, pid=10845, tid=140362698929920
#
# JRE version: 6.0_31-b31
# Java VM: OpenJDK 64-Bit Server VM (23.25-b01 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# C [libz.so.1+0x4c14] uncompress+0x774
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# If you would like to submit a bug report, please include
# instructions how to reproduce the bug and visit:
# https://bugs.launchpad.net/ubuntu/+source/openjdk-6/
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#
--------------- T H R E A D ---------------
Current thread (0x00007fa844010800): JavaThread "http-bio-80-exec-87" daemon [_thread_in_native, id=27442, stack(0x00007fa8bcbf3000,0x00007fa8bccf4000)]
siginfo:si_signo=SIGSEGV: si_errno=0, si_code=2 (SEGV_ACCERR), si_addr=0x00007fa7c0055000
...........
...........
Stack: [0x00007fa8bcbf3000,0x00007fa8bccf4000], sp=0x00007fa8bccf0110, free space=1012k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C [libz.so.1+0x4c14] uncompress+0x774
Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
J java.util.zip.Deflater.deflateBytes(J[BII)I
J java.util.zip.Deflater.deflate([BII)I
j java.util.zip.GZIPOutputStream.finish()V+41
j org.apache.coyote.http11.filters.FlushableGZIPOutputStream.finish()V+9
j org.apache.coyote.http11.filters.GzipOutputFilter.end()J+26
j org.apache.coyote.http11.AbstractOutputBuffer.endRequest()V+43
j org.apache.coyote.http11.InternalOutputBuffer.endRequest()V+1
j org.apache.coyote.http11.AbstractHttp11Processor.action(Lorg/apache/coyote/ActionCode;Ljava/lang/Object;)V+124
j org.apache.coyote.Response.action(Lorg/apache/coyote/ActionCode;Ljava/lang/Object;)V+31
j org.apache.coyote.Response.finish()V+5
j org.apache.catalina.connector.OutputBuffer.close()V+138
j org.apache.catalina.connector.CoyoteWriter.close()V+4
j org.apache.catalina.core.ApplicationDispatcher.doForward(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V+467
j org.apache.catalina.core.ApplicationDispatcher.forward(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V+55
j org.springframework.web.servlet.view.InternalResourceView.renderMergedOutputModel(Ljava/util/Map;Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V+244
j org.springframework.web.servlet.view.AbstractView.render(Ljava/util/Map;Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V+157
j org.springframework.web.servlet.DispatcherServlet.render(Lorg/springframework/web/servlet/ModelAndView;Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V+225
j org.springframework.web.servlet.DispatcherServlet.doDispatch(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V+388
j org.springframework.web.servlet.DispatcherServlet.doService(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V+219
j org.springframework.web.servlet.FrameworkServlet.processRequest(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V+10
j org.springframework.web.servlet.FrameworkServlet.doGet(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V+3
j javax.servlet.http.HttpServlet.service(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V+35
j javax.servlet.http.HttpServlet.service(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V+30
j org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V+446
j org.apache.catalina.core.ApplicationFilterChain.doFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V+101
j org.displaytag.filter.ResponseOverrideFilter.doFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;Ljavax/servlet/FilterChain;)V+40
j org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V+151
j org.apache.catalina.core.ApplicationFilterChain.doFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V+101
j org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V+76
j org.springframework.security.intercept.web.FilterSecurityInterceptor.invoke(Lorg/springframework/security/intercept/web/FilterInvocation;)V+87
j org.springframework.security.intercept.web.FilterSecurityInterceptor.doFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;Ljavax/servlet/FilterChain;)V+15
j org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V+199
j org.springframework.security.ui.SessionFixationProtectionFilter.doFilterHttp(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;Ljavax/servlet/FilterChain;)V+93
j org.springframework.security.ui.SpringSecurityFilter.doFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;Ljavax/servlet/FilterChain;)V+44
j org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V+199
j org.springframework.security.ui.ExceptionTranslationFilter.doFilterHttp(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;Ljavax/servlet/FilterChain;)V+3
j org.springframework.security.ui.SpringSecurityFilter.doFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;Ljavax/servlet/FilterChain;)V+44
j org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V+199
j org.springframework.security.providers.anonymous.AnonymousProcessingFilter.doFilterHttp(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;Ljavax/servlet/FilterChain;)V+148
j org.springframework.security.ui.SpringSecurityFilter.doFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;Ljavax/servlet/FilterChain;)V+44
j org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V+199
j org.springframework.security.ui.AbstractProcessingFilter.doFilterHttp(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;Ljavax/servlet/FilterChain;)V+86
j org.joget.apps.workflow.security.WorkflowHttpAuthProcessingFilter.doFilterHttp(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;Ljavax/servlet/FilterChain;)V+49
j org.springframework.security.ui.SpringSecurityFilter.doFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;Ljavax/servlet/FilterChain;)V+44
j org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V+199
j org.joget.apps.workflow.controller.WorkflowBasicProcessingFilter.doFilterHttp(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;Ljavax/servlet/FilterChain;)V+532
j org.springframework.security.ui.SpringSecurityFilter.doFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;Ljavax/servlet/FilterChain;)V+44
j org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V+199
j org.springframework.security.ui.logout.LogoutFilter.doFilterHttp(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;Ljavax/servlet/FilterChain;)V+120
j org.springframework.security.ui.SpringSecurityFilter.doFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;Ljavax/servlet/FilterChain;)V+44
j org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V+199
j org.springframework.security.context.HttpSessionContextIntegrationFilter.doFilterHttp(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;Ljavax/servlet/FilterChain;)V+180
j org.springframework.security.ui.SpringSecurityFilter.doFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;Ljavax/servlet/FilterChain;)V+44
j org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V+199
j org.springframework.security.concurrent.ConcurrentSessionFilter.doFilterHttp(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;Ljavax/servlet/FilterChain;)V+134
j org.springframework.security.ui.SpringSecurityFilter.doFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;Ljavax/servlet/FilterChain;)V+44
j org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V+199
j org.springframework.security.util.FilterChainProxy.doFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;Ljavax/servlet/FilterChain;)V+165
j org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(Ljavax/servlet/Filter;Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;Ljavax/servlet/FilterChain;)V+5
j org.springframework.web.filter.DelegatingFilterProxy.doFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;Ljavax/servlet/FilterChain;)V+75
j org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V+151
j org.apache.catalina.core.ApplicationFilterChain.doFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V+101
j org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;Ljavax/servlet/FilterChain;)V+59
j org.springframework.web.filter.OncePerRequestFilter.doFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;Ljavax/servlet/FilterChain;)V+90
j org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V+151
j org.apache.catalina.core.ApplicationFilterChain.doFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V+101
j org.apache.catalina.core.StandardWrapperValve.invoke(Lorg/apache/catalina/connector/Request;Lorg/apache/catalina/connector/Response;)V+783
j org.apache.catalina.core.StandardContextValve.invoke(Lorg/apache/catalina/connector/Request;Lorg/apache/catalina/connector/Response;)V+166
j org.apache.catalina.authenticator.AuthenticatorBase.invoke(Lorg/apache/catalina/connector/Request;Lorg/apache/catalina/connector/Response;)V+449
j org.apache.catalina.core.StandardHostValve.invoke(Lorg/apache/catalina/connector/Request;Lorg/apache/catalina/connector/Response;)V+158
j org.apache.catalina.valves.ErrorReportValve.invoke(Lorg/apache/catalina/connector/Request;Lorg/apache/catalina/connector/Response;)V+6
j org.apache.catalina.core.StandardEngineValve.invoke(Lorg/apache/catalina/connector/Request;Lorg/apache/catalina/connector/Response;)V+71
j org.apache.catalina.connector.CoyoteAdapter.service(Lorg/apache/coyote/Request;Lorg/apache/coyote/Response;)V+196
j org.apache.coyote.http11.AbstractHttp11Processor.process(Lorg/apache/tomcat/util/net/SocketWrapper;)Lorg/apache/tomcat/util/net/AbstractEndpoint$Handler$SocketState;+632
j org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(Lorg/apache/tomcat/util/net/SocketWrapper;Lorg/apache/tomcat/util/net/SocketStatus;)Lorg/apache/tomcat/util/net/AbstractEndpoint$Handler$SocketState;+270
j org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run()V+130
J java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V
j java.util.concurrent.ThreadPoolExecutor$Worker.run()V+5
j org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run()V+4
j java.lang.Thread.run()V+11
v ~StubRoutines::call_stub
Related
kdump no crash log in centos 7
rpm -qa | grep kexec kexec-tools-2.0.15-13.el7.x86_64 cat /proc/cmdline BOOT_IMAGE=/vmlinuz-3.10.0-862.el7.x86_64 root=/dev/mapper/centos-root ro crashkernel=auto rd.lvm.lv=centos/root rd.lvm.lv=centos/swap rhgb quiet dmesg | grep -i crash [ 0.000000] Command line: BOOT_IMAGE=/vmlinuz-3.10.0-862.el7.x86_64 root=/dev/mapper/centos-root ro crashkernel=auto rd.lvm.lv=centos/root rd.lvm.lv=centos/swap rhgb quiet [ 0.000000] Reserving 161MB of memory at 688MB for crashkernel (System RAM: 16383MB) [ 0.000000] Kernel command line: BOOT_IMAGE=/vmlinuz-3.10.0-862.el7.x86_64 root=/dev/mapper/centos-root ro crashkernel=auto rd.lvm.lv=centos/root rd.lvm.lv=centos/swap rhgb quiet [ 1.033253] crash memory driver: version 1.1 grep -v ^# /etc/kdump.conf path /var/crash core_collector makedumpfile -l --message-level 1 -d 31 systemctl status kdump ● kdump.service - Crash recovery kernel arming Loaded: loaded (/usr/lib/systemd/system/kdump.service; enabled; vendor preset: enabled) Active: active (exited) since Fri 2022-08-26 07:34:03 CST; 1h 34min ago Process: 996 ExecStart=/usr/bin/kdumpctl start (code=exited, status=0/SUCCESS) Main PID: 996 (code=exited, status=0/SUCCESS) CGroup: /system.slice/kdump.service Aug 26 07:34:01 c-1 systemd[1]: Starting Crash recovery kernel arming... Aug 26 07:34:03 c-1 kdumpctl[996]: kexec: loaded kdump kernel Aug 26 07:34:03 c-1 kdumpctl[996]: Starting kdump: [OK] Aug 26 07:34:03 c-1 systemd[1]: Started Crash recovery kernel arming. kdumpctl status Kdump is operational everything looks fine,but when echo c > /proc/sysrq-trigger to test function,it doesn't work.plz help,thanks!
rabbitmq-server doesn't start (status=64)
Running sudo service rabbitmq-server start results in the following output: Job for rabbitmq-server.service failed because the control process exited with error code. See "systemctl status rabbitmq-server.service" and "journalctl -xe" for details. systemctl status output for rabbitmq-server.service is as follows: ● rabbitmq-server.service - RabbitMQ broker Loaded: loaded (/lib/systemd/system/rabbitmq-server.service; enabled; vendor preset: enabled) Active: activating (auto-restart) (Result: exit-code) since Wed 2019-01-30 13:45:25 +08; 4s ago Process: 32710 ExecStop=/usr/lib/rabbitmq/bin/rabbitmqctl shutdown (code=exited, status=69) Process: 6836 ExecStart=/usr/lib/rabbitmq/bin/rabbitmq-server (code=exited, status=64) Main PID: 6836 (code=exited, status=64) Jan 30 13:45:25 ip-172-31-1-92 systemd[1]: rabbitmq-server.service: Main process exited, code=exited, status=64/n/a Jan 30 13:45:25 ip-172-31-1-92 systemd[1]: rabbitmq-server.service: Failed with result 'exit-code'. Jan 30 13:45:25 ip-172-31-1-92 systemd[1]: Failed to start RabbitMQ broker.
Turns out the problem was I had set the environment variable RABBITMQ_CONFIG_FILE to /etc/rabbitmq/rabbitmq.config but created that file as rabbitmq.conf. The server crashed as a result trying to look for a file that didn't exist.
redis-cli unix socket - import rcmd failing
I am using redis server 3.0.6 on ubuntu 16.04 desktop. Installed using apt-get install redis-server It was working fine for a long time. Today, when I tried to import a rcmd file using unix socket in redis-cli I am getting out of memory error. $ sudo redis-cli -s /run/redis/redis.sock < a1.rcmd Error: Out of memory a1.rcmd contents: SET Api:Server:Name "server1" /etc/redis/redis.conf was confiured with /run/redis/redis.sock. All other configurations are default. GET, SET commands with unix socket option are working fine. If I run without unix socket, the redis-cli < a1.rcmd command is working fine. What causes this error? Will upgrade to latest version of redis resolve this? I dont see apt-get way to install latest redis version 5. Any help would be really appreciated. ** EDIT ** Some more inputs and GDB logs. $ which redis-server /usr/bin/redis-server $ sudo service redis-server status ● redis-server.service - Advanced key-value store Loaded: loaded (/lib/systemd/system/redis-server.service; enabled; vendor preset: enabled) Active: active (running) since Mon 2018-12-17 11:29:14 EST; 16min ago Docs: http://redis.io/documentation, man:redis-server(1) Process: 8986 ExecStopPost=/bin/run-parts --verbose /etc/redis/redis-server.post-down.d (code=exited, status=0/SUCCESS) Process: 8981 ExecStop=/bin/kill -s TERM $MAINPID (code=exited, status=0/SUCCESS) Process: 8977 ExecStop=/bin/run-parts --verbose /etc/redis/redis-server.pre-down.d (code=exited, status=0/SUCCESS) Process: 9001 ExecStartPost=/bin/run-parts --verbose /etc/redis/redis-server.post-up.d (code=exited, status=0/SUCCESS) Process: 8997 ExecStart=/usr/bin/redis-server /etc/redis/redis.conf (code=exited, status=0/SUCCESS) Process: 8992 ExecStartPre=/bin/run-parts --verbose /etc/redis/redis-server.pre-up.d (code=exited, status=0/SUCCESS) Main PID: 9000 (redis-server) Tasks: 3 Memory: 884.0K CPU: 304ms CGroup: /system.slice/redis-server.service └─9000 /usr/bin/redis-server 127.0.0.1:637 $ sudo gdb /usr/bin/redis-server 9000 GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.5) 7.11.1 Copyright (C) 2016 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>. For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from /usr/bin/redis-server...(no debugging symbols found)...done. Attaching to program: /usr/bin/redis-server, process 9000 [New LWP 9002] [New LWP 9003] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". 0x00007fe8f5158a13 in epoll_wait () at ../sysdeps/unix/syscall-template.S:84 84 ../sysdeps/unix/syscall-template.S: No such file or directory. (gdb) continue Continuing. After this step I ran following command in another terminal: $ sudo redis-cli -s /run/redis/redis.sock < a1.rcmd Segmentation fault (core dumped) (gdb) bt #0 0x00007fe8f5158a13 in epoll_wait () at ../sysdeps/unix/syscall-template.S:84 #1 0x000055b238abe15f in aeProcessEvents () #2 0x000055b238abe58b in aeMain () #3 0x000055b238abd204 in main () (gdb) info registers rax 0xfffffffffffffffc -4 rbx 0x7fe8f48e5300 140638512108288 rcx 0x7fe8f5158a13 140638520969747 rdx 0x1060 4192 rsi 0x7fe8f4910000 140638512283648 rdi 0x3 3 rbp 0x7fe8f487b068 0x7fe8f487b068 rsp 0x7ffd9a5ca920 0x7ffd9a5ca920 r8 0x284 644 r9 0x284 644 r10 0x64 100 r11 0x293 659 r12 0x7ffd9a5caae8 140727193217768 r13 0x7fe8f480e1a0 140638511227296 r14 0x7fe8f48101f0 140638511235568 r15 0xfffffffe 4294967294 rip 0x7fe8f5158a13 0x7fe8f5158a13 <epoll_wait+51> eflags 0x293 [ CF AF SF IF ] cs 0x33 51 ss 0x2b 43 ds 0x0 0 es 0x0 0 fs 0x0 0 gs 0x0 0 (gdb)
JVM crash with SIGSEGV (0xb) at pc=0x0000000000090000
We faced an issue on server where the JVM crashed and seems like most threads been in blocked state. I dont think it is GC which is causing it since also the GC daemon is blocked. From the GC info, does not seems like memory issue.. and there are no errors in the /var/log/messages file. Unfortunately, even when it is saying that a core dump written, there is no core file on the server. no storage issue or so. Any help is more than welcome :) I will paste the hs_err_pid file below. Thanks
# # A fatal error has been detected by the Java Runtime Environment: # # SIGSEGV (0xb) at pc=0x0000000000090000, pid=10320, tid=140491055642368 # # JRE version: Java(TM) SE Runtime Environment (8.0_66-b17) (build 1.8.0_66-b17) # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.66-b17 mixed mode linux-amd64 ) # Problematic frame: # C 0x0000000000090000 # # Core dump written. Default location: /opt/CSCOlumos/core or core.10320 # # If you would like to submit a bug report, please visit: # http://bugreport.java.com/bugreport/crash.jsp # --------------- T H R E A D --------------- Current thread (0x00007fc5a0f60000): JavaThread "http-nio-443-exec-39" daemon [_thread_in_Java, id=1520, stack(0x00007fc69f6f3000,0x00007fc69f774000)] siginfo: si_signo: 11 (SIGSEGV), si_code: 1 (SEGV_MAPERR), si_addr: 0x0000000000090000 Registers: RAX=0x00007fc754276be0, RBX=0x00007fc5ef526f68, RCX=0x0000000000000006, RDX=0x00007fcb0ea9e188 RSP=0x00007fc69f770bc0, RBP=0x0000000000000007, RSI=0x00007fcb0ea9e188, RDI=0x00007fc69f770b40 R8 =0x0000000000090000, R9 =0x00007fcb0358d000, R10=0x0000000000000000, R11=0x00007fcb18a69600 R12=0x00007fc5b9f27f20, R13=0x00007fc754276be0, R14=0x00007fc69f770820, R15=0x00007fc5a0f60000 RIP=0x0000000000090000, EFLAGS=0x0000000000010206, CSGSFS=0x0000000000000033, ERR=0x0000000000000014 TRAPNO=0x000000000000000e Top of Stack: (sp=0x00007fc69f770bc0) 0x00007fc69f770bc0: 00007fc754274938 00007fc7542727d8 0x00007fc69f770bd0: 00007fc7542727d8 00007fcb18c6e1e9 0x00007fc69f770be0: 00007fc5a0f60000 00007fcb0d642ef8 0x00007fc69f770bf0: 00007fc5a0f60000 00007fcb035949b0 0x00007fc69f770c00: 00007fc69f770c68 00007fcb03594670 0x00007fc69f770c10: 00007fcb03594670 00007fc754274938 0x00007fc69f770c20: 00007fc7542727d8 00007fc69f770c28 0x00007fc69f770c30: 00007fc5daf2d079 00007fc69f770c98 0x00007fc69f770c40: 00007fc5daf2d838 00007fc5b7d11e80 0x00007fc69f770c50: 00007fc5daf2d108 00007fc69f770c18 0x00007fc69f770c60: 00007fc69f770c90 00007fc69f770ce0 0x00007fc69f770c70: 00007fcb035949b0 0000000000000000 0x00007fc69f770c80: 00007fc754274938 00007fc7542748e0 0x00007fc69f770c90: 0000000000001000 00007fc7542727d8 0x00007fc69f770ca0: 00007fc69f770ca0 00007fc5daf2cecc 0x00007fc69f770cb0: 00007fc69f770d10 00007fc5daf2d838 0x00007fc69f770cc0: 00007fc5b7d11ce8 00007fc5daf2cf20 0x00007fc69f770cd0: 00007fc69f770c90 00007fc69f770d00 0x00007fc69f770ce0: 00007fc69f770d58 00007fcb035949b0 0x00007fc69f770cf0: 00007fc7542727d8 0000000000000000 0x00007fc69f770d00: 00007fc84956a4b8 0000000000001000 0x00007fc69f770d10: 00007fc754251018 00007fc69f770d18 0x00007fc69f770d20: 00007fc5daf2cd35 00007fc69f770d70 0x00007fc69f770d30: 00007fc5daf2d838 00007fc5b7d123e0 0x00007fc69f770d40: 00007fc5daf2cd60 00007fc69f770d00 0x00007fc69f770d50: 00007fc69f770d68 00007fc69f770db8 0x00007fc69f770d60: 00007fcb035949b0 00007fc84956a4b8 0x00007fc69f770d70: 00007fc754251018 00007fc69f770d78 0x00007fc69f770d80: 00007fc5f4ff6819 00007fc69f770df8 0x00007fc69f770d90: 00007fc5f4ff6e18 00007fc5b7d11aa8 0x00007fc69f770da0: 00007fc5f4ff6888 00007fc69f770d68 0x00007fc69f770db0: 00007fc69f770e00 00007fc75426fd60 Instructions: (pc=0x0000000000090000) 0x000000000008ffe0: [error occurred during error reporting (printing registers, top of stack, instructions near pc), id 0xb] Register to memory mapping: RAX= [error occurred during error reporting (printing register info), id 0xb] Stack: [0x00007fc69f6f3000,0x00007fc69f774000], sp=0x00007fc69f770bc0, free space=502k Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code) C 0x0000000000090000
--------------- P R O C E S S --------------- Java Threads: ( => current thread ) 0x00007fcaf4cba000 JavaThread "http-nio-443-exec-49" daemon [_thread_blocked, id=6679, stack(0x00007fc5d2bdc000,0x00007fc5d2c5d000)] 0x00007fcafdac8800 JavaThread "http-nio-443-exec-48" daemon [_thread_blocked, id=6652, stack(0x00007fc5eca18000,0x00007fc5eca99000)] 0x00007fcaf4bfd000 JavaThread "http-nio-443-exec-47" daemon [_thread_blocked, id=6651, stack(0x00007fc5ed3ab000,0x00007fc5ed42c000)] 0x00000000051ac000 JavaThread "seqtaskexecutor-29735" [_thread_blocked, id=6338, stack(0x00007fc5bc73c000,0x00007fc5bc7bd000)] 0x00007fcaf5299800 JavaThread "seqtaskexecutor-29735" [_thread_blocked, id=6337, stack(0x00007fc5ef870000,0x00007fc5ef8f1000)] 0x00007fc6c1a3c800 JavaThread "seqtaskexecutor-29735" [_thread_blocked, id=6336, stack(0x00007fc5ca167000,0x00007fc5ca1e8000)] 0x00007fc57cd80800 JavaThread "seqtaskexecutor-29735" [_thread_blocked, id=6329, stack(0x00007fc5d2ada000,0x00007fc5d2b5b000)] 0x00007fc6c1acf000 JavaThread "seqtaskexecutor-29735" [_thread_blocked, id=6328, stack(0x00007fc69ebd9000,0x00007fc69ec5a000)] 0x00007fc6c112b000 JavaThread "seqtaskexecutor-29735" [_thread_blocked, id=6327, stack(0x00007fc5e1731000,0x00007fc5e17b2000)] 0x00007fc5a0e34800 JavaThread "seqtaskexecutor-29735" [_thread_blocked, id=6326, stack(0x00007fc5e1833000,0x00007fc5e18b4000)] 0x00007fc5a5eac000 JavaThread "seqtaskexecutor-29735" [_thread_blocked, id=6325, stack(0x00007fc5bcd4c000,0x00007fc5bcdcd000)] 0x0000000007616800 JavaThread "seqtaskexecutor-29735" [_thread_blocked, id=6324, stack(0x00007fc5ebe00000,0x00007fc5ebe81000)] 0x0000000001b5c800 JavaThread "seqtaskexecutor-29735" [_thread_blocked, id=6323, stack(0x00007fc5d9803000,0x00007fc5d9884000)] 0x00007fc5a5e68000 JavaThread "seqtaskexecutor-29735" [_thread_blocked, id=6322, stack(0x00007fc69f351000,0x00007fc69f3d2000)] 0x00007fcaf5d5e800 JavaThread "seqtaskexecutor-29735" [_thread_blocked, id=6321, stack(0x00007fc5bb1c2000,0x00007fc5bb243000)] 0x0000000005d12000 JavaThread "seqtaskexecutor-29735" [_thread_blocked, id=6320, stack(0x00007fc5bbf30000,0x00007fc5bbfb1000)] 0x0000000002b45000 JavaThread "seqtaskexecutor-29735" [_thread_blocked, id=6319, stack(0x00007fc5d18b7000,0x00007fc5d1938000)] 0x00000000015f6000 JavaThread "seqtaskexecutor-29735" [_thread_blocked, id=6318, stack(0x00007fc69e9d5000,0x00007fc69ea56000)] 0x00007fc5d404b800 JavaThread "seqtaskexecutor-29735" [_thread_blocked, id=6317, stack(0x00007fc5b9a1e000,0x00007fc5b9a9f000)] 0x0000000001f93000 JavaThread "seqtaskexecutor-29735" [_thread_blocked, id=6316, stack(0x00007fc5b8171000,0x00007fc5b81f2000)] 0x00000000040c7000 JavaThread "WebSocket background processing" daemon [_thread_blocked, id=6315, stack(0x00007fc5bb2c4000,0x00007fc5bb345000)] 0x00007fcafe0d2800 JavaThread "seqtaskexecutor-29735" [_thread_blocked, id=6314, stack(0x00007fc5ebafa000,0x00007fc5ebb7b000)] 0x00007fcafd073000 JavaThread "seqtaskexecutor-29735" [_thread_blocked, id=6313, stack(0x00007fc69eb58000,0x00007fc69ebd9000)] 0x00007fcafc535000 JavaThread "seqtaskexecutor-2023" [_thread_blocked, id=6312, stack(0x00007fc5e4a9b000,0x00007fc5e4b1c000)] 0x0000000000a0a000 JavaThread "seqtaskexecutor-29735" [_thread_blocked, id=6311, stack(0x00007fc5c9eef000,0x00007fc5c9f70000)] 0x00007fcaf4562000 JavaThread "seqtaskexecutor-29735" [_thread_blocked, id=6310, stack(0x00007fc69f1ec000,0x00007fc69f26d000)] 0x00007fc6c3e3a000 JavaThread "seqtaskexecutor-29735" [_thread_blocked, id=6309, stack(0x00007fc68eb8a000,0x00007fc68ec0b000)] 0x00007fcaf408f000 JavaThread "seqtaskexecutor-29735" [_thread_blocked, id=6308, stack(0x00007fc5ed932000,0x00007fc5ed9b3000)] 0x0000000003518800 JavaThread "seqtaskexecutor-29735" [_thread_blocked, id=6307, stack(0x00007fc5e1b39000,0x00007fc5e1bba000)] 0x00007fcaf6414000 JavaThread "seqtaskexecutor-29735" [_thread_blocked, id=6306, stack(0x00007fc5ca6ff000,0x00007fc5ca780000)] 0x00007fc6c2d72000 JavaThread "seqtaskexecutor-29735" [_thread_blocked, id=6305, stack(0x00007fc5cd9a2000,0x00007fc5cda23000)] 0x00007fcafdb80800 JavaThread "seqtaskexecutor-29735" [_thread_blocked, id=6303, stack(0x00007fc5ea1c8000,0x00007fc5ea249000)] 0x00007fc6c0b93800 JavaThread "seqtaskexecutor-29735" [_thread_blocked, id=6302, stack(0x00007fc5e2cdc000,0x00007fc5e2d5d000)] 0x00007fc6c3f14000 JavaThread "seqtaskexecutor-29735" [_thread_blocked, id=6301, stack(0x00007fc5cd71d000,0x00007fc5cd79e000)] 0x00007fc5a2a76000 JavaThread "seqtaskexecutor-29735" [_thread_blocked, id=6300, stack(0x00007fc5bb924000,0x00007fc5bb9a5000)] 0x00007fcafeaf6000 JavaThread "seqtaskexecutor-29735" [_thread_blocked, id=6299, stack(0x00007fc5f0688000,0x00007fc5f0709000)] 0x00000000068c9000 JavaThread "seqtaskexecutor-29735" [_thread_blocked, id=6298, stack(0x00007fc5c9ff1000,0x00007fc5ca072000)] 0x00000000073ee000 JavaThread "seqtaskexecutor-29735" [_thread_blocked, id=6297, stack(0x00007fc5d25ac000,0x00007fc5d262d000)] 0x00007fcafca4e800 JavaThread "seqtaskexecutor-29735" [_thread_blocked, id=6296, stack(0x00007fc69e8e2000,0x00007fc69e963000)] 0x00007fc6992e1800 JavaThread "seqtaskexecutor-29735" [_thread_blocked, id=6295, stack(0x00007fc5e2e5f000,0x00007fc5e2ee0000)] 0x0000000004417800 JavaThread "seqtaskexecutor-29735" [_thread_blocked, id=6294, stack(0x00007fc5bbe2e000,0x00007fc5bbeaf000)] 0x00000000096a1000 JavaThread "seqtaskexecutor-29735" [_thread_blocked, id=6293, stack(0x00007fc5bb3c6000,0x00007fc5bb447000)] 0x0000000001b6e000 JavaThread "seqtaskexecutor-29736" [_thread_blocked, id=6292, stack(0x00007fc5e1e3f000,0x00007fc5e1ec0000)] 0x00007fcafea3b800 JavaThread "seqtaskexecutor-29736" [_thread_blocked, id=6291, stack(0x00007fc5d2046000,0x00007fc5d20c7000)] 0x00000000084f4000 JavaThread "seqtaskexecutor-29737" [_thread_blocked, id=6290, stack(0x00007fc5ccf8e000,0x00007fc5cd00f000)] 0x00007fcafc785000 JavaThread "seqtaskexecutor-15029" [_thread_blocked, id=6289, stack(0x00007fc68b3ae000,0x00007fc68b42f000)] 0x00007fc5a5eff800 JavaThread "seqtaskexecutor-6045" [_thread_blocked, id=6285, stack(0x00007fc5d14b3000,0x00007fc5d1534000)] 0x00007fc597cb3800 JavaThread "seqtaskexecutor-6045" [_thread_blocked, id=6284, stack(0x00007fc5ee6d1000,0x00007fc5ee752000)] 0x0000000005c90800 JavaThread "seqtaskexecutor-6045" [_thread_blocked, id=6283, stack(0x00007fc5bbeaf000,0x00007fc5bbf30000)] 0x00007fc5a1a88800 JavaThread "seqtaskexecutor-6045" [_thread_blocked, id=6282, stack(0x00007fc5e1ec0000,0x00007fc5e1f41000)] 0x00007fc672b3c800 JavaThread "seqtaskexecutor-6045" [_thread_blocked, id=6281, stack(0x00007fc5e16b0000,0x00007fc5e1731000)] 0x00007fc69964e800 JavaThread "seqtaskexecutor-6045" [_thread_blocked, id=6280, stack(0x00007fc5d272f000,0x00007fc5d27b0000)] 0x0000000000bb1000 JavaThread "seqtaskexecutor-6045" [_thread_blocked, id=6279, stack(0x00007fc590343000,0x00007fc5903c4000)] 0x00007fcaf7135000 JavaThread "seqtaskexecutor-6045" [_thread_blocked, id=6278, stack(0x00007fc5bad39000,0x00007fc5badba000)] 0x00007fc6c0b05800 JavaThread "seqtaskexecutor-6045" [_thread_blocked, id=6277, stack(0x00007fc5e2853000,0x00007fc5e28d4000)] 0x00007fc57d5dd000 JavaThread "seqtaskexecutor-6045" [_thread_blocked, id=6276, stack(0x00007fc69ee5c000,0x00007fc69eedd000)] 0x00007fc6c3c8b000 JavaThread "seqtaskexecutor-6045" [_thread_blocked, id=6275, stack(0x00007fc5ccc88000,0x00007fc5ccd09000)] 0x00007fc59d124000 JavaThread "seqtaskexecutor-6045" [_thread_blocked, id=6274, stack(0x00007fc69fb70000,0x00007fc69fbf1000)] 0x00007fcaf6a93000 JavaThread "seqtaskexecutor-6045" [_thread_blocked, id=6246, stack(0x00007fc69f3d2000,0x00007fc69f453000)] 0x0000000001248000 JavaThread "ActiveMQ BrokerService[ServiceDiscoveryNotificationBroker] Task-50495" daemon [_thread_blocked, id=6245, stack(0x00007fc5e3b7d000,0x00007fc5e3bfe000)] 0x00000000097d8000 JavaThread "ActiveMQ BrokerService[XMP_Message_Broker] Task-26689" daemon [_thread_blocked, id=6244, stack(0x00007fc5bb03f000,0x00007fc5bb0c0000)] 0x00007fcafc700800 JavaThread "seqtaskexecutor-6045" [_thread_blocked, id=6243, stack(0x00007fc69ea56000,0x00007fc69ead7000)] 0x0000000000b98800 JavaThread "seqtaskexecutor-6045" [_thread_blocked, id=6242, stack(0x00007fc5ca67e000,0x00007fc5ca6ff000)] 0x0000000004254000 JavaThread "seqtaskexecutor-6045" [_thread_blocked, id=6241, stack(0x00007fc5bb8a3000,0x00007fc5bb924000)] 0x00007fc5a63d6000 JavaThread "seqtaskexecutor-6045" [_thread_blocked, id=6240, stack(0x00007fc5df36e000,0x00007fc5df3ef000)] 0x00007fcaf5be3800 JavaThread "seqtaskexecutor-6045" [_thread_blocked, id=6239, stack(0x00007fc5bcccb000,0x00007fc5bcd4c000)] 0x0000000006274000 JavaThread "seqtaskexecutor-6045" [_thread_blocked, id=6238, stack(0x00007fc5d12af000,0x00007fc5d1330000)] 0x0000000001033000 JavaThread "seqtaskexecutor-6045" [_thread_blocked, id=6237, stack(0x00007fc5d9884000,0x00007fc5d9905000)] 0x0000000002240800 JavaThread "seqtaskexecutor-6045" [_thread_blocked, id=6236, stack(0x00007fc5f6206000,0x00007fc5f6287000)] 0x00007fc5d5abe800 JavaThread "seqtaskexecutor-6045" [_thread_blocked, id=6235, stack(0x00007fc5bb141000,0x00007fc5bb1c2000)] 0x0000000001a90000 JavaThread "seqtaskexecutor-6045" [_thread_blocked, id=6234, stack(0x00007fc69faef000,0x00007fc69fb70000)] 0x00007fcafe8a3000 JavaThread "seqtaskexecutor-6045" [_thread_blocked, id=6233, stack(0x00007fc677dfb000,0x00007fc677e7c000)] 0x00007fcafc880800 JavaThread "seqtaskexecutor-6045" [_thread_blocked, id=6232, stack(0x00007fc5bb69f000,0x00007fc5bb720000)] 0x0000000001cda800 JavaThread "ActiveMQ BrokerService[ServiceDiscoveryNotificationBroker] Task-50494" daemon [_thread_blocked, id=6087, stack(0x00007fc5cce8c000,0x00007fc5ccf0d000)] 0x000000000651d000 JavaThread "ActiveMQ BrokerService[ServiceDiscoveryNotificationBroker] Task-50491" daemon [_thread_blocked, id=5690, stack(0x00007fc5da640000,0x00007fc5da6c1000)] 0x00000000072f3000 JavaThread "ActiveMQ BrokerService[ServiceDiscoveryNotificationBroker] Task-50489" daemon [_thread_blocked, id=5454, stack(0x00007fc5e2dde000,0x00007fc5e2e5f000)] 0x0000000001fc3000 JavaThread "ActiveMQ BrokerService[ServiceDiscoveryNotificationBroker] Task-50487" daemon [_thread_blocked, id=5292, stack(0x00007fc69f5b8000,0x00007fc69f639000)] 0x0000000000b52800 JavaThread "ActiveMQ BrokerService[ServiceDiscoveryNotificationBroker] Task-50468" daemon [_thread_blocked, id=1532, stack(0x00007fc69f97b000,0x00007fc69f9fc000)] 0x00007fcaf4d7a000 JavaThread "http-nio-443-exec-46" daemon [_thread_blocked, id=28551, stack(0x00007fc5d9e90000,0x00007fc5d9f11000)] 0x00007fcaf61a5800 JavaThread "http-nio-443-exec-44" daemon [_thread_blocked, id=28550, stack(0x00007fc69fd80000,0x00007fc69fe01000)] 0x00007fc6c1596000 JavaThread "ActiveMQ Transport: ssl:///127.0.0.1:15452" daemon [_thread_in_native, id=4056, stack(0x00007fc5bd2d3000,0x00007fc5bd354000)] 0x00007fc698b9d000 JavaThread "ActiveMQ Transport: ssl://localhost/127.0.0.1:61617" [_thread_in_native, id=4055, stack(0x00007fc5b7752000,0x00007fc5b77d3000)] 0x00007fcaf5f8b800 JavaThread "TQ-Publisher - 6" [_thread_blocked, id=24797, stack(0x00007fc5ccd09000,0x00007fc5ccd8a000)] 0x0000000006615800 JavaThread "allTriggeredJobsExecutor-26" [_thread_blocked, id=722, stack(0x00007fc5e1ab8000,0x00007fc5e1b39000)] 0x00007fcafe822800 JavaThread "http-nio-443-exec-42" daemon [_thread_blocked, id=6188, stack(0x00007fc5cd417000,0x00007fc5cd498000)] =>0x00007fc5a0f60000 JavaThread "http-nio-443-exec-39" daemon [_thread_in_Java, id=1520, stack(0x00007fc69f6f3000,0x00007fc69f774000)] 0x00007fcaf575a800 JavaThread "http-nio-443-exec-36" daemon [_thread_blocked, id=7400, stack(0x00007fc593265000,0x00007fc5932e6000)] 0x00007fcafeffb800 JavaThread "http-nio-443-exec-35" daemon [_thread_blocked, id=7340, stack(0x00007fc5b48f2000,0x00007fc5b4973000)] 0x00007fc5a2b04000 JavaThread "pool-68-thread-3" [_thread_blocked, id=26240, stack(0x00007fc5bb243000,0x00007fc5bb2c4000)] 0x00007fc6c35ad800 JavaThread "pool-32-thread-1" [_thread_blocked, id=19944, stack(0x00007fc5aa257000,0x00007fc5aa2d8000)] 0x00007fc59f4e2000 JavaThread "CDBPool-OpticalDwdm15-thread-4" [_thread_blocked, id=1631, stack(0x00007fc5d1d40000,0x00007fc5d1dc1000)] 0x00007fc5a4400000 JavaThread "CDBPool-OpticalDwdm15-thread-3" [_thread_blocked, id=30974, stack(0x00007fc5b940e000,0x00007fc5b948f000)] 0x0000000009636800 JavaThread "CDBPool-OpticalDwdm15-thread-2" [_thread_blocked, id=30596, stack(0x00007fc5eab5b000,0x00007fc5eabdc000)] 0x00007fcaf46b4800 JavaThread "CDBPool-OpticalDwdm15-thread-1" [_thread_blocked, id=27729, stack(0x00007fc5bd1d1000,0x00007fc5bd252000)] 0x00007fcafcbc2000 JavaThread "Thread-1508" [_thread_blocked, id=11679, stack(0x00007fc5d1ec3000,0x00007fc5d1f44000)] 0x00000000017f1800 JavaThread "OnDemandWiredClientPoller" daemon [_thread_blocked, id=11349, stack(0x00007fc5e1f41000,0x00007fc5e1fc2000)] 0x00007fcaf578a000 JavaThread "ActiveMQ Data File Writer" daemon [_thread_blocked, id=3770, stack(0x00007fc5b930c000,0x00007fc5b938d000)] 0x00000000030df000 JavaThread "CorrIsoThread_DuplicateEventSuppresion_t0" [_thread_blocked, id=3413, stack(0x00007fc5b64a9000,0x00007fc5b652a000)] 0x00007fc698dc6800 JavaThread "CorrIsoThread_AnyAlarmFlapping_t0" [_thread_blocked, id=3412, stack(0x00007fc5bc03b000,0x00007fc5bc0bc000)] 0x00007fc59e66a800 JavaThread "ICE Service - CPU 39" [_thread_blocked, id=900, stack(0x00007fc5d2a59000,0x00007fc5d2ada000)] 0x00007fc5a48b5800 JavaThread "ICE Service - CPU 38" [_thread_blocked, id=32103, stack(0x00007fc5eda38000,0x00007fc5edab9000)] 0x00007fc5a4ed2800 JavaThread "ICE Service - CPU 37" [_thread_blocked, id=31722, stack(0x00007fc5cd921000,0x00007fc5cd9a2000)] 0x00007fc6c21a1800 JavaThread "ICE Service - CPU 36" [_thread_blocked, id=31486, stack(0x00007fc5bc0bc000,0x00007fc5bc13d000)] 0x00007fcafec9c800 JavaThread "ICE Service - CPU 35" [_thread_blocked, id=30882, stack(0x00007fc5b6830000,0x00007fc5b68b1000)] 0x00007fcafc235000 JavaThread "ICE Service - CPU 34" [_thread_blocked, id=30726, stack(0x00007fc5e2f65000,0x00007fc5e2fe6000)] 0x00007fcafdbcf800 JavaThread "ICE Service - CPU 33" [_thread_blocked, id=30686, stack(0x00007fc5d9986000,0x00007fc5d9a07000)] 0x00007fc6c38ae800 JavaThread "ICE Service - CPU 32" [_thread_blocked, id=29962, stack(0x00007fc5cd8a0000,0x00007fc5cd921000)] 0x00007fc6700bc800 JavaThread "ICE Service - CPU 31" [_thread_blocked, id=28867, stack(0x00007fc5b66ad000,0x00007fc5b672e000)] 0x0000000006277800 JavaThread "ICE Service - CPU 30" [_thread_blocked, id=28562, stack(0x00007fc5cd090000,0x00007fc5cd111000)] 0x00007fc6997ee800 JavaThread "ICE Service - CPU 29" [_thread_blocked, id=28151, stack(0x00007fc5cd519000,0x00007fc5cd59a000)] 0x00007fcaf659b000 JavaThread "ICE Service - CPU 28" [_thread_blocked, id=27996, stack(0x00007fc5cd00f000,0x00007fc5cd090000)] 0x00007fcafdbd1000 JavaThread "configParallelExecutor-10" [_thread_blocked, id=27986, stack(0x00007fc5d122e000,0x00007fc5d12af000)] 0x00007fc698a00800 JavaThread "detectingDetailSyncUp" daemon [_thread_blocked, id=27454, stack(0x00007fc5f0e9c000,0x00007fc5f0f1d000)] 0x00007fcaf705e000 JavaThread "refreshRogueAp" daemon [_thread_blocked, id=27453, stack(0x00007fc5ed72e000,0x00007fc5ed7af000)] 0x00007fcaf4ccd800 JavaThread "deleteDetectingDetail" daemon [_thread_blocked, id=27452, stack(0x00007fc5b4f7f000,0x00007fc5b5000000)] 0x00007fc586328000 JavaThread "RemoveFromIgnoreList" daemon [_thread_blocked, id=27451, stack(0x00007fc5b73c7000,0x00007fc5b7448000)] 0x00007fc5a2576000 JavaThread "DiscoverRogues" daemon [_thread_blocked, id=27450, stack(0x00007fc5e3877000,0x00007fc5e38f8000)] 0x0000000007b78800 JavaThread "RogueDataProcessor" daemon [_thread_blocked, id=27449, stack(0x00007fc5d90f5000,0x00007fc5d9176000)] 0x00007fc69925f000 JavaThread "RoguePoller1" [_thread_blocked, id=27448, stack(0x00007fc5d1534000,0x00007fc5d15b5000)] 0x00007fcaf79d7000 JavaThread "ICE Service - CPU 27" [_thread_blocked, id=27065, stack(0x00007fc5b4dfc000,0x00007fc5b4e7d000)] 0x00007fcaf7070000 JavaThread "configParallelExecutor-9" [_thread_blocked, id=27063, stack(0x00007fc5b7a54000,0x00007fc5b7ad5000)] 0x00007fcafc1f4000 JavaThread "ICE Service - CPU 26" [_thread_blocked, id=27018, stack(0x00007fc5b6d3a000,0x00007fc5b6dbb000)] 0x00007fcaf6db6800 JavaThread "configParallelExecutor-8" [_thread_blocked, id=27009, stack(0x00007fc5ad19d000,0x00007fc5ad21e000)] 0x00007fc594a6f000 JavaThread "http-nio-443-exec-33" daemon [_thread_blocked, id=26803, stack(0x00007fc5da844000,0x00007fc5da8c5000)] 0x00007fc5a3d18000 JavaThread "http-nio-443-exec-26" daemon [_thread_blocked, id=26796, stack(0x00007fc5b43e8000,0x00007fc5b4469000)] 0x00007fc699653000 JavaThread "http-nio-443-exec-14" daemon [_thread_blocked, id=26741, stack(0x00007fc5b44ea000,0x00007fc5b456b000)] 0x0000000003b2b000 JavaThread "ICE Service - CPU 25" [_thread_blocked, id=26649, stack(0x00007fc5b60a1000,0x00007fc5b6122000)] 0x00007fc6c3920000 JavaThread "configParallelExecutor-7" [_thread_blocked, id=26648, stack(0x00007fc5ea4ce000,0x00007fc5ea54f000)] 0x00007fcafceca000 JavaThread "ThreadManager-ConfigArchive-20" [_thread_blocked, id=26637, stack(0x00007fc5edd4d000,0x00007fc5edd86000)] 0x00007fcaff4c0000 JavaThread "ICE Service - CPU 24" [_thread_blocked, id=26635, stack(0x00007fc5af23f000,0x00007fc5af2c0000)] 0x00007fc6c2a20800 JavaThread "configParallelExecutor-6" [_thread_blocked, id=26625, stack(0x00007fc5b5891000,0x00007fc5b5912000)] 0x00007fc5a5cae000 JavaThread "ThreadManager-ConfigArchive-19" [_thread_blocked, id=25596, stack(0x00007fc5eed21000,0x00007fc5eed5a000)] 0x00007fc5a5f42800 JavaThread "ICE Service - CPU 23" [_thread_blocked, id=25595, stack(0x00007fc5b5b16000,0x00007fc5b5b97000)] 0x00007fc6c33f3000 JavaThread "configParallelExecutor-5" [_thread_blocked, id=25589, stack(0x00007fc5b79d3000,0x00007fc5b7a54000)] 0x00007fc5a4a7b800 JavaThread "ThreadManager-ConfigArchive-18" [_thread_blocked, id=25506, stack(0x00007fc5eece8000,0x00007fc5eed21000)] 0x0000000001654000 JavaThread "ICE Service - CPU 22" [_thread_blocked, id=25505, stack(0x00007fc5d27b0000,0x00007fc5d2831000)] 0x00007fcafe242800 JavaThread "configParallelExecutor-4" [_thread_blocked, id=25496, stack(0x00007fc5b5000000,0x00007fc5b5081000)] 0x00007fc5d5a90800 JavaThread "ThreadManager-ConfigArchive-17" [_thread_blocked, id=24495, stack(0x00007fc5f089f000,0x00007fc5f08d8000)] 0x00007fc6988ae000 JavaThread "ICE Service - CPU 21" [_thread_blocked, id=24494, stack(0x00007fc5b71c3000,0x00007fc5b7244000)] 0x00007fc6c0a9a000 JavaThread "configParallelExecutor-3" [_thread_blocked, id=24483, stack(0x00007fc5ad6a7000,0x00007fc5ad728000)] 0x00007fc5a521d800 JavaThread "ThreadManager-ConfigArchive-16" [_thread_blocked, id=23793, stack(0x00007fc5f5d90000,0x00007fc5f5dc9000)] 0x00007fc6c31fc000 JavaThread "ICE Service - CPU 20" [_thread_blocked, id=23792, stack(0x00007fc5b6326000,0x00007fc5b63a7000)] 0x00007fc6c3df5000 JavaThread "configParallelExecutor-2" [_thread_blocked, id=23783, stack(0x00007fc5b6cb9000,0x00007fc5b6d3a000)] 0x00007fcaf6ea7000 JavaThread "http-nio-443-exec-12" daemon [_thread_blocked, id=23630, stack(0x00007fc5b6e3c000,0x00007fc5b6ebd000)] 0x00007fcafcd40000 JavaThread "ThreadManager-ConfigArchive-15" [_thread_blocked, id=23499, stack(0x00007fc5f5dc9000,0x00007fc5f5e02000)] 0x00007fc6c3bce800 JavaThread "ICE Service - CPU 19" [_thread_blocked, id=23497, stack(0x00007fc5b754a000,0x00007fc5b75cb000)] 0x00007fcafd531000 JavaThread "CorrIsoThread_tl1_granular_inventory_rule_t0" [_thread_blocked, id=23475, stack(0x00007fc5b5993000,0x00007fc5b5a14000)] 0x00007fcaf5934000 JavaThread "configParallelExecutor-1" [_thread_blocked, id=23473, stack(0x00007fc5b5912000,0x00007fc5b5993000)] 0x00007fcafd374000 JavaThread "pool-69-thread-1" [_thread_blocked, id=16655, stack(0x00007fc5b6c38000,0x00007fc5b6cb9000)] 0x00007fcafd6f5000 JavaThread "pool-69-thread-3" [_thread_blocked, id=16654, stack(0x00007fc5d2c5d000,0x00007fc5d2cde000)] 0x00007fcaf7928800 JavaThread "pool-69-thread-2" [_thread_blocked, id=16653, stack(0x00007fc5b6dbb000,0x00007fc5b6e3c000)] 0x00007fc6c1a6c000 JavaThread "pool-70-thread-2" [_thread_blocked, id=16201, stack(0x00007fc5ad09b000,0x00007fc5ad11c000)] 0x00007fc6cd815800 JavaThread "pool-70-thread-3" [_thread_blocked, id=16200, stack(0x00007fc5af0bc000,0x00007fc5af13d000)] 0x00007fcafc70f800 JavaThread "pool-70-thread-1" [_thread_blocked, id=16199, stack(0x00007fc5bb822000,0x00007fc5bb8a3000)] 0x00007fcafea57000 JavaThread "CDBPool-DVAVAILABILITY-thread-4" [_thread_blocked, id=15970, stack(0x00007fc5da111000,0x00007fc5da192000)] 0x00007fc6c2538800 JavaThread "CDBPool-ENVTEMP-thread-4" [_thread_blocked, id=15966, stack(0x00007fc5bca46000,0x00007fc5bcac7000)] 0x00007fc670770800 JavaThread "CleanAirAlarmAnalyzer" daemon [_thread_blocked, id=15763, stack(0x00007fc5b568d000,0x00007fc5b570e000)] 0x0000000001ced800 JavaThread "CDBPool-DVAVAILABILITY-thread-3" [_thread_blocked, id=15629, stack(0x00007fc5ace97000,0x00007fc5acf18000)] 0x00007fcaf4571000 JavaThread "CDBPool-ENVTEMP-thread-3" [_thread_blocked, id=15625, stack(0x00007fc5b7142000,0x00007fc5b71c3000)] 0x00007fcaf4664800 JavaThread "pool-61-thread-25" [_thread_blocked, id=14848, stack(0x00007fc5e2043000,0x00007fc5e20c4000)] 0x00007fcaf4ab2800 JavaThread "pool-61-thread-24" [_thread_blocked, id=14843, stack(0x00007fc5b42e6000,0x00007fc5b4367000)] 0x00007fcaf4aba000 JavaThread "pool-61-thread-23" [_thread_blocked, id=14838, stack(0x00007fc5ad626000,0x00007fc5ad6a7000)] 0x00007fcaf7087800 JavaThread "pool-61-thread-22" [_thread_blocked, id=14836, stack(0x00007fc5bb7a1000,0x00007fc5bb822000)] 0x00007fcaf534e000 JavaThread "pool-61-thread-21" [_thread_blocked, id=14798, stack(0x00007fc5d2957000,0x00007fc5d29d8000)] 0x00007fc6cd52f000 JavaThread "CDBPool-DVAVAILABILITY-thread-2" [_thread_blocked, id=14790, stack(0x00007fc5b3fe0000,0x00007fc5b4061000)] 0x0000000006c1f000 JavaThread "CDBPool-ENVTEMP-thread-2" [_thread_blocked, id=14786, stack(0x00007fc5b5d9b000,0x00007fc5b5e1c000)] 0x00007fcaf60ea000 JavaThread "pool-61-thread-20" [_thread_blocked, id=14724, stack(0x00007fc5bbfb1000,0x00007fc5bc032000)] 0x00007fcaf60bd000 JavaThread "pool-61-thread-19" [_thread_blocked, id=14643, stack(0x00007fc5ada2e000,0x00007fc5adaaf000)] 0x00007fcaf72bf000 JavaThread "pool-61-thread-18" [_thread_blocked, id=14614, stack(0x00007fc5b18ce000,0x00007fc5b194f000)] 0x00007fcaf508b000 JavaThread "pool-61-thread-17" [_thread_blocked, id=14613, stack(0x00007fc5af13d000,0x00007fc5af1be000)] 0x00007fc6983cb000 JavaThread "pool-61-thread-16" [_thread_blocked, id=14611, stack(0x00007fc5aef39000,0x00007fc5aefba000)] 0x00007fcafcb7d000 JavaThread "ThreadManager-ConfigArchive-14" [_thread_blocked, id=14610, stack(0x00007fc5ace25000,0x00007fc5ace5e000)] 0x00007fcaf5ffa800 JavaThread "pool-61-thread-15" [_thread_blocked, id=14599, stack(0x00007fc5b1a51000,0x00007fc5b1ad2000)] 0x00007fcaf5d55000 JavaThread "pool-61-thread-14" [_thread_blocked, id=14594, stack(0x00007fc5b47f0000,0x00007fc5b4871000)] 0x00007fcaffc59000 JavaThread "ThreadManager-ConfigArchive-13" [_thread_blocked, id=14520, stack(0x00007fc5bc9c5000,0x00007fc5bca46000)] 0x00007fcafcb7d800 JavaThread "ThreadManager-ConfigArchive-12" [_thread_blocked, id=14405, stack(0x00007fc5af341000,0x00007fc5af3c2000)] 0x00007fcaf7240000 JavaThread "pool-61-thread-13" [_thread_blocked, id=14328, stack(0x00007fc5b184d000,0x00007fc5b18ce000)] 0x00007fcaf46b3800 JavaThread "pool-61-thread-12" [_thread_blocked, id=14289, stack(0x00007fc5aeeb8000,0x00007fc5aef39000)] 0x00000000096c9800 JavaThread "ICE Service - CPU 18" [_thread_blocked, id=14283, stack(0x00007fc5ca780000,0x00007fc5ca801000)] 0x00007fcaf4157800 JavaThread "pool-61-thread-11" [_thread_blocked, id=14257, stack(0x00007fc5e1d3d000,0x00007fc5e1dbe000)] 0x00000000096c8000 JavaThread "ICE Service - CPU 17" [_thread_blocked, id=14227, stack(0x00007fc5af1be000,0x00007fc5af23f000)] 0x00007fcaf46b5000 JavaThread "pool-61-thread-10" [_thread_blocked, id=14226, stack(0x00007fc5b1ad2000,0x00007fc5b1b53000)] 0x00007fcaf4059800 JavaThread "pool-61-thread-9" [_thread_blocked, id=14225, stack(0x00007fc5af2c0000,0x00007fc5af341000)] 0x00007fcaf5f7d000 JavaThread "pool-61-thread-8" [_thread_blocked, id=14219, stack(0x00007fc5bb0c0000,0x00007fc5bb141000)] 0x00007fcaf6c92800 JavaThread "pool-61-thread-7" [_thread_blocked, id=14181, stack(0x00007fc5b5f1e000,0x00007fc5b5f9f000)] 0x00007fcaf4f50000 JavaThread "pool-61-thread-6" [_thread_blocked, id=13929, stack(0x00007fc5ef366000,0x00007fc5ef3e7000)]
Java kill suddenly
I am getting this error when writer.optimize() called I have caugh all exceptions but hopeless .writer is an instance of apache lucene Indexwriter and tomcat collapse when optimizing the indexwriter.I am trying to index a large number of file its works for a few number of file but when number of files increase it cause to tomcat fail. logger.info("Optimizing optimazing..."); this.writer.optimize(); logger.info("Optimizing closing..."); this.writer.close(); logger.info("Optimazide and closed succesfully..."); # # A fatal error has been detected by the Java Runtime Environment: # # SIGSEGV (0xb) at pc=0x00007fe6f8c38e90, pid=10316, tid=140629887768320 # # JRE version: 6.0_20-b20 # Java VM: OpenJDK 64-Bit Server VM (19.0-b09 mixed mode linux-amd64 compressed oops) # Derivative: IcedTea6 1.9.7 # Distribution: Ubuntu 10.10, package 6b20-1.9.7-0ubuntu1 # Problematic frame: # V [libjvm.so+0x54ae90] # # If you would like to submit a bug report, please include # instructions how to reproduce the bug and visit: # https://bugs.launchpad.net/ubuntu/+source/openjdk-6/ # --------------- T H R E A D --------------- Current thread (0x00000000023e0000): JavaThread "CompilerThread0" daemon [_thread_in_native, id=10333, stack(0x00007fe6f2715000,0x00007fe6f2816000)] siginfo:si_signo=SIGSEGV: si_errno=0, si_code=1 (SEGV_MAPERR), si_addr=0x0000000000000008 Registers: RAX=0x0000000000000000, RBX=0x00007fe6f2812930, RCX=0x00007fe6ec03e9e0, RDX=0x0000000000002000 RSP=0x00007fe6f2811150, RBP=0x00007fe6f2811190, RSI=0x00007fe6e43a20f0, RDI=0x0000000000000000 R8 =0x00007fe6e43f5a70, R9 =0x00007fe6f2812930, R10=0x00007fe6ec6f7948, R11=0x0000000000000000 R12=0x00007fe6edd326b0, R13=0x00007fe6ec6f7948, R14=0x00007fe6f2812950, R15=0x00007fe6ec068990 RIP=0x00007fe6f8c38e90, EFL=0x0000000000010206, CSGSFS=0x0000000000000033, ERR=0x0000000000000004 TRAPNO=0x000000000000000e Register to memory mapping: RAX=0x0000000000000000 0x0000000000000000 is pointing to unknown location RBX=0x00007fe6f2812930 0x00007fe6f2812930 is pointing into the stack for thread: 0x00000000023e0000 "CompilerThread0" daemon prio=10 tid=0x00000000023e0000 nid=0x285d runnable [0x0000000000000000] java.lang.Thread.State: RUNNABLE RCX=0x00007fe6ec03e9e0 0x00007fe6ec03e9e0 is pointing to unknown location RDX=0x0000000000002000 0x0000000000002000 is pointing to unknown location RSP=0x00007fe6f2811150 0x00007fe6f2811150 is pointing into the stack for thread: 0x00000000023e0000 "CompilerThread0" daemon prio=10 tid=0x00000000023e0000 nid=0x285d runnable [0x0000000000000000] java.lang.Thread.State: RUNNABLE RBP=0x00007fe6f2811190 0x00007fe6f2811190 is pointing into the stack for thread: 0x00000000023e0000 "CompilerThread0" daemon prio=10 tid=0x00000000023e0000 nid=0x285d runnable [0x0000000000000000] java.lang.Thread.State: RUNNABLE RSI=0x00007fe6e43a20f0 0x00007fe6e43a20f0 is pointing to unknown location RDI=0x0000000000000000 0x0000000000000000 is pointing to unknown location R8 =0x00007fe6e43f5a70 0x00007fe6e43f5a70 is pointing to unknown location R9 =0x00007fe6f2812930 0x00007fe6f2812930 is pointing into the stack for thread: 0x00000000023e0000 "CompilerThread0" daemon prio=10 tid=0x00000000023e0000 nid=0x285d runnable [0x0000000000000000] java.lang.Thread.State: RUNNABLE R10=0x00007fe6ec6f7948 0x00007fe6ec6f7948 is pointing to unknown location R11=0x0000000000000000 0x0000000000000000 is pointing to unknown location R12=0x00007fe6edd326b0 0x00007fe6edd326b0 is pointing to unknown location R13=0x00007fe6ec6f7948 0x00007fe6ec6f7948 is pointing to unknown location R14=0x00007fe6f2812950 0x00007fe6f2812950 is pointing into the stack for thread: 0x00000000023e0000 "CompilerThread0" daemon prio=10 tid=0x00000000023e0000 nid=0x285d runnable [0x0000000000000000] java.lang.Thread.State: RUNNABLE R15=0x00007fe6ec068990 0x00007fe6ec068990 is pointing to unknown location Top of Stack: (sp=0x00007fe6f2811150) 0x00007fe6f2811150: 00007fe6e4522cd0 00007fe6f2811420 0x00007fe6f2811160: 00007fe6f2811190 00007fe6f2812930 0x00007fe6f2811170: 0000000000000002 00007fe6edd326b0 0x00007fe6f2811180: 00007fe6ec5d6430 00007fe6f2811420 0x00007fe6f2811190: 00007fe6f2811200 00007fe6f8c3941b 0x00007fe6f28111a0: 0000000000000002 00007fe600000100 0x00007fe6f28111b0: 00007fe600000001 00007fe6f28132d0 w-p 00021000 08:01 17301749 /lib/ld-2.12.1.so 7fe6fa020000-7fe6fa021000 rw-p 00000000 00:00 0 7fffd5558000-7fffd5579000 rw-p 00000000 00:00 0 [stack] 7fffd55ff000-7fffd5600000 r-xp 00000000 00:00 0 [vdso] ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall] VM Arguments: jvm_args: -Djava.util.logging.config.file=/home/murat/Desktop/servers/apache-tomcat-6.0.24/conf/logging.properties -Dhttp.nonProxyHosts=localhost|127.0.0.1|expertPC -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -agentlib:jdwp=transport=dt_socket,address=11550,server=y,suspend=n -Djava.endorsed.dirs=/home/murat/Desktop/servers/apache-tomcat-6.0.24/endorsed -Dcatalina.base=/home/murat/Desktop/servers/apache-tomcat-6.0.24 -Dcatalina.home=/home/murat/Desktop/servers/apache-tomcat-6.0.24 -Djava.io.tmpdir=/home/murat/Desktop/servers/apache-tomcat-6.0.24/temp java_command: org.apache.catalina.startup.Bootstrap start Launcher Type: SUN_STANDARD Environment Variables: JAVA_HOME=/usr/lib/jvm/java-6-openjdk JRE_HOME=/usr/lib/jvm/java-6-openjdk PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games USERNAME=murat LD_LIBRARY_PATH=/usr/lib/jvm/java-6-openjdk/jre/lib/amd64/server:/usr/lib/jvm/java-6-openjdk/jre/lib/amd64:/usr/lib/jvm/java-6-openjdk/jre/../lib/amd64 SHELL=/bin/bash DISPLAY=:0.0 Signal Handlers: SIGSEGV: [libjvm.so+0x723630], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004 SIGBUS: [libjvm.so+0x723630], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004 SIGFPE: [libjvm.so+0x5e0000], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004 SIGPIPE: SIG_IGN, sa_mask[0]=0x00000000, sa_flags=0x00000000 SIGXFSZ: [libjvm.so+0x5e0000], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004 SIGILL: [libjvm.so+0x5e0000], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004 SIGUSR1: SIG_DFL, sa_mask[0]=0x00000000, sa_flags=0x00000000 SIGUSR2: [libjvm.so+0x5df710], sa_mask[0]=0x00000004, sa_flags=0x10000004 SIGHUP: [libjvm.so+0x5e2180], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004 SIGINT: SIG_IGN, sa_mask[0]=0x00000000, sa_flags=0x00000000 SIGTERM: [libjvm.so+0x5e2180], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004 SIGQUIT: [libjvm.so+0x5e2180], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004 --------------- S Y S T E M --------------- OS:Ubuntu 10.10 (maverick) uname:Linux 2.6.35-28-generic #50-Ubuntu SMP Fri Mar 18 18:42:20 UTC 2011 x86_64 libc:glibc 2.12.1 NPTL 2.12.1 rlimit: STACK 8192k, CORE 0k, NPROC infinity, NOFILE 1024, AS infinity load average:0.77 0.30 0.14 /proc/meminfo: MemTotal: 4054828 kB MemFree: 176928 kB Buffers: 207640 kB Cached: 1332820 kB SwapCached: 17608 kB Active: 2419624 kB Inactive: 1004992 kB Active(anon): 1834536 kB Inactive(anon): 67792 kB Active(file): 585088 kB Inactive(file): 937200 kB Unevictable: 16 kB Mlocked: 16 kB SwapTotal: 11876348 kB SwapFree: 11687616 kB Dirty: 3508 kB Writeback: 32 kB AnonPages: 1873148 kB Mapped: 197036 kB Shmem: 18240 kB Slab: 157916 kB SReclaimable: 131452 kB SUnreclaim: 26464 kB KernelStack: 3928 kB PageTables: 32140 kB NFS_Unstable: 0 kB Bounce: 0 kB WritebackTmp: 0 kB CommitLimit: 13903760 kB Committed_AS: 3427468 kB VmallocTotal: 34359738367 kB VmallocUsed: 323536 kB VmallocChunk: 34359412360 kB HardwareCorrupted: 0 kB HugePages_Total: 0 HugePages_Free: 0 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 2048 kB DirectMap4k: 272384 kB DirectMap2M: 3919872 kB CPU:total 2 (2 cores per cpu, 1 threads per core) family 6 model 23 stepping 10, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, sse4.1 Memory: 4k page, physical 4054828k(176928k free), swap 11876348k(11687616k free) vm_info: OpenJDK 64-Bit Server VM (19.0-b09) for linux-amd64 JRE (1.6.0_20-b20), built on Feb 23 2011 09:05:53 by "buildd" with gcc 4.4.5 time: Mon Jun 6 22:24:11 2011 elapsed time: 2076 seconds
As Stéphane says, try different JREs to see if you can get a different error message. There's a chance (but hard to quantify) that this is related to reaching a memory limit, but it'll be hard to be sure unless you do get an error saying which one!
Use sun/oracle java jdk, not the open jdk... Regards, Stéphane
Since it is most likely a bug in the JVM, the first thing I would try is updating your JVM to Java 6 update 25. You could try the Sun/Oracle JVM, but it likely to be the same. I am assuming you are not using an JNI libraries?