I'm profiling (using instrumentation) a Java web app on Tomcat. When I attach a JProfiler session it first goes through a period of Retransforming Classes. During this period I am seeing numerous messages in the Tomcat log like these:
JProfiler> ERROR: could not retransform class Lcom/zaxxer/hikari/pool/HikariPool$HouseKeeper$$Lambda$36/1482601803; (62).
JProfiler> ERROR: could not retransform class Lcom/zaxxer/hikari/pool/HikariPool$HouseKeeper$$Lambda$35/545889432; (62).
JProfiler> ERROR: could not retransform class Lcom/sun/jersey/atom/rome/impl/provider/entity/AtomEntryProvider; (113).
JProfiler> ERROR: could not retransform class Lcom/sun/jersey/atom/rome/impl/provider/entity/AtomFeedProvider; (113).
What is the cause and implication of these errors?
I'm using:
Tomcat 7.0.55
Java 8
JProfiler 8.0.7
The used version of JProfiler is too old and does not fully support Java 8. Updating to 8.1.4 will fix this issue.
Related
Iam facing JVM Crash cosistently while enabling hotdeploy (USING below java options on starting up JAVA_OPTS -Xmx4096m -XX:MetaspaceSize=512m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=crash -XX:ThreadStackSize=512 -XX:+UseConcMarkSweepGC -XX:ParallelGCThreads=5 -XX:NewRatio=2 -XX:+UnlockDiagnosticVMOptions -XX:-UseLoopPredicate -Xdebug -Xrunjdwp:transport=dt_socket,address=$DEBUG_PORT,server=y,suspend=n -XX:NewRatio=2 -Dspringloaded.synchronize=true JAVA_OPTS=`echo $JAVA_OPTS -Dspringloaded.synchronize=true -javaagent:springloaded-1.2.1.jar -noverify
)
Environment : JDK 1.8 U 66, RHEL 6.7
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x00007faee9a1e27c, pid=27208, tid=140379827795712
#
# JRE version: Java(TM) SE Runtime Environment (8.0_66-b17) (build 1.8.0_66-b17)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.66-b17 mixed mode linux-amd64 )
# Problematic frame:
# V [libjvm.so+0x35027c] Canonicalizer::do_If(If*)+0x1c
#
# Core dump written. Default location: core.27208
#
# An error report file with more information is saved as:
# hs_err_pid27208.log
# [ timer expired, abort... ]
I've noticed both -javaagent and -noverify in Java options list.
It looks like springloaded agent generates invalid bytecode, while the bytecode verification is explicitly turned off. No surprise, this may lead to unpredictable results including JVM crash.
This is not a JVM problem, but most likely a bug in springloaded agent. Try to remove -noverify option.
-XX:-TieredCompilation may also work around this particular problem, but don't expect application to work correctly if the bytecode fails to pass verification. It's better to stay away from the buggy agent libraries.
4.2.1 Crash in HotSpot Compiler Thread or Compiled Code
If the fatal error log indicates that the crash occurred in a compiler
thread, then it is possible (but not always the case) that you have
encountered a compiler bug. Similarly, if the crash is in compiled
code then it is possible that the compiler has generated incorrect
code.
In the case of the HotSpot Client VM (-client option), the compiler
thread appears in the error log as CompilerThread0. With the HotSpot
Server VM there are multiple compiler threads and these appear in the
error log file as CompilerThread0, CompilerThread1, and AdapterThread.
Below is a fragment of an error log for a compiler bug that was
encountered and fixed during the development of J2SE 5.0. The log file
shows that the HotSpot Server VM is used and the crash occurred in
CompilerThread1. In addition, the log file shows that the Current
CompileTask was the compilation of the java.lang.Thread.setPriority
method.
An unexpected error has been detected by HotSpot Virtual Machine:
:
Java VM: Java HotSpot(TM) Server VM (1.5-internal-debug mixed mode) :
--------------- T H R E A D ---------------
Current thread (0x001e9350): JavaThread "CompilerThread1" daemon
[_thread_in_vm, id=20]
Stack: [0xb2500000,0xb2580000), sp=0xb257e500, free space=505k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code,
C=native code) V [libjvm.so+0xc3b13c] :
Current CompileTask: opto: 11 java.lang.Thread.setPriority(I)V
(53 bytes)
--------------- P R O C E S S ---------------
Java Threads: ( => current thread ) 0x00229930 JavaThread "Low
Memory Detector" daemon [_thread_blocked, id=21]
=>0x001e9350 JavaThread "CompilerThread1" daemon [_thread_in_vm, id=20] :
In this case there are two potential workarounds:
The brute force approach: change the configuration so that the application is run with the -client option to specify the HotSpot
Client VM.
Assume that the bug only occurs during the compilation of the setPriority method and exclude this method from compilation.
The first approach (to use the -client option) might be trivial to
configure in some environments. In others, it might be more difficult
if the configuration is complex or if the command line to configure
the VM is not readily accessible. In general, switching from the
HotSpot Server VM to the HotSpot Client VM also reduces the peak
performance of an application. Depending on the environment, this
might be acceptable until the actual issue is diagnosed and fixed.
The second approach (exclude the method from compilation) requires
creating the file .hotspot_compiler in the working directory of the
application. Below is an example of this file:
exclude java/lang/Thread setPriority
In general the format of this file is exclude CLASS METHOD, where
CLASS is the class (fully qualified with the package name) and METHOD
is the name of the method. Constructor methods are specified as
and static initializers are specified as .
Note - The .hotspot_compiler file is an unsupported interface. It is
documented here solely for the purposes of troubleshooting and finding
a temporary workaround.
Once the application is restarted, the compiler will not attempt to
compile any of the methods listed as excluded in the .hotspot_compiler
file. In some cases this can provide temporary relief until the root
cause of the crash is diagnosed and the bug is fixed.
In order to verify that the HotSpot VM correctly located and processed
the .hotspot_compiler file that is shown in the example above, look
for the following log information at runtime. Note that the file name
separator is a dot, not a slash.
Excluding compile: java.lang.Thread::setPriority
Source
Agree with #apangin, In the program you are doing bytecode intrumentation (-agent) but specifies -noverify. When verification is turned off, you may end up such crashes.
You should not use -noverify or -Xverify:none during byte code intrumentation.
For those of you unfamiliar with bytecode verification, it is simply part of the JVM's classloading process that checks the code for certain dangerous and disallowed behavior. You can (but shouldn't) disable this protection on many JVMs by adding -Xverify:none or -noverify to the Java command line. https://blogs.oracle.com/buck/entry/never_disable_bytecode_verification_in
I 've started a Java application with the args :
"-agentpath:C:\Program Files\jprofiler7\bin\windows-x64\jprofilerti.dll=port=8849,id=134,nowait"
then the console showed:
JProfiler> Protocol version 37
JProfiler> Using JVMTI
JProfiler> JVMTI version 1.1 detected.
JProfiler> 64-bit library
JProfiler> Don't wait for frontend to connect.
JProfiler> Using config file C:\Users\user\.jprofiler7\config.xml (id: 134)
JProfiler> Listening on port: 8849.
JProfiler> Instrumenting native methods.
JProfiler> Can retransform classes.
JProfiler> Can retransform any class.
JProfiler> Native library initialized
JProfiler> VM initialized
JProfiler> Using dynamic instrumentation
JProfiler> Time measurement: elapsed time
JProfiler> CPU profiling enabled
JProfiler> Hotspot compiler enabled
Then I could connect by JProfiler GUI now.
But I need connect that via another Java application with JProfiler API.
How could I connect that ? By any VM argument ?
I just know maybe I have to create a connection first , but I don't know how to choose the session ID.
Connection connection = ConnectionFactory.createRemoteConnection("localhost", 8849, 30);
If I ran the agent application then connect via JProfiler GUi firstly ,I could get the cpu information by my Java application , including then running my JProfiler API application .
Profiling data :
Hot spots:
Top 5 hot spots:
Hot spot 1: ProfilerTest.main(java.lang.String[ ]): 34 ms (100 %)
Backtraces:
ProfilerTest.main(java.lang.String[ ]): 34722 (0)
***: 34722 (0)
Cpu tree:
***: 34722 (0)
ProfilerTest.main(java.lang.String[ ]): 34722 (0)
java.util.Scanner.next(): 34722 (0)
But I'll get nothing if I ran the agent application then connect via my JProfiler API application firstly.
Profiling data :
Hot spots:
Top 5 hot spots:
Cpu tree:
***: 0 (0)
After ",nowait", add ",config=[path to config file],id=nnn" where "[path to config file]" is %USERPROFILE%.jprofiler8\config.xml and "nnn" is the session ID that is visible in the top-right corner of the application settings tab in the session settings dialog.
all
I met an error when I running the sample of JProfiler7 offline on my windows PC , the code is from "jprofiler7\api\samples\offline\src\TestProgram.java" ,and got the message as thus:
JProfiler> A different instance of the native library has been
JProfiler> loaded. Please check the appropriate environment
JProfiler> variable. (PATH, LD_LIBRARY_PATH, DYLD_LIBRARY_PATH)
JProfiler> Exiting.
I'm sure there is no any JProfiler process running, and I've imported the referenced libraries by "agent.jar" and set my PATH of JProfiler：
C:\Users\user>echo %PATH%
C:\Program Files\Java\jdk1.6.0_33\bin;C:\Program Files\java\jdk1.6.0_33\jre\bin;C:\Program Files\jprofiler7\bin\windows-x64;C:\Program Files\jprofiler7\bin;
I wanna use the JProfiler API for remoting doing something ,could anybody know how to fix that?
JProfiler:7.2.3
Windows: Win7 Enterprise SP1
JDK:1.6.0_33
And if I use the arguments as following:
-agentlib:jprofilerti=offline,id=4321,config=C:\Program Files\jprofiler7\config\config.xml "-Xbootclasspath/a:C:\Program Files\jprofiler7\bin\agent.jar"
there will be another message:
Exception in thread "main" java.lang.NoClassDefFoundError: com/jprofiler/agent/ControllerImpl
at com.jprofiler.api.agent.Controller.startCPURecording(Controller.java:87)
at TestProgram.main(TestProgram.java:31)
Caused by: java.lang.ClassNotFoundException: com.jprofiler.agent.ControllerImpl
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
... 2 more
____________edited 2014.05.08 17:04 CST__________________________________
I tried to run the demo of platform (jprofiler7\api\samples\platform\src-profiler\TestProfiler.java) , got the same message both without args and with args(-agentpath:C:\Program Files\jprofiler7\bin\windows\jprofilerti.dll=offline,id=112,config="C:\Program Files\jprofiler7\config\config.xml -Xbootclasspath/a:C:\Program Files\jprofiler7\bin\agent.jar):
JProfiler> A different instance of the native library has been
JProfiler> loaded. Please check the appropriate environment
JProfiler> variable. (PATH, LD_LIBRARY_PATH, DYLD_LIBRARY_PATH)
JProfiler> Exiting.
———————————————————ending—————————————————————————————————————————————
solved that.
In my way, firstly, I start a Java application with VM args
"-agentpath:C:\Program Files\jprofiler7\bin\windows-x64\jprofilerti.dll=port=8849"
Then I run a JProfiler platfrom program with args :
-agentpath:C:\Program Files\jprofiler7\bin\windows\jprofilerti.dll=offline,id=112,config="C:\Program Files\jprofiler7\config\config.xml -Xbootclasspath/a:C:\Program Files\jprofiler7\bin\agent.jar
then I could catch some information , with null value.
Profiling data :
Memory data:
Exception in thread "main" java.lang.NullPointerException
at com.jprofiler.core.comm.d.a.getTotalHeapUsage(ejt:100)
at TestProfiler.dumpMemoryData(TestProfiler.java:82)
at TestProfiler.dumpProfilingData(TestProfiler.java:55)
at TestProfiler.main(TestProfiler.java:47)
In a word , it could run...so ending the question.
Replace the -agentlib VM parameter with
-agentpath:C:\Program Files\jprofiler7\bin\windows\jprofilerti.dll=offline,...
and remove the -Xbootclasspath VM parameter
I have written a jvmti agent to trace method invocations. I code it with C and jvmti and jni functions. Our os is Fedora 15 and the agent is compiled into a .so file. When I test it with a non-trivial java program, it crashes and gives the following error message:
A fatal error has been detected by the Java Runtime Environment:
SIGSEGV (0xb) at pc=0x4e8e4e28, pid=24294, tid=3065949040.
JRE version: 6.0_32-b05.
Java VM: Java HotSpot (TM) Server VM (20.7-b02 mixed mode linux-x86).
**Problematic frame:
C [libc.so.6+0x7ae68] strcpy+0x18.**
IGSEGV is an abbreviation for Signal Segmentation Violation. On
POSIX-compliant platforms, SIGSEGV is the signal sent to a process
when it makes an invalid memory reference, or segmentation fault.
You have to check the pointers in your JVMTI agent. In all probability you make some unclean pointer operations.
I want to increase the heap size of my Glassfish. For that, I know that I can go up to 4GB:
java -Xmx4000M -version
java version "1.6.0_26"
Java(TM) SE Runtime Environment (build 1.6.0_26-b03-384-10M3425)
Java HotSpot(TM) 64-Bit Server VM (build 20.1-b02-384, mixed mode)
I tried to set in domain.xml file the -Xmx to 2GB:
<jvm-options>-Xmx2000m</jvm-options>
But I am getting the following error:
asadmin> start-domain
Waiting for ... to start .Error starting domain ...
The server exited prematurely with exit code 1.
Before it died, it produced the following output:
Error occurred during initialization of VM
The size of the object heap + VM data exceeds the maximum representable size
launchctl bsexec failed: Inappropriate ioctl for device
Launching the command with -v option gives this:
12 oct. 2011 11:46:34 com.sun.enterprise.admin.launcher.GFLauncherLogger info
INFO: JVM invocation command line:
/System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home/bin/java
-XX:+UnlockDiagnosticVMOptions
-XX:MaxPermSize=512m
-XX:NewRatio=2
-XX:+CMSClassUnloadingEnabled
-Xmx2000m
-Xms1000m
...
12 oct. 2011 11:46:35 com.sun.enterprise.admin.launcher.GFLauncherLogger info
INFO: Successfully launched in 45 msec.
Error occurred during initialization of VM
Could not reserve enough space for object heap
Could not create the Java virtual machine.
Command start-domain failed.
I found the problem. For some reasons which I do not understand, the "-d32" argument was passed in the command, asking to start in 32 bits mode. When adding
<jvm-options>-d64</jvm-options>
to the domain.xml file, glassfish starts. Please note that this option is not present by default in the file.
you must be using a different Java install when doing the "-version" and from within GlassFish. I can reproduce the same error with the default "java" in Mac OS' path but not when using an absolute path :
% /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home/bin/java -Xmx2300m -version
java version "1.6.0_26"
Java(TM) SE Runtime Environment (build 1.6.0_26-b03-383-11A511)
Java HotSpot(TM) 64-Bit Server VM (build 20.1-b02-383, mixed mode)
% java -Xmx2300m -version
Error occurred during initialization of VM
Could not reserve enough space for object heap
Could not create the Java virtual machine.
Could it be that you don't have enough disk space on your system? Reserving 4GB of RAM may require MAC OS to expand swap space, and you may not have the disk space to do this. Unfortunately, this has burned me before :-/