I've got a plain vanilla install of ignite 2.14, with the binaries downloaded from https://ignite.apache.org/download.cgi (exact link https://dlcdn.apache.org/ignite/2.14.0/apache-ignite-2.14.0-bin.zip). I'm on Windows 10, IGNITE_HOME is not set in the PATH (this is optional), and Ignite's using this java runtime:
OpenJDK Runtime Environment 1.8.0_201-2-redhat-b09 Oracle Corporation
OpenJDK 64-Bit Server VM 25.201-b09
I start an ignite node using the default configuration provided in the downloaded zip apache-ignite-2.14.0-bin.zip :
ignite.bat ..\config\default-config.xml
This starts fine. Following the instructions at https://ignite.apache.org/docs/latest/tools/control-script I can check the state and see I've got a single node cluster in state ACTIVE (the default-config.xml must not have native persistence enabled, so the cluster goes to ACTIVE state automatically).
I can then set the state to INACTIVE like so:
control.bat --set-state INACTIVE
This works fine. However if I set the state to active again like so:
control.bat --set-state ACTIVE
I get the error pasted below and the cluster stays in the INACTIVE state. I first came across this error when using Ignite in embedded server mode, but I can still reproduce it with a fresh out-of-the-box ignite install (not using embedded). I'm surprised that a plain vanilla install just calling a couple of basic commands falls over like this. Any idea what's happening?
This is the error:
C:\temp\apache-ignite-2.14.0-bin\bin>control.bat --set-state ACTIVE
Nov 17, 2022 4:27:17 PM
org.apache.ignite.internal.client.impl.connection.GridClientNioTcpConnection
INFO: Client TCP connection established: /127.0.0.1:11211 Nov
17, 2022 4:27:17 PM
org.apache.ignite.internal.client.impl.connection.GridClientNioTcpConnection
close INFO: Client TCP connection closed: /127.0.0.1:11211 Nov 17,
2022 4:27:17 PM org.apache.ignite.internal.client.util.GridClientUtils
shutdownNow WARNING: Runnable tasks outlived thread pool executor
service [owner=GridClientConnectionManager,
tasks=[java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask#6d7b4f4c]]
Control utility [ver. 2.14.0#20220929-sha1:951e8deb] 2022 Copyright(C)
Apache Software Foundation User: info Time: 2022-11-17T16:27:16.344
null suppressed:
Command [SET-STATE] finished with code: 4 Error stack trace: class
org.apache.ignite.internal.client.GridClientException: null
suppressed:
at org.apache.ignite.internal.client.impl.connection.GridClientNioTcpConnection.handleClientResponse(GridClientNioTcpConnection.java:632)
at org.apache.ignite.internal.client.impl.connection.GridClientNioTcpConnection.handleResponse(GridClientNioTcpConnection.java:563)
at org.apache.ignite.internal.client.impl.connection.GridClientConnectionManagerAdapter$NioListener.onMessage(GridClientConnectionManagerAdapter.java:691)
at org.apache.ignite.internal.util.nio.GridNioFilterChain$TailFilter.onMessageReceived(GridNioFilterChain.java:279)
at org.apache.ignite.internal.util.nio.GridNioFilterAdapter.proceedMessageReceived(GridNioFilterAdapter.java:109)
at org.apache.ignite.internal.util.nio.GridNioCodecFilter.onMessageReceived(GridNioCodecFilter.java:116)
at org.apache.ignite.internal.util.nio.GridNioFilterAdapter.proceedMessageReceived(GridNioFilterAdapter.java:109)
at org.apache.ignite.internal.util.nio.GridNioServer$HeadFilter.onMessageReceived(GridNioServer.java:3734)
at org.apache.ignite.internal.util.nio.GridNioFilterChain.onMessageReceived(GridNioFilterChain.java:175)
at org.apache.ignite.internal.util.nio.GridNioServer$ByteBufferNioClientWorker.processRead(GridNioServer.java:1211)
at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2508)
at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2273)
at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1910)
at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:125)
at java.lang.Thread.run(Thread.java:748)
Control utility has completed execution at: 2022-11-17T16:27:17.642
Execution time: 1298 ms Press any key to continue . . .
It's a known issue, which is, unfortunately, not fixed yet.
As a workaround, you can execute the command with the autoconfirmation flag --yes, as shown below:
control.bat --set-state ACTIVE --yes
when I deploy a Mule API and after deployment, I do some changes and save it then my API gets redeploy.
during redeployment, I am getting the below error in the console that stops the deployment.
java.lang.OutOfMemoryError: Metaspace
Dumping heap to java_pid19656.hprof ...
Heap dump file created [197920637 bytes in 0.811 secs]
#
# java.lang.OutOfMemoryError: Metaspace
# -XX:OnOutOfMemoryError="taskkill /F /PID %p"
# Executing "taskkill /F /PID 19656"...
JVM exited unexpectedly.
Automatic JVM Restarts disabled. Shutting down.
<-- Wrapper Stopped
anyone can help me to fix this issue?
Thanks
Probably you are having a known issue related to redeployments that cause the metaspace are to get exhausted. It is recommended to use the latest version of Mule and the latest version of every connector used in your applications to mitigate that issue.
Also ensure that the MetaspaceSize is half of MaxMetaspaceSize. You can increase MaxMetaspaceSize if you feel you are deploying lots of classes or applications, but keep the proportion mentioned.
wrapper.java.additional.7=-XX:MetaspaceSize=128m
wrapper.java.additional.8=-XX:MaxMetaspaceSize=256m
i installed gem5 simulator on ubuntu 14.04. then i used the youtube guide (https://www.youtube.com/watch?v=gd_DtxQD5kc) to run gem5 in full system mode in ARM architecture. first i downloaded arm-system-2011-08.tar.bz2 as mentioned in the video then i run below command:
build/ARM/gem5.opt configs/example/fs.py --disk-image=/home/morteza/full_system_images/disks/arm-ubuntu-natty-headless.img --kernel=/home/morteza/full_system_images/binaries/vmlinux.arm.smp.fb.2.6.38.8
but i encountered this output. can abybody please help me?
p.s: i added --kernel option and rename bootloader in /fulls_system_image/binaries from boot.arm to boot_emm.arm because of some errors about not finding bootloader and kernel. this is my final output which i brought hereunder. i' ll appreciate if anybody tell what is the problem.
OUTPUT:
gem5 Simulator System. http://gem5.org
gem5 is copyrighted software; use the --copyright option for details.
gem5 compiled Jan 3 2020 05:49:20
gem5 started Jan 3 2020 17:16:17
gem5 executing on morteza-pc, pid 2499
command line: build/ARM/gem5.opt configs/example/fs.py --disk-image=/home/morteza/full_system_images/disks/arm-ubuntu-natty-headless.img --kernel=/home/morteza/full_system_images/binaries/vmlinux.arm.smp.fb.2.6.38.8
warn: Can only correctly generate a dtb for VExpress_GEM5_V1 platforms, unless custom hardware models have been equipped with generation functionality.
Global frequency set at 1000000000000 ticks per second
warn: DRAM device capacity (8192 Mbytes) does not match the address range assigned (512 Mbytes)
info: kernel located at: /home/morteza/full_system_images/binaries/vmlinux.arm.smp.fb.2.6.38.8
warn: Bootloader entry point 0x80000000 overriding reset address 0
system.vncserver: Listening for connections on port 5900
system.terminal: Listening for connections on port 3456
0: system.remote_gdb: listening for remote gdb on port 7000
info: Using bootloader at address 0x80000000
info: Using kernel entry physical address at 0x80008000
warn: DTB file specified, but no device tree support in kernel
**** REAL SIMULATION ****
warn: Existing EnergyCtrl, but no enabled DVFSHandler found.
info: Entering event queue # 0. Starting simulation...
warn: Device system.membus.badaddr_responder accessed by read to address 0x10009018 size=4
gem5.opt: build/ARM/cpu/simple/atomic.cc:418: virtual Fault AtomicSimpleCPU::readMem(Addr, uint8_t*, unsigned int, Request::Flags, const std::vector<bool>&): Assertion `!pkt.isError()' failed.
Program aborted at tick 30500
--- BEGIN LIBC BACKTRACE ---
build/ARM/gem5.opt(_Z15print_backtracev+0x15)[0x1d505e5]
build/ARM/gem5.opt(_Z12abortHandleri+0x36)[0x1d5a796]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x10330)[0x7f41e3962330]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x37)[0x7f41e1eacc37]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x148)[0x7f41e1eb0028]
/lib/x86_64-linux-gnu/libc.so.6(+0x2fbf6)[0x7f41e1ea5bf6]
/lib/x86_64-linux-gnu/libc.so.6(+0x2fca2)[0x7f41e1ea5ca2]
build/ARM/gem5.opt(_ZN15AtomicSimpleCPU7readMemEmPhj5FlagsImERKSt6vectorIbSaIbEE+0x538)[0x1e4eca8]
build/ARM/gem5.opt(_ZN17SimpleExecContext7readMemEmPhj5FlagsImERKSt6vectorIbSaIbEE+0x21)[0x1e5c5b1]
build/ARM/gem5.opt(_Z13readMemAtomicI11ExecContextjESt10shared_ptrI9FaultBaseEPT_PN5Trace10InstRecordEmRT0_5FlagsImE+0x64)[0x1972e14]
build/ARM/gem5.opt(_ZNK10ArmISAInst27LOAD_IMM_AY_PN_SN_UN_WN_SZ47executeEP11ExecContextPN5Trace10InstRecordE+0x12d)[0x14f95cd]
build/ARM/gem5.opt(_ZN15AtomicSimpleCPU4tickEv+0x428)[0x1e4da58]
build/ARM/gem5.opt(_ZN10EventQueue10serviceOneEv+0xa1)[0x1d55f51]
build/ARM/gem5.opt(_Z9doSimLoopP10EventQueue+0x38)[0x1d65fc8]
build/ARM/gem5.opt(_Z8simulatem+0xaae)[0x1d66dfe]
build/ARM/gem5.opt[0x1dbbd3d]
build/ARM/gem5.opt[0xe08e85]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x45f7)[0x7f41e3579be7]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x80d)[0x7f41e357b63d]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x48d8)[0x7f41e3579ec8]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x80d)[0x7f41e357b63d]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x48d8)[0x7f41e3579ec8]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x80d)[0x7f41e357b63d]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x48d8)[0x7f41e3579ec8]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x80d)[0x7f41e357b63d]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalCode+0x32)[0x7f41e357b772]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x563e)[0x7f41e357ac2e]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x80d)[0x7f41e357b63d]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x48d8)[0x7f41e3579ec8]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x80d)[0x7f41e357b63d]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalCode+0x32)[0x7f41e357b772]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyRun_StringFlags+0x79)[0x7f41e35755a9]
--- END LIBC BACKTRACE ---
Aborted (core dumped)
I am using a Cloudera quickstart VM 5.13.0.0 to run Spark applications in yarn-client mode. I have allocated 10GB and 3 cores to my Cloudera VM. When I submit the application, the application is ACCEPTED but never moves on to RUNNING. When I try to look for logs using yarn logs -applicationId I do not see anything. Its absolutely blank.
I have looked up this issue on:
here
here
here
here
here
here
here
I have practically meddled with all the configs that these links see a problem with. I still do not have an answer to my problem which on the face of it looks like the ones in the links above. Here are the config parameters of my cloudera cluster:
mapreduce.map.memory.mb 128M
mapreduce.reduce.memory.mb 128M
mapreduce.job.heap.memory-mb.ratio 0.8
yarn.nodemanager.resource.memory-mb 1900M
yarn.nodemanager.resource.percentage-physical-cpu-limit 100
yarn.nodemanager.resource.cpu-vcores 1
yarn.scheduler.minimum-allocation-mb 1M
yarn.scheduler.increment-allocation-mb 100M
yarn.scheduler.maximum-allocation-mb 1600M
yarn.scheduler.minimum-allocation-vcores 1
yarn.scheduler.increment-allocation-vcores 1
yarn.scheduler.maximum-allocation-vcores 2
yarn.scheduler.fair.continuous-scheduling-enabled unchecked
mapreduce.am.max-attempts 1
yarn.resourcemanager.am.max-retries, yarn.resourcemanager.am.max-attempts 1
yarn.app.mapreduce.am.resource.mb 1G
yarn.app.mapreduce.am.resource.cpu-vcores 1
ApplicationMaster Java Maximum Heap Size 512M
yarn.resourcemanager.scheduler.class org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler
yarn.scheduler.fair.user-as-default-queue unchecked
yarn.scheduler.fair.preemption unchecked
yarn.scheduler.fair.preemption.cluster-utilization-threshold 0.8
yarn.scheduler.fair.sizebasedweight unchecked
Fair Scheduler Allocations (deployed) {"defaultFairSharePreemptionThreshold":null,"defaultFairSharePreemptionTimeout":null,"defaultMinSharePreemptionTimeout":null,"defaultQueueSchedulingPolicy":"drf","queueMaxAMShareDefault":-1.0,"queueMaxAppsDefault":null,"queuePlacementRules":[{"create":true,"name":"specified","queue":null,"rules":null},{"create":null,"name":"nestedUserQueue","queue":null,"rules":[{"create":true,"name":"default","queue":"users","rules":null}]},{"create":null,"name":"default","queue":null,"rules":null}],"queues":[{"aclAdministerApps":null,"aclSubmitApps":null,"allowPreemptionFrom":null,"fairSharePreemptionThreshold":null,"fairSharePreemptionTimeout":null,"minSharePreemptionTimeout":null,"name":"root","queues":[{"aclAdministerApps":null,"aclSubmitApps":null,"allowPreemptionFrom":null,"fairSharePreemptionThreshold":null,"fairSharePreemptionTimeout":null,"minSharePreemptionTimeout":null,"name":"default","queues":[],"schedulablePropertiesList":[{"impalaDefaultQueryMemLimit":null,"impalaDefaultQueryOptions":null,"impalaMaxMemory":null,"impalaMaxQueuedQueries":null,"impalaMaxRunningQueries":null,"impalaQueueTimeout":null,"maxAMShare":-1.0,"maxChildResources":null,"maxResources":null,"maxRunningApps":null,"minResources":null,"scheduleName":"default","weight":1.0}],"schedulingPolicy":"drf","type":null},{"aclAdministerApps":null,"aclSubmitApps":null,"allowPreemptionFrom":null,"fairSharePreemptionThreshold":null,"fairSharePreemptionTimeout":null,"minSharePreemptionTimeout":null,"name":"users","queues":[],"schedulablePropertiesList":[{"impalaDefaultQueryMemLimit":null,"impalaDefaultQueryOptions":null,"impalaMaxMemory":null,"impalaMaxQueuedQueries":null,"impalaMaxRunningQueries":null,"impalaQueueTimeout":null,"maxAMShare":-1.0,"maxChildResources":null,"maxResources":null,"maxRunningApps":null,"minResources":null,"scheduleName":"default","weight":1.0}],"schedulingPolicy":"drf","type":"parent"}],"schedulablePropertiesList":[{"impalaDefaultQueryMemLimit":null,"impalaDefaultQueryOptions":null,"impalaMaxMemory":null,"impalaMaxQueuedQueries":null,"impalaMaxRunningQueries":null,"impalaQueueTimeout":null,"maxAMShare":null,"maxChildResources":null,"maxResources":null,"maxRunningApps":null,"minResources":null,"scheduleName":"default","weight":1.0}],"schedulingPolicy":"drf","type":null}],"userMaxAppsDefault":1,"users":[]}
Here is what the queue description looks like when the application is still in ACCEPTED state:
Likewise, here is the record from the Yarn RM UI (Note that the resources are allocated (memory/cpu) and Running Containers shows 1 container running):
Here is the Application Summary:
Here are the application logs (empty):
And, lastly, here is what the driver sees:
enter code here19/12/26 00:16:42 INFO Client:
client token: N/A
diagnostics: Application application_1577297544619_0002 failed 1 times due to AM Container for appattempt_1577297544619_0002_000001 exited with exitCode: 10
For more detailed output, check application tracking page:http://quickstart.cloudera:8088/proxy/application_1577297544619_0002/Then, click on links to logs of each attempt.
Diagnostics: Exception from container-launch.
Container id: container_1577297544619_0002_01_000001
Exit code: 10
Stack trace: ExitCodeException exitCode=10:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:604)
at org.apache.hadoop.util.Shell.run(Shell.java:507)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:789)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:213)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Container exited with a non-zero exit code 10
Failing this attempt. Failing the application.
ApplicationMaster host: N/A
ApplicationMaster RPC port: -1
queue: root.default
start time: 1577299469533
final status: FAILED
tracking URL: http://quickstart.cloudera:8088/cluster/app/application_1577297544619_0002
user: shepanch
19/12/26 00:16:42 ERROR SparkContext: Error initializing SparkContext.
org.apache.spark.SparkException: Yarn application has already ended! It might have been killed or unable to launch application master.
at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.waitForApplication(YarnClientSchedulerBackend.scala:85)
at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:62)
at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:165)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:512)
at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2511)
at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:909)
at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:901)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:901)
at cloudera.jobs.ClouderaSampleJob$.delayedEndpoint$cloudera$jobs$ClouderaSampleJob$1(ClouderaSampleJob.scala:17)
at cloudera.jobs.ClouderaSampleJob$delayedInit$body.apply(ClouderaSampleJob.scala:6)
at scala.Function0$class.apply$mcV$sp(Function0.scala:34)
at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
at scala.App$$anonfun$main$1.apply(App.scala:76)
at scala.App$$anonfun$main$1.apply(App.scala:76)
at scala.collection.immutable.List.foreach(List.scala:392)
at scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:35)
at scala.App$class.main(App.scala:76)
at cloudera.jobs.ClouderaSampleJob$.main(ClouderaSampleJob.scala:6)
at cloudera.jobs.ClouderaSampleJob.main(ClouderaSampleJob.scala)
Is there anything that can be done to solve this issue?
After all the research and apart from the reasons mentioned in the links I have mentioned in the question, I found that this can happen due to various reasons:
when you have different versions of spark in the client (driver) and the cluster. Once you ensure that both bundle the same version of spark, it runs fine.
you might need to mention the property spark.driver.host. Make sure the IP passed in here can be pinged from the guest VM.
Iam facing JVM Crash cosistently while enabling hotdeploy (USING below java options on starting up JAVA_OPTS -Xmx4096m -XX:MetaspaceSize=512m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=crash -XX:ThreadStackSize=512 -XX:+UseConcMarkSweepGC -XX:ParallelGCThreads=5 -XX:NewRatio=2 -XX:+UnlockDiagnosticVMOptions -XX:-UseLoopPredicate -Xdebug -Xrunjdwp:transport=dt_socket,address=$DEBUG_PORT,server=y,suspend=n -XX:NewRatio=2 -Dspringloaded.synchronize=true JAVA_OPTS=`echo $JAVA_OPTS -Dspringloaded.synchronize=true -javaagent:springloaded-1.2.1.jar -noverify
)
Environment : JDK 1.8 U 66, RHEL 6.7
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x00007faee9a1e27c, pid=27208, tid=140379827795712
#
# JRE version: Java(TM) SE Runtime Environment (8.0_66-b17) (build 1.8.0_66-b17)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.66-b17 mixed mode linux-amd64 )
# Problematic frame:
# V [libjvm.so+0x35027c] Canonicalizer::do_If(If*)+0x1c
#
# Core dump written. Default location: core.27208
#
# An error report file with more information is saved as:
# hs_err_pid27208.log
# [ timer expired, abort... ]
I've noticed both -javaagent and -noverify in Java options list.
It looks like springloaded agent generates invalid bytecode, while the bytecode verification is explicitly turned off. No surprise, this may lead to unpredictable results including JVM crash.
This is not a JVM problem, but most likely a bug in springloaded agent. Try to remove -noverify option.
-XX:-TieredCompilation may also work around this particular problem, but don't expect application to work correctly if the bytecode fails to pass verification. It's better to stay away from the buggy agent libraries.
4.2.1 Crash in HotSpot Compiler Thread or Compiled Code
If the fatal error log indicates that the crash occurred in a compiler
thread, then it is possible (but not always the case) that you have
encountered a compiler bug. Similarly, if the crash is in compiled
code then it is possible that the compiler has generated incorrect
code.
In the case of the HotSpot Client VM (-client option), the compiler
thread appears in the error log as CompilerThread0. With the HotSpot
Server VM there are multiple compiler threads and these appear in the
error log file as CompilerThread0, CompilerThread1, and AdapterThread.
Below is a fragment of an error log for a compiler bug that was
encountered and fixed during the development of J2SE 5.0. The log file
shows that the HotSpot Server VM is used and the crash occurred in
CompilerThread1. In addition, the log file shows that the Current
CompileTask was the compilation of the java.lang.Thread.setPriority
method.
An unexpected error has been detected by HotSpot Virtual Machine:
:
Java VM: Java HotSpot(TM) Server VM (1.5-internal-debug mixed mode) :
--------------- T H R E A D ---------------
Current thread (0x001e9350): JavaThread "CompilerThread1" daemon
[_thread_in_vm, id=20]
Stack: [0xb2500000,0xb2580000), sp=0xb257e500, free space=505k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code,
C=native code) V [libjvm.so+0xc3b13c] :
Current CompileTask: opto: 11 java.lang.Thread.setPriority(I)V
(53 bytes)
--------------- P R O C E S S ---------------
Java Threads: ( => current thread ) 0x00229930 JavaThread "Low
Memory Detector" daemon [_thread_blocked, id=21]
=>0x001e9350 JavaThread "CompilerThread1" daemon [_thread_in_vm, id=20] :
In this case there are two potential workarounds:
The brute force approach: change the configuration so that the application is run with the -client option to specify the HotSpot
Client VM.
Assume that the bug only occurs during the compilation of the setPriority method and exclude this method from compilation.
The first approach (to use the -client option) might be trivial to
configure in some environments. In others, it might be more difficult
if the configuration is complex or if the command line to configure
the VM is not readily accessible. In general, switching from the
HotSpot Server VM to the HotSpot Client VM also reduces the peak
performance of an application. Depending on the environment, this
might be acceptable until the actual issue is diagnosed and fixed.
The second approach (exclude the method from compilation) requires
creating the file .hotspot_compiler in the working directory of the
application. Below is an example of this file:
exclude java/lang/Thread setPriority
In general the format of this file is exclude CLASS METHOD, where
CLASS is the class (fully qualified with the package name) and METHOD
is the name of the method. Constructor methods are specified as
and static initializers are specified as .
Note - The .hotspot_compiler file is an unsupported interface. It is
documented here solely for the purposes of troubleshooting and finding
a temporary workaround.
Once the application is restarted, the compiler will not attempt to
compile any of the methods listed as excluded in the .hotspot_compiler
file. In some cases this can provide temporary relief until the root
cause of the crash is diagnosed and the bug is fixed.
In order to verify that the HotSpot VM correctly located and processed
the .hotspot_compiler file that is shown in the example above, look
for the following log information at runtime. Note that the file name
separator is a dot, not a slash.
Excluding compile: java.lang.Thread::setPriority
Source
Agree with #apangin, In the program you are doing bytecode intrumentation (-agent) but specifies -noverify. When verification is turned off, you may end up such crashes.
You should not use -noverify or -Xverify:none during byte code intrumentation.
For those of you unfamiliar with bytecode verification, it is simply part of the JVM's classloading process that checks the code for certain dangerous and disallowed behavior. You can (but shouldn't) disable this protection on many JVMs by adding -Xverify:none or -noverify to the Java command line. https://blogs.oracle.com/buck/entry/never_disable_bytecode_verification_in