Simplest way to auto-restart a JVM on StackOverflowError - error-handling

Question

It does not seem that there is a -XX option to restart a JVM on StackOverflowError. What is the simplest way to auto-restart a JVM when it gets a StackOverflowError?

Answer 1

HotSpot JVM has built-in -XX:AbortVMOnException=java.lang.StackOverflowError option, but unfortunately this flag is available only in debug builds of JVM.
The working solution is to use JVM TI agent that will intercept all exceptions and abort the process whenever the exception belongs to the specified class. Here is an example of such agent.
#include <jvmti.h>
#include <string.h>
#include <stdio.h>
#include <stdlib.h>
static const char* fatal_error_class;
void JNICALL ExceptionCallback(jvmtiEnv* jvmti, JNIEnv* env, jthread thread,
jmethodID method, jlocation location, jobject exception,
jmethodID catch_method, jlocation catch_location) {
char* class_name;
jclass exception_class = env->GetObjectClass(exception);
jvmti->GetClassSignature(exception_class, &class_name, NULL);
class_name[strlen(class_name) - 1] = 0;
if (strcmp(class_name + 1, fatal_error_class) == 0) {
printf("Abort on fatal error\n");
exit(1);
}
jvmti->Deallocate((unsigned char*)class_name);
}
extern "C" JNIEXPORT jint JNICALL Agent_OnLoad(JavaVM* vm, char* options, void* unused) {
if (options == NULL || options[0] == 0) {
printf("Usage: -agentpath:/path/to/libabort.so=java/lang/StackOverflowError\n");
return 1;
}
fatal_error_class = strdup(options);
jvmtiEnv* jvmti;
vm->GetEnv((void**)&jvmti, JVMTI_VERSION_1_0);
jvmtiCapabilities capabilities = {0};
capabilities.can_generate_exception_events = 1;
jvmti->AddCapabilities(&capabilities);
jvmtiEventCallbacks callbacks = {0};
callbacks.Exception = ExceptionCallback;
jvmti->SetEventCallbacks(&callbacks, sizeof(callbacks));
jvmti->SetEventNotificationMode(JVMTI_ENABLE, JVMTI_EVENT_EXCEPTION, NULL);
return 0;
}
How to compile it:
g++ -I $JAVA_HOME/include -I $JAVA_HOME/include/linux -fPIC -shared -olibabort.so abort.cpp
How to run:
java -agentpath:/path/to/libabort.so=java/lang/StackOverflowError ...