Using Janino in YARN with Apache Twill causes "Imported class x.y could not be loaded" - hadoop-yarn

Question

I'm porting an open source project, which uses Janino for dynamic compilation of classes, to YARN, using Apache Twill. This works great except one last error. When Janino is used with twill, I'm getting an exception that a class cannot be found, although the class in in the Classpath and even used.
The exception I'm getting is:
2014-06-09T18:30:40,093Z ERROR o.a.d.e.p.i.p.ProjectRecordBatch [zk1]
[37daf04b-7d82-4d2f-987c-59851f2aeafe:frag:0:0]
AbstractSingleRecordBatch:next(AbstractSingleRecordBatch.java:60) -
Failure during query
org.apache.drill.exec.exception.SchemaChangeException: Failure while
attempting to load generated class
at org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.setupNewSchema(ProjectRecordBatch.java:243)
at org.apache.drill.exec.record.AbstractSingleRecordBatch.next(AbstractSingleRecordBatch.java:57)
at org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.next(ProjectRecordBatch.java:83)
at org.apache.drill.exec.record.AbstractSingleRecordBatch.next(AbstractSingleRecordBatch.java:45)
at org.apache.drill.exec.physical.impl.limit.LimitRecordBatch.next(LimitRecordBatch.java:99)
at org.apache.drill.exec.record.AbstractSingleRecordBatch.next(AbstractSingleRecordBatch.java:45)
at org.apache.drill.exec.physical.impl.svremover.RemovingRecordBatch.next(RemovingRecordBatch.java:94)
at org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.next(ScreenCreator.java:80)
at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:104)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744) Caused by: org.apache.drill.exec.exception.ClassTransformationException: Failure
Generating transformation classes for value:
package org.apache.drill.exec.test.generated;
import org.apache.drill.exec.exception.SchemaChangeException;
import org.apache.drill.exec.expr.holders.BitHolder;
import org.apache.drill.exec.expr.holders.VarCharHolder;
import org.apache.drill.exec.ops.FragmentContext;
import org.apache.drill.exec.record.RecordBatch;
import org.apache.drill.exec.vector.RepeatedVarCharVector;
import org.apache.drill.exec.vector.VarCharVector;
import org.apache.drill.exec.vector.complex.impl.RepeatedVarCharReaderImpl;
public class ProjectorGen0 {
RepeatedVarCharVector vv0;
RepeatedVarCharReaderImpl reader4;
VarCharVector vv5;
public boolean doEval(int inIndex, int outIndex)
throws SchemaChangeException
{
{
VarCharHolder out3 = new VarCharHolder();
complex:
vv0 .getAccessor().getReader().setPosition((inIndex));
reader4 .read(0, out3);
BitHolder out8 = new BitHolder();
out8 .value = 1;
if (!vv5 .getMutator().setSafe((outIndex), out3)) {
out8 .value = 0;
}
if (out8 .value == 0) {
return false;
}
}
{
return true;
}
}
public void doSetup(FragmentContext context, RecordBatch incoming, RecordBatch outgoing)
throws SchemaChangeException
{
{
int[] fieldIds1 = new int[ 1 ] ;
fieldIds1 [ 0 ] = 0;
Object tmp2 = (incoming).getValueAccessorById(RepeatedVarCharVector.class, fieldIds1).getValueVector();
if (tmp2 == null) {
throw new SchemaChangeException("Failure while loading vector vv0 with id: org.apache.drill.exec.record.TypedFieldId#1cf4a5a0.");
}
vv0 = ((RepeatedVarCharVector) tmp2);
reader4 = ((RepeatedVarCharReaderImpl) vv0 .getAccessor().getReader());
int[] fieldIds6 = new int[ 1 ] ;
fieldIds6 [ 0 ] = 0;
Object tmp7 = (outgoing).getValueAccessorById(VarCharVector.class, fieldIds6).getValueVector();
if (tmp7 == null) {
throw new SchemaChangeException("Failure while loading vector vv5 with id: org.apache.drill.exec.record.TypedFieldId#1ce776c0.");
}
vv5 = ((VarCharVector) tmp7);
}
}
}
at
org.apache.drill.exec.compile.ClassTransformer.getImplementationClass(ClassTransformer.java:302)
at
org.apache.drill.exec.ops.FragmentContext.getImplementationClass(FragmentContext.java:185)
at
org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.setupNewSchema(ProjectRecordBatch.java:240)
at
org.apache.drill.exec.record.AbstractSingleRecordBatch.next(AbstractSingleRecordBatch.java:57)
at
org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.next(ProjectRecordBatch.java:83)
at
org.apache.drill.exec.record.AbstractSingleRecordBatch.next(AbstractSingleRecordBatch.java:45)
at
org.apache.drill.exec.physical.impl.limit.LimitRecordBatch.next(LimitRecordBatch.java:99)
at
org.apache.drill.exec.record.AbstractSingleRecordBatch.next(AbstractSingleRecordBatch.java:45)
at
org.apache.drill.exec.physical.impl.svremover.RemovingRecordBatch.next(RemovingRecordBatch.java:94)
at
org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.next(ScreenCreator.java:80)
at
org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:104)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744) Caused by:
org.codehaus.commons.compiler.CompileException: Line 4, Column 8:
Imported class "org.apache.drill.exec.exception.SchemaChangeException"
could not be loaded at
org.codehaus.janino.UnitCompiler.compileError(UnitCompiler.java:9014)
at org.codehaus.janino.UnitCompiler.import2(UnitCompiler.java:192) at
org.codehaus.janino.UnitCompiler.access$000(UnitCompiler.java:104) at
org.codehaus.janino.UnitCompiler$1.visitSingleTypeImportDeclaration(UnitCompiler.java:166)
at
org.codehaus.janino.Java$CompilationUnit$SingleTypeImportDeclaration.accept(Java.java:171)
at org.codehaus.janino.UnitCompiler.(UnitCompiler.java:164) at
org.apache.drill.exec.compile.JaninoClassCompiler.getClassByteCode(JaninoClassCompiler.java:53)
at
org.apache.drill.exec.compile.QueryClassLoader.getClassByteCode(QueryClassLoader.java:69)
at
org.apache.drill.exec.compile.ClassTransformer.getImplementationClass(ClassTransformer.java:256)
at
org.apache.drill.exec.ops.FragmentContext.getImplementationClass(FragmentContext.java:185)
at
org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.setupNewSchema(ProjectRecordBatch.java:240)
at
org.apache.drill.exec.record.AbstractSingleRecordBatch.next(AbstractSingleRecordBatch.java:57)
at
org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.next(ProjectRecordBatch.java:83)
at
org.apache.drill.exec.record.AbstractSingleRecordBatch.next(AbstractSingleRecordBatch.java:45)
at
org.apache.drill.exec.physical.impl.limit.LimitRecordBatch.next(LimitRecordBatch.java:99)
at
org.apache.drill.exec.record.AbstractSingleRecordBatch.next(AbstractSingleRecordBatch.java:45)
at
org.apache.drill.exec.physical.impl.svremover.RemovingRecordBatch.next(RemovingRecordBatch.java:94)
at
org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.next(ScreenCreator.java:80)
at
org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:104)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
As you can see, the type of the exception is SchemaChangeException but the internal exception is a ClassNotFoundException for SchemaChangeException:
Line 4, Column 8: Imported class
"org.apache.drill.exec.exception.SchemaChangeException" could not be
loaded
So there is something wrong with the class loader, which changes when the application is run with Apache Twill. It works stand alone, but in both cases the underlying jars are identical.
Apache Twill also has a function to add additional resources, but adding my jar there didn't work either, instead I got an exception, that the jar is already included:
Exception in thread "ServiceDelegate STARTING" java.lang.RuntimeException: java.util.zip.ZipException: duplicate entry: lib/drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar
at com.google.common.base.Throwables.propagate(Throwables.java:160) at
org.apache.twill.yarn.YarnTwillController.doStartUp(YarnTwillController.java:133)
at
org.apache.twill.internal.AbstractZKServiceController.startUp(AbstractZKServiceController.java:82)
at
org.apache.twill.internal.AbstractExecutionServiceController$ServiceDelegate.startUp(AbstractExecutionServiceController.java:109)
at
com.google.common.util.concurrent.AbstractIdleService$1$1.run(AbstractIdleService.java:43)
at java.lang.Thread.run(Thread.java:744) Caused by:
java.util.zip.ZipException: duplicate entry:
lib/drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar at
java.util.zip.ZipOutputStream.putNextEntry(ZipOutputStream.java:215)
at
java.util.jar.JarOutputStream.putNextEntry(JarOutputStream.java:109)
at
org.apache.twill.internal.ApplicationBundler.copyResource(ApplicationBundler.java:347)
at
org.apache.twill.internal.ApplicationBundler.createBundle(ApplicationBundler.java:140)
at
org.apache.twill.yarn.YarnTwillPreparer.createContainerJar(YarnTwillPreparer.java:388)
at
org.apache.twill.yarn.YarnTwillPreparer.access$300(YarnTwillPreparer.java:106)
at
org.apache.twill.yarn.YarnTwillPreparer$1.call(YarnTwillPreparer.java:264)
at
org.apache.twill.yarn.YarnTwillPreparer$1.call(YarnTwillPreparer.java:253)
at
org.apache.twill.yarn.YarnTwillController.doStartUp(YarnTwillController.java:98)
... 4 more
The underlying classloader used is the URLClassLoader. It's initialized with an empty array, but it works for the stand alone application, the problem is only when it runs with Apache Twill, where does it get the URLs it should look up from? How could I check it?
The classloader definition:
public class QueryClassLoader extends URLClassLoader {
static final org.slf4j.Logger logger = org.slf4j.LoggerFactory.getLogger(QueryClassLoader.class);
private final ClassCompiler classCompiler;
private AtomicLong index = new AtomicLong(0);
private ConcurrentMap<String, byte[]> customClasses = new MapMaker().concurrencyLevel(4).makeMap();
public QueryClassLoader(boolean useJanino) {
super(new URL[0]);
if (useJanino) {
this.classCompiler = new JaninoClassCompiler(this);
} else {
throw new UnsupportedOperationException("Drill no longer supports using the JDK class compiler.");
}
}
...
Any ideas where I could look into, why the error occurs or how to solve it?

Answer 1

The same question was asked in the Apache Twill mailing list. Here is the discussion and proposed solution to it.
http://mail-archives.apache.org/mod_mbox/twill-dev/201406.mbox/%3CCAHqY-MOa8jBYs%3DEZENxxNZg-9YGMR5SASg76P_k6%2Bm6p2L9JuQ%40mail.gmail.com%3E
Repeat my answer in the mail content:
I am not familiar with how janino works, but it seems to me that it may not be using context ClassLoader to load classes or as least the thread that is compiling the generated class does not have the context
ClassLoader set properly.
The way that Twill works is pretty straightforward. It creates a "launcher.jar", which has no dependency on any library and start the JVM in a YARN container like this:
java -cp launcher.jar ....
Hence the system classloader has no user/library classes, but only the Launcher class.
Then in the Launcher.main() method, it creates a URLClassLoader, using all the jars + .class files inside the "container.jar" file, to load the user TwillRunnable. It also sets it as the context ClassLoader of the thread that calls the "run()" method. So, if you want to load class manually (through ClassLoader or Class.forName) in a different thread than the "run()" thread, you'll have to use set the context ClassLoader of that thread or explicitly construct the ClassLoader with the correct parent ClassLoader.