Apache Twill HelloWorld example fails - hadoop-yarn

Question

When I run Apache twill HelloWorld example as stated in http://twill.incubator.apache.org/GettingStarted.html I get this log:
10:44:47.888 [ STARTING] DEBUG o.a.twill.yarn.YarnTwillController -
Yarn application status for HelloWorldRunnable
application_1443786884805_0185: ACCEPTED
10:44:48.383 [ STARTING-SendThread(hadice.dev:2181)] DEBUG org.apache.zookeeper.ClientCnxn -
Got ping response for sessionid: 0x15028da0ff0009d after 0ms
10:44:48.889 [IPC Parameter Sending Thread #0] DEBUG org.apache.hadoop.ipc.Client -
IPC Client (431687661) connection to gin1.dev/10.0.22.129:8032 from root sending #45
10:44:48.894 [IPC Client (431687661) connection to gin1.dev/10.0.22.129:8032 from root]
DEBUG org.apache.hadoop.ipc.Client -
IPC Client (431687661) connection to gin1.dev/10.0.22.129:8032 from root got value #45
10:44:48.894 [ STARTING] DEBUG o.a.hadoop.ipc.ProtobufRpcEngine - Call: getApplicationReport took 6ms
10:44:48.895 [ STARTING] DEBUG o.a.twill.yarn.YarnTwillController -
Yarn application status for HelloWorldRunnable application_1443786884805_0185:
ACCEPTED
10:44:49.711 [Kafka-Consumer-log-0] DEBUG o.a.t.i.k.client.SimpleKafkaConsumer -
No leader for topic partition TopicPartition{topic=log, partition=0}.
10:44:49.895 [IPC Parameter Sending Thread #0] DEBUG org.apache.hadoop.ipc.Client -
IPC Client (431687661) connection to gin1.dev/10.0.22.129:8032 from root sending #46
10:44:49.902 [IPC Client (431687661) connection to gin1.dev/10.0.22.129:8032 from root] DEBUG org.apache.hadoop.ipc.Client -
IPC Client (431687661) connection to gin1.dev/10.0.22.129:8032 from root got value #46
10:44:49.902 [ STARTING] DEBUG o.a.hadoop.ipc.ProtobufRpcEngine - Call: getApplicationReport took 7ms
10:44:49.902 [ STARTING] DEBUG o.a.twill.yarn.YarnTwillController -
Yarn application status for HelloWorldRunnable application_1443786884805_0185:
FAILED
10:44:50.902 [ STARTING] INFO o.a.twill.yarn.YarnTwillController -
Yarn application HelloWorldRunnable application_1443786884805_0185 is in state
FAILED
10:44:50.903 [ STARTING] INFO o.a.twill.yarn.YarnTwillController -
Yarn application HelloWorldRunnable application_1443786884805_0185 is not in running state. Shutting down controller.
10:44:50.907 [IPC Parameter Sending Thread #0] DEBUG org.apache.hadoop.ipc.Client -
IPC Client (431687661) connection to gin1.dev/10.0.22.129:8032 from root sending #47
10:44:50.908 [ STARTING-SendThread(hadice.dev:2181)] DEBUG org.apache.zookeeper.ClientCnxn -
Reading reply sessionid:0x15028da0ff0009d, packet::
clientPath:/HelloWorldRunnable/instances/5e72cb8c-cf94-4718-a44b-ec983304efa0
serverPath:/HelloWorldRunnable/instances/5e72cb8c-cf94-4718-a44b-ec983304efa0
finished:false header:: 10,3 replyHeader:: 10,1797,-101 request:: '/HelloWorldRunnable/instances/5e72cb8c-cf94-4718-a44b-ec983304efa0,T response::
10:44:50.913 [IPC Client (431687661) connection to gin1.dev/10.0.22.129:8032 from root] DEBUG org.apache.hadoop.ipc.Client -
IPC Client (431687661) connection to gin1.dev/10.0.22.129:8032 from root got value #47
10:44:50.913 [ STOPPING] DEBUG o.a.hadoop.ipc.ProtobufRpcEngine - Call: getApplicationReport took 6ms
10:44:50.916 [ STOPPING] DEBUG o.a.twill.yarn.YarnTwillController -
Yarn application HelloWorldRunnable application_1443786884805_0185 completed with status
FAILED
The Application gets ACCEPTed but then transitions to the "FAILED" state.
The YARN Web UI shows this as the error (very unspecific):
Application application_1443786884805_0185 failed 2 times due to AM Container for appattempt_1443786884805_0185_000002 exited with exitCode: 1
For more detailed output, check application tracking page:http://gin1.dev:8088/proxy/application_1443786884805_0185/Then, click on links to logs of each attempt.
Diagnostics: Exception from container-launch.
Container id: container_1443786884805_0185_02_000001
Exit code: 1
Stack trace: ExitCodeException exitCode=1:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
at org.apache.hadoop.util.Shell.run(Shell.java:455)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:211)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Container exited with a non-zero exit code 1
Failing this attempt. Failing the application.
And the node log shows:
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/conf/Configuration
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Class.java:2693)
at java.lang.Class.privateGetMethodRecursive(Class.java:3040)
at java.lang.Class.getMethod0(Class.java:3010)
at java.lang.Class.getMethod(Class.java:1776)
at org.apache.twill.launcher.TwillLauncher.main(TwillLauncher.java:85)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.conf.Configuration
at java.net.URLClassLoader$1.run(URLClassLoader.java:372)
at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 6 more
What can be wrong? I supposed that hadoop classes should be on classpath in yarn application. How to fix that?

Answer 1

I assume you are launching the application with the result of hadoop classpath in the classpath (as shown in the example). You need to make sure the result of running hadoop classpath on launcher box points to local paths that contain hadoop jars. The other thing you might want to check is the stdout file in the container log directory. It prints out classpath that it uses to launch the application and see if you find hadoop jars there.