pig aws emr jython serialization error - jython

Question

I am trying to run a trivial Python UDF in Pig on Amazon EMR and it throws a java serialization error:
java.io.IOException: Deserialization error: could not instantiate 'org.apache.pig.scripting.jython.JythonFunction' with arguments '[/tmp/pig4877832484731242596tmp/simple.py, aprs]'
I've searched here and elsewhere and seen somewhat related questions and solutions posted, but none of the solutions seem to apply, including one post over a year ago that seemed to indicate this was working OK with Pig 0.9.1 on Amazon EMR.
$ pig --version
Apache Pig version 0.9.2-amzn (rexported)
compiled Aug 06 2012, 20:34:29
$ hadoop version
Hadoop 1.0.3
Here's my trivial python UDF:
#/usr/bin/python
#outputSchema("data:chararray")
def aprs(l):
return l
And here's the pig script invocation that shows the UDF is loaded and the #outputSchema did the right thing:
grunt> Register 's3n://n2ygk/simple.py' using jython as myudf;
grunt> raw = LOAD 's3n://aprs-is/small-sample.log' USING TextLoader as (line:chararray);
grunt> cooked = LIMIT raw 1000;
grunt> aprs = FOREACH cooked GENERATE FLATTEN(myudf.aprs(line));
grunt> DESCRIBE aprs;
aprs: {data: chararray}
grunt> dump aprs;
Any suggestions?

Answer 1

The short answer is to download, build and use Pig 0.11.1 as described with this post on the AWS forum. Once I did this, the Python UDF code "just worked".
I first did this naively by using the above-referenced script that downloads and builds Pig 0.11.1. I assumed this was likely overkill to build pig from scratch since I've been reading up more on Pig versions and believe that 0.11.1 is already available via --pig-versions. (Although the doc doesn't list 0.11.1, the pig installer script does include it.) I've used the following command to test using the EMR-distro of pig-0.11.1:
./elastic-mapreduce --create --name 'Pig11 2013-06-03-21:35:24' --alive \
--num-instances 1 --instance-type m1.small --pig-interactive \
--pig-versions 0.11.1 \
--bootstrap-action s3n://us-west-2.elasticmapreduce/bootstrap-actions/configurations/latest/memory-intensive \
--bootstrap-name 'memory intensive'
Unfortunately, this throws the following error which appears to perhaps be related to java version 1.6 vs. 1.7:
hadoop#ip-10-253-41-55:~$ pig
Exception in thread "main" java.lang.UnsupportedClassVersionError: org/apache/pig/Main : Unsupported major.minor version 51.0
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631)
at java.lang.ClassLoader.defineClass(ClassLoader.java:615)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at org.apache.hadoop.util.RunJar.main(RunJar.java:180)
hadoop#ip-10-253-41-55:~$ hadoop version
Hadoop 1.0.3
When I retested the above without specifying --pig-versions 0.11.1 it turns out that only 0.9.2 is installed in response to the default value for --pig-versions latest.
I see from (s3://us-west-2.elasticmapreduce/libs/pig/pig-script) that Pig 0.11.1 and hadoop 1.0.3 require java 7 whereas 0.9.2 requires hadoop 1.03 with java 6. I'm guessing that somehow java 7 did not get installed.
If someone can straighten me out on what I'm doing wrong, I'd really appreciate it. Otherwise, I'll live with the workaround of downloading and building it as it wastes a little time but does work.