ERROR 1066: Unable to open iterator for alias - Pig - apache-pig

Just started Pig; trying to load the data from a file and dump it henceforth. Loading seems to be proper, no error is thrown. Below is the query:
NYSE = LOAD '/root/Desktop/Works/NYSE-2000-2001.tsv' USING
PigStorage() AS (exchange:chararray, stock_symbol:chararray,
date:chararray, stock_price_open:float, stock_price_high:float,
stock_price_low:float, stock_price_close:float, stock_volume:int,
stock_price_adj_close:float);
When I try to do the Dump, it throws the following error:
Pig Stack Trace
ERROR 1066: Unable to open iterator for alias NYSE
org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to open iterator for alias NYSE
at org.apache.pig.PigServer.openIterator(PigServer.java:857)
at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:682)
at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:303)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:189)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:165)
at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:69)
at org.apache.pig.Main.run(Main.java:490)
at org.apache.pig.Main.main(Main.java:111)
Caused by: java.io.IOException: Job terminated with anomalous status FAILED
at org.apache.pig.PigServer.openIterator(PigServer.java:849)"
Any idea what's causing the issue?

Are you running a pig 0.12.0 or earlier jar against hadoop 2.2, if this is the case then
I managed to get around this error by recompiling the pig jar from src, here is a summary of the steps involved on a debian type box
download the pig-0.12.0.tar.gz
unpack the jar and set permissions
then inside the unpacked directory compile the src with 'ant clean jar -Dhadoopversion=23'
then you need to get the jar on your class-path in maven, for example, in the
same directory
mvn install:install-file -Dfile=pig.jar -DgroupId={set a groupId}-
DartifactId={set a artifactId} -Dversion=1.0 -Dpackaging=jar
or if in eclipse then add jar as external libary/dependency
I was getting your exact trace trying to run pig 12 in a hadoop 2.2.0 and the above steps worked for me
UPDATE
I posted my issue on the pig jira and they responded. They have a pig jar already compiled for hadoop2 pig-h2.jar here http://search.maven.org/#artifactdetails|org.apache.pig|pig|0.12.0|jar
a maven tag for this jar is
<dependency>
<groupId>org.apache.pig</groupId>
<artifactId>pig</artifactId>
<classifier>h2</classifier>
<version>0.12.0</version>
<scope>provided</scope>
</dependency>

This could be due to a change in the Pig Version starting 0.12. The specific change is that Pig used to be permissive and automatically ignore the first line in the data file or it would interpret that line as column names, in the new version of Pig this permissiveness was removed. The work around is to delete the column names from the input file and this should solve the problem
Kapil

I also meet this problem. And then I see this link: http://www.fanli7.net/a/JAVAbiancheng/ANT/20140325/441264.html
I just replace pig version from 0.12.0 to 0.13.0 and the problem is solved. (Here, my hadoop version is 2.3.0)

You can place breakpoint to class PigServer to method store().
for(JobStats js : stats.getJobGraph()){
if(js.getException() != null) {
ex = js.getException();
}
}
Inside the js object there is field errorMessage and it may contain description of the problem

Related

Liquibase CLI cannot execute custom changes written in Kotlin

We write our custom changes in Kotlin. But we cannot execute these custom changes using the Liquibase CLI, because the kotlin-stdlib is missing from the classpath.
Here's my liquibase.properties file:
changelog-file:db/changelog.yaml
driver:org.postgresql.Driver
url:jdbc:postgresql://localhost:5432/project
username:postgres
password:postgres
classpath:build/classes/kotlin/main:src/main/resources
includeSystemClasspath:true
In src/main/resources we keep the changelog.yaml and the changesets, while in build/classes/kotlin/main we have the custom changes written in Kotlin.
Running liquibase update --log-level=debug return the following output:
[...]
Caused by: java.lang.ClassNotFoundException: kotlin.jvm.internal.Intrinsics
at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:471)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:589)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
... 70 more
Caused by: liquibase.exception.ValidationFailedException: Validation Failed:
2 changes have validation errors
liquibase.exception.UnexpectedLiquibaseException: liquibase.exception.CustomChangeException: liquibase.exception.UnexpectedLiquibaseException: java.lang.reflect.InvocationTargetException
liquibase.exception.UnexpectedLiquibaseException: liquibase.exception.CustomChangeException: liquibase.exception.UnexpectedLiquibaseException: java.lang.reflect.InvocationTargetException
If I hard code the path to the kotlin-stdlib in the liquibase.properties file, then gradle update works fine.
classpath:build/classes/kotlin/main:src/main/resources:/Home/user/.gradle/caches/modules-2/files-2.1/org.jetbrains.kotlin/kotlin-stdlib/1.6.21/11ef67f1900634fd951bad28c53ec957fabbe5b8/kotlin-stdlib-1.6.21.jar
But we cannot hard code it in liquibase.properties as the path to the kotlin-stdlib would be different for each machine. I was expecting that exporting it in the system CLASSPATH would solve this issue:
export CLASSPATH=/Home/user/.gradle/caches/modules-2/files-2.1/org.jetbrains.kotlin/kotlin-stdlib/1.6.21
This doesn't seem to work, even when we specified includeSystemClasspath:true in the liquibase.properties file.
Using the LIQUIBASE_CLASSPATH env variable requires a Liquibase Pro or Labs license.
Liquibase Community detected and ignored the following environment variables:
- LIQUIBASE_CLASSPATH
To configure Liquibase with environment variables requires a Liquibase Pro or Liquibase Labs license. Get a free trial at https://liquibase.com/trial. Options include the liquibase.licenseKey in the defaults file, adding a flag in the CLI, and more. Learn more at https://docs.liquibase.com.

sbt.compiler.EvalException: Type Error in Expression

I'm quite new with IntelliJ (and on this site too tbh) and have been using it for only a couple of days. When I tried to build my project this exception came up. I tried to solve the problem but to no avail. I tried to see if the same problem occured on a newly created project and yes it does. So the code I've written should have no influence on the build. IƤve also checked if there might be a problem with the sbt version but I am up to date.
This is what the error looks like. I have absolutely no clue how to solve this so any help would be very much appreciated.
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=256m; support was removed in 8.0
[warn] Executing in batch mode.
[warn] For better performance, hit [ENTER] to switch to interactive mode, or
[warn] consider launching sbt without any commands, or explicitly passing 'shell'
[info] Loading global plugins from C:\Users\Johner-Institut\.sbt\0.13\plugins
Waiting for lock on C:\Users\Johner-Institut\.ivy2\.sbt.ivy.lock to be available...
[info] Loading project definition from C:\Users\Johner-Institut\Desktop\Project\project
C:\Users\Johner-Institut\Desktop\Project\build.sbt:5: error: recursive value project needs type
lazy val `project` = (project in file(".")).enablePlugins(PlayScala)
^
sbt.compiler.EvalException: Type error in expression
[error] sbt.compiler.EvalException: Type error in expression
[error] Use 'last' for the full log.
Project loading failed: (r)etry, (q)uit, (l)ast, or (i)gnore?
Don't use project as a name for sbt project, that's why the recursion error. Try something like:
lazy val root = (project in file(".")).enablePlugins(PlayScala)

java.lang.ClassNotFoundException: com.microsoft.azure.storage.blob.BlobListingDetails Exception

I am trying to read a table that is on a azure blob storage via pyspark and the below exception is raised even though I have added the below jars in the pyspark --jars.
azure-storage-2.0.0.jar
hadoop-azure-2.7.0.jar
Exception:
py4j.protocol.Py4JJavaError: An error occurred while calling o38.showString.
: java.lang.NoClassDefFoundError: com/microsoft/azure/storage/blob/BlobListingDetails
Caused by: java.lang.ClassNotFoundException: com.microsoft.azure.storage.blob.BlobListingDetails
Any idea as which specific jar needs to be added to resolve the issue and read azure tables in spark?
My suggestion is that as below.
Please download the jar files of the newest version of Azure Storage Java Client & Hadoop Azure Support instead of their old version.
Check whether the path of these jars were added into the SPARK_CLASSPATH environment variable in the conf/spark-env file, or you can programmatically add the jar path via code SparkContext.addJar("Path to jar created from maven [hint: mvn package]").
Hope it helps.

NullPointerException org.gradle.wrapper.BootstrapMainStarter.findLauncherJar(BootstrapMainStarter.java:37)

Got the following stacktrace when launching gradle 1.1, anyone know how to resolve them:
Exception in thread "main" java.lang.NullPointerException
at org.gradle.wrapper.BootstrapMainStarter.findLauncherJar(BootstrapMainStarter.java:37)
at org.gradle.wrapper.BootstrapMainStarter.start(BootstrapMainStarter.java:28)
at org.gradle.wrapper.WrapperExecutor.execute(WrapperExecutor.java:130)
at org.gradle.wrapper.GradleWrapperMain.main(GradleWrapperMain.java:47)
I think the automatic unzip of the dists/gradle-1.1-bin/13d7lnhcrghv2i5e54el41jpgr/gradle-1.1-bin.zip might be failing. I checked permissions and that I have access to that directory.
If I unzip manually, then I get the following error:
Exception in thread "main" java.lang.RuntimeException: Gradle distribution 'http://services.gradle.org/distributions/gradle-1.1-bin.zip' contains too many directories. Expected to find exactly 1 directory.
at org.gradle.wrapper.Install.createDist(Install.java:73)
at org.gradle.wrapper.WrapperExecutor.execute(WrapperExecutor.java:129)
at org.gradle.wrapper.GradleWrapperMain.main(GradleWrapperMain.java:47)
I did a google search for gradle nullpointerexception and it mentioned the JAVA_HOME needs to be set for compiling, but I've already checked it is set correctly and been able to compile stuff with ant in that environment.
I was getting exactly same error and I changed the version of gradle that I was using. Inside my gradle-wrapper.properties, changed version 2.4 to 2.2.1 and error is gone.

BuildException when executing Ant task

I have installed Apache Ant and set the correctly the Path, CLASS_PATH and ANT_PATH variables. When I run a jar task in a command line within my project folder where the build.xml file is, I get the following error:
BUILD FAILED
C:\Documents and Settings\TonyS\My Documents\My Workspace\code\build.xml: 32:
The following error occurred while executing this line:
jar:file:/C:/Documents%20and%20Settings/TonyS/My%20Documents/Ant/apache-ant-1.8.1/lib/ant.jar!org/apache/tools/ant/antlib.xml:37: Could not create task or type: componentdef.
Ant could not find the task or a class this task relies upon.
I had this error and found it was caused by having an older version of ant.jar in the classpath.
One possibility is you may have another version of ant installed on your system and your PATH pointing to it.
Check what is ANT_HOME set to.
Run ant -version and see if it shows up 1.8.1
Add the following config in ~/.bashrc
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
Don't forget source ~/.bashrc
Hope useful for you!
you have duplicate ant in your environment.
My case was:
I am using 1.9.7 ant, but ant -version gave me 1.7.1
I installed weblogic and it has 1.7.1 ant, after rename MW_HOME, the issue was gone.
I found this while I was using 1.7.1 to test junit. it shows a warning.
junit:
[junit] WARNING: multiple versions of ant detected in path for junit
[junit] jar:file:/D:/wls12120/oracle_common/modules/org.apache.ant_1.7.1/lib/ant.jar!/org/apache/tools/ant/
Project.class
[junit] and jar:file:/D:/apache-ant-1.7.1/lib/ant.jar!/org/apache/tools/ant/Project.class
[junit] Running HelloWorldTest