I am trying to read a table that is on a azure blob storage via pyspark and the below exception is raised even though I have added the below jars in the pyspark --jars.
azure-storage-2.0.0.jar
hadoop-azure-2.7.0.jar
Exception:
py4j.protocol.Py4JJavaError: An error occurred while calling o38.showString.
: java.lang.NoClassDefFoundError: com/microsoft/azure/storage/blob/BlobListingDetails
Caused by: java.lang.ClassNotFoundException: com.microsoft.azure.storage.blob.BlobListingDetails
Any idea as which specific jar needs to be added to resolve the issue and read azure tables in spark?
My suggestion is that as below.
Please download the jar files of the newest version of Azure Storage Java Client & Hadoop Azure Support instead of their old version.
Check whether the path of these jars were added into the SPARK_CLASSPATH environment variable in the conf/spark-env file, or you can programmatically add the jar path via code SparkContext.addJar("Path to jar created from maven [hint: mvn package]").
Hope it helps.
Related
I have a situation wherein all the event data is getting stored in an s3 bucket and I need to fetch that from S3 to Kafka topic on ec2. I am using CamelAWSS3Connector and am facing issues of the connector not working.
Following is the error I am facing
[2023-01-06 10:11:21,048] ERROR Failed to create job for config/s3_connect.properties (org.apache.kafka.connect.cli.ConnectStandalone:107)
[2023-01-06 10:11:21,053] ERROR Stopping after connector error (org.apache.kafka.connect.cli.ConnectStandalone:117)
java.util.concurrent.ExecutionException: java.lang.NoClassDefFoundError: org/jctools/queues/MessagePassingQueue$Supplier
at org.apache.kafka.connect.util.ConvertingFutureCallback.result(ConvertingFutureCallback.java:115)
at org.apache.kafka.connect.util.ConvertingFutureCallback.get(ConvertingFutureCallback.java:99)
at org.apache.kafka.connect.cli.ConnectStandalone.main(ConnectStandalone.java:114)
Caused by: java.lang.NoClassDefFoundError: org/jctools/queues/MessagePassingQueue$Supplier
I was expecting the publisher to push msg to topic from s3 to kafka
Following is my properties files
name=CamelAwss3SourceConnector
connector.class=org.apache.camel.kafkaconnector.aws2s3.CamelAws2s3SourceConnector
key.converter=org.apache.kafka.connect.storage.StringConverter
value.converter=org.apache.kafka.connect.converters.ByteArrayConverter
camel.source.maxPollDuration=10000
topics=mytopic
camel.component.aws-s3.access-key=XXXXXXXX
camel.component.aws-s3.region=ap-south-1
camel.source.path.bucketNameOrArn=poc-s3-kafkatopic
camel.source.endpoint.autocloseBody=true
camel.source.endpoint.deleteAfterRead=true
After using export command and adding jars location before calling the publisher following is the error
[2023-01-11 06:43:05,528] ERROR Stopping after connector error (org.apache.kafka.connect.cli.ConnectStandalone:117) java.util.concurrent.ExecutionException: java.lang.NoClassDefFoundError: org/apache/camel/kafkaconnector/CamelSourceConnectorConfig
at org.apache.kafka.connect.util.ConvertingFutureCallback.result(ConvertingFutureCallback.java:115)
at org.apache.kafka.connect.util.ConvertingFutureCallback.get(ConvertingFutureCallback.java:99)
at org.apache.kafka.connect.cli.ConnectStandalone.main(ConnectStandalone.java:114) Caused by: java.lang.NoClassDefFoundError: org/apache/camel/kafkaconnector/CamelSourceConnectorConfig
Make sure you have added plugin.path=/path/to/extracted-camel-connector to the connect-standalone.properties file.
And if that doesn't work, you'll need to export CLASSPATH environment variable to include the jar files in that path.
I'm trying to migrate my project from 12.1a weblogic to 12c and encountered error during ear deployment.
I have one ear module which contains two war submodules. during the process while I was trying to deploy ear:exploded artifact to weblogic 12c. I got nullpointer exception which related to BaseJ2eeAnnotationProcessor.getClassPersistenceContextRefs
The error log is as follows:
weblogic.management.DeploymentException:
java.lang.NullPointerException at
weblogic.application.internal.BaseDeployment.throwAppException(BaseDeployment.java:132)
at
weblogic.application.internal.BaseDeployment.prepare(BaseDeployment.java:244)
at
weblogic.application.internal.EarDeployment.prepare(EarDeployment.java:66)
at
weblogic.application.internal.DeploymentStateChecker.prepare(DeploymentStateChecker.java:158)
at
weblogic.deploy.internal.targetserver.AppContainerInvoker.prepare(AppContainerInvoker.java:65)
at
weblogic.deploy.internal.targetserver.operations.ActivateOperation.createAndPrepareContainer(ActivateOperation.java:229)
at
weblogic.deploy.internal.targetserver.operations.ActivateOperation.doPrepare(ActivateOperation.java:103)
at
weblogic.deploy.internal.targetserver.operations.AbstractOperation.prepare(AbstractOperation.java:242)
at
weblogic.deploy.internal.targetserver.DeploymentManager.handleDeploymentPrepare(DeploymentManager.java:800)
at
weblogic.deploy.internal.targetserver.DeploymentManager.prepareDeploymentList(DeploymentManager.java:1352)
at
weblogic.deploy.internal.targetserver.DeploymentManager.handlePrepare(DeploymentManager.java:270)
at
weblogic.deploy.internal.targetserver.DeploymentServiceDispatcher.prepare(DeploymentServiceDispatcher.java:177)
at
weblogic.deploy.service.internal.targetserver.DeploymentReceiverCallbackDeliverer.doPrepareCallback(DeploymentReceiverCallbackDeliverer.java:186)
at
weblogic.deploy.service.internal.targetserver.DeploymentReceiverCallbackDeliverer.access$000(DeploymentReceiverCallbackDeliverer.java:14)
at
weblogic.deploy.service.internal.targetserver.DeploymentReceiverCallbackDeliverer$1.run(DeploymentReceiverCallbackDeliverer.java:47)
at
weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:681)
at
weblogic.invocation.ComponentInvocationContextManager._runAs(ComponentInvocationContextManager.java:352)
at
weblogic.invocation.ComponentInvocationContextManager.runAs(ComponentInvocationContextManager.java:337)
at
weblogic.work.LivePartitionUtility.doRunWorkUnderContext(LivePartitionUtility.java:57)
at
weblogic.work.PartitionUtility.runWorkUnderContext(PartitionUtility.java:41)
at
weblogic.work.SelfTuningWorkManagerImpl.runWorkUnderContext(SelfTuningWorkManagerImpl.java:655)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:420) at
weblogic.work.ExecuteThread.run(ExecuteThread.java:360)
Blockquote
Caused By:
java.lang.NullPointerException at
weblogic.j2ee.dd.xml.BaseJ2eeAnnotationProcessor.getClassPersistenceContextRefs(BaseJ2eeAnnotationProcessor.java:1180)
at
weblogic.j2ee.dd.xml.J2eeAnnotationProcessor.processJ2eeAnnotations(J2eeAnnotationProcessor.java:44)
at
weblogic.j2ee.dd.xml.BaseJ2eeAnnotationProcessor.processJ2eeAnnotations(BaseJ2eeAnnotationProcessor.java:120)
at
weblogic.j2ee.dd.xml.PojoAnnotationProcessorImpl.processJ2eeAnnotations(PojoAnnotationProcessorImpl.java:93)
at
weblogic.application.internal.flow.PojoAnnotationProcessingFlow.processAnnotations(PojoAnnotationProcessingFlow.java:304)
at
weblogic.application.internal.flow.PojoAnnotationProcessingFlow.processPOJOsInModuleScopes(PojoAnnotationProcessingFlow.java:229)
at
weblogic.application.internal.flow.PojoAnnotationProcessingFlow.prepare(PojoAnnotationProcessingFlow.java:73)
at
weblogic.application.internal.BaseDeployment$1.next(BaseDeployment.java:729)
at
weblogic.application.utils.StateMachineDriver.nextState(StateMachineDriver.java:45)
at
weblogic.application.internal.BaseDeployment.prepare(BaseDeployment.java:241)
at
weblogic.application.internal.EarDeployment.prepare(EarDeployment.java:66)
at
weblogic.application.internal.DeploymentStateChecker.prepare(DeploymentStateChecker.java:158)
at
weblogic.deploy.internal.targetserver.AppContainerInvoker.prepare(AppContainerInvoker.java:65)
at
weblogic.deploy.internal.targetserver.operations.ActivateOperation.createAndPrepareContainer(ActivateOperation.java:229)
at
weblogic.deploy.internal.targetserver.operations.ActivateOperation.doPrepare(ActivateOperation.java:103)
at
weblogic.deploy.internal.targetserver.operations.AbstractOperation.prepare(AbstractOperation.java:242)
at
weblogic.deploy.internal.targetserver.DeploymentManager.handleDeploymentPrepare(DeploymentManager.java:800)
at
weblogic.deploy.internal.targetserver.DeploymentManager.prepareDeploymentList(DeploymentManager.java:1352)
at
weblogic.deploy.internal.targetserver.DeploymentManager.handlePrepare(DeploymentManager.java:270)
at
weblogic.deploy.internal.targetserver.DeploymentServiceDispatcher.prepare(DeploymentServiceDispatcher.java:177)
at
weblogic.deploy.service.internal.targetserver.DeploymentReceiverCallbackDeliverer.doPrepareCallback(DeploymentReceiverCallbackDeliverer.java:186)
at
weblogic.deploy.service.internal.targetserver.DeploymentReceiverCallbackDeliverer.access$000(DeploymentReceiverCallbackDeliverer.java:14)
at
weblogic.deploy.service.internal.targetserver.DeploymentReceiverCallbackDeliverer$1.run(DeploymentReceiverCallbackDeliverer.java:47)
at
weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:681)
at
weblogic.invocation.ComponentInvocationContextManager._runAs(ComponentInvocationContextManager.java:352)
at
weblogic.invocation.ComponentInvocationContextManager.runAs(ComponentInvocationContextManager.java:337)
at
weblogic.work.LivePartitionUtility.doRunWorkUnderContext(LivePartitionUtility.java:57)
at
weblogic.work.PartitionUtility.runWorkUnderContext(PartitionUtility.java:41)
at
weblogic.work.SelfTuningWorkManagerImpl.runWorkUnderContext(SelfTuningWorkManagerImpl.java:655)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:420) at
weblogic.work.ExecuteThread.run(ExecuteThread.java:360)
I've added several datasources in weblogic console and nothing else was added.
It would be great if somebody could help!
Match up the jpa version of the server and your application . WebLogic defaults to jpa 2.0 check in your project facets the JPA version and update WebLogic runtime to have the same JPA version
Just started Pig; trying to load the data from a file and dump it henceforth. Loading seems to be proper, no error is thrown. Below is the query:
NYSE = LOAD '/root/Desktop/Works/NYSE-2000-2001.tsv' USING
PigStorage() AS (exchange:chararray, stock_symbol:chararray,
date:chararray, stock_price_open:float, stock_price_high:float,
stock_price_low:float, stock_price_close:float, stock_volume:int,
stock_price_adj_close:float);
When I try to do the Dump, it throws the following error:
Pig Stack Trace
ERROR 1066: Unable to open iterator for alias NYSE
org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to open iterator for alias NYSE
at org.apache.pig.PigServer.openIterator(PigServer.java:857)
at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:682)
at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:303)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:189)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:165)
at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:69)
at org.apache.pig.Main.run(Main.java:490)
at org.apache.pig.Main.main(Main.java:111)
Caused by: java.io.IOException: Job terminated with anomalous status FAILED
at org.apache.pig.PigServer.openIterator(PigServer.java:849)"
Any idea what's causing the issue?
Are you running a pig 0.12.0 or earlier jar against hadoop 2.2, if this is the case then
I managed to get around this error by recompiling the pig jar from src, here is a summary of the steps involved on a debian type box
download the pig-0.12.0.tar.gz
unpack the jar and set permissions
then inside the unpacked directory compile the src with 'ant clean jar -Dhadoopversion=23'
then you need to get the jar on your class-path in maven, for example, in the
same directory
mvn install:install-file -Dfile=pig.jar -DgroupId={set a groupId}-
DartifactId={set a artifactId} -Dversion=1.0 -Dpackaging=jar
or if in eclipse then add jar as external libary/dependency
I was getting your exact trace trying to run pig 12 in a hadoop 2.2.0 and the above steps worked for me
UPDATE
I posted my issue on the pig jira and they responded. They have a pig jar already compiled for hadoop2 pig-h2.jar here http://search.maven.org/#artifactdetails|org.apache.pig|pig|0.12.0|jar
a maven tag for this jar is
<dependency>
<groupId>org.apache.pig</groupId>
<artifactId>pig</artifactId>
<classifier>h2</classifier>
<version>0.12.0</version>
<scope>provided</scope>
</dependency>
This could be due to a change in the Pig Version starting 0.12. The specific change is that Pig used to be permissive and automatically ignore the first line in the data file or it would interpret that line as column names, in the new version of Pig this permissiveness was removed. The work around is to delete the column names from the input file and this should solve the problem
Kapil
I also meet this problem. And then I see this link: http://www.fanli7.net/a/JAVAbiancheng/ANT/20140325/441264.html
I just replace pig version from 0.12.0 to 0.13.0 and the problem is solved. (Here, my hadoop version is 2.3.0)
You can place breakpoint to class PigServer to method store().
for(JobStats js : stats.getJobGraph()){
if(js.getException() != null) {
ex = js.getException();
}
}
Inside the js object there is field errorMessage and it may contain description of the problem
I'm trying to deploy Guvnor (guvnor-5.5.0.Final-tomcat-6.0) to a fresh
glassfish server ( 3.1.2.2 build 5) and am getting the error below.
I did try the other WARs without any luck and I get the same error on
OSX and windows.
Error occurred during deployment: Exception while loading the app :
java.lang.IllegalStateException: ContainerBase.addChild: start:
org.apache.catalina.LifecycleException:
java.lang.NoClassDefFoundError: org/apache/AnnotationProcessor. Please
see server.log for more details
Are there any work-around for this ?
Add the jasper-xxx.jar to your lib. It should work. If you deploy in tomcat, it should be in catalina.jar.
Unfortunately JBOSS 7 doesn't use tomcat instead it uses JBossWeb. That's why you have this problem.
Take the guvnor-5.5.0.Final-jboss-as-7.0.war binary war file
add dom4j-1.6.1.jar
remove javassit*.jars
add resteasy-jaxb-provider-2.2.3.GA.jar
add resteasy-jaxrs-2.2.3.GA.jar
Add the a glassfish-web.xml file to WEB-INF https://github.com/snowch/glassfish-guvnor/blob/master/src/main/webapp/WEB-INF/glassfish-web.xml
Replace WEB-INF/web.xml with this https://github.com/snowch/glassfish-guvnor/blob/master/src/main/webapp/WEB-INF/web.xml
Repackage the binary war file and deploy to glassfish
I created a new build configuration and tried to run the same. However I encounter errors saying the artifacts could not be found in the repository. If I run the maven build from command line for the same workspace, the build works well. The proxy configuration has been mentioned in ./m2/settings.xml and the settings file has been mentioned in the build configuration. I tried even placing the proxy configuration in TeamCity\buildAgent\plugins\maven\conf\settings.xml but it does not seem to help. Kindly let me know if you have any pointers.
Caused by: org.apache.maven.plugin.InvalidPluginException: Unable to build project for plugin 'org.apache.maven.plugins:maven-site-plugin': POM 'org.apache.maven.plugins:maven-site-plugin' not found in repository: Unable to download the artifact from any repository[16:59:48]: org.apache.maven.plugins:maven-site-plugin:pom:2.0-beta-7[16:59:48]: from the specified remote repositories:[16:59:48]: central (http://repo1.maven.org/maven2)[16:59:48]: for project org.apache.maven.plugins:maven-site-plugin[16:59:48]: at org.apache.maven.plugin.DefaultPluginManager.checkRequiredMavenVersion(DefaultPluginManager.java:293)[16:59:48]: at org.apache.maven.plugin.DefaultPluginManager.verifyVersionedPlugin(DefaultPluginManager.java:205)[16:59:48]: at org.apache.maven.plugin.DefaultPluginManager.verifyPlugin(DefaultPluginManager.java:184)[16:59:48]: at org.apache.maven.plugin.DefaultPluginManager.loadPluginDescriptor(DefaultPluginManager.java:1642)[16:59:48]: at org.apache.maven.lifecycle.DefaultLifecycleExecutor.verifyPlugin(DefaultLifecycleExecutor.java:1540)[16:59:48]: ... 19 more[16:59:48]: Caused by: org.apache.maven.project.ProjectBuildingException: POM 'org.apache.maven.plugins:maven-site-plugin' not found in repository: Unable to download the artifact from any repository[16:59:48]: org.apache.maven.plugins:maven-site-plugin:pom:2.0-beta-7[16:59:48]: from the specified remote repositories:[16:59:48]: central (http://repo1.maven.org/maven2)[16:59:48]: for project org.apache.maven.plugins:maven-site-plugin[16:59:48]: at org.apache.maven.project.DefaultMavenProjectBuilder.findModelFromRepository(DefaultMavenProjectBuilder.java:605)[16:59:48]: at org.apache.maven.project.DefaultMavenProjectBuilder.buildFromRepository(DefaultMavenProjectBuilder.java:251)[16:59:48]: at org.apache.maven.plugin.DefaultPluginManager.checkRequiredMavenVersion(DefaultPluginManager.java:277)[16:59:48]: ... 23 more[16:59:48]: Caused by: org.apache.maven.artifact.resolver.ArtifactNotFoundException: Unable to download the artifact from any repository[16:59:48]: org.apache.maven.plugins:maven-site-plugin:pom:2.0-beta-7[16:59:48]: from the specified remote repositories:[16:59:48]: central (http://repo1.maven.org/maven2)[16:59:48]: at org.apache.maven.artifact.resolver.DefaultArtifactResolver.resolve(DefaultArtifactResolver.java:228)[16:59:48]: at org.apache.maven.artifact.resolver.DefaultArtifactResolver.resolve(DefaultArtifactResolver.java:90)[16:59:48]: at org.apache.maven.project.DefaultMavenProjectBuilder.findModelFromRepository(DefaultMavenProjectBuilder.java:558)[16:59:48]: ... 25 more[16:59:48]: Caused by: org.apache.maven.wagon.ResourceDoesNotExistException: Unable to download the artifact from any repository[16:59:48]: at org.apache.maven.artifact.manager.DefaultWagonManager.getArtifact(DefaultWagonManager.java:404)[16:59:48]: at org.apache.maven.artifact.resolver.DefaultArtifactResolver.resolve(DefaultArtifactResolver.java:216)[16:59:48]: ... 27 more
Have you tried setting the settings.xml within the Maven installation
e.g. for my TeamCity installation /usr/local/apache-maven/apache-maven-2.2.1/conf/settings.xml
Don't forget that TeamCity will probably be running as its own user, so ~/.m2/settings.xml may not be where you think it is.