spark.read.parquet not working in Colab - amazon-s3

Py4JJavaError: An error occurred while calling o188.parquet.
: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.s3a.S3AFileSystem not found
I tried adding the missing hadoop-aws jar file using spark-submit to the classpath but was unable to add it. This is what I tried:
!spark-submit --jars /content/hadoop-aws-2.7.1.jar
Exception in thread "main" java.lang.IllegalArgumentException: Missing application resource.

os.environ['PYSPARK_SUBMIT_ARGS'] = "--packages=org.apache.hadoop:hadoop-aws:2.7.3 pyspark-shell"

Related

Endeca deployment - Initialize service.bat command failed

Please find the error when I run the initialize_services command in cmd
SEVERE: Caught an exception while invoking method 'run' on object 'InitialSetup'. Releasing locks.
Caused by java.lang.reflect.InvocationTargetException
sun.reflect.NativeMethodAccessorImpl invoke0 - null
Caused by com.endeca.soleng.eac.toolkit.exception.AppControlException
com.endeca.soleng.eac.toolkit.script.Script runBeanShellScript - Unknown error executing a BeanShell script.
Caused by bsh.EvalError
bsh.BSHMethodInvocation eval - Sourced file: inline evaluation of: `` IFCR.provisionSite(); CAS.importDimensionValueIdMappings("Disco . . . '' : Error in method invocation: Method importDimensionValueIdMappings( java.lang.String, java.lang.String ) not found in class'com.endeca.soleng.eac.toolkit.component.CustomComponent'
Failure to initialize EAC application.
According to this adding casStubs.jar from CAS_ROOT\lib\cas-dt\ to classpath in runcommand script will solve the problem.
set CLASSPATH=%CLASSPATH%;C:\Endeca\CAS\11.1.0\lib\cas-dt\casStubs.jar

ClassNotFoundException when trying to execute a jar file in serenity-jbehave

I'm getting the error below:
Exception in thread "main"java.lang.NoClassDefFoundError:net/serenitybdd/jbehave/SerenityStories
Caused by: java.lang.ClassNotFoundException: net.serenitybdd.jbehave.SerenityStories
That's a dependency conflict. Check that you are using the latest versions for all of the serenity jars.

While compiling jar file i'm getting " A JNI error has occurred" error Exception in thread "main" java.lang.NoClassDefFoundError: org/openqa/selenium

While compiling jar file i'm getting
" A JNI error has occurred, please check your installation and try
again"
Exception in thread "main" java.lang.NoClassDefFoundError:
org/openqa/selenium/I nvalidArgumentException
I've created my Jar file of my made selenium functions. I've not able to run my jar file. I've set Selenium jars in CLASS PATH though I'm getting this error. need Help.

Not able to start Datastax Cassandra due to missing SystemInfo class

I have installed Datastax Cassandra 5.0.4 on my mac but when i try to start, it is giving me error for SystemInfo class.
Here is related log:
Exception (com.google.inject.ProvisionException) encountered during startup: Unable to provision, see the following errors:
1) Error injecting constructor, java.lang.NoClassDefFoundError: com/datastax/hadoop/hive/util/SystemInfo
at com.datastax.bdp.plugin.HiveMetaStorePlugin.(HiveMetaStorePlugin.java:39)
at com.datastax.bdp.DseHiveModule.configure(Unknown Source) (via modules: com.datastax.bdp.DseModule -> com.datastax.bdp.DseAnalyticsModule -> com.datastax.bdp.DseHiveModule)
while locating com.datastax.bdp.plugin.HiveMetaStorePlugin
1 error
com.google.inject.ProvisionException: Unable to provision, see the following errors:
1) Error injecting constructor, java.lang.NoClassDefFoundError: com/datastax/hadoop/hive/util/SystemInfo
at com.datastax.bdp.plugin.HiveMetaStorePlugin.(HiveMetaStorePlugin.java:39)
at com.datastax.bdp.DseHiveModule.configure(Unknown Source) (via modules: com.datastax.bdp.DseModule -> com.datastax.bdp.DseAnalyticsModule -> com.datastax.bdp.DseHiveModule)
while locating com.datastax.bdp.plugin.HiveMetaStorePlugin
1 error
at com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1025)
at com.google.inject.internal.InjectorImpl.getInstance(InjectorImpl.java:1051)
at com.datastax.bdp.plugin.PluginManager.register(PluginManager.java:139)
at com.datastax.bdp.plugin.PluginManager.postSetup(PluginManager.java:68)
at com.datastax.bdp.server.DseDaemon.postSetup(DseDaemon.java:824)
at com.datastax.bdp.server.DseDaemon.setup(DseDaemon.java:447)
at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:568)
at com.datastax.bdp.DseModule.main(DseModule.java:91)
Caused by: java.lang.NoClassDefFoundError: com/datastax/hadoop/hive/util/SystemInfo
at com.datastax.bdp.plugin.HiveMetaStorePlugin.(HiveMetaStorePlugin.java:40)
at com.datastax.bdp.plugin.HiveMetaStorePlugin$$FastClassByGuice$$7e2893c3.newInstance()
at com.google.inject.internal.cglib.reflect.$FastConstructor.newInstance(FastConstructor.java:40)

Oozie hive action fails

I am creating oozie workflow for hive create table command.
I have added hive-site.xml in hdfs location.
I am getting below error:-
Launcher ERROR, reason: Main class [org.apache.oozie.action.hadoop.HiveMain], main() threw exception, com/facebook/fb303/FacebookService$Iface
java.lang.NoClassDefFoundError: com/facebook/fb303/FacebookService$Iface
at java.lang.ClassLoader.defineClass1(Native Method)
This might be because you are missing Thrift jar or version mismatch.
Refer the following
Error while executing program with Hive JDBC