Read Remote S3 File Using Databricks Connect - amazon-s3

I am trying to read a file in an S3 bucket using Spark through Databricks Connect.
This is the code that I am using,
from pyspark import SparkConf
from pyspark.sql import SparkSession
conf = SparkConf()
conf.set('spark.jars.packages', 'org.apache.hadoop:hadoop-aws:3.3.0')
conf.set('spark.hadoop.fs.s3a.access.key', access_key)
conf.set('spark.hadoop.fs.s3a.secret.key', secret_access_key)
spark = SparkSession.builder.config(conf=conf).getOrCreate()
df = spark.read.format("csv").option("header",True).load('s3a://container/path/to/file.csv')
df.show()
This works completely fine when I execute it using a Docker container that I spin up, however, it fails with Databricks Connect with the following error,
pyspark.dbutils.ExecutionError: An error occurred while calling o48.ls.
: com.databricks.service.SparkServiceRemoteException: java.nio.file.AccessDeniedException: getFileStatus on com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden; request: HEAD Forbidden
at shaded.databricks.org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:244)
at shaded.databricks.org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:155)
at shaded.databricks.org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:2870)
at shaded.databricks.org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:2840)
at shaded.databricks.org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:2779)
at shaded.databricks.org.apache.hadoop.fs.s3a.S3AFileSystem.innerListStatus(S3AFileSystem.java:2449)
at shaded.databricks.org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$listStatus$11(S3AFileSystem.java:2428)
at shaded.databricks.org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:118)
at shaded.databricks.org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:112)
at shaded.databricks.org.apache.hadoop.fs.s3a.S3AFileSystem.listStatus(S3AFileSystem.java:2428)
at com.databricks.service.SparkServiceImpl$.$anonfun$fileSystemOperation0$2(SparkServiceImpl.scala:617)
at com.databricks.service.SparkServiceImpl$.withFileSystemExceptionHandler(SparkServiceImpl.scala:647)
at com.databricks.service.SparkServiceImpl$.fileSystemOperation0(SparkServiceImpl.scala:617)
at com.databricks.service.SparkServiceImpl$.$anonfun$fileSystemOperation$1(SparkServiceImpl.scala:184)
at com.databricks.logging.UsageLogging.$anonfun$recordOperation$4(UsageLogging.scala:431)
at com.databricks.logging.UsageLogging.$anonfun$withAttributionContext$1(UsageLogging.scala:239)
at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62)
at com.databricks.logging.UsageLogging.withAttributionContext(UsageLogging.scala:234)
at com.databricks.logging.UsageLogging.withAttributionContext$(UsageLogging.scala:231)
at com.databricks.spark.util.PublicDBLogging.withAttributionContext(DatabricksSparkUsageLogger.scala:19)
at com.databricks.logging.UsageLogging.withAttributionTags(UsageLogging.scala:276)
at com.databricks.logging.UsageLogging.withAttributionTags$(UsageLogging.scala:269)
at com.databricks.spark.util.PublicDBLogging.withAttributionTags(DatabricksSparkUsageLogger.scala:19)
at com.databricks.logging.UsageLogging.recordOperation(UsageLogging.scala:412)
at com.databricks.logging.UsageLogging.recordOperation$(UsageLogging.scala:338)
at com.databricks.spark.util.PublicDBLogging.recordOperation(DatabricksSparkUsageLogger.scala:19)
at com.databricks.spark.util.PublicDBLogging.recordOperation0(DatabricksSparkUsageLogger.scala:56)
at com.databricks.spark.util.DatabricksSparkUsageLogger.recordOperation(DatabricksSparkUsageLogger.scala:131)
at com.databricks.spark.util.UsageLogger.recordOperation(UsageLogger.scala:71)
at com.databricks.spark.util.UsageLogger.recordOperation$(UsageLogger.scala:58)
at com.databricks.spark.util.DatabricksSparkUsageLogger.recordOperation(DatabricksSparkUsageLogger.scala:85)
at com.databricks.spark.util.UsageLogging.recordOperation(UsageLogger.scala:401)
at com.databricks.spark.util.UsageLogging.recordOperation$(UsageLogger.scala:380)
at com.databricks.service.SparkServiceImpl$.recordOperation(SparkServiceImpl.scala:92)
at com.databricks.service.SparkServiceImpl$.fileSystemOperation(SparkServiceImpl.scala:184)
at com.databricks.service.SparkServiceRPCHandler.execute0(SparkServiceRPCHandler.scala:663)
at com.databricks.service.SparkServiceRPCHandler.$anonfun$executeRPC0$1(SparkServiceRPCHandler.scala:451)
at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62)
at com.databricks.service.SparkServiceRPCHandler.executeRPC0(SparkServiceRPCHandler.scala:351)
at com.databricks.service.SparkServiceRPCHandler$$anon$2.call(SparkServiceRPCHandler.scala:302)
at com.databricks.service.SparkServiceRPCHandler$$anon$2.call(SparkServiceRPCHandler.scala:288)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at com.databricks.service.SparkServiceRPCHandler.$anonfun$executeRPC$1(SparkServiceRPCHandler.scala:338)
at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62)
at com.databricks.service.SparkServiceRPCHandler.executeRPC(SparkServiceRPCHandler.scala:315)
at com.databricks.service.SparkServiceRPCServlet.doPost(SparkServiceRPCServer.scala:152)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:873)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:542)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:205)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at org.eclipse.jetty.server.Server.handle(Server.java:505)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:370)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:267)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:698)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:804)
at java.lang.Thread.run(Thread.java:750)
Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden; request: HEAD
Note: I've scrubbed some information related to the path of my file in the above error thread.

Related

Dbutils() commands throwing strange permissions exception

I have created a databricks account and added my account via root AWS access keys (I know this is not recommended but had tried via cross acount role and this was not working either). These root access keys should have access to my entire AWS account, however when I call.
dbutils.fs.ls('/mnt')
Just a basic listing of the files in my directory, I get this error:
ExecutionError: An error occurred while calling z:com.databricks.backend.daemon.dbutils.FSUtils.ls.
: java.lang.RuntimeException: java.io.IOException: Failed to perform 'getMountFileState(forceRefresh=true)' for mounts after 3 attempts. Please, retry the operation.
Original exception: 'com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden; request: HEAD https://auto-emailer.s3.amazonaws.com oregon-prod/3323621803827190.meta/mounts {} aws-sdk-java/1.11.602 Linux/4.4.0-1104-aws OpenJDK_64-Bit_Server_VM/25.242-b08 java/1.8.0_242 scala/2.12.10 vendor/Private_Build com.amazonaws.services.s3.model.GetObjectMetadataRequest; Request ID: 640F00A5A05F37B3, Extended Request ID: GMSFv0QNFog8PqLBYz8Hre9W5ttMpCLOKNQPPingLc2t1/XFUxn5RdkdBtIYGUbeHYVvIkIM7IY=, Cloud Provider: AWS, Instance ID: i-0acc5354538e76431 (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden; Request ID: 640F00A5A05F37B3; S3 Extended Request ID: GMSFv0QNFog8PqLBYz8Hre9W5ttMpCLOKNQPPingLc2t1/XFUxn5RdkdBtIYGUbeHYVvIkIM7IY=), S3 Extended Request ID: GMSFv0QNFog8PqLBYz8Hre9W5ttMpCLOKNQPPingLc2t1/XFUxn5RdkdBtIYGUbeHYVvIkIM7IY=
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1712)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1367)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1113)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:770)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:744)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:726)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:686)
at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:668)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:532)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:512)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4921)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4867)
at com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1320)
at com.databricks.s3a.aws.EnforcingDatabricksS3Client.$anonfun$getObjectMetadata$1(EnforcingDatabricksS3Client.scala:223)
at scala.util.Try$.apply(Try.scala:213)
at com.databricks.s3a.aws.DatabricksS3Client.retryRequest(DatabricksS3Client.scala:137)
at com.databricks.s3a.aws.DatabricksS3Client.withExponentialBackoff(DatabricksS3Client.scala:111)
at com.databricks.s3a.aws.DatabricksS3Client.withExponentialBackoff$(DatabricksS3Client.scala:109)
at com.databricks.s3a.aws.EnforcingDatabricksS3Client.withExponentialBackoff(EnforcingDatabricksS3Client.scala:28)
at com.databricks.s3a.aws.EnforcingDatabricksS3Client.getObjectMetadata(EnforcingDatabricksS3Client.scala:223)
at com.databricks.s3a.S3AFileSystem.getObjectMetadata(S3AFileSystem.java:1948)
at com.databricks.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:1450)
at com.databricks.s3a.S3AFileSystem.open(S3AFileSystem.java:663)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:769)
at com.databricks.backend.daemon.data.server.DefaultMetadataManager.liftedTree1$1(MetadataManager.scala:156)
at com.databricks.backend.daemon.data.server.DefaultMetadataManager.$anonfun$getMountFileState$1(MetadataManager.scala:151)
at com.databricks.backend.daemon.data.server.DefaultMetadataManager.withRetries(MetadataManager.scala:326)
at com.databricks.backend.daemon.data.server.DefaultMetadataManager.getMountFileState(MetadataManager.scala:150)
at com.databricks.backend.daemon.data.server.DefaultMetadataManager.getMounts(MetadataManager.scala:200)
at com.databricks.backend.daemon.data.filesystem.MountEntryResolver.retrieveFilesystems(MountEntryResolver.scala:76)
at com.databricks.backend.daemon.data.filesystem.MountEntryResolver.<init>(MountEntryResolver.scala:33)
at com.databricks.backend.daemon.data.server.DbfsServerBackend.makeSessionContext(DbfsServerBackend.scala:242)
at com.databricks.backend.daemon.data.server.DbfsServerBackend.$anonfun$receive$2(DbfsServerBackend.scala:280)
at com.databricks.backend.daemon.data.server.session.SessionManager.createSession(SessionManager.scala:193)
at com.databricks.backend.daemon.data.server.DbfsServerBackend$$anonfun$receive$3.applyOrElse(DbfsServerBackend.scala:285)
at com.databricks.backend.daemon.data.server.DbfsServerBackend$$anonfun$receive$3.applyOrElse(DbfsServerBackend.scala:282)
at com.databricks.rpc.ServerBackend.$anonfun$internalReceive$2(ServerBackend.scala:52)
at com.databricks.rpc.ServerBackend$$anonfun$commonReceive$1.applyOrElse(ServerBackend.scala:79)
at com.databricks.rpc.ServerBackend$$anonfun$commonReceive$1.applyOrElse(ServerBackend.scala:79)
at com.databricks.rpc.ServerBackend.$anonfun$internalReceive$1(ServerBackend.scala:48)
at com.databricks.logging.UsageLogging.$anonfun$recordOperation$4(UsageLogging.scala:428)
at com.databricks.logging.UsageLogging.$anonfun$withAttributionContext$1(UsageLogging.scala:238)
at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62)
at com.databricks.logging.UsageLogging.withAttributionContext(UsageLogging.scala:233)
at com.databricks.logging.UsageLogging.withAttributionContext$(UsageLogging.scala:230)
at com.databricks.rpc.ServerBackend.withAttributionContext(ServerBackend.scala:15)
at com.databricks.logging.UsageLogging.withAttributionTags(UsageLogging.scala:275)
at com.databricks.logging.UsageLogging.withAttributionTags$(UsageLogging.scala:268)
at com.databricks.rpc.ServerBackend.withAttributionTags(ServerBackend.scala:15)
at com.databricks.logging.UsageLogging.recordOperation(UsageLogging.scala:409)
at com.databricks.logging.UsageLogging.recordOperation$(UsageLogging.scala:336)
at com.databricks.rpc.ServerBackend.recordOperation(ServerBackend.scala:15)
at com.databricks.rpc.ServerBackend.internalReceive(ServerBackend.scala:47)
at com.databricks.rpc.JettyServer$RequestManager.$anonfun$handleRPC$2(JettyServer.scala:611)
at scala.util.Try$.apply(Try.scala:213)
at com.databricks.rpc.JettyServer$RequestManager.handleRPC(JettyServer.scala:611)
at com.databricks.rpc.JettyServer$RequestManager.handleRequestAndRespond(JettyServer.scala:534)
at com.databricks.rpc.JettyServer$RequestManager.$anonfun$handleHttp$4(JettyServer.scala:321)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at com.databricks.logging.UsageLogging.$anonfun$withAttributionContext$1(UsageLogging.scala:238)
at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62)
at com.databricks.logging.UsageLogging.withAttributionContext(UsageLogging.scala:233)
at com.databricks.logging.UsageLogging.withAttributionContext$(UsageLogging.scala:230)
at com.databricks.rpc.JettyServer$.withAttributionContext(JettyServer.scala:152)
at com.databricks.logging.UsageLogging.withAttributionTags(UsageLogging.scala:275)
at com.databricks.logging.UsageLogging.withAttributionTags$(UsageLogging.scala:268)
at com.databricks.rpc.JettyServer$.withAttributionTags(JettyServer.scala:152)
at com.databricks.rpc.JettyServer$RequestManager.handleHttp(JettyServer.scala:310)
at com.databricks.rpc.JettyServer$RequestManager.doPost(JettyServer.scala:217)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at com.databricks.rpc.HttpServletWithPatch.service(HttpServletWithPatch.scala:33)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:848)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
at org.eclipse.jetty.server.Server.handle(Server.java:539)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
at java.lang.Thread.run(Thread.java:748)
'
at org.apache.hadoop.mapred.JobConf.getWorkingDirectory(JobConf.java:660)
at org.apache.hadoop.mapred.FileInputFormat.setInputPaths(FileInputFormat.java:438)
at org.apache.hadoop.mapred.FileInputFormat.setInputPaths(FileInputFormat.java:411)
at org.apache.spark.api.python.PythonSecurityUtils$.resolvePaths(PythonSecurityUtils.scala:246)
at org.apache.spark.api.python.PythonSecurityUtils$.checkPathStringFileSystemSafety(PythonSecurityUtils.scala:258)
at com.databricks.backend.daemon.dbutils.FSUtils$.com$databricks$backend$daemon$dbutils$FSUtils$$withFsSafetyCheck(DBUtilsCore.scala:80)
at com.databricks.backend.daemon.dbutils.FSUtils$.ls(DBUtilsCore.scala:85)
at com.databricks.backend.daemon.dbutils.FSUtils.ls(DBUtilsCore.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:380)
at py4j.Gateway.invoke(Gateway.java:295)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.GatewayConnection.run(GatewayConnection.java:251)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: Failed to perform 'getMountFileState(forceRefresh=true)' for mounts after 3 attempts. Please, retry the operation.
Original exception: 'com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden; request: HEAD https://auto-emailer.s3.amazonaws.com oregon-prod/3323621803827190.meta/mounts {} aws-sdk-java/1.11.602 Linux/4.4.0-1104-aws OpenJDK_64-Bit_Server_VM/25.242-b08 java/1.8.0_242 scala/2.12.10 vendor/Private_Build com.amazonaws.services.s3.model.GetObjectMetadataRequest; Request ID: 640F00A5A05F37B3, Extended Request ID: GMSFv0QNFog8PqLBYz8Hre9W5ttMpCLOKNQPPingLc2t1/XFUxn5RdkdBtIYGUbeHYVvIkIM7IY=, Cloud Provider: AWS, Instance ID: i-0acc5354538e76431 (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden; Request ID: 640F00A5A05F37B3; S3 Extended Request ID: GMSFv0QNFog8PqLBYz8Hre9W5ttMpCLOKNQPPingLc2t1/XFUxn5RdkdBtIYGUbeHYVvIkIM7IY=), S3 Extended Request ID: GMSFv0QNFog8PqLBYz8Hre9W5ttMpCLOKNQPPingLc2t1/XFUxn5RdkdBtIYGUbeHYVvIkIM7IY=
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1712)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1367)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1113)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:770)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:744)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:726)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:686)
at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:668)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:532)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:512)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4921)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4867)
at com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1320)
at com.databricks.s3a.aws.EnforcingDatabricksS3Client.$anonfun$getObjectMetadata$1(EnforcingDatabricksS3Client.scala:223)
at scala.util.Try$.apply(Try.scala:213)
at com.databricks.s3a.aws.DatabricksS3Client.retryRequest(DatabricksS3Client.scala:137)
at com.databricks.s3a.aws.DatabricksS3Client.withExponentialBackoff(DatabricksS3Client.scala:111)
at com.databricks.s3a.aws.DatabricksS3Client.withExponentialBackoff$(DatabricksS3Client.scala:109)
at com.databricks.s3a.aws.EnforcingDatabricksS3Client.withExponentialBackoff(EnforcingDatabricksS3Client.scala:28)
at com.databricks.s3a.aws.EnforcingDatabricksS3Client.getObjectMetadata(EnforcingDatabricksS3Client.scala:223)
at com.databricks.s3a.S3AFileSystem.getObjectMetadata(S3AFileSystem.java:1948)
at com.databricks.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:1450)
at com.databricks.s3a.S3AFileSystem.open(S3AFileSystem.java:663)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:769)
at com.databricks.backend.daemon.data.server.DefaultMetadataManager.liftedTree1$1(MetadataManager.scala:156)
at com.databricks.backend.daemon.data.server.DefaultMetadataManager.$anonfun$getMountFileState$1(MetadataManager.scala:151)
at com.databricks.backend.daemon.data.server.DefaultMetadataManager.withRetries(MetadataManager.scala:326)
at com.databricks.backend.daemon.data.server.DefaultMetadataManager.getMountFileState(MetadataManager.scala:150)
at com.databricks.backend.daemon.data.server.DefaultMetadataManager.getMounts(MetadataManager.scala:200)
at com.databricks.backend.daemon.data.filesystem.MountEntryResolver.retrieveFilesystems(MountEntryResolver.scala:76)
at com.databricks.backend.daemon.data.filesystem.MountEntryResolver.<init>(MountEntryResolver.scala:33)
at com.databricks.backend.daemon.data.server.DbfsServerBackend.makeSessionContext(DbfsServerBackend.scala:242)
at com.databricks.backend.daemon.data.server.DbfsServerBackend.$anonfun$receive$2(DbfsServerBackend.scala:280)
at com.databricks.backend.daemon.data.server.session.SessionManager.createSession(SessionManager.scala:193)
at com.databricks.backend.daemon.data.server.DbfsServerBackend$$anonfun$receive$3.applyOrElse(DbfsServerBackend.scala:285)
at com.databricks.backend.daemon.data.server.DbfsServerBackend$$anonfun$receive$3.applyOrElse(DbfsServerBackend.scala:282)
at com.databricks.rpc.ServerBackend.$anonfun$internalReceive$2(ServerBackend.scala:52)
at com.databricks.rpc.ServerBackend$$anonfun$commonReceive$1.applyOrElse(ServerBackend.scala:79)
at com.databricks.rpc.ServerBackend$$anonfun$commonReceive$1.applyOrElse(ServerBackend.scala:79)
at com.databricks.rpc.ServerBackend.$anonfun$internalReceive$1(ServerBackend.scala:48)
at com.databricks.logging.UsageLogging.$anonfun$recordOperation$4(UsageLogging.scala:428)
at com.databricks.logging.UsageLogging.$anonfun$withAttributionContext$1(UsageLogging.scala:238)
at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62)
at com.databricks.logging.UsageLogging.withAttributionContext(UsageLogging.scala:233)
at com.databricks.logging.UsageLogging.withAttributionContext$(UsageLogging.scala:230)
at com.databricks.rpc.ServerBackend.withAttributionContext(ServerBackend.scala:15)
at com.databricks.logging.UsageLogging.withAttributionTags(UsageLogging.scala:275)
at com.databricks.logging.UsageLogging.withAttributionTags$(UsageLogging.scala:268)
at com.databricks.rpc.ServerBackend.withAttributionTags(ServerBackend.scala:15)
at com.databricks.logging.UsageLogging.recordOperation(UsageLogging.scala:409)
at com.databricks.logging.UsageLogging.recordOperation$(UsageLogging.scala:336)
at com.databricks.rpc.ServerBackend.recordOperation(ServerBackend.scala:15)
at com.databricks.rpc.ServerBackend.internalReceive(ServerBackend.scala:47)
at com.databricks.rpc.JettyServer$RequestManager.$anonfun$handleRPC$2(JettyServer.scala:611)
at scala.util.Try$.apply(Try.scala:213)
at com.databricks.rpc.JettyServer$RequestManager.handleRPC(JettyServer.scala:611)
at com.databricks.rpc.JettyServer$RequestManager.handleRequestAndRespond(JettyServer.scala:534)
at com.databricks.rpc.JettyServer$RequestManager.$anonfun$handleHttp$4(JettyServer.scala:321)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at com.databricks.logging.UsageLogging.$anonfun$withAttributionContext$1(UsageLogging.scala:238)
at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62)
at com.databricks.logging.UsageLogging.withAttributionContext(UsageLogging.scala:233)
at com.databricks.logging.UsageLogging.withAttributionContext$(UsageLogging.scala:230)
at com.databricks.rpc.JettyServer$.withAttributionContext(JettyServer.scala:152)
at com.databricks.logging.UsageLogging.withAttributionTags(UsageLogging.scala:275)
at com.databricks.logging.UsageLogging.withAttributionTags$(UsageLogging.scala:268)
at com.databricks.rpc.JettyServer$.withAttributionTags(JettyServer.scala:152)
at com.databricks.rpc.JettyServer$RequestManager.handleHttp(JettyServer.scala:310)
at com.databricks.rpc.JettyServer$RequestManager.doPost(JettyServer.scala:217)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at com.databricks.rpc.HttpServletWithPatch.service(HttpServletWithPatch.scala:33)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:848)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
at org.eclipse.jetty.server.Server.handle(Server.java:539)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
at java.lang.Thread.run(Thread.java:748)
'
at com.databricks.backend.daemon.data.client.DbfsClient.send0(DbfsClient.scala:117)
at com.databricks.backend.daemon.data.client.DbfsClient.sendIdempotent(DbfsClient.scala:63)
at com.databricks.backend.daemon.data.client.DatabricksMountsStore.com$databricks$backend$daemon$data$client$DatabricksMountsStore$$refreshMounts(DatabricksMountsStore.scala:92)
at com.databricks.backend.daemon.data.client.DatabricksMountsStore.forceRefreshMounts(DatabricksMountsStore.scala:127)
at com.databricks.backend.daemon.data.client.SharedDatabricksMountsStore$.getOrCreate(DatabricksMountsStore.scala:189)
at com.databricks.backend.daemon.data.client.DBFSV2$$anonfun$initialize$2.apply(DatabricksFileSystemV2.scala:72)
at com.databricks.backend.daemon.data.client.DBFSV2$$anonfun$initialize$2.apply(DatabricksFileSystemV2.scala:64)
at com.databricks.logging.UsageLogging$$anonfun$recordOperation$1.apply(UsageLogging.scala:428)
at com.databricks.logging.UsageLogging$$anonfun$withAttributionContext$1.apply(UsageLogging.scala:238)
at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58)
at com.databricks.logging.UsageLogging$class.withAttributionContext(UsageLogging.scala:233)
at com.databricks.backend.daemon.data.client.DatabricksFileSystemV2.withAttributionContext(DatabricksFileSystemV2.scala:450)
at com.databricks.logging.UsageLogging$class.withAttributionTags(UsageLogging.scala:275)
at com.databricks.backend.daemon.data.client.DatabricksFileSystemV2.withAttributionTags(DatabricksFileSystemV2.scala:450)
at com.databricks.logging.UsageLogging$class.recordOperation(UsageLogging.scala:409)
at com.databricks.backend.daemon.data.client.DatabricksFileSystemV2.recordOperation(DatabricksFileSystemV2.scala:450)
at com.databricks.backend.daemon.data.client.DBFSV2.initialize(DatabricksFileSystemV2.scala:64)
at com.databricks.backend.daemon.data.client.DatabricksFileSystem.initialize(DatabricksFileSystem.scala:226)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2669)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:94)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2703)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2685)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:373)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:172)
at org.apache.hadoop.mapred.JobConf.getWorkingDirectory(JobConf.java:656)
... 18 more
Caused by: java.io.IOException: Failed to perform 'getMountFileState(forceRefresh=true)' for mounts after 3 attempts. Please, retry the operation.
Original exception: 'com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden; request: HEAD https://auto-emailer.s3.amazonaws.com oregon-prod/3323621803827190.meta/mounts {} aws-sdk-java/1.11.602 Linux/4.4.0-1104-aws OpenJDK_64-Bit_Server_VM/25.242-b08 java/1.8.0_242 scala/2.12.10 vendor/Private_Build com.amazonaws.services.s3.model.GetObjectMetadataRequest; Request ID: 640F00A5A05F37B3, Extended Request ID: GMSFv0QNFog8PqLBYz8Hre9W5ttMpCLOKNQPPingLc2t1/XFUxn5RdkdBtIYGUbeHYVvIkIM7IY=, Cloud Provider: AWS, Instance ID: i-0acc5354538e76431 (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden; Request ID: 640F00A5A05F37B3; S3 Extended Request ID: GMSFv0QNFog8PqLBYz8Hre9W5ttMpCLOKNQPPingLc2t1/XFUxn5RdkdBtIYGUbeHYVvIkIM7IY=), S3 Extended Request ID: GMSFv0QNFog8PqLBYz8Hre9W5ttMpCLOKNQPPingLc2t1/XFUxn5RdkdBtIYGUbeHYVvIkIM7IY=
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1712)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1367)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1113)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:770)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:744)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:726)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:686)
at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:668)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:532)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:512)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4921)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4867)
at com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1320)
at com.databricks.s3a.aws.EnforcingDatabricksS3Client.$anonfun$getObjectMetadata$1(EnforcingDatabricksS3Client.scala:223)
at scala.util.Try$.apply(Try.scala:213)
at com.databricks.s3a.aws.DatabricksS3Client.retryRequest(DatabricksS3Client.scala:137)
at com.databricks.s3a.aws.DatabricksS3Client.withExponentialBackoff(DatabricksS3Client.scala:111)
at com.databricks.s3a.aws.DatabricksS3Client.withExponentialBackoff$(DatabricksS3Client.scala:109)
at com.databricks.s3a.aws.EnforcingDatabricksS3Client.withExponentialBackoff(EnforcingDatabricksS3Client.scala:28)
at com.databricks.s3a.aws.EnforcingDatabricksS3Client.getObjectMetadata(EnforcingDatabricksS3Client.scala:223)
at com.databricks.s3a.S3AFileSystem.getObjectMetadata(S3AFileSystem.java:1948)
at com.databricks.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:1450)
at com.databricks.s3a.S3AFileSystem.open(S3AFileSystem.java:663)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:769)
at com.databricks.backend.daemon.data.server.DefaultMetadataManager.liftedTree1$1(MetadataManager.scala:156)
at com.databricks.backend.daemon.data.server.DefaultMetadataManager.$anonfun$getMountFileState$1(MetadataManager.scala:151)
at com.databricks.backend.daemon.data.server.DefaultMetadataManager.withRetries(MetadataManager.scala:326)
at com.databricks.backend.daemon.data.server.DefaultMetadataManager.getMountFileState(MetadataManager.scala:150)
at com.databricks.backend.daemon.data.server.DefaultMetadataManager.getMounts(MetadataManager.scala:200)
at com.databricks.backend.daemon.data.filesystem.MountEntryResolver.retrieveFilesystems(MountEntryResolver.scala:76)
at com.databricks.backend.daemon.data.filesystem.MountEntryResolver.<init>(MountEntryResolver.scala:33)
at com.databricks.backend.daemon.data.server.DbfsServerBackend.makeSessionContext(DbfsServerBackend.scala:242)
at com.databricks.backend.daemon.data.server.DbfsServerBackend.$anonfun$receive$2(DbfsServerBackend.scala:280)
at com.databricks.backend.daemon.data.server.session.SessionManager.createSession(SessionManager.scala:193)
at com.databricks.backend.daemon.data.server.DbfsServerBackend$$anonfun$receive$3.applyOrElse(DbfsServerBackend.scala:285)
at com.databricks.backend.daemon.data.server.DbfsServerBackend$$anonfun$receive$3.applyOrElse(DbfsServerBackend.scala:282)
at com.databricks.rpc.ServerBackend.$anonfun$internalReceive$2(ServerBackend.scala:52)
at com.databricks.rpc.ServerBackend$$anonfun$commonReceive$1.applyOrElse(ServerBackend.scala:79)
at com.databricks.rpc.ServerBackend$$anonfun$commonReceive$1.applyOrElse(ServerBackend.scala:79)
at com.databricks.rpc.ServerBackend.$anonfun$internalReceive$1(ServerBackend.scala:48)
at com.databricks.logging.UsageLogging.$anonfun$recordOperation$4(UsageLogging.scala:428)
at com.databricks.logging.UsageLogging.$anonfun$withAttributionContext$1(UsageLogging.scala:238)
at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62)
at com.databricks.logging.UsageLogging.withAttributionContext(UsageLogging.scala:233)
at com.databricks.logging.UsageLogging.withAttributionContext$(UsageLogging.scala:230)
at com.databricks.rpc.ServerBackend.withAttributionContext(ServerBackend.scala:15)
at com.databricks.logging.UsageLogging.withAttributionTags(UsageLogging.scala:275)
at com.databricks.logging.UsageLogging.withAttributionTags$(UsageLogging.scala:268)
at com.databricks.rpc.ServerBackend.withAttributionTags(ServerBackend.scala:15)
at com.databricks.logging.UsageLogging.recordOperation(UsageLogging.scala:409)
at com.databricks.logging.UsageLogging.recordOperation$(UsageLogging.scala:336)
at com.databricks.rpc.ServerBackend.recordOperation(ServerBackend.scala:15)
at com.databricks.rpc.ServerBackend.internalReceive(ServerBackend.scala:47)
at com.databricks.rpc.JettyServer$RequestManager.$anonfun$handleRPC$2(JettyServer.scala:611)
at scala.util.Try$.apply(Try.scala:213)
at com.databricks.rpc.JettyServer$RequestManager.handleRPC(JettyServer.scala:611)
at com.databricks.rpc.JettyServer$RequestManager.handleRequestAndRespond(JettyServer.scala:534)
I see in the stack trace that original exception: 'com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden; but I don't understand why things are forbidden, this account is set up and the cluster was launched using my root access keys, so Databricks should have access to everything. Any help here would be greatly appreciated!
To be clear, the end goal here is to mount a bucket from s3 to databricks so I can access it, but when I tried mounting I got the same error
There're more than one ways to limit access to a resource in AWS:
limit user/role permissions (not your case as you're using root)
limit operations on resource side, for example allow only VPC traffic (i.e. no external traffic) or allow GET but not HEAD, allow only this specific user to do HEAD, and so on.
Check your S3 bucket access (or provide it here): explicitly allow your user to do list, get, put objects.

muutanja error in aws server and not in localhost

Hopefully it makes sense to make a question like this here:
I have an python3 api which it uses Flask to serve. I also have an clojure app that makes a post request targeting the python3 api. To do so I use clj-http with something like:
(client/post url-link
{:basic-auth ["username"
"password"]
:form-params json
:content-type :json})
Now, when I test in my machine serving the python3 api locally, i.e,
(def url-link "localhost:port/api-name")
it works smoothly. The issue comes because I will serve the python3 api from the amazon server. So my thought was to use the link of the amazon server. But when I do so, clojure complains with
HTTP ERROR 500 clojure.lang.ExceptionInfo: Malformed application/json in :muuntaja/encode {:type :muuntaja/encode, :format "application/json"}
URI: /api/sell-call
STATUS: 500
MESSAGE: clojure.lang.ExceptionInfo: Malformed application/json in :muuntaja/encode {:type :muuntaja/encode, :format "application/json"}
SERVLET: -
CAUSED BY: clojure.lang.ExceptionInfo: Malformed application/json in :muuntaja/encode {:type :muuntaja/encode, :format "application/json"}
CAUSED BY: com.fasterxml.jackson.databind.JsonMappingException: (was java.lang.UnsupportedOperationException) (through reference chain: clojure.lang.PersistentArrayMap[":full_message"]->clojure.lang.ExceptionInfo["data"]->clojure.lang.PersistentHashMap[":http-client"]->org.apache.http.impl.client.InternalHttpClient["params"])
CAUSED BY: java.lang.UnsupportedOperationException
clojure.lang.ExceptionInfo: Malformed application/json in :muuntaja/encode {:type :muuntaja/encode, :format "application/json"}
at muuntaja.core$on_exception.invokeStatic(core.clj:284)
at muuntaja.core$on_exception.invoke(core.clj:267)
at clojure.core$partial$fn__5824.invoke(core.clj:2626)
at muuntaja.core$create_coder$encode__10562.invoke(core.clj:346)
at clojure.core$update.invokeStatic(core.clj:6198)
at clojure.core$update.invoke(core.clj:6188)
at muuntaja.core$create$_handle_response__10629.invoke(core.clj:443)
at muuntaja.core$create$reify__10631.format_response(core.clj:486)
at muuntaja.middleware$wrap_format_response$fn__10702.invoke(middleware.clj:132)
at muuntaja.middleware$wrap_format_negotiate$fn__10695.invoke(middleware.clj:96)
at ring.middleware.keyword_params$wrap_keyword_params$fn__9403.invoke(keyword_params.clj:53)
at ring.middleware.nested_params$wrap_nested_params$fn__9461.invoke(nested_params.clj:89)
at ring.middleware.params$wrap_params$fn__9535.invoke(params.clj:67)
at compojure.api.middleware$wrap_inject_data$fn__11689.invoke(middleware.clj:96)
at compojure.api.routes.Route.invoke(routes.clj:89)
at ring.middleware.basic_authentication$wrap_basic_authentication$fn__16405.invoke(basic_authentication.clj:157)
at clojure.lang.Var.invoke(Var.java:384)
at ring.middleware.reload$wrap_reload$fn__1065.invoke(reload.clj:39)
at ring.adapter.jetty$proxy_handler$fn__480.invoke(jetty.clj:27)
at ring.adapter.jetty.proxy$org.eclipse.jetty.server.handler.AbstractHandler$ff19274a.handle(Unknown Source)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.Server.handle(Server.java:500)
at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:386)
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:562)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:378)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:270)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: com.fasterxml.jackson.databind.JsonMappingException: (was java.lang.UnsupportedOperationException) (through reference chain: clojure.lang.PersistentArrayMap[":full_message"]->clojure.lang.ExceptionInfo["data"]->clojure.lang.PersistentHashMap[":http-client"]->org.apache.http.impl.client.InternalHttpClient["params"])
at com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:394)
at com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:353)
at com.fasterxml.jackson.databind.ser.std.StdSerializer.wrapAndThrow(StdSerializer.java:316)
at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:727)
at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:155)
at com.fasterxml.jackson.databind.ser.std.MapSerializer.serializeFields(MapSerializer.java:722)
at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:643)
at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:33)
at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:727)
at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:719)
at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:155)
at com.fasterxml.jackson.databind.ser.std.MapSerializer.serializeFields(MapSerializer.java:722)
at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:643)
at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:33)
at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider._serialize(DefaultSerializerProvider.java:480)
at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:319)
at com.fasterxml.jackson.databind.ObjectMapper._configAndWriteValue(ObjectMapper.java:3905)
at com.fasterxml.jackson.databind.ObjectMapper.writeValueAsBytes(ObjectMapper.java:3243)
at jsonista.core$write_value_as_bytes.invokeStatic(core.clj:229)
at jsonista.core$write_value_as_bytes.invoke(core.clj:221)
at muuntaja.format.json$encoder$reify__10016.encode_to_bytes(json.clj:43)
at muuntaja.core$create_coder$encode__10562.invoke(core.clj:344)
... 28 more
Caused by: java.lang.UnsupportedOperationException
at org.apache.http.impl.client.InternalHttpClient.getParams(InternalHttpClient.java:211)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:688)
at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:719)
... 46 more
com.fasterxml.jackson.databind.JsonMappingException: (was java.lang.UnsupportedOperationException) (through reference chain: clojure.lang.PersistentArrayMap[":full_message"]->clojure.lang.ExceptionInfo["data"]->clojure.lang.PersistentHashMap[":http-client"]->org.apache.http.impl.client.InternalHttpClient["params"])
at com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:394)
at com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:353)
at com.fasterxml.jackson.databind.ser.std.StdSerializer.wrapAndThrow(StdSerializer.java:316)
at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:727)
at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:155)
at com.fasterxml.jackson.databind.ser.std.MapSerializer.serializeFields(MapSerializer.java:722)
at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:643)
at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:33)
at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:727)
at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:719)
at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:155)
at com.fasterxml.jackson.databind.ser.std.MapSerializer.serializeFields(MapSerializer.java:722)
at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:643)
at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:33)
at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider._serialize(DefaultSerializerProvider.java:480)
at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:319)
at com.fasterxml.jackson.databind.ObjectMapper._configAndWriteValue(ObjectMapper.java:3905)
at com.fasterxml.jackson.databind.ObjectMapper.writeValueAsBytes(ObjectMapper.java:3243)
at jsonista.core$write_value_as_bytes.invokeStatic(core.clj:229)
at jsonista.core$write_value_as_bytes.invoke(core.clj:221)
at muuntaja.format.json$encoder$reify__10016.encode_to_bytes(json.clj:43)
at muuntaja.core$create_coder$encode__10562.invoke(core.clj:344)
at clojure.core$update.invokeStatic(core.clj:6198)
at clojure.core$update.invoke(core.clj:6188)
at muuntaja.core$create$_handle_response__10629.invoke(core.clj:443)
at muuntaja.core$create$reify__10631.format_response(core.clj:486)
at muuntaja.middleware$wrap_format_response$fn__10702.invoke(middleware.clj:132)
at muuntaja.middleware$wrap_format_negotiate$fn__10695.invoke(middleware.clj:96)
at ring.middleware.keyword_params$wrap_keyword_params$fn__9403.invoke(keyword_params.clj:53)
at ring.middleware.nested_params$wrap_nested_params$fn__9461.invoke(nested_params.clj:89)
at ring.middleware.params$wrap_params$fn__9535.invoke(params.clj:67)
at compojure.api.middleware$wrap_inject_data$fn__11689.invoke(middleware.clj:96)
at compojure.api.routes.Route.invoke(routes.clj:89)
at ring.middleware.basic_authentication$wrap_basic_authentication$fn__16405.invoke(basic_authentication.clj:157)
at clojure.lang.Var.invoke(Var.java:384)
at ring.middleware.reload$wrap_reload$fn__1065.invoke(reload.clj:39)
at ring.adapter.jetty$proxy_handler$fn__480.invoke(jetty.clj:27)
at ring.adapter.jetty.proxy$org.eclipse.jetty.server.handler.AbstractHandler$ff19274a.handle(Unknown Source)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.Server.handle(Server.java:500)
at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:386)
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:562)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:378)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:270)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.lang.UnsupportedOperationException
at org.apache.http.impl.client.InternalHttpClient.getParams(InternalHttpClient.java:211)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:688)
at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:719)
... 46 more
java.lang.UnsupportedOperationException
at org.apache.http.impl.client.InternalHttpClient.getParams(InternalHttpClient.java:211)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:688)
at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:719)
at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:155)
at com.fasterxml.jackson.databind.ser.std.MapSerializer.serializeFields(MapSerializer.java:722)
at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:643)
at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:33)
at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:727)
at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:719)
at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:155)
at com.fasterxml.jackson.databind.ser.std.MapSerializer.serializeFields(MapSerializer.java:722)
at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:643)
at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:33)
at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider._serialize(DefaultSerializerProvider.java:480)
at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:319)
at com.fasterxml.jackson.databind.ObjectMapper._configAndWriteValue(ObjectMapper.java:3905)
at com.fasterxml.jackson.databind.ObjectMapper.writeValueAsBytes(ObjectMapper.java:3243)
at jsonista.core$write_value_as_bytes.invokeStatic(core.clj:229)
at jsonista.core$write_value_as_bytes.invoke(core.clj:221)
at muuntaja.format.json$encoder$reify__10016.encode_to_bytes(json.clj:43)
at muuntaja.core$create_coder$encode__10562.invoke(core.clj:344)
at clojure.core$update.invokeStatic(core.clj:6198)
at clojure.core$update.invoke(core.clj:6188)
at muuntaja.core$create$_handle_response__10629.invoke(core.clj:443)
at muuntaja.core$create$reify__10631.format_response(core.clj:486)
at muuntaja.middleware$wrap_format_response$fn__10702.invoke(middleware.clj:132)
at muuntaja.middleware$wrap_format_negotiate$fn__10695.invoke(middleware.clj:96)
at ring.middleware.keyword_params$wrap_keyword_params$fn__9403.invoke(keyword_params.clj:53)
at ring.middleware.nested_params$wrap_nested_params$fn__9461.invoke(nested_params.clj:89)
at ring.middleware.params$wrap_params$fn__9535.invoke(params.clj:67)
at compojure.api.middleware$wrap_inject_data$fn__11689.invoke(middleware.clj:96)
at compojure.api.routes.Route.invoke(routes.clj:89)
at ring.middleware.basic_authentication$wrap_basic_authentication$fn__16405.invoke(basic_authentication.clj:157)
at clojure.lang.Var.invoke(Var.java:384)
at ring.middleware.reload$wrap_reload$fn__1065.invoke(reload.clj:39)
at ring.adapter.jetty$proxy_handler$fn__480.invoke(jetty.clj:27)
at ring.adapter.jetty.proxy$org.eclipse.jetty.server.handler.AbstractHandler$ff19274a.handle(Unknown Source)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.Server.handle(Server.java:500)
at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:386)
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:562)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:378)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:270)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
at java.base/java.lang.Thread.run(Thread.java:834)
Powered by Jetty:// 9.4.22.v20191022
So, is it possible to say what I am doing wrong? Is there a way to work around it from the client side?
Note: The problem is that this is a code from my work and I would like to not share much of the specifics from it. I am asking because maybe I am missing a simple solution. If this question does not make sense to be in here, let me know and I will delete it.
I appreciate any help!
I have figured out what happened.
There was an internal error in the python application in the amazon server, so when it returned to the clojure app, it did not know how to transform the error into a JSON (which is expected). So the solution was to deal with the error in the python application to return a JSON regarding the problem.
In my local machine worked because I had all non python dependencies to make the application work.
Thanks for the help!

SSL .pk12 importing error using ikeyman

I have generated CSR and downloaded .pk12 file from the CA. While I'm trying to import this getting error as
An IO Exception has occurred:
DerInputStream.getLength(): lengthTag=47, too big.
Error details are as ::
Error Code:
58
Stack Trace:
com.ibm.gsk.ikeyman.error.KeyManagerException: java.io.IOException: DerInputStream.getLength(): lengthTag=47, too big.
at com.ibm.gsk.ikeyman.keystore.KeyStoreProxyCreatorFactory$FileKeyStoreProxy.load(KeyStoreProxyCreatorFactory.java:264)
at com.ibm.gsk.ikeyman.keystore.KeyStoreProxyCreatorFactory$AbstractKeyStoreProxy.getKeyStore(KeyStoreProxyCreatorFactory.java:187)
at com.ibm.gsk.ikeyman.keystore.KeyStoreItemFactory.getPKCS12KeyStoreItem(KeyStoreItemFactory.java:338)
at com.ibm.gsk.ikeyman.keystore.KeyStoreItemFactory.newPKCS12KeyStoreItem(KeyStoreItemFactory.java:332)
at com.ibm.gsk.ikeyman.keystore.KeyStoreItemFactory.loadKeystore(KeyStoreItemFactory.java:141)
at com.ibm.gsk.ikeyman.command.KeyStoreLoaderFactory$BasicKeyStoreLoader.getKeyStoreItem(KeyStoreLoaderFactory.java:48)
at com.ibm.gsk.ikeyman.command.KeyStoreLoaderFactory$CachedKeyStoreLoader.getKeyStoreItem(KeyStoreLoaderFactory.java:83)
at com.ibm.gsk.ikeyman.command.KeyStoreLoaderFactory$BasicKeyStoreLoader.getValidKeyStoreItem(KeyStoreLoaderFactory.java:53)
at com.ibm.gsk.ikeyman.command.Command.getValidKeystoreItem(Command.java:104)
at com.ibm.gsk.ikeyman.command.CommandFactory$LoadDbCommand.run(CommandFactory.java:531)
at com.ibm.gsk.ikeyman.command.Command.invoke(Command.java:198)
at com.ibm.gsk.ikeyman.command.gui.KeymanController$Invoker.invoke(KeymanController.java:261)
at com.ibm.gsk.ikeyman.command.gui.RetrieverFactory$ImportRetriever.getImportParameters(RetrieverFactory.java:509)
at com.ibm.gsk.ikeyman.command.gui.RetrieverFactory$ExportRetriever.getParameters(RetrieverFactory.java:556)
at com.ibm.gsk.ikeyman.command.gui.RetrieverFactory$CompoundRetriever.getParameters(RetrieverFactory.java:149)
at com.ibm.gsk.ikeyman.command.gui.KeymanController.getParameters(KeymanController.java:139)
at com.ibm.gsk.ikeyman.command.gui.KeymanController.getParameters(KeymanController.java:130)
at com.ibm.gsk.ikeyman.command.gui.KeymanController.actionPerformed(KeymanController.java:91)
at com.ibm.gsk.ikeyman.gui.panels.KeymanPanel.fireActionPerformed(KeymanPanel.java:54)
at com.ibm.gsk.ikeyman.gui.panels.KeymanPanel.actionPerformed(KeymanPanel.java:48)
at com.ibm.gsk.ikeyman.gui.panels.KeymanPanel.fireActionPerformed(KeymanPanel.java:54)
at com.ibm.gsk.ikeyman.gui.panels.KeymanPanel.actionPerformed(KeymanPanel.java:48)
at com.ibm.gsk.ikeyman.gui.panels.ContentsPanel.actionPerformed(ContentsPanel.java:334)
at com.ibm.gsk.ikeyman.gui.panels.KeymanPanel.fireActionPerformed(KeymanPanel.java:54)
at com.ibm.gsk.ikeyman.gui.panels.KeymanPanel.actionPerformed(KeymanPanel.java:48)
at javax.swing.AbstractButton.fireActionPerformed(AbstractButton.java:2006)
at javax.swing.AbstractButton$Handler.actionPerformed(AbstractButton.java:2329)
at javax.swing.DefaultButtonModel.fireActionPerformed(DefaultButtonModel.java:398)
at javax.swing.DefaultButtonModel.setPressed(DefaultButtonModel.java:253)
at javax.swing.plaf.basic.BasicButtonListener.mouseReleased(BasicButtonListener.java:247)
at java.awt.AWTEventMulticaster.mouseReleased(AWTEventMulticaster.java:283)
at java.awt.Component.processMouseEvent(Component.java:6052)
at javax.swing.JComponent.processMouseEvent(JComponent.java:3276)
at java.awt.Component.processEvent(Component.java:5817)
at java.awt.Container.processEvent(Container.java:2069)
at java.awt.Component.dispatchEventImpl(Component.java:4424)
at java.awt.Container.dispatchEventImpl(Container.java:2127)
at java.awt.Component.dispatchEvent(Component.java:4254)
at java.awt.LightweightDispatcher.retargetMouseEvent(Container.java:4333)
at java.awt.LightweightDispatcher.processMouseEvent(Container.java:3997)
at java.awt.LightweightDispatcher.dispatchEvent(Container.java:3927)
at java.awt.Container.dispatchEventImpl(Container.java:2113)
at java.awt.Window.dispatchEventImpl(Window.java:2451)
at java.awt.Component.dispatchEvent(Component.java:4254)
at java.awt.EventQueue.dispatchEvent(EventQueue.java:610)
at java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:284)
at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:194)
at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:184)
at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:179)
at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:171)
at java.awt.EventDispatchThread.run(EventDispatchThread.java:132)
Caused by: java.io.IOException: DerInputStream.getLength(): lengthTag=47, too big.
at com.ibm.security.util.DerInputStream.getLength(DerInputStream.java:724)
at com.ibm.security.util.DerInputStream.getLength(DerInputStream.java:698)
at com.ibm.security.util.DerValue.(DerValue.java:253)
at com.ibm.security.util.DerInputStream.getDerValue(DerInputStream.java:499)
at com.ibm.security.pkcsutil.PKCSDerObject.decode(PKCSDerObject.java:258)
at com.ibm.security.pkcs12.PFX.(PFX.java:134)
at com.ibm.crypto.provider.PKCS12KeyStore.engineLoad(Unknown Source)
at java.security.KeyStore.load(Unknown Source)
at com.ibm.gsk.ikeyman.keystore.KeyStoreProxyCreatorFactory$FileKeyStoreProxy.performLoad(KeyStoreProxyCreatorFactory.java:296)
at com.ibm.gsk.ikeyman.keystore.KeyStoreProxyCreatorFactory$FileKeyStoreProxy.load(KeyStoreProxyCreatorFactory.java:245)
... 50 more
What is causing the error ?

Spark submit failing with Hive

I am trying to get a Spark 1.1.0 program written in Scala to work, but I'm having a hard time with it. I have a Hive query that is very simple:
select json, score from data
When I run the following command from spark-shell everything works (I need the MYSQL_CONN in the driver class path as I'm using Hive with a MySQL metadata store)
bin/spark-shell --master $SPARK_URL --driver-class-path $MYSQL_CONN
import org.apache.spark.sql.hive.HiveContext
val sqlContext = new HiveContext(sc)
sqlContext.sql("select json from data").map(t => t.getString(0)).take(10).foreach(println)
I get ten lines of json just like I want. However, when I run this with spark-submit as follows I get a problem
bin/spark-submit --master $SPARK_URL --class spark.Main --driver-class-path $MYSQL_CONN target/spark-testing-1.0-SNAPSHOT.jar
Here is my whole Spark program
package spark
import org.apache.spark.sql.hive.HiveContext
import org.apache.spark.{SparkContext, SparkConf}
object Main {
def main(args: Array[String]) {
val sc = new SparkContext(new SparkConf().setAppName("Gathering Data"))
val sqlContext = new HiveContext(sc)
sqlContext.sql("select json from data").map(t => t.getString(0)).take(10).foreach(println)
}
}
And here is the resultant stack
14/12/01 21:30:04 WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0, match1hd17.dc1): java.lang.ClassNotFoundException: spark.Main$$anonfun$main$1
java.net.URLClassLoader$1.run(URLClassLoader.java:200)
java.security.AccessController.doPrivileged(Native Method)
java.net.URLClassLoader.findClass(URLClassLoader.java:188)
java.lang.ClassLoader.loadClass(ClassLoader.java:307)
java.lang.ClassLoader.loadClass(ClassLoader.java:252)
java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
java.lang.Class.forName0(Native Method)
java.lang.Class.forName(Class.java:247)
org.apache.spark.serializer.JavaDeserializationStream$$anon$1.resolveClass(JavaSerializer.scala:59)
java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1575)
java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1496)
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1732)
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1947)
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1871)
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1753)
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1947)
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1871)
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1753)
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:62)
org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:87)
org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:57)
org.apache.spark.scheduler.Task.run(Task.scala:54)
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:177)
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
java.lang.Thread.run(Thread.java:619)
14/12/01 21:30:10 ERROR TaskSetManager: Task 0 in stage 0.0 failed 4 times; aborting job
Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage 0.0 (TID 3, match1hd12.dc1m): java.lang.ClassNotFoundException: spark.Main$$anonfun$main$1
java.net.URLClassLoader$1.run(URLClassLoader.java:200)
java.security.AccessController.doPrivileged(Native Method)
java.net.URLClassLoader.findClass(URLClassLoader.java:188)
java.lang.ClassLoader.loadClass(ClassLoader.java:307)
java.lang.ClassLoader.loadClass(ClassLoader.java:252)
java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
java.lang.Class.forName0(Native Method)
java.lang.Class.forName(Class.java:247)
org.apache.spark.serializer.JavaDeserializationStream$$anon$1.resolveClass(JavaSerializer.scala:59)
java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1575)
java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1496)
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1732)
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1947)
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1871)
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1753)
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1947)
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1871)
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1753)
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:62)
org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:87)
org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:57)
org.apache.spark.scheduler.Task.run(Task.scala:54)
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:177)
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
java.lang.Thread.run(Thread.java:619)
Driver stacktrace:
at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1185)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1174)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1173)
at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1173)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:688)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:688)
at scala.Option.foreach(Option.scala:236)
at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:688)
at org.apache.spark.scheduler.DAGSchedulerEventProcessActor$$anonfun$receive$2.applyOrElse(DAGScheduler.scala:1391)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:498)
at akka.actor.ActorCell.invoke(ActorCell.scala:456)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:237)
at akka.dispatch.Mailbox.run(Mailbox.scala:219)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
I have spent hours on this already, and I have no idea why this only works with spark-shell. I looked at the stderr output on the individual nodes and they have the same cryptic error message. If anyone can shed some light as to why this only works with spark-shell and not spark-submit that would be awesome.
Thanks
UPDATE:
I've been playing around and the following program works fine.
package spark
import org.apache.spark.sql.hive.HiveContext
import org.apache.spark.{SparkContext, SparkConf}
object Main {
def main(args: Array[String]) {
val sc = new SparkContext(new SparkConf().setAppName("Gathering Data"))
val sqlContext = new HiveContext(sc)
sqlContext.sql("select json from data").take(10).map(t => t.getString(0)).foreach(println)
}
}
Obviously this won't work for a large amount of data, but it shows that the problem appears to be in the ScehmaRDD.map() function.
It seems there is a problem with the spark context initialization.
Please try the below code:
val sparkConf = new SparkConf().setAppName("Gathering Data");
val sc = new SparkContext(sparkConf);

soap client not able to access webservice with saml authentication over SSL enabled

I used the example given at => http://metro.java.net/nonav/1.2/guide/Example_Applications.html#ahies
to try saml authentication over SSL in netbeans 7.3.1 with jdk 7 and glassfish 4.0.
However when I run my client code I'm getting an exception
Exception
javax.xml.ws.WebServiceException)javax.xml.ws.WebServiceException: Failed to access the WSDL at: https://:8181/CalculatorApplication/CalculatorWS?wsdl. It failed with:
java.security.cert.CertificateException: No name matching found.
The exception came in my JSP when I was trying to create the service :
org.me.calculator.client.CalculatorWS_Service service = new org.me.calculator.client.CalculatorWS_Service();
Can anyone who has tried this or has faced similar issue help me with this.
Complete stack trace =>
SEVERE: javax.xml.ws.WebServiceException: Failed to access the WSDL at: https://:8181/CalculatorApplication/CalculatorWS?wsdl. It failed with:
java.security.cert.CertificateException: No name matching found.
at com.sun.xml.ws.wsdl.parser.RuntimeWSDLParser.tryWithMex(RuntimeWSDLParser.java:249)
at com.sun.xml.ws.wsdl.parser.RuntimeWSDLParser.parse(RuntimeWSDLParser.java:230)
at com.sun.xml.ws.wsdl.parser.RuntimeWSDLParser.parse(RuntimeWSDLParser.java:193)
at com.sun.xml.ws.wsdl.parser.RuntimeWSDLParser.parse(RuntimeWSDLParser.java:162)
at com.sun.xml.ws.client.WSServiceDelegate.parseWSDL(WSServiceDelegate.java:359)
at com.sun.xml.ws.client.WSServiceDelegate.(WSServiceDelegate.java:321)
at com.sun.xml.ws.client.WSServiceDelegate.(WSServiceDelegate.java:230)
at com.sun.xml.ws.client.WSServiceDelegate.(WSServiceDelegate.java:212)
at com.sun.xml.ws.client.WSServiceDelegate.(WSServiceDelegate.java:208)
at com.sun.xml.ws.spi.ProviderImpl.createServiceDelegate(ProviderImpl.java:112)
at javax.xml.ws.Service.(Service.java:92)
at org.me.calculator.client.CalculatorWS_Service.(CalculatorWS_Service.java:42)
at org.apache.jsp.index_jsp._jspService(index_jsp.java:58)
at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:111)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:411)
at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:473)
at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:377)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at org.apache.catalina.core.StandardWrapper.service(StandardWrapper.java:1682)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:318)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:160)
at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:734)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:673)
at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:99)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:174)
at org.apache.catalina.connector.CoyoteAdapter.doService(CoyoteAdapter.java:357)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:260)
at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:188)
at org.glassfish.grizzly.http.server.HttpHandler.runService(HttpHandler.java:191)
at org.glassfish.grizzly.http.server.HttpHandler.doHandle(HttpHandler.java:168)
at org.glassfish.grizzly.http.server.HttpServerFilter.handleRead(HttpServerFilter.java:189)
at org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:119)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:288)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:206)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:136)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:114)
at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77)
at org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:838)
at org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:113)
at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.run0(WorkerThreadIOStrategy.java:115)
at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.access$100(WorkerThreadIOStrategy.java:55)
at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy$WorkerThreadRunnable.run(WorkerThreadIOStrategy.java:135)
at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:564)
at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:544)
at java.lang.Thread.run(Thread.java:724)
Caused by: javax.net.ssl.SSLHandshakeException: java.security.cert.CertificateException: No name matching found
at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1884)
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:276)
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:270)
at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1341)
at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:153)
at sun.security.ssl.Handshaker.processLoop(Handshaker.java:868)
at sun.security.ssl.Handshaker.process_record(Handshaker.java:804)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1016)
at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1312)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1339)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1323)
at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:563)
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1300)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:254)
at java.net.URL.openStream(URL.java:1037)
at com.sun.xml.ws.wsdl.parser.RuntimeWSDLParser.createReader(RuntimeWSDLParser.java:983)
at com.sun.xml.ws.wsdl.parser.RuntimeWSDLParser.resolveWSDL(RuntimeWSDLParser.java:384)
at com.sun.xml.ws.wsdl.parser.RuntimeWSDLParser.parse(RuntimeWSDLParser.java:215)
... 44 more
Caused by: java.security.cert.CertificateException: No name matching found
at sun.security.util.HostnameChecker.matchDNS(HostnameChecker.java:208)
at sun.security.util.HostnameChecker.match(HostnameChecker.java:93)
at sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:347)
at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:203)
at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:126)
at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1323)
... 59 more
There is a solution available here[1]. Looks like this is a known issue. I have not tried the solution he is proposing but it looks like the issue is verifying the hostname. He is overwriting the verification method.
public boolean verify(String hostname,
javax.net.ssl.SSLSession sslSession) {
if (hostname.equals("localhost")) {
return true;
}
return false;
}
[1] - http://www.mkyong.com/webservices/jax-ws/java-security-cert-certificateexception-no-name-matching-localhost-found/