I am trying to read a file in an S3 bucket using Spark through Databricks Connect.
This is the code that I am using,
from pyspark import SparkConf
from pyspark.sql import SparkSession
conf = SparkConf()
conf.set('spark.jars.packages', 'org.apache.hadoop:hadoop-aws:3.3.0')
conf.set('spark.hadoop.fs.s3a.access.key', access_key)
conf.set('spark.hadoop.fs.s3a.secret.key', secret_access_key)
spark = SparkSession.builder.config(conf=conf).getOrCreate()
df = spark.read.format("csv").option("header",True).load('s3a://container/path/to/file.csv')
df.show()
This works completely fine when I execute it using a Docker container that I spin up, however, it fails with Databricks Connect with the following error,
pyspark.dbutils.ExecutionError: An error occurred while calling o48.ls.
: com.databricks.service.SparkServiceRemoteException: java.nio.file.AccessDeniedException: getFileStatus on com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden; request: HEAD Forbidden
at shaded.databricks.org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:244)
at shaded.databricks.org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:155)
at shaded.databricks.org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:2870)
at shaded.databricks.org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:2840)
at shaded.databricks.org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:2779)
at shaded.databricks.org.apache.hadoop.fs.s3a.S3AFileSystem.innerListStatus(S3AFileSystem.java:2449)
at shaded.databricks.org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$listStatus$11(S3AFileSystem.java:2428)
at shaded.databricks.org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:118)
at shaded.databricks.org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:112)
at shaded.databricks.org.apache.hadoop.fs.s3a.S3AFileSystem.listStatus(S3AFileSystem.java:2428)
at com.databricks.service.SparkServiceImpl$.$anonfun$fileSystemOperation0$2(SparkServiceImpl.scala:617)
at com.databricks.service.SparkServiceImpl$.withFileSystemExceptionHandler(SparkServiceImpl.scala:647)
at com.databricks.service.SparkServiceImpl$.fileSystemOperation0(SparkServiceImpl.scala:617)
at com.databricks.service.SparkServiceImpl$.$anonfun$fileSystemOperation$1(SparkServiceImpl.scala:184)
at com.databricks.logging.UsageLogging.$anonfun$recordOperation$4(UsageLogging.scala:431)
at com.databricks.logging.UsageLogging.$anonfun$withAttributionContext$1(UsageLogging.scala:239)
at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62)
at com.databricks.logging.UsageLogging.withAttributionContext(UsageLogging.scala:234)
at com.databricks.logging.UsageLogging.withAttributionContext$(UsageLogging.scala:231)
at com.databricks.spark.util.PublicDBLogging.withAttributionContext(DatabricksSparkUsageLogger.scala:19)
at com.databricks.logging.UsageLogging.withAttributionTags(UsageLogging.scala:276)
at com.databricks.logging.UsageLogging.withAttributionTags$(UsageLogging.scala:269)
at com.databricks.spark.util.PublicDBLogging.withAttributionTags(DatabricksSparkUsageLogger.scala:19)
at com.databricks.logging.UsageLogging.recordOperation(UsageLogging.scala:412)
at com.databricks.logging.UsageLogging.recordOperation$(UsageLogging.scala:338)
at com.databricks.spark.util.PublicDBLogging.recordOperation(DatabricksSparkUsageLogger.scala:19)
at com.databricks.spark.util.PublicDBLogging.recordOperation0(DatabricksSparkUsageLogger.scala:56)
at com.databricks.spark.util.DatabricksSparkUsageLogger.recordOperation(DatabricksSparkUsageLogger.scala:131)
at com.databricks.spark.util.UsageLogger.recordOperation(UsageLogger.scala:71)
at com.databricks.spark.util.UsageLogger.recordOperation$(UsageLogger.scala:58)
at com.databricks.spark.util.DatabricksSparkUsageLogger.recordOperation(DatabricksSparkUsageLogger.scala:85)
at com.databricks.spark.util.UsageLogging.recordOperation(UsageLogger.scala:401)
at com.databricks.spark.util.UsageLogging.recordOperation$(UsageLogger.scala:380)
at com.databricks.service.SparkServiceImpl$.recordOperation(SparkServiceImpl.scala:92)
at com.databricks.service.SparkServiceImpl$.fileSystemOperation(SparkServiceImpl.scala:184)
at com.databricks.service.SparkServiceRPCHandler.execute0(SparkServiceRPCHandler.scala:663)
at com.databricks.service.SparkServiceRPCHandler.$anonfun$executeRPC0$1(SparkServiceRPCHandler.scala:451)
at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62)
at com.databricks.service.SparkServiceRPCHandler.executeRPC0(SparkServiceRPCHandler.scala:351)
at com.databricks.service.SparkServiceRPCHandler$$anon$2.call(SparkServiceRPCHandler.scala:302)
at com.databricks.service.SparkServiceRPCHandler$$anon$2.call(SparkServiceRPCHandler.scala:288)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at com.databricks.service.SparkServiceRPCHandler.$anonfun$executeRPC$1(SparkServiceRPCHandler.scala:338)
at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62)
at com.databricks.service.SparkServiceRPCHandler.executeRPC(SparkServiceRPCHandler.scala:315)
at com.databricks.service.SparkServiceRPCServlet.doPost(SparkServiceRPCServer.scala:152)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:873)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:542)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:205)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at org.eclipse.jetty.server.Server.handle(Server.java:505)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:370)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:267)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:698)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:804)
at java.lang.Thread.run(Thread.java:750)
Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden; request: HEAD
Note: I've scrubbed some information related to the path of my file in the above error thread.
Related
I have created a databricks account and added my account via root AWS access keys (I know this is not recommended but had tried via cross acount role and this was not working either). These root access keys should have access to my entire AWS account, however when I call.
dbutils.fs.ls('/mnt')
Just a basic listing of the files in my directory, I get this error:
ExecutionError: An error occurred while calling z:com.databricks.backend.daemon.dbutils.FSUtils.ls.
: java.lang.RuntimeException: java.io.IOException: Failed to perform 'getMountFileState(forceRefresh=true)' for mounts after 3 attempts. Please, retry the operation.
Original exception: 'com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden; request: HEAD https://auto-emailer.s3.amazonaws.com oregon-prod/3323621803827190.meta/mounts {} aws-sdk-java/1.11.602 Linux/4.4.0-1104-aws OpenJDK_64-Bit_Server_VM/25.242-b08 java/1.8.0_242 scala/2.12.10 vendor/Private_Build com.amazonaws.services.s3.model.GetObjectMetadataRequest; Request ID: 640F00A5A05F37B3, Extended Request ID: GMSFv0QNFog8PqLBYz8Hre9W5ttMpCLOKNQPPingLc2t1/XFUxn5RdkdBtIYGUbeHYVvIkIM7IY=, Cloud Provider: AWS, Instance ID: i-0acc5354538e76431 (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden; Request ID: 640F00A5A05F37B3; S3 Extended Request ID: GMSFv0QNFog8PqLBYz8Hre9W5ttMpCLOKNQPPingLc2t1/XFUxn5RdkdBtIYGUbeHYVvIkIM7IY=), S3 Extended Request ID: GMSFv0QNFog8PqLBYz8Hre9W5ttMpCLOKNQPPingLc2t1/XFUxn5RdkdBtIYGUbeHYVvIkIM7IY=
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1712)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1367)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1113)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:770)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:744)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:726)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:686)
at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:668)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:532)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:512)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4921)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4867)
at com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1320)
at com.databricks.s3a.aws.EnforcingDatabricksS3Client.$anonfun$getObjectMetadata$1(EnforcingDatabricksS3Client.scala:223)
at scala.util.Try$.apply(Try.scala:213)
at com.databricks.s3a.aws.DatabricksS3Client.retryRequest(DatabricksS3Client.scala:137)
at com.databricks.s3a.aws.DatabricksS3Client.withExponentialBackoff(DatabricksS3Client.scala:111)
at com.databricks.s3a.aws.DatabricksS3Client.withExponentialBackoff$(DatabricksS3Client.scala:109)
at com.databricks.s3a.aws.EnforcingDatabricksS3Client.withExponentialBackoff(EnforcingDatabricksS3Client.scala:28)
at com.databricks.s3a.aws.EnforcingDatabricksS3Client.getObjectMetadata(EnforcingDatabricksS3Client.scala:223)
at com.databricks.s3a.S3AFileSystem.getObjectMetadata(S3AFileSystem.java:1948)
at com.databricks.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:1450)
at com.databricks.s3a.S3AFileSystem.open(S3AFileSystem.java:663)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:769)
at com.databricks.backend.daemon.data.server.DefaultMetadataManager.liftedTree1$1(MetadataManager.scala:156)
at com.databricks.backend.daemon.data.server.DefaultMetadataManager.$anonfun$getMountFileState$1(MetadataManager.scala:151)
at com.databricks.backend.daemon.data.server.DefaultMetadataManager.withRetries(MetadataManager.scala:326)
at com.databricks.backend.daemon.data.server.DefaultMetadataManager.getMountFileState(MetadataManager.scala:150)
at com.databricks.backend.daemon.data.server.DefaultMetadataManager.getMounts(MetadataManager.scala:200)
at com.databricks.backend.daemon.data.filesystem.MountEntryResolver.retrieveFilesystems(MountEntryResolver.scala:76)
at com.databricks.backend.daemon.data.filesystem.MountEntryResolver.<init>(MountEntryResolver.scala:33)
at com.databricks.backend.daemon.data.server.DbfsServerBackend.makeSessionContext(DbfsServerBackend.scala:242)
at com.databricks.backend.daemon.data.server.DbfsServerBackend.$anonfun$receive$2(DbfsServerBackend.scala:280)
at com.databricks.backend.daemon.data.server.session.SessionManager.createSession(SessionManager.scala:193)
at com.databricks.backend.daemon.data.server.DbfsServerBackend$$anonfun$receive$3.applyOrElse(DbfsServerBackend.scala:285)
at com.databricks.backend.daemon.data.server.DbfsServerBackend$$anonfun$receive$3.applyOrElse(DbfsServerBackend.scala:282)
at com.databricks.rpc.ServerBackend.$anonfun$internalReceive$2(ServerBackend.scala:52)
at com.databricks.rpc.ServerBackend$$anonfun$commonReceive$1.applyOrElse(ServerBackend.scala:79)
at com.databricks.rpc.ServerBackend$$anonfun$commonReceive$1.applyOrElse(ServerBackend.scala:79)
at com.databricks.rpc.ServerBackend.$anonfun$internalReceive$1(ServerBackend.scala:48)
at com.databricks.logging.UsageLogging.$anonfun$recordOperation$4(UsageLogging.scala:428)
at com.databricks.logging.UsageLogging.$anonfun$withAttributionContext$1(UsageLogging.scala:238)
at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62)
at com.databricks.logging.UsageLogging.withAttributionContext(UsageLogging.scala:233)
at com.databricks.logging.UsageLogging.withAttributionContext$(UsageLogging.scala:230)
at com.databricks.rpc.ServerBackend.withAttributionContext(ServerBackend.scala:15)
at com.databricks.logging.UsageLogging.withAttributionTags(UsageLogging.scala:275)
at com.databricks.logging.UsageLogging.withAttributionTags$(UsageLogging.scala:268)
at com.databricks.rpc.ServerBackend.withAttributionTags(ServerBackend.scala:15)
at com.databricks.logging.UsageLogging.recordOperation(UsageLogging.scala:409)
at com.databricks.logging.UsageLogging.recordOperation$(UsageLogging.scala:336)
at com.databricks.rpc.ServerBackend.recordOperation(ServerBackend.scala:15)
at com.databricks.rpc.ServerBackend.internalReceive(ServerBackend.scala:47)
at com.databricks.rpc.JettyServer$RequestManager.$anonfun$handleRPC$2(JettyServer.scala:611)
at scala.util.Try$.apply(Try.scala:213)
at com.databricks.rpc.JettyServer$RequestManager.handleRPC(JettyServer.scala:611)
at com.databricks.rpc.JettyServer$RequestManager.handleRequestAndRespond(JettyServer.scala:534)
at com.databricks.rpc.JettyServer$RequestManager.$anonfun$handleHttp$4(JettyServer.scala:321)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at com.databricks.logging.UsageLogging.$anonfun$withAttributionContext$1(UsageLogging.scala:238)
at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62)
at com.databricks.logging.UsageLogging.withAttributionContext(UsageLogging.scala:233)
at com.databricks.logging.UsageLogging.withAttributionContext$(UsageLogging.scala:230)
at com.databricks.rpc.JettyServer$.withAttributionContext(JettyServer.scala:152)
at com.databricks.logging.UsageLogging.withAttributionTags(UsageLogging.scala:275)
at com.databricks.logging.UsageLogging.withAttributionTags$(UsageLogging.scala:268)
at com.databricks.rpc.JettyServer$.withAttributionTags(JettyServer.scala:152)
at com.databricks.rpc.JettyServer$RequestManager.handleHttp(JettyServer.scala:310)
at com.databricks.rpc.JettyServer$RequestManager.doPost(JettyServer.scala:217)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at com.databricks.rpc.HttpServletWithPatch.service(HttpServletWithPatch.scala:33)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:848)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
at org.eclipse.jetty.server.Server.handle(Server.java:539)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
at java.lang.Thread.run(Thread.java:748)
'
at org.apache.hadoop.mapred.JobConf.getWorkingDirectory(JobConf.java:660)
at org.apache.hadoop.mapred.FileInputFormat.setInputPaths(FileInputFormat.java:438)
at org.apache.hadoop.mapred.FileInputFormat.setInputPaths(FileInputFormat.java:411)
at org.apache.spark.api.python.PythonSecurityUtils$.resolvePaths(PythonSecurityUtils.scala:246)
at org.apache.spark.api.python.PythonSecurityUtils$.checkPathStringFileSystemSafety(PythonSecurityUtils.scala:258)
at com.databricks.backend.daemon.dbutils.FSUtils$.com$databricks$backend$daemon$dbutils$FSUtils$$withFsSafetyCheck(DBUtilsCore.scala:80)
at com.databricks.backend.daemon.dbutils.FSUtils$.ls(DBUtilsCore.scala:85)
at com.databricks.backend.daemon.dbutils.FSUtils.ls(DBUtilsCore.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:380)
at py4j.Gateway.invoke(Gateway.java:295)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.GatewayConnection.run(GatewayConnection.java:251)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: Failed to perform 'getMountFileState(forceRefresh=true)' for mounts after 3 attempts. Please, retry the operation.
Original exception: 'com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden; request: HEAD https://auto-emailer.s3.amazonaws.com oregon-prod/3323621803827190.meta/mounts {} aws-sdk-java/1.11.602 Linux/4.4.0-1104-aws OpenJDK_64-Bit_Server_VM/25.242-b08 java/1.8.0_242 scala/2.12.10 vendor/Private_Build com.amazonaws.services.s3.model.GetObjectMetadataRequest; Request ID: 640F00A5A05F37B3, Extended Request ID: GMSFv0QNFog8PqLBYz8Hre9W5ttMpCLOKNQPPingLc2t1/XFUxn5RdkdBtIYGUbeHYVvIkIM7IY=, Cloud Provider: AWS, Instance ID: i-0acc5354538e76431 (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden; Request ID: 640F00A5A05F37B3; S3 Extended Request ID: GMSFv0QNFog8PqLBYz8Hre9W5ttMpCLOKNQPPingLc2t1/XFUxn5RdkdBtIYGUbeHYVvIkIM7IY=), S3 Extended Request ID: GMSFv0QNFog8PqLBYz8Hre9W5ttMpCLOKNQPPingLc2t1/XFUxn5RdkdBtIYGUbeHYVvIkIM7IY=
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1712)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1367)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1113)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:770)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:744)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:726)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:686)
at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:668)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:532)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:512)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4921)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4867)
at com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1320)
at com.databricks.s3a.aws.EnforcingDatabricksS3Client.$anonfun$getObjectMetadata$1(EnforcingDatabricksS3Client.scala:223)
at scala.util.Try$.apply(Try.scala:213)
at com.databricks.s3a.aws.DatabricksS3Client.retryRequest(DatabricksS3Client.scala:137)
at com.databricks.s3a.aws.DatabricksS3Client.withExponentialBackoff(DatabricksS3Client.scala:111)
at com.databricks.s3a.aws.DatabricksS3Client.withExponentialBackoff$(DatabricksS3Client.scala:109)
at com.databricks.s3a.aws.EnforcingDatabricksS3Client.withExponentialBackoff(EnforcingDatabricksS3Client.scala:28)
at com.databricks.s3a.aws.EnforcingDatabricksS3Client.getObjectMetadata(EnforcingDatabricksS3Client.scala:223)
at com.databricks.s3a.S3AFileSystem.getObjectMetadata(S3AFileSystem.java:1948)
at com.databricks.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:1450)
at com.databricks.s3a.S3AFileSystem.open(S3AFileSystem.java:663)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:769)
at com.databricks.backend.daemon.data.server.DefaultMetadataManager.liftedTree1$1(MetadataManager.scala:156)
at com.databricks.backend.daemon.data.server.DefaultMetadataManager.$anonfun$getMountFileState$1(MetadataManager.scala:151)
at com.databricks.backend.daemon.data.server.DefaultMetadataManager.withRetries(MetadataManager.scala:326)
at com.databricks.backend.daemon.data.server.DefaultMetadataManager.getMountFileState(MetadataManager.scala:150)
at com.databricks.backend.daemon.data.server.DefaultMetadataManager.getMounts(MetadataManager.scala:200)
at com.databricks.backend.daemon.data.filesystem.MountEntryResolver.retrieveFilesystems(MountEntryResolver.scala:76)
at com.databricks.backend.daemon.data.filesystem.MountEntryResolver.<init>(MountEntryResolver.scala:33)
at com.databricks.backend.daemon.data.server.DbfsServerBackend.makeSessionContext(DbfsServerBackend.scala:242)
at com.databricks.backend.daemon.data.server.DbfsServerBackend.$anonfun$receive$2(DbfsServerBackend.scala:280)
at com.databricks.backend.daemon.data.server.session.SessionManager.createSession(SessionManager.scala:193)
at com.databricks.backend.daemon.data.server.DbfsServerBackend$$anonfun$receive$3.applyOrElse(DbfsServerBackend.scala:285)
at com.databricks.backend.daemon.data.server.DbfsServerBackend$$anonfun$receive$3.applyOrElse(DbfsServerBackend.scala:282)
at com.databricks.rpc.ServerBackend.$anonfun$internalReceive$2(ServerBackend.scala:52)
at com.databricks.rpc.ServerBackend$$anonfun$commonReceive$1.applyOrElse(ServerBackend.scala:79)
at com.databricks.rpc.ServerBackend$$anonfun$commonReceive$1.applyOrElse(ServerBackend.scala:79)
at com.databricks.rpc.ServerBackend.$anonfun$internalReceive$1(ServerBackend.scala:48)
at com.databricks.logging.UsageLogging.$anonfun$recordOperation$4(UsageLogging.scala:428)
at com.databricks.logging.UsageLogging.$anonfun$withAttributionContext$1(UsageLogging.scala:238)
at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62)
at com.databricks.logging.UsageLogging.withAttributionContext(UsageLogging.scala:233)
at com.databricks.logging.UsageLogging.withAttributionContext$(UsageLogging.scala:230)
at com.databricks.rpc.ServerBackend.withAttributionContext(ServerBackend.scala:15)
at com.databricks.logging.UsageLogging.withAttributionTags(UsageLogging.scala:275)
at com.databricks.logging.UsageLogging.withAttributionTags$(UsageLogging.scala:268)
at com.databricks.rpc.ServerBackend.withAttributionTags(ServerBackend.scala:15)
at com.databricks.logging.UsageLogging.recordOperation(UsageLogging.scala:409)
at com.databricks.logging.UsageLogging.recordOperation$(UsageLogging.scala:336)
at com.databricks.rpc.ServerBackend.recordOperation(ServerBackend.scala:15)
at com.databricks.rpc.ServerBackend.internalReceive(ServerBackend.scala:47)
at com.databricks.rpc.JettyServer$RequestManager.$anonfun$handleRPC$2(JettyServer.scala:611)
at scala.util.Try$.apply(Try.scala:213)
at com.databricks.rpc.JettyServer$RequestManager.handleRPC(JettyServer.scala:611)
at com.databricks.rpc.JettyServer$RequestManager.handleRequestAndRespond(JettyServer.scala:534)
at com.databricks.rpc.JettyServer$RequestManager.$anonfun$handleHttp$4(JettyServer.scala:321)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at com.databricks.logging.UsageLogging.$anonfun$withAttributionContext$1(UsageLogging.scala:238)
at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62)
at com.databricks.logging.UsageLogging.withAttributionContext(UsageLogging.scala:233)
at com.databricks.logging.UsageLogging.withAttributionContext$(UsageLogging.scala:230)
at com.databricks.rpc.JettyServer$.withAttributionContext(JettyServer.scala:152)
at com.databricks.logging.UsageLogging.withAttributionTags(UsageLogging.scala:275)
at com.databricks.logging.UsageLogging.withAttributionTags$(UsageLogging.scala:268)
at com.databricks.rpc.JettyServer$.withAttributionTags(JettyServer.scala:152)
at com.databricks.rpc.JettyServer$RequestManager.handleHttp(JettyServer.scala:310)
at com.databricks.rpc.JettyServer$RequestManager.doPost(JettyServer.scala:217)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at com.databricks.rpc.HttpServletWithPatch.service(HttpServletWithPatch.scala:33)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:848)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
at org.eclipse.jetty.server.Server.handle(Server.java:539)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
at java.lang.Thread.run(Thread.java:748)
'
at com.databricks.backend.daemon.data.client.DbfsClient.send0(DbfsClient.scala:117)
at com.databricks.backend.daemon.data.client.DbfsClient.sendIdempotent(DbfsClient.scala:63)
at com.databricks.backend.daemon.data.client.DatabricksMountsStore.com$databricks$backend$daemon$data$client$DatabricksMountsStore$$refreshMounts(DatabricksMountsStore.scala:92)
at com.databricks.backend.daemon.data.client.DatabricksMountsStore.forceRefreshMounts(DatabricksMountsStore.scala:127)
at com.databricks.backend.daemon.data.client.SharedDatabricksMountsStore$.getOrCreate(DatabricksMountsStore.scala:189)
at com.databricks.backend.daemon.data.client.DBFSV2$$anonfun$initialize$2.apply(DatabricksFileSystemV2.scala:72)
at com.databricks.backend.daemon.data.client.DBFSV2$$anonfun$initialize$2.apply(DatabricksFileSystemV2.scala:64)
at com.databricks.logging.UsageLogging$$anonfun$recordOperation$1.apply(UsageLogging.scala:428)
at com.databricks.logging.UsageLogging$$anonfun$withAttributionContext$1.apply(UsageLogging.scala:238)
at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58)
at com.databricks.logging.UsageLogging$class.withAttributionContext(UsageLogging.scala:233)
at com.databricks.backend.daemon.data.client.DatabricksFileSystemV2.withAttributionContext(DatabricksFileSystemV2.scala:450)
at com.databricks.logging.UsageLogging$class.withAttributionTags(UsageLogging.scala:275)
at com.databricks.backend.daemon.data.client.DatabricksFileSystemV2.withAttributionTags(DatabricksFileSystemV2.scala:450)
at com.databricks.logging.UsageLogging$class.recordOperation(UsageLogging.scala:409)
at com.databricks.backend.daemon.data.client.DatabricksFileSystemV2.recordOperation(DatabricksFileSystemV2.scala:450)
at com.databricks.backend.daemon.data.client.DBFSV2.initialize(DatabricksFileSystemV2.scala:64)
at com.databricks.backend.daemon.data.client.DatabricksFileSystem.initialize(DatabricksFileSystem.scala:226)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2669)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:94)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2703)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2685)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:373)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:172)
at org.apache.hadoop.mapred.JobConf.getWorkingDirectory(JobConf.java:656)
... 18 more
Caused by: java.io.IOException: Failed to perform 'getMountFileState(forceRefresh=true)' for mounts after 3 attempts. Please, retry the operation.
Original exception: 'com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden; request: HEAD https://auto-emailer.s3.amazonaws.com oregon-prod/3323621803827190.meta/mounts {} aws-sdk-java/1.11.602 Linux/4.4.0-1104-aws OpenJDK_64-Bit_Server_VM/25.242-b08 java/1.8.0_242 scala/2.12.10 vendor/Private_Build com.amazonaws.services.s3.model.GetObjectMetadataRequest; Request ID: 640F00A5A05F37B3, Extended Request ID: GMSFv0QNFog8PqLBYz8Hre9W5ttMpCLOKNQPPingLc2t1/XFUxn5RdkdBtIYGUbeHYVvIkIM7IY=, Cloud Provider: AWS, Instance ID: i-0acc5354538e76431 (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden; Request ID: 640F00A5A05F37B3; S3 Extended Request ID: GMSFv0QNFog8PqLBYz8Hre9W5ttMpCLOKNQPPingLc2t1/XFUxn5RdkdBtIYGUbeHYVvIkIM7IY=), S3 Extended Request ID: GMSFv0QNFog8PqLBYz8Hre9W5ttMpCLOKNQPPingLc2t1/XFUxn5RdkdBtIYGUbeHYVvIkIM7IY=
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1712)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1367)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1113)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:770)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:744)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:726)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:686)
at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:668)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:532)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:512)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4921)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4867)
at com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1320)
at com.databricks.s3a.aws.EnforcingDatabricksS3Client.$anonfun$getObjectMetadata$1(EnforcingDatabricksS3Client.scala:223)
at scala.util.Try$.apply(Try.scala:213)
at com.databricks.s3a.aws.DatabricksS3Client.retryRequest(DatabricksS3Client.scala:137)
at com.databricks.s3a.aws.DatabricksS3Client.withExponentialBackoff(DatabricksS3Client.scala:111)
at com.databricks.s3a.aws.DatabricksS3Client.withExponentialBackoff$(DatabricksS3Client.scala:109)
at com.databricks.s3a.aws.EnforcingDatabricksS3Client.withExponentialBackoff(EnforcingDatabricksS3Client.scala:28)
at com.databricks.s3a.aws.EnforcingDatabricksS3Client.getObjectMetadata(EnforcingDatabricksS3Client.scala:223)
at com.databricks.s3a.S3AFileSystem.getObjectMetadata(S3AFileSystem.java:1948)
at com.databricks.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:1450)
at com.databricks.s3a.S3AFileSystem.open(S3AFileSystem.java:663)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:769)
at com.databricks.backend.daemon.data.server.DefaultMetadataManager.liftedTree1$1(MetadataManager.scala:156)
at com.databricks.backend.daemon.data.server.DefaultMetadataManager.$anonfun$getMountFileState$1(MetadataManager.scala:151)
at com.databricks.backend.daemon.data.server.DefaultMetadataManager.withRetries(MetadataManager.scala:326)
at com.databricks.backend.daemon.data.server.DefaultMetadataManager.getMountFileState(MetadataManager.scala:150)
at com.databricks.backend.daemon.data.server.DefaultMetadataManager.getMounts(MetadataManager.scala:200)
at com.databricks.backend.daemon.data.filesystem.MountEntryResolver.retrieveFilesystems(MountEntryResolver.scala:76)
at com.databricks.backend.daemon.data.filesystem.MountEntryResolver.<init>(MountEntryResolver.scala:33)
at com.databricks.backend.daemon.data.server.DbfsServerBackend.makeSessionContext(DbfsServerBackend.scala:242)
at com.databricks.backend.daemon.data.server.DbfsServerBackend.$anonfun$receive$2(DbfsServerBackend.scala:280)
at com.databricks.backend.daemon.data.server.session.SessionManager.createSession(SessionManager.scala:193)
at com.databricks.backend.daemon.data.server.DbfsServerBackend$$anonfun$receive$3.applyOrElse(DbfsServerBackend.scala:285)
at com.databricks.backend.daemon.data.server.DbfsServerBackend$$anonfun$receive$3.applyOrElse(DbfsServerBackend.scala:282)
at com.databricks.rpc.ServerBackend.$anonfun$internalReceive$2(ServerBackend.scala:52)
at com.databricks.rpc.ServerBackend$$anonfun$commonReceive$1.applyOrElse(ServerBackend.scala:79)
at com.databricks.rpc.ServerBackend$$anonfun$commonReceive$1.applyOrElse(ServerBackend.scala:79)
at com.databricks.rpc.ServerBackend.$anonfun$internalReceive$1(ServerBackend.scala:48)
at com.databricks.logging.UsageLogging.$anonfun$recordOperation$4(UsageLogging.scala:428)
at com.databricks.logging.UsageLogging.$anonfun$withAttributionContext$1(UsageLogging.scala:238)
at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62)
at com.databricks.logging.UsageLogging.withAttributionContext(UsageLogging.scala:233)
at com.databricks.logging.UsageLogging.withAttributionContext$(UsageLogging.scala:230)
at com.databricks.rpc.ServerBackend.withAttributionContext(ServerBackend.scala:15)
at com.databricks.logging.UsageLogging.withAttributionTags(UsageLogging.scala:275)
at com.databricks.logging.UsageLogging.withAttributionTags$(UsageLogging.scala:268)
at com.databricks.rpc.ServerBackend.withAttributionTags(ServerBackend.scala:15)
at com.databricks.logging.UsageLogging.recordOperation(UsageLogging.scala:409)
at com.databricks.logging.UsageLogging.recordOperation$(UsageLogging.scala:336)
at com.databricks.rpc.ServerBackend.recordOperation(ServerBackend.scala:15)
at com.databricks.rpc.ServerBackend.internalReceive(ServerBackend.scala:47)
at com.databricks.rpc.JettyServer$RequestManager.$anonfun$handleRPC$2(JettyServer.scala:611)
at scala.util.Try$.apply(Try.scala:213)
at com.databricks.rpc.JettyServer$RequestManager.handleRPC(JettyServer.scala:611)
at com.databricks.rpc.JettyServer$RequestManager.handleRequestAndRespond(JettyServer.scala:534)
I see in the stack trace that original exception: 'com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden; but I don't understand why things are forbidden, this account is set up and the cluster was launched using my root access keys, so Databricks should have access to everything. Any help here would be greatly appreciated!
To be clear, the end goal here is to mount a bucket from s3 to databricks so I can access it, but when I tried mounting I got the same error
There're more than one ways to limit access to a resource in AWS:
limit user/role permissions (not your case as you're using root)
limit operations on resource side, for example allow only VPC traffic (i.e. no external traffic) or allow GET but not HEAD, allow only this specific user to do HEAD, and so on.
Check your S3 bucket access (or provide it here): explicitly allow your user to do list, get, put objects.
Hopefully it makes sense to make a question like this here:
I have an python3 api which it uses Flask to serve. I also have an clojure app that makes a post request targeting the python3 api. To do so I use clj-http with something like:
(client/post url-link
{:basic-auth ["username"
"password"]
:form-params json
:content-type :json})
Now, when I test in my machine serving the python3 api locally, i.e,
(def url-link "localhost:port/api-name")
it works smoothly. The issue comes because I will serve the python3 api from the amazon server. So my thought was to use the link of the amazon server. But when I do so, clojure complains with
HTTP ERROR 500 clojure.lang.ExceptionInfo: Malformed application/json in :muuntaja/encode {:type :muuntaja/encode, :format "application/json"}
URI: /api/sell-call
STATUS: 500
MESSAGE: clojure.lang.ExceptionInfo: Malformed application/json in :muuntaja/encode {:type :muuntaja/encode, :format "application/json"}
SERVLET: -
CAUSED BY: clojure.lang.ExceptionInfo: Malformed application/json in :muuntaja/encode {:type :muuntaja/encode, :format "application/json"}
CAUSED BY: com.fasterxml.jackson.databind.JsonMappingException: (was java.lang.UnsupportedOperationException) (through reference chain: clojure.lang.PersistentArrayMap[":full_message"]->clojure.lang.ExceptionInfo["data"]->clojure.lang.PersistentHashMap[":http-client"]->org.apache.http.impl.client.InternalHttpClient["params"])
CAUSED BY: java.lang.UnsupportedOperationException
clojure.lang.ExceptionInfo: Malformed application/json in :muuntaja/encode {:type :muuntaja/encode, :format "application/json"}
at muuntaja.core$on_exception.invokeStatic(core.clj:284)
at muuntaja.core$on_exception.invoke(core.clj:267)
at clojure.core$partial$fn__5824.invoke(core.clj:2626)
at muuntaja.core$create_coder$encode__10562.invoke(core.clj:346)
at clojure.core$update.invokeStatic(core.clj:6198)
at clojure.core$update.invoke(core.clj:6188)
at muuntaja.core$create$_handle_response__10629.invoke(core.clj:443)
at muuntaja.core$create$reify__10631.format_response(core.clj:486)
at muuntaja.middleware$wrap_format_response$fn__10702.invoke(middleware.clj:132)
at muuntaja.middleware$wrap_format_negotiate$fn__10695.invoke(middleware.clj:96)
at ring.middleware.keyword_params$wrap_keyword_params$fn__9403.invoke(keyword_params.clj:53)
at ring.middleware.nested_params$wrap_nested_params$fn__9461.invoke(nested_params.clj:89)
at ring.middleware.params$wrap_params$fn__9535.invoke(params.clj:67)
at compojure.api.middleware$wrap_inject_data$fn__11689.invoke(middleware.clj:96)
at compojure.api.routes.Route.invoke(routes.clj:89)
at ring.middleware.basic_authentication$wrap_basic_authentication$fn__16405.invoke(basic_authentication.clj:157)
at clojure.lang.Var.invoke(Var.java:384)
at ring.middleware.reload$wrap_reload$fn__1065.invoke(reload.clj:39)
at ring.adapter.jetty$proxy_handler$fn__480.invoke(jetty.clj:27)
at ring.adapter.jetty.proxy$org.eclipse.jetty.server.handler.AbstractHandler$ff19274a.handle(Unknown Source)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.Server.handle(Server.java:500)
at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:386)
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:562)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:378)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:270)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: com.fasterxml.jackson.databind.JsonMappingException: (was java.lang.UnsupportedOperationException) (through reference chain: clojure.lang.PersistentArrayMap[":full_message"]->clojure.lang.ExceptionInfo["data"]->clojure.lang.PersistentHashMap[":http-client"]->org.apache.http.impl.client.InternalHttpClient["params"])
at com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:394)
at com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:353)
at com.fasterxml.jackson.databind.ser.std.StdSerializer.wrapAndThrow(StdSerializer.java:316)
at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:727)
at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:155)
at com.fasterxml.jackson.databind.ser.std.MapSerializer.serializeFields(MapSerializer.java:722)
at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:643)
at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:33)
at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:727)
at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:719)
at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:155)
at com.fasterxml.jackson.databind.ser.std.MapSerializer.serializeFields(MapSerializer.java:722)
at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:643)
at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:33)
at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider._serialize(DefaultSerializerProvider.java:480)
at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:319)
at com.fasterxml.jackson.databind.ObjectMapper._configAndWriteValue(ObjectMapper.java:3905)
at com.fasterxml.jackson.databind.ObjectMapper.writeValueAsBytes(ObjectMapper.java:3243)
at jsonista.core$write_value_as_bytes.invokeStatic(core.clj:229)
at jsonista.core$write_value_as_bytes.invoke(core.clj:221)
at muuntaja.format.json$encoder$reify__10016.encode_to_bytes(json.clj:43)
at muuntaja.core$create_coder$encode__10562.invoke(core.clj:344)
... 28 more
Caused by: java.lang.UnsupportedOperationException
at org.apache.http.impl.client.InternalHttpClient.getParams(InternalHttpClient.java:211)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:688)
at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:719)
... 46 more
com.fasterxml.jackson.databind.JsonMappingException: (was java.lang.UnsupportedOperationException) (through reference chain: clojure.lang.PersistentArrayMap[":full_message"]->clojure.lang.ExceptionInfo["data"]->clojure.lang.PersistentHashMap[":http-client"]->org.apache.http.impl.client.InternalHttpClient["params"])
at com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:394)
at com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:353)
at com.fasterxml.jackson.databind.ser.std.StdSerializer.wrapAndThrow(StdSerializer.java:316)
at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:727)
at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:155)
at com.fasterxml.jackson.databind.ser.std.MapSerializer.serializeFields(MapSerializer.java:722)
at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:643)
at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:33)
at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:727)
at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:719)
at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:155)
at com.fasterxml.jackson.databind.ser.std.MapSerializer.serializeFields(MapSerializer.java:722)
at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:643)
at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:33)
at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider._serialize(DefaultSerializerProvider.java:480)
at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:319)
at com.fasterxml.jackson.databind.ObjectMapper._configAndWriteValue(ObjectMapper.java:3905)
at com.fasterxml.jackson.databind.ObjectMapper.writeValueAsBytes(ObjectMapper.java:3243)
at jsonista.core$write_value_as_bytes.invokeStatic(core.clj:229)
at jsonista.core$write_value_as_bytes.invoke(core.clj:221)
at muuntaja.format.json$encoder$reify__10016.encode_to_bytes(json.clj:43)
at muuntaja.core$create_coder$encode__10562.invoke(core.clj:344)
at clojure.core$update.invokeStatic(core.clj:6198)
at clojure.core$update.invoke(core.clj:6188)
at muuntaja.core$create$_handle_response__10629.invoke(core.clj:443)
at muuntaja.core$create$reify__10631.format_response(core.clj:486)
at muuntaja.middleware$wrap_format_response$fn__10702.invoke(middleware.clj:132)
at muuntaja.middleware$wrap_format_negotiate$fn__10695.invoke(middleware.clj:96)
at ring.middleware.keyword_params$wrap_keyword_params$fn__9403.invoke(keyword_params.clj:53)
at ring.middleware.nested_params$wrap_nested_params$fn__9461.invoke(nested_params.clj:89)
at ring.middleware.params$wrap_params$fn__9535.invoke(params.clj:67)
at compojure.api.middleware$wrap_inject_data$fn__11689.invoke(middleware.clj:96)
at compojure.api.routes.Route.invoke(routes.clj:89)
at ring.middleware.basic_authentication$wrap_basic_authentication$fn__16405.invoke(basic_authentication.clj:157)
at clojure.lang.Var.invoke(Var.java:384)
at ring.middleware.reload$wrap_reload$fn__1065.invoke(reload.clj:39)
at ring.adapter.jetty$proxy_handler$fn__480.invoke(jetty.clj:27)
at ring.adapter.jetty.proxy$org.eclipse.jetty.server.handler.AbstractHandler$ff19274a.handle(Unknown Source)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.Server.handle(Server.java:500)
at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:386)
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:562)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:378)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:270)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.lang.UnsupportedOperationException
at org.apache.http.impl.client.InternalHttpClient.getParams(InternalHttpClient.java:211)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:688)
at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:719)
... 46 more
java.lang.UnsupportedOperationException
at org.apache.http.impl.client.InternalHttpClient.getParams(InternalHttpClient.java:211)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:688)
at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:719)
at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:155)
at com.fasterxml.jackson.databind.ser.std.MapSerializer.serializeFields(MapSerializer.java:722)
at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:643)
at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:33)
at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:727)
at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:719)
at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:155)
at com.fasterxml.jackson.databind.ser.std.MapSerializer.serializeFields(MapSerializer.java:722)
at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:643)
at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:33)
at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider._serialize(DefaultSerializerProvider.java:480)
at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:319)
at com.fasterxml.jackson.databind.ObjectMapper._configAndWriteValue(ObjectMapper.java:3905)
at com.fasterxml.jackson.databind.ObjectMapper.writeValueAsBytes(ObjectMapper.java:3243)
at jsonista.core$write_value_as_bytes.invokeStatic(core.clj:229)
at jsonista.core$write_value_as_bytes.invoke(core.clj:221)
at muuntaja.format.json$encoder$reify__10016.encode_to_bytes(json.clj:43)
at muuntaja.core$create_coder$encode__10562.invoke(core.clj:344)
at clojure.core$update.invokeStatic(core.clj:6198)
at clojure.core$update.invoke(core.clj:6188)
at muuntaja.core$create$_handle_response__10629.invoke(core.clj:443)
at muuntaja.core$create$reify__10631.format_response(core.clj:486)
at muuntaja.middleware$wrap_format_response$fn__10702.invoke(middleware.clj:132)
at muuntaja.middleware$wrap_format_negotiate$fn__10695.invoke(middleware.clj:96)
at ring.middleware.keyword_params$wrap_keyword_params$fn__9403.invoke(keyword_params.clj:53)
at ring.middleware.nested_params$wrap_nested_params$fn__9461.invoke(nested_params.clj:89)
at ring.middleware.params$wrap_params$fn__9535.invoke(params.clj:67)
at compojure.api.middleware$wrap_inject_data$fn__11689.invoke(middleware.clj:96)
at compojure.api.routes.Route.invoke(routes.clj:89)
at ring.middleware.basic_authentication$wrap_basic_authentication$fn__16405.invoke(basic_authentication.clj:157)
at clojure.lang.Var.invoke(Var.java:384)
at ring.middleware.reload$wrap_reload$fn__1065.invoke(reload.clj:39)
at ring.adapter.jetty$proxy_handler$fn__480.invoke(jetty.clj:27)
at ring.adapter.jetty.proxy$org.eclipse.jetty.server.handler.AbstractHandler$ff19274a.handle(Unknown Source)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.Server.handle(Server.java:500)
at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:386)
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:562)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:378)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:270)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
at java.base/java.lang.Thread.run(Thread.java:834)
Powered by Jetty:// 9.4.22.v20191022
So, is it possible to say what I am doing wrong? Is there a way to work around it from the client side?
Note: The problem is that this is a code from my work and I would like to not share much of the specifics from it. I am asking because maybe I am missing a simple solution. If this question does not make sense to be in here, let me know and I will delete it.
I appreciate any help!
I have figured out what happened.
There was an internal error in the python application in the amazon server, so when it returned to the clojure app, it did not know how to transform the error into a JSON (which is expected). So the solution was to deal with the error in the python application to return a JSON regarding the problem.
In my local machine worked because I had all non python dependencies to make the application work.
Thanks for the help!
I have generated CSR and downloaded .pk12 file from the CA. While I'm trying to import this getting error as
An IO Exception has occurred:
DerInputStream.getLength(): lengthTag=47, too big.
Error details are as ::
Error Code:
58
Stack Trace:
com.ibm.gsk.ikeyman.error.KeyManagerException: java.io.IOException: DerInputStream.getLength(): lengthTag=47, too big.
at com.ibm.gsk.ikeyman.keystore.KeyStoreProxyCreatorFactory$FileKeyStoreProxy.load(KeyStoreProxyCreatorFactory.java:264)
at com.ibm.gsk.ikeyman.keystore.KeyStoreProxyCreatorFactory$AbstractKeyStoreProxy.getKeyStore(KeyStoreProxyCreatorFactory.java:187)
at com.ibm.gsk.ikeyman.keystore.KeyStoreItemFactory.getPKCS12KeyStoreItem(KeyStoreItemFactory.java:338)
at com.ibm.gsk.ikeyman.keystore.KeyStoreItemFactory.newPKCS12KeyStoreItem(KeyStoreItemFactory.java:332)
at com.ibm.gsk.ikeyman.keystore.KeyStoreItemFactory.loadKeystore(KeyStoreItemFactory.java:141)
at com.ibm.gsk.ikeyman.command.KeyStoreLoaderFactory$BasicKeyStoreLoader.getKeyStoreItem(KeyStoreLoaderFactory.java:48)
at com.ibm.gsk.ikeyman.command.KeyStoreLoaderFactory$CachedKeyStoreLoader.getKeyStoreItem(KeyStoreLoaderFactory.java:83)
at com.ibm.gsk.ikeyman.command.KeyStoreLoaderFactory$BasicKeyStoreLoader.getValidKeyStoreItem(KeyStoreLoaderFactory.java:53)
at com.ibm.gsk.ikeyman.command.Command.getValidKeystoreItem(Command.java:104)
at com.ibm.gsk.ikeyman.command.CommandFactory$LoadDbCommand.run(CommandFactory.java:531)
at com.ibm.gsk.ikeyman.command.Command.invoke(Command.java:198)
at com.ibm.gsk.ikeyman.command.gui.KeymanController$Invoker.invoke(KeymanController.java:261)
at com.ibm.gsk.ikeyman.command.gui.RetrieverFactory$ImportRetriever.getImportParameters(RetrieverFactory.java:509)
at com.ibm.gsk.ikeyman.command.gui.RetrieverFactory$ExportRetriever.getParameters(RetrieverFactory.java:556)
at com.ibm.gsk.ikeyman.command.gui.RetrieverFactory$CompoundRetriever.getParameters(RetrieverFactory.java:149)
at com.ibm.gsk.ikeyman.command.gui.KeymanController.getParameters(KeymanController.java:139)
at com.ibm.gsk.ikeyman.command.gui.KeymanController.getParameters(KeymanController.java:130)
at com.ibm.gsk.ikeyman.command.gui.KeymanController.actionPerformed(KeymanController.java:91)
at com.ibm.gsk.ikeyman.gui.panels.KeymanPanel.fireActionPerformed(KeymanPanel.java:54)
at com.ibm.gsk.ikeyman.gui.panels.KeymanPanel.actionPerformed(KeymanPanel.java:48)
at com.ibm.gsk.ikeyman.gui.panels.KeymanPanel.fireActionPerformed(KeymanPanel.java:54)
at com.ibm.gsk.ikeyman.gui.panels.KeymanPanel.actionPerformed(KeymanPanel.java:48)
at com.ibm.gsk.ikeyman.gui.panels.ContentsPanel.actionPerformed(ContentsPanel.java:334)
at com.ibm.gsk.ikeyman.gui.panels.KeymanPanel.fireActionPerformed(KeymanPanel.java:54)
at com.ibm.gsk.ikeyman.gui.panels.KeymanPanel.actionPerformed(KeymanPanel.java:48)
at javax.swing.AbstractButton.fireActionPerformed(AbstractButton.java:2006)
at javax.swing.AbstractButton$Handler.actionPerformed(AbstractButton.java:2329)
at javax.swing.DefaultButtonModel.fireActionPerformed(DefaultButtonModel.java:398)
at javax.swing.DefaultButtonModel.setPressed(DefaultButtonModel.java:253)
at javax.swing.plaf.basic.BasicButtonListener.mouseReleased(BasicButtonListener.java:247)
at java.awt.AWTEventMulticaster.mouseReleased(AWTEventMulticaster.java:283)
at java.awt.Component.processMouseEvent(Component.java:6052)
at javax.swing.JComponent.processMouseEvent(JComponent.java:3276)
at java.awt.Component.processEvent(Component.java:5817)
at java.awt.Container.processEvent(Container.java:2069)
at java.awt.Component.dispatchEventImpl(Component.java:4424)
at java.awt.Container.dispatchEventImpl(Container.java:2127)
at java.awt.Component.dispatchEvent(Component.java:4254)
at java.awt.LightweightDispatcher.retargetMouseEvent(Container.java:4333)
at java.awt.LightweightDispatcher.processMouseEvent(Container.java:3997)
at java.awt.LightweightDispatcher.dispatchEvent(Container.java:3927)
at java.awt.Container.dispatchEventImpl(Container.java:2113)
at java.awt.Window.dispatchEventImpl(Window.java:2451)
at java.awt.Component.dispatchEvent(Component.java:4254)
at java.awt.EventQueue.dispatchEvent(EventQueue.java:610)
at java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:284)
at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:194)
at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:184)
at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:179)
at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:171)
at java.awt.EventDispatchThread.run(EventDispatchThread.java:132)
Caused by: java.io.IOException: DerInputStream.getLength(): lengthTag=47, too big.
at com.ibm.security.util.DerInputStream.getLength(DerInputStream.java:724)
at com.ibm.security.util.DerInputStream.getLength(DerInputStream.java:698)
at com.ibm.security.util.DerValue.(DerValue.java:253)
at com.ibm.security.util.DerInputStream.getDerValue(DerInputStream.java:499)
at com.ibm.security.pkcsutil.PKCSDerObject.decode(PKCSDerObject.java:258)
at com.ibm.security.pkcs12.PFX.(PFX.java:134)
at com.ibm.crypto.provider.PKCS12KeyStore.engineLoad(Unknown Source)
at java.security.KeyStore.load(Unknown Source)
at com.ibm.gsk.ikeyman.keystore.KeyStoreProxyCreatorFactory$FileKeyStoreProxy.performLoad(KeyStoreProxyCreatorFactory.java:296)
at com.ibm.gsk.ikeyman.keystore.KeyStoreProxyCreatorFactory$FileKeyStoreProxy.load(KeyStoreProxyCreatorFactory.java:245)
... 50 more
What is causing the error ?
I am trying to get a Spark 1.1.0 program written in Scala to work, but I'm having a hard time with it. I have a Hive query that is very simple:
select json, score from data
When I run the following command from spark-shell everything works (I need the MYSQL_CONN in the driver class path as I'm using Hive with a MySQL metadata store)
bin/spark-shell --master $SPARK_URL --driver-class-path $MYSQL_CONN
import org.apache.spark.sql.hive.HiveContext
val sqlContext = new HiveContext(sc)
sqlContext.sql("select json from data").map(t => t.getString(0)).take(10).foreach(println)
I get ten lines of json just like I want. However, when I run this with spark-submit as follows I get a problem
bin/spark-submit --master $SPARK_URL --class spark.Main --driver-class-path $MYSQL_CONN target/spark-testing-1.0-SNAPSHOT.jar
Here is my whole Spark program
package spark
import org.apache.spark.sql.hive.HiveContext
import org.apache.spark.{SparkContext, SparkConf}
object Main {
def main(args: Array[String]) {
val sc = new SparkContext(new SparkConf().setAppName("Gathering Data"))
val sqlContext = new HiveContext(sc)
sqlContext.sql("select json from data").map(t => t.getString(0)).take(10).foreach(println)
}
}
And here is the resultant stack
14/12/01 21:30:04 WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0, match1hd17.dc1): java.lang.ClassNotFoundException: spark.Main$$anonfun$main$1
java.net.URLClassLoader$1.run(URLClassLoader.java:200)
java.security.AccessController.doPrivileged(Native Method)
java.net.URLClassLoader.findClass(URLClassLoader.java:188)
java.lang.ClassLoader.loadClass(ClassLoader.java:307)
java.lang.ClassLoader.loadClass(ClassLoader.java:252)
java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
java.lang.Class.forName0(Native Method)
java.lang.Class.forName(Class.java:247)
org.apache.spark.serializer.JavaDeserializationStream$$anon$1.resolveClass(JavaSerializer.scala:59)
java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1575)
java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1496)
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1732)
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1947)
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1871)
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1753)
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1947)
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1871)
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1753)
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:62)
org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:87)
org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:57)
org.apache.spark.scheduler.Task.run(Task.scala:54)
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:177)
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
java.lang.Thread.run(Thread.java:619)
14/12/01 21:30:10 ERROR TaskSetManager: Task 0 in stage 0.0 failed 4 times; aborting job
Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage 0.0 (TID 3, match1hd12.dc1m): java.lang.ClassNotFoundException: spark.Main$$anonfun$main$1
java.net.URLClassLoader$1.run(URLClassLoader.java:200)
java.security.AccessController.doPrivileged(Native Method)
java.net.URLClassLoader.findClass(URLClassLoader.java:188)
java.lang.ClassLoader.loadClass(ClassLoader.java:307)
java.lang.ClassLoader.loadClass(ClassLoader.java:252)
java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
java.lang.Class.forName0(Native Method)
java.lang.Class.forName(Class.java:247)
org.apache.spark.serializer.JavaDeserializationStream$$anon$1.resolveClass(JavaSerializer.scala:59)
java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1575)
java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1496)
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1732)
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1947)
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1871)
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1753)
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1947)
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1871)
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1753)
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:62)
org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:87)
org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:57)
org.apache.spark.scheduler.Task.run(Task.scala:54)
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:177)
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
java.lang.Thread.run(Thread.java:619)
Driver stacktrace:
at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1185)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1174)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1173)
at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1173)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:688)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:688)
at scala.Option.foreach(Option.scala:236)
at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:688)
at org.apache.spark.scheduler.DAGSchedulerEventProcessActor$$anonfun$receive$2.applyOrElse(DAGScheduler.scala:1391)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:498)
at akka.actor.ActorCell.invoke(ActorCell.scala:456)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:237)
at akka.dispatch.Mailbox.run(Mailbox.scala:219)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
I have spent hours on this already, and I have no idea why this only works with spark-shell. I looked at the stderr output on the individual nodes and they have the same cryptic error message. If anyone can shed some light as to why this only works with spark-shell and not spark-submit that would be awesome.
Thanks
UPDATE:
I've been playing around and the following program works fine.
package spark
import org.apache.spark.sql.hive.HiveContext
import org.apache.spark.{SparkContext, SparkConf}
object Main {
def main(args: Array[String]) {
val sc = new SparkContext(new SparkConf().setAppName("Gathering Data"))
val sqlContext = new HiveContext(sc)
sqlContext.sql("select json from data").take(10).map(t => t.getString(0)).foreach(println)
}
}
Obviously this won't work for a large amount of data, but it shows that the problem appears to be in the ScehmaRDD.map() function.
It seems there is a problem with the spark context initialization.
Please try the below code:
val sparkConf = new SparkConf().setAppName("Gathering Data");
val sc = new SparkContext(sparkConf);
I used the example given at => http://metro.java.net/nonav/1.2/guide/Example_Applications.html#ahies
to try saml authentication over SSL in netbeans 7.3.1 with jdk 7 and glassfish 4.0.
However when I run my client code I'm getting an exception
Exception
javax.xml.ws.WebServiceException)javax.xml.ws.WebServiceException: Failed to access the WSDL at: https://:8181/CalculatorApplication/CalculatorWS?wsdl. It failed with:
java.security.cert.CertificateException: No name matching found.
The exception came in my JSP when I was trying to create the service :
org.me.calculator.client.CalculatorWS_Service service = new org.me.calculator.client.CalculatorWS_Service();
Can anyone who has tried this or has faced similar issue help me with this.
Complete stack trace =>
SEVERE: javax.xml.ws.WebServiceException: Failed to access the WSDL at: https://:8181/CalculatorApplication/CalculatorWS?wsdl. It failed with:
java.security.cert.CertificateException: No name matching found.
at com.sun.xml.ws.wsdl.parser.RuntimeWSDLParser.tryWithMex(RuntimeWSDLParser.java:249)
at com.sun.xml.ws.wsdl.parser.RuntimeWSDLParser.parse(RuntimeWSDLParser.java:230)
at com.sun.xml.ws.wsdl.parser.RuntimeWSDLParser.parse(RuntimeWSDLParser.java:193)
at com.sun.xml.ws.wsdl.parser.RuntimeWSDLParser.parse(RuntimeWSDLParser.java:162)
at com.sun.xml.ws.client.WSServiceDelegate.parseWSDL(WSServiceDelegate.java:359)
at com.sun.xml.ws.client.WSServiceDelegate.(WSServiceDelegate.java:321)
at com.sun.xml.ws.client.WSServiceDelegate.(WSServiceDelegate.java:230)
at com.sun.xml.ws.client.WSServiceDelegate.(WSServiceDelegate.java:212)
at com.sun.xml.ws.client.WSServiceDelegate.(WSServiceDelegate.java:208)
at com.sun.xml.ws.spi.ProviderImpl.createServiceDelegate(ProviderImpl.java:112)
at javax.xml.ws.Service.(Service.java:92)
at org.me.calculator.client.CalculatorWS_Service.(CalculatorWS_Service.java:42)
at org.apache.jsp.index_jsp._jspService(index_jsp.java:58)
at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:111)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:411)
at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:473)
at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:377)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at org.apache.catalina.core.StandardWrapper.service(StandardWrapper.java:1682)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:318)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:160)
at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:734)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:673)
at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:99)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:174)
at org.apache.catalina.connector.CoyoteAdapter.doService(CoyoteAdapter.java:357)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:260)
at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:188)
at org.glassfish.grizzly.http.server.HttpHandler.runService(HttpHandler.java:191)
at org.glassfish.grizzly.http.server.HttpHandler.doHandle(HttpHandler.java:168)
at org.glassfish.grizzly.http.server.HttpServerFilter.handleRead(HttpServerFilter.java:189)
at org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:119)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:288)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:206)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:136)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:114)
at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77)
at org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:838)
at org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:113)
at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.run0(WorkerThreadIOStrategy.java:115)
at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.access$100(WorkerThreadIOStrategy.java:55)
at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy$WorkerThreadRunnable.run(WorkerThreadIOStrategy.java:135)
at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:564)
at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:544)
at java.lang.Thread.run(Thread.java:724)
Caused by: javax.net.ssl.SSLHandshakeException: java.security.cert.CertificateException: No name matching found
at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1884)
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:276)
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:270)
at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1341)
at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:153)
at sun.security.ssl.Handshaker.processLoop(Handshaker.java:868)
at sun.security.ssl.Handshaker.process_record(Handshaker.java:804)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1016)
at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1312)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1339)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1323)
at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:563)
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1300)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:254)
at java.net.URL.openStream(URL.java:1037)
at com.sun.xml.ws.wsdl.parser.RuntimeWSDLParser.createReader(RuntimeWSDLParser.java:983)
at com.sun.xml.ws.wsdl.parser.RuntimeWSDLParser.resolveWSDL(RuntimeWSDLParser.java:384)
at com.sun.xml.ws.wsdl.parser.RuntimeWSDLParser.parse(RuntimeWSDLParser.java:215)
... 44 more
Caused by: java.security.cert.CertificateException: No name matching found
at sun.security.util.HostnameChecker.matchDNS(HostnameChecker.java:208)
at sun.security.util.HostnameChecker.match(HostnameChecker.java:93)
at sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:347)
at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:203)
at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:126)
at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1323)
... 59 more
There is a solution available here[1]. Looks like this is a known issue. I have not tried the solution he is proposing but it looks like the issue is verifying the hostname. He is overwriting the verification method.
public boolean verify(String hostname,
javax.net.ssl.SSLSession sslSession) {
if (hostname.equals("localhost")) {
return true;
}
return false;
}
[1] - http://www.mkyong.com/webservices/jax-ws/java-security-cert-certificateexception-no-name-matching-localhost-found/