I have created a databricks account and added my account via root AWS access keys (I know this is not recommended but had tried via cross acount role and this was not working either). These root access keys should have access to my entire AWS account, however when I call.
dbutils.fs.ls('/mnt')
Just a basic listing of the files in my directory, I get this error:
ExecutionError: An error occurred while calling z:com.databricks.backend.daemon.dbutils.FSUtils.ls.
: java.lang.RuntimeException: java.io.IOException: Failed to perform 'getMountFileState(forceRefresh=true)' for mounts after 3 attempts. Please, retry the operation.
Original exception: 'com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden; request: HEAD https://auto-emailer.s3.amazonaws.com oregon-prod/3323621803827190.meta/mounts {} aws-sdk-java/1.11.602 Linux/4.4.0-1104-aws OpenJDK_64-Bit_Server_VM/25.242-b08 java/1.8.0_242 scala/2.12.10 vendor/Private_Build com.amazonaws.services.s3.model.GetObjectMetadataRequest; Request ID: 640F00A5A05F37B3, Extended Request ID: GMSFv0QNFog8PqLBYz8Hre9W5ttMpCLOKNQPPingLc2t1/XFUxn5RdkdBtIYGUbeHYVvIkIM7IY=, Cloud Provider: AWS, Instance ID: i-0acc5354538e76431 (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden; Request ID: 640F00A5A05F37B3; S3 Extended Request ID: GMSFv0QNFog8PqLBYz8Hre9W5ttMpCLOKNQPPingLc2t1/XFUxn5RdkdBtIYGUbeHYVvIkIM7IY=), S3 Extended Request ID: GMSFv0QNFog8PqLBYz8Hre9W5ttMpCLOKNQPPingLc2t1/XFUxn5RdkdBtIYGUbeHYVvIkIM7IY=
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1712)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1367)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1113)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:770)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:744)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:726)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:686)
at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:668)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:532)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:512)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4921)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4867)
at com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1320)
at com.databricks.s3a.aws.EnforcingDatabricksS3Client.$anonfun$getObjectMetadata$1(EnforcingDatabricksS3Client.scala:223)
at scala.util.Try$.apply(Try.scala:213)
at com.databricks.s3a.aws.DatabricksS3Client.retryRequest(DatabricksS3Client.scala:137)
at com.databricks.s3a.aws.DatabricksS3Client.withExponentialBackoff(DatabricksS3Client.scala:111)
at com.databricks.s3a.aws.DatabricksS3Client.withExponentialBackoff$(DatabricksS3Client.scala:109)
at com.databricks.s3a.aws.EnforcingDatabricksS3Client.withExponentialBackoff(EnforcingDatabricksS3Client.scala:28)
at com.databricks.s3a.aws.EnforcingDatabricksS3Client.getObjectMetadata(EnforcingDatabricksS3Client.scala:223)
at com.databricks.s3a.S3AFileSystem.getObjectMetadata(S3AFileSystem.java:1948)
at com.databricks.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:1450)
at com.databricks.s3a.S3AFileSystem.open(S3AFileSystem.java:663)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:769)
at com.databricks.backend.daemon.data.server.DefaultMetadataManager.liftedTree1$1(MetadataManager.scala:156)
at com.databricks.backend.daemon.data.server.DefaultMetadataManager.$anonfun$getMountFileState$1(MetadataManager.scala:151)
at com.databricks.backend.daemon.data.server.DefaultMetadataManager.withRetries(MetadataManager.scala:326)
at com.databricks.backend.daemon.data.server.DefaultMetadataManager.getMountFileState(MetadataManager.scala:150)
at com.databricks.backend.daemon.data.server.DefaultMetadataManager.getMounts(MetadataManager.scala:200)
at com.databricks.backend.daemon.data.filesystem.MountEntryResolver.retrieveFilesystems(MountEntryResolver.scala:76)
at com.databricks.backend.daemon.data.filesystem.MountEntryResolver.<init>(MountEntryResolver.scala:33)
at com.databricks.backend.daemon.data.server.DbfsServerBackend.makeSessionContext(DbfsServerBackend.scala:242)
at com.databricks.backend.daemon.data.server.DbfsServerBackend.$anonfun$receive$2(DbfsServerBackend.scala:280)
at com.databricks.backend.daemon.data.server.session.SessionManager.createSession(SessionManager.scala:193)
at com.databricks.backend.daemon.data.server.DbfsServerBackend$$anonfun$receive$3.applyOrElse(DbfsServerBackend.scala:285)
at com.databricks.backend.daemon.data.server.DbfsServerBackend$$anonfun$receive$3.applyOrElse(DbfsServerBackend.scala:282)
at com.databricks.rpc.ServerBackend.$anonfun$internalReceive$2(ServerBackend.scala:52)
at com.databricks.rpc.ServerBackend$$anonfun$commonReceive$1.applyOrElse(ServerBackend.scala:79)
at com.databricks.rpc.ServerBackend$$anonfun$commonReceive$1.applyOrElse(ServerBackend.scala:79)
at com.databricks.rpc.ServerBackend.$anonfun$internalReceive$1(ServerBackend.scala:48)
at com.databricks.logging.UsageLogging.$anonfun$recordOperation$4(UsageLogging.scala:428)
at com.databricks.logging.UsageLogging.$anonfun$withAttributionContext$1(UsageLogging.scala:238)
at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62)
at com.databricks.logging.UsageLogging.withAttributionContext(UsageLogging.scala:233)
at com.databricks.logging.UsageLogging.withAttributionContext$(UsageLogging.scala:230)
at com.databricks.rpc.ServerBackend.withAttributionContext(ServerBackend.scala:15)
at com.databricks.logging.UsageLogging.withAttributionTags(UsageLogging.scala:275)
at com.databricks.logging.UsageLogging.withAttributionTags$(UsageLogging.scala:268)
at com.databricks.rpc.ServerBackend.withAttributionTags(ServerBackend.scala:15)
at com.databricks.logging.UsageLogging.recordOperation(UsageLogging.scala:409)
at com.databricks.logging.UsageLogging.recordOperation$(UsageLogging.scala:336)
at com.databricks.rpc.ServerBackend.recordOperation(ServerBackend.scala:15)
at com.databricks.rpc.ServerBackend.internalReceive(ServerBackend.scala:47)
at com.databricks.rpc.JettyServer$RequestManager.$anonfun$handleRPC$2(JettyServer.scala:611)
at scala.util.Try$.apply(Try.scala:213)
at com.databricks.rpc.JettyServer$RequestManager.handleRPC(JettyServer.scala:611)
at com.databricks.rpc.JettyServer$RequestManager.handleRequestAndRespond(JettyServer.scala:534)
at com.databricks.rpc.JettyServer$RequestManager.$anonfun$handleHttp$4(JettyServer.scala:321)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at com.databricks.logging.UsageLogging.$anonfun$withAttributionContext$1(UsageLogging.scala:238)
at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62)
at com.databricks.logging.UsageLogging.withAttributionContext(UsageLogging.scala:233)
at com.databricks.logging.UsageLogging.withAttributionContext$(UsageLogging.scala:230)
at com.databricks.rpc.JettyServer$.withAttributionContext(JettyServer.scala:152)
at com.databricks.logging.UsageLogging.withAttributionTags(UsageLogging.scala:275)
at com.databricks.logging.UsageLogging.withAttributionTags$(UsageLogging.scala:268)
at com.databricks.rpc.JettyServer$.withAttributionTags(JettyServer.scala:152)
at com.databricks.rpc.JettyServer$RequestManager.handleHttp(JettyServer.scala:310)
at com.databricks.rpc.JettyServer$RequestManager.doPost(JettyServer.scala:217)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at com.databricks.rpc.HttpServletWithPatch.service(HttpServletWithPatch.scala:33)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:848)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
at org.eclipse.jetty.server.Server.handle(Server.java:539)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
at java.lang.Thread.run(Thread.java:748)
'
at org.apache.hadoop.mapred.JobConf.getWorkingDirectory(JobConf.java:660)
at org.apache.hadoop.mapred.FileInputFormat.setInputPaths(FileInputFormat.java:438)
at org.apache.hadoop.mapred.FileInputFormat.setInputPaths(FileInputFormat.java:411)
at org.apache.spark.api.python.PythonSecurityUtils$.resolvePaths(PythonSecurityUtils.scala:246)
at org.apache.spark.api.python.PythonSecurityUtils$.checkPathStringFileSystemSafety(PythonSecurityUtils.scala:258)
at com.databricks.backend.daemon.dbutils.FSUtils$.com$databricks$backend$daemon$dbutils$FSUtils$$withFsSafetyCheck(DBUtilsCore.scala:80)
at com.databricks.backend.daemon.dbutils.FSUtils$.ls(DBUtilsCore.scala:85)
at com.databricks.backend.daemon.dbutils.FSUtils.ls(DBUtilsCore.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:380)
at py4j.Gateway.invoke(Gateway.java:295)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.GatewayConnection.run(GatewayConnection.java:251)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: Failed to perform 'getMountFileState(forceRefresh=true)' for mounts after 3 attempts. Please, retry the operation.
Original exception: 'com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden; request: HEAD https://auto-emailer.s3.amazonaws.com oregon-prod/3323621803827190.meta/mounts {} aws-sdk-java/1.11.602 Linux/4.4.0-1104-aws OpenJDK_64-Bit_Server_VM/25.242-b08 java/1.8.0_242 scala/2.12.10 vendor/Private_Build com.amazonaws.services.s3.model.GetObjectMetadataRequest; Request ID: 640F00A5A05F37B3, Extended Request ID: GMSFv0QNFog8PqLBYz8Hre9W5ttMpCLOKNQPPingLc2t1/XFUxn5RdkdBtIYGUbeHYVvIkIM7IY=, Cloud Provider: AWS, Instance ID: i-0acc5354538e76431 (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden; Request ID: 640F00A5A05F37B3; S3 Extended Request ID: GMSFv0QNFog8PqLBYz8Hre9W5ttMpCLOKNQPPingLc2t1/XFUxn5RdkdBtIYGUbeHYVvIkIM7IY=), S3 Extended Request ID: GMSFv0QNFog8PqLBYz8Hre9W5ttMpCLOKNQPPingLc2t1/XFUxn5RdkdBtIYGUbeHYVvIkIM7IY=
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1712)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1367)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1113)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:770)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:744)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:726)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:686)
at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:668)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:532)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:512)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4921)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4867)
at com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1320)
at com.databricks.s3a.aws.EnforcingDatabricksS3Client.$anonfun$getObjectMetadata$1(EnforcingDatabricksS3Client.scala:223)
at scala.util.Try$.apply(Try.scala:213)
at com.databricks.s3a.aws.DatabricksS3Client.retryRequest(DatabricksS3Client.scala:137)
at com.databricks.s3a.aws.DatabricksS3Client.withExponentialBackoff(DatabricksS3Client.scala:111)
at com.databricks.s3a.aws.DatabricksS3Client.withExponentialBackoff$(DatabricksS3Client.scala:109)
at com.databricks.s3a.aws.EnforcingDatabricksS3Client.withExponentialBackoff(EnforcingDatabricksS3Client.scala:28)
at com.databricks.s3a.aws.EnforcingDatabricksS3Client.getObjectMetadata(EnforcingDatabricksS3Client.scala:223)
at com.databricks.s3a.S3AFileSystem.getObjectMetadata(S3AFileSystem.java:1948)
at com.databricks.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:1450)
at com.databricks.s3a.S3AFileSystem.open(S3AFileSystem.java:663)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:769)
at com.databricks.backend.daemon.data.server.DefaultMetadataManager.liftedTree1$1(MetadataManager.scala:156)
at com.databricks.backend.daemon.data.server.DefaultMetadataManager.$anonfun$getMountFileState$1(MetadataManager.scala:151)
at com.databricks.backend.daemon.data.server.DefaultMetadataManager.withRetries(MetadataManager.scala:326)
at com.databricks.backend.daemon.data.server.DefaultMetadataManager.getMountFileState(MetadataManager.scala:150)
at com.databricks.backend.daemon.data.server.DefaultMetadataManager.getMounts(MetadataManager.scala:200)
at com.databricks.backend.daemon.data.filesystem.MountEntryResolver.retrieveFilesystems(MountEntryResolver.scala:76)
at com.databricks.backend.daemon.data.filesystem.MountEntryResolver.<init>(MountEntryResolver.scala:33)
at com.databricks.backend.daemon.data.server.DbfsServerBackend.makeSessionContext(DbfsServerBackend.scala:242)
at com.databricks.backend.daemon.data.server.DbfsServerBackend.$anonfun$receive$2(DbfsServerBackend.scala:280)
at com.databricks.backend.daemon.data.server.session.SessionManager.createSession(SessionManager.scala:193)
at com.databricks.backend.daemon.data.server.DbfsServerBackend$$anonfun$receive$3.applyOrElse(DbfsServerBackend.scala:285)
at com.databricks.backend.daemon.data.server.DbfsServerBackend$$anonfun$receive$3.applyOrElse(DbfsServerBackend.scala:282)
at com.databricks.rpc.ServerBackend.$anonfun$internalReceive$2(ServerBackend.scala:52)
at com.databricks.rpc.ServerBackend$$anonfun$commonReceive$1.applyOrElse(ServerBackend.scala:79)
at com.databricks.rpc.ServerBackend$$anonfun$commonReceive$1.applyOrElse(ServerBackend.scala:79)
at com.databricks.rpc.ServerBackend.$anonfun$internalReceive$1(ServerBackend.scala:48)
at com.databricks.logging.UsageLogging.$anonfun$recordOperation$4(UsageLogging.scala:428)
at com.databricks.logging.UsageLogging.$anonfun$withAttributionContext$1(UsageLogging.scala:238)
at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62)
at com.databricks.logging.UsageLogging.withAttributionContext(UsageLogging.scala:233)
at com.databricks.logging.UsageLogging.withAttributionContext$(UsageLogging.scala:230)
at com.databricks.rpc.ServerBackend.withAttributionContext(ServerBackend.scala:15)
at com.databricks.logging.UsageLogging.withAttributionTags(UsageLogging.scala:275)
at com.databricks.logging.UsageLogging.withAttributionTags$(UsageLogging.scala:268)
at com.databricks.rpc.ServerBackend.withAttributionTags(ServerBackend.scala:15)
at com.databricks.logging.UsageLogging.recordOperation(UsageLogging.scala:409)
at com.databricks.logging.UsageLogging.recordOperation$(UsageLogging.scala:336)
at com.databricks.rpc.ServerBackend.recordOperation(ServerBackend.scala:15)
at com.databricks.rpc.ServerBackend.internalReceive(ServerBackend.scala:47)
at com.databricks.rpc.JettyServer$RequestManager.$anonfun$handleRPC$2(JettyServer.scala:611)
at scala.util.Try$.apply(Try.scala:213)
at com.databricks.rpc.JettyServer$RequestManager.handleRPC(JettyServer.scala:611)
at com.databricks.rpc.JettyServer$RequestManager.handleRequestAndRespond(JettyServer.scala:534)
at com.databricks.rpc.JettyServer$RequestManager.$anonfun$handleHttp$4(JettyServer.scala:321)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at com.databricks.logging.UsageLogging.$anonfun$withAttributionContext$1(UsageLogging.scala:238)
at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62)
at com.databricks.logging.UsageLogging.withAttributionContext(UsageLogging.scala:233)
at com.databricks.logging.UsageLogging.withAttributionContext$(UsageLogging.scala:230)
at com.databricks.rpc.JettyServer$.withAttributionContext(JettyServer.scala:152)
at com.databricks.logging.UsageLogging.withAttributionTags(UsageLogging.scala:275)
at com.databricks.logging.UsageLogging.withAttributionTags$(UsageLogging.scala:268)
at com.databricks.rpc.JettyServer$.withAttributionTags(JettyServer.scala:152)
at com.databricks.rpc.JettyServer$RequestManager.handleHttp(JettyServer.scala:310)
at com.databricks.rpc.JettyServer$RequestManager.doPost(JettyServer.scala:217)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at com.databricks.rpc.HttpServletWithPatch.service(HttpServletWithPatch.scala:33)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:848)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
at org.eclipse.jetty.server.Server.handle(Server.java:539)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
at java.lang.Thread.run(Thread.java:748)
'
at com.databricks.backend.daemon.data.client.DbfsClient.send0(DbfsClient.scala:117)
at com.databricks.backend.daemon.data.client.DbfsClient.sendIdempotent(DbfsClient.scala:63)
at com.databricks.backend.daemon.data.client.DatabricksMountsStore.com$databricks$backend$daemon$data$client$DatabricksMountsStore$$refreshMounts(DatabricksMountsStore.scala:92)
at com.databricks.backend.daemon.data.client.DatabricksMountsStore.forceRefreshMounts(DatabricksMountsStore.scala:127)
at com.databricks.backend.daemon.data.client.SharedDatabricksMountsStore$.getOrCreate(DatabricksMountsStore.scala:189)
at com.databricks.backend.daemon.data.client.DBFSV2$$anonfun$initialize$2.apply(DatabricksFileSystemV2.scala:72)
at com.databricks.backend.daemon.data.client.DBFSV2$$anonfun$initialize$2.apply(DatabricksFileSystemV2.scala:64)
at com.databricks.logging.UsageLogging$$anonfun$recordOperation$1.apply(UsageLogging.scala:428)
at com.databricks.logging.UsageLogging$$anonfun$withAttributionContext$1.apply(UsageLogging.scala:238)
at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58)
at com.databricks.logging.UsageLogging$class.withAttributionContext(UsageLogging.scala:233)
at com.databricks.backend.daemon.data.client.DatabricksFileSystemV2.withAttributionContext(DatabricksFileSystemV2.scala:450)
at com.databricks.logging.UsageLogging$class.withAttributionTags(UsageLogging.scala:275)
at com.databricks.backend.daemon.data.client.DatabricksFileSystemV2.withAttributionTags(DatabricksFileSystemV2.scala:450)
at com.databricks.logging.UsageLogging$class.recordOperation(UsageLogging.scala:409)
at com.databricks.backend.daemon.data.client.DatabricksFileSystemV2.recordOperation(DatabricksFileSystemV2.scala:450)
at com.databricks.backend.daemon.data.client.DBFSV2.initialize(DatabricksFileSystemV2.scala:64)
at com.databricks.backend.daemon.data.client.DatabricksFileSystem.initialize(DatabricksFileSystem.scala:226)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2669)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:94)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2703)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2685)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:373)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:172)
at org.apache.hadoop.mapred.JobConf.getWorkingDirectory(JobConf.java:656)
... 18 more
Caused by: java.io.IOException: Failed to perform 'getMountFileState(forceRefresh=true)' for mounts after 3 attempts. Please, retry the operation.
Original exception: 'com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden; request: HEAD https://auto-emailer.s3.amazonaws.com oregon-prod/3323621803827190.meta/mounts {} aws-sdk-java/1.11.602 Linux/4.4.0-1104-aws OpenJDK_64-Bit_Server_VM/25.242-b08 java/1.8.0_242 scala/2.12.10 vendor/Private_Build com.amazonaws.services.s3.model.GetObjectMetadataRequest; Request ID: 640F00A5A05F37B3, Extended Request ID: GMSFv0QNFog8PqLBYz8Hre9W5ttMpCLOKNQPPingLc2t1/XFUxn5RdkdBtIYGUbeHYVvIkIM7IY=, Cloud Provider: AWS, Instance ID: i-0acc5354538e76431 (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden; Request ID: 640F00A5A05F37B3; S3 Extended Request ID: GMSFv0QNFog8PqLBYz8Hre9W5ttMpCLOKNQPPingLc2t1/XFUxn5RdkdBtIYGUbeHYVvIkIM7IY=), S3 Extended Request ID: GMSFv0QNFog8PqLBYz8Hre9W5ttMpCLOKNQPPingLc2t1/XFUxn5RdkdBtIYGUbeHYVvIkIM7IY=
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1712)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1367)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1113)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:770)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:744)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:726)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:686)
at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:668)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:532)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:512)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4921)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4867)
at com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1320)
at com.databricks.s3a.aws.EnforcingDatabricksS3Client.$anonfun$getObjectMetadata$1(EnforcingDatabricksS3Client.scala:223)
at scala.util.Try$.apply(Try.scala:213)
at com.databricks.s3a.aws.DatabricksS3Client.retryRequest(DatabricksS3Client.scala:137)
at com.databricks.s3a.aws.DatabricksS3Client.withExponentialBackoff(DatabricksS3Client.scala:111)
at com.databricks.s3a.aws.DatabricksS3Client.withExponentialBackoff$(DatabricksS3Client.scala:109)
at com.databricks.s3a.aws.EnforcingDatabricksS3Client.withExponentialBackoff(EnforcingDatabricksS3Client.scala:28)
at com.databricks.s3a.aws.EnforcingDatabricksS3Client.getObjectMetadata(EnforcingDatabricksS3Client.scala:223)
at com.databricks.s3a.S3AFileSystem.getObjectMetadata(S3AFileSystem.java:1948)
at com.databricks.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:1450)
at com.databricks.s3a.S3AFileSystem.open(S3AFileSystem.java:663)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:769)
at com.databricks.backend.daemon.data.server.DefaultMetadataManager.liftedTree1$1(MetadataManager.scala:156)
at com.databricks.backend.daemon.data.server.DefaultMetadataManager.$anonfun$getMountFileState$1(MetadataManager.scala:151)
at com.databricks.backend.daemon.data.server.DefaultMetadataManager.withRetries(MetadataManager.scala:326)
at com.databricks.backend.daemon.data.server.DefaultMetadataManager.getMountFileState(MetadataManager.scala:150)
at com.databricks.backend.daemon.data.server.DefaultMetadataManager.getMounts(MetadataManager.scala:200)
at com.databricks.backend.daemon.data.filesystem.MountEntryResolver.retrieveFilesystems(MountEntryResolver.scala:76)
at com.databricks.backend.daemon.data.filesystem.MountEntryResolver.<init>(MountEntryResolver.scala:33)
at com.databricks.backend.daemon.data.server.DbfsServerBackend.makeSessionContext(DbfsServerBackend.scala:242)
at com.databricks.backend.daemon.data.server.DbfsServerBackend.$anonfun$receive$2(DbfsServerBackend.scala:280)
at com.databricks.backend.daemon.data.server.session.SessionManager.createSession(SessionManager.scala:193)
at com.databricks.backend.daemon.data.server.DbfsServerBackend$$anonfun$receive$3.applyOrElse(DbfsServerBackend.scala:285)
at com.databricks.backend.daemon.data.server.DbfsServerBackend$$anonfun$receive$3.applyOrElse(DbfsServerBackend.scala:282)
at com.databricks.rpc.ServerBackend.$anonfun$internalReceive$2(ServerBackend.scala:52)
at com.databricks.rpc.ServerBackend$$anonfun$commonReceive$1.applyOrElse(ServerBackend.scala:79)
at com.databricks.rpc.ServerBackend$$anonfun$commonReceive$1.applyOrElse(ServerBackend.scala:79)
at com.databricks.rpc.ServerBackend.$anonfun$internalReceive$1(ServerBackend.scala:48)
at com.databricks.logging.UsageLogging.$anonfun$recordOperation$4(UsageLogging.scala:428)
at com.databricks.logging.UsageLogging.$anonfun$withAttributionContext$1(UsageLogging.scala:238)
at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62)
at com.databricks.logging.UsageLogging.withAttributionContext(UsageLogging.scala:233)
at com.databricks.logging.UsageLogging.withAttributionContext$(UsageLogging.scala:230)
at com.databricks.rpc.ServerBackend.withAttributionContext(ServerBackend.scala:15)
at com.databricks.logging.UsageLogging.withAttributionTags(UsageLogging.scala:275)
at com.databricks.logging.UsageLogging.withAttributionTags$(UsageLogging.scala:268)
at com.databricks.rpc.ServerBackend.withAttributionTags(ServerBackend.scala:15)
at com.databricks.logging.UsageLogging.recordOperation(UsageLogging.scala:409)
at com.databricks.logging.UsageLogging.recordOperation$(UsageLogging.scala:336)
at com.databricks.rpc.ServerBackend.recordOperation(ServerBackend.scala:15)
at com.databricks.rpc.ServerBackend.internalReceive(ServerBackend.scala:47)
at com.databricks.rpc.JettyServer$RequestManager.$anonfun$handleRPC$2(JettyServer.scala:611)
at scala.util.Try$.apply(Try.scala:213)
at com.databricks.rpc.JettyServer$RequestManager.handleRPC(JettyServer.scala:611)
at com.databricks.rpc.JettyServer$RequestManager.handleRequestAndRespond(JettyServer.scala:534)
I see in the stack trace that original exception: 'com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden; but I don't understand why things are forbidden, this account is set up and the cluster was launched using my root access keys, so Databricks should have access to everything. Any help here would be greatly appreciated!
To be clear, the end goal here is to mount a bucket from s3 to databricks so I can access it, but when I tried mounting I got the same error
There're more than one ways to limit access to a resource in AWS:
limit user/role permissions (not your case as you're using root)
limit operations on resource side, for example allow only VPC traffic (i.e. no external traffic) or allow GET but not HEAD, allow only this specific user to do HEAD, and so on.
Check your S3 bucket access (or provide it here): explicitly allow your user to do list, get, put objects.
Related
I am trying to read a file in an S3 bucket using Spark through Databricks Connect.
This is the code that I am using,
from pyspark import SparkConf
from pyspark.sql import SparkSession
conf = SparkConf()
conf.set('spark.jars.packages', 'org.apache.hadoop:hadoop-aws:3.3.0')
conf.set('spark.hadoop.fs.s3a.access.key', access_key)
conf.set('spark.hadoop.fs.s3a.secret.key', secret_access_key)
spark = SparkSession.builder.config(conf=conf).getOrCreate()
df = spark.read.format("csv").option("header",True).load('s3a://container/path/to/file.csv')
df.show()
This works completely fine when I execute it using a Docker container that I spin up, however, it fails with Databricks Connect with the following error,
pyspark.dbutils.ExecutionError: An error occurred while calling o48.ls.
: com.databricks.service.SparkServiceRemoteException: java.nio.file.AccessDeniedException: getFileStatus on com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden; request: HEAD Forbidden
at shaded.databricks.org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:244)
at shaded.databricks.org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:155)
at shaded.databricks.org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:2870)
at shaded.databricks.org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:2840)
at shaded.databricks.org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:2779)
at shaded.databricks.org.apache.hadoop.fs.s3a.S3AFileSystem.innerListStatus(S3AFileSystem.java:2449)
at shaded.databricks.org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$listStatus$11(S3AFileSystem.java:2428)
at shaded.databricks.org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:118)
at shaded.databricks.org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:112)
at shaded.databricks.org.apache.hadoop.fs.s3a.S3AFileSystem.listStatus(S3AFileSystem.java:2428)
at com.databricks.service.SparkServiceImpl$.$anonfun$fileSystemOperation0$2(SparkServiceImpl.scala:617)
at com.databricks.service.SparkServiceImpl$.withFileSystemExceptionHandler(SparkServiceImpl.scala:647)
at com.databricks.service.SparkServiceImpl$.fileSystemOperation0(SparkServiceImpl.scala:617)
at com.databricks.service.SparkServiceImpl$.$anonfun$fileSystemOperation$1(SparkServiceImpl.scala:184)
at com.databricks.logging.UsageLogging.$anonfun$recordOperation$4(UsageLogging.scala:431)
at com.databricks.logging.UsageLogging.$anonfun$withAttributionContext$1(UsageLogging.scala:239)
at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62)
at com.databricks.logging.UsageLogging.withAttributionContext(UsageLogging.scala:234)
at com.databricks.logging.UsageLogging.withAttributionContext$(UsageLogging.scala:231)
at com.databricks.spark.util.PublicDBLogging.withAttributionContext(DatabricksSparkUsageLogger.scala:19)
at com.databricks.logging.UsageLogging.withAttributionTags(UsageLogging.scala:276)
at com.databricks.logging.UsageLogging.withAttributionTags$(UsageLogging.scala:269)
at com.databricks.spark.util.PublicDBLogging.withAttributionTags(DatabricksSparkUsageLogger.scala:19)
at com.databricks.logging.UsageLogging.recordOperation(UsageLogging.scala:412)
at com.databricks.logging.UsageLogging.recordOperation$(UsageLogging.scala:338)
at com.databricks.spark.util.PublicDBLogging.recordOperation(DatabricksSparkUsageLogger.scala:19)
at com.databricks.spark.util.PublicDBLogging.recordOperation0(DatabricksSparkUsageLogger.scala:56)
at com.databricks.spark.util.DatabricksSparkUsageLogger.recordOperation(DatabricksSparkUsageLogger.scala:131)
at com.databricks.spark.util.UsageLogger.recordOperation(UsageLogger.scala:71)
at com.databricks.spark.util.UsageLogger.recordOperation$(UsageLogger.scala:58)
at com.databricks.spark.util.DatabricksSparkUsageLogger.recordOperation(DatabricksSparkUsageLogger.scala:85)
at com.databricks.spark.util.UsageLogging.recordOperation(UsageLogger.scala:401)
at com.databricks.spark.util.UsageLogging.recordOperation$(UsageLogger.scala:380)
at com.databricks.service.SparkServiceImpl$.recordOperation(SparkServiceImpl.scala:92)
at com.databricks.service.SparkServiceImpl$.fileSystemOperation(SparkServiceImpl.scala:184)
at com.databricks.service.SparkServiceRPCHandler.execute0(SparkServiceRPCHandler.scala:663)
at com.databricks.service.SparkServiceRPCHandler.$anonfun$executeRPC0$1(SparkServiceRPCHandler.scala:451)
at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62)
at com.databricks.service.SparkServiceRPCHandler.executeRPC0(SparkServiceRPCHandler.scala:351)
at com.databricks.service.SparkServiceRPCHandler$$anon$2.call(SparkServiceRPCHandler.scala:302)
at com.databricks.service.SparkServiceRPCHandler$$anon$2.call(SparkServiceRPCHandler.scala:288)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at com.databricks.service.SparkServiceRPCHandler.$anonfun$executeRPC$1(SparkServiceRPCHandler.scala:338)
at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62)
at com.databricks.service.SparkServiceRPCHandler.executeRPC(SparkServiceRPCHandler.scala:315)
at com.databricks.service.SparkServiceRPCServlet.doPost(SparkServiceRPCServer.scala:152)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:873)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:542)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:205)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at org.eclipse.jetty.server.Server.handle(Server.java:505)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:370)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:267)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:698)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:804)
at java.lang.Thread.run(Thread.java:750)
Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden; request: HEAD
Note: I've scrubbed some information related to the path of my file in the above error thread.
Hopefully it makes sense to make a question like this here:
I have an python3 api which it uses Flask to serve. I also have an clojure app that makes a post request targeting the python3 api. To do so I use clj-http with something like:
(client/post url-link
{:basic-auth ["username"
"password"]
:form-params json
:content-type :json})
Now, when I test in my machine serving the python3 api locally, i.e,
(def url-link "localhost:port/api-name")
it works smoothly. The issue comes because I will serve the python3 api from the amazon server. So my thought was to use the link of the amazon server. But when I do so, clojure complains with
HTTP ERROR 500 clojure.lang.ExceptionInfo: Malformed application/json in :muuntaja/encode {:type :muuntaja/encode, :format "application/json"}
URI: /api/sell-call
STATUS: 500
MESSAGE: clojure.lang.ExceptionInfo: Malformed application/json in :muuntaja/encode {:type :muuntaja/encode, :format "application/json"}
SERVLET: -
CAUSED BY: clojure.lang.ExceptionInfo: Malformed application/json in :muuntaja/encode {:type :muuntaja/encode, :format "application/json"}
CAUSED BY: com.fasterxml.jackson.databind.JsonMappingException: (was java.lang.UnsupportedOperationException) (through reference chain: clojure.lang.PersistentArrayMap[":full_message"]->clojure.lang.ExceptionInfo["data"]->clojure.lang.PersistentHashMap[":http-client"]->org.apache.http.impl.client.InternalHttpClient["params"])
CAUSED BY: java.lang.UnsupportedOperationException
clojure.lang.ExceptionInfo: Malformed application/json in :muuntaja/encode {:type :muuntaja/encode, :format "application/json"}
at muuntaja.core$on_exception.invokeStatic(core.clj:284)
at muuntaja.core$on_exception.invoke(core.clj:267)
at clojure.core$partial$fn__5824.invoke(core.clj:2626)
at muuntaja.core$create_coder$encode__10562.invoke(core.clj:346)
at clojure.core$update.invokeStatic(core.clj:6198)
at clojure.core$update.invoke(core.clj:6188)
at muuntaja.core$create$_handle_response__10629.invoke(core.clj:443)
at muuntaja.core$create$reify__10631.format_response(core.clj:486)
at muuntaja.middleware$wrap_format_response$fn__10702.invoke(middleware.clj:132)
at muuntaja.middleware$wrap_format_negotiate$fn__10695.invoke(middleware.clj:96)
at ring.middleware.keyword_params$wrap_keyword_params$fn__9403.invoke(keyword_params.clj:53)
at ring.middleware.nested_params$wrap_nested_params$fn__9461.invoke(nested_params.clj:89)
at ring.middleware.params$wrap_params$fn__9535.invoke(params.clj:67)
at compojure.api.middleware$wrap_inject_data$fn__11689.invoke(middleware.clj:96)
at compojure.api.routes.Route.invoke(routes.clj:89)
at ring.middleware.basic_authentication$wrap_basic_authentication$fn__16405.invoke(basic_authentication.clj:157)
at clojure.lang.Var.invoke(Var.java:384)
at ring.middleware.reload$wrap_reload$fn__1065.invoke(reload.clj:39)
at ring.adapter.jetty$proxy_handler$fn__480.invoke(jetty.clj:27)
at ring.adapter.jetty.proxy$org.eclipse.jetty.server.handler.AbstractHandler$ff19274a.handle(Unknown Source)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.Server.handle(Server.java:500)
at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:386)
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:562)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:378)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:270)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: com.fasterxml.jackson.databind.JsonMappingException: (was java.lang.UnsupportedOperationException) (through reference chain: clojure.lang.PersistentArrayMap[":full_message"]->clojure.lang.ExceptionInfo["data"]->clojure.lang.PersistentHashMap[":http-client"]->org.apache.http.impl.client.InternalHttpClient["params"])
at com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:394)
at com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:353)
at com.fasterxml.jackson.databind.ser.std.StdSerializer.wrapAndThrow(StdSerializer.java:316)
at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:727)
at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:155)
at com.fasterxml.jackson.databind.ser.std.MapSerializer.serializeFields(MapSerializer.java:722)
at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:643)
at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:33)
at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:727)
at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:719)
at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:155)
at com.fasterxml.jackson.databind.ser.std.MapSerializer.serializeFields(MapSerializer.java:722)
at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:643)
at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:33)
at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider._serialize(DefaultSerializerProvider.java:480)
at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:319)
at com.fasterxml.jackson.databind.ObjectMapper._configAndWriteValue(ObjectMapper.java:3905)
at com.fasterxml.jackson.databind.ObjectMapper.writeValueAsBytes(ObjectMapper.java:3243)
at jsonista.core$write_value_as_bytes.invokeStatic(core.clj:229)
at jsonista.core$write_value_as_bytes.invoke(core.clj:221)
at muuntaja.format.json$encoder$reify__10016.encode_to_bytes(json.clj:43)
at muuntaja.core$create_coder$encode__10562.invoke(core.clj:344)
... 28 more
Caused by: java.lang.UnsupportedOperationException
at org.apache.http.impl.client.InternalHttpClient.getParams(InternalHttpClient.java:211)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:688)
at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:719)
... 46 more
com.fasterxml.jackson.databind.JsonMappingException: (was java.lang.UnsupportedOperationException) (through reference chain: clojure.lang.PersistentArrayMap[":full_message"]->clojure.lang.ExceptionInfo["data"]->clojure.lang.PersistentHashMap[":http-client"]->org.apache.http.impl.client.InternalHttpClient["params"])
at com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:394)
at com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:353)
at com.fasterxml.jackson.databind.ser.std.StdSerializer.wrapAndThrow(StdSerializer.java:316)
at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:727)
at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:155)
at com.fasterxml.jackson.databind.ser.std.MapSerializer.serializeFields(MapSerializer.java:722)
at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:643)
at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:33)
at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:727)
at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:719)
at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:155)
at com.fasterxml.jackson.databind.ser.std.MapSerializer.serializeFields(MapSerializer.java:722)
at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:643)
at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:33)
at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider._serialize(DefaultSerializerProvider.java:480)
at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:319)
at com.fasterxml.jackson.databind.ObjectMapper._configAndWriteValue(ObjectMapper.java:3905)
at com.fasterxml.jackson.databind.ObjectMapper.writeValueAsBytes(ObjectMapper.java:3243)
at jsonista.core$write_value_as_bytes.invokeStatic(core.clj:229)
at jsonista.core$write_value_as_bytes.invoke(core.clj:221)
at muuntaja.format.json$encoder$reify__10016.encode_to_bytes(json.clj:43)
at muuntaja.core$create_coder$encode__10562.invoke(core.clj:344)
at clojure.core$update.invokeStatic(core.clj:6198)
at clojure.core$update.invoke(core.clj:6188)
at muuntaja.core$create$_handle_response__10629.invoke(core.clj:443)
at muuntaja.core$create$reify__10631.format_response(core.clj:486)
at muuntaja.middleware$wrap_format_response$fn__10702.invoke(middleware.clj:132)
at muuntaja.middleware$wrap_format_negotiate$fn__10695.invoke(middleware.clj:96)
at ring.middleware.keyword_params$wrap_keyword_params$fn__9403.invoke(keyword_params.clj:53)
at ring.middleware.nested_params$wrap_nested_params$fn__9461.invoke(nested_params.clj:89)
at ring.middleware.params$wrap_params$fn__9535.invoke(params.clj:67)
at compojure.api.middleware$wrap_inject_data$fn__11689.invoke(middleware.clj:96)
at compojure.api.routes.Route.invoke(routes.clj:89)
at ring.middleware.basic_authentication$wrap_basic_authentication$fn__16405.invoke(basic_authentication.clj:157)
at clojure.lang.Var.invoke(Var.java:384)
at ring.middleware.reload$wrap_reload$fn__1065.invoke(reload.clj:39)
at ring.adapter.jetty$proxy_handler$fn__480.invoke(jetty.clj:27)
at ring.adapter.jetty.proxy$org.eclipse.jetty.server.handler.AbstractHandler$ff19274a.handle(Unknown Source)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.Server.handle(Server.java:500)
at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:386)
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:562)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:378)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:270)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.lang.UnsupportedOperationException
at org.apache.http.impl.client.InternalHttpClient.getParams(InternalHttpClient.java:211)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:688)
at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:719)
... 46 more
java.lang.UnsupportedOperationException
at org.apache.http.impl.client.InternalHttpClient.getParams(InternalHttpClient.java:211)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:688)
at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:719)
at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:155)
at com.fasterxml.jackson.databind.ser.std.MapSerializer.serializeFields(MapSerializer.java:722)
at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:643)
at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:33)
at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:727)
at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:719)
at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:155)
at com.fasterxml.jackson.databind.ser.std.MapSerializer.serializeFields(MapSerializer.java:722)
at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:643)
at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:33)
at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider._serialize(DefaultSerializerProvider.java:480)
at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:319)
at com.fasterxml.jackson.databind.ObjectMapper._configAndWriteValue(ObjectMapper.java:3905)
at com.fasterxml.jackson.databind.ObjectMapper.writeValueAsBytes(ObjectMapper.java:3243)
at jsonista.core$write_value_as_bytes.invokeStatic(core.clj:229)
at jsonista.core$write_value_as_bytes.invoke(core.clj:221)
at muuntaja.format.json$encoder$reify__10016.encode_to_bytes(json.clj:43)
at muuntaja.core$create_coder$encode__10562.invoke(core.clj:344)
at clojure.core$update.invokeStatic(core.clj:6198)
at clojure.core$update.invoke(core.clj:6188)
at muuntaja.core$create$_handle_response__10629.invoke(core.clj:443)
at muuntaja.core$create$reify__10631.format_response(core.clj:486)
at muuntaja.middleware$wrap_format_response$fn__10702.invoke(middleware.clj:132)
at muuntaja.middleware$wrap_format_negotiate$fn__10695.invoke(middleware.clj:96)
at ring.middleware.keyword_params$wrap_keyword_params$fn__9403.invoke(keyword_params.clj:53)
at ring.middleware.nested_params$wrap_nested_params$fn__9461.invoke(nested_params.clj:89)
at ring.middleware.params$wrap_params$fn__9535.invoke(params.clj:67)
at compojure.api.middleware$wrap_inject_data$fn__11689.invoke(middleware.clj:96)
at compojure.api.routes.Route.invoke(routes.clj:89)
at ring.middleware.basic_authentication$wrap_basic_authentication$fn__16405.invoke(basic_authentication.clj:157)
at clojure.lang.Var.invoke(Var.java:384)
at ring.middleware.reload$wrap_reload$fn__1065.invoke(reload.clj:39)
at ring.adapter.jetty$proxy_handler$fn__480.invoke(jetty.clj:27)
at ring.adapter.jetty.proxy$org.eclipse.jetty.server.handler.AbstractHandler$ff19274a.handle(Unknown Source)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.Server.handle(Server.java:500)
at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:386)
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:562)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:378)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:270)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
at java.base/java.lang.Thread.run(Thread.java:834)
Powered by Jetty:// 9.4.22.v20191022
So, is it possible to say what I am doing wrong? Is there a way to work around it from the client side?
Note: The problem is that this is a code from my work and I would like to not share much of the specifics from it. I am asking because maybe I am missing a simple solution. If this question does not make sense to be in here, let me know and I will delete it.
I appreciate any help!
I have figured out what happened.
There was an internal error in the python application in the amazon server, so when it returned to the clojure app, it did not know how to transform the error into a JSON (which is expected). So the solution was to deal with the error in the python application to return a JSON regarding the problem.
In my local machine worked because I had all non python dependencies to make the application work.
Thanks for the help!
I have a Spring Boot project using the fabric-java-sdk pointing to the BYFN network that I've stood up locally.
The BYFN network does not have a CA container.
I am trying to connect to the peer and orderer with grpcs since tls is enabled on my local BYFN network. Grpc fails.
However, it appears that it doesn't like the CN or some other field in the cert presented by the peer during the initial TLS handshake. I think it doesn't like that I am pointing to localhost for the grpcs url, but the CN is not 'localhost' in the cert presented by the peer, nor are there any 'localhost' SANs. (This based on my weak understanding of TLS).
Am I missing something built into the Fabric Java SDK to get this to work? Or maybe I need to add some config to the peer and orderer nodes so that their tls certs have 'localhost' SANs?
Code and trace below, thank you.
private static void initHfClient() throws Exception {
hfClient = HFClient.createNewInstance();
CryptoSuite cs = CryptoSuite.Factory.getCryptoSuite();
hfClient.setCryptoSuite(cs);
User context = new LocalUser();
hfClient.setUserContext(context);
Channel channel = hfClient.newChannel("mychannel");
Peer p = hfClient.newPeer("peer0.org1.example.com", "grpcs://localhost:7051");
channel.addPeer(p);
channel.addOrderer(hfClient.newOrderer("orderer.example.com", "grpcs://localhost:7050"));
channel.initialize();
}
Trace:
org.hyperledger.fabric.sdk.exception.ProposalException: org.hyperledger.fabric.sdk.exception.TransactionException: org.hyperledger.fabric.sdk.exception.ProposalException: getConfigBlock for channel mychannel failed with peer peer0.org1.example.com. Status FAILURE, details: Channel Channel{id: 1, name: mychannel} Sending proposal with transaction: 8d926388b9bc4d16e97260f0905076d8cce4abb5499c324d85cc012c82e534a2 to Peer{ id: 2, name: peer0.org1.example.com, channelName: mychannel, url: grpcs://localhost:7051} failed because of: gRPC failure=Status{code=UNAVAILABLE, description=io exception, cause=javax.net.ssl.SSLHandshakeException: General OpenSslEngine problem
at io.netty.handler.ssl.ReferenceCountedOpenSslContext$AbstractCertificateVerifier.verify(ReferenceCountedOpenSslContext.java:665)
at io.netty.internal.tcnative.SSL.readFromSSL(Native Method)
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.readPlaintextData(ReferenceCountedOpenSslEngine.java:565)
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1114)
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1226)
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1269)
at io.netty.handler.ssl.SslHandler$SslEngineType$1.unwrap(SslHandler.java:216)
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1297)
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1199)
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1243)
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:648)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:583)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:500)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:462)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.security.cert.CertificateException: No subject alternative DNS name matching localhost found.
at sun.security.util.HostnameChecker.matchDNS(HostnameChecker.java:214)
at sun.security.util.HostnameChecker.match(HostnameChecker.java:96)
at sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:455)
at sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:436)
at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:252)
at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:136)
at io.netty.handler.ssl.OpenSslTlsv13X509ExtendedTrustManager.checkServerTrusted(OpenSslTlsv13X509ExtendedTrustManager.java:239)
at io.netty.handler.ssl.ReferenceCountedOpenSslClientContext$ExtendedTrustManagerVerifyCallback.verify(ReferenceCountedOpenSslClientContext.java:242)
at io.netty.handler.ssl.ReferenceCountedOpenSslContext$AbstractCertificateVerifier.verify(ReferenceCountedOpenSslContext.java:661)
... 27 more
}
at org.hyperledger.fabric.sdk.Channel.sendProposalToPeers(Channel.java:4059)
at org.hyperledger.fabric.sdk.Channel.sendProposal(Channel.java:4030)
at org.hyperledger.fabric.sdk.Channel.queryByChaincode(Channel.java:3940)
at org.hyperledger.fabric.sdk.Channel.queryByChaincode(Channel.java:3926)
at com.comcast.iotledger.fabricAPI.services.FabricIotledgerService.checkAuth(FabricIotledgerService.java:37)
at com.comcast.iotledger.fabricAPI.controllers.MainController.checkAuth(MainController.java:41)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:189)
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:138)
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:102)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:895)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:800)
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1038)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:942)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1005)
at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:897)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:634)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:882)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:92)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:93)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:200)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:490)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:408)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:791)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1417)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.hyperledger.fabric.sdk.exception.TransactionException: org.hyperledger.fabric.sdk.exception.ProposalException: getConfigBlock for channel mychannel failed with peer peer0.org1.example.com. Status FAILURE, details: Channel Channel{id: 1, name: mychannel} Sending proposal with transaction: 8d926388b9bc4d16e97260f0905076d8cce4abb5499c324d85cc012c82e534a2 to Peer{ id: 2, name: peer0.org1.example.com, channelName: mychannel, url: grpcs://localhost:7051} failed because of: gRPC failure=Status{code=UNAVAILABLE, description=io exception, cause=javax.net.ssl.SSLHandshakeException: General OpenSslEngine problem
at io.netty.handler.ssl.ReferenceCountedOpenSslContext$AbstractCertificateVerifier.verify(ReferenceCountedOpenSslContext.java:665)
at io.netty.internal.tcnative.SSL.readFromSSL(Native Method)
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.readPlaintextData(ReferenceCountedOpenSslEngine.java:565)
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1114)
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1226)
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1269)
at io.netty.handler.ssl.SslHandler$SslEngineType$1.unwrap(SslHandler.java:216)
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1297)
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1199)
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1243)
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:648)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:583)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:500)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:462)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.security.cert.CertificateException: No subject alternative DNS name matching localhost found.
at sun.security.util.HostnameChecker.matchDNS(HostnameChecker.java:214)
at sun.security.util.HostnameChecker.match(HostnameChecker.java:96)
at sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:455)
at sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:436)
at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:252)
at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:136)
at io.netty.handler.ssl.OpenSslTlsv13X509ExtendedTrustManager.checkServerTrusted(OpenSslTlsv13X509ExtendedTrustManager.java:239)
at io.netty.handler.ssl.ReferenceCountedOpenSslClientContext$ExtendedTrustManagerVerifyCallback.verify(ReferenceCountedOpenSslClientContext.java:242)
at io.netty.handler.ssl.ReferenceCountedOpenSslContext$AbstractCertificateVerifier.verify(ReferenceCountedOpenSslContext.java:661)
... 27 more
}
at org.hyperledger.fabric.sdk.Channel.parseConfigBlock(Channel.java:1796)
at org.hyperledger.fabric.sdk.Channel.loadCACertificates(Channel.java:1616)
at org.hyperledger.fabric.sdk.Channel.sendProposalToPeers(Channel.java:4057)
... 59 more
Caused by: org.hyperledger.fabric.sdk.exception.ProposalException: getConfigBlock for channel mychannel failed with peer peer0.org1.example.com. Status FAILURE, details: Channel Channel{id: 1, name: mychannel} Sending proposal with transaction: 8d926388b9bc4d16e97260f0905076d8cce4abb5499c324d85cc012c82e534a2 to Peer{ id: 2, name: peer0.org1.example.com, channelName: mychannel, url: grpcs://localhost:7051} failed because of: gRPC failure=Status{code=UNAVAILABLE, description=io exception, cause=javax.net.ssl.SSLHandshakeException: General OpenSslEngine problem
at io.netty.handler.ssl.ReferenceCountedOpenSslContext$AbstractCertificateVerifier.verify(ReferenceCountedOpenSslContext.java:665)
at io.netty.internal.tcnative.SSL.readFromSSL(Native Method)
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.readPlaintextData(ReferenceCountedOpenSslEngine.java:565)
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1114)
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1226)
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1269)
at io.netty.handler.ssl.SslHandler$SslEngineType$1.unwrap(SslHandler.java:216)
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1297)
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1199)
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1243)
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:648)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:583)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:500)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:462)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.security.cert.CertificateException: No subject alternative DNS name matching localhost found.
at sun.security.util.HostnameChecker.matchDNS(HostnameChecker.java:214)
at sun.security.util.HostnameChecker.match(HostnameChecker.java:96)
at sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:455)
at sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:436)
at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:252)
at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:136)
at io.netty.handler.ssl.OpenSslTlsv13X509ExtendedTrustManager.checkServerTrusted(OpenSslTlsv13X509ExtendedTrustManager.java:239)
at io.netty.handler.ssl.ReferenceCountedOpenSslClientContext$ExtendedTrustManagerVerifyCallback.verify(ReferenceCountedOpenSslClientContext.java:242)
at io.netty.handler.ssl.ReferenceCountedOpenSslContext$AbstractCertificateVerifier.verify(ReferenceCountedOpenSslContext.java:661)
... 27 more
}
at org.hyperledger.fabric.sdk.Channel.getConfigBlock(Channel.java:867)
at org.hyperledger.fabric.sdk.Channel.parseConfigBlock(Channel.java:1779)
... 61 more
I finally figured it out. Here's the high level for getting this to work:
The cryptogen tool from Fabric consumes a yaml file, for me in crypto-config.yaml. You can set SANs in there. So I added "localhost" as a SAN for all peers and orderers.
The JVM needs a custom truststore that accepts the certs that are going to be presented by the peer and orderer during grpcs tls handshake. Use the Java keytool to create a truststore with the .crt files in: crypto-config/peerOrganizations/org1.example.com/peers/peer0.org1.example.com/tls and crypto-config/ordererOrganizations/example.com/orderers/orderer.example.com/tls. Note that YMMV on the paths to the .crt files, and perhaps not all are required to be in the truststore to get it to work. I just threw all of them in there.
Pass the truststore to the JVM with -Djava.net.ssl.trustStore=truststore.jks -Djava.net.ssl.trustStorePassword=changeit, or in Spring like: System.setProperty("javax.net.ssl.trustStore", pathToTrustStore) System.setProperty("javax.net.ssl.trustStorePassword", pw)
Keytool create truststore instructions: https://docs.oracle.com/cd/E19509-01/820-3503/ggfka/index.html
(the aliases used when adding certs to the store don't appear to matter).
-- by request, here's an example of setting SANs in crypto-config.yaml:
# ---------------------------------------------------------------------------
# "PeerOrgs" - Definition of organizations managing peer nodes
# ---------------------------------------------------------------------------
PeerOrgs:
# ---------------------------------------------------------------------------
# Org1
# ---------------------------------------------------------------------------
- Name: Org1
Domain: example.com
EnableNodeOUs: true
# ---------------------------------------------------------------------------
# "Specs"
# ---------------------------------------------------------------------------
# Uncomment this section to enable the explicit definition of hosts in your
# configuration. Most users will want to use Template, below
#
# Specs is an array of Spec entries. Each Spec entry consists of two fields:
# - Hostname: (Required) The desired hostname, sans the domain.
# - CommonName: (Optional) Specifies the template or explicit override for
# the CN. By default, this is the template:
#
# "{{.Hostname}}.{{.Domain}}"
#
# which obtains its values from the Spec.Hostname and
# Org.Domain, respectively.
# ---------------------------------------------------------------------------
Specs:
- Hostname: peer0
CommonName: peer0.example.com # overrides Hostname-based FQDN set above
SANS:
- "localhost"
- Hostname: peer1
CommonName: peer1.example.com
SANS:
- "localhost"
nexus ldap config,verify user mapping success,
verify login error
error message screenshot
and the ldap roles can get from roles->create role->external role mapping->ldap->mapped roles
the log as below:
2018-09-27 20:20:11,402+0800 ERROR [qtp1824645432-178] admin
org.sonatype.nexus.extdirect.internal.ExtDirectServlet - Failed to
invoke action method: ldap_LdapServer.verifyLogin, java-method:
org.sonatype.nexus.ldap.internal.ui.LdapServerComponent.verifyLogin
java.lang.Exception: Failed to connect to LDAP Server: No such user: A
user with username 'root' does not exist at
sun.reflect.GeneratedConstructorAccessor230.newInstance(Unknown
Source) at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at
org.codehaus.groovy.reflection.CachedConstructor.invoke(CachedConstructor.java:83)
at
org.codehaus.groovy.runtime.callsite.ConstructorSite$ConstructorSiteNoUnwrapNoCoerce.callConstructor(ConstructorSite.java:105)
at
org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:247)
at
org.sonatype.nexus.ldap.internal.ui.LdapServerComponent.verifyLogin(LdapServerComponent.groovy:252)
at
com.palominolabs.metrics.guice.ExceptionMeteredInterceptor.invoke(ExceptionMeteredInterceptor.java:49)
at
com.palominolabs.metrics.guice.TimedInterceptor.invoke(TimedInterceptor.java:47)
at
org.sonatype.nexus.validation.internal.ValidationInterceptor.invoke(ValidationInterceptor.java:53)
at
org.apache.shiro.guice.aop.AopAllianceMethodInvocationAdapter.proceed(AopAllianceMethodInvocationAdapter.java:49)
at
org.apache.shiro.authz.aop.AuthorizingAnnotationMethodInterceptor.invoke(AuthorizingAnnotationMethodInterceptor.java:68)
at
org.apache.shiro.guice.aop.AopAllianceMethodInterceptorAdapter.invoke(AopAllianceMethodInterceptorAdapter.java:36)
at
org.apache.shiro.guice.aop.AopAllianceMethodInvocationAdapter.proceed(AopAllianceMethodInvocationAdapter.java:49)
at
org.apache.shiro.authz.aop.AuthorizingAnnotationMethodInterceptor.invoke(AuthorizingAnnotationMethodInterceptor.java:68)
at
org.apache.shiro.guice.aop.AopAllianceMethodInterceptorAdapter.invoke(AopAllianceMethodInterceptorAdapter.java:36)
at sun.reflect.GeneratedMethodAccessor539.invoke(Unknown Source) at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498) at
com.softwarementors.extjs.djn.router.dispatcher.DispatcherBase.invokeJavaMethod(DispatcherBase.java:142)
at
com.softwarementors.extjs.djn.router.dispatcher.DispatcherBase.invokeMethod(DispatcherBase.java:133)
at
org.sonatype.nexus.extdirect.internal.ExtDirectServlet$3.invokeMethod(ExtDirectServlet.java:233)
at
com.softwarementors.extjs.djn.router.dispatcher.DispatcherBase.dispatch(DispatcherBase.java:63)
at
com.softwarementors.extjs.djn.router.processor.standard.StandardRequestProcessorBase.dispatchStandardMethod(StandardRequestProcessorBase.java:73)
at
com.softwarementors.extjs.djn.router.processor.standard.json.JsonRequestProcessor.processIndividualRequest(JsonRequestProcessor.java:502)
at
com.softwarementors.extjs.djn.router.processor.standard.json.JsonRequestProcessor.processIndividualRequestsInThisThread(JsonRequestProcessor.java:150)
at
com.softwarementors.extjs.djn.router.processor.standard.json.JsonRequestProcessor.process(JsonRequestProcessor.java:133)
at
com.softwarementors.extjs.djn.router.RequestRouter.processJsonRequest(RequestRouter.java:83)
at
com.softwarementors.extjs.djn.servlet.DirectJNgineServlet.processRequest(DirectJNgineServlet.java:617)
at
com.softwarementors.extjs.djn.servlet.DirectJNgineServlet.doPost(DirectJNgineServlet.java:580)
at
org.sonatype.nexus.extdirect.internal.ExtDirectServlet.doPost(ExtDirectServlet.java:138)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) at
javax.servlet.http.HttpServlet.service(HttpServlet.java:790) at
com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:286)
at
com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:276)
at
com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:181)
at
com.google.inject.servlet.DynamicServletPipeline.service(DynamicServletPipeline.java:71)
at
com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85)
at
org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:112)
at
com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
at
org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:61)
at
org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)
at
org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)
at
org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
at
org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
at
org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)
at
org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)
at
org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
at
org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
at
org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:449)
at
org.sonatype.nexus.security.SecurityFilter.executeChain(SecurityFilter.java:85)
at
org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365)
at
org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90)
at
org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83)
at
org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:383)
at
org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:362)
at
org.sonatype.nexus.security.SecurityFilter.doFilterInternal(SecurityFilter.java:101)
at
org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
at
com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
at
com.sonatype.nexus.licensing.internal.LicensingRedirectFilter.doFilter(LicensingRedirectFilter.java:108)
at
com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
at
com.codahale.metrics.servlet.AbstractInstrumentedFilter.doFilter(AbstractInstrumentedFilter.java:97)
at
com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
at
org.sonatype.nexus.internal.web.ErrorPageFilter.doFilter(ErrorPageFilter.java:68)
at
com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
at
org.sonatype.nexus.internal.web.EnvironmentFilter.doFilter(EnvironmentFilter.java:101)
at
com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
at
org.sonatype.nexus.internal.web.HeaderPatternFilter.doFilter(HeaderPatternFilter.java:98)
at
com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
at
com.google.inject.servlet.DynamicFilterPipeline.dispatch(DynamicFilterPipeline.java:104)
at
com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:135)
at
org.sonatype.nexus.bootstrap.osgi.DelegatingFilter.doFilter(DelegatingFilter.java:73)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1629)
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:190)
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188)
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1253)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:168)
at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:166)
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1155)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at
com.codahale.metrics.jetty9.InstrumentedHandler.handle(InstrumentedHandler.java:175)
at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at org.eclipse.jetty.server.Server.handle(Server.java:530) at
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:347) at
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:256)
at
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)
at
org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:247)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:140)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
at
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:382)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
at java.lang.Thread.run(Thread.java:748)
my detail config as below:
realms config
connection config
user and group config,and veriry success image
verify all ldap user , can not found the username from ldap
Have you tried to log into the LDAP server with the given proxy user uid=admin,ou=system? Because I think that DN should be uid=admin,ou=system,dc=dinfo,dc=com instead.
When the proxy user fails, all user logins will likely fail.
Try swapping User ID attribute with an attribute that corresponds to your user name (e.g. cn). If that does not help check the logs of your LDAP provider and analyze the search query.
I am loading data from kafka to ignite data streamers in parallel in multiple ignite nodes. But when I kill one of these nodes(for checking fault tolerance), I am getting following error. Some Messages are also getting lost.
class org.apache.ignite.IgniteCheckedException: Failed to finish operation (too many remaps): 32
at org.apache.ignite.internal.processors.datastreamer.DataStreamerImpl$5.apply(DataStreamerImpl.java:863)
at org.apache.ignite.internal.processors.datastreamer.DataStreamerImpl$5.apply(DataStreamerImpl.java:828)
at org.apache.ignite.internal.util.future.GridFutureAdapter$ArrayListener.apply(GridFutureAdapter.java:456)
at org.apache.ignite.internal.util.future.GridFutureAdapter$ArrayListener.apply(GridFutureAdapter.java:439)
at org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:271)
at org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListeners(GridFutureAdapter.java:259)
at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:389)
at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:355)
at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:343)
at org.apache.ignite.internal.processors.datastreamer.DataStreamerImpl$Buffer.submit(DataStreamerImpl.java:1716)
at org.apache.ignite.internal.processors.datastreamer.DataStreamerImpl$Buffer.update(DataStreamerImpl.java:1416)
at org.apache.ignite.internal.processors.datastreamer.DataStreamerImpl.load0(DataStreamerImpl.java:932)
at org.apache.ignite.internal.processors.datastreamer.DataStreamerImpl.addData(DataStreamerImpl.java:576)
at org.apache.ignite.internal.processors.datastreamer.DataStreamerImpl.addData(DataStreamerImpl.java:544)
at org.apache.ignite.stream.StreamAdapter.addMessage(StreamAdapter.java:184)
at org.apache.ignite.stream.kafka.KafkaStreamer.access$100(KafkaStreamer.java:47)
at org.apache.ignite.stream.kafka.KafkaStreamer$1.run(KafkaStreamer.java:156)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: class org.apache.ignite.internal.cluster.ClusterTopologyCheckedException: Failed to send request (node has left): 0a04411f-a797-4526-826a-96c52549b0d0
... 11 more
[21:45:24] (err) Failed to execute compound future reducer: GridCompoundFuture [rdc=null, initFlag=1, lsnrCalls=1, done=false, cancelled=false, err=null, futs=[false, false, true, false, false, true, false, false]]class org.apache.ignite.IgniteCheckedException: DataStreamer request failed [node=f48edaa8-74df-4503-bf9b-1fdc6d642744]
at org.apache.ignite.internal.processors.datastreamer.DataStreamerImpl$Buffer.onResponse(DataStreamerImpl.java:1777)
at org.apache.ignite.internal.processors.datastreamer.DataStreamerImpl$3.onMessage(DataStreamerImpl.java:335)
at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1215)
at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:843)
at org.apache.ignite.internal.managers.communication.GridIoManager.access$2100(GridIoManager.java:108)
at org.apache.ignite.internal.managers.communication.GridIoManager$6.run(GridIoManager.java:783)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)