I'm trying out Spinnaker and since we use Helm quite regularly I'm trying to deploy it using stable/spinnaker ( https://github.com/helm/charts/tree/master/stable/spinnaker)
Running it with the default configuration works, no complaints at all. The issue comes when I try to replace Minio with S3. To do so I go to the values file and edit the following:
minio:
enabled: false
and
s3:
enabled: true
bucket: "mybucketname"
rootFolder: "front50"
region: "eu-west-1"
accessKey: "myaccesskey"
secretKey: "mysecretkey"
However, front50 is not happy. I'm getting an UnknownHostException.
Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.netflix.spinnaker.front50.model.S3StorageService]: Factory method 's3StorageService' threw exception; nested exception is com.amazonaws.SdkClientException: Unable to execute HTTP request: mybucket-dev.spinnaker-minio
at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:185) ~[spring-beans-5.2.4.RELEASE.jar:5.2.4.RELEASE]
at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:651) ~[spring-beans-5.2.4.RELEASE.jar:5.2.4.RELEASE]
... 134 common frames omitted
Caused by: com.amazonaws.SdkClientException: Unable to execute HTTP request: mybucket-dev.spinnaker-minio
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleRetryableException(AmazonHttpClient.java:1207) ~[aws-java-sdk-core-1.11.802.jar:na]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1153) ~[aws-java-sdk-core-1.11.802.jar:na]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:802) ~[aws-java-sdk-core-1.11.802.jar:na]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:770) ~[aws-java-sdk-core-1.11.802.jar:na]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:744) ~[aws-java-sdk-core-1.11.802.jar:na]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:704) ~[aws-java-sdk-core-1.11.802.jar:na]
at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:686) ~[aws-java-sdk-core-1.11.802.jar:na]
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:550) ~[aws-java-sdk-core-1.11.802.jar:na]
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:530) ~[aws-java-sdk-core-1.11.802.jar:na]
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5062) ~[aws-java-sdk-s3-1.11.802.jar:na]
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5008) ~[aws-java-sdk-s3-1.11.802.jar:na]
at com.amazonaws.services.s3.AmazonS3Client.headBucket(AmazonS3Client.java:1416) ~[aws-java-sdk-s3-1.11.802.jar:na]
at com.netflix.spinnaker.front50.model.S3StorageService.ensureBucketExists(S3StorageService.java:75) ~[front50-s3.jar:na]
at com.netflix.spinnaker.front50.config.S3Config.s3StorageService(S3Config.java:57) ~[front50-s3.jar:na]
at com.netflix.spinnaker.front50.config.S3Config$$EnhancerBySpringCGLIB$$3f80523e.CGLIB$s3StorageService$2(<generated>) ~[front50-s3.jar:na]
at com.netflix.spinnaker.front50.config.S3Config$$EnhancerBySpringCGLIB$$3f80523e$$FastClassBySpringCGLIB$$5007d3c3.invoke(<generated>) ~[front50-s3.jar:na]
at org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:244) ~[spring-core-5.2.4.RELEASE.jar:5.2.4.RELEASE]
at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:331) ~[spring-context-5.2.4.RELEASE.jar:5.2.4.RELEASE]
at com.netflix.spinnaker.front50.config.S3Config$$EnhancerBySpringCGLIB$$3f80523e.s3StorageService(<generated>) ~[front50-s3.jar:na]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:na]
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]
at java.base/java.lang.reflect.Method.invoke(Method.java:566) ~[na:na]
at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:154) ~[spring-beans-5.2.4.RELEASE.jar:5.2.4.RELEASE]
... 135 common frames omitted
Caused by: java.net.UnknownHostException: mybucket-dev.spinnaker-minio
at java.base/java.net.InetAddress$CachedAddresses.get(InetAddress.java:797) ~[na:na]
at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1505) ~[na:na]
at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1364) ~[na:na]
at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1298) ~[na:na]
at com.amazonaws.SystemDefaultDnsResolver.resolve(SystemDefaultDnsResolver.java:27) ~[aws-java-sdk-core-1.11.802.jar:na]
at com.amazonaws.http.DelegatingDnsResolver.resolve(DelegatingDnsResolver.java:38) ~[aws-java-sdk-core-1.11.802.jar:na]
at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:112) ~[httpclient-4.5.12.jar:4.5.12]
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) ~[httpclient-4.5.12.jar:4.5.12]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:na]
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]
at java.base/java.lang.reflect.Method.invoke(Method.java:566) ~[na:na]
at com.amazonaws.http.conn.ClientConnectionManagerFactory$Handler.invoke(ClientConnectionManagerFactory.java:76) ~[aws-java-sdk-core-1.11.802.jar:na]
at com.amazonaws.http.conn.$Proxy125.connect(Unknown Source) ~[na:na]
at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) ~[httpclient-4.5.12.jar:4.5.12]
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[httpclient-4.5.12.jar:4.5.12]
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) ~[httpclient-4.5.12.jar:4.5.12]
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[httpclient-4.5.12.jar:4.5.12]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[httpclient-4.5.12.jar:4.5.12]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56) ~[httpclient-4.5.12.jar:4.5.12]
at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72) ~[aws-java-sdk-core-1.11.802.jar:na]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1330) ~[aws-java-sdk-core-1.11.802.jar:na]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1145) ~[aws-java-sdk-core-1.11.802.jar:na]
... 157 common frames omitted
My bucket is called mybucket-dev I don't understand where .spinnaker-minio is coming from.
I've also checked the install-using-hal pod logs:
hal --daemon-endpoint http://spinnaker-spinnaker-halyard:8064 config storage s3 edit --bucket mybucket-dev --root-folder front50 --region eu-west-1 --access-key-id mykey --secret-access-key
The bucket name is properly set there.
I've tried:
Halyard: 1.32.0 Spinnaker 1.19.4
Halyard: 1.39.0 Spinnaker 1.22.1
With the same result.
The front50 Docker image is automatically set to:
gcr.io/spinnaker-marketplace/front50:0.25.1-20200831095512
The AWS permissions should be OK. Is there anything I'm missing on the config side?
Thanks
Related
I have a mock server running in Postman and I can access it with GET commands etc.
I can also view the GET commands from my web browser, for example
http://5cff3352-bd89-44c2-90b9-23b980d75f30.mock.pstmn.io//users/username/organisation
This all works fine, my problem is when I trying and access the mock server from IntelliJ. The process times out as shown below.
Any ideas please, thanks in advance.
java.net.ConnectException: Connection timed out: connect
at java.base/java.net.PlainSocketImpl.connect0(Native Method)
at java.base/java.net.PlainSocketImpl.socketConnect(PlainSocketImpl.java:101)
at java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:399)
at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:242)
at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:224)
at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:403)
at java.base/java.net.Socket.connect(Socket.java:591)
at org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:121)
at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180)
at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:326)
at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:610)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:445)
at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:835)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
at org.apache.http.client.HttpClient$execute$0.call(Unknown Source)
at io.restassured.internal.RequestSpecificationImpl$RestAssuredHttpBuilder.doRequest(RequestSpecificationImpl.groovy:2054)
at io.restassured.internal.http.HTTPBuilder.post(HTTPBuilder.java:349)
at io.restassured.internal.http.HTTPBuilder$post$2.call(Unknown Source)
at io.restassured.internal.RequestSpecificationImpl.sendRequest(RequestSpecificationImpl.groovy:1180)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:101)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1217)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1041)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:821)
at io.restassured.internal.RequestSpecificationImpl.invokeMethod(RequestSpecificationImpl.groovy)
at org.codehaus.groovy.runtime.callsite.PogoInterceptableSite.call(PogoInterceptableSite.java:45)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:151)
at io.restassured.internal.filter.SendRequestFilter.filter(SendRequestFilter.groovy:30)
at io.restassured.filter.Filter$filter$0.call(Unknown Source)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47)
at io.restassured.filter.Filter$filter.call(Unknown Source)
at io.restassured.internal.filter.FilterContextImpl.next(FilterContextImpl.groovy:72)
at io.restassured.filter.time.TimingFilter.filter(TimingFilter.java:56)
at io.restassured.filter.Filter$filter.call(Unknown Source)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47)
at io.restassured.filter.Filter$filter.call(Unknown Source)
at io.restassured.internal.filter.FilterContextImpl.next(FilterContextImpl.groovy:72)
at io.restassured.filter.log.StatusCodeBasedLoggingFilter.filter(StatusCodeBasedLoggingFilter.java:93)
at io.restassured.filter.log.ResponseLoggingFilter.filter(ResponseLoggingFilter.java:31)
at io.restassured.filter.Filter$filter.call(Unknown Source)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47)
at io.restassured.filter.Filter$filter.call(Unknown Source)
at io.restassured.internal.filter.FilterContextImpl.next(FilterContextImpl.groovy:72)
at io.restassured.filter.log.RequestLoggingFilter.filter(RequestLoggingFilter.java:122)
at io.restassured.filter.Filter$filter.call(Unknown Source)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47)
at io.restassured.filter.Filter$filter$0.call(Unknown Source)
at io.restassured.internal.filter.FilterContextImpl.next(FilterContextImpl.groovy:72)
at io.restassured.filter.FilterContext$next.call(Unknown Source)
at io.restassured.internal.RequestSpecificationImpl.applyPathParamsAndSendRequest(RequestSpecificationImpl.groovy:1654)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:101)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1217)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1041)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:821)
at io.restassured.internal.RequestSpecificationImpl.invokeMethod(RequestSpecificationImpl.groovy)
at org.codehaus.groovy.runtime.callsite.PogoInterceptableSite.call(PogoInterceptableSite.java:45)
at org.codehaus.groovy.runtime.callsite.PogoInterceptableSite.callCurrent(PogoInterceptableSite.java:55)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:184)
at io.restassured.internal.RequestSpecificationImpl.applyPathParamsAndSendRequest(RequestSpecificationImpl.groovy:1660)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:101)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1217)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1041)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:821)
at io.restassured.internal.RequestSpecificationImpl.invokeMethod(RequestSpecificationImpl.groovy)
at org.codehaus.groovy.runtime.callsite.PogoInterceptableSite.call(PogoInterceptableSite.java:45)
at org.codehaus.groovy.runtime.callsite.PogoInterceptableSite.callCurrent(PogoInterceptableSite.java:55)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:184)
at io.restassured.internal.RequestSpecificationImpl.post(RequestSpecificationImpl.groovy:175)
at io.restassured.internal.RequestSpecificationImpl.post(RequestSpecificationImpl.groovy)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:101)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1217)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1041)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:821)
at io.restassured.internal.RequestSpecificationImpl.invokeMethod(RequestSpecificationImpl.groovy)
at org.codehaus.groovy.runtime.callsite.PogoInterceptableSite.call(PogoInterceptableSite.java:45)
at org.codehaus.groovy.runtime.callsite.PogoInterceptableSite.callCurrent(PogoInterceptableSite.java:55)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:168)
at io.restassured.internal.RequestSpecificationImpl.post(RequestSpecificationImpl.groovy:259)
at io.restassured.internal.RequestSpecificationImpl.post(RequestSpecificationImpl.groovy)
at uk.co.kevsbox.automation.core.factory.CreateRequestFactory.postRequest(CreateRequestFactory.java:25)
at uk.co.kevsbox.automation.restassured_userservice.specs.PatchUserDetailsRequestSpec.callApiRequest(PatchUserDetailsRequestSpec.java:48)
at uk.co.kevsbox.automation.restassured_userservice.steps.UpdateUserDetailsSteps.patchUserDetails(UpdateUserDetailsSteps.java:51)
at uk.co.kevsbox.automation.restassured_userservice.steps.UpdateUserDetailsSteps.iMakePatchRequestToUsersEndpoint(UpdateUserDetailsSteps.java:40)
at ✽.When I make patch request to users endpoint with relationships(C:/Dev/RestAssured/restassured_userservice/src/test/resources/features/CreatePatchForOrganisation.feature:9)
Disconnected from the target VM, address: '127.0.0.1:55142', transport: 'socket'
Process finished with exit code 1
It was a proxy setting and a silly typo. All sorted now.
Hopefully it makes sense to make a question like this here:
I have an python3 api which it uses Flask to serve. I also have an clojure app that makes a post request targeting the python3 api. To do so I use clj-http with something like:
(client/post url-link
{:basic-auth ["username"
"password"]
:form-params json
:content-type :json})
Now, when I test in my machine serving the python3 api locally, i.e,
(def url-link "localhost:port/api-name")
it works smoothly. The issue comes because I will serve the python3 api from the amazon server. So my thought was to use the link of the amazon server. But when I do so, clojure complains with
HTTP ERROR 500 clojure.lang.ExceptionInfo: Malformed application/json in :muuntaja/encode {:type :muuntaja/encode, :format "application/json"}
URI: /api/sell-call
STATUS: 500
MESSAGE: clojure.lang.ExceptionInfo: Malformed application/json in :muuntaja/encode {:type :muuntaja/encode, :format "application/json"}
SERVLET: -
CAUSED BY: clojure.lang.ExceptionInfo: Malformed application/json in :muuntaja/encode {:type :muuntaja/encode, :format "application/json"}
CAUSED BY: com.fasterxml.jackson.databind.JsonMappingException: (was java.lang.UnsupportedOperationException) (through reference chain: clojure.lang.PersistentArrayMap[":full_message"]->clojure.lang.ExceptionInfo["data"]->clojure.lang.PersistentHashMap[":http-client"]->org.apache.http.impl.client.InternalHttpClient["params"])
CAUSED BY: java.lang.UnsupportedOperationException
clojure.lang.ExceptionInfo: Malformed application/json in :muuntaja/encode {:type :muuntaja/encode, :format "application/json"}
at muuntaja.core$on_exception.invokeStatic(core.clj:284)
at muuntaja.core$on_exception.invoke(core.clj:267)
at clojure.core$partial$fn__5824.invoke(core.clj:2626)
at muuntaja.core$create_coder$encode__10562.invoke(core.clj:346)
at clojure.core$update.invokeStatic(core.clj:6198)
at clojure.core$update.invoke(core.clj:6188)
at muuntaja.core$create$_handle_response__10629.invoke(core.clj:443)
at muuntaja.core$create$reify__10631.format_response(core.clj:486)
at muuntaja.middleware$wrap_format_response$fn__10702.invoke(middleware.clj:132)
at muuntaja.middleware$wrap_format_negotiate$fn__10695.invoke(middleware.clj:96)
at ring.middleware.keyword_params$wrap_keyword_params$fn__9403.invoke(keyword_params.clj:53)
at ring.middleware.nested_params$wrap_nested_params$fn__9461.invoke(nested_params.clj:89)
at ring.middleware.params$wrap_params$fn__9535.invoke(params.clj:67)
at compojure.api.middleware$wrap_inject_data$fn__11689.invoke(middleware.clj:96)
at compojure.api.routes.Route.invoke(routes.clj:89)
at ring.middleware.basic_authentication$wrap_basic_authentication$fn__16405.invoke(basic_authentication.clj:157)
at clojure.lang.Var.invoke(Var.java:384)
at ring.middleware.reload$wrap_reload$fn__1065.invoke(reload.clj:39)
at ring.adapter.jetty$proxy_handler$fn__480.invoke(jetty.clj:27)
at ring.adapter.jetty.proxy$org.eclipse.jetty.server.handler.AbstractHandler$ff19274a.handle(Unknown Source)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.Server.handle(Server.java:500)
at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:386)
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:562)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:378)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:270)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: com.fasterxml.jackson.databind.JsonMappingException: (was java.lang.UnsupportedOperationException) (through reference chain: clojure.lang.PersistentArrayMap[":full_message"]->clojure.lang.ExceptionInfo["data"]->clojure.lang.PersistentHashMap[":http-client"]->org.apache.http.impl.client.InternalHttpClient["params"])
at com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:394)
at com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:353)
at com.fasterxml.jackson.databind.ser.std.StdSerializer.wrapAndThrow(StdSerializer.java:316)
at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:727)
at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:155)
at com.fasterxml.jackson.databind.ser.std.MapSerializer.serializeFields(MapSerializer.java:722)
at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:643)
at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:33)
at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:727)
at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:719)
at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:155)
at com.fasterxml.jackson.databind.ser.std.MapSerializer.serializeFields(MapSerializer.java:722)
at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:643)
at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:33)
at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider._serialize(DefaultSerializerProvider.java:480)
at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:319)
at com.fasterxml.jackson.databind.ObjectMapper._configAndWriteValue(ObjectMapper.java:3905)
at com.fasterxml.jackson.databind.ObjectMapper.writeValueAsBytes(ObjectMapper.java:3243)
at jsonista.core$write_value_as_bytes.invokeStatic(core.clj:229)
at jsonista.core$write_value_as_bytes.invoke(core.clj:221)
at muuntaja.format.json$encoder$reify__10016.encode_to_bytes(json.clj:43)
at muuntaja.core$create_coder$encode__10562.invoke(core.clj:344)
... 28 more
Caused by: java.lang.UnsupportedOperationException
at org.apache.http.impl.client.InternalHttpClient.getParams(InternalHttpClient.java:211)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:688)
at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:719)
... 46 more
com.fasterxml.jackson.databind.JsonMappingException: (was java.lang.UnsupportedOperationException) (through reference chain: clojure.lang.PersistentArrayMap[":full_message"]->clojure.lang.ExceptionInfo["data"]->clojure.lang.PersistentHashMap[":http-client"]->org.apache.http.impl.client.InternalHttpClient["params"])
at com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:394)
at com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:353)
at com.fasterxml.jackson.databind.ser.std.StdSerializer.wrapAndThrow(StdSerializer.java:316)
at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:727)
at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:155)
at com.fasterxml.jackson.databind.ser.std.MapSerializer.serializeFields(MapSerializer.java:722)
at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:643)
at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:33)
at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:727)
at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:719)
at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:155)
at com.fasterxml.jackson.databind.ser.std.MapSerializer.serializeFields(MapSerializer.java:722)
at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:643)
at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:33)
at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider._serialize(DefaultSerializerProvider.java:480)
at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:319)
at com.fasterxml.jackson.databind.ObjectMapper._configAndWriteValue(ObjectMapper.java:3905)
at com.fasterxml.jackson.databind.ObjectMapper.writeValueAsBytes(ObjectMapper.java:3243)
at jsonista.core$write_value_as_bytes.invokeStatic(core.clj:229)
at jsonista.core$write_value_as_bytes.invoke(core.clj:221)
at muuntaja.format.json$encoder$reify__10016.encode_to_bytes(json.clj:43)
at muuntaja.core$create_coder$encode__10562.invoke(core.clj:344)
at clojure.core$update.invokeStatic(core.clj:6198)
at clojure.core$update.invoke(core.clj:6188)
at muuntaja.core$create$_handle_response__10629.invoke(core.clj:443)
at muuntaja.core$create$reify__10631.format_response(core.clj:486)
at muuntaja.middleware$wrap_format_response$fn__10702.invoke(middleware.clj:132)
at muuntaja.middleware$wrap_format_negotiate$fn__10695.invoke(middleware.clj:96)
at ring.middleware.keyword_params$wrap_keyword_params$fn__9403.invoke(keyword_params.clj:53)
at ring.middleware.nested_params$wrap_nested_params$fn__9461.invoke(nested_params.clj:89)
at ring.middleware.params$wrap_params$fn__9535.invoke(params.clj:67)
at compojure.api.middleware$wrap_inject_data$fn__11689.invoke(middleware.clj:96)
at compojure.api.routes.Route.invoke(routes.clj:89)
at ring.middleware.basic_authentication$wrap_basic_authentication$fn__16405.invoke(basic_authentication.clj:157)
at clojure.lang.Var.invoke(Var.java:384)
at ring.middleware.reload$wrap_reload$fn__1065.invoke(reload.clj:39)
at ring.adapter.jetty$proxy_handler$fn__480.invoke(jetty.clj:27)
at ring.adapter.jetty.proxy$org.eclipse.jetty.server.handler.AbstractHandler$ff19274a.handle(Unknown Source)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.Server.handle(Server.java:500)
at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:386)
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:562)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:378)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:270)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.lang.UnsupportedOperationException
at org.apache.http.impl.client.InternalHttpClient.getParams(InternalHttpClient.java:211)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:688)
at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:719)
... 46 more
java.lang.UnsupportedOperationException
at org.apache.http.impl.client.InternalHttpClient.getParams(InternalHttpClient.java:211)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:688)
at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:719)
at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:155)
at com.fasterxml.jackson.databind.ser.std.MapSerializer.serializeFields(MapSerializer.java:722)
at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:643)
at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:33)
at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:727)
at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:719)
at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:155)
at com.fasterxml.jackson.databind.ser.std.MapSerializer.serializeFields(MapSerializer.java:722)
at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:643)
at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:33)
at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider._serialize(DefaultSerializerProvider.java:480)
at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:319)
at com.fasterxml.jackson.databind.ObjectMapper._configAndWriteValue(ObjectMapper.java:3905)
at com.fasterxml.jackson.databind.ObjectMapper.writeValueAsBytes(ObjectMapper.java:3243)
at jsonista.core$write_value_as_bytes.invokeStatic(core.clj:229)
at jsonista.core$write_value_as_bytes.invoke(core.clj:221)
at muuntaja.format.json$encoder$reify__10016.encode_to_bytes(json.clj:43)
at muuntaja.core$create_coder$encode__10562.invoke(core.clj:344)
at clojure.core$update.invokeStatic(core.clj:6198)
at clojure.core$update.invoke(core.clj:6188)
at muuntaja.core$create$_handle_response__10629.invoke(core.clj:443)
at muuntaja.core$create$reify__10631.format_response(core.clj:486)
at muuntaja.middleware$wrap_format_response$fn__10702.invoke(middleware.clj:132)
at muuntaja.middleware$wrap_format_negotiate$fn__10695.invoke(middleware.clj:96)
at ring.middleware.keyword_params$wrap_keyword_params$fn__9403.invoke(keyword_params.clj:53)
at ring.middleware.nested_params$wrap_nested_params$fn__9461.invoke(nested_params.clj:89)
at ring.middleware.params$wrap_params$fn__9535.invoke(params.clj:67)
at compojure.api.middleware$wrap_inject_data$fn__11689.invoke(middleware.clj:96)
at compojure.api.routes.Route.invoke(routes.clj:89)
at ring.middleware.basic_authentication$wrap_basic_authentication$fn__16405.invoke(basic_authentication.clj:157)
at clojure.lang.Var.invoke(Var.java:384)
at ring.middleware.reload$wrap_reload$fn__1065.invoke(reload.clj:39)
at ring.adapter.jetty$proxy_handler$fn__480.invoke(jetty.clj:27)
at ring.adapter.jetty.proxy$org.eclipse.jetty.server.handler.AbstractHandler$ff19274a.handle(Unknown Source)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.Server.handle(Server.java:500)
at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:386)
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:562)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:378)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:270)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
at java.base/java.lang.Thread.run(Thread.java:834)
Powered by Jetty:// 9.4.22.v20191022
So, is it possible to say what I am doing wrong? Is there a way to work around it from the client side?
Note: The problem is that this is a code from my work and I would like to not share much of the specifics from it. I am asking because maybe I am missing a simple solution. If this question does not make sense to be in here, let me know and I will delete it.
I appreciate any help!
I have figured out what happened.
There was an internal error in the python application in the amazon server, so when it returned to the clojure app, it did not know how to transform the error into a JSON (which is expected). So the solution was to deal with the error in the python application to return a JSON regarding the problem.
In my local machine worked because I had all non python dependencies to make the application work.
Thanks for the help!
When I execute a spark process in oozie I have the following error. The database not found.
2018-09-26 15:27:23,576 INFO [main] org.apache.spark.deploy.yarn.Client:
client token: Token { kind: YARN_CLIENT_TOKEN, service: }
diagnostics: User class threw exception: org.apache.spark.sql.catalyst.analysis.NoSuchDatabaseException: Database 'zdm_datos_externos' not found;
ApplicationMaster host: 10.74.234.6
ApplicationMaster RPC port: 0
queue: default
queue user: administrador
start time: 1537986410475
final status: FAILED
tracking URL: https://BR-PC-CentOS-02:26001/proxy/application_1537467570666_4127/
user: administrador
This is my spark config
String warehouseLocation = new File("spark-warehouse").getAbsolutePath();
SparkSession spark = SparkSession
.builder()
.appName("Java Spark Hive Example")
.master("yarn")
.config("spark.sql.warehouse.dir", warehouseLocation)
.config("spark.driver.maxResultSize", "3g")
.config("spark.debug.maxToStringFields", "10000")
.config("spark.sql.crossJoin.enabled", "true")
.enableHiveSupport()
.getOrCreate();
spark.conf().set("spark.driver.maxResultSize", "3g");
metastore, current connections: 1 2018-09-26 17:31:42,598 WARN [main]
hive.metastore: set_ugi() not successful, Likely cause: new client
talking to old server. Continuing without it.
org.apache.thrift.transport.TTransportException
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
at org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:380)
at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:230)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_set_ugi(ThriftHiveMetastore.java:3748)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.set_ugi(ThriftHiveMetastore.java:3734)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:557)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(HiveMetaStoreClient.java:249)
at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.(SessionHiveMetaStoreClient.java:74)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1533)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.(RetryingMetaStoreClient.java:86)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:132)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:104)
at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:3157)
at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3176)
at org.apache.hadoop.hive.ql.metadata.Hive.getAllFunctions(Hive.java:3409)
at org.apache.hadoop.hive.ql.metadata.Hive.reloadFunctions(Hive.java:178)
at org.apache.hadoop.hive.ql.metadata.Hive.(Hive.java:170)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.yarn.security.HiveCredentialProvider$$anonfun$obtainCredentials$1.apply$mcV$sp(HiveCredentialProvider.scala:91)
at org.apache.spark.deploy.yarn.security.HiveCredentialProvider$$anonfun$obtainCredentials$1.apply(HiveCredentialProvider.scala:90)
at org.apache.spark.deploy.yarn.security.HiveCredentialProvider$$anonfun$obtainCredentials$1.apply(HiveCredentialProvider.scala:90)
at org.apache.spark.deploy.yarn.security.HiveCredentialProvider$$anon$1.run(HiveCredentialProvider.scala:124)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1778)
at org.apache.spark.deploy.yarn.security.HiveCredentialProvider.doAsRealUser(HiveCredentialProvider.scala:123)
at org.apache.spark.deploy.yarn.security.HiveCredentialProvider.obtainCredentials(HiveCredentialProvider.scala:90)
at org.apache.spark.deploy.yarn.security.ConfigurableCredentialManager$$anonfun$obtainCredentials$2.apply(ConfigurableCredentialManager.scala:82)
at org.apache.spark.deploy.yarn.security.ConfigurableCredentialManager$$anonfun$obtainCredentials$2.apply(ConfigurableCredentialManager.scala:80)
at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
at scala.collection.Iterator$class.foreach(Iterator.scala:893)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
at scala.collection.MapLike$DefaultValuesIterable.foreach(MapLike.scala:206)
at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:241)
at scala.collection.AbstractTraversable.flatMap(Traversable.scala:104)
at org.apache.spark.deploy.yarn.security.ConfigurableCredentialManager.obtainCredentials(ConfigurableCredentialManager.scala:80)
at org.apache.spark.deploy.yarn.Client.prepareLocalResources(Client.scala:430)
at org.apache.spark.deploy.yarn.Client.createContainerLaunchContext(Client.scala:915)
at org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:195)
at org.apache.spark.deploy.yarn.Client.run(Client.scala:1205)
at org.apache.spark.deploy.yarn.Client$.main(Client.scala:1261)
at org.apache.spark.deploy.yarn.Client.main(Client.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:761)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:190)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:215)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:129)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
at org.apache.oozie.action.hadoop.SparkMain.runSpark(SparkMain.java:113)
at org.apache.oozie.action.hadoop.SparkMain.run(SparkMain.java:104)
at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:47)
at org.apache.oozie.action.hadoop.SparkMain.main(SparkMain.java:38)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:238)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:459)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:187)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
What version of Spark are you using? Did you enable Hive support on your sparkSession?
sparkBuilder.enableHiveSupport().appName(appName).getOrCreate()
I am running API Tests using REST Assured and Swagger, using the Swagger Request Validator package of Atlassian (https://bitbucket.org/atlassian/swagger-request-validator)
I experince weird problem trying to test some PUT Api:
given()
.pathParam("id", "1")
.headers("Content-Type", "application/x-www-form-urlencoded")
.when()
.queryParams("id", "1",
"package", "Yogev")
.put("/packages/{id}.json")
.then()
.assertThat()
.statusCode(200);
}
Here is the swagger json:
"/packages/{id}.json": {
"put": {
"description": "Updates p....
And this is the error:
com.atlassian.oai.validator.restassured.SwaggerValidationFilter$SwaggerValidationException: Validation failed.
[ERROR] PUT operation not allowed on path '/packages/1.json'.
at com.atlassian.oai.validator.restassured.SwaggerValidationFilter.filter(SwaggerValidationFilter.java:61)
at io.restassured.filter.Filter$filter.call(Unknown Source)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
at io.restassured.filter.Filter$filter$0.call(Unknown Source)
at io.restassured.internal.filter.FilterContextImpl.next(FilterContextImpl.groovy:72)
at io.restassured.filter.FilterContext$next.call(Unknown Source)
at io.restassured.internal.RequestSpecificationImpl.applyPathParamsAndSendRequest(RequestSpecificationImpl.groovy:1731)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:564)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1218)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1027)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:815)
at io.restassured.internal.RequestSpecificationImpl.invokeMethod(RequestSpecificationImpl.groovy)
at org.codehaus.groovy.runtime.callsite.PogoInterceptableSite.call(PogoInterceptableSite.java:48)
at org.codehaus.groovy.runtime.callsite.PogoInterceptableSite.callCurrent(PogoInterceptableSite.java:58)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:182)
at io.restassured.internal.RequestSpecificationImpl.applyPathParamsAndSendRequest(RequestSpecificationImpl.groovy:1737)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:564)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1218)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1027)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:815)
at io.restassured.internal.RequestSpecificationImpl.invokeMethod(RequestSpecificationImpl.groovy)
at org.codehaus.groovy.runtime.callsite.PogoInterceptableSite.call(PogoInterceptableSite.java:48)
at org.codehaus.groovy.runtime.callsite.PogoInterceptableSite.callCurrent(PogoInterceptableSite.java:58)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:52)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:154)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:182)
at io.restassured.internal.RequestSpecificationImpl.put(RequestSpecificationImpl.groovy:178)
at io.restassured.internal.RequestSpecificationImpl.put(RequestSpecificationImpl.groovy)
at ApiTests.PatientsTest.PatientProfileTest.updatePatientProfile(PatientProfileTest.java:27)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:564)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252)
at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141)
at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:564)
at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115)
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)
The main thing:
[ERROR] PUT operation not allowed on path '/packages/1.json'.
As you can see, I do have the PUT operation specified in my json, but still, the validation filter doesn't recognize it.
POST and GET works great for now.
You are using queryParams, so your request is smth like this: PUT /packages/1.json?id=1&package=Yogev with an empty request body. Maybe you should use formParams?
I am trying to use HikariCP with h2 database. Here is my class responsible for sql
#Singleton
#Log
class SqlPersistence {
def config
{
def props = new Properties()
new File("build/resources/main/db.properties").withInputStream {
props.load(it)
}
log.info "Loaded properties: $props"
config = new HikariConfig(props)
}
#Lazy def ds = new HikariDataSource(config)
#Lazy def sql = new Sql(ds)
}
But when I try to run it using gradle task I get this error
Exception in thread "main" java.lang.RuntimeException: java.lang.ClassNotFoundException: 'org.h2.jdbcx.JdbcDataSource'
at com.zaxxer.hikari.util.UtilityElf.createInstance(UtilityElf.java:89)
at com.zaxxer.hikari.pool.PoolBase.initializeDataSource(PoolBase.java:293)
at com.zaxxer.hikari.pool.PoolBase.<init>(PoolBase.java:90)
at com.zaxxer.hikari.pool.HikariPool.<init>(HikariPool.java:101)
at com.zaxxer.hikari.HikariDataSource.<init>(HikariDataSource.java:71)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at org.codehaus.groovy.reflection.CachedConstructor.invoke(CachedConstructor.java:77)
at org.codehaus.groovy.runtime.callsite.ConstructorSite$ConstructorSiteNoUnwrapNoCoerce.callConstructor(ConstructorS
ite.java:102)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallConstructor(CallSiteArray.java:57)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:232)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:244)
at repository.persistence.SqlPersistence.getDs(SqlPersistence.groovy:27)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:324)
at org.codehaus.groovy.runtime.metaclass.MethodMetaProperty$GetBeanMethodMetaProperty.getProperty(MethodMetaProperty
.java:73)
at org.codehaus.groovy.runtime.callsite.GetEffectivePogoPropertySite.getProperty(GtivePogoPropertySite.java:82)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callGroovyObjectGetProperty(AbstractCallSite.java:304)
at repository.persistence.SqlPersistence.getSql(SqlPersistence.groovy:28)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:324)
at org.codehaus.groovy.runtime.metaclass.MethodMetaProperty$GetBeanMethodMetaProperty.getProperty(MethodMetaProperty
.java:73)
at org.codehaus.groovy.runtime.callsite.GetEffectivePogoPropertySite.getProperty(GetEffectivePogoPropertySite.java:8
2)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callGroovyObjectGetProperty(AbstractCallSite.java:304)
at repository.persistence.SqlPersistence.selectAll(SqlPersistence.groovy:42)
at repository.persistence.SqlPersistence$selectAll.call(Unknown Source)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:110)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:122)
at repository.BookRepository.findAll(BookRepository.groovy:11)
at repository.BookRepository$findAll.call(Unknown Source)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:110)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:114)
at service.BookService.findAll(BookService.groovy:13)
at service.BookService$findAll.call(Unknown Source)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45)
org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:110)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:114)
at script.run(script.groovy:7)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:324)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1207)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1016)
at org.codehaus.groovy.runtime.InvokerHelper.invokePogoMethod(InvokerHelper.java:914)
at org.codehaus.groovy.runtime.InvokerHelper.invokeMethod(InvokerHelper.java:897)
at org.codehaus.groovy.runtime.InvokerHelper.runScript(InvokerHelper.java:407)
at org.codehaus.groovy.runtime.InvokerHelper$runScript.call(Unknown Source)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:110)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:130)
at script.main(script.groovy)
Caused by: java.lang.ClassNotFoundException: 'org.h2.jdbcx.JdbcDataSource'
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at com.zaxxer.hikari.util.UtilityElf.createInstance(UtilityElf.java:76)
... 65 more
My build.gradle looks like this
apply plugin: 'groovy'
repositories {
jcenter()
}
dependencies {
compile 'org.codehaus.groovy:groovy-all:2.4.3'
compile 'com.h2database:h2:1.4.191'
compile 'com.zaxxer:HikariCP:2.4.5'
testCompile 'org.spockframework:spock-core:1.0-groovy-2.4'
testCompile 'junit:junit:4.12'
testCompile 'cglib:cglib:3.2.1'
}
task runScript(dependsOn: 'compileJava', type: JavaExec) {
main = 'script'
classpath = sourceSets.main.runtimeClasspath + sourceSets.main.output
}
Why it's unable to find driver?
P.S. there is always some issue with using sql drivers...
It looks like your script is Java, but the snippet you posted is Groovy. So I think this is either an issue of how you call the Groovy part so that the H2 lib is not in Groovy classpath, because if I transform your code to Java and put it in script.java it works fine with your build.gradle. But it could also be a Groovy class loading problem. When I tried to access a database from a Gradle script (which essentially is also a Groovy script I had to do the following to get it working correctly:
// load JDBC drivers to the Groovy class loader to overcome java.sql.DriverManager quirks
Sql.classLoader.addURL new File('driver.jar').toURI().toURL()
Sql.withInstance('my:connection:string', 'user', 'pass', 'my.jdbc.DriverClass') {
it.execute 'select * from foo'
}
Maybe you need something similar.