I've been debugging a JDBC Connection error in hive, similar to what was asked here:
Hive JDBC getConnection does not return.
By turning on log4j properly, i finally got down to seeing this , before the getConnection() hangs. What is thrift waiting for ? If this is related to using the wrong thrift APIs, how can I determine versioning differences between client/server?
I have tried copying all libraries from my hive server onto my client app to test if it is some kind of minor thrift class versioning error, but that didnt solve the problem, my JDBC connection still hangs.
0 [main] DEBUG org.apache.thrift.transport.TSaslTransport - opening transport org.apache.thrift.transport.TSaslClientTransport#219ba640
0 [main] DEBUG org.apache.thrift.transport.TSaslTransport - opening transport org.apache.thrift.transport.TSaslClientTransport#219ba640
3 [main] DEBUG org.apache.thrift.transport.TSaslClientTransport - Sending mechanism name PLAIN and initial response of length 14
3 [main] DEBUG org.apache.thrift.transport.TSaslClientTransport - Sending mechanism name PLAIN and initial response of length 14
5 [main] DEBUG org.apache.thrift.transport.TSaslTransport - CLIENT: Writing message with status START and payload length 5
5 [main] DEBUG org.apache.thrift.transport.TSaslTransport - CLIENT: Writing message with status START and payload length 5
5 [main] DEBUG org.apache.thrift.transport.TSaslTransport - CLIENT: Writing message with status COMPLETE and payload length 14
5 [main] DEBUG org.apache.thrift.transport.TSaslTransport - CLIENT: Writing message with status COMPLETE and payload length 14
5 [main] DEBUG org.apache.thrift.transport.TSaslTransport - CLIENT: Start message handled
5 [main] DEBUG org.apache.thrift.transport.TSaslTransport - CLIENT: Start message handled
5 [main] DEBUG org.apache.thrift.transport.TSaslTransport - CLIENT: Main negotiation loop complete
5 [main] DEBUG org.apache.thrift.transport.TSaslTransport - CLIENT: Main negotiation loop complete
6 [main] DEBUG org.apache.thrift.transport.TSaslTransport - CLIENT: SASL Client receiving last message
6 [main] DEBUG org.apache.thrift.transport.TSaslTransport - CLIENT: SASL Client receiving last message
Related
I am working on automating a API which is hosted in China server and when I send a request it throws me timeout exception ( org.apache.http.conn.HttpHostConnectException).
My feature file:
Background:
* url 'http://myurl'
* configure connectTimeout = 500000
Scenario: Get Client details
Given path '/clients'
And header Authorization = 'sdssSSLwWDSD'
When method get
Then match response.client_id == 'TestId'
Error details:
11:22:30.347 [main] DEBUG org.apache.http.impl.conn.PoolingHttpClientConnectionManager - Connection request: [route: {s}->https://myurl.com][total kept alive: 0; route allocated: 0 of 5; total allocated: 0 of 10]
11:22:30.365 [main] DEBUG org.apache.http.impl.conn.PoolingHttpClientConnectionManager - Connection leased: [id: 0][route: {s}->https://myurl.com][total kept alive: 0; route allocated: 1 of 5; total allocated: 1 of 10]
11:22:30.365 [main] DEBUG org.apache.http.impl.execchain.MainClientExec - Opening connection {s}->https://myurl.com
11:22:30.384 [main] DEBUG org.apache.http.impl.conn.DefaultHttpClientConnectionOperator - Connecting to myurl.com/54.223.191.33:443
11:22:30.384 [main] DEBUG org.apache.http.conn.ssl.SSLConnectionSocketFactory - Connecting socket to myurl.com/54.223.191.33:443 with timeout 500000
11:22:51.407 [main] DEBUG org.apache.http.impl.conn.DefaultHttpClientConnectionOperator - Connect to myurl.com/54.223.191.33:443 timed out. Connection will be retried using another IP address
11:22:51.407 [main] DEBUG org.apache.http.impl.conn.DefaultHttpClientConnectionOperator - Connecting to myurl.com/52.80.167.86:443
11:22:51.408 [main] DEBUG org.apache.http.conn.ssl.SSLConnectionSocketFactory - Connecting socket to myurl.com/52.80.167.86:443 with timeout 500000
11:23:12.438 [main] DEBUG org.apache.http.impl.conn.DefaultManagedHttpClientConnection - http-outgoing-0: Shutdown connection
11:23:12.439 [main] DEBUG org.apache.http.impl.execchain.MainClientExec - Connection discarded
11:23:12.440 [main] DEBUG org.apache.http.impl.conn.PoolingHttpClientConnectionManager - Connection released: [id: 0][route: {s}->https://myurl.com:443][total kept alive: 0; route allocated: 0 of 5; total allocated: 0 of 10]
11:23:12.440 [main] ERROR com.intuit.karate - org.apache.http.conn.HttpHostConnectException: Connect to myurl.com [myurl.com/54.223.191.33, myurl.com/52.80.167.86] failed: Connection timed out: connect, http call failed after 42094 milliseconds for URL: https://myurl.com/tnc/v1/tnc/all
11:23:12.441 [main] ERROR com.intuit.karate - http request failed:
org.apache.http.conn.HttpHostConnectException: Connect to myurl.com:443 [myurl.com/54.223.191.33, myurl.com/52.80.167.86] failed: Connection timed out: connect
11:23:12.601 [Finalizer] DEBUG org.apache.http.impl.conn.PoolingHttpClientConnectionManager - Connection manager is shutting down
11:23:12.601 [Finalizer] DEBUG org.apache.http.impl.conn.PoolingHttpClientConnectionManager - Connection manager shut down
HTML report: (paste into browser to view) | Karate version: 0.9.4
What I have tried yet:
I tried to configure timeout to max as below:
* configure connectTimeout = 500000
It didn't work.
However, the same request when tried from Postman, works fine and I get response within 2000 ms.
Not sure where I am wrong.
Most likely you have a corporate HTTP proxy.
Refer: https://github.com/intuit/karate#configure
* configure proxy = 'http://my.proxy.host:8080'
So I've been using karate for a while now, and there has been an issue we were facing since over the last year: org.apache.http.conn.ConnectTimeoutException
Other threads about that mentioned connectionTimeout exception were solvable by specifying proxy, but taht did not help us.
After tons of investigation, it turned out that our Azure SNAT was exhausted, meaning Karate was opening way too many connections.
To verify this I enabled log debugging and used this feature
Background:
* url "https://www.karatelabs.io/"
Scenario:
* method GET
* method GET
the logs then had following lines
13:10:17.868 [main] DEBUG com.intuit.karate - request:
1 > GET https://www.karatelabs.io/
1 > Host: www.karatelabs.io
1 > Connection: Keep-Alive
1 > User-Agent: Apache-HttpClient/4.5.13 (Java/17.0.4.1)
1 > Accept-Encoding: gzip,deflate
13:10:17.868 [main] DEBUG o.a.h.i.c.PoolingHttpClientConnectionManager - Connection request: [route: {s}->https://www.karatelabs.io:443][total available: 0; route allocated: 0 of 5; total allocated: 0 of 10]
13:10:17.874 [main] DEBUG o.a.h.i.c.PoolingHttpClientConnectionManager - Connection leased: [id: 0][route: {s}->https://www.karatelabs.io:443][total available: 0; route allocated: 1 of 5; total allocated: 1 of 10]
13:10:17.875 [main] DEBUG o.a.h.impl.execchain.MainClientExec - Opening connection {s}->https://www.karatelabs.io:443
13:10:17.883 [main] DEBUG o.a.h.i.c.DefaultHttpClientConnectionOperator - Connecting to www.karatelabs.io/34.149.87.45:443
13:10:17.883 [main] DEBUG o.a.h.c.s.SSLConnectionSocketFactory - Connecting socket to www.karatelabs.io/34.149.87.45:443 with timeout 30000
13:10:17.924 [main] DEBUG o.a.h.c.s.SSLConnectionSocketFactory - Enabled protocols: [TLSv1.3, TLSv1.2]
13:10:17.924 [main] DEBUG o.a.h.c.s.SSLConnectionSocketFactory - Enabled cipher suites:[...]
13:10:17.924 [main] DEBUG o.a.h.c.s.SSLConnectionSocketFactory - Starting handshake
13:10:18.012 [main] DEBUG o.a.h.c.s.SSLConnectionSocketFactory - Secure session established
13:10:18.012 [main] DEBUG o.a.h.c.s.SSLConnectionSocketFactory - negotiated protocol: TLSv1.3
13:10:18.012 [main] DEBUG o.a.h.c.s.SSLConnectionSocketFactory - negotiated cipher suite: TLS_AES_256_GCM_SHA384
13:10:18.012 [main] DEBUG o.a.h.c.s.SSLConnectionSocketFactory - peer principal: CN=karatelabs.io
13:10:18.012 [main] DEBUG o.a.h.c.s.SSLConnectionSocketFactory - peer alternative names: [karatelabs.io, www.karatelabs.io]
13:10:18.012 [main] DEBUG o.a.h.c.s.SSLConnectionSocketFactory - issuer principal: CN=Sectigo RSA Domain Validation Secure Server CA, O=Sectigo Limited, L=Salford, ST=Greater Manchester, C=GB
13:10:18.014 [main] DEBUG o.a.h.i.c.DefaultHttpClientConnectionOperator - Connection established localIp<->serverIp
13:10:18.015 [main] DEBUG o.a.h.i.c.DefaultManagedHttpClientConnection - http-outgoing-0: set socket timeout to 120000
13:10:18.015 [main] DEBUG o.a.h.impl.execchain.MainClientExec - Executing request GET / HTTP/1.1
...
13:10:18.066 [main] DEBUG o.a.h.impl.execchain.MainClientExec - Connection can be kept alive indefinitely
...
...
13:10:18.196 [main] DEBUG com.intuit.karate - request:
2 > GET https://www.karatelabs.io/
13:10:18.196 [main] DEBUG o.a.h.i.c.PoolingHttpClientConnectionManager - Connection request: [route: {s}->https://www.karatelabs.io:443][total available: 0; route allocated: 0 of 5; total allocated: 0 of 10]
13:10:18.196 [main] DEBUG o.a.h.i.c.PoolingHttpClientConnectionManager - Connection leased: [id: 1][route: {s}->https://www.karatelabs.io:443][total available: 0; route allocated: 1 of 5; total allocated: 1 of 10]
13:10:18.196 [main] DEBUG o.a.h.impl.execchain.MainClientExec - Opening connection {s}->https://www.karatelabs.io:443
13:10:18.196 [main] DEBUG o.a.h.i.c.DefaultHttpClientConnectionOperator - Connecting to www.karatelabs.io/34.149.87.45:443
13:10:18.196 [main] DEBUG o.a.h.c.s.SSLConnectionSocketFactory - Connecting socket to www.karatelabs.io/34.149.87.45:443 with timeout 30000
13:10:18.206 [main] DEBUG o.a.h.c.s.SSLConnectionSocketFactory - Enabled protocols: [TLSv1.3, TLSv1.2]
13:10:18.206 [main] DEBUG o.a.h.c.s.SSLConnectionSocketFactory - Enabled cipher suites:[...]
13:10:18.206 [main] DEBUG o.a.h.c.s.SSLConnectionSocketFactory - Starting handshake
13:10:18.236 [main] DEBUG o.a.h.c.s.SSLConnectionSocketFactory - Secure session established
13:10:18.236 [main] DEBUG o.a.h.c.s.SSLConnectionSocketFactory - negotiated protocol: TLSv1.3
13:10:18.236 [main] DEBUG o.a.h.c.s.SSLConnectionSocketFactory - negotiated cipher suite: TLS_AES_256_GCM_SHA384
13:10:18.236 [main] DEBUG o.a.h.c.s.SSLConnectionSocketFactory - peer principal: CN=karatelabs.io
13:10:18.236 [main] DEBUG o.a.h.c.s.SSLConnectionSocketFactory - peer alternative names: [karatelabs.io, www.karatelabs.io]
13:10:18.236 [main] DEBUG o.a.h.c.s.SSLConnectionSocketFactory - issuer principal: CN=Sectigo RSA Domain Validation Secure Server CA, O=Sectigo Limited, L=Salford, ST=Greater Manchester, C=GB
13:10:18.236 [main] DEBUG o.a.h.i.c.DefaultHttpClientConnectionOperator - Connection established localIp<->serverIp
13:10:18.236 [main] DEBUG o.a.h.i.c.DefaultManagedHttpClientConnection - http-outgoing-1: set socket timeout to 120000
...
13:10:18.279 [main] DEBUG o.a.h.impl.execchain.MainClientExec - Connection can be kept alive indefinitely
...
...
13:10:18.609 [Finalizer] DEBUG o.a.h.i.c.PoolingHttpClientConnectionManager - Connection manager is shutting down
13:10:18.610 [Finalizer] DEBUG o.a.h.i.c.DefaultManagedHttpClientConnection - http-outgoing-1: Shutdown connection
13:10:18.611 [Finalizer] DEBUG o.a.h.i.c.PoolingHttpClientConnectionManager - Connection manager shut down
13:10:18.612 [Finalizer] DEBUG o.a.h.i.c.PoolingHttpClientConnectionManager - Connection manager is shutting down
13:10:18.612 [Finalizer] DEBUG o.a.h.i.c.DefaultManagedHttpClientConnection - http-outgoing-2: Shutdown connection
13:10:18.612 [Finalizer] DEBUG o.a.h.i.c.PoolingHttpClientConnectionManager - Connection manager shut down
13:10:18.612 [Finalizer] DEBUG o.a.h.i.c.PoolingHttpClientConnectionManager - Connection manager is shutting down
"Connecting to socket" and "handshake" indicate that karate is establishing a new connection instead of using an already opened one, even though I am sending a request to the same host.
On the other hand, on longer scenarios, I was seeing "http-outgoing-x: Shutdown connection" after about ~1s from opening it, in the middle of the run, despite having "karate.configure('readTimeout', 120000)" specified.
I don't think that was intentional, especially after seeing the "keep-alive" header and the "Connection can be kept alive indefinitely" in the log"
That being said, is there any way to force karate to use the same connection instead of establishing a new one each request?
As far as we know, we use the Apache HTTP Client API the right way.
But you never know. The best thing is for you to dive into the code and see what we could be missing. Or you could provide a way to replicate following these instructions: https://github.com/karatelabs/karate/wiki/How-to-Submit-an-Issue
I have upgraded to version 1.0.0 and I am encountering an error that was working fine when using 0.9.6.
Below is a provided Karate sample code which I modified a bit to show how I get the error.
Background:
* url 'https://jsonplaceholder.typicode.com'
* def id = read("id-list.json")
Scenario Outline: [id: <id>] get all users and then get the first user by id
Given path 'users', <id>
When method get
Then status 200
* print id
Examples:
| id |
id-list.json
[{"id":1},{"id":2}]
Error:
Testing started at 11:12 ...
command: --plugin org.jetbrains.plugins.cucumber.java.run.CucumberJvmSMFormatter --monochrome --name ^\[id\: .*\] get all users and then get the first user by id$ --glue com.intuit.karate C:/Users/danna/IdeaProjects/YXC/src/test/java/examples/users/users.feature
11:12:19.580 [main] DEBUG com.intuit.karate.Suite - [config] classpath:karate-config.js
11:12:19.585 [main] INFO com.intuit.karate.Suite - backed up existing 'target\karate-reports' dir to: target\karate-reports_1616551939583
11:12:19.606 [main] INFO c.intuit.karate.core.FeatureRuntime - found scenario at line: 18 - ^\[id\: .*\] get all users and then get the first user by id$
11:12:20.265 [main] INFO com.intuit.karate - karate.env system property was: null
11:12:20.333 [main] INFO c.intuit.karate.core.FeatureRuntime - found scenario at line: 18 - ^\[id\: .*\] get all users and then get the first user by id$
11:12:20.338 [main] INFO com.intuit.karate - karate.env system property was: null
11:12:20.595 [main] ERROR com.intuit.karate - karate.org.apache.http.ProtocolException: Target host is not specified, http call failed after 250 milliseconds for url: /users/1
11:12:20.596 [main] ERROR com.intuit.karate - src/test/java/examples/users/users.feature:12
When method get
http call failed after 250 milliseconds for url: /users/1
src/test/java/examples/users/users.feature:12
http call failed after 250 milliseconds for url: /users/1
src/test/java/examples/users/users.feature:12
11:12:20.596 [main] INFO c.intuit.karate.core.FeatureRuntime - found scenario at line: 18 - ^\[id\: .*\] get all users and then get the first user by id$
11:12:20.600 [main] INFO com.intuit.karate - karate.env system property was: null
11:12:20.601 [main] ERROR com.intuit.karate - karate.org.apache.http.ProtocolException: Target host is not specified, http call failed after 0 milliseconds for url: /users/2
11:12:20.601 [main] ERROR com.intuit.karate - src/test/java/examples/users/users.feature:12
When method get
http call failed after 0 milliseconds for url: /users/2
src/test/java/examples/users/users.feature:12
http call failed after 0 milliseconds for url: /users/2
src/test/java/examples/users/users.feature:12
I think the error was related to this issue but I'm not quite sure .
Any help would be appreciated.
Thanks for reporting this. This sounds like it is related to the same bug, so we have noted it: https://github.com/intuit/karate/issues/1517#issuecomment-805439326
For now you can work around by moving the url line into the first line of the Scenario.
I try to run a post request that does not have an output body. The output is void. I get the 200 answer but then it fails. I have already spoken to my colleagues at the company and they have indicated that it is not possible to execute these types of requests in karate. Is it a bug on the part of karate's developers? Here's the error:
[ForkJoinPool-1-worker-1] DEBUG org.apache.http.impl.conn.PoolingHttpClientConnectionManager - Connection [id: 3][route: {s}->REMOVED:443] can be kept alive indefinitely
[ForkJoinPool-1-worker-1] DEBUG org.apache.http.impl.conn.DefaultManagedHttpClientConnection - http-outgoing-3: set socket timeout to 0
[ForkJoinPool-1-worker-1] DEBUG org.apache.http.impl.conn.PoolingHttpClientConnectionManager - Connection released: [id: 3][route: {s}->REMOVED:443][total available: 1; route allocated: 1 of 5; total allocated: 1 of 10]
[ForkJoinPool-1-worker-1] ERROR com.intuit.karate - java.lang.RuntimeException: java.io.EOFException, http call failed after 1601 milliseconds for URL:REMOVED
[ForkJoinPool-1-worker-1] ERROR com.intuit.karate - http request failed:
java.lang.RuntimeException: java.io.EOFException
There's a new version just released: https://github.com/intuit/karate/releases/tag/v1.0.0 - so try that first.
Do update here if that works. That said - the log you provided does not help at all. The best thing you can do is follow this process, so that the developers get a decent chance to understand what you mean by "does not have an output body" and "void": https://github.com/intuit/karate/wiki/How-to-Submit-an-Issue
We have a Kerberised cluster and I'm trying to run a Java action in Oozie where I make a JDBC connection to Hive. This JDBC connections works fine on the Sandbox without Kerberos.
The connection string is as simple as the following, where I'm providing username and password in it:
Connection con = DriverManager.getConnection("jdbc:hive2://W12345:10000/control;principal=hive/W12345.companynet.net#COMPANYNET.NET","user123","passw123");
The Oozie action (strangely) completes succesfully, and the Java action log does not present any error:
1742 [main] INFO org.apache.hive.jdbc.Utils - Supplied authorities: W12345:10000
1742 [main] INFO org.apache.hive.jdbc.Utils - Resolved authority: W12345:10000
1766 [main] INFO org.apache.hive.jdbc.HiveConnection - Will try to open client transport with JDBC Uri: jdbc:hive2://W12345:10000/control;principal=hive/W12345.companynet.net#COMPANYNET.NET
<<< Invocation of Main class completed <<<
Oozie Launcher ends
1785 [main] INFO org.apache.hadoop.mapred.Task - Task:attempt_1464245290012_0129_m_000000_0 is done. And is in the process of committing
1847 [main] INFO org.apache.hadoop.mapred.Task - Task attempt_1464245290012_0129_m_000000_0 is allowed to commit now
1854 [main] INFO org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter - Saved output of task 'attempt_1464245290012_0129_m_000000_0' to hdfs://danskehadoop/user/user123/oozie-oozi/0000013-160527101253015-oozie-oozi-W/JavaAction--java/output/_temporary/1/task_1464245290012_0129_m_000000
1909 [main] INFO org.apache.hadoop.mapred.Task - Task 'attempt_1464245290012_0129_m_000000_0' done.
But in reality the Java main does not complete correctly the execution (and does not execute the needed queries) because the JDBC connection fails with an exception that I can see only in the Hive log:
ERROR [HiveServer2-Handler-Pool: Thread-78363]: server.TThreadPoolServer (TThreadPoolServer.java:run(296)) - Error occurred during processing of message.
java.lang.RuntimeException: org.apache.thrift.transport.TSaslTransportException: No data or no sasl data in the stream
at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219)
at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:739)
at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:736)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:360)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1637)
at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory.getTransport(HadoopThriftAuthBridge.java:736)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:268)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.thrift.transport.TSaslTransportException: No data or no sasl data in the stream
at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:328)
at org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41)
at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216)
... 10 more
I'm actually connected to the cluster, and already done further kinit on my username.
Does anybody know what could the cause of this exception be?
Thanks in advance for the help!
Antonio
This happened to me on MapR hadoop distribution platform.
In my case it was Keepalived checking Hive port every 5 seconds and producing such error. I simply used "nc" command to check if Hive port is in use and did not use any authentication method. Later I switched to "maprcli" command which uses SASL authentication and the error was gone.