Karate - Cookie rejected when it contains special characters - karate

We are struggling to pass the response cookie to the subsequent request as we are getting Cookie Rejected error. We are also not able to print the cookie response:
Here are more details: giving req and response. See the Print at the very end prints nothing and at the start of response there are warning which we think is related to not printing response cookies.
Scenario:
* configure ssl = true
Given url 'https://dvabc-qa.kdc.abc.com'
Given path 'api/dp/v1/apps/user/login'
And request {username: "test#abc.com", password: "1234"}
When method post
Then status 200
* print responseCookies
Request
1 > POST https://dvabc-qa.kdc.abc.com/api/dp/v1/apps/user/login
1 > Accept-Encoding: gzip,deflate
1 > Connection: Keep-Alive
1 > Content-Length: 52
1 > Content-Type: application/json
1 > Host: dvabc-qa.abc.com
1 > User-Agent: Apache-HttpClient/4.5.3 (Java/1.8.0_121)
{"username":"test#capone.com","password":"test1234"} Response 14:48:30.700 [main] WARN o.a.h.c.p.ResponseProcessCookies - Cookie rejected [X-AUTH-TOKEN="eyJ1c2VySW5mbyI6eyJ1c2VyRmlyc3ROYW1lIjoiVGVzdCIsInVzZXJMYXN0TmFtZSI6Ilh0ZXN0IiwidXNlcklEIjoiMTMxMTMx...", version:1, domain:.abc.com, path:/, expiry:null] Domain attribute ".abc.com" violates RFC 2109: host minus domain may not contain any dots
14:48:30.704 [main] WARN o.a.h.c.p.ResponseProcessCookies - Cookie rejected [SSOID="test#abc.com", version:1, domain:.abc.com, path:/, expiry:null] Domain attribute ".abc.com" violates RFC 2109: host minus domain may not contain any dots
14:48:30.718 [main] DEBUG com.intuit.karate -
1 < 200
1 < CUSTOMER_APP_DATA_REF_KEY: $2a$10$qnUQ.paqXvMEJSmu6G/BhL7d08oOFnSAVmOBVQjec3Umlme
1 < Cache-Control: no-cache, no-store, must-revalidate
1 < Connection: keep-alive
1 < Content-Type: application/json;charset=UTF-8
1 < Date: Mon, 23 Oct 2017 18:48:32 GMT
1 < Expires: Thu, 01 Jan 1970 00:00:00 GMT
1 < Pragma: no-cache
1 < SSOID: test#abc.com
1 < Set-Cookie: [X-AUTH-TOKEN="eyJ1c2VySW5joiVGVzdCwTaG93IjpudWxsLCJoYXNEaWFtb25kRGVhbGVyc2hpcHMiOmZhbHNlLCJub25EaWFtb25kRGVhbGVyQWNjZXNzRW5hYmxlZCI6ZmFsc2UsIm1mYVRva2VuIjpmYWxzZSwicmVzcG9uZGVkVG9BbGxUYyInVzZXJuYW1lIjoidGVzdEBjYXBvbmUuY29tIn0=.HzxYTNQGXk8n0HePKBcuk1ND5h6P8z4xvfaoJ5qah94="; Version=1; Domain=.abc.com; Path=/; HttpOnly, CUSTOMER_APP_DATA_REF_KEY=$2a$10$qnUQ.paPnsI9gqXvMEJSmu6G/BhL7d08oOFnSAVmOBVQjec3Umlme; Domain=.abc.com; Path=/; HttpOnly, SSOID="test#abc.com"; Version=1; Domain=.abc.com; Path=/; HttpOnly]
1 < Vary: Accept-Encoding,User-Agent
1 < X-AUTH-TOKEN: eyJ1c2VySW5mbyI6eyJ1c2VyRmlyc3ROYW1lIjoiVGVzdCIsInVzZXJMYXN0TmFtZSI6Ilh0ZXN0IiwidOiJ0ZXN0QGNhcG9uZS5jb20iLCJyYW5kb21UZXh0IjpudWxsLCJpbnRlcm5hbFVzZXJFbWFpbCI6bnVsbCwidXNlckRCJqb2JUaXRsZSI6IlNhbGVzIE1hbmFnZXIifSwiYWN0aXZlRGlyZWN0b3J5QWNjb3VudCI6ZmFsc2UsInRjVG9TaG93IjpudWxsLCJoYXNEaWFtb25kRGVhbGVyc2hpcHMiOmZhbHNlLCJub25EaWFtb25kRGVhbGVyQWNjZXNzRW5hYmxlZCI6ZmFsc2UsIm1mYVRva2VuIjpmYWxzZSwicmVzcG9uZGVkVG9BbGxUYyI6ZmFsc2UsInVzZXJuYW1lIjoidGVzdEBjYXBvbmUuY29tIn0=.HzxYTNk1ND5h6P8z4xvfaoJ5qah94=
1 < X-Frame-Options: SAMEORIGIN
14:48:30.719 [main] DEBUG com.intuit.karate - response time in milliseconds: 583
14:48:30.726 [main] INFO com.intuit.karate - [print]

Okay, this may need a fix or enhancement to Karate. Can you kindly file an enhancement request.
Violates RFC 2109: host > minus domain may not contain any dots
Meanwhile, can you try switching from karate-apache to karate-jersey in your pom.xml and see if that makes a difference.

Related

Having trouble getting content-encoding to show up in response header for get request

I was trying to write a karate test that validated that a particular response header contained the Content-Encoding header field with a value of gzip. I tried on my api where both times the content-encoding field was missing from karate's response. Both of these endpoints returned the content-encoding field on postman and curl commands.
I then tried to hit the postman-echo service to see if it was my api endpoint that karate was having issues with and it seems that it is not only my api. Could someone take a look at my code and see if I am doing something incorrectly to get the header field to show up in the response?
Feature: test getting Content-Encoding
Background:
* url 'https://postman-echo.com/gzip'
Scenario:
Given header Accept-Encoding = 'gzip'
When method get
Then status 200
And match responseHeaders contains {'Content-Encoding':'#present'}
This is what karate request looks like
1 > GET https://postman-echo.com/gzip
1 > Accept-Encoding: gzip
1 > Connection: Keep-Alive
1 > Host: postman-echo.com
1 > User-Agent: Apache-HttpClient/4.5.12 (Java/1.8.0_252)
and the response looks like
1 < 200
1 < Connection: keep-alive
1 < Content-Type: application/json; charset=utf-8
1 < Date: Fri, 08 May 2020 16:18:42 GMT
1 < ETag: W/"ef-7kclc8pzXTvQiPUaEOf6j95iFaE"
1 < Vary: Accept-Encoding
1 < set-cookie: sails.sid=s%3A6G_FShPRZH4V1G-tVDfUEEfMwQQmolo5.T2Cb37zqYA21FTyRyIGutVWQWo9ta4EWiod36%2FkM88I; Path=/; HttpOnly
{
"gzipped": true,
"headers": {
"x-forwarded-proto": "https",
"x-forwarded-port": "443",
"host": "postman-echo.com",
"x-amzn-trace-id": "Root=1-5eb58662-c4aaeec26efd116ac0544a18",
"accept-encoding": "gzip",
"user-agent": "Apache-HttpClient/4.5.12 (Java/1.8.0_252)"
},
"method": "GET"
}
The curl response header is
curl --location --request GET 'https://postman-echo.com/gzip' \
> --header 'Accept-Encoding: gzip' -I
HTTP/2 200
date: Fri, 08 May 2020 16:21:53 GMT
content-type: application/json; charset=utf-8
content-length: 220
content-encoding: gzip
etag: W/"dc-BuD8DN1qXT7trYUQtZOuSvbq1pM"
vary: Accept-Encoding
set-cookie: sails.sid=s%3Aj86lznX3nK20fnEN4B3nbHESrfWqVJ3M.236VrsmQp7V%2F7%2BrvG%2FEtlc9yUVLTtylh1yyIAdQJSiY; Path=/; HttpOnly
This seems to be the way the Apache HttpClient works or how it is configured for Karate. I just found you get the header back with karate-jersey:
1 > GET https://postman-echo.com/gzip
1 > Accept: */*
1 > Accept-Encoding: gzip
1 > User-Agent: Jersey/2.30 (HttpUrlConnection 1.8.0_231)
22:40:48.981 [ForkJoinPool-1-worker-1] DEBUG com.intuit.karate - response time in milliseconds: 1177.37
1 < 200
1 < Connection: keep-alive
1 < Content-Encoding: gzip
1 < Content-Length: 248
1 < Content-Type: application/json; charset=utf-8
1 < Date: Fri, 08 May 2020 17:10:48 GMT
1 < ETag: W/"f8-sigbV4PuNI2Fx08AqzMEqW1WIYY"
1 < Vary: Accept-Encoding
So if Jersey is ok for the rest of your tests, maybe that's all you need. Getting this to work for karate-apache is not a priority for me so if you or anyone else is willing to investigate or fix it, that would be great.

Karate test follows 303 dummy URL when `Expires: 0` header is missing

I have a Karate test that sends a POST that then will send back a redirect to a dummy address. With a recent Spring upgrade, I am now seeing the redirect coming back as a 303 rather than a 302.
I do not want this 302 URL to be followed with a request. The test just verified that we got a 302 back, and continued with more validation. This is what I have been seeing up until the Spring upgrade. However, with the 303 returned, it is then followed by a GET request to the dummy URL, which then fails with an UnknownHostException. I think perhaps due to the presence of Expires: 0 in the headers of the 302. The 303 does not contain this header (or Pragma: no-cache but I can't see how that would affect it). Below are the reponse headers for each, with differences highlighted with *
302
5 < 302
* 5 < Cache-Control: no-cache, no-store, max-age=0, must-revalidate
5 < Connection: keep-alive
5 < Content-Language: en-US
5 < Content-Length: 0
5 < Date: Mon, 16 Mar 2020 12:04:55 GMT
* 5 < Expires: 0
5 < Location: http://anywhere?key=value&code=iXAnzC
* 5 < Pragma: no-cache
5 < Server: nginx/1.17.9
5 < X-Content-Type-Options: nosniff
5 < X-Frame-Options: DENY
5 < X-XSS-Protection: 1; mode=block
303
5 < 303
5 < Cache-Control: no-store
5 < Connection: keep-alive
5 < Content-Language: en-US
5 < Content-Length: 0
5 < Date: Fri, 13 Mar 2020 11:06:31 GMT
5 < Location: http://anywhere?key=value&code=05O7lL
5 < Server: nginx/1.17.9
5 < X-Content-Type-Options: nosniff
5 < X-Frame-Options: DENY
5 < X-XSS-Protection: 1; mode=block
We do not explicitly set these headers so it will be Tomcat/NGinx behaviour, which means it's going to be standard on many servers. Is this generally expected behaviour with 303's or is this related to the differences in the headers?
Can you look at the configure followRedirects option: https://github.com/intuit/karate#configure - but I guess you know this already and maybe that works only for 302-s.
If you have already set that to false and still see Karate make the second call automatically, it may need a fix. Any pointers to what changes we need to make in the karate-apache code will help, here is the implementation for followRedirects.
You can also try switch to karate-jersey and see if that client works the way you expect.

Disable follow redirect in Karate BDD framework?

I have a part of my application flow that reuses an old login service. This old login service uses a 302 redirect with a Location header that includes a query param I have to capture and use as an auth code.
Right now, the Karate framework is helpfully forwarding me to the location specified in the Location header. How can I disable this so that instead I can capture the values myself?
09:39:26.096 [main] DEBUG com.intuit.karate -
2 < 302
2 < Connection: keep-alive
2 < Content-Length: 0
2 < Content-Type: application/json;charset=ISO-8859-1
2 < Date: Fri, 08 Sep 2017 15:39:25 GMT
2 < Server: nginx/1.4.6 (Ubuntu)
2 < Set-Cookie: [JSESSIONID=0000UDF76JJ7ZGsFlTs9FPYcVoN:1bj4v930o; Path=/; HttpOnly, PD-S-SESSION-ID=1_2_1_upREX6Zqm0NTxJ5dNQeSyVeaiUFYnO1OdgUVYXYnekLFY-A7; Path=/; Secure; HttpOnly, PD-ID=ZugwEuE0PZmTNhZkIlog5FqfvlDKD2nfvKAZ+ebfTvVx2dzw0JAgyOMeviZx3xsxYndwQxGmURx6n32fscUXMw3X8P9FpNDssgIz355kfTeda/25qxPf+uaEvs8JpE2FxsMWkzQ25qM2KmrhAK9QTRu31oflpD6eL+klFfDahSSTQlFMXZmB4Yjsc8DOnoGAhbv19Yh9A8OuzDFXimwML2a+Yn9UYN/ZjRXF45veNQGsA9/dtP/nfZl6eAPyy7mdSKunhHuWggBtYFP9kAgmB1LJ87MFWsOhZEwcnmLsJM078tRStGLeOFo7KX+1FtMhhcV76UkTxDFj1VR68fSPdzF+84oeyxf0WhQ0UZB+oc76kQtl94B7uW4X/lp9jADlGUwrE8hPQ5asPcPblt6E0yPyi+TvouK/0XHpy7QfAeXZqyifs+yeUOKHBdHUXHI7QhnadEX0GViHPsqZ5amztnYkF5mMQLkdXYIO22oSHCtci7M08gr1usTLrHc/hIBm; Path=/; Secure; HttpOnly, BigIPWebSeal=rd3o00000000000000000000ffff0a1bf852o443; path=/]
2 < X-Application-Context: sv:dev:8080
2 < X-B3-Sampled: 1
2 < X-B3-SpanId: 3229b43a709ac873
2 < X-B3-TraceId: 3229b43a709ac873
2 < X-Backside-Transport: OK OK
2 < X-Client-IP: x.x.x.x
2 < X-Global-Transaction-ID: 1890531591
2 < cache-control: no-cache="set-cookie, set-cookie2"
2 < content-language: en-US
2 < expires: Thu, 01 Dec 1994 16:00:00 GMT
2 < iis_server_name: APPA014D
2 < location: https://a-url.com/path?code=a51769a482e5
2 < p3p: CP="NON CUR OTPi OUR NOR UNI"
2 < strict-transport-security: max-age=31536000; includeSubDomains
2 < x-powered-by: Servlet/3.0
2 < x-xss-protection: 1;mode=block
Wow, to be honest no-one has asked for this. Until now. Looks like we'll have to add this support: http://www.baeldung.com/httpclient-stop-follow-redirect
This can be implemented rather quickly, can you raise a feature request ? If this is a blocker, you could temporarily call into Java and use an instance of the Apache Http Client or something. I was wondering if you had the option to access a lastRedirectUrl instead would that have helped.
edit: this has been implemented in dev and will be available in version 0.6.0

Pulling data from Netflix through private account

I want to pull a list of all the movies and shows I have seen on Netflix for a personal project, which Netflix has a page for.
Results from trying curl:
curl https://www.netflix.com/MoviesYouveSeen -v
* Trying 50.112.92.119...
* Connected to www.netflix.com (50.112.92.119) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
* ALPN/NPN, server did not agree to a protocol
* SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate:
* subject: CN=www.netflix.com,OU=Operations,O="Netflix, Inc.",L=Los Gatos,ST=CALIFORNIA,C=US
* start date: Apr 14 00:00:00 2015 GMT
* expire date: Apr 12 23:59:59 2017 GMT
* common name: www.netflix.com
* issuer: CN=Symantec Class 3 Secure Server CA - G4,OU=Symantec Trust Network,O=Symantec Corporation,C=US
> GET /MoviesYouveSeen HTTP/1.1
> Host: www.netflix.com
> User-Agent: curl/7.43.0
> Accept: */*
>
< HTTP/1.1 302 Found
< Cache-Control: no-cache, no-store
< Date: Tue, 26 Apr 2016 14:47:16 GMT
< Edge-Control: no-cache, no-store
< location: https://www.netflix.com/Login?nextpage=https%3A%2F%2Fwww.netflix.com%2FMoviesYouveSeen
< req_id: 2a134cc9-7f77-4a35-9d83-0099fc7a2466
< Server: shakti-prod i-8cf6164a
< Set-Cookie: nflx-rgn=uw2|1461682036196; Max-Age=-1; Expires=Tue, 26 Apr 2016 14:47:15 GMT; Path=/; Domain=.netflix.com
< Set-Cookie: memclid=b40d0e2c-27b3-4d72-9b14-4477fcf5fa39; Max-Age=31536000; Expires=Wed, 26 Apr 2017 14:47:16 GMT; Path=/; Domain=.netflix.com
< Set-Cookie: nfvdid=BQFmAAEBEDgFjrzXIIi7X6rTj6vmSYUwYpekhXXCCx5ywGWHaOvo0%2BmNx86oMCsliwERTTbRi6FwmgZM3YhqFUBfffSwJ0Kd; Max-Age=31536000; Expires=Wed, 26 Apr 2017 14:47:16 GMT; Path=/; Domain=.netflix.com
< Strict-Transport-Security: max-age=31536
< Via: 1.1 i-6af8eaad (us-west-2)
< X-Content-Type-Options: nosniff
< X-Frame-Options: DENY
< X-Netflix-From-Zuul: true
< X-Netflix.nfstatus: 1_1
< X-Originating-URL: https://www.netflix.com/MoviesYouveSeen
< X-Xss-Protection: 1; mode=block; report=https://ichnaea.netflix.com/log/freeform/xssreport
< Content-Length: 256
< Connection: keep-alive
<
* Connection #0 to host www.netflix.com left intact
I also tried wget:
wget https://www.netflix.com/MoviesYouveSeen
--2016-04-26 10:57:23-- https://www.netflix.com/MoviesYouveSeen
Resolving www.netflix.com (www.netflix.com)... 54.244.126.7, 50.112.115.177, 54.214.7.82, ...
Connecting to www.netflix.com (www.netflix.com)|54.244.126.7|:443... connected.
HTTP request sent, awaiting response... 302 Found
Syntax error in Set-Cookie: nflx-rgn=uw2|1461682643973; Max-Age=-1; Expires=Tue, 26 Apr 2016 14:57:23 GMT; Path=/; Domain=.netflix.com at position 39.
Location: https://www.netflix.com/Login?nextpage=https%3A%2F%2Fwww.netflix.com%2FMoviesYouveSeen [following]
--2016-04-26 10:57:24-- https://www.netflix.com/Login?nextpage=https%3A%2F%2Fwww.netflix.com%2FMoviesYouveSeen
Reusing existing connection to www.netflix.com:443.
HTTP request sent, awaiting response... 200 OK
Syntax error in Set-Cookie: nflx-rgn=uw2|1461682644112; Max-Age=-1; Expires=Tue, 26 Apr 2016 14:57:23 GMT; Path=/; Domain=.netflix.com at position 39.
Length: unspecified [text/html]
Saving to: ‘MoviesYouveSeen’
MoviesYouveSeen [ <=> ] 41.63K 220KB/s in 0.2s
2016-04-26 10:57:24 (220 KB/s) - ‘MoviesYouveSeen’ saved [42629]
It looks like I am not being properly authenticated. Inside my browser if I view source I can see the list of movies. Any suggestions for getting the data?
That 302 response is redirecting you to the login page. You'd need to be logged in for the query to work correctly.

will Accept Http Headers digest any other formats other than the one specified?

Per my understanding:
the Accept header is used by HTTP clients to tell the server what content types they'll accept. The server will then send back a response, which will include a Content-Type header telling the client what the content type of the returned content actually is.
With this understanding, I tried the following:
curl -X GET -H "Accept: application/xml" http://www.google.com -v
* About to connect() to www.google.com port 80 (#0)
* Trying 173.194.33.81...
* connected
* Connected to www.google.com (173.194.33.81) port 80 (#0)
> GET / HTTP/1.1
> User-Agent: curl/7.24.0 (x86_64-apple-darwin12.0) libcurl/7.24.0 OpenSSL/0.9.8y zlib/1.2.5
> Host: www.google.com
> Accept: application/xml
>
< HTTP/1.1 200 OK
< Date: Tue, 02 Sep 2014 17:58:05 GMT
< Expires: -1
< Cache-Control: private, max-age=0
< Content-Type: text/html; charset=ISO-8859-1
< Set-Cookie: PREF=ID=5c30672b67a74789:FF=0:TM=1409680685:LM=1409680685:S=PsGclk3vR4HWjann; expires=Thu, 01-Sep-2016 17:58:05 GMT; path=/; domain=.google.com
< Set-Cookie: NID=67=rPuxpwUu5UNuapzCdbD5iwVyjjC9TzP_Ado29h3ucjEq4A_2qkSM4nQM3RO02rfyuHmrh-hvmwmgFCmOvISttFfHv06f8ay4_6Gl4pXRjqxihNhJSGbvujjDRzaSibfy; expires=Wed, 04-Mar-2015 17:58:05 GMT; path=/; domain=.google.com; HttpOnly
< P3P: CP="This is not a P3P policy! See http://www.google.com/support/accounts/bin/answer.py?hl=en&answer=151657 for more info."
< Server: gws
< X-XSS-Protection: 1; mode=block
< X-Frame-Options: SAMEORIGIN
< Alternate-Protocol: 80:quic
< Transfer-Encoding: chunked
<
<!doctype html><html itemscope="" itemtype="http://schema.org/WebPage" lang="en"><
As you can notice in the response, I am sent Content-Type: text/html; charset=ISO-8859-1 which is not what I asked for?
why does a different representation (HTML in this case) is sent, although I asked for xml?
Thanks
From RFC 2616:
If an Accept header field is present,
and if the server cannot send a response which is acceptable
according to the combined Accept field value, then the server SHOULD
send a 406 (not acceptable) response.
Here, "should" means that Google aren't actually obliged to throw a 406 error. But since you're receiving an HTML response, it has effectively the same meaning.