I have a part of my application flow that reuses an old login service. This old login service uses a 302 redirect with a Location header that includes a query param I have to capture and use as an auth code.
Right now, the Karate framework is helpfully forwarding me to the location specified in the Location header. How can I disable this so that instead I can capture the values myself?
09:39:26.096 [main] DEBUG com.intuit.karate -
2 < 302
2 < Connection: keep-alive
2 < Content-Length: 0
2 < Content-Type: application/json;charset=ISO-8859-1
2 < Date: Fri, 08 Sep 2017 15:39:25 GMT
2 < Server: nginx/1.4.6 (Ubuntu)
2 < Set-Cookie: [JSESSIONID=0000UDF76JJ7ZGsFlTs9FPYcVoN:1bj4v930o; Path=/; HttpOnly, PD-S-SESSION-ID=1_2_1_upREX6Zqm0NTxJ5dNQeSyVeaiUFYnO1OdgUVYXYnekLFY-A7; Path=/; Secure; HttpOnly, PD-ID=ZugwEuE0PZmTNhZkIlog5FqfvlDKD2nfvKAZ+ebfTvVx2dzw0JAgyOMeviZx3xsxYndwQxGmURx6n32fscUXMw3X8P9FpNDssgIz355kfTeda/25qxPf+uaEvs8JpE2FxsMWkzQ25qM2KmrhAK9QTRu31oflpD6eL+klFfDahSSTQlFMXZmB4Yjsc8DOnoGAhbv19Yh9A8OuzDFXimwML2a+Yn9UYN/ZjRXF45veNQGsA9/dtP/nfZl6eAPyy7mdSKunhHuWggBtYFP9kAgmB1LJ87MFWsOhZEwcnmLsJM078tRStGLeOFo7KX+1FtMhhcV76UkTxDFj1VR68fSPdzF+84oeyxf0WhQ0UZB+oc76kQtl94B7uW4X/lp9jADlGUwrE8hPQ5asPcPblt6E0yPyi+TvouK/0XHpy7QfAeXZqyifs+yeUOKHBdHUXHI7QhnadEX0GViHPsqZ5amztnYkF5mMQLkdXYIO22oSHCtci7M08gr1usTLrHc/hIBm; Path=/; Secure; HttpOnly, BigIPWebSeal=rd3o00000000000000000000ffff0a1bf852o443; path=/]
2 < X-Application-Context: sv:dev:8080
2 < X-B3-Sampled: 1
2 < X-B3-SpanId: 3229b43a709ac873
2 < X-B3-TraceId: 3229b43a709ac873
2 < X-Backside-Transport: OK OK
2 < X-Client-IP: x.x.x.x
2 < X-Global-Transaction-ID: 1890531591
2 < cache-control: no-cache="set-cookie, set-cookie2"
2 < content-language: en-US
2 < expires: Thu, 01 Dec 1994 16:00:00 GMT
2 < iis_server_name: APPA014D
2 < location: https://a-url.com/path?code=a51769a482e5
2 < p3p: CP="NON CUR OTPi OUR NOR UNI"
2 < strict-transport-security: max-age=31536000; includeSubDomains
2 < x-powered-by: Servlet/3.0
2 < x-xss-protection: 1;mode=block
Wow, to be honest no-one has asked for this. Until now. Looks like we'll have to add this support: http://www.baeldung.com/httpclient-stop-follow-redirect
This can be implemented rather quickly, can you raise a feature request ? If this is a blocker, you could temporarily call into Java and use an instance of the Apache Http Client or something. I was wondering if you had the option to access a lastRedirectUrl instead would that have helped.
edit: this has been implemented in dev and will be available in version 0.6.0
Related
I am trying to do a perfectly conventional thing: I am using
CloudFront / S3 to host a static website, but I also want to host
another website in a subdirectory. Following the instruction, I
believe I got S3 to work
% curl -v http://mydomain.me.s3-website-us-west-1.amazonaws.com/c
> GET /c HTTP/1.1
> Host: mydomain.me.s3-website-us-west-1.amazonaws.com
> User-Agent: curl/7.64.1
> Accept: */*
>
< HTTP/1.1 302 Moved Temporarily
< x-amz-error-code: Found
< x-amz-error-message: Resource Found
< x-amz-request-id: 9BB13A73FFB4503E
< x-amz-id-2: 3JX26tNdHi1irPbFJS7E1BifwliygqRZsZIc/qZptjBqBjjmGL7YGK6xfG23GZR70R0Ou+3ZAiM=
< Location: /c/
< Content-Type: text/html; charset=utf-8
< Content-Length: 313
< Date: Tue, 01 Dec 2020 01:58:08 GMT
< Server: AmazonS3
So /c is redirecting to /c/, which I believe is correct, and that new location definitely serves correctly:
% curl -v http://mydomain.me.s3-website-us-west-1.amazonaws.com/c/
> GET /c/ HTTP/1.1
> Host: mydomain.me.s3-website-us-west-1.amazonaws.com
> User-Agent: curl/7.64.1
> Accept: */*
>
< HTTP/1.1 200 OK
< x-amz-id-2: BD0wdDnhonp7Y5i2b7mUDVbIXKYu4O52YPUKVQx5GDaLW5hmDzcrsF/EixdksCtkt/NK6Bg24hY=
< x-amz-request-id: 7F11B109218EF9ED
< Date: Tue, 01 Dec 2020 01:58:11 GMT
< Last-Modified: Tue, 01 Dec 2020 01:31:59 GMT
< x-amz-version-id: zSq5IxE3Ug8oG5SSW.lZsCYydp42.h.4
< ETag: "7999ccd49fe930021167ae6f8fe95eb6"
< Content-Type: text/html
< Content-Length: 36
< Server: AmazonS3
<
And it actually gives me my file. But when I try to go through CloudFront for /c:
% curl -v https://mydomain.me/c
> GET /c HTTP/2
> Host: mydomain.me
> User-Agent: curl/7.64.1
> Accept: */*
>
< HTTP/2 403
< content-type: application/xml
< date: Tue, 01 Dec 2020 01:59:43 GMT
< server: AmazonS3
< x-cache: Error from cloudfront
< via: 1.1 58b53da3f7d231b76d30fcffbf4945a1.cloudfront.net (CloudFront)
< x-amz-cf-pop: SFO20-C1
< x-amz-cf-id: PSjqsinkkfheUfhEPVYbbujMqemugFbrYxM-pQMIihMk3dpp2W4Bmw==
and it downloads the familiar S3 access denied. For /c/, it is even weirder:
% curl -v https://mydomain.me/c/
> GET /c/ HTTP/2
> Host: mydomain.me
> User-Agent: curl/7.64.1
> Accept: */*
>
< HTTP/2 200
< content-type: application/x-directory; charset=UTF-8
< content-length: 0
< last-modified: Tue, 01 Dec 2020 01:30:44 GMT
< x-amz-version-id: 4L.jn6WG3emcGutRuwEZv_lE0aO07AGR
< accept-ranges: bytes
< server: AmazonS3
< date: Tue, 01 Dec 2020 02:00:31 GMT
< etag: "d41d8cd98f00b204e9800998ecf8427e"
< x-cache: RefreshHit from cloudfront
< via: 1.1 37d64bca4c93552139fb3a85c9c4a119.cloudfront.net (CloudFront)
< x-amz-cf-pop: SFO20-C1
< x-amz-cf-id: r5lS4QTmg07XhIXRlXsNJ4qcJaWXfj5Ik9fXZPY_dzLjED-A2MhBiA==
It "works", but it returns an empty file, which it says is a directory listing.
I have logging turned on, and that last one returns:
b5063beaaa3c80c2ad85635ddb1c5fac3da6b5510e9ef332c9e0df0c9abdd45a mydomain.me [01/Dec/2020:01:57:47 +0000] 73.202.134.48 b5063beaaa3c80c2ad85635ddb1c5fac3da6b5510e9ef332c9e0df0c9abdd45a 116EA2ED16AA56DE REST.GET.NOTIFICATION - "GET /mydomain.me?notification= HTTP/1.1" 200 - 115 - 15 - "-" "S3Console/0.4, aws-internal/3 aws-sdk-java/1.11.888 Linux/4.9.217-0.3.ac.206.84.332.metal1.x86_64 OpenJDK_64-Bit_Server_VM/25.262-b10 java/1.8.0_262 vendor/Oracle_Corporation" - noe+YUO+FeYaIukSpTTKl9npt1R0+uAr4Hqzx/mQge2bfhydBiiquR9EWG3iGanDRjK/EagN5Ss= SigV4 ECDHE-RSA-AES128-SHA AuthHeader s3-us-west-1.amazonaws.com TLSv1.2
CloudFront is running some Java library?
curl -v https://mydomain.me/c/index.html works fine.
I assume I have misconfigured CloudFront, but cannot figure out how. Any suggestions?
Click on the CloudFront Distribution ID
Select the tab "Origins and Origin Groups"
Click the checkbox for the first item under "Origins" (assuming you only have one)
Click "Edit"
Change the "Origin Domain Name" to
"mydomain.me.s3-website-us-west-1.amazonaws.com" (following your
example)
Click "Yes, Edit"
I've done this a hundred times, I know this is a requirement, and it bites me every time!
I have a Karate test that sends a POST that then will send back a redirect to a dummy address. With a recent Spring upgrade, I am now seeing the redirect coming back as a 303 rather than a 302.
I do not want this 302 URL to be followed with a request. The test just verified that we got a 302 back, and continued with more validation. This is what I have been seeing up until the Spring upgrade. However, with the 303 returned, it is then followed by a GET request to the dummy URL, which then fails with an UnknownHostException. I think perhaps due to the presence of Expires: 0 in the headers of the 302. The 303 does not contain this header (or Pragma: no-cache but I can't see how that would affect it). Below are the reponse headers for each, with differences highlighted with *
302
5 < 302
* 5 < Cache-Control: no-cache, no-store, max-age=0, must-revalidate
5 < Connection: keep-alive
5 < Content-Language: en-US
5 < Content-Length: 0
5 < Date: Mon, 16 Mar 2020 12:04:55 GMT
* 5 < Expires: 0
5 < Location: http://anywhere?key=value&code=iXAnzC
* 5 < Pragma: no-cache
5 < Server: nginx/1.17.9
5 < X-Content-Type-Options: nosniff
5 < X-Frame-Options: DENY
5 < X-XSS-Protection: 1; mode=block
303
5 < 303
5 < Cache-Control: no-store
5 < Connection: keep-alive
5 < Content-Language: en-US
5 < Content-Length: 0
5 < Date: Fri, 13 Mar 2020 11:06:31 GMT
5 < Location: http://anywhere?key=value&code=05O7lL
5 < Server: nginx/1.17.9
5 < X-Content-Type-Options: nosniff
5 < X-Frame-Options: DENY
5 < X-XSS-Protection: 1; mode=block
We do not explicitly set these headers so it will be Tomcat/NGinx behaviour, which means it's going to be standard on many servers. Is this generally expected behaviour with 303's or is this related to the differences in the headers?
Can you look at the configure followRedirects option: https://github.com/intuit/karate#configure - but I guess you know this already and maybe that works only for 302-s.
If you have already set that to false and still see Karate make the second call automatically, it may need a fix. Any pointers to what changes we need to make in the karate-apache code will help, here is the implementation for followRedirects.
You can also try switch to karate-jersey and see if that client works the way you expect.
We are struggling to pass the response cookie to the subsequent request as we are getting Cookie Rejected error. We are also not able to print the cookie response:
Here are more details: giving req and response. See the Print at the very end prints nothing and at the start of response there are warning which we think is related to not printing response cookies.
Scenario:
* configure ssl = true
Given url 'https://dvabc-qa.kdc.abc.com'
Given path 'api/dp/v1/apps/user/login'
And request {username: "test#abc.com", password: "1234"}
When method post
Then status 200
* print responseCookies
Request
1 > POST https://dvabc-qa.kdc.abc.com/api/dp/v1/apps/user/login
1 > Accept-Encoding: gzip,deflate
1 > Connection: Keep-Alive
1 > Content-Length: 52
1 > Content-Type: application/json
1 > Host: dvabc-qa.abc.com
1 > User-Agent: Apache-HttpClient/4.5.3 (Java/1.8.0_121)
{"username":"test#capone.com","password":"test1234"} Response 14:48:30.700 [main] WARN o.a.h.c.p.ResponseProcessCookies - Cookie rejected [X-AUTH-TOKEN="eyJ1c2VySW5mbyI6eyJ1c2VyRmlyc3ROYW1lIjoiVGVzdCIsInVzZXJMYXN0TmFtZSI6Ilh0ZXN0IiwidXNlcklEIjoiMTMxMTMx...", version:1, domain:.abc.com, path:/, expiry:null] Domain attribute ".abc.com" violates RFC 2109: host minus domain may not contain any dots
14:48:30.704 [main] WARN o.a.h.c.p.ResponseProcessCookies - Cookie rejected [SSOID="test#abc.com", version:1, domain:.abc.com, path:/, expiry:null] Domain attribute ".abc.com" violates RFC 2109: host minus domain may not contain any dots
14:48:30.718 [main] DEBUG com.intuit.karate -
1 < 200
1 < CUSTOMER_APP_DATA_REF_KEY: $2a$10$qnUQ.paqXvMEJSmu6G/BhL7d08oOFnSAVmOBVQjec3Umlme
1 < Cache-Control: no-cache, no-store, must-revalidate
1 < Connection: keep-alive
1 < Content-Type: application/json;charset=UTF-8
1 < Date: Mon, 23 Oct 2017 18:48:32 GMT
1 < Expires: Thu, 01 Jan 1970 00:00:00 GMT
1 < Pragma: no-cache
1 < SSOID: test#abc.com
1 < Set-Cookie: [X-AUTH-TOKEN="eyJ1c2VySW5joiVGVzdCwTaG93IjpudWxsLCJoYXNEaWFtb25kRGVhbGVyc2hpcHMiOmZhbHNlLCJub25EaWFtb25kRGVhbGVyQWNjZXNzRW5hYmxlZCI6ZmFsc2UsIm1mYVRva2VuIjpmYWxzZSwicmVzcG9uZGVkVG9BbGxUYyInVzZXJuYW1lIjoidGVzdEBjYXBvbmUuY29tIn0=.HzxYTNQGXk8n0HePKBcuk1ND5h6P8z4xvfaoJ5qah94="; Version=1; Domain=.abc.com; Path=/; HttpOnly, CUSTOMER_APP_DATA_REF_KEY=$2a$10$qnUQ.paPnsI9gqXvMEJSmu6G/BhL7d08oOFnSAVmOBVQjec3Umlme; Domain=.abc.com; Path=/; HttpOnly, SSOID="test#abc.com"; Version=1; Domain=.abc.com; Path=/; HttpOnly]
1 < Vary: Accept-Encoding,User-Agent
1 < X-AUTH-TOKEN: eyJ1c2VySW5mbyI6eyJ1c2VyRmlyc3ROYW1lIjoiVGVzdCIsInVzZXJMYXN0TmFtZSI6Ilh0ZXN0IiwidOiJ0ZXN0QGNhcG9uZS5jb20iLCJyYW5kb21UZXh0IjpudWxsLCJpbnRlcm5hbFVzZXJFbWFpbCI6bnVsbCwidXNlckRCJqb2JUaXRsZSI6IlNhbGVzIE1hbmFnZXIifSwiYWN0aXZlRGlyZWN0b3J5QWNjb3VudCI6ZmFsc2UsInRjVG9TaG93IjpudWxsLCJoYXNEaWFtb25kRGVhbGVyc2hpcHMiOmZhbHNlLCJub25EaWFtb25kRGVhbGVyQWNjZXNzRW5hYmxlZCI6ZmFsc2UsIm1mYVRva2VuIjpmYWxzZSwicmVzcG9uZGVkVG9BbGxUYyI6ZmFsc2UsInVzZXJuYW1lIjoidGVzdEBjYXBvbmUuY29tIn0=.HzxYTNk1ND5h6P8z4xvfaoJ5qah94=
1 < X-Frame-Options: SAMEORIGIN
14:48:30.719 [main] DEBUG com.intuit.karate - response time in milliseconds: 583
14:48:30.726 [main] INFO com.intuit.karate - [print]
Okay, this may need a fix or enhancement to Karate. Can you kindly file an enhancement request.
Violates RFC 2109: host > minus domain may not contain any dots
Meanwhile, can you try switching from karate-apache to karate-jersey in your pom.xml and see if that makes a difference.
I want to pull a list of all the movies and shows I have seen on Netflix for a personal project, which Netflix has a page for.
Results from trying curl:
curl https://www.netflix.com/MoviesYouveSeen -v
* Trying 50.112.92.119...
* Connected to www.netflix.com (50.112.92.119) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
* ALPN/NPN, server did not agree to a protocol
* SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate:
* subject: CN=www.netflix.com,OU=Operations,O="Netflix, Inc.",L=Los Gatos,ST=CALIFORNIA,C=US
* start date: Apr 14 00:00:00 2015 GMT
* expire date: Apr 12 23:59:59 2017 GMT
* common name: www.netflix.com
* issuer: CN=Symantec Class 3 Secure Server CA - G4,OU=Symantec Trust Network,O=Symantec Corporation,C=US
> GET /MoviesYouveSeen HTTP/1.1
> Host: www.netflix.com
> User-Agent: curl/7.43.0
> Accept: */*
>
< HTTP/1.1 302 Found
< Cache-Control: no-cache, no-store
< Date: Tue, 26 Apr 2016 14:47:16 GMT
< Edge-Control: no-cache, no-store
< location: https://www.netflix.com/Login?nextpage=https%3A%2F%2Fwww.netflix.com%2FMoviesYouveSeen
< req_id: 2a134cc9-7f77-4a35-9d83-0099fc7a2466
< Server: shakti-prod i-8cf6164a
< Set-Cookie: nflx-rgn=uw2|1461682036196; Max-Age=-1; Expires=Tue, 26 Apr 2016 14:47:15 GMT; Path=/; Domain=.netflix.com
< Set-Cookie: memclid=b40d0e2c-27b3-4d72-9b14-4477fcf5fa39; Max-Age=31536000; Expires=Wed, 26 Apr 2017 14:47:16 GMT; Path=/; Domain=.netflix.com
< Set-Cookie: nfvdid=BQFmAAEBEDgFjrzXIIi7X6rTj6vmSYUwYpekhXXCCx5ywGWHaOvo0%2BmNx86oMCsliwERTTbRi6FwmgZM3YhqFUBfffSwJ0Kd; Max-Age=31536000; Expires=Wed, 26 Apr 2017 14:47:16 GMT; Path=/; Domain=.netflix.com
< Strict-Transport-Security: max-age=31536
< Via: 1.1 i-6af8eaad (us-west-2)
< X-Content-Type-Options: nosniff
< X-Frame-Options: DENY
< X-Netflix-From-Zuul: true
< X-Netflix.nfstatus: 1_1
< X-Originating-URL: https://www.netflix.com/MoviesYouveSeen
< X-Xss-Protection: 1; mode=block; report=https://ichnaea.netflix.com/log/freeform/xssreport
< Content-Length: 256
< Connection: keep-alive
<
* Connection #0 to host www.netflix.com left intact
I also tried wget:
wget https://www.netflix.com/MoviesYouveSeen
--2016-04-26 10:57:23-- https://www.netflix.com/MoviesYouveSeen
Resolving www.netflix.com (www.netflix.com)... 54.244.126.7, 50.112.115.177, 54.214.7.82, ...
Connecting to www.netflix.com (www.netflix.com)|54.244.126.7|:443... connected.
HTTP request sent, awaiting response... 302 Found
Syntax error in Set-Cookie: nflx-rgn=uw2|1461682643973; Max-Age=-1; Expires=Tue, 26 Apr 2016 14:57:23 GMT; Path=/; Domain=.netflix.com at position 39.
Location: https://www.netflix.com/Login?nextpage=https%3A%2F%2Fwww.netflix.com%2FMoviesYouveSeen [following]
--2016-04-26 10:57:24-- https://www.netflix.com/Login?nextpage=https%3A%2F%2Fwww.netflix.com%2FMoviesYouveSeen
Reusing existing connection to www.netflix.com:443.
HTTP request sent, awaiting response... 200 OK
Syntax error in Set-Cookie: nflx-rgn=uw2|1461682644112; Max-Age=-1; Expires=Tue, 26 Apr 2016 14:57:23 GMT; Path=/; Domain=.netflix.com at position 39.
Length: unspecified [text/html]
Saving to: ‘MoviesYouveSeen’
MoviesYouveSeen [ <=> ] 41.63K 220KB/s in 0.2s
2016-04-26 10:57:24 (220 KB/s) - ‘MoviesYouveSeen’ saved [42629]
It looks like I am not being properly authenticated. Inside my browser if I view source I can see the list of movies. Any suggestions for getting the data?
That 302 response is redirecting you to the login page. You'd need to be logged in for the query to work correctly.
My .htaccess file:
Header add X-Hello "time %D"
Now I check to see what it looks like, check out the X-Hello header:
$curl -v http://foo.com/bar.php
...
< HTTP/1.1 200 OK
< Date: Wed, 06 Oct 2010 20:43:39 GMT
< Server: Apache
< Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
< Expires: Thu, 19 Nov 1981 08:52:00 GMT
< Pragma: no-cache
< Set-Cookie: PHPSESSID=7d32d8fch8qrnnuvoqqopr74c0; path=/
< X-Hello: time %D
< Transfer-Encoding: chunked
< Content-Type: text/html
Which is clearly contrary to the Apache documentation below. Is there some other directive that mod_headers needs in order to interpolate environment variables?
http://httpd.apache.org/docs/2.0/mod/mod_headers.html#examples
Without note otherwise, hosting provider admitted to still using Apache 1.3 which does not interpolate values in mod_header.