curl changes the URI in the authorization header for digest behind proxy - apache

The bounty expires in 4 days. Answers to this question are eligible for a +50 reputation bounty.
Mirza Prangon is looking for an answer from a reputable source:
Details how what is going wrong, where it is going wrong and how to fix it.
I am trying to use curl for a http request.
I have to use it behind a enterprise proxy server. The remote host uses digest authentication.
I am using the following curl command.
curl -x "http://proxy_username:proxy_pass#proxyIp.xxx.xxx.xxx:8080" -L -X GET "https://remote-host.something.com:443/tomcat_servlet/UploadServlet" --digest -u digest_auth_user:digest_auth_pass -v -k
But I get 400 bad request from apache httpd. The full output from curl is
* Trying proxyIp.xxx.xxx.xxx:8080...
* Connected to proxyIp.xxx.xxx.xxx (proxyIp.xxx.xxx.xxx) port 8080 (#0)
* allocate connect buffer
* Establish HTTP proxy tunnel to remote-host.something.com:443
* Proxy auth using Basic with user 'proxy_username'
* Server auth using Digest with user 'digest_auth_user'
> CONNECT remote-host.something.com:443 HTTP/1.1
> Host: remote-host.something.com:443
> Proxy-Authorization: Basic <redacted>
> User-Agent: curl/7.83.1
> Proxy-Connection: Keep-Alive
>
< HTTP/1.1 200 Connection established
< Via:HTTP/1.1 s_proxy_nrt
<
* Proxy replied 200 to CONNECT request
* CONNECT phase completed
* schannel: disabled automatic use of client certificate
* ALPN: offers http/1.1
* ALPN: server did not agree on a protocol. Uses default.
* Server auth using Digest with user 'digest_auth_user'
> GET /tomcat_servlet/UploadServlet HTTP/1.1
> Host: remote-host.something.com
> User-Agent: curl/7.83.1
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 307 Temporary Redirect
< Server: Cisco Umbrella
< Date: Tue, 14 Feb 2023 02:52:03 GMT
< Content-Type: text/html
< Content-Length: 190
< Connection: keep-alive
< Set-Cookie: swg_https_a2bc=1; Path=/; Expires=Tue, 14-Feb-23 03:02:03 GMT; domain=remote-host.something.com; SameSite=None; Secure
< Location: https://remote-host.something.com/tomcat_servlet/UploadServlet?swg_a2bc=1
< Via: HTTP/1.1 s_proxy_nrt
<
* Ignoring the response-body
* Connection #0 to host proxyIp.xxx.xxx.xxx left intact
* Issue another request to this URL: 'https://remote-host.something.com/tomcat_servlet/UploadServlet?swg_a2bc=1'
* Found bundle for host: 0x1a0ed47d970 [serially]
* Re-using existing connection #0 with proxy proxyIp.xxx.xxx.xxx
* Connected to proxyIp.xxx.xxx.xxx (proxyIp.xxx.xxx.xxx) port 8080 (#0)
* Server auth using Digest with user 'digest_auth_user'
> GET /tomcat_servlet/UploadServlet?swg_a2bc=1 HTTP/1.1
> Host: remote-host.something.com
> User-Agent: curl/7.83.1
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 401 Unauthorized
< Date: Tue, 14 Feb 2023 02:52:03 GMT
< Content-Type: text/html; charset=iso-8859-1
< Content-Length: 381
< Connection: keep-alive
< Server: Apache/2.4.48 (Win64) OpenSSL/1.1.1k
< WWW-Authenticate: Digest realm="https_transfer", nonce="redacted", algorithm=MD5, qop="auth"
< Via: HTTP/1.1 m_proxy_nrt
<
* Ignoring the response-body
* Connection #0 to host proxyIp.xxx.xxx.xxx left intact
* Issue another request to this URL: 'https://remote-host.something.com/tomcat_servlet/UploadServlet?swg_a2bc=1'
* Found bundle for host: 0x1a0ed47d970 [serially]
* Re-using existing connection #0 with proxy proxyIp.xxx.xxx.xxx
* Connected to proxyIp.xxx.xxx.xxx (proxyIp.xxx.xxx.xxx) port 8080 (#0)
* Server auth using Digest with user 'digest_auth_user'
> GET /tomcat_servlet/UploadServlet?swg_a2bc=1 HTTP/1.1
> Host: remote-host.something.com
> Authorization: Digest username="digest_auth_user",realm="https_transfer",nonce="redacted",uri="/tomcat_servlet/UploadServlet?swg_a2bc=1",cnonce="redacted",nc=00000001,algorithm=MD5,response="redacted",qop="redacted"
> User-Agent: curl/7.83.1
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 400 Bad Request
< Date: Tue, 14 Feb 2023 02:52:03 GMT
< Content-Type: text/html; charset=iso-8859-1
< Content-Length: 226
< Connection: keep-alive
< Server: Apache/2.4.48 (Win64) OpenSSL/1.1.1k
< Via: HTTP/1.1 m_proxy_nrt
<
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>400 Bad Request</title>
</head><body>
<h1>Bad Request</h1>
<p>Your browser sent a request that this server could not understand.<br />
</p>
</body></html>
* Connection #0 to host proxyIp.xxx.xxx.xxx left intact
Is the server side, I get the following in httpd log.
[auth_digest:error] [pid 3052:tid 1928] [client xxx.xxx.xxx.xxx:xxx] AH01786: uri mismatch - </tomcat_servlet/UploadServlet?swg_a2bc=1> does not match request-uri </tomcat_servlet/UploadServlet>
Indeed, cURL is adding some query it is getting from the proxy server in the authentication header.
Settings of my httpd
<Location /tomcat_servlet>
ProxyPass http://localhost:8080/tomcat_servlet
ProxyPassReverse http://localhost:8080/tomcat_servlet
AuthType Digest
AuthName https_transfer
AuthUserFile ${SRVROOT}/conf/.htpasswd
Require valid-user
</Location>
How do I use cURL in this situation? Or should I change some settings in the httpd side?

Related

PHP prevent header overwriting by Proxy

I want to access a PHP script hosted on dnsserver.icu via this proxy 207.154.231.211:8080 with curl.
The problem is that the Proxy server seems to do overwrite the HTTP 200 code with a 302 code making it impossible to reach the script.
curl -v dnsserver.icu gives the following output:
* Rebuilt URL to: dnsserver.icu/
* Trying 134.122.73.150...
* TCP_NODELAY set
* Connected to dnsserver.icu (134.122.73.150) port 80 (#0)
> GET / HTTP/1.1
> Host: dnsserver.icu
> User-Agent: curl/7.58.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Date: Wed, 15 Apr 2020 20:05:18 GMT
< Server: Apache/2.4.29 (Ubuntu)
< Content-Length: 31
< Content-Type: text/html; charset=UTF-8
<
* Connection #0 to host dnsserver.icu left intact
whereas curl -v -x 207.154.231.211:8080 dnsserver.icu gives the unexpected result of:
* Rebuilt URL to: dnsserver.icu/
* Trying 207.154.231.211...
* TCP_NODELAY set
* Connected to 207.154.231.211 (207.154.231.211) port 8080 (#0)
> GET http://dnsserver.icu/ HTTP/1.1
> Host: dnsserver.icu
> User-Agent: curl/7.58.0
> Accept: */*
> Proxy-Connection: Keep-Alive
>
< HTTP/1.1 302 Found
< Location: http://206.189.153.135
< Date: Wed, 15 Apr 2020 20:08:37 GMT
< Connection: keep-alive
< Transfer-Encoding: chunked
<
* Connection #0 to host 207.154.231.211 left intact
The address in the Location header is also changing sometimes.
I already experimented with different header configurations but I couldn't get it to work. When I log every call to the PHP script it doesn't look like the server is even reached by the proxy (no call logged). Futhermore the apache access log is empty.
Strangely this is not the case for all domains. I'm able to access e.g. ip-api.com, google.com or also less popular domains like proxyjudge.us (but not the ip equivalent of 45.33.35.141) through the proxy.
I have no idea what the reason for this behaviour is. Is there any 'trick' in terms of header setting or apache configuration that makes it possible to also access dnsserver.icu through this proxy? Something I havent tought of?
I appreciate any help.

Cannot able to access API in on-premise

I have installed Tyk( dashboard, gateway & pump) as a docker image on our local machine.
We have created API by ( System Management -> APIs -> Add New API) with below-mentioned configuration via Tyk Dashboard UI.
API-Name: My API
Listen Path: /test-api/
Target URL: http://httpbin.org/
Now the problem is that I am getting "Not Found" error when we access the API.
Could someone help me to resolve this issue?
Request: curl -X GET http://api-dashboard:3000/test-api/get -v
Response: 404 (Not Found)
Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to api-dashboard (127.0.0.1) port 3000 (#0)
> GET /test-api/get HTTP/1.1
> Host: api-dashboard:3000
> User-Agent: curl/7.58.0
> Accept: */*
>
< HTTP/1.1 404 Not Found
< Access-Control-Allow-Credentials: true
< Cache-Control: no-store, no-cache, private
< Strict-Transport-Security: max-age=63072000; includeSubDomains
< X-Content-Type-Options: nosniff
< X-Frame-Options: DENY
< Date: Wed, 24 Apr 2019 08:58:35 GMT
< Content-Length: 9
< Content-Type: text/plain; charset=utf-8
<
* Connection #0 to host api-dashboard left intact
You are calling the dashboard, you should be calling your gateway url instead.
E.g. http://api-gateway:8080/test-api/get
Tyk gateway default port is 8080.

Invalid client type getting verification code for limited input devices with google oauth

I'm trying to add google login for limited input devices as described here for a Web application type.
Using my client_id I cannot get the verification code because I keep getting this error:
$> curl -d "client_id=887293527777-tf5uf5q5skss8sbktp1vpo67p2v5b7i7.apps.googleusercontent.com&scope=email%20profile" https://accounts.google.com/o/oauth2/device/code
{
"error" : "invalid_client",
"error_description" : "Invalid client type."
}
And with verbose output:
$> curl -d "client_id=887293527777-tf5uf5q5skss8sbktp1vpo67p2v5b7i7.apps.googleusercontent.com&scope=email%20profile" https://accounts.google.com/o/oauth2/device/code -vvv
* Trying 209.85.203.84...
* TCP_NODELAY set
* Connected to accounts.google.com (209.85.203.84) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
* ALPN, server accepted to use h2
* SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate:
* subject: CN=accounts.google.com,O=Google Inc,L=Mountain View,ST=California,C=US
* start date: Feb 07 21:22:30 2018 GMT
* expire date: May 02 21:11:00 2018 GMT
* common name: accounts.google.com
* issuer: CN=Google Internet Authority G2,O=Google Inc,C=US
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x56450ea69cd0)
> POST /o/oauth2/device/code HTTP/1.1
> Host: accounts.google.com
> User-Agent: curl/7.51.0
> Accept: */*
> Content-Length: 104
> Content-Type: application/x-www-form-urlencoded
>
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
* We are completely uploaded and fine
< HTTP/2 401
< content-type: application/json; charset=utf-8
< x-content-type-options: nosniff
< cache-control: no-cache, no-store, max-age=0, must-revalidate
< pragma: no-cache
< expires: Mon, 01 Jan 1990 00:00:00 GMT
< date: Fri, 23 Feb 2018 14:29:41 GMT
< server: ESF
< x-xss-protection: 1; mode=block
< x-frame-options: SAMEORIGIN
< alt-svc: hq=":443"; ma=2592000; quic=51303431; quic=51303339; quic=51303338; quic=51303337; quic=51303335,quic=":443"; ma=2592000; v="41,39,38,37,35"
< accept-ranges: none
< vary: Accept-Encoding
<
{
"error" : "invalid_client",
"error_description" : "Invalid client type."
* Curl_http_done: called premature == 0
* Connection #0 to host accounts.google.com left intact
}
This is really annoying given they give a curl example on their guide. I've tried from javascript too but with no luck.
Edit: I can curl using Other as the type so I don't think the problem is on my side, but using Other is no good to me because I need to use Web Application in order to set CORS.
That flow is only supported for client type "TVs and Limited Input devices". See https://developers.google.com/identity/sign-in/devices#get_a_client_id_and_client_secret

Authenticate via api-key in chrome web store api

I would like to automate chrome extension deployment and followed this guide for the most part: https://developer.chrome.com/webstore/using_webstore_api. I created a project in my developer console, activated the Chrome Web Store API. Next step is creating the right credentials. There are three different ones, when i let google help me choose it tells me that API-key is just fine to use for this API, so i did.
My question: How can i use this API-key to make API calls? I tried to include the key in the params as pointed out in the docs but that doesn't work. If this is the wrong approach i really don't understand, that these sort of authentication is suggested by the developer console.
As i try to automate things on a build server without using chrome or some other browser i cannot really use OAuth credentials.
curl \
-H "x-goog-api-version: 2" \
-H "Content-Length: 0" \
-H "Expect:" \
-X GET \
-v \
"https://www.googleapis.com/chromewebstore/v1.1/items/ITEM_ID?projection=draft&key=API_KEY"
results in
https://www.googleapis.com/chromewebstore/v1.1/items/ITEM_ID\?projection\=draft\&key\=API_KEY
* Trying 172.217.17.234...
* Connected to www.googleapis.com (172.217.17.234) port 443 (#0)
* TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate: *.googleapis.com
* Server certificate: Google Internet Authority G2
* Server certificate: GeoTrust Global CA
> GET /chromewebstore/v1.1/items/ITEM_ID?projection=draft&key=API_KEY HTTP/1.1
> Host: www.googleapis.com
> User-Agent: curl/7.43.0
> Accept: */*
> x-goog-api-version: 2
> Content-Length: 0
>
< HTTP/1.1 401 Unauthorized
< Vary: X-Origin
< WWW-Authenticate: Bearer realm="https://accounts.google.com/"
< Content-Type: application/json; charset=UTF-8
< Date: Thu, 15 Sep 2016 14:14:18 GMT
< Expires: Thu, 15 Sep 2016 14:14:18 GMT
< Cache-Control: private, max-age=0
< X-Content-Type-Options: nosniff
< X-Frame-Options: SAMEORIGIN
< X-XSS-Protection: 1; mode=block
< Server: GSE
< Alt-Svc: quic=":443"; ma=2592000; v="36,35,34,33,32"
< Accept-Ranges: none
< Vary: Origin,Accept-Encoding
< Transfer-Encoding: chunked
<
* Connection #0 to host www.googleapis.com left intact
{"error":{"errors":[{"domain":"global","reason":"required","message":"Login Required","locationType":"header","location":"Authorization"}],"code":401,"message":"Login Required"}}
I also tried to set it as header but neither of these worked.

Apache2 proxypass does NOT work for POST request

I'm using Apache2 to route incoming requests to backend python flask web service and Splunk web service, there are running on the same server. I need to use POST to upload my file.
Here's my apache proxy setup:
ProxyPass /api/ http://10.68.57.166:5000/api/
ProxyPass / http://10.68.57.166:8000/
The test shows POST request has not been successfully routed, GET request is working fine although, the message says the method is not allowed, because my python flask only allows POST and OPTIONS.
Please help give me some thoughts of how to get it fixed ? very appreciated.
Gent79 tmp $ curl -v -X POST http://10.68.57.166/api/upload
Trying 10.68.57.166...
Connected to 10.68.57.166 (10.68.57.166) port 80 (#0)
> POST /api/upload HTTP/1.1
> Host: 10.68.57.166
> User-Agent: curl/7.45.0
> Accept: */*
>
< HTTP/1.1 400 BAD REQUEST
< Date: Fri, 15 Jul 2016 02:28:28 GMT
< Server: Werkzeug/0.9.6 Python/2.7.9
< Content-Type: text/html
< Content-Length: 192
< Connection: close
<
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<title>400 Bad Request</title>
<h1>Bad Request</h1>
<p>The browser (or proxy) sent a request that this server could not understand.</p>
* Closing connection 0
Gent79 tmp $ curl -v -X GET http://10.68.57.166/api/upload
Note: Unnecessary use of -X or --request, GET is already inferred.
* Trying 10.68.57.166...
* Connected to 10.68.57.166 (10.68.57.166) port 80 (#0)
> GET /api/upload HTTP/1.1
> Host: 10.68.57.166
> User-Agent: curl/7.45.0
> Accept: */*
>
< HTTP/1.1 405 METHOD NOT ALLOWED
< Date: Fri, 15 Jul 2016 02:28:41 GMT
< Server: Werkzeug/0.9.6 Python/2.7.9
< Content-Type: text/html
< Allow: POST, OPTIONS
< Content-Length: 178
<
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<title>405 Method Not Allowed</title>
<h1>Method Not Allowed</h1>
<p>The method is not allowed for the requested URL.</p>
* Connection #0 to host 10.68.57.166 left intact*