Download only the HTTP headers in Cocoa - objective-c

I'm trying to port one of my Android applications to work natively on Mac OS X.
For the initialisation of the application, it needs to connect to a server and read only the headers of the server's response. The server (3rd party server) will respond with 82274 bytes of data, but the only useful data to me are the headers; specifically I only need to read the session cookie and retrieve its value. This means that all of the other data is redundant.
Through Googling, the only working looking response is as follows:
// Create the request.
NSMutableURLRequest *theRequest = [NSMutableURLRequest requestWithURL:[NSURL URLWithString:#"http://www.grooveshark.com/"]];
[theRequest setHTTPMethod:#"HEAD"];
[theRequest setValue:#"MySpecialUserAgent/1.0" forHTTPHeaderField:#"User-Agent"];
[theRequest setTimeoutInterval:15.0];
[theRequest setCachePolicy:NSURLRequestReloadIgnoringCacheData];
However this still downloads the entire page.
Can anyone point me in the right direction?

Let's see what happens if we hit that URL.
› curl -v -X HEAD http://www.grooveshark.com
* About to connect() to www.grooveshark.com port 80 (#0)
* Trying 8.20.213.76...
* connected
* Connected to www.grooveshark.com (8.20.213.76) port 80 (#0)
> HEAD / HTTP/1.1
> User-Agent: curl/7.24.0 (x86_64-apple-darwin12.0) libcurl/7.24.0 OpenSSL/0.9.8r zlib/1.2.5
> Host: www.grooveshark.com
> Accept: */*
>
< HTTP/1.1 301 Moved Permanently
< Server: richhickey
< Date: Sat, 15 Dec 2012 20:27:38 GMT
< Content-Type: text/html; charset=UTF-8
< Connection: close
< Location: http://grooveshark.com
< Vary: Accept-Encoding
< X-Hostname: rhl081
< X-Hostname: rhl081
<
* Closing connection #0
So www.grooveshark.com redirects to grooveshark.com. Let's see if that page honors HEAD requests correctly.
› curl -v -X HEAD http://grooveshark.com
* About to connect() to grooveshark.com port 80 (#0)
* Trying 8.20.213.76...
* connected
* Connected to grooveshark.com (8.20.213.76) port 80 (#0)
> HEAD / HTTP/1.1
> User-Agent: curl/7.24.0 (x86_64-apple-darwin12.0) libcurl/7.24.0 OpenSSL/0.9.8r zlib/1.2.5
> Host: grooveshark.com
> Accept: */*
>
< HTTP/1.1 200 OK
< Server: richhickey
< Date: Sat, 15 Dec 2012 20:28:06 GMT
< Content-Type: text/html; charset=UTF-8
< Connection: close
< Vary: Accept-Encoding
< Set-Cookie: PHPSESSID=844a5e6bdd6d84a97afd8f42faf4eb95; expires=Sat, 22-Dec-2012 20:28:06 GMT; path=/; domain=.grooveshark.com
< Expires: Thu, 19 Nov 1981 08:52:00 GMT
< Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
< Pragma: no-cache
< Vary: Accept-Encoding
< X-Hostname: rhl061
< Set-Cookie: ismobile=no;domain=.grooveshark.com;path=/
< X-country: US
<
* Closing connection #0
That looks good. I suspect that your request is falling back to a GET when following that redirect. It looks like Chris Suter ran into the same thing and gave an example solution: http://sutes.co.uk/2009/12/nsurlconnection-using-head-met.html
In the future you might want to try running your requests through a local proxy so that you can see them in flight. That would probably reveal that you make a HEAD request to www.grooveshark.com followed by a GET to grooveshark.com.

Related

curl changes the URI in the authorization header for digest behind proxy

The bounty expires in 4 days. Answers to this question are eligible for a +50 reputation bounty.
Mirza Prangon is looking for an answer from a reputable source:
Details how what is going wrong, where it is going wrong and how to fix it.
I am trying to use curl for a http request.
I have to use it behind a enterprise proxy server. The remote host uses digest authentication.
I am using the following curl command.
curl -x "http://proxy_username:proxy_pass#proxyIp.xxx.xxx.xxx:8080" -L -X GET "https://remote-host.something.com:443/tomcat_servlet/UploadServlet" --digest -u digest_auth_user:digest_auth_pass -v -k
But I get 400 bad request from apache httpd. The full output from curl is
* Trying proxyIp.xxx.xxx.xxx:8080...
* Connected to proxyIp.xxx.xxx.xxx (proxyIp.xxx.xxx.xxx) port 8080 (#0)
* allocate connect buffer
* Establish HTTP proxy tunnel to remote-host.something.com:443
* Proxy auth using Basic with user 'proxy_username'
* Server auth using Digest with user 'digest_auth_user'
> CONNECT remote-host.something.com:443 HTTP/1.1
> Host: remote-host.something.com:443
> Proxy-Authorization: Basic <redacted>
> User-Agent: curl/7.83.1
> Proxy-Connection: Keep-Alive
>
< HTTP/1.1 200 Connection established
< Via:HTTP/1.1 s_proxy_nrt
<
* Proxy replied 200 to CONNECT request
* CONNECT phase completed
* schannel: disabled automatic use of client certificate
* ALPN: offers http/1.1
* ALPN: server did not agree on a protocol. Uses default.
* Server auth using Digest with user 'digest_auth_user'
> GET /tomcat_servlet/UploadServlet HTTP/1.1
> Host: remote-host.something.com
> User-Agent: curl/7.83.1
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 307 Temporary Redirect
< Server: Cisco Umbrella
< Date: Tue, 14 Feb 2023 02:52:03 GMT
< Content-Type: text/html
< Content-Length: 190
< Connection: keep-alive
< Set-Cookie: swg_https_a2bc=1; Path=/; Expires=Tue, 14-Feb-23 03:02:03 GMT; domain=remote-host.something.com; SameSite=None; Secure
< Location: https://remote-host.something.com/tomcat_servlet/UploadServlet?swg_a2bc=1
< Via: HTTP/1.1 s_proxy_nrt
<
* Ignoring the response-body
* Connection #0 to host proxyIp.xxx.xxx.xxx left intact
* Issue another request to this URL: 'https://remote-host.something.com/tomcat_servlet/UploadServlet?swg_a2bc=1'
* Found bundle for host: 0x1a0ed47d970 [serially]
* Re-using existing connection #0 with proxy proxyIp.xxx.xxx.xxx
* Connected to proxyIp.xxx.xxx.xxx (proxyIp.xxx.xxx.xxx) port 8080 (#0)
* Server auth using Digest with user 'digest_auth_user'
> GET /tomcat_servlet/UploadServlet?swg_a2bc=1 HTTP/1.1
> Host: remote-host.something.com
> User-Agent: curl/7.83.1
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 401 Unauthorized
< Date: Tue, 14 Feb 2023 02:52:03 GMT
< Content-Type: text/html; charset=iso-8859-1
< Content-Length: 381
< Connection: keep-alive
< Server: Apache/2.4.48 (Win64) OpenSSL/1.1.1k
< WWW-Authenticate: Digest realm="https_transfer", nonce="redacted", algorithm=MD5, qop="auth"
< Via: HTTP/1.1 m_proxy_nrt
<
* Ignoring the response-body
* Connection #0 to host proxyIp.xxx.xxx.xxx left intact
* Issue another request to this URL: 'https://remote-host.something.com/tomcat_servlet/UploadServlet?swg_a2bc=1'
* Found bundle for host: 0x1a0ed47d970 [serially]
* Re-using existing connection #0 with proxy proxyIp.xxx.xxx.xxx
* Connected to proxyIp.xxx.xxx.xxx (proxyIp.xxx.xxx.xxx) port 8080 (#0)
* Server auth using Digest with user 'digest_auth_user'
> GET /tomcat_servlet/UploadServlet?swg_a2bc=1 HTTP/1.1
> Host: remote-host.something.com
> Authorization: Digest username="digest_auth_user",realm="https_transfer",nonce="redacted",uri="/tomcat_servlet/UploadServlet?swg_a2bc=1",cnonce="redacted",nc=00000001,algorithm=MD5,response="redacted",qop="redacted"
> User-Agent: curl/7.83.1
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 400 Bad Request
< Date: Tue, 14 Feb 2023 02:52:03 GMT
< Content-Type: text/html; charset=iso-8859-1
< Content-Length: 226
< Connection: keep-alive
< Server: Apache/2.4.48 (Win64) OpenSSL/1.1.1k
< Via: HTTP/1.1 m_proxy_nrt
<
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>400 Bad Request</title>
</head><body>
<h1>Bad Request</h1>
<p>Your browser sent a request that this server could not understand.<br />
</p>
</body></html>
* Connection #0 to host proxyIp.xxx.xxx.xxx left intact
Is the server side, I get the following in httpd log.
[auth_digest:error] [pid 3052:tid 1928] [client xxx.xxx.xxx.xxx:xxx] AH01786: uri mismatch - </tomcat_servlet/UploadServlet?swg_a2bc=1> does not match request-uri </tomcat_servlet/UploadServlet>
Indeed, cURL is adding some query it is getting from the proxy server in the authentication header.
Settings of my httpd
<Location /tomcat_servlet>
ProxyPass http://localhost:8080/tomcat_servlet
ProxyPassReverse http://localhost:8080/tomcat_servlet
AuthType Digest
AuthName https_transfer
AuthUserFile ${SRVROOT}/conf/.htpasswd
Require valid-user
</Location>
How do I use cURL in this situation? Or should I change some settings in the httpd side?

PHP prevent header overwriting by Proxy

I want to access a PHP script hosted on dnsserver.icu via this proxy 207.154.231.211:8080 with curl.
The problem is that the Proxy server seems to do overwrite the HTTP 200 code with a 302 code making it impossible to reach the script.
curl -v dnsserver.icu gives the following output:
* Rebuilt URL to: dnsserver.icu/
* Trying 134.122.73.150...
* TCP_NODELAY set
* Connected to dnsserver.icu (134.122.73.150) port 80 (#0)
> GET / HTTP/1.1
> Host: dnsserver.icu
> User-Agent: curl/7.58.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Date: Wed, 15 Apr 2020 20:05:18 GMT
< Server: Apache/2.4.29 (Ubuntu)
< Content-Length: 31
< Content-Type: text/html; charset=UTF-8
<
* Connection #0 to host dnsserver.icu left intact
whereas curl -v -x 207.154.231.211:8080 dnsserver.icu gives the unexpected result of:
* Rebuilt URL to: dnsserver.icu/
* Trying 207.154.231.211...
* TCP_NODELAY set
* Connected to 207.154.231.211 (207.154.231.211) port 8080 (#0)
> GET http://dnsserver.icu/ HTTP/1.1
> Host: dnsserver.icu
> User-Agent: curl/7.58.0
> Accept: */*
> Proxy-Connection: Keep-Alive
>
< HTTP/1.1 302 Found
< Location: http://206.189.153.135
< Date: Wed, 15 Apr 2020 20:08:37 GMT
< Connection: keep-alive
< Transfer-Encoding: chunked
<
* Connection #0 to host 207.154.231.211 left intact
The address in the Location header is also changing sometimes.
I already experimented with different header configurations but I couldn't get it to work. When I log every call to the PHP script it doesn't look like the server is even reached by the proxy (no call logged). Futhermore the apache access log is empty.
Strangely this is not the case for all domains. I'm able to access e.g. ip-api.com, google.com or also less popular domains like proxyjudge.us (but not the ip equivalent of 45.33.35.141) through the proxy.
I have no idea what the reason for this behaviour is. Is there any 'trick' in terms of header setting or apache configuration that makes it possible to also access dnsserver.icu through this proxy? Something I havent tought of?
I appreciate any help.

Cannot able to access API in on-premise

I have installed Tyk( dashboard, gateway & pump) as a docker image on our local machine.
We have created API by ( System Management -> APIs -> Add New API) with below-mentioned configuration via Tyk Dashboard UI.
API-Name: My API
Listen Path: /test-api/
Target URL: http://httpbin.org/
Now the problem is that I am getting "Not Found" error when we access the API.
Could someone help me to resolve this issue?
Request: curl -X GET http://api-dashboard:3000/test-api/get -v
Response: 404 (Not Found)
Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to api-dashboard (127.0.0.1) port 3000 (#0)
> GET /test-api/get HTTP/1.1
> Host: api-dashboard:3000
> User-Agent: curl/7.58.0
> Accept: */*
>
< HTTP/1.1 404 Not Found
< Access-Control-Allow-Credentials: true
< Cache-Control: no-store, no-cache, private
< Strict-Transport-Security: max-age=63072000; includeSubDomains
< X-Content-Type-Options: nosniff
< X-Frame-Options: DENY
< Date: Wed, 24 Apr 2019 08:58:35 GMT
< Content-Length: 9
< Content-Type: text/plain; charset=utf-8
<
* Connection #0 to host api-dashboard left intact
You are calling the dashboard, you should be calling your gateway url instead.
E.g. http://api-gateway:8080/test-api/get
Tyk gateway default port is 8080.

will Accept Http Headers digest any other formats other than the one specified?

Per my understanding:
the Accept header is used by HTTP clients to tell the server what content types they'll accept. The server will then send back a response, which will include a Content-Type header telling the client what the content type of the returned content actually is.
With this understanding, I tried the following:
curl -X GET -H "Accept: application/xml" http://www.google.com -v
* About to connect() to www.google.com port 80 (#0)
* Trying 173.194.33.81...
* connected
* Connected to www.google.com (173.194.33.81) port 80 (#0)
> GET / HTTP/1.1
> User-Agent: curl/7.24.0 (x86_64-apple-darwin12.0) libcurl/7.24.0 OpenSSL/0.9.8y zlib/1.2.5
> Host: www.google.com
> Accept: application/xml
>
< HTTP/1.1 200 OK
< Date: Tue, 02 Sep 2014 17:58:05 GMT
< Expires: -1
< Cache-Control: private, max-age=0
< Content-Type: text/html; charset=ISO-8859-1
< Set-Cookie: PREF=ID=5c30672b67a74789:FF=0:TM=1409680685:LM=1409680685:S=PsGclk3vR4HWjann; expires=Thu, 01-Sep-2016 17:58:05 GMT; path=/; domain=.google.com
< Set-Cookie: NID=67=rPuxpwUu5UNuapzCdbD5iwVyjjC9TzP_Ado29h3ucjEq4A_2qkSM4nQM3RO02rfyuHmrh-hvmwmgFCmOvISttFfHv06f8ay4_6Gl4pXRjqxihNhJSGbvujjDRzaSibfy; expires=Wed, 04-Mar-2015 17:58:05 GMT; path=/; domain=.google.com; HttpOnly
< P3P: CP="This is not a P3P policy! See http://www.google.com/support/accounts/bin/answer.py?hl=en&answer=151657 for more info."
< Server: gws
< X-XSS-Protection: 1; mode=block
< X-Frame-Options: SAMEORIGIN
< Alternate-Protocol: 80:quic
< Transfer-Encoding: chunked
<
<!doctype html><html itemscope="" itemtype="http://schema.org/WebPage" lang="en"><
As you can notice in the response, I am sent Content-Type: text/html; charset=ISO-8859-1 which is not what I asked for?
why does a different representation (HTML in this case) is sent, although I asked for xml?
Thanks
From RFC 2616:
If an Accept header field is present,
and if the server cannot send a response which is acceptable
according to the combined Accept field value, then the server SHOULD
send a 406 (not acceptable) response.
Here, "should" means that Google aren't actually obliged to throw a 406 error. But since you're receiving an HTML response, it has effectively the same meaning.

How do I force apache to deliver a file in chunked encoded format

I am verifying if my application handles file content delivered through chunked-encoding mode. I am not sure what change to make to the httpd.conf file to force chunked encoding through Apache. Is it even possible to do this with Apache server, if not what would be an easier solution? I am using Apache 2.4.2 and HTTP 1.1.
By default, keep-alive is On in Apache and I do not see the data as chunked when testing with wireshark.
EDIT: Added more info:
Only way I managed to do this was by enabling the deflate module.
Then I configured my client to send "Accept-Encoding: gzip, deflate" header and apache would compress and send the file back in chunked mode.
I had to enable the file type in the module though.
AddOutputFilterByType DEFLATE image/png
See example:
curl --raw -v --header "Accept-Encoding: gzip, deflate" http://localhost/image.png | more
* Connected to localhost (127.0.0.1) port 80 (#0)
> GET /image.png HTTP/1.1
> User-Agent: curl/7.35.0
> Host: localhost
> Accept: */*
> Accept-Encoding: gzip, deflate
>
< HTTP/1.1 200 OK
< Date: Mon, 13 Apr 2015 10:08:45 GMT
* Server Apache/2.4.7 (Ubuntu) is not blacklisted
< Server: Apache/2.4.7 (Ubuntu)
< Last-Modified: Mon, 13 Apr 2015 09:48:53 GMT
< ETag: "3b5306-5139805976dae-gzip"
< Accept-Ranges: bytes
< Vary: Accept-Encoding
< Content-Encoding: gzip
< Transfer-Encoding: chunked
< Content-Type: image/png
<
This resource produces chunked results http://www.httpwatch.com/httpgallery/chunked/
which is very useful for testing clients. You can see this by running
$ curl --raw -i http://www.httpwatch.com/httpgallery/chunked/
HTTP/1.1 200 OK
Cache-Control: private,Public
Transfer-Encoding: chunked
Content-Type: text/html
Server: Microsoft-IIS/7.5
X-AspNet-Version: 4.0.30319
X-Powered-By: ASP.NET
Date: Mon, 22 Jul 2013 09:41:04 GMT
7b
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
2d
<html xmlns="http://www.w3.org/1999/xhtml">
....
I tried this way to get HTTP chunked encoded data in Ubuntu, it might help.
In apache server create a file index.php in your directory where index page is there ( ex : /var/www/html/) and paste below content (should have php installed):
<?php phpinfo(); ?>
Then try to curl the page as below :
root#ubuntu-16:~# curl -v http://10.11.0.230:2222/index.php
* Trying 10.11.0.230...
* Connected to 10.11.0.230 (10.11.0.230) port 2222 (#0)
> GET /index.php HTTP/1.1
> Host: 10.11.0.230:2222
> User-Agent: curl/7.47.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Date: Wed, 01 Jul 2020 07:51:24 GMT
< Server: Apache/2.4.18 (Ubuntu)
< Vary: Accept-Encoding
< Transfer-Encoding: chunked
< Content-Type: text/html; charset=UTF-8
<
<!DOCTYPE html>
<html>
<body>
...
...
...