HTTP keep-alive: when client should stop reusing connection? - apache

I understand that the client should stop reusing http keepalive socket after timeout or the max usage. However two popular server implementations seem to keep socket open for the longer duration than the timeout and hence this question to learn what strategies a client should adapt to optimally reuse the connection.
We were testing our implementation with various servers, and while apache seems to be adhering to timeout, lighttpd and nginx seem to keep connection open for longer duration than the timeout. As you can see from the logs below, despite of client-server agreed on 15 second timeout, the socket was kept open for more than 60 seconds and second request succeeded even after 60 seconds.
1) First request and response with timestamp
GET /en/CHANGES HTTP/1.1
Host: nginx.org
Accept: */*
Accept-Encoding: identity
Content-Length: 0
Connection: Keep-Alive
Keep-Alive: timeout=15, max=1000
E(2410-130624-216): HTTP/1.1 200 OK
E(2410-130624-216): Server: nginx/1.13.3
E(2410-130624-216): Date: Tue, 24 Oct 2017 07:36:23 GMT
E(2410-130624-216): Content-Type: text/plain; charset=utf-8
E(2410-130624-216): Content-Length: 282456
E(2410-130624-216): Last-Modified: Tue, 10 Oct 2017 15:39:18 GMT
E(2410-130624-216): Connection: keep-alive
E(2410-130624-216): Keep-Alive: timeout=15
E(2410-130624-216): ETag: "59dce9a6-44f58"
E(2410-130624-216): Accept-Ranges: bytes
E(2410-130624-216):
2) Second request on the same socket, 60 seconds later
GET /en/CHANGES HTTP/1.1
Host: nginx.org
Accept: */*
Accept-Encoding: identity
Content-Length: 0
Connection: Keep-Alive
Keep-Alive: timeout=15, max=1000
E(2410-130726-881): HTTP/1.1 200 OK
E(2410-130726-881): Server: nginx/1.13.3
E(2410-130726-881): Date: Tue, 24 Oct 2017 07:37:26 GMT
E(2410-130726-881): Content-Type: text/plain; charset=utf-8
E(2410-130726-881): Content-Length: 282456
E(2410-130726-881): Last-Modified: Tue, 10 Oct 2017 15:39:18 GMT
E(2410-130726-881): Connection: keep-alive
E(2410-130726-881): Keep-Alive: timeout=15
E(2410-130726-881): ETag: "59dce9a6-44f58"
E(2410-130726-881): Accept-Ranges: bytes
E(2410-130726-881):
There comes the question, when should client stop reusing connection in the optimal manner. Should client keep reusing the connection till it gives error? or should we discard the connection after timeout? What strategies browsers are using?
Thanks

Related

Why can't googlebot fetch my JavaScript file?

https://api-staging-weld.freetls.fastly.net/scripts/customdomain_weld.19f3e9ec.js
The error I get in Google Search Console is Temporarily unreachable with no further explanation. I think it's something with my http headers but can't figure out which one. Here they are:
accept-ranges: bytes
access-control-allow-origin: *
age: 4905
cache-control: public, max-age=31536000
content-encoding: gzip
content-length: 29990
content-type: application/javascript; charset=UTF-8
date: Fri, 20 Apr 2018 12:46:37 GMT
etag: W/"1a018-162e269bfe0"
last-modified: Fri, 20 Apr 2018 09:36:44 GMT
server: Cowboy
status: 200
vary: Accept-Encoding
via: 1.1 vegur
via: 1.1 varnish
x-cache: HIT
x-cache-hits: 1
x-powered-by: Express
x-served-by: cache-bma7030-BMA
x-timer: S1524228398.925110,VS0,VE6
What's wrong or what am i Missing?

Why does the web server sent the file instead of a 304 http: not modified?

My browser send to the server the following request:
Host: www.imprimante.be
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:38.0) Gecko/20100101 Firefox/38.0
Accept: */*
Accept-Language: fr,fr-FR;q=0.8,en-US;q=0.5,en;q=0.3
Accept-Encoding: gzip, deflate
If-Modified-Since: Fri, 29 May 2015 14:22:44 GMT
If-None-Match: "90-5173935ad3a1a-gzip"
Referer: http://www.imprimante.be/premier-avis-gratuit/
Cookie: <hidden>
Connection: keep-alive
The url used is http://www.imprimante.be/wp-content/themes/mch_imprimante/js/theme.min.js? (note: www.imprimante.be is not accessible trough wlan yet)
And the server send me the file with this (status 200) http header:
Accept-Ranges: bytes
Connection: Keep-Alive
Content-Encoding: gzip
Content-Length: 137
Content-Type: application/javascript
Date: Wed, 03 Jun 2015 07:18:03 GMT
Etag: "90-5173935ad3a1a-gzip"
Keep-Alive: timeout=5, max=99
Last-Modified: Fri, 29 May 2015 14:22:44 GMT
Server: Apache/2.4.10 (Debian)
Vary: Accept-Encoding
As you might notice (Last-Modified: Fri, 29 May 2015 14:22:44 GMT) the file hasn't been modified since the last request.
So I don't get why the response isn't a 304 status: not modified.
I'd really like to know why the caching of this files (and some others) doesn't work as I expect it.
It is bug in Apache. Turn off mod_deflate.

How to configure Apache to send gzipped and chunked encoded response for static content?

For the purpose of unit testing, I want to configure Apache to gzip a static file and then chunk encode it and send it as a response. I tried following so far:
# Created a file httpd-deflate.conf
# Set to gzip all output
SetOutputFilter DEFLATE
#set compression level
DeflateCompressionLevel 9
# Deflate in chunks
DeflateBufferSize 100
Included this file in httpd.conf
Include httpd-deflate.conf
Send request as follows:
GET / HTTP/1.1
Host: test-server
Accept-Encoding: gzip
Connection: Keep-Alive
Got following response:
HTTP/1.1 200 OK
Date: Wed, 05 Nov 2014 19:24:23 GMT
Server: Apache/2.2.15 (Scientific Linux)
Last-Modified: Wed, 05 Nov 2014 18:17:21 GMT
ETag: "5e28c-5df-5072097390e40"
Accept-Ranges: bytes
Vary: Accept-Encoding
Content-Encoding: gzip
Content-Length: 1186
Connection: close
Content-Type: text/html; charset=UTF-8
<binary-data>
Is there a way by which Apache can start sending the gziped data in chunked format. I thought DeflateBufferSize can help me do that. However, no luck so far.
Any suggestions?
Thanks.

Apache truncates HTTP header

I'm experimenting a very strange issue: using apache-2.2.25 on an Arch Linux server I get a truncated HTTP header in response, but only under windows machines.
I left untouched the httpd.conf file and I have created a file /srv/http/index.html with inside Hello.
With a web browser on a Linux machine I get correctly visualized the hello page, but under Windows I get
.1 200 OK Date: Fri, 15 Nov 2013 08:44:28 GMT Server: Apache/2.2.25 (Unix) mod_ssl/2.2.25 OpenSSL/1.0.1e DAV/2 Last-Modified: Fri, 15 Nov 2013 08:15:09 GMT ETag: "27c-a-4eb32cbf8b950" Accept-Ranges: bytes Content-Length: 10 Keep-Alive: timeout=5, max=100 Connection: Keep-Alive Content-Type: text/html; charset=utf-8 Hello
It seems that the http header is truncated for the first characters (HTTP/1).
Have you got any advices?

HTTP pipelining request text example

Below is an example HTTP 1.1 call with a single page requested :
GET /jq.js HTTP/1.1
Host: 127.0.0.1
Accept: */*
I understand with HTTP Pipelining, multiple requests can be sent without breaking the connection.
Can someone post, some text example of how this request will be sent to the server, I want to be able to do it over the command line or with PHP sockets.
Does support for pipelining need to enabled on the web-server as well?
Is pipelining supported by major Web-servers(apache, nginx) by default or does it need to be enabled
From w3c protocol details:
8.1.2.2 Pipelining
A client that supports persistent connections MAY "pipeline" its requests (i.e., send multiple requests without waiting for each response). A server MUST send its responses to those requests in the same order that the requests were received.
Clients which assume persistent connections and pipeline immediately after connection establishment SHOULD be prepared to retry their connection if the first pipelined attempt fails. If a client does such a retry, it MUST NOT pipeline before it knows the connection is persistent. Clients MUST also be prepared to resend their requests if the server closes the connection before sending all of the corresponding responses.
Clients SHOULD NOT pipeline requests using non-idempotent methods or non-idempotent sequences of methods (see section 9.1.2). Otherwise, a premature termination of the transport connection could lead to indeterminate results. A client wishing to send a non-idempotent request SHOULD wait to send that request until it has received the response status for the previous request.
So, first fact is that you should be in a KeepAlive status. So you should add Connection: keep-alive keyword in your request headers, but some webservers may still accept pipelining without this keep alive status. On the other hand, this could be rejected by the server, the server may or may not accept your connection in keepalive mode. So, at any time, being in keepalived or not, you may send 3 requests pipelined in one connection, and get only one response.
From this gist we can find a nice way to test it with telnet.
Asking for keepalive with Connection: keep-alive header:
(echo -en "GET /index.html HTTP/1.1\nHost: foo.com\nConnection: keep-alive\n\nGET /index.html HTTP/1.1\nHost: foo.com\n\n"; sleep 10) | telnet localhost 80
Trying 127.0.0.1...
Connected to localhost.lan.
Escape character is '^]'.
HTTP/1.1 200 OK
Date: Sun, 27 Oct 2013 17:51:58 GMT
Server: Apache/2.2.22 (Debian)
Last-Modified: Sun, 04 Mar 2012 15:00:29 GMT
ETag: "56176e-3e-4ba6c121c4761"
Accept-Ranges: bytes
Content-Length: 62
Vary: Accept-Encoding
Keep-Alive: timeout=5, max=100 <======= Keepalive!
Connection: Keep-Alive
Content-Type: text/html; charset=utf-8
<html>
<body>
<h1>test</h1>
</body>
</html>
HTTP/1.1 200 OK
Date: Sun, 27 Oct 2013 17:51:58 GMT
Server: Apache/2.2.22 (Debian)
Last-Modified: Sun, 04 Mar 2012 15:00:29 GMT
ETag: "56176e-3e-4ba6c121c4761"
Accept-Ranges: bytes
Content-Length: 62
Vary: Accept-Encoding
Content-Type: text/html; charset=utf-8
<html>
<body>
<h1>test</h1>
</body>
</html>
It works.
Without asking for Keepalive:
(echo -en "GET /index.html HTTP/1.1\nHost: foo.com\nConnection: keep-alive\n\nGET /index.html HTTP/1.1\nHost: foo.com\n\n"; sleep 10) | telnet localhost 80
Trying 127.0.0.1...
Connected to localhost.lan.
Escape character is '^]'.
HTTP/1.1 200 OK
Date: Sun, 27 Oct 2013 17:49:37 GMT
Server: Apache/2.2.22 (Debian)
Last-Modified: Sun, 04 Mar 2012 15:00:29 GMT
ETag: "56176e-3e-4ba6c121c4761"
Accept-Ranges: bytes
Content-Length: 62
Vary: Accept-Encoding
Content-Type: text/html; charset=utf-8
<html>
<body>
<h1>test</h1>
</body>
</html>
HTTP/1.1 200 OK
Date: Sun, 27 Oct 2013 17:49:37 GMT
Server: Apache/2.2.22 (Debian)
Last-Modified: Sun, 04 Mar 2012 15:00:29 GMT
ETag: "56176e-3e-4ba6c121c4761"
Accept-Ranges: bytes
Content-Length: 62
Vary: Accept-Encoding
Content-Type: text/html; charset=utf-8
<html>
<body>
<h1>test</h1>
</body>
</html>
Connection closed by foreign host.
Same result, I did not ask for it but it looks like a Keepalive answer (closing after 5s which is the value set in Apache). And a pipelined answer, I get my two pages.
Now if I prevent usage of any Keepalive connection in Apache by setting:
Keepalive Off
And restarting it:
(echo -en "GET /index.html HTTP/1.1\nHost: foo.com\nConnection: keep-alive\n\nGET /index.html HTTP/1.1\nHost: foo.com\n\n"; sleep 10) | telnet localhost 80
Trying 127.0.0.1...
Connected to localhost.lan.
Escape character is '^]'.
HTTP/1.1 200 OK
Date: Sun, 27 Oct 2013 18:02:41 GMT
Server: Apache/2.2.22 (Debian)
Last-Modified: Sun, 04 Mar 2012 15:00:29 GMT
ETag: "56176e-3e-4ba6c121c4761"
Accept-Ranges: bytes
Content-Length: 62
Vary: Accept-Encoding
Connection: close
Content-Type: text/html; charset=utf-8
<html>
<body>
<h1>test</h1>
</body>
</html>
Connection closed by foreign host.
Only one answer... So the server can reject my request for pipelining.
Now, for support on servers and browsers, I think your wikipedia source tells enough :-)