Below is an example HTTP 1.1 call with a single page requested :
GET /jq.js HTTP/1.1
Host: 127.0.0.1
Accept: */*
I understand with HTTP Pipelining, multiple requests can be sent without breaking the connection.
Can someone post, some text example of how this request will be sent to the server, I want to be able to do it over the command line or with PHP sockets.
Does support for pipelining need to enabled on the web-server as well?
Is pipelining supported by major Web-servers(apache, nginx) by default or does it need to be enabled
From w3c protocol details:
8.1.2.2 Pipelining
A client that supports persistent connections MAY "pipeline" its requests (i.e., send multiple requests without waiting for each response). A server MUST send its responses to those requests in the same order that the requests were received.
Clients which assume persistent connections and pipeline immediately after connection establishment SHOULD be prepared to retry their connection if the first pipelined attempt fails. If a client does such a retry, it MUST NOT pipeline before it knows the connection is persistent. Clients MUST also be prepared to resend their requests if the server closes the connection before sending all of the corresponding responses.
Clients SHOULD NOT pipeline requests using non-idempotent methods or non-idempotent sequences of methods (see section 9.1.2). Otherwise, a premature termination of the transport connection could lead to indeterminate results. A client wishing to send a non-idempotent request SHOULD wait to send that request until it has received the response status for the previous request.
So, first fact is that you should be in a KeepAlive status. So you should add Connection: keep-alive keyword in your request headers, but some webservers may still accept pipelining without this keep alive status. On the other hand, this could be rejected by the server, the server may or may not accept your connection in keepalive mode. So, at any time, being in keepalived or not, you may send 3 requests pipelined in one connection, and get only one response.
From this gist we can find a nice way to test it with telnet.
Asking for keepalive with Connection: keep-alive header:
(echo -en "GET /index.html HTTP/1.1\nHost: foo.com\nConnection: keep-alive\n\nGET /index.html HTTP/1.1\nHost: foo.com\n\n"; sleep 10) | telnet localhost 80
Trying 127.0.0.1...
Connected to localhost.lan.
Escape character is '^]'.
HTTP/1.1 200 OK
Date: Sun, 27 Oct 2013 17:51:58 GMT
Server: Apache/2.2.22 (Debian)
Last-Modified: Sun, 04 Mar 2012 15:00:29 GMT
ETag: "56176e-3e-4ba6c121c4761"
Accept-Ranges: bytes
Content-Length: 62
Vary: Accept-Encoding
Keep-Alive: timeout=5, max=100 <======= Keepalive!
Connection: Keep-Alive
Content-Type: text/html; charset=utf-8
<html>
<body>
<h1>test</h1>
</body>
</html>
HTTP/1.1 200 OK
Date: Sun, 27 Oct 2013 17:51:58 GMT
Server: Apache/2.2.22 (Debian)
Last-Modified: Sun, 04 Mar 2012 15:00:29 GMT
ETag: "56176e-3e-4ba6c121c4761"
Accept-Ranges: bytes
Content-Length: 62
Vary: Accept-Encoding
Content-Type: text/html; charset=utf-8
<html>
<body>
<h1>test</h1>
</body>
</html>
It works.
Without asking for Keepalive:
(echo -en "GET /index.html HTTP/1.1\nHost: foo.com\nConnection: keep-alive\n\nGET /index.html HTTP/1.1\nHost: foo.com\n\n"; sleep 10) | telnet localhost 80
Trying 127.0.0.1...
Connected to localhost.lan.
Escape character is '^]'.
HTTP/1.1 200 OK
Date: Sun, 27 Oct 2013 17:49:37 GMT
Server: Apache/2.2.22 (Debian)
Last-Modified: Sun, 04 Mar 2012 15:00:29 GMT
ETag: "56176e-3e-4ba6c121c4761"
Accept-Ranges: bytes
Content-Length: 62
Vary: Accept-Encoding
Content-Type: text/html; charset=utf-8
<html>
<body>
<h1>test</h1>
</body>
</html>
HTTP/1.1 200 OK
Date: Sun, 27 Oct 2013 17:49:37 GMT
Server: Apache/2.2.22 (Debian)
Last-Modified: Sun, 04 Mar 2012 15:00:29 GMT
ETag: "56176e-3e-4ba6c121c4761"
Accept-Ranges: bytes
Content-Length: 62
Vary: Accept-Encoding
Content-Type: text/html; charset=utf-8
<html>
<body>
<h1>test</h1>
</body>
</html>
Connection closed by foreign host.
Same result, I did not ask for it but it looks like a Keepalive answer (closing after 5s which is the value set in Apache). And a pipelined answer, I get my two pages.
Now if I prevent usage of any Keepalive connection in Apache by setting:
Keepalive Off
And restarting it:
(echo -en "GET /index.html HTTP/1.1\nHost: foo.com\nConnection: keep-alive\n\nGET /index.html HTTP/1.1\nHost: foo.com\n\n"; sleep 10) | telnet localhost 80
Trying 127.0.0.1...
Connected to localhost.lan.
Escape character is '^]'.
HTTP/1.1 200 OK
Date: Sun, 27 Oct 2013 18:02:41 GMT
Server: Apache/2.2.22 (Debian)
Last-Modified: Sun, 04 Mar 2012 15:00:29 GMT
ETag: "56176e-3e-4ba6c121c4761"
Accept-Ranges: bytes
Content-Length: 62
Vary: Accept-Encoding
Connection: close
Content-Type: text/html; charset=utf-8
<html>
<body>
<h1>test</h1>
</body>
</html>
Connection closed by foreign host.
Only one answer... So the server can reject my request for pipelining.
Now, for support on servers and browsers, I think your wikipedia source tells enough :-)
Related
I understand that the client should stop reusing http keepalive socket after timeout or the max usage. However two popular server implementations seem to keep socket open for the longer duration than the timeout and hence this question to learn what strategies a client should adapt to optimally reuse the connection.
We were testing our implementation with various servers, and while apache seems to be adhering to timeout, lighttpd and nginx seem to keep connection open for longer duration than the timeout. As you can see from the logs below, despite of client-server agreed on 15 second timeout, the socket was kept open for more than 60 seconds and second request succeeded even after 60 seconds.
1) First request and response with timestamp
GET /en/CHANGES HTTP/1.1
Host: nginx.org
Accept: */*
Accept-Encoding: identity
Content-Length: 0
Connection: Keep-Alive
Keep-Alive: timeout=15, max=1000
E(2410-130624-216): HTTP/1.1 200 OK
E(2410-130624-216): Server: nginx/1.13.3
E(2410-130624-216): Date: Tue, 24 Oct 2017 07:36:23 GMT
E(2410-130624-216): Content-Type: text/plain; charset=utf-8
E(2410-130624-216): Content-Length: 282456
E(2410-130624-216): Last-Modified: Tue, 10 Oct 2017 15:39:18 GMT
E(2410-130624-216): Connection: keep-alive
E(2410-130624-216): Keep-Alive: timeout=15
E(2410-130624-216): ETag: "59dce9a6-44f58"
E(2410-130624-216): Accept-Ranges: bytes
E(2410-130624-216):
2) Second request on the same socket, 60 seconds later
GET /en/CHANGES HTTP/1.1
Host: nginx.org
Accept: */*
Accept-Encoding: identity
Content-Length: 0
Connection: Keep-Alive
Keep-Alive: timeout=15, max=1000
E(2410-130726-881): HTTP/1.1 200 OK
E(2410-130726-881): Server: nginx/1.13.3
E(2410-130726-881): Date: Tue, 24 Oct 2017 07:37:26 GMT
E(2410-130726-881): Content-Type: text/plain; charset=utf-8
E(2410-130726-881): Content-Length: 282456
E(2410-130726-881): Last-Modified: Tue, 10 Oct 2017 15:39:18 GMT
E(2410-130726-881): Connection: keep-alive
E(2410-130726-881): Keep-Alive: timeout=15
E(2410-130726-881): ETag: "59dce9a6-44f58"
E(2410-130726-881): Accept-Ranges: bytes
E(2410-130726-881):
There comes the question, when should client stop reusing connection in the optimal manner. Should client keep reusing the connection till it gives error? or should we discard the connection after timeout? What strategies browsers are using?
Thanks
I'm puzzled why this FE doesn't seem to connect me to the BE through my HAproxy:
defaults
mode http
log global
option httplog
option dontlognull
source 0.0.0.0 usesrc clientip # transparent proxy mode
frontend fe-kb
bind :8081 ssl crt /etc/haproxy/ssl/ssl-key.pem
default_backend be-kb
backend be-kb
server afnB afnB:1080 check
I get this in HA http log:
Jan 9 17:25:04 localhost haproxy[17266]: <ip redacted>:51396 [09/Jan/2016:17:24:44.544] fe-kb~ be-kb/afnB 31/0/-1/-1/20036 503 212 - - cC-- 0/0/0/0/3 0/0 "GET / HTTP/1.1"
I can connect fine from HAproxy CLI (selinux is disabled):
[root#hapA ~]# telnet afnB 1080
Trying 10.45.69.14...
Connected to afnB.
Escape character is '^]'.
GET / HTTP/1.0
HTTP/1.1 200 OK
Server: nginx/1.9.9
Date: Sat, 09 Jan 2016 16:40:44 GMT
Content-Type: text/html
Content-Length: 612
Last-Modified: Wed, 09 Dec 2015 15:05:19 GMT
Connection: close
ETag: "5668432f-264"
Accept-Ranges: bytes
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
...
Figured it out, the default route wasn't through my HAproxy and as I was trying to do transparency of the source IPs well ;)
Thanks for watching!
I'm experimenting a very strange issue: using apache-2.2.25 on an Arch Linux server I get a truncated HTTP header in response, but only under windows machines.
I left untouched the httpd.conf file and I have created a file /srv/http/index.html with inside Hello.
With a web browser on a Linux machine I get correctly visualized the hello page, but under Windows I get
.1 200 OK Date: Fri, 15 Nov 2013 08:44:28 GMT Server: Apache/2.2.25 (Unix) mod_ssl/2.2.25 OpenSSL/1.0.1e DAV/2 Last-Modified: Fri, 15 Nov 2013 08:15:09 GMT ETag: "27c-a-4eb32cbf8b950" Accept-Ranges: bytes Content-Length: 10 Keep-Alive: timeout=5, max=100 Connection: Keep-Alive Content-Type: text/html; charset=utf-8 Hello
It seems that the http header is truncated for the first characters (HTTP/1).
Have you got any advices?
I use gwan for image generation, so I need to set correct content type, but G-WAN 3.12.26 after some load adds its own header with content type text/html and returns page with 2 http headers.
How to reproduce this:
use setheaders.c servlet from gwan package, start gwan and open this page, lets say http://localhost/?setheaders.c and you will get this (correct response):
HTTP/1.1 200 OK
Date: Sat, 29 Dec 2012 20:37:52 GMT
Last-Modified: Sat, 29 Dec 2012 20:37:52 GMT
Content-type: text/html
Content-Length: 371
Connection: close
<!DOCTYPE HTML><html lang="en"><head><title>Setting response headers</title><meta http-equiv="Content-Type" content="text/html; charset=utf-8"><link href="imgs/style.css" rel="stylesheet" type="text/css"></head><body style="margin:16px;"><h1>Setting response headers</h1><br>This reply was made with custom HTTP headers, look at the servlet source code.<br></body></html>`
now run apache bench: ab -n 1000 'http://localhost/?setheaders.c' (1000 requests were enough for my system).
DO NOT RESTART GWAN, open http://localhost/?setheaders.c again and this is what you should get (incorrect response, 2 http headers):
HTTP/1.1 200 OK
Server: G-WAN
Date: Sat, 29 Dec 2012 20:43:34 GMT
Last-Modified: Fri, 16 Jan 1970 16:53:33 GMT
ETag: "be86ada7-14b40d-16f"
Vary: Accept-Encoding
Accept-Ranges: bytes
Content-Type: text/html; charset=UTF-8
Content-Length: 367
Content-Encoding: gzip
Connection: close
HTTP/1.1 200 OK
Date: Sat, 29 Dec 2012 20:43:34 GMT
Last-Modified: Sat, 29 Dec 2012 20:43:34 GMT
Content-type: text/html
Content-Length: 371
Connection: close
<!DOCTYPE HTML><html lang="en"><head><title>Setting response headers</title><meta http-equiv="Content-Type" content="text/html; charset=utf-8"><link href="imgs/style.css" rel="stylesheet" type="text/css"></head><body style="margin:16px;"><h1>Setting response headers</h1><br>This reply was made with custom HTTP headers, look at the servlet source code.<br></body></html>
GWAN returns correct response if gzip and x-gzip are not set as acceptable encoding in request header (Accept-Encoding: gzip, x-gzip).
Is it possible to solve this modifying just servlet? If yes, then how?
Are you setting the MIME type like shown in fractal.c:
// -------------------------------------------------------------------------
// specify a MIME type so we don't have to build custom HTTP headers
// -------------------------------------------------------------------------
char *mime = (char*)get_env(argv, REPLY_MIME_TYPE);
// note that we setup the FILE EXTENTION, not the MIME type:
mime[0] = '.'; mime[1] = 'g'; mime[2] = 'i'; mime[3] = 'f'; mime[4] = 0;
If you do so then there's no way to confuse the automatic headers feature.
Other than that, v3.12 has had many instability problems (file time failures, pthread failures, signals failures, etc.) due to our direct syscalls and GLIBC wrappers - an effort initially intended to make the program run on all versions of Linux.
We have found (thanks to the many reports like yours) that rather than trying to fix those issues one by one (pointlessly fighting GLIBC, a moving target with many different releases each having its own bugs and specificities) a much safer path is to ditch GLIBC.
That's what G-WAN v4 will do, just a few days from now.
I'm using Nginx + uwsgi + python3
Sending any header via start_response goes well, but when I want to send more than one header, it becomes mad.
For example, if I write:
start_response('200 OK', [('Last-Modified', 'Wed, 11 Jan 2012 00:00:00 GMT'), ('Content-Type', 'text/html; charset=windows-1251')])
The headers sent are:
HTTP/1.1 200 OK
Transfer-Encoding: chunked
Server: nginx/1.0.11
Connection: close
Date: Wed, 11 Jan 2012 04:17:22 GMT
Content-Type: text/html; charset=windows-1251
Content-Type: text/html; charset=windows-12
uwsgi sends the same header twice and even more the second one is broken.
which uWSGI and nginx version ? In both 0.9.8.x and 1.0.x i cannot reproduce your error.
You can check the real headers sent by uWSGI putting it in http mode with --http/--http-socket