We are getting lots of 408 status code in apache access log and these are coming after migration from http to https .
Our web server is behind loadbalancer and we are using keepalive on and keepalivetimeout value is 15 sec.
Can someone please help to resolve this.
Same problem here, after migration from http to https. Do not panic, it is not a bug but a client feature ;)
I suppose that you find these log entries only in the logs of the default (or alphabetically first) apache ssl conf and that you have a low timeout (<20).
As of my tests these are clients establishing pre-connected/speculative sockets to your web server for fast next page/resource load.
Since they only establish the initial socket connection or handshake ( 150 bytes or few thousands) the connect to the ip and do not specify a vhost name, and got logged in the default/firs apache conf log.
After few secs from the initial connection they drop the socket if not needed or the use is for faster further request.
If your timeout is lower than these few secs you get the 408 if is higher apache doesn't bother.
So either you ignore them / add a different default conf for apache, or you rise the timeout having more apache processes busy waiting from the client to drop or use the socket.
see https://bugs.chromium.org/p/chromium/issues/detail?id=85229 for some related discussions
We have two recently upgraded Plone 4.3.2 instances behind a haproxy load balancer which itself is behind Apache.
We limit each Plone instance to serving two concurrent requests using haproxy configuration.
We recently encountered an issue whereby a client sent 4 byte-range requests in quick succession for a PDF that each took between 6 and 8 minutes to get a response. This locked up all available requests for 6 minutes and so haproxy timed out other requests in the queue. The PDF is stored an ATFile object in Plone which I believe should have been migrated to blob storage in our recent upgrade.
My question is what steps should we take to prevent a similar scenario in the future?
I'm also interested in:
how to debug why the byte-range requests on an otherwise lightly loaded server should take so long to respond
how plone.app.blob deals with byte-range requests
is it possible to configure Apache such that byte-range requests are served from its cache but not from the back-end server
As requested here is the haproxy.cfg with superfluous configuration stripped out.
global
maxconn 450
spread-checks 3
defaults
log /dev/log local0
mode http
option http-server-close
option abortonclose
option redispatch
option httplog
timeout connect 7s
timeout client 300s
timeout queue 120s
timeout server 300s
listen cms 127.0.0.1:18181
id 3
balance leastconn
option httpchk
http-check send-state
timeout check 10s
acl cms_edit url_dom xxx.xxx.xxx.xxx
acl cms_not_ok nbsrv() lt 2
block if cms_edit cms_not_ok
server cms_instance1 app:18081 check downinter 10s maxconn 2 rise 1 slowstart 300s
server cms_instance2 app:18082 check downinter 10s maxconn 2 rise 1 slowstart 300s
You can install https://pypi.python.org/pypi/Products.LongRequestLogger and check its log file to see where the request gets stuck.
I've opted to disable byte-range requests to the back-end Zope server. I've added the following to the CMS listen section in haproxy.
reqidel ^Range:.*
I have an odd issue, and have managed to replicate this problem at different locations with different installs of Squid.
I will base my "problem" with my squid server at home.
Running Fedora 20 (32bit), with Squid 3.3.11, firewall and iptables uninstalled/disabled. The network is IPv4. I have a couple of Windows 7 machines with IE11, and 1x Windows 8.1 machine with IE11.
My problem is, on my Windows 8.1 machine with IPv6 protocols turned off, trying to load SSL based web pages (such as https://www.google.co.uk or https://www.facebook.com), the initial page load results in an error. Subsiquent loads either fail, part fail (IE main body of the site loads, but further SSL connections fail, such as image loads) or allow the page to load.). Oddly enough though, I do not re-call having a problem with my banks website! I would suggest that some websites seem to struggle more than others.
A friend of mine also managed to replicate the fault on a squid server he setup with Windows 8.1. He commented that using another browser such as Firefox, the problem is resolved so it seems limited to Windows 8.1, and IE11
Using Wireshark, during the failed attempts, towards the end my machine sends back a load of TCP RST commands.
However loading the same websites on my Windows 7 using IE11, or Windows XP with IE8, the problem does not appear, and until I moved to Windows 8.1, I had 0 problems with my Squid server.
My Squid Config is fairly basic as I just use it for filtering adverts using a block list using SquidGuard, although an experiment ruled out SquidGuard as being the problem when I removed the relevant line from squid.conf.
Thanks for reading and hope we can get to the bottom of this!
Copy of my squid config.
#squid normally listens to port 3128
http_port 3128
#Allow local machine
#acl manager proto cache_object
acl localhost src 192.168.20.6
# Only allow cachemgr access from localhost
http_access allow manager localhost
http_access deny manager
http_access allow localhost
# Define Local network
acl localnet src 192.168.20.0/24
http_access allow localnet
#Redirect for SquidGuard
redirect_program /usr/bin/squidGuard -c /etc/squid/squidGuard.conf
# And finally deny all other access to this proxy
http_access deny all
I found it: Disable SPDY/3 Protokol in IE11 (Extras....)
It might be related to IE11 disabling RC4 and using TLS1.2 in the initial handshake, see http://blogs.msdn.com/b/ie/archive/2013/11/12/ie11-automatically-makes-over-40-of-the-web-more-secure-while-making-sure-sites-continue-to-work.aspx. Unfortunatly lots of hosts still require RC4 and might fail if the client does not offer this cipher. Other hosts will also croak when using TLS1.2, e.g they just close the connection (like currently bmwi.de) or just drop the request (usually an older F5 BIG-IP load balancer in front).
I know from Chrome that it just retries a failed TLS 1.2 request immediately with TLS 1.1. Maybe IE11 does this different, e.g. does not retry immediately but only remembers the failure and work around it if the user retries. Or it behaves this way only when used with a proxy.
That you sometimes get the full page and sometimes only the html on the second request might be the case if the images are loaded from a different host name, e.g. images.whatever instead of www.whatever. In this case it gets at first a failed handshake with www.whatever and after a successful retry it with downgraded SSL it receives the HTML with the includes from images.whatever. It then will access images.whatever and will run in the same problems with SSL, so that the page stays w/o images until another retry.
If my theory is correct, you should see in wireshark in the initial failed connect an SSL Client Hello message with version TLS 1.2 and it should not have RC4 in the list of offered ciphers. On the second (successful) connect you should see RC4 in the cipher list of the Client Hello and maybe it will also use TLS 1.1 or oven TLS 1.0 instead of TLS 1.2.
I need to setup Load balancer as an alternative for ELB for Amazon as they have issue in connection timeout.
Currently, Im using HAProxy and it works normally. However, I need to use SSL for users who wants to connect in https (port 443) to the backend apache servers plus sticky session.
What will be the configuration would looks like? I heard that HAProxy doesn't support SSL in native and can use stunnel or nginx / apache to handle the SSL termination.
I would appreciate anyone to share their knowledge and experiences.
Thanks.
James
To http use something like that.
Change the XXX.XXX.XXX.XXX to your IP address.
listen example-cluster XXX.XXX.XXX.XXX:80
mode http
stats enable
stats auth user:password
stick store-request src
stick-table type ip size 200k expire 2m
balance source
cookie JSESSIONID prefix
option httplog
option httpclose
option forwardfor
option persist
option redispatch
option httpchk HEAD /check.txt HTTP/1.0
server example-webl XXX.XXX.XXX.XXX:80 cookie A check
server example-web2 XXX.XXX.XXX.XXX:80 cookie B check
server example-web3 XXX.XXX.XXX.XXX:80 cookie C check
server example-web4 XXX.XXX.XXX.XXX:80 cookie D check
server example-web5 XXX.XXX.XXX.XXX:80 cookie E check
To your SSL use the mode tcp with balance source:
listen example-cluster-ssl XXX.XXX.XXX.XXX:443
mode tcp
reqadd X-Forwarded-Proto:\ https
stick store-request src
stick-table type ip size 200k expire 2m
option persist
option redispatch
option ssl-hello-chk
balance source
server example-webl XXX.XXX.XXX.XXX:443 check
server example-web2 XXX.XXX.XXX.XXX:443 check
server example-web3 XXX.XXX.XXX.XXX:443 check
server example-web4 XXX.XXX.XXX.XXX:443 check
server example-web5 XXX.XXX.XXX.XXX:443 check
Another way is your upgrade your haproxy to version 1.5, in that version have support to ssl but isn't stable yet.
Take a look at the Stud project on github, which combines extremely well with haproxy, is very performant, scalable, and uses very little resource. Many users are switching to it right now because it's simple and efficient.
At my workplace, the traffic blocker/firewall has been getting progressively worse. I can't connect to my home machine on port 22, and lack of ssh access makes me sad. I was previously able to use SSH by moving it to port 5050, but I think some recent filters now treat this traffic as IM and redirect it through another proxy, maybe. That's my best guess; in any case, my ssh connections now terminate before I get to log in.
These days I've been using Ajaxterm over HTTPS, as port 443 is still unmolested, but this is far from ideal. (Sucky terminal emulation, lack of port forwarding, my browser leaks memory at an amazing rate...) I tried setting up mod_proxy_connect on top of mod_ssl, with the idea that I could send a CONNECT localhost:22 HTTP/1.1 request through HTTPS, and then I'd be all set. Sadly, this seems to not work; the HTTPS connection works, up until I finish sending my request; then SSL craps out. It appears as though mod_proxy_connect takes over the whole connection instead of continuing to pipe through mod_ssl, confusing the heck out of the HTTPS client.
Is there a way to get this to work? I don't want to do this over plain HTTP, for several reasons:
Leaving a big fat open proxy like that just stinks
A big fat open proxy is not good over HTTPS either, but with authentication required it feels fine to me
HTTP goes through a proxy -- I'm not too concerned about my traffic being sniffed, as it's ssh that'll be going "plaintext" through the tunnel -- but it's a lot more likely to be mangled than HTTPS, which fundamentally cannot be proxied
Requirements:
Must work over port 443, without disturbing other HTTPS traffic (i.e. I can't just put the ssh server on port 443, because I would no longer be able to serve pages over HTTPS)
I have or can write a simple port forwarder client that runs under Windows (or Cygwin)
Edit
DAG: Tunnelling SSH over HTTP(S) has been pointed out to me, but it doesn't help: at the end of the article, they mention Bug 29744 - CONNECT does not work over existing SSL connection preventing tunnelling over HTTPS, exactly the problem I was running into. At this point, I am probably looking at some CGI script, but I don't want to list that as a requirement if there's better solutions available.
Find out why the company has such a restrictive policy. It might be for a good reason.
If you still find that you want to bypass the policy, you could write a small proxy that will listen on your server on port 443 and then, depending on the request, will forward the traffic either to your web server or to the SSH daemon. There are two catches though.
To determine whether it's an HTTPS request or an SSH request, you need to try to read some data with a (small) timeout, this is because TLS/SSL handshakes start with the client sending some data, whereas the SSH handshake starts with the server sending some data. The timeout has to be big enough to delays in delivering the initial data from the client in the TLS/SSL handshake, so it'll make establishing SSH connections slower.
If the HTTP proxy in your company is smart, it'll actually eavesdrop on the expected TLS/SSL "handshake" when you CONNECT to port 443, and, when it detects that it's not an TLS/SSL handshake, it might terminate the SSH connection attempt. To address that, you could wrap the SSH daemon into an TLS/SSL tunnel (e.g., stunnel), but then you'll need to differentiate requests based on the TLS/SSL version in your client request to determine whether to route the TLS/SSL connection to the web server or to the TLS/SSL-tunneled SSH daemon.
You should be able to use iptables to forward ssh traffic from your work machines to ssh while all other machines attaching to your home server on port 443 get the Apache server.
Try a rule like this:
iptables -t nat -A PREROUTING -p tcp -s 111.111.111.111 --dport 443 -j REDIRECT --to-port 22
Where 111.111.111.111 is your office computer's ip address.
That all assumes you're running Linux >= 2.4, which you should be by now. It's been out for almost a decade.
Documentation for iptables is at http://www.netfilter.org.
Set up OpenVPN 2.1 server at home, use port 443 (if you set up your home any HTTPS service at port 443, trigger OpenVPN's port-share option to handle both OpenVPN and HTTPS transactions at port 443; this feature is only available to non-Windows OS)
Then, set up your OpenVPN client on your laptop in road-warrior mode to access the OpenVPN server at home. You will be able to call home or anywhere you like within a secure VPN network you've created with OpenVPN. It is no longer required to use SSH for this purpose.
I'm really sorry for being the Devil's advocate here, but if they are blocking ports at your work, its likely because they don't want people breaching security.
Now if you get permission to open a tunnel from your boss, that's fine, but IF something happens, ANYTHING, and they figure out you have a tunnel, I can almost assure you, you'll become the scapegoat. So if I were you I'd not be opening tunnels at work if they are setting up firewalls against it.
How about using 2 IP adresses on your machine?
Bind apache/https on one IP_1:443 and your sshd on the other IP_2:443?
Could you set up a middle man?
Run a small/free/cheap instance in the cloud listening on 443 for SSH, then though that cloud instance tunnel to your home box on your favorite port - 22 or whatever.
It'll add some latency I'm sure, but it solves the problem of leaving the original home setup intact.
I think you'll have to find a port that you're not using currently that you can get out on, and listen on that. 443 is the obvious candidate, but you say that's not possible. What about mail (25, 110, 143), telnet (23), ftp (21), DNS (53), or even whois (43)?
Proxy tunnel may be your answer
http://proxytunnel.sourceforge.net/
lets say my ssh server is host.domain.tld and my works proxy server is 10.2.4.37
I would add this to my local ssh config
Host host.domain.tld
ProxyCommand /usr/local/bin/proxytunnel -q -p 10.2.4.37:3128 -d %h:%p
ProtocolKeepAlives 30
See:
SSH Through or Over Proxy
http://daniel.haxx.se/docs/sshproxy.html
http://www.agroman.net/corkscrew/
Since apache has no problem whatsoever with CONNECT when no SSL is involved, I turn off SSL features and I use stunnel to serve an https version of my site. This does not require any recompilation, and allows your site to serve https normally. So far, the cleanest workaround I know.
See http://chm.duquesne.free.fr/blog/?p=281 for details.
Must work over port 443, without disturbing other HTTPS traffic (i.e. I can't just put the ssh server on port 443, because I would no longer be able to serve pages over HTTPS)
Is it possible to bind your HTTPS server to a different port? Depending on what it's used for, you may even be able to get around the problem of not being able to directly access it from work by just SSHing home and then using lynx from there.
So, then, give proxifier a try (- it supports HTTP Proxy Server)!
http://www.proxifier.com/documentation/intro.htm
I managed to bypass my company's firewall using the following design via AjaxTerm, it works for me.
PC on company network --> company's proxy via https --> INTERNET --> My home Apache reverse proxy server on SSL + .htpasswd protection --> AjaxTerm Server(From here on ward, I can SSH to any other servers ).
Still not the perfect world... would be good if I can can tunneling to my home network via HTTPS.