I currently have a server running apache httpd 2.2 serving ~1000 websocket connections. I'm trying to scale this up to around ~10K websockets on the the same hardware. I thought I'd be able to place an nginx reverse proxy on the front end, and that nginx would only connect to the backend when there was incoming traffic, and would maintain the connection to the outside world. However, right now the connection seems to continuous (i.e., once the websocket upgrade is complete, a httpd process is tied up until the connection is broken. Am I misunderstanding how nginx should do websockets proxying, or do I have something misconfigured?
NGINX supports WebSockets by creating a tunnel between the client and the backend server and so the nginx will not terminate the connections to the backend/frontend until the client/server terminates the connections.
See: https://www.nginx.com/blog/websocket-nginx/ for more info.
Related
I've setup Nginx server to act as a reverse proxy for Apache. Now I'm wondering how TCP connections are established and closed in this workflow. My assumption is that the first TCP connection is established between a browser and Nginx, then Nginx establishes second TCP connection to the Apache. When response from Apache is returned, Nginx closes this connection and returns response to the browser. Is this how it's actually done?
Yes, you are right.
Nginx creates 2 connections for 1 request from a client: Client <---> Nginx <---> Apache.
For better performance you should configure buffering (as example https://www.digitalocean.com/community/tutorials/understanding-nginx-http-proxying-load-balancing-buffering-and-caching)
And look on proxy_redirect directive.
Nginx creates two connections, that's correct. except that it does not close the connection, but keeps it open so that it can use it for other requests as well.
I have a server that listens for WebSocket connections on port 80, using Twisted and Autobahn. I want to have it also serve static HTML pages, for when the client doesn't want to use a WebSocket. Is it possible to do both things at the same time, using Twisted and Autobahn?
Sure, have a look here and here. You can run Twisted Web and add a Autobahn based WebSocket Twisted Web resource on a path. You can add any number of Twisted Web resources into your resource tree.
Briefly the technique is to start your WebSocketServerFactory manually by invoking startFactory(), then wrap it within a autobahn.twisted.resource.WebSocketResource resource, which you can then register with putChild anywhere within a Twisted Web hierarchy.
I think you have to add haproxy to the mix. If you want to just use twisted and autobahn then I don't think you can share the port. Having said that, I've got both my websockets and web server listening on the same external port. I had to use haproxy to do the trick, though...haproxy handle the inbound connection, then distributes the connection based upon things it pulls from its environment. Every environment is different. Basically, you get haproxy running, then make your web service and web socket listen on private, different ports. In my case I put my web socket server on 127.0.0.1:9000, and my web service on 127.0.0.1:8080. Then you create a haproxy.conf file for haproxy's configuration, in this example, something like:
global
maxconn 100
mode http
frontend myfrontend
bind *:80
acl is_websocket hdr(Upgrade) -i WebSocket
use_backend ws if is_websocket
default_backend mybackend
backend mybackend
server s3 127.0.0.1:8080
backend ws
timeout server 600s
reqrep ^Host:\ .* \0:9000
server ws1 127.0.0.1:9000
I had to remove a bunch of unrelated stuff from the haproxy.conf file, but this gets the idea across. It was important to me to have only one port visible from the outside instead of managing two.
haproxy is awesome !
-g
For reasons we're not going to change, our application needs to handle the SSL connection, and not the ELB. The goal of using the Proxy Protocol is to get the client's IP address over an SSL connection.
http://aws.typepad.com/aws/2013/07/elastic-load-balancing-adds-support-for-proxy-protocol.html?ref_=9 indicates "Alternatively, you can use it if you are sending HTTPS requests and do not want to terminate the SSL connection on the load balancer. For more information, please visit the Elastic Load Balancing Guide."
Unfortunately, it appears the guide that's linked to doesn't actually elaborate on this, and the basic documentation for the Proxy Protocol ( http://docs.aws.amazon.com/ElasticLoadBalancing/latest/DeveloperGuide/enable-proxy-protocol.html ) fails in our environment when configured as described.
Does anyone have steps or a link for this?
The proxy protocol (version 1) injects a single line into the data stream at the beginning of the connection, before SSL is negotiated by your server. You don't get this information "over" an SSL connection; you get the information prior to SSL handshaking. Your server has to implement this capability and specifically be configured so that it can accept and understand it. For an IPv4 connection, it looks like this:
PROXY TCP4 source-ip dest-ip source-port dest-port\r\n
The standard for the protocol is here:
http://haproxy.1wt.eu/download/1.5/doc/proxy-protocol.txt
Additional info in the ELB docs here:
http://docs.aws.amazon.com/ElasticLoadBalancing/latest/DeveloperGuide/TerminologyandKeyConcepts.html#proxy-protocol
Regarding Apache support, at least at the time AWS announced support for the proxy protocol...
“neither Apache nor Nginx currently support the Proxy Protocol header inserted by the ELB”
— http://aws.typepad.com/aws/2013/07/elastic-load-balancing-adds-support-for-proxy-protocol.html?ref_=9
That is subject to change, of course, but I didn't successfully google for any Apache support of the proxy protocol. Of course, since Apache is open source, you could presumably hack it in there, though I am unfamiliar with the Apache source code.
Realizing that you don't want to change what you're doing now, I still would suggest that depending on your motivation for not wanting to change, there may still be a relatively simple solution. It's a change, but not involving SSL on ELB. What about running HAProxy behind ELB to terminate the SSL in front of Apache? Since HAProxy 1.5 can terminate SSL and appears to be able to translate the proxy protocol string from ELB into an X-Forwarded-For header, as well as generate X-SSL headers to give your application information about the client's SSL cert (perhaps that's your motivation for terminating SSL at the app server instead of on the ELB?) ... so this might be an alternative.
Otherwise, I don't have suggestions unless Apache implements support in the future, or we can find some documentation to indicate that they already have.
For the newer Network Load Balancers which allow your application servers to terminate the TLS connections, you can still get the real IP addresses of your clients and avoid all the work of configuring proxy protocol on the ELBs and in the web server config by simply configuring the target groups to use the servers' instance ids rather than their IP addresses. Regardless of which web server you use, the real IPs of the clients will show up in the logs with no translation needed.
Just to follow up on Michael - sqlbot's answer discussing the AWS support for proxy protocol on EC2 instances behind classic TCP elastic load balancers, the Apache module to use that implements the proxy protocol is mod_remoteip. Enabling it and updating the configuration properly will correct the problem of logging IP addresses of users rather than the elastic load balancer's IPs.
To enable proxy protocol on the elastic load balancer you could use these aws cli commands described in the aws documentation:
aws elb create-load-balancer-policy --load-balancer-name my-elb-name --policy-name my-elb-name-ProxyProtocol-policy --policy-type-name ProxyProtocolPolicyType --policy-attributes AttributeName=ProxyProtocol,AttributeValue=true
aws elb set-load-balancer-policies-for-backend-server --load-balancer-name my-elb-name --instance-port 443 --policy-names my-elb-name-ProxyProtocol-policy
aws elb set-load-balancer-policies-for-backend-server --load-balancer-name my-elb-name --instance-port 80 --policy-names my-elb-name-ProxyProtocol-policy
To enable use of proxy protocol in apache, in a server-wide or VirtualHost context, follow the mod_remoteip documentation such as below:
<IfModule mod_remoteip.c>
RemoteIPProxyProtocol On
RemoteIPHeader X-Forwarded-For
# The IPs or IP range of your ELB:
RemoteIPInternalProxy 192.168.1.0/24
# The IPs of hosts that may need to connect directly to the web server, bypassing the ELB (if applicable):
RemoteIPProxyProtocolExceptions 127.0.0.1
</IfModule>
You'll need to update the LogFormat wherever you have those defined (e.g. httpd.conf) to use %a rather than %h else the load balancer IP addresses will still appear.
I've used both Apache and nginx as a reverse proxy performing HTTPS termination (listening on port 443) and forwarding the unencrypted HTTP traffic to Tomcat on port 8080 before.
However, what I need to do now is do the opposite. I have some client applications running on localhost that are (for simplicity) just talking plain HTTP. I want to be able to tell these client apps to use a forward proxy (on localhost) that will convert them to HTTPS and use a client-side certificate for the communication to the origin. Ie, the client will think it is communicating plain HTTP on port 80, but the traffic will actually leave the host as HTTPS on port 443.
Does anyone know how to configure mod_proxy to do this (or even if it is possible)?
At a further stage, I may need to configure the proxy to use different client certificates based on headers set by the client and also have mod_proxy use RFC 5077 (quick session resumption).
It doesn't have to be Apache (so if nginx or squid can do the function I'm happy with that) as long as it's not a resource hog. We already have Apache running as a reverse proxy anyway so it would be handy if Apache can do it.
I am in the process of migrating a webapp from one server to another, and the IP address of the servers are different. I need to change the A record for the domain to the new server. To make sure the traffic to the old server lands up in the new server I need to setup reverse proxy in the old server which will redirect all traffic to new server.
I have several domains in the same server with different SSL certificates. The client will have a SSL connection the old apache server and the new server will accept connections through SSL only. So I need to setup reverse proxy with SSL connection. Is this possible in apache ? How do I achieve this? I am also worried about slowness due to two SSL connection setup times.
Yes, it is perfectly possible using mod_proxy and mod_ssl. See :
https://httpd.apache.org/docs/2.2/mod/mod_ssl.html#sslproxyengine
As mentioned by Remi, it is possible and to add, you do not have to worry about 2 SSL connections since I am assuming that the reverse proxy connection would have keep alive switched on and therefore the SSL connection setup between the two servers would only be carried out intermittently.
So remember to setup keep-alive. Any server worth its salt would support that and Apache does too.
In the old server I would perform browser redirection "301 moved permanently" to the other IP/hostname (probably on the app root too, just to annoy them enough to change their bookmark and stop using ip addresses in favor of hostnames) . Light and effective.