I have a F5 Virtual Server configured with client-side https encryption, in front of a web server.
I would like this VS to manage also HTTP CONNECT requests, so that clients can request it either as a web server, or as a proxy.
That is to say, the VS should decrypt all TCP connections, but if the first TCP packet starts with "CONNECT", it should first respond HTTP 200, then wait for the next packet (that must be "client hello") and process the SSL handshake.
It is certainly possible with some iRule, but I can't easily get a solution, and I can't find any help on Internet, since F5 doc is not open. Does anyboy know how to do it ?
I finally found a solution.
The VS default behavour is to process the SSL decryption right from the first TCP packet :
so one must look at the first TCP packet, and, if it starts with CONNECT,
disable SSL decryption,
respond with HTTP 200,
then reenable SSL decryption for the "client hello" that should come right after
It works in both context :
in TCP context, before SSL decryption, to detect CONNECT request
and in HTTP context, to properly respond to the CONNECT request
when CLIENT_ACCEPTED { # TCP CONTEXT
TCP::collect 7 # look at the first 7 bytes of TCP stream
}
when CLIENT_DATA {
if { [TCP::payload] starts_with "CONNECT" } {
SSL::disable # disable SSL decryption
}
}
when HTTP_REQUEST { # HTTP CONTEXT
if { [HTTP::method] eq "CONNECT" } {
HTTP::respond 200 # send HTTP 200
SSL::enable # re-enable SSL decryption for next "client hello"
}
}
Related
I'm having a hard time with this setup. I have a node.js box serving HTTP on 3000, websockets on 3001, and secure websockets on 3002. Out in front of that I have a remote Hitch/Varnish caching proxy on its own server that's listening on 443/80 and connecting the first server as its default backend via 3000. A user who visits the site URL https://foo.tld hits the varnish proxy and sees the site, where some javascript on the site tells their browser to connect to wss://foo.tld:3002 for secure websockets.
My problem is getting websockets to pass transparently through to the backend. In the VCL I have the standard
if (req.http.upgrade ~ "(?i)websocket") {
return (pipe);
}
and
sub vcl_pipe {
#Declare pipe handler for websockets
if (req.http.upgrade) {
set bereq.http.upgrade = req.http.upgrade;
set bereq.http.connection = req.http.connection;
}
}
Which doesn't work in this case. To list what I have tried so far with no success:
1: Creating a second backend in VCL named "websockets" that is the same backend IP but on either port 3001 or 3002 and adding "set req.backend_hint = websockets;" before the pipe summon in the first snippet above.
2: Turning off HTTPS and trying to connect it over pure HTTP.
3: Modifying varnish.service to try and make varnish listen on ports other than, or in addition to, -a :80 and -a :8443,proxy, in which cases Varnish simply refuses to start. One attempt was to simply use HTTP only and attempt to run varnish on 3001 to get ws:// working without SSL but varnish refuses to start.
4: Most recently I attempted the following in VCL to try and pick up client connections coming in on 3001:
if (std.port(server.ip) == 3001) {
set req.backend_hint = websockets;
}
My goal is for the Varnish box to pick up secure websocket traffic (wss://) on 3002 (via hitch at 443 using the normal secure websocket connection protocol) and have that passed transparently to the backend websocket server, whether SSL encrypted across that leg of the connection or not. I have set up other, smaller servers like this before and getting websockets working is trivial if Varnish and the backend service are either on the same machine or behind a regulating CDN like Cloudflare, so it has been extra frustrating trying to figure out just what this remote proxy setup needs. I feel like part of the solution is having Varnish or Hitch (not sure) listening on 3002 to accept the connections at which point the normal req.http.upgrade and pipe functions would come into play, but the software refuses to cooperate.
--------Update--
I have broken down the problem into the simplest form I can. The main server (backend) is now serving plain HTTP on 8080 and WS:// on 6081. I have removed hitch and TLS from the equation entirely, but even in this simplified form it is impossible to connect to websockets through Varnish. I can verify that the Websocket server is working correctly on the backend. Connecting to the backend IP address with a browser shows websockets functioning perfectly there. It's Varnish that's the problem.
My current hitch.conf (not relevant here but provided per request):
frontend = "[*]:443"
frontend = "[*]:3001"
backend = "[127.0.0.1]:8443" # 6086 is the default Varnish PROXY port.
workers = 4 # number of CPU cores
daemon = on
# We strongly recommend you create a separate non-privileged hitch
# user and group
user = "redacted"
group = "redacted"
# Enable to let clients negotiate HTTP/2 with ALPN. (default off)
# alpn-protos = "h2, http/1.1"
# run Varnish as backend over PROXY; varnishd -a :80 -a localhost:6086,PROXY ..
write-proxy-v2 = on # Write PROXY header
syslog = on
log-level = 1
# Add pem files to this directory
# pem-dir = "/etc/pki/tls/private"
pem-file = "/redacted/hitch-bundle.pem"
Current default.vcl (stripped down to almost nothing just for testing this. The backend is NOT running on the same machine, it is remote):
# Marker to tell the VCL compiler that this VCL has been adapted to the
# new 4.0 format.
vcl 4.0;
# Default backend definition. Set this to point to your content server.
backend default {
.host = "remote.server.ip";
.port = "8080";
}
backend websockets {
.host = "remote.server.ip";
.port = "6081";
}
sub vcl_recv {
# Happens before we check if we have this in cache already.
#
# Typically you clean up the request here, removing cookies you don't need,
# rewriting the request, etc.
#Allow websockets to pass through the cache (summons pipe handler below)
if (req.http.Upgrade ~ "(?i)websocket") {
set req.backend_hint = websockets;
return (pipe);
} else {
set req.backend_hint = default;
}
}
sub vcl_pipe {
if (req.http.upgrade) {
set bereq.http.upgrade = req.http.upgrade;
set bereq.http.connection = req.http.connection;
}
return (pipe);
}
Varnish's systemd exec parameters:
ExecStart=/usr/sbin/varnishd \
-a http=:80 \
-a proxy=localhost:8443,PROXY \
-a ws=:6081 \
-p feature=+http2 \
-f /etc/varnish/default.vcl \
-s malloc,256m \
-p pipe_timeout=1800
Working in plain HTTP and insecure websockets like this, it should be very simple to get a working model. I don't understand what could possibly be going wrong.
Varnish Cache, the open source version of Varnish, doesn't support backend connections over TLS.
While you can offload TLS using Hitch, the connection to your websocket server will not be encrypted.
Basic VCL example
Here's a very basic VCL example where web & websocket requests are split and sent to separate backends:
vcl 4.1;
backend web {
.port = "3000";
}
backend ws {
.port = "3001";
}
sub vcl_recv {
if (req.http.Upgrade ~ "(?i)websocket") {
set req.backend_hint = ws;
return (pipe);
} else {
set req.backend_hint = web;
}
}
sub vcl_pipe {
if (req.http.upgrade) {
set bereq.http.upgrade = req.http.upgrade;
}
return (pipe);
}
Need more input
However, I'm probably missing a lot of context. I also didn't specify a .host parameter in the backends, so the assumption is that all services are hosted locally.
Please add your full VCL, your Hitch config and the varnishd runtime parameters to your question. This will add context and allows me to come up with a better solution.
What about Hitch?
If you terminate TLS in Hitch, both HTTPS & secure websockets will be handled by Hitch where the plain-text HTTP & websockets will still be directly handeled by Varnish.
See https://www.varnish-software.com/developers/tutorials/terminate-tls-varnish-hitch for a Hitch tutorial that also explains how Varnish should be configured.
I'm a big advocate of using the PROXY protocol in Varnish. The hitch tutorial has a specific section about this: https://www.varnish-software.com/developers/tutorials/terminate-tls-varnish-hitch/#enable-the-proxy-protocol-in-varnish
Custom ports
The standard ports to access the service are 80 for HTTP and insecure websockets and 443 for HTTPS and secure websockets.
If you want to use custom ports for the websockets, it is possible to configure them in Hitch and Varnish.
Let's say you want to main ports 3001 and 3002 for your websockets. This means you need 2 frontends in Hitch:
One for HTTPs on 443
One for secure WS on 3002
See https://www.varnish-software.com/developers/tutorials/terminate-tls-varnish-hitch/#listening-address for more information about the frontend config.
Varnish on the other hand needs to have 3 listening addresses:
One for HTTP on port 80 (-a http=:80)
One for offloaded HTTPS & secure WS with PROXY support on port 8443 (-a proxy=:8443,PROXY)
One for insecure WS on port 3001 (-a ws=:3001)
Next steps
Please use the information and see if this helps to find a solution. If not, please share your VCL file, your Hitch config and varnishd runtime.
Update
Now that you provided more input, the picture starts to become more clear. The fact that you eliminated the TLS part for now will make it a lot easier to debug.
Assuming the names of your listening interfaces for varnishd are http and ws (as mentioned in your systemd unit file), we can use the following varnishlog commands to debug:
varnishlog -g request -q "ReqStart[3] eq 'http'"
This command will show logs for all log transactions where the http listening interface is used.
If you want to make it more granular, you can also add the request URL as a filtering criterium. This will narrow down the number of transactions:
varnishlog -g request -q "ReqStart[3] eq 'http' and ReqUrl eq '/'"
Please add a complete log transaction for one of the failed requests. This will help us understand why requests are failing.
You can do the same for requests on the ws listening interface by using the commands below:
varnishlog -g request -q "ReqStart[3] eq 'ws'"
varnishlog -g request -q "ReqStart[3] eq 'ws' and ReqUrl eq '/'"
I'm assuming you're successful at starting the varnishd program but unsuccessful at getting decent output out of Varnish. The varnishlog program will provide the insight we need. Please add the logging output to your question so I can look into it.
I've configured X_FORWARDED_FOR to capture client IP for a HTTPS request and it works as expected.
However, for MQTT, the data is sent over SSL and HTTP/S does not come into the picture.
ssl://<HOST_NAME>:<PORT>
I've tried adding the following to the backend server on HAproxy config. No luck so far.
backend TestServer
mode tcp
server TestServer01 10.6.186.24:48080 send-proxy-v2
------
server TestServer01 10.6.186.24:48080 send-proxy
------
server TestServer01 10.6.186.24:48080 send-proxy-v2-ssl
Is there a way to capture client (source) IP for an incoming MQTT request by changing HAProxy configuration?
No, there is no where in the MQTT protocol to store the original client IP address (like adding extra headers to HTTP requests).
The proxy is literally just forwarding packets that arrive on it's public port to the backend servers (with the possible exception of doing SSL termination) it doesn't change the packets at all.
If you wanted the IP address to do stick-table based abuse protection, you will need to key your stick-table with the MQTT client identifier.
For example this will reject clients if their connection rate is greater than 1 per second, over a 10s window.
tcp-request content set-var(txn.client_id) req.payload(0,0),mqtt_field_value(connect,client_identifier) if data_in_buffer
stick-table type string len 64 size 100k expire 5m store gpc0,gpc0_rate(10s)
tcp-request content track-sc0 var(txn.client_id)
tcp-request content sc-inc-gpc0(0)
tcp-request content reject if { sc0_gpc0_rate gt 10 }
I am creating a 'firewall' type device (i.e. sitting in the middle of a communication) that in some cases need to intercept a HTTPS request and return a message to the client browser (like e.g. : sorry this is blocked).
I can do this for HTTP by redirecting (with iptables DNAT) to another port on the device where netcat is listening:
while true; do echo -e "HTTP/1.1 200 OK\n\nsorry this is blocked"|nc -l -p 8000; done
(so nc is listening on port 8000 and returning a normal code 200 reply. Could of course also be some other return code like 403 Forbidden etc.)
But what to do for HTTPS?
The whole thing is encapsulated in SSL/TLS and if intercepted the browser will just display a message that the secure connection failed.
I tried responding with a HTTP 307 Temporary Redirect with a Location pointing to http://127.0.0.1 (which would then give the above message). But the browser doesn't like this.
I need to display some sort of customized message (not necessarily HTML).
I realize that it would be a huge security issue if a HTTPS request could be changed to HTTP, thus stripping the security without the client noticing, but can a popup message or something not be forced in the client? Or at least a standard code like '403 Forbidden'..?
Is there something in the SSL or TLS protocols that I can (ab)use?
Thanks.
So you are developing a transparent proxy. When it comes to HTTPS traffic every proxy has the choice:
Pass it without decryption
Block it completely
Perform a man-in-the-middle attack for getting access to the content
If you performing the man-in-the-middle attack and the client does not trust the certificate used by the proxy it will get a certificate warning. You can not send anything HTTP related to the client because SSL/TLS already fails to establish the tunnel. No tunnel means that you will not be able to transmit a single "HTTP byte" (this also means that you can not redirect the client somewhere else).
And on SSL/TLS level there is AFAIK no way to send a custom message. The "TLS alert message" only allows pre-defined constant values.
I have built an application called Tun2Socks GUI. It's program to make Socks proxy o be transparent.
Usually it use SSH port forward or TOR as SOCKS service, but I want it can use HTTP proxy too. So I build SOCKS5 proxy my self that connect to that HTTP proxy. It's working good with capturing HTTP request from client to be sent to HTTP Proxy.
The problem when the client send SSL request, I cannot capture the request to be forwarded. How the best method to make SSL request from SOCKS proxy through HTTP Proxy?
Schema of request transportation like here :
Client SSL request > SOCKS Proxy > HTTP Proxy > Internet
Thanks
When a client intentionally wants to establish an SSL session with a target server through a proxy, it does not establish an SSL session with the proxy itself. The client first tells the proxy to establish a connection to the target server, and THEN the client initiates an SSL session with the target server. In that situation, it is not possible for the proxy to sniff the traffic as it is encrypted, nor should it be trying to. A proxy is just a pass-through, it exchanges raw data back and forth between client and server as needed. The proxy should not care what kind of requests the client is sending, since the client tells the proxy where to connect.
If you have injected your proxy in between the client and server in such a way that the client has no knowledge that your proxy exists, the client will not know that it needs to adjust its requests to make them proxy-friendly. The client will be connected to your proxy but it will think it is connected to the target server, and thus will initiate an SSL handshake that your proxy will have to respond to. Only then will your proxy have access to the client's request data (provided the handshake is successful, such as if the client does not verify peer certificates), and can then tunnel the unencrypted data to the next proxy as needed.
Update: I just thought of another scenario that should work for both cleartext and SSL connections. Regardless of whether you are transparently redirecting the client's outbound connection to your SOCKS proxy without the client knowing about it, or the client intentionally connects to the SOCKS proxy and tells it where to go, the SOCKS proxy knows the client's target host/IP:port. The SOCKS proxy can either connect directly to the target, or it can connect to the HTTP proxy and ask it to create a tunnel to the target via the HTTP CONNECT method. If successful, the client has a viable connection to the target, and any data the client sends, SSL or otherwise, will flow as-is to the target, and vice versa. Neither the SOCKS proxy nor the HTTP proxy needs to know anything about the client's request other than the target host/IP:port. That is in the initial SOCKS request, either captured from the intercepted TCP header, or explicit from the client.
Let's say you want to perform an https request to a certain website but you have a proxy on the middle.
The aforesaid proxy doesn't look into the request but just relay all the traffic to the actual HTTPS server after the user-agent has used the HTTP CONNECT method (as in http://www.web-cache.com/Writings/Internet-Drafts/draft-luotonen-web-proxy-tunneling-01.txt).
Now my question is the following: after the proxy opens a SSL connection to the destination webserver, should it also upgrade the socket which handles the connection with the client to SSL as well? And if so, how would it forward packets to the server without sniffing the actual content?
What I mean here is that if the proxy actually reads data from SSL client socket and forwards them to SSL server socket, the data will be not encrypted to it.
The proxy has a plaintext connection open to the client, via which it received the CONNECT command. It opens a plaintext connection to the server. Thereafter it just copies bytes in both directions. The bytes coming from both client and server are SSL, so this works without the proxy knowing what's inside the ciphertext.