Apache websocket connection:upgrade replaced by keep-alive - apache

Question

I am trying to figure out a mind-numbing apache issue where requests are going through two layers of apache reverse proxies before hitting upstream services. Most traffic seems to make it through fine. The notable exception is websockets.
In particular, this is a test request
curl -i -H 'Connection: Upgrade' -H 'Upgrade: websocket' localhost:80/test.html
When proxying requests from port 80 to port 8080, I notice (using tcpdump and Wireshark) that the Upgrade header has been removed and Connection: Keep-Alive has been set instead. Moreover, any attempts I have made at resetting the Connection header to Upgrade and Upgrade: websocket have been impotent.
Note that the upstream service needs Connection: Upgrade and Upgrade: websocket to initiate the websocket (I get a 404 error without those headers).
Why is Apache forcing Connection: Keep-Alive when proxying to itself? Is there any way to force it to pass along Connection / Upgrade headers or set those values manually? RequestHeader and friends have unfortunately not been helpful. Without Connection and Upgrade making it through the reverse proxy, the upstream service pukes and throws a 404/Not found at the websocket endpoint.
ServerRoot "/usr/local/apache2"
Listen 80
Listen 8080
LogLevel rewrite:trace8
LoadModule mpm_event_module modules/mod_mpm_event.so
LoadModule authn_file_module modules/mod_authn_file.so
LoadModule authn_core_module modules/mod_authn_core.so
LoadModule authz_host_module modules/mod_authz_host.so
LoadModule authz_groupfile_module modules/mod_authz_groupfile.so
LoadModule authz_user_module modules/mod_authz_user.so
LoadModule authz_core_module modules/mod_authz_core.so
LoadModule access_compat_module modules/mod_access_compat.so
LoadModule auth_basic_module modules/mod_auth_basic.so
LoadModule reqtimeout_module modules/mod_reqtimeout.so
LoadModule filter_module modules/mod_filter.so
LoadModule xml2enc_module modules/mod_xml2enc.so
LoadModule proxy_html_module modules/mod_proxy_html.so
LoadModule mime_module modules/mod_mime.so
LoadModule log_config_module modules/mod_log_config.so
LoadModule env_module modules/mod_env.so
LoadModule headers_module modules/mod_headers.so
LoadModule setenvif_module modules/mod_setenvif.so
LoadModule version_module modules/mod_version.so
LoadModule proxy_module modules/mod_proxy.so
LoadModule proxy_http_module modules/mod_proxy_http.so
LoadModule proxy_wstunnel_module modules/mod_proxy_wstunnel.so
LoadModule unixd_module modules/mod_unixd.so
LoadModule status_module modules/mod_status.so
LoadModule autoindex_module modules/mod_autoindex.so
LoadModule auth_mellon_module modules/mod_auth_mellon.so
LoadModule rewrite_module modules/mod_rewrite.so
LoadModule ssl_module modules/mod_ssl.so
<IfModule unixd_module>
User daemon
Group daemon
</IfModule>
<VirtualHost *:80>
LogLevel rewrite:trace8
ServerName localhost
RewriteEngine On
RewriteCond %{HTTP:Upgrade} =websocket
RewriteRule /(.*) ws://localhost:8080/$1 [P,L]
RewriteCond %{HTTP:Upgrade} !=websocket
RewriteRule /(.*) http://localhost:8080/$1 [P,L]
ProxyPass / http://localhost:8080/
ProxyPassReverse / http://localhost:8080/
ProxyRequests Off
</VirtualHost>
<VirtualHost *:8080>
LogLevel rewrite:trace8
ServerName localhost
UseCanonicalName On
RewriteEngine On
ProxyPass / http://proxy-debug:8080/
ProxyPassReverse / http://proxy-debug:8080/
ProxyRequests Off
</VirtualHost>
ServerAdmin you#example.com
ErrorLog /proc/self/fd/2
DocumentRoot "/"

Answer 1

I do not completely understand how this works, but the nuggets of wisdom I have gleaned and the solution I have devised are as follows:
Apache does not understand websockets in one sense (i.e. when it is talking to itself). As a result, you need to use mod_rewrite and set the protocol to ws:// in order to forward websockets upstream (it seems that no amount of setting headers will help you here).
If you set the protocol to ws://, then Apache will set the appropriate headers (Connection: Upgrade and Upgrade: websocket) when proxying requests upstream. However, for some reason, it does not seem to do this when proxying requests to itself / another VirtualHost.
You can see a blurb in VirtualHost *:80 (copied below) that tries to discern whether a websocket is in order and then change the protocol accordingly. This will not work for VirtualHost *:8080 (see the above bullet). Other methods are necessary.
RewriteCond %{HTTP:Upgrade} =websocket
RewriteRule /(.*) ws://localhost:8080/$1 [P,L]
RewriteCond %{HTTP:Upgrade} !=websocket
RewriteRule /(.*) http://localhost:8080/$1 [P,L]
Put all of this together, and you have to communicate to VirtualHost *:8080 that a websocket connection is needed upstream. Luckily, we are in control of VirtualHost *:80 and can discern this information / pass it along. We must do so without touching the Connection or Upgrade headers, since Apache does weird things with them. The protocol may be traceable, but I am not sure how to do that. As a result, I use a bogus, internal, custom header for the transmission. Probably best to name it in such a way that collisions are unlikely.
In VirtualHost *:80, we add a block like:
SetEnvIf Upgrade ^websocket$ websock=true
RequestHeader set X-Is-Websocket %{websock}e
And then we read that header in VirtualHost *:8080, changing the protocol if necessary:
SetEnvIf X-Is-Websocket ^true$ websock=true
RequestHeader unset X-Is-Websocket
# change protocol if necessary
RewriteCond %{ENV:websock} =true
RewriteRule /(.*) ws://proxy-debug:8080/$1 [P,L]
RewriteCond %{ENV:websock} !=true
RewriteRule /(.*) http://proxy-debug:8080/$1 [P,L]
Hope it helps! :)