I have endpoint termination setup on my Google Cloud Platform HTTP Load Balancer and HTTPS set as the protocol for communication with my backends.
This means that all requests, HTTP or HTTPS, is HTTPS to apache. The problem with this is that the HTTPS environment variable is set to on even when X-Forwarded-Proto is set to http.
All of my research and testing only points to the inverse case (setting HTTPS to on when X-Forwarded-Proto is https via a SetEnvIf X-Forwarded-Proto https HTTPS=on rule).
But, I need something to unset HTTPS when X-Forwarded-Proto is http.
I've tried setting SSLOptions -StdEnvVars as well as many combinations of SetEnvIf, SetEnv, and UnsetEnv. Setting it via mod_rewrite is not an option for me (I don't know if it would work anyway). An interesting note about turning off StdEnvVars is that even when it is off, all the SSL related variables are gone except HTTPS and I can confirm nothing else is setting it in any of my config files.
Edit:
Some examples of directives I've tried in my server config, vhost, and htaccess:
SetEnvIf X-Forwarded-Proto http HTTPS=Off
SetEnvIf X-Forwarded-Proto http HTTPS=0
SetEnvIf X-Forwarded-Proto http !HTTPS
SetEnv HTTPS Off
SetEnv HTTPS 0
SetEnv HTTPS
UnsetEnv HTTPS
Using these directives with other variables, including tests like foo works just fine.
Using these directives with other variables, including tests like foo works just fine.
Just an idea first (gladly retracted if someone has a better idea)
https://cloud.google.com/compute/docs/load-balancing/http/ says:
Target proxies
Target proxies terminate HTTP(S) connections from clients, and
are referenced by one or more global forwarding rules and route the
incoming requests to a URL map.
The proxies set HTTP request/response headers as follows:
Via: 1.1 google (requests and responses)
X-Forwarded-Proto: [http | https] (requests only)
X-Forwarded-For: <client IP(s)>, <global forwarding rule external IP> (requests only)
Can be a comma-separated list of IP addresses depending on the X-Forwarded-For entries appended by the intermediaries the client is
traveling through. The first element in the section
shows the origin address.
The question is where this is set. If in the apache config files, you could just alter the config. If it is set somewhere else, you need to find out where.
The TargetHttpsProxies resource did not list any ways to alter it either. So how about you post the config files that lead to above behavior?
Related
I have multiple domains on the same server, all setup with Cloudflare DNS and no problems, but for one domain I get this error whenever I try switching DNS to Cloudflare :
[Sat Dec 05 11:41:22.471013 2020] [pagespeed:warn] [pid 26313:tid 140310743021312] [mod_pagespeed 1.11.33.5-0 #26313] [1205/114122:WARNING:instaweb_context.cc(402)] Unsupported X-Forwarded-Proto: https, https for URL http://mydomain/page.php? protocol not changed .
I thought it was maybe CF forced HTTPS rewrite, but it does the same when I disable it.
I looked also in the .htaccess but found nothing related to HTTPS rewrite. I even deleted the .htaccess to test but it did not stop the warnings.
I tried changing all the settings on CF but nothing made a difference.
I really think its an issue on my server side, but its weird that none of the other domains suffer the same issue.
I don't have much control on the apache config of my host
Unsupported X-Forwarded-Proto: https, https
This would imply you have a proxy (or "load balancer") in addition to Cloudflare. Both will set an X-Forwarded-Proto header and one or other is merging them (arguably incorrectly, according to the "HTTP standard" at least, although X-Forwarded-Proto is only a defacto standard so there are no official rules governing how multiple headers should be merged, so https, https isn't necessarily wrong.)
mod_pagespeed appears to be hardcoded to issue this warning if this header (when set) is not exactly http or https:
if (!STR_CASE_EQ_LITERAL(*x_forwarded_proto_header, "http") &&
!STR_CASE_EQ_LITERAL(*x_forwarded_proto_header, "https")) {
LOG(WARNING) << "Unsupported X-Forwarded-Proto: " << x_forwarded_proto
<< " for URL " << url << " protocol not changed.";
return false;
}
Try setting the following in your .htaccess (or server config) to change this to simply https:
RequestHeader edit X-Forwarded-Proto "^https,\s?https$" "https" early
If this doesn't work, try removing the early argument.
I've been experimenting with Apache mod_proxy and mod_remoteip in order to confirm my understanding of the handling of X-Forwarded-For headers, particularly around how internal IP Addresses (e.g. 10.x.x.x or 192.168.x.x ranges) are being handled.
It seems like mod_proxy doesn't always add the internal IP addresses to the X-Forwarded-For header, but I've been unable to find any documentation explaining the expected behavior for this.
As far as I can tell, when the request initiates from an internal IP address, then mod_proxy will add internal IP addresses to the X-Forwarded-For header, but when the initial request comes from a public IP, mod_proxy does not seem to add any internal IP addresses to the X-Forwarded-For.
The Question
My question is: What are the rules that govern whether or not mod_proxy will append the calling IP address to the X-Forwarded-For header.
The documentation on mod_proxy says:
When acting in a reverse-proxy mode (using the ProxyPass directive, for example), mod_proxy_http adds several request headers in order to pass information to the origin server. These headers are:
X-Forwarded-For -
The IP address of the client.
X-Forwarded-Host -
The original host requested by the client in the Host HTTP request header.
X-Forwarded-Server -
The hostname of the proxy server.
Be careful when using these headers on the origin server, since they will contain more than one (comma-separated) value if the original request already contained one of these headers. For example, you can use %{X-Forwarded-For}i in the log format string of the origin server to log the original clients IP address, but you may get more than one address if the request passes through several proxies.
I read this as saying that the client IP address will always be appended to the X-Forwarded-For header, but that's not the behavior I'm observing.
The rest of this question is the tests I've conducted and the behavior I've observed.
The Setup
I've setup two servers both running Apache with mod_proxy installed. I'll refer to these as One and Two.
One has the (internal) IP address 10.0.7.1
Two has the (internal) IP address 10.0.7.2
One has the following ProxyPass directive so that requests to sub-paths of /proxyToTwo are sent to the equivalent sub-path under /proxyToOne on Two
<Location "/proxyToTwo">
ProxyPass http://10.0.7.2/proxyToOne
</Location>
Two has the following ProxyPass directive so that requests to sub-paths of /proxyToOne are sent back to One but without the /proxyToOne prefix
<Location "/proxyToOne">
ProxyPass http://10.0.7.1
</Location>
The effect of this is that when I issue requests to http://One/proxyToTwo/foo it's proxied as follows
One receives the request, issues the following request to Two http://10.0.7.2/proxyToOne/foo
Two receives the request, issues the following request back to One http://10.0.7.1/foo
One receives a request for /foo and actually serves the resource
So every request is bounced from one to two and back to one before being responded two.
Calling with an Internal IP
Using the above setup, I call One from Two using it's internal IP address:
curl http://10.0.7.1/proxyToTwo/foo
The X-Forwarded-For and X-Forwarded-Host headers received when One eventually gets the request for the /foo resource is what I expect below:
X-Forwarded-For: 10.0.7.2, 10.0.7.1
X-Forwarded-Host: 10.0.7.1, 10.0.7.2
This is what I expect, that the request was proxied first through One then Two, and the requesting IP addresses are first the initial request from Two (curl) then the request from One (mod_proxy) and the final request (not in the header because it's the client IP of the connection being from Two (mod_proxy)
Calling with external IP
The unexpected behavior is that mod_proxy seems to behave differently when called from public IP. So Instead of calling One from Two, I call One from my local machine using the public address
curl http://35.162.28.102/proxyToTwo/foo
The X-Forwarded-Host is still what I expect:
X-Forwarded-Host: 35.162.28.102, 10.0.7.2
That is, the request was first proxied through One (using it's external address) then through Two.
But the X-Forwarded-For header is showing only my (external) IP address:
X-Forwarded-For: 35.163.25.76
This suggests to me that the initial execution of mod_proxy is adding the X-Forwarded-For header with the client IP address. But then the subsequent proxying by Two doesn't append the address of One.
I think this behavior is probably more useful than blindly appending the internal IP addresses to the header, but I can't find it documented anywhere so would like to ensure I fully understand it.
Answering this so it's available in-case others make a similar mistake.
In short: mod_proxy always appends the client IP to the X-Forwarded-For header (or adds an X-Forwarded-For header if there is no existing header).
But, other Apache modules can manipulate the X-Forwarded-For header prior to mod_proxy processing it. Mod_proxy will append the client IP to what it sees as the X-Forwarded-For header, which can be the output of other Apache modules.
In my tests, the other module affecting the result was mod_remoteip. The reason for the difference in resulting behavior was that I was using the RemoteIpTrustedProxy directive to specify my trusted proxies and this will allow trusting of private IPs as the first connection but will not process private IPs in the X-Forwarded-For header.
As a result for the External IP case there was the following handling:
My Machine (35.163.25.76) connects to One with no X-Forwarded-For
One sends this request to Two with my IP in X-Forwarded-For header (X-Forwarded-For: 35.163.25.76)
Two receives this request, mod_remoteip processes the X-Forwarded-For header, because One is trusted, the client IP effectively is 35.163.25.76 and there is effectively no X-Forward-For header passed on to mod_proxy
Two sends this request to One with my IP in X-Forwarded-For header (X-Forwarded-For: 35.163.25.76)
This is what presents as "not appending the private IP" but is actually processing the X-Forward-For header and producing an identical one.
The Private ID case behaves differently because mod_remoteip isn't accepting the private IP address in the X-Forward for. So it's handled by:
Two (10.0.7.2) connects to One with no X-Forwarded-For
One sends this request to Two with Two's IP in X-Forwarded-For header (X-Forwarded-For: 10.0.7.2)
Two receives this request, mod_remoteip processes the X-Forwarded-For header but does not trust the value so leaves it as is. So the client IP remains One's IP (10.0.7.1) and the X-Forward-For header passed on to mod_proxy is unmodified
Two sends this request to One appending the client IP (One's) to X-Forwarded-For header resulting in X-Forwarded-For: 10.0.7.2, 10.0.7.1
I verified this by actually sending in some X-Forwarded-For headers from trusted hosts. E.g.
curl http://10.0.7.1/proxyToTwo/foo --header "X-Forwarded-For: TrustedHost1, TrustedHost2"
This results in the final X-Forward-For header containing only TrustedHost1 indicating that the header really is being processed by mod_remoteip and re-issued by mod_proxy.
In the following scenario,
[client]---https--->[Nginx]---http--->[app server]
How (and what) would I pass down to the app server to uniquely identify the certificate? That is, Nginx validates the certificate, but app server doesn't see it. I need to distinguish between users at the app server, so they can't impersonate each other.
You could adapt the same technique as what's described in this question for Apache Httpd. You'd need the Nginx equivalent of something like:
RequestHeader set X-ClientCert ""
RequestHeader set X-ClientCert "%{SSL_CLIENT_CERT}s"
I haven't tried, but the documentation for the Nginx SSL module has a section about "Embedded Variables". More specifically:
$ssl_client_cert returns the client certificate in the PEM format for an established SSL connection, with each line except the first prepended
with the tab character; this is intended for the use in the
proxy_set_header directive;
This looks like what you need with a reverse-proxy setting, like the one you have.
Note that it's very important to clear this header on its way in, otherwise clients could just set the headers themselves and use any certificate they like.
How you then want to check this in your application server depends on the platform you're using. In Java, for example, you could write a Filter (or a Tomcat Valve) that sets the parameter in the request from this custom HTTP header.
It sounds like you want to use Nginx for SSL termination, but you want the backend servers to be able to tell with the original request was over HTTPS or HTTP.
I think something this could work:
server {
ssl on;
listen 443;
add_header X-Forwarded-Proto https;
proxy_pass ...
}
# If you need insecure requests as well
server {
listen 80;
add_header X-Forwarded-Proto http;
proxy_pass ...
}
Then your app server can check the value of the X-Forwarded-Proto header.
This is the same design pattern that Amazon Web Services uses for terminating SSL at their Elastic Load Balancers. They also set the X-Forwarded-Proto header for backend servers to check.
When acting as a reverse proxy, apache adds x-forwarded headers as described here.
http://httpd.apache.org/docs/2.2/mod/mod_proxy.html#x-headers
In my configuration I have configured server A as a forward proxy. There is a rule like this:
RewriteRule proxy:(.*example.com)/(.*) $1.mysecondserver.com/$2 [P]
This rule lets the server request the resource from one of my other servers.
On the second server (origin) I have a virtual host container for the resource and another rewrite rule like this:
RewriteRule some-regex some-url [P]
It may not seem to make sense like this but there is a lot of other stuff going on that I left out as it is not part of the problem.
However that final request has these headers:
[X-Forwarded-For] => ip of 1st server
[X-Forwarded-Host] => example.myseconserver.com
[X-Forwarded-Server] => example.com
I want those headers gone.
I seem to be unable to unset them with mod_headers. I can add more entries to them, but I can not remove them.
Any ideas?
Since Apache 2, as this pretty answer says, the
ProxyAddHeaders Off
theoretically disables it. In my experiences, it had no effect. However, combined with
<Proxy *>
ProxyAddHeaders Off
</Proxy>
and, with
RequestHeader unset X-Forwarded-Host
RequestHeader unset X-Forwarded-For
RequestHeader unset X-Forwarded-Server
somewhere it started to work.
corrected answer: there is no way to do that since its hardcoded
to fix this in the source code of mod_proxy_http.c search for the following part:
apr_table_mergen(r->headers_in, "X-Forwarded-Server",
r->server->server_hostname);
}
and immediately after that add this code:
// remove any X-Forwarded headers
apr_table_unset(r->headers_in, "X-Forwarded-For");
apr_table_unset(r->headers_in, "X-Forwarded-Host");
apr_table_unset(r->headers_in, "X-Forwarded-Server");
then compile by running apxs2 -cia mod_proxy_http.c
I had the same problem on httpd 2.2 on CentOS 5. Installing httpd 2.4 wasn't possible. But because of some reasons I couldn't switch to nginx completly. So I did it by inserting nginx proxy between httpd and the destination address. So I had: httpd(localhost:80/path) -> nginx(localhost:81/path) -> http://your.destination/path. Installation steps are the following:
Install nginx according to these instructions
Configure nginx to avoid security problems.
Add an location in nginx that will remove those httpd's reverse proxy request headers. It can look like this:
location /path {
proxy_set_header x-forwarded-for "";
proxy_set_header x-forwarded-host "";
proxy_set_header x-forwarded-server "";
proxy_pass http://your.destination/path;
}
I have a Tomcat connected via mod_proxy_ajp to an Apache2.2 instance. Apache does the authentication via mod_auth_kerb, and Tomcat uses request.getRemoteUser() to get the authenticated user.
This basically works, but I want to rewrite the user. However, none of the headers I set affect what is returned by request.getRemoteUser(), I only see them as additional headers, what do I have to do?
# Rewrite Magic: change REMOTE_USER to something Alfresco expects
RewriteEngine On
RewriteMap domain_map txt:/etc/apache2/rewrite-map.txt
# Grab the REMOTE_USER apache environment variable for HTTP forwarding (requires sub-request!)
RewriteCond %{LA-U:REMOTE_USER} (.*)#(.*)
# change the format and replace the domain, e.g.:
# user#some.domain ==> other.domain_user
RewriteRule . - [E=RU:${domain_map:%2|%2}_%1]
# copy processed user to HTTP headers
RequestHeader set REMOTE_USER %{RU}e
RequestHeader set HTTP_REMOTE_USER %{RU}e
RequestHeader set AJP_REMOTE_USER %{RU}e
RequestHeader set AJP_HTTP_REMOTE_USER %{RU}e
Thanks!
I suspect that the headers are not being set as you expect them to be set, and they are getting to Tomcat empty.
I have experienced some puzzling processing order issues that caused RequestHeader to ignore the environment variables set by a RewriteRule. Take a look at https://stackoverflow.com/a/9303018/239408 in case it helps
It seems the getRemoteUser() value can not be overwritten by Apache header directives, as the AJP protocol handler gets the username from some internal Apache structure. I worked around this by sending the username via http header and modifying the Java code to use that instead of using getRemoteUser().