My setup is as follows:
Load Balancer → nginx → Traefik
The load balancer in place does not support Proxy Protocol. Instead it adds the real IP of the client to the TCP options field (yikes, I know! Details). That's something Traefik does not support.
To get the real IP to Traefik, I added an nginx inbetween that does nothing more than accepting connections on ports 80 and 443 and adding Proxy Protocol when using SSL. Traefik is configured for Proxy Protocol. Things work as expected.
However I'd like to set the X-Real-IP header to the correct IP when Proxy Protocol is used. When I try setting the header manually through curl, that one is used, so clients can overwrite it.
How can I tell Traefik to always set X-Real-IP to the IP as adviced by Proxy Protocol?
I solved my problem and can see clearer now.
It depdends on which node in your configuration (Load Balancer → nginx → Traefik) terminates the clients request. In my setup (Load Balancer → Traefik) the Load Balancer uses NATing to send the request to the Traefik. Traefik then takes the client´s request and sends a new request to the corresponding backend.
So I had to configure Traefik to never trust the X-Real-Ip header but always set the request´s source ip in the X-Real-Ip header.
Configuration is something like this:
[entryPoints.http.proxyProtocol]
insecure = true
trustedIPs = ["10.10.10.1", "10.10.10.2"]
[entryPoints.http.forwardedHeaders]
trustedIPs = ["10.10.10.1", "10.10.10.2"]
The mostly found configuration (I think) would be that the Load Balancer takes the client´s request and then sends a new request to nginx (reverse proxy load balancer). In this case the Load Balancer must set the X-Real-Ip Header, nginx must propagate the header to Traefik and Traefik must be configured to trust nginx as source for the X-Real-Ip header.
I just looked into the source code because of a similar problem.
Traefik sets the header X-Real-Ip with the source IP address of the request being forwarded. If the header X-Real-Ip already exists, it will be passed through unchanged.
I hope that answers the question.
if req.Header.Get (XRealIp) == "" {
req.Header.Set (XRealIp, clientIP)
}
Related
We are using mulesoft proxy api as proxy for another api. We need to know the X-Forwarded-For of the originating requestor.
If you are creating the proxy with API Manager you could add a Header Injection policy to it to add the header. You could use as the value the expression #[attributes.remoteAddress] so it takes the IP of the client of the IP.
If the proxy is deployed to CloudHub it might be receiving the IP of CloudHub load balancer and the load balancer may be already adding the X-Forwarded-For header.
I have a few questions about cloudflare service before I use it.
Are there any difference on http request header between a web server use cf-connecting-ip,x-forwarded-for etc and a web don't use these kinds of thing?
Can/How client knows that a web server uses cf-connecting-ip, x-forwarded-for etc?
These request headers are specific to webproxies, and appended after the request was received by Cloudflare, when it's relaying to your origin. The user cannot see it, unless your webserver for some reason sends it back.
x-forwarded-for: https://en.wikipedia.org/wiki/X-Forwarded-For
Used by proxy servers to tell the origin any HTTP servers involved in relaying the request between the user and the origin. you may see 1 or multiple sets of IP addresses in this header.
cf-connecting-ip:
Relays the IP of the user connecting to Cloudflare to the origin webserver
From: https://support.cloudflare.com/hc/en-us/articles/200170986-How-does-Cloudflare-handle-HTTP-Request-headers-
CF-Connecting-IP Provides the client (visitor) IP address (connecting
to Cloudflare) to the origin web server. cf-connecting-ip contains a
special Cloudflare IP 2a06:98c0:3600:0:0:0:0:103 when the request
originates from a Cloudflare Workers subrequest instead of the
visitor's true IP.
Example:
CF-Connecting-IP: 203.0.113.1
X-Forwarded-For Maintains proxy server and original visitor IP
addresses. If there was no existing X-Forwarded-For header in the
request sent to Cloudflare, X-Forwarded-For has an identical value to
the CF-Connecting-IP header:
Example:
X-Forwarded-For: 203.0.113.1 If an X-Forwarded-For header was already
present in the request to Cloudflare, Cloudflare appends the IP
address of the HTTP proxy to the header:
Example:
X-Forwarded-For: 203.0.113.1,198.51.100.101,198.51.100.102 In the
examples above, 203.0.113.1 is the original visitor IP address and
198.51.100.101 and 198.51.100.102 are proxy server IP addresses provided to Cloudflare via the X-Forwarded-For header.
So for advice, review your application and see if it depends on the IP of the user, if so you will need to modify your web-server configuration to relay the correct IP.
To restore original visitor IP addresses at your origin web server,
Cloudflare recommends your logs or applications look at
CF-Connecting-IP or True-Client-IP instead of X-Forwarded-For since
CF-Connecting-IP and True-Client-IP have a consistent format
containing only one IP.
I've been experimenting with Apache mod_proxy and mod_remoteip in order to confirm my understanding of the handling of X-Forwarded-For headers, particularly around how internal IP Addresses (e.g. 10.x.x.x or 192.168.x.x ranges) are being handled.
It seems like mod_proxy doesn't always add the internal IP addresses to the X-Forwarded-For header, but I've been unable to find any documentation explaining the expected behavior for this.
As far as I can tell, when the request initiates from an internal IP address, then mod_proxy will add internal IP addresses to the X-Forwarded-For header, but when the initial request comes from a public IP, mod_proxy does not seem to add any internal IP addresses to the X-Forwarded-For.
The Question
My question is: What are the rules that govern whether or not mod_proxy will append the calling IP address to the X-Forwarded-For header.
The documentation on mod_proxy says:
When acting in a reverse-proxy mode (using the ProxyPass directive, for example), mod_proxy_http adds several request headers in order to pass information to the origin server. These headers are:
X-Forwarded-For -
The IP address of the client.
X-Forwarded-Host -
The original host requested by the client in the Host HTTP request header.
X-Forwarded-Server -
The hostname of the proxy server.
Be careful when using these headers on the origin server, since they will contain more than one (comma-separated) value if the original request already contained one of these headers. For example, you can use %{X-Forwarded-For}i in the log format string of the origin server to log the original clients IP address, but you may get more than one address if the request passes through several proxies.
I read this as saying that the client IP address will always be appended to the X-Forwarded-For header, but that's not the behavior I'm observing.
The rest of this question is the tests I've conducted and the behavior I've observed.
The Setup
I've setup two servers both running Apache with mod_proxy installed. I'll refer to these as One and Two.
One has the (internal) IP address 10.0.7.1
Two has the (internal) IP address 10.0.7.2
One has the following ProxyPass directive so that requests to sub-paths of /proxyToTwo are sent to the equivalent sub-path under /proxyToOne on Two
<Location "/proxyToTwo">
ProxyPass http://10.0.7.2/proxyToOne
</Location>
Two has the following ProxyPass directive so that requests to sub-paths of /proxyToOne are sent back to One but without the /proxyToOne prefix
<Location "/proxyToOne">
ProxyPass http://10.0.7.1
</Location>
The effect of this is that when I issue requests to http://One/proxyToTwo/foo it's proxied as follows
One receives the request, issues the following request to Two http://10.0.7.2/proxyToOne/foo
Two receives the request, issues the following request back to One http://10.0.7.1/foo
One receives a request for /foo and actually serves the resource
So every request is bounced from one to two and back to one before being responded two.
Calling with an Internal IP
Using the above setup, I call One from Two using it's internal IP address:
curl http://10.0.7.1/proxyToTwo/foo
The X-Forwarded-For and X-Forwarded-Host headers received when One eventually gets the request for the /foo resource is what I expect below:
X-Forwarded-For: 10.0.7.2, 10.0.7.1
X-Forwarded-Host: 10.0.7.1, 10.0.7.2
This is what I expect, that the request was proxied first through One then Two, and the requesting IP addresses are first the initial request from Two (curl) then the request from One (mod_proxy) and the final request (not in the header because it's the client IP of the connection being from Two (mod_proxy)
Calling with external IP
The unexpected behavior is that mod_proxy seems to behave differently when called from public IP. So Instead of calling One from Two, I call One from my local machine using the public address
curl http://35.162.28.102/proxyToTwo/foo
The X-Forwarded-Host is still what I expect:
X-Forwarded-Host: 35.162.28.102, 10.0.7.2
That is, the request was first proxied through One (using it's external address) then through Two.
But the X-Forwarded-For header is showing only my (external) IP address:
X-Forwarded-For: 35.163.25.76
This suggests to me that the initial execution of mod_proxy is adding the X-Forwarded-For header with the client IP address. But then the subsequent proxying by Two doesn't append the address of One.
I think this behavior is probably more useful than blindly appending the internal IP addresses to the header, but I can't find it documented anywhere so would like to ensure I fully understand it.
Answering this so it's available in-case others make a similar mistake.
In short: mod_proxy always appends the client IP to the X-Forwarded-For header (or adds an X-Forwarded-For header if there is no existing header).
But, other Apache modules can manipulate the X-Forwarded-For header prior to mod_proxy processing it. Mod_proxy will append the client IP to what it sees as the X-Forwarded-For header, which can be the output of other Apache modules.
In my tests, the other module affecting the result was mod_remoteip. The reason for the difference in resulting behavior was that I was using the RemoteIpTrustedProxy directive to specify my trusted proxies and this will allow trusting of private IPs as the first connection but will not process private IPs in the X-Forwarded-For header.
As a result for the External IP case there was the following handling:
My Machine (35.163.25.76) connects to One with no X-Forwarded-For
One sends this request to Two with my IP in X-Forwarded-For header (X-Forwarded-For: 35.163.25.76)
Two receives this request, mod_remoteip processes the X-Forwarded-For header, because One is trusted, the client IP effectively is 35.163.25.76 and there is effectively no X-Forward-For header passed on to mod_proxy
Two sends this request to One with my IP in X-Forwarded-For header (X-Forwarded-For: 35.163.25.76)
This is what presents as "not appending the private IP" but is actually processing the X-Forward-For header and producing an identical one.
The Private ID case behaves differently because mod_remoteip isn't accepting the private IP address in the X-Forward for. So it's handled by:
Two (10.0.7.2) connects to One with no X-Forwarded-For
One sends this request to Two with Two's IP in X-Forwarded-For header (X-Forwarded-For: 10.0.7.2)
Two receives this request, mod_remoteip processes the X-Forwarded-For header but does not trust the value so leaves it as is. So the client IP remains One's IP (10.0.7.1) and the X-Forward-For header passed on to mod_proxy is unmodified
Two sends this request to One appending the client IP (One's) to X-Forwarded-For header resulting in X-Forwarded-For: 10.0.7.2, 10.0.7.1
I verified this by actually sending in some X-Forwarded-For headers from trusted hosts. E.g.
curl http://10.0.7.1/proxyToTwo/foo --header "X-Forwarded-For: TrustedHost1, TrustedHost2"
This results in the final X-Forward-For header containing only TrustedHost1 indicating that the header really is being processed by mod_remoteip and re-issued by mod_proxy.
Is there a way to get the origin IP of the user from the HTTP load balancing w/ GCloud? We are currently using just Network Load Balancing, and are needing to move to a cross region balancer although we need to user's IP for compliance and logging.
Does it pass in a header or something along those lines?
Thanks ~Z
The documentation (https://cloud.google.com/compute/docs/load-balancing/http/) says it's the first IP address of the X-Forwarded-For header.
X-Forwarded-For: <client IP(s)>, <global forwarding rule external IP>
If you are sure that you do not run any other proxy (that append additional IPs into X-Forwarded-For) behind Google Cloud Balancing, you can get the second to last IP from X-Forwarded-For as immediate client IP. Or even if you have some proxies but know the exact number of additional IPs that will be appended, you can also add those into account.
From https://cloud.google.com/compute/docs/load-balancing/http/#components:
X-Forwarded-For: <unverified IP(s)>, <immediate client IP>, <global forwarding rule external IP>, <proxies running in GCP> (requests only)
Only the <immediate client IP> and <global forwarding rule external IP> entries are provided by the load balancer. All other entries in
the list are passed along without verification.
IPs that comes before immediate client IP could be spoofed IPs or IPs coming from client proxies. Even if the client spoofs X-Forwarded-For header, the load balancer still appends the actual IP that hits the load balancer.
Ok, so after digging though headers and other things I found the following header that is passing the origin IP and thee IP for the user.
$_SERVER['HTTP_X_FORWARDED_FOR']
You will need to split it by the ',' and take the first part of the string. This is the user IP, that is being pushed by the Google Cloud HTTP Balancer.
Based on HTTP_X_FORWARDED_FOR header, a nice Nginx rule to split the IPs chain :
set $realip $remote_addr;
if ($http_x_forwarded_for ~ "^(\d+\.\d+\.\d+\.\d+)") {
set $realip $1;
}
fastcgi_param REMOTE_ADDR $realip;
Paste it after include fastcgi_params; directive to be effective.
If you're using Cloudflare, you can get original client IP from HTTP_CF_CONNECTING_IP.
I found this article
https://geko.cloud/forward-real-ip-to-a-nginx-behind-a-gcp-load-balancer/
You can whitelist/ignore IPs that are known to GCP like the static ip needed for registering the loadbalancer
set_real_ip_from 36.129.221.25/32; // LB Public IP address
set_real_ip_from 130.211.0.0/22; // Private IP range for GCP Load Balancers
set_real_ip_from 35.191.0.0/16; //Private IP range for GCP Load Balancers
real_ip_header X-Forwarded-For;
real_ip_recursive on;
In the following scenario,
[client]---https--->[Nginx]---http--->[app server]
How (and what) would I pass down to the app server to uniquely identify the certificate? That is, Nginx validates the certificate, but app server doesn't see it. I need to distinguish between users at the app server, so they can't impersonate each other.
You could adapt the same technique as what's described in this question for Apache Httpd. You'd need the Nginx equivalent of something like:
RequestHeader set X-ClientCert ""
RequestHeader set X-ClientCert "%{SSL_CLIENT_CERT}s"
I haven't tried, but the documentation for the Nginx SSL module has a section about "Embedded Variables". More specifically:
$ssl_client_cert returns the client certificate in the PEM format for an established SSL connection, with each line except the first prepended
with the tab character; this is intended for the use in the
proxy_set_header directive;
This looks like what you need with a reverse-proxy setting, like the one you have.
Note that it's very important to clear this header on its way in, otherwise clients could just set the headers themselves and use any certificate they like.
How you then want to check this in your application server depends on the platform you're using. In Java, for example, you could write a Filter (or a Tomcat Valve) that sets the parameter in the request from this custom HTTP header.
It sounds like you want to use Nginx for SSL termination, but you want the backend servers to be able to tell with the original request was over HTTPS or HTTP.
I think something this could work:
server {
ssl on;
listen 443;
add_header X-Forwarded-Proto https;
proxy_pass ...
}
# If you need insecure requests as well
server {
listen 80;
add_header X-Forwarded-Proto http;
proxy_pass ...
}
Then your app server can check the value of the X-Forwarded-Proto header.
This is the same design pattern that Amazon Web Services uses for terminating SSL at their Elastic Load Balancers. They also set the X-Forwarded-Proto header for backend servers to check.