I have a GCP instance group with 2 instances. Both are up and running. I want to configure a load balancer (HTTPS) to manage the traffic.
I've set up a forwarding rule with the HTTP-protocol and a certificate managed by google. This all works, but only when the traffic between the load balancer and the backend (the instances) is plain HTTP.
Steps I did so far
I create a template and this template is just a normal N1 series machine. I checked the boxes to create firewall rules for allowing http and https traffic.
I create a firewall rule named "allow-ports". This firewall rule targets all instances in the network, has a 0.0.0.0/0 IP-range and allow port tcp = 80, 443. How I see this, this firewall rule should open both the http (80) and https (443) port.
I create an instance group with port mapping. "http-port" = 80, "https-port" = 443. I use the template I just created.
When the instance group is created, I check if this is running. With SSH, I get access to the instances and install apache (sudo apt-get install -y apache2) on the both. When navigating to their external IP's in the browser, I see them both.
I create a HTTP(S) load balancer, with the option "From internet to my VMs". For backend configuration, I add a backend service with my instance group, protocol HTTP, named port "http-port". For frontend configuration, I set up the HTTPS protocol, create an IPv4 IP address, create a google-managed ssl certificate, and I'm done. I also added health checks btw.
Now... these steps work (after a few minutes). With the cloud DNS, I have set up a domain name which points to the IP address of the load balancer. When going to , I see the apache page.
What doesn't work?
When I change the backend configuration to HTTPS (and named port "https-port"), I get a 502 server error. So it seems to me that there is some connection, but there is an error. Could this be an apache error?
I have spent a whole day, creating and recreating instance groups, firewall rules, load balancers, ... but nothing seems to work. I'm surely missing something, probably something dumb, but I have no clue what it could be.
What do I want to achieve?
I do not only want a secure (HTTPS) connection between the client and my load balancer, I also want a secure connection between the load balancer and the backend service (the instance group). Because GCP offers the option to use the HTTPS protocol when creating a backend service, I feel that this could be done.
To be honest: I'm reading some articles about the fact that the internal traffic is secured, so a HTTPS connection is not necessary. But that doesn't matter to me, I really want to know how this works!
EDIT
I'm using the correct VPC (default). I also edited the firewall rule from 0.0.0.0/0 to 130.211.0.0/22 and 35.191.0.0/16 (see: https://cloud.google.com/compute/docs/tutorials/globally-autoscaling-a-web-service-on-compute-engine?hl=nl#configure_the_load_balancer).
In addition to my previous comment. I followed your steps at my test project to find out the cause of your issue. I installed the same configuration and checked it with HTTP at the back-end. As it was expected, I found no errors. After that, I installed SSL certificates to the back-end and to the load balancer. Then I switched my back-end, load balancer and health checks to HTTPS and disabled HTTP at the back-end. At this point, I found no errors also.
So, I decided to get 502 error in my test configuration in some way. I switched my health check at the load balancer to HTTP. A few minutes later I tried to reach my test service again and got 502 error. When I switched back my health check to HTTPS 502 error gone away.
During this test, I didn't change firewall rules, but allowed HTTP and HTTPS traffic in my instance template and I used default network.
Related
I'm trying to figure out the best way to manage HTTPS for an EB docker application.
At the moment I'm using the following approach.
ELB accepts HTTPS connections on 443, and forwards to HTTP port 80 on the instance.
ELB accepts HTTP connections on 80, and forwards to HTTP port 8080 on the instance.
Instance accepts HTTP connections on port 80 and forwards to docker app.
Instance accepts HTTP connections on port 8080 and redirects them to HTTPS.
This all works reasonably well. It means the docker app doesn't have to worry about redirects at all. It just listens on port 80, and does it's thing. The ELB and docker host do the rest.
My only concern with this setup, is the docker app doesn't know that it's running secure. If within my application I check for this, it'll fail.
I want to completely separate my docker app from domain names, and SSL certificates that may change, so I would prefer to continue terminating the original HTTPS connection at the ELB. I'm wondering if there is some way I can get the docker host (or ELB) to forward (re-encrypt) request in HTTPS protocol, but using a self-signed certificate, so I can keep it completely generic.
Just to be clear, this would only be between the ELB and/or docker host, and my docker app, not to the browser.
If I create a non-expiring self-signed certificate, and then register this with the web-server in the docker app (currently using Apache2, but could potentially use nginx), and then simply tell the ELB, or docker host to forward requests as HTTPS, will this work? Or would it fall over at some point because the certificate isn't trusted?
Or is there some way to be able to terminate a HTTPS connection at the docker app web-server without actually needing to pre-generate a certificate (I'm guessing no on this, as presumably it'd need to generate a certificate on the fly or something).
Is there a recommended best practice way to do this kind of thing?
A common solution when you have a load balancer terminating client connections and forwarding to backend is for the load balancer to add headers onto the backend requests to fill in any information stripped by having the load balancer there.
ELB has a page on this and uses the following headers:
X-Forwarded-For - The client IP
X-Forwarded-Proto - The scheme/protocol
X-Forwarded-Port - The incoming port.
You would generally not allow these headers directly from the client, unless they were a trusted client. I assume ELB takes care of that for you.
I am trying to use a load balancer to direct traffic to a container backend. The service in the containers hosts web traffic on port 80. My health checks are all passing. If I ssh into the Kubernetes host for the containers, I can curl each container and get correct responses over port 80. When I try to access them through the load balancers external IP, however, I receive a 502 response. I have firewall rules allowing traffic from 130.211.0.0/22 on tcp:1-5000 and on the NodePort port. I've also tried adding firewall rules from 0.0.0.0/0 ports 80 and 443 to those nodes.
When in the Kubernetes host, capturing with tcpdump, I see the health check requests to my containers, but no traffic is coming through when I make an external request.
I have an identical configuration that points to a single Compute Engine VM that works perfectly. This leads me to believe that the issue might be in the container setup rather than the load balancer.
Does anyone have any advice on resolving this issue?
I was able to resolve the problem by changing the Named Port that the Load Balancer was connecting to. By default, the Load Balancer connected to Named Port "http", which pointed to port 80. It was my assumption (always a bad thing) that this matched since my application serves on port 80. Not so. Since I'd exposed the containers though NodePort, it was assigned another port. This is the port I had my Health Check pointing to. By going into "Compute Engine -> Instance groups" selecting the group, and then "Edit Group", I was able to change the Named Port "http" to match my NodePort number. Once I did that, traffic started flowing.
I have uploaded my SSL certificates to IAM purchased from Comodo and evrything looks fine in chrome and opera. But mozilla is giving an error: "Connection Partially encrypted". I am not able gauge why this is happening.
Link : https://www.advisorcircuit.com/
Please tell me what is the possible culprit for this?
and also i want to know , how can i redirect my users to HTTPS ebven if they type http as even if i type http the website loads and opens.
I am using AWS t2.medium instance. So is there any configuration i need to do in my console??
Redirection:
You have a few options:
Block HTTP traffic, only allow HTTPS on the Security Group level ( Not the nicest solution.
Use an Elastic Load balancer, Listening only on HTTPS port. ( Same as above)
The webserver ( most of them like Tomcat, IIS, etc) supports a redirection, so it sends back "HTTP/1.1 301 Moved Permanently", then the client browser does the call again on HTTPS.
If you use Elastic Load Balancer with SSL termination ( which is a good practice, less load on your server, easier setup of the SSL Certificate). Then all your traffic inside your VPC goes on port 80. In this case you need to setup your webserver to redirect differently. Instead of the incoming port, the trigger for the redirection should be the based on the "X-Forwarded-Proto" header value, which is the original protocol what the client is using.
For production environment the last setup is an AWS Best practice. ( Of course there are also other solutions)
Your site is running Apache/2.2.29. You can redirect your virtual host traffic from 80->443 in Apache itself. That way if someone goes to http://www.yourdomain.com then get redirected to https://www.yourdomain.com
ServerFault has an post explaining how to use Apache mod_rewrite to accomplish this
https://serverfault.com/a/554183/280448
Also you need to adjust the SSL cipher suites that your site accepts. Your ELB has an option to change cipher suites and you can deselect some there. The two you definitely want deselected are RC4 and SSL3.
Here's the full report if you want to make more changes
https://www.ssllabs.com/ssltest/analyze.html?d=www.advisorcircuit.com&s=52.7.154.196&latest
When I setup the LBS on the aws, I set SSL offload to forward requests from port 443 on ELB to port 80 on the EC2 instances.
I am not sure this is right or not since I saw many people did that. But after I did that I got lots of errors within the browser console:
"This request has been blocked; the content must be served over HTTPS."
Should I change the forward port 80 to 443? and install ssl for each instance?
443 to 80 is correct. Your EC2 instance serves plain data, the ELB encrypts it before sending it out of AWS.
There are a few nuances to this- the server thinks it is serving plain content on 80, so if it creates URLs they will typically be http:// without some configuration. (it depends on your framework, server, etc)
The second nuance is that hardcoded URLs will break it in a similar manner. That error message has been explained in a stackoverflow question about Ajax, some form of that is causing your problem.
For reasons we're not going to change, our application needs to handle the SSL connection, and not the ELB. The goal of using the Proxy Protocol is to get the client's IP address over an SSL connection.
http://aws.typepad.com/aws/2013/07/elastic-load-balancing-adds-support-for-proxy-protocol.html?ref_=9 indicates "Alternatively, you can use it if you are sending HTTPS requests and do not want to terminate the SSL connection on the load balancer. For more information, please visit the Elastic Load Balancing Guide."
Unfortunately, it appears the guide that's linked to doesn't actually elaborate on this, and the basic documentation for the Proxy Protocol ( http://docs.aws.amazon.com/ElasticLoadBalancing/latest/DeveloperGuide/enable-proxy-protocol.html ) fails in our environment when configured as described.
Does anyone have steps or a link for this?
The proxy protocol (version 1) injects a single line into the data stream at the beginning of the connection, before SSL is negotiated by your server. You don't get this information "over" an SSL connection; you get the information prior to SSL handshaking. Your server has to implement this capability and specifically be configured so that it can accept and understand it. For an IPv4 connection, it looks like this:
PROXY TCP4 source-ip dest-ip source-port dest-port\r\n
The standard for the protocol is here:
http://haproxy.1wt.eu/download/1.5/doc/proxy-protocol.txt
Additional info in the ELB docs here:
http://docs.aws.amazon.com/ElasticLoadBalancing/latest/DeveloperGuide/TerminologyandKeyConcepts.html#proxy-protocol
Regarding Apache support, at least at the time AWS announced support for the proxy protocol...
“neither Apache nor Nginx currently support the Proxy Protocol header inserted by the ELB”
— http://aws.typepad.com/aws/2013/07/elastic-load-balancing-adds-support-for-proxy-protocol.html?ref_=9
That is subject to change, of course, but I didn't successfully google for any Apache support of the proxy protocol. Of course, since Apache is open source, you could presumably hack it in there, though I am unfamiliar with the Apache source code.
Realizing that you don't want to change what you're doing now, I still would suggest that depending on your motivation for not wanting to change, there may still be a relatively simple solution. It's a change, but not involving SSL on ELB. What about running HAProxy behind ELB to terminate the SSL in front of Apache? Since HAProxy 1.5 can terminate SSL and appears to be able to translate the proxy protocol string from ELB into an X-Forwarded-For header, as well as generate X-SSL headers to give your application information about the client's SSL cert (perhaps that's your motivation for terminating SSL at the app server instead of on the ELB?) ... so this might be an alternative.
Otherwise, I don't have suggestions unless Apache implements support in the future, or we can find some documentation to indicate that they already have.
For the newer Network Load Balancers which allow your application servers to terminate the TLS connections, you can still get the real IP addresses of your clients and avoid all the work of configuring proxy protocol on the ELBs and in the web server config by simply configuring the target groups to use the servers' instance ids rather than their IP addresses. Regardless of which web server you use, the real IPs of the clients will show up in the logs with no translation needed.
Just to follow up on Michael - sqlbot's answer discussing the AWS support for proxy protocol on EC2 instances behind classic TCP elastic load balancers, the Apache module to use that implements the proxy protocol is mod_remoteip. Enabling it and updating the configuration properly will correct the problem of logging IP addresses of users rather than the elastic load balancer's IPs.
To enable proxy protocol on the elastic load balancer you could use these aws cli commands described in the aws documentation:
aws elb create-load-balancer-policy --load-balancer-name my-elb-name --policy-name my-elb-name-ProxyProtocol-policy --policy-type-name ProxyProtocolPolicyType --policy-attributes AttributeName=ProxyProtocol,AttributeValue=true
aws elb set-load-balancer-policies-for-backend-server --load-balancer-name my-elb-name --instance-port 443 --policy-names my-elb-name-ProxyProtocol-policy
aws elb set-load-balancer-policies-for-backend-server --load-balancer-name my-elb-name --instance-port 80 --policy-names my-elb-name-ProxyProtocol-policy
To enable use of proxy protocol in apache, in a server-wide or VirtualHost context, follow the mod_remoteip documentation such as below:
<IfModule mod_remoteip.c>
RemoteIPProxyProtocol On
RemoteIPHeader X-Forwarded-For
# The IPs or IP range of your ELB:
RemoteIPInternalProxy 192.168.1.0/24
# The IPs of hosts that may need to connect directly to the web server, bypassing the ELB (if applicable):
RemoteIPProxyProtocolExceptions 127.0.0.1
</IfModule>
You'll need to update the LogFormat wherever you have those defined (e.g. httpd.conf) to use %a rather than %h else the load balancer IP addresses will still appear.