I'm implementing a Frontend Loadbalancer which passthrough the traffic coming to port 80 and 443 to different backend ports. SSL termination is happening in the backend and HAproxy should not engage with anything other than forwarding the traffic coming to the frontend port 80 and 443 to the respective backend ports.
Port 80 forwarding seems fine and 443 is not working as expected and giving SSL handshake failure. Even my backend service is not coming up on the web browser with a warning saying this is not trusted. I have no clue why this is happening and my HAProxy experience is not that high and below is the current configuration. Please correct me if I'm wrong.
HAProxy is installed on Ubuntu 18.04.5 LTS
Config after the defaults section
frontend k8s_lb
mode tcp
bind x.x.x.x:80
default_backend kube_minions
frontend k8s_lb_https
mode tcp
bind x.x.x.x:443
default_backend kube_minions_https
backend kube_minions
mode tcp
balance roundrobin
server k8s_worker-01 x.x.x.x:32080
server k8s_worker-02 x.x.x.x:32080
backend kube_minions_https
mode tcp
balance roundrobin
server k8s_worker-01 x.x.x.x:32443
server k8s_worker-02 x.x.x.x:32443
The backend story:
I have a k8s cluster and traefik ingress which is running as a DaemonSet on each and every node, and minions are my backend servers. CertManager is in place to do the cert automation with Let's encrypt ACME protocol in the ingress resources, hence SSL termination should be happening through the ingress resources.
I have completed the certificates and everything seems perfect as I have already implemented a similar setup on AWS with a TCP loadbalancer and everything is perfectly working and running prod workloads.
So, I need to mention that backend services are all good and up and running. In this I replaced the AWS loadbalancer with HAProxy and need to implement the same.
Please assist me to fix this as I'm struggling with this and still no luck with the issue.
Thank you.
Sorry, I was able to figure it out and there is nothing to do with traefik and HAProxy for this SSL issue. My Client's DNS is configured in CloudFlare and they have enabled the universal SSL and it caused the issue.
I checked with a new DNS record from route53 working as expected so my HAProxy config do what I need.
I have a Kubernetes Ingress, pointing to a headless service, pointing finally to an Endpoints object that routes to an external IP address. The following is the configuration for the endpoint
apiVersion: v1
kind: Endpoints
metadata:
name: my-chart
subsets:
- addresses:
- ip: **.**.**.**
ports:
- port: 443
However, the upstream connection fails with 'connection reset by peer', and on looking at the logs I see the following error in the Kubernetes nginx-ingress-controller:
2020/01/15 14:39:50 [error] 24546#24546: *240425068 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: *****, server: dev.somehost.com, request: "GET / HTTP/1.1", upstream: "http://**.**.**.**:443/", host: "dev.somehost.com"
My theory is that the combination of http:// and the 443 port is what is triggering this (tested with cURL commands). How do I either 1) Specify a different protocol for the endpoint object or 2) just prevent the prepending of http://
Additional notes:
1) SSL is enabled on the target IP, and if I curl it I can set up a secure connection
2) SSL passthrough doesn't really work here. The incoming and outgoing requests will use two different SSL connections with two different certificates.
3) I want the Ingress host to be the SNI (and it looks like this may default to being the case)
Edit: Ingress controller version: 0.21.0-rancher3
We were able to solve this by adding the following to the metadata of our Ingress
annotations:
nginx.ingress.kubernetes.io/backend-protocol: HTTPS
nginx.ingress.kubernetes.io/configuration-snippet: |-
proxy_ssl_server_name on;
proxy_ssl_name $host;
The first command turns on HTTPS for the backend protocol, and the second command enables SNI
I am using Cloudflare's flexible SSL to secure my website, and it works fine when running my vue.js frontend. Essentially what it is doing is sharing the SSL cert with 50 random customers, and it does so by piping my DNS through their edge network. Basically, it did the thing I needed and was fine, but now that I am trying to tie it to a phoenix/elixir backend it is breaking.
The problem is that you can't make an http request from inside an ssl page, because you'll get this error:
Blocked loading mixed active content
This makes sense - if it's ssl on load, it needs to be ssl all the way down. So now I need to add SSL to elixir.
This site (https://elixirforum.com/t/run-phoenix-https-behind-cloudflare-without-key-keyfile/12660/2) seemed to have the solution! Their answer was:
configs = Keyword.put(config, :http, [:inet6, port: "80"])
|> Keyword.put(:url, [scheme: "https", host: hostname, port: "443"])
So I made my config like this:
config :albatross, AlbatrossWeb.Endpoint,
http: [:inet6, port: "4000"],
url: [scheme: "https", host: "my.website", port: "443"],
secret_key_base: "SUPERSECRET",
render_errors: [view: AlbatrossWeb.ErrorView, accepts: ~w(html json)],
pubsub: [name: Albatross.PubSub,
adapter: Phoenix.PubSub.PG2]
That only allows me to get to http:etc! So I've also tried this:
config :albatross, AlbatrossWeb.Endpoint,
http: [:inet6, port: "4000"],
https: [ :inet6, port: "4443"],
url: [scheme: "https", host: "my.website", port: "443"],
secret_key_base: "SUPERSECRET",
render_errors: [view: AlbatrossWeb.ErrorView, accepts: ~w(html json)],
pubsub: [name: Albatross.PubSub,
adapter: Phoenix.PubSub.PG2]
Which doesn't work of course because there's no PEM files. Since I'm only using elixir as an API (and not a DNS) I can't use solutions like this (http://51percent.tech/blog/uncategorized/serving-phoenix-apps-ssl-and-lets-encrypt/), because letsencrypt does not allow IP address only auth (https://www.digitalocean.com/community/questions/ssl-for-ip-address).
So at this point I'm very confused. Does anyone have any advice?
EDIT:
Someone mentioned that you can go on to cloudflare and generate TLS certs by going to crypto>Origin Certificates>Create Certificate. I did that, downloaded the files, saved them in my project and ran this:
config :albatross, AlbatrossWeb.Endpoint,
http: [:inet6, port: "4000"],
https: [ port: "4443",
keyfile: "priv/ssl/cloudflare/private.key",
certfile: "priv/ssl/cloudflare/public.pem"],
url: [scheme: "https", host: "website.me", port: "443"],
secret_key_base: "SUPERSECRET",
render_errors: [view: AlbatrossWeb.ErrorView, accepts: ~w(html json)],
pubsub: [name: Albatross.PubSub,
adapter: Phoenix.PubSub.PG2]
So what are the results of all the possible ways to query the backend?
Well I'm running docker-compose so https://backendservice:4443 is what I query from the frontend. That gives me -
Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at https://backendservice:4443/getComments?postnum=6. (Reason: CORS request did not succeed).[Learn More]
value of error : Error: "Network Error"
exports https://mywebsi.te/js/chunk-vendors.4345f11a.js:49:15423
onerror https://mywebsi.te/js/chunk-vendors.4345f11a.js:65:538540
actions.js:61:12
So that clearly doesn't work.
I can go to http://my.ip.address:4000, but I cannot go to https://my.ip.address:4443.
As far as I can tell cloudflare TLS certificates do not work.
Or, more likely, I am doing something stupid in writing the elixir config.
FURTHER CLARIFICATION:
Yes, there is a CORS header error above. However please note that it is only firing for https request and NOT http requests. Why this is happening is very confusing. I have a cors plugin for elixir in the entrypoint of my application that is currently allowing * incoming requests. This is it - it should be pretty straight forward:
plug CORSPlug, origin: "*"
More information can be found here (https://github.com/mschae/cors_plug).
I seem to have been able to make my website work with the following configuration:
config :my_app, MyAppWeb.Endpoint,
force_ssl: [hsts: true],
url: [host: "my.website", port: 443],
http: [:inet6, port: 4000],
https: [
:inet6,
port: 8443,
cipher_suite: :strong,
keyfile: "private.key.pem",
certfile: "public.cert.pem",
cacertfile: "intermediate.cert.pem"
],
cache_static_manifest: "priv/static/cache_manifest.json"
where private.key.pem and public.cert.pem are the origin certificate downloaded from Cloudflare.
(Note that the origin certificate you can download from Cloudflare is only useful for encrypting the connection between your website and Cloudflare.)
I also had to add routing rules via iptables.
iptables -A PREROUTING -t nat -i eth0 -p tcp --dport 80 -j REDIRECT --to-port 4000
iptables -A PREROUTING -t nat -i eth0 -p tcp --dport 443 -j REDIRECT --to-port 8443
This configuration also worked before for a setup using letsencrypt certificates.
I'm not sure if the cacertfile: "intermediate.cert.pem" part is needed. I obtained it from "Step 4" of the Cloudflare documentation.
I never used Cloudflare but I know that phoenix can pretend it is serving https content if x-forwarded-proto is set in request headers. Your configuration is OK (without https part, you don't need it).
Try to add force_ssl option in endpoint configuration. This will instruct to Plug.SSL to force SSL and redirect any http request to https.
config :albatross, AlbatrossWeb.Endpoint,
http: [:inet6, port: "4000"],
url: [scheme: "https", host: "my.website", port: "443"],
force_ssl: [rewrite_on: [:x_forwarded_proto], host: nil],
secret_key_base: "SUPERSECRET",
render_errors: [view: AlbatrossWeb.ErrorView, accepts: ~w(html json)],
pubsub: [name: Albatross.PubSub,
adapter: Phoenix.PubSub.PG2]
In my experience with Cloudflare and web backends (and without seeing the exact request that is causing the issue), this is usually caused by hard-coding a CSS/JS dependency with http:// or by making an AJAX request with http:// hard-coded. If the back end you are requesting with http:// doesn't automatically redirect to https://, you will get the error you're seeing.
If you're using Cloudflare's "One-Click SSL", this is how requests are being made to your server:
[ Client Browser ] <-- HTTPS --> [ Cloudflare ] <-- HTTP --> [ Your Server ]
Since the communication between Cloudflare and your server is all over http://, you should not need to change any Phoenix configuration at all, but there are 2 potential sources of error:
Automatic redirection to https:// is not being done by Cloudflare (or Cloudflare is not serving your content)
You have some CSS/JS dependencies or Ajax requests that are hard-coded to http:// and the server responding to the requests is not automatically redirecting to https://.
Troubleshooting point 1: According to Cloudflare's docs for their "One-Click SSL", they should automatically redirect http:// requests to https:// (although this might be a setting that you need to change). If a request comes to your domain over http://, Cloudflare should automatically redirect to https://. You can check whether this is happening by requesting a page from your backend in your browser with http:// explicitly included at the front of the URL. If you are not redirected to https://, it could mean that a) Cloudflare is not actually in between your browser and the backend or b) that Cloudflare is not automatically redirecting to https://. In either case, you have some hints about your next step in solving the problem.
Troubleshooting point 2: You can check this by using the "Network" tab in the developer tools in your browser, or by manually going through your website code looking for CSS/JS dependencies or Ajax requests hard-coded with http://. In the developer tools, you can check for exactly which request(s) are causing the problem (the line should be red and have an error about mixed content). From there you can check where this request is coming from in your code. When searching through your code for hard-coded http:// requests, most of the time you can simply replace it with https:// directly, but you should double-check the URL in your browser to make sure that the URL is actually available over https://. If the CSS/JS dependency or Ajax endpoint is not actually available over https:// then you will need to remove/replace it.
Hope this helps.
So I have a docker application that runs on port 9000, and I'd like to have this accessed only via https rather than http, however I don't appear to be making any sense of how amazon handles ports. In short I'd like only expose port 443 and not 80 (on the load balancer layer and the instance layer), but haven't been able to do this.
So my Dockerfile has:
EXPOSE 9000
and my Dockerrun.aws.json has:
{
"AWSEBDockerrunVersion": "1",
"Ports": [{
"ContainerPort": "9000"
}]
}
and I cannot seem to access things via port 9000, but by 80 only.
When I ssh into the instance that the docker container is running and look for the ports with netstat I get ports 80 and 22 and some other udp ports, but no port 9000. How on earth does Amazon manage this? More importantly how does a user get expected behaviour?
Attempting this with ssl and https also yields the same thing. Certificates are set and mapped to port 443, I have even created a case in the .ebextensions config file to open port 443 on the instance and still no ssl
sslSecurityGroupIngress:
Type: AWS::EC2::SecurityGroupIngress
Properties:
GroupName: {Ref : AWSEBSecurityGroup}
IpProtocol: tcp
ToPort: 443
FromPort: 443
CidrIp: 0.0.0.0/0
The only way that I can get SSL to work is to have the Load Balancer use port 443 (ssl) forwarding to the instance port 80 (non https) but this is ridiculous. How on earth do I open the ssl port on the instance and set docker to use the given port? Has anyone ever done this successfully?
I'd appreciate any help on this - I've combed through the docs and got this far with it, but this just plain puzzles me. In short I'd like only expose port 443 and not 80 (on the load balancer layer and the instance layer), but haven't been able to do this.
Have a great day
Cheers
It's known problem, from http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/create_deploy_docker_image.html:
You can specify multiple container ports, but Elastic Beanstalk uses only the first one to connect your container to the host's reverse proxy and route requests from the public Internet.
So, if you need multiple ports, AWS Elastic Beanstalk is probably not the best choice. At least Docker option.
Regarding SSL - we solved it by using dedicated nginx instance and proxy_pass'ing to Elastic Beanstalk environment URL.
I have a problem to add https to my EC2 instance and maybe you guys can have the answer to make it work.
I have a load balancer that is forwarding the connection to my EC2 instance, I've add the SSL certificate to the load balancer and everything went fine, I've add a listener to the port 443 that will forward to the port 443 of my instance and I've configured Apache to listen on both port 443 and 80, now here the screenshot of my load balancer:
The SSL certificate is valid and on port 80 (HTTP) everything is fine, but if I try the with https the request does not got through.
Any idea?
Cheers
Elastic Load Balancer can not forward your HTTPS requests to the server. This is why SSL is there : to prevent a man in the middle attack (amongst others)
The way you can get this working is the following :
configure your ELB to accept 443 TCP connection and install an SSL certificate through IAM (just like you did)
relay traffic on TCP 80 to your fleet of web servers
configure your web server to accept traffic on TCP 80 (having SSL between the load balancer and the web servers is also supported, but not required most of the time)
configure your web servers Security Group to only accept traffic from the load balancer.
(optional) be sure your Web Servers are running in a private subnet, i.e. with only private IP addressed and no route to the Internet Gateway
If you really need to have an end-to-end SSL tunnel between your client and you backend servers (for example, to perform client side SSL authentication), then you'll have to configure your load balancer in TCP mode, not in HTTP mode (see Support for two-way TLS/HTTPS with ELB for more details)
More details :
SSL Load Balancers : http://docs.aws.amazon.com/ElasticLoadBalancing/latest/DeveloperGuide/US_SettingUpLoadBalancerHTTPS.html
Load Balancers in VPC :
http://docs.aws.amazon.com/ElasticLoadBalancing/latest/DeveloperGuide/UserScenariosForVPC.html
Do you have an HTTPS listener on your EC2 instance? If not, your instance port should be 80 for both load balancer listeners.