Phoenix in Production on EC2 not rendering in HTTPS with AWS Load Balancer - ssl

I have followed this tutorial to set up my phoenix app on EC2, and later I added the load balancer for SSL.
I used ACM (Amazon Certificate Manager) to get the public certificate and applied on the Amazon Load Balancer (ALB).
I'm still a bit fuzzy on the port mapping, so I suppose it might be the cause.
# config/prod.exs
host = System.get_env("HOST") || "example.com"
config :app_web, AppWeb.Endpoint,
force_ssl: [rewrite_on: [:x_forwarded_proto]],
load_from_system_env: true,
http: [port: 80],
url: [host: host, port: 80],
url: [host: host, port: 443, scheme: "https"],
server: true,
secret_key_base: System.get_env("SECRET_KEY_BASE")
# docker-compose.yml
version: '2'
services:
kroo:
image: [image url]
environment:
- HOST=0.0.0.0
ports:
- '443:443'
- '80:80'
$ docker ps
PORTS
0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp
$ docker logs
01:56:30.177 [info] Running AppWeb.Endpoint with cowboy 2.7.0 at 0.0.0.0:80 (http)
01:56:30.177 [info] Access AppWeb.Endpoint at https://example.com
Running Release tasks
[]
01:56:31.316 [info] Already up
01:56:33.085 [info] Plug.SSL is redirecting GET / to https://example.com with status 301
When I don't include force_ssl: [rewrite_on: [:x_forwarded_proto]], I'm able to have the page displayed fine in http, but when I include force_ssl, it redirects the https which is working fine, but I'm getting unable to connect error.
My confusion is that, since the load balancer is taking care of the SSL, I don't have the key and the certificate for SSL, which is why I don't have https: [] option in prod.exs.
Could someone point out what I'm doing wrong here?
Thanks
UPDATE: I finally got it working, below is my working configs in case anyone would find it helpful.
# config/prod.exs
# https config is not needed since ALB is handling the SSL
# Phoenix app serving in http is fine
config :app_web, AppWeb.Endpoint,
load_from_system_env: true,
http: [port: 8080],
url: [host: "example.com"],
server: true,
secret_key_base: System.get_env("SECRET_KEY_BASE")
# docker-compose.yml
# map phoenix port 8080 to docker 8080
ports:
- '8080:8080'
Since I'm not providing SSL certificates, but I still want to force ssl, like #jamesvl suggested in answer, use your load balancer to redirect http traffic to https.
If you need help setting up SSL on ALB, I followed this guide
If somehow your app still not showing up under your domain, make sure that you have an A Record with an alias map to the DNS name of your load balancer

I would suggest setting the listen port of your docker container to something other than 80, and don't listen on 443 at all.
Rationale
I think the issue may lie in the fact that your http: configuration is listening on port 80.
With force_ssl: enabled, you're indicating that you want http connections to go to port 443, but when something arrives on 443 (via the load balancer), you send it to your (listening) port 80... which redirects it back to 443?
Fix
Let Phoenix listen on an arbitrary port (say... 4010) for http only connections. (Since the load balancer does your SSL termination, all your communication with the load balancer will be over http.) This involves changing your Docker container to forward connections to that port as well - you don't want to listen on 80 or 443 at all in your container.
Your url: configuration would then be looking only at headers, redirecting http requests to https as needed.
By the way, Amazon's ALB can also do 80 -> 443 redirection for you if you setup the rules; this saves Phoenix from even having to have a config url: setup for port 80 at all

Related

HAProxy TCP (443) Loadbalancing with different backend ports

I'm implementing a Frontend Loadbalancer which passthrough the traffic coming to port 80 and 443 to different backend ports. SSL termination is happening in the backend and HAproxy should not engage with anything other than forwarding the traffic coming to the frontend port 80 and 443 to the respective backend ports.
Port 80 forwarding seems fine and 443 is not working as expected and giving SSL handshake failure. Even my backend service is not coming up on the web browser with a warning saying this is not trusted. I have no clue why this is happening and my HAProxy experience is not that high and below is the current configuration. Please correct me if I'm wrong.
HAProxy is installed on Ubuntu 18.04.5 LTS
Config after the defaults section
frontend k8s_lb
mode tcp
bind x.x.x.x:80
default_backend kube_minions
frontend k8s_lb_https
mode tcp
bind x.x.x.x:443
default_backend kube_minions_https
backend kube_minions
mode tcp
balance roundrobin
server k8s_worker-01 x.x.x.x:32080
server k8s_worker-02 x.x.x.x:32080
backend kube_minions_https
mode tcp
balance roundrobin
server k8s_worker-01 x.x.x.x:32443
server k8s_worker-02 x.x.x.x:32443
The backend story:
I have a k8s cluster and traefik ingress which is running as a DaemonSet on each and every node, and minions are my backend servers. CertManager is in place to do the cert automation with Let's encrypt ACME protocol in the ingress resources, hence SSL termination should be happening through the ingress resources.
I have completed the certificates and everything seems perfect as I have already implemented a similar setup on AWS with a TCP loadbalancer and everything is perfectly working and running prod workloads.
So, I need to mention that backend services are all good and up and running. In this I replaced the AWS loadbalancer with HAProxy and need to implement the same.
Please assist me to fix this as I'm struggling with this and still no luck with the issue.
Thank you.
Sorry, I was able to figure it out and there is nothing to do with traefik and HAProxy for this SSL issue. My Client's DNS is configured in CloudFlare and they have enabled the universal SSL and it caused the issue.
I checked with a new DNS record from route53 working as expected so my HAProxy config do what I need.

Don't prepend http:// to Endpoint Subset IP

I have a Kubernetes Ingress, pointing to a headless service, pointing finally to an Endpoints object that routes to an external IP address. The following is the configuration for the endpoint
apiVersion: v1
kind: Endpoints
metadata:
name: my-chart
subsets:
- addresses:
- ip: **.**.**.**
ports:
- port: 443
However, the upstream connection fails with 'connection reset by peer', and on looking at the logs I see the following error in the Kubernetes nginx-ingress-controller:
2020/01/15 14:39:50 [error] 24546#24546: *240425068 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: *****, server: dev.somehost.com, request: "GET / HTTP/1.1", upstream: "http://**.**.**.**:443/", host: "dev.somehost.com"
My theory is that the combination of http:// and the 443 port is what is triggering this (tested with cURL commands). How do I either 1) Specify a different protocol for the endpoint object or 2) just prevent the prepending of http://
Additional notes:
1) SSL is enabled on the target IP, and if I curl it I can set up a secure connection
2) SSL passthrough doesn't really work here. The incoming and outgoing requests will use two different SSL connections with two different certificates.
3) I want the Ingress host to be the SNI (and it looks like this may default to being the case)
Edit: Ingress controller version: 0.21.0-rancher3
We were able to solve this by adding the following to the metadata of our Ingress
annotations:
nginx.ingress.kubernetes.io/backend-protocol: HTTPS
nginx.ingress.kubernetes.io/configuration-snippet: |-
proxy_ssl_server_name on;
proxy_ssl_name $host;
The first command turns on HTTPS for the backend protocol, and the second command enables SNI

Cloudflare SSL Does Not Work With Elixir/Phoenix Backend

I am using Cloudflare's flexible SSL to secure my website, and it works fine when running my vue.js frontend. Essentially what it is doing is sharing the SSL cert with 50 random customers, and it does so by piping my DNS through their edge network. Basically, it did the thing I needed and was fine, but now that I am trying to tie it to a phoenix/elixir backend it is breaking.
The problem is that you can't make an http request from inside an ssl page, because you'll get this error:
Blocked loading mixed active content
This makes sense - if it's ssl on load, it needs to be ssl all the way down. So now I need to add SSL to elixir.
This site (https://elixirforum.com/t/run-phoenix-https-behind-cloudflare-without-key-keyfile/12660/2) seemed to have the solution! Their answer was:
configs = Keyword.put(config, :http, [:inet6, port: "80"])
|> Keyword.put(:url, [scheme: "https", host: hostname, port: "443"])
So I made my config like this:
config :albatross, AlbatrossWeb.Endpoint,
http: [:inet6, port: "4000"],
url: [scheme: "https", host: "my.website", port: "443"],
secret_key_base: "SUPERSECRET",
render_errors: [view: AlbatrossWeb.ErrorView, accepts: ~w(html json)],
pubsub: [name: Albatross.PubSub,
adapter: Phoenix.PubSub.PG2]
That only allows me to get to http:etc! So I've also tried this:
config :albatross, AlbatrossWeb.Endpoint,
http: [:inet6, port: "4000"],
https: [ :inet6, port: "4443"],
url: [scheme: "https", host: "my.website", port: "443"],
secret_key_base: "SUPERSECRET",
render_errors: [view: AlbatrossWeb.ErrorView, accepts: ~w(html json)],
pubsub: [name: Albatross.PubSub,
adapter: Phoenix.PubSub.PG2]
Which doesn't work of course because there's no PEM files. Since I'm only using elixir as an API (and not a DNS) I can't use solutions like this (http://51percent.tech/blog/uncategorized/serving-phoenix-apps-ssl-and-lets-encrypt/), because letsencrypt does not allow IP address only auth (https://www.digitalocean.com/community/questions/ssl-for-ip-address).
So at this point I'm very confused. Does anyone have any advice?
EDIT:
Someone mentioned that you can go on to cloudflare and generate TLS certs by going to crypto>Origin Certificates>Create Certificate. I did that, downloaded the files, saved them in my project and ran this:
config :albatross, AlbatrossWeb.Endpoint,
http: [:inet6, port: "4000"],
https: [ port: "4443",
keyfile: "priv/ssl/cloudflare/private.key",
certfile: "priv/ssl/cloudflare/public.pem"],
url: [scheme: "https", host: "website.me", port: "443"],
secret_key_base: "SUPERSECRET",
render_errors: [view: AlbatrossWeb.ErrorView, accepts: ~w(html json)],
pubsub: [name: Albatross.PubSub,
adapter: Phoenix.PubSub.PG2]
So what are the results of all the possible ways to query the backend?
Well I'm running docker-compose so https://backendservice:4443 is what I query from the frontend. That gives me -
Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at https://backendservice:4443/getComments?postnum=6. (Reason: CORS request did not succeed).[Learn More]
value of error : Error: "Network Error"
exports https://mywebsi.te/js/chunk-vendors.4345f11a.js:49:15423
onerror https://mywebsi.te/js/chunk-vendors.4345f11a.js:65:538540
actions.js:61:12
So that clearly doesn't work.
I can go to http://my.ip.address:4000, but I cannot go to https://my.ip.address:4443.
As far as I can tell cloudflare TLS certificates do not work.
Or, more likely, I am doing something stupid in writing the elixir config.
FURTHER CLARIFICATION:
Yes, there is a CORS header error above. However please note that it is only firing for https request and NOT http requests. Why this is happening is very confusing. I have a cors plugin for elixir in the entrypoint of my application that is currently allowing * incoming requests. This is it - it should be pretty straight forward:
plug CORSPlug, origin: "*"
More information can be found here (https://github.com/mschae/cors_plug).
I seem to have been able to make my website work with the following configuration:
config :my_app, MyAppWeb.Endpoint,
force_ssl: [hsts: true],
url: [host: "my.website", port: 443],
http: [:inet6, port: 4000],
https: [
:inet6,
port: 8443,
cipher_suite: :strong,
keyfile: "private.key.pem",
certfile: "public.cert.pem",
cacertfile: "intermediate.cert.pem"
],
cache_static_manifest: "priv/static/cache_manifest.json"
where private.key.pem and public.cert.pem are the origin certificate downloaded from Cloudflare.
(Note that the origin certificate you can download from Cloudflare is only useful for encrypting the connection between your website and Cloudflare.)
I also had to add routing rules via iptables.
iptables -A PREROUTING -t nat -i eth0 -p tcp --dport 80 -j REDIRECT --to-port 4000
iptables -A PREROUTING -t nat -i eth0 -p tcp --dport 443 -j REDIRECT --to-port 8443
This configuration also worked before for a setup using letsencrypt certificates.
I'm not sure if the cacertfile: "intermediate.cert.pem" part is needed. I obtained it from "Step 4" of the Cloudflare documentation.
I never used Cloudflare but I know that phoenix can pretend it is serving https content if x-forwarded-proto is set in request headers. Your configuration is OK (without https part, you don't need it).
Try to add force_ssl option in endpoint configuration. This will instruct to Plug.SSL to force SSL and redirect any http request to https.
config :albatross, AlbatrossWeb.Endpoint,
http: [:inet6, port: "4000"],
url: [scheme: "https", host: "my.website", port: "443"],
force_ssl: [rewrite_on: [:x_forwarded_proto], host: nil],
secret_key_base: "SUPERSECRET",
render_errors: [view: AlbatrossWeb.ErrorView, accepts: ~w(html json)],
pubsub: [name: Albatross.PubSub,
adapter: Phoenix.PubSub.PG2]
In my experience with Cloudflare and web backends (and without seeing the exact request that is causing the issue), this is usually caused by hard-coding a CSS/JS dependency with http:// or by making an AJAX request with http:// hard-coded. If the back end you are requesting with http:// doesn't automatically redirect to https://, you will get the error you're seeing.
If you're using Cloudflare's "One-Click SSL", this is how requests are being made to your server:
[ Client Browser ] <-- HTTPS --> [ Cloudflare ] <-- HTTP --> [ Your Server ]
Since the communication between Cloudflare and your server is all over http://, you should not need to change any Phoenix configuration at all, but there are 2 potential sources of error:
Automatic redirection to https:// is not being done by Cloudflare (or Cloudflare is not serving your content)
You have some CSS/JS dependencies or Ajax requests that are hard-coded to http:// and the server responding to the requests is not automatically redirecting to https://.
Troubleshooting point 1: According to Cloudflare's docs for their "One-Click SSL", they should automatically redirect http:// requests to https:// (although this might be a setting that you need to change). If a request comes to your domain over http://, Cloudflare should automatically redirect to https://. You can check whether this is happening by requesting a page from your backend in your browser with http:// explicitly included at the front of the URL. If you are not redirected to https://, it could mean that a) Cloudflare is not actually in between your browser and the backend or b) that Cloudflare is not automatically redirecting to https://. In either case, you have some hints about your next step in solving the problem.
Troubleshooting point 2: You can check this by using the "Network" tab in the developer tools in your browser, or by manually going through your website code looking for CSS/JS dependencies or Ajax requests hard-coded with http://. In the developer tools, you can check for exactly which request(s) are causing the problem (the line should be red and have an error about mixed content). From there you can check where this request is coming from in your code. When searching through your code for hard-coded http:// requests, most of the time you can simply replace it with https:// directly, but you should double-check the URL in your browser to make sure that the URL is actually available over https://. If the CSS/JS dependency or Ajax endpoint is not actually available over https:// then you will need to remove/replace it.
Hope this helps.

AWS Beanstalk and Docker ports = what manner of tomfoolery is this?

So I have a docker application that runs on port 9000, and I'd like to have this accessed only via https rather than http, however I don't appear to be making any sense of how amazon handles ports. In short I'd like only expose port 443 and not 80 (on the load balancer layer and the instance layer), but haven't been able to do this.
So my Dockerfile has:
EXPOSE 9000
and my Dockerrun.aws.json has:
{
"AWSEBDockerrunVersion": "1",
"Ports": [{
"ContainerPort": "9000"
}]
}
and I cannot seem to access things via port 9000, but by 80 only.
When I ssh into the instance that the docker container is running and look for the ports with netstat I get ports 80 and 22 and some other udp ports, but no port 9000. How on earth does Amazon manage this? More importantly how does a user get expected behaviour?
Attempting this with ssl and https also yields the same thing. Certificates are set and mapped to port 443, I have even created a case in the .ebextensions config file to open port 443 on the instance and still no ssl
sslSecurityGroupIngress:
Type: AWS::EC2::SecurityGroupIngress
Properties:
GroupName: {Ref : AWSEBSecurityGroup}
IpProtocol: tcp
ToPort: 443
FromPort: 443
CidrIp: 0.0.0.0/0
The only way that I can get SSL to work is to have the Load Balancer use port 443 (ssl) forwarding to the instance port 80 (non https) but this is ridiculous. How on earth do I open the ssl port on the instance and set docker to use the given port? Has anyone ever done this successfully?
I'd appreciate any help on this - I've combed through the docs and got this far with it, but this just plain puzzles me. In short I'd like only expose port 443 and not 80 (on the load balancer layer and the instance layer), but haven't been able to do this.
Have a great day
Cheers
It's known problem, from http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/create_deploy_docker_image.html:
You can specify multiple container ports, but Elastic Beanstalk uses only the first one to connect your container to the host's reverse proxy and route requests from the public Internet.
So, if you need multiple ports, AWS Elastic Beanstalk is probably not the best choice. At least Docker option.
Regarding SSL - we solved it by using dedicated nginx instance and proxy_pass'ing to Elastic Beanstalk environment URL.

AWS - SSL/HTTPS on load balancer

I have a problem to add https to my EC2 instance and maybe you guys can have the answer to make it work.
I have a load balancer that is forwarding the connection to my EC2 instance, I've add the SSL certificate to the load balancer and everything went fine, I've add a listener to the port 443 that will forward to the port 443 of my instance and I've configured Apache to listen on both port 443 and 80, now here the screenshot of my load balancer:
The SSL certificate is valid and on port 80 (HTTP) everything is fine, but if I try the with https the request does not got through.
Any idea?
Cheers
Elastic Load Balancer can not forward your HTTPS requests to the server. This is why SSL is there : to prevent a man in the middle attack (amongst others)
The way you can get this working is the following :
configure your ELB to accept 443 TCP connection and install an SSL certificate through IAM (just like you did)
relay traffic on TCP 80 to your fleet of web servers
configure your web server to accept traffic on TCP 80 (having SSL between the load balancer and the web servers is also supported, but not required most of the time)
configure your web servers Security Group to only accept traffic from the load balancer.
(optional) be sure your Web Servers are running in a private subnet, i.e. with only private IP addressed and no route to the Internet Gateway
If you really need to have an end-to-end SSL tunnel between your client and you backend servers (for example, to perform client side SSL authentication), then you'll have to configure your load balancer in TCP mode, not in HTTP mode (see Support for two-way TLS/HTTPS with ELB for more details)
More details :
SSL Load Balancers : http://docs.aws.amazon.com/ElasticLoadBalancing/latest/DeveloperGuide/US_SettingUpLoadBalancerHTTPS.html
Load Balancers in VPC :
http://docs.aws.amazon.com/ElasticLoadBalancing/latest/DeveloperGuide/UserScenariosForVPC.html
Do you have an HTTPS listener on your EC2 instance? If not, your instance port should be 80 for both load balancer listeners.