How to Correct 'nginx: [emerg] "stream" directive is not allowed here' - ssl

The Question
Why does the following Nginx configuration return nginx: [emerg] "stream" directive is not allowed here in /etc/nginx/sites-enabled/default:1?
Nginx Configuration...
stream {
map $ssl_preread_server_name $upstream {
example.com 1051;
}
upstream 1051 {
server 127.0.0.1:1051;
}
server {
listen 443;
proxy_pass $upstream;
ssl_preread on;
}
}
Version / Build information...
OS: Debian 10
Here is the stripped down nginx -V output confirming the presence of the modules I understand I need...
nginx version: nginx/1.14.2
TLS SNI support enabled
configure arguments: ... --with-stream=dynamic --with-stream_ssl_module --with-stream_ssl_preread_module ...
The Context
I have a single static IP address. At the static IP address, I am setting up a reverse proxy Nginx server to forward traffic to a variety of backend services. Several of the services are websites with unique domain names.
+-----+ +----------------------+ +---------+
| WAN | <----> | Nginx Reverse Proxy | <----> | Service |
+-----+ +----------------------+ +---------+
At boot, the service uses systemd to run this port forwarding ssh command to connect to the reverse proxy: ssh -N -R 1051:localhost:443 tunnel#example.com (That is working well.)
I want the certificate to reside on the service - not the reverse proxy. From what I understand I need to leverage SNI on Nginx to passthrough the SSL connections bases on domain name. But I cannot get the Nginx reverse proxy to passthrough SSL.
Resources
Here are a few of the resources I have pored over...
https://serverfault.com/questions/625362/can-a-reverse-proxy-use-sni-with-ssl-pass-through
https://nginx.org/en/docs/stream/ngx_stream_core_module.html
https://nginx.org/en/docs/stream/ngx_stream_ssl_preread_module.html
https://www.amitnepal.com/nginx-ssl-passthrough-reverse-proxy
https://serverfault.com/questions/1049158/nginx-how-to-combine-ssl-preread-protocol-with-ssl-preread-server-name-ssh-mul

The problem was I tried to embed a stream block inside an http block. I was not properly accounting for the include in /etc/nginx/nignx.conf file.

Related

Phoenix in Production on EC2 not rendering in HTTPS with AWS Load Balancer

I have followed this tutorial to set up my phoenix app on EC2, and later I added the load balancer for SSL.
I used ACM (Amazon Certificate Manager) to get the public certificate and applied on the Amazon Load Balancer (ALB).
I'm still a bit fuzzy on the port mapping, so I suppose it might be the cause.
# config/prod.exs
host = System.get_env("HOST") || "example.com"
config :app_web, AppWeb.Endpoint,
force_ssl: [rewrite_on: [:x_forwarded_proto]],
load_from_system_env: true,
http: [port: 80],
url: [host: host, port: 80],
url: [host: host, port: 443, scheme: "https"],
server: true,
secret_key_base: System.get_env("SECRET_KEY_BASE")
# docker-compose.yml
version: '2'
services:
kroo:
image: [image url]
environment:
- HOST=0.0.0.0
ports:
- '443:443'
- '80:80'
$ docker ps
PORTS
0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp
$ docker logs
01:56:30.177 [info] Running AppWeb.Endpoint with cowboy 2.7.0 at 0.0.0.0:80 (http)
01:56:30.177 [info] Access AppWeb.Endpoint at https://example.com
Running Release tasks
[]
01:56:31.316 [info] Already up
01:56:33.085 [info] Plug.SSL is redirecting GET / to https://example.com with status 301
When I don't include force_ssl: [rewrite_on: [:x_forwarded_proto]], I'm able to have the page displayed fine in http, but when I include force_ssl, it redirects the https which is working fine, but I'm getting unable to connect error.
My confusion is that, since the load balancer is taking care of the SSL, I don't have the key and the certificate for SSL, which is why I don't have https: [] option in prod.exs.
Could someone point out what I'm doing wrong here?
Thanks
UPDATE: I finally got it working, below is my working configs in case anyone would find it helpful.
# config/prod.exs
# https config is not needed since ALB is handling the SSL
# Phoenix app serving in http is fine
config :app_web, AppWeb.Endpoint,
load_from_system_env: true,
http: [port: 8080],
url: [host: "example.com"],
server: true,
secret_key_base: System.get_env("SECRET_KEY_BASE")
# docker-compose.yml
# map phoenix port 8080 to docker 8080
ports:
- '8080:8080'
Since I'm not providing SSL certificates, but I still want to force ssl, like #jamesvl suggested in answer, use your load balancer to redirect http traffic to https.
If you need help setting up SSL on ALB, I followed this guide
If somehow your app still not showing up under your domain, make sure that you have an A Record with an alias map to the DNS name of your load balancer
I would suggest setting the listen port of your docker container to something other than 80, and don't listen on 443 at all.
Rationale
I think the issue may lie in the fact that your http: configuration is listening on port 80.
With force_ssl: enabled, you're indicating that you want http connections to go to port 443, but when something arrives on 443 (via the load balancer), you send it to your (listening) port 80... which redirects it back to 443?
Fix
Let Phoenix listen on an arbitrary port (say... 4010) for http only connections. (Since the load balancer does your SSL termination, all your communication with the load balancer will be over http.) This involves changing your Docker container to forward connections to that port as well - you don't want to listen on 80 or 443 at all in your container.
Your url: configuration would then be looking only at headers, redirecting http requests to https as needed.
By the way, Amazon's ALB can also do 80 -> 443 redirection for you if you setup the rules; this saves Phoenix from even having to have a config url: setup for port 80 at all

HAProxy with SSL passthrough to multiple domains with multiple backends

I need to setup a load balancer for all our applications.
At the moment all our applications are clustered (2-node appservers, and 1 apache on each node as well) and we do not have a LB so we just point our DNS alias to the first webserver of each node, making the second node useless (have to manually do a DNS switch in case of a failure of node1, and we don't have load balanced https queries).
Each application uses SSL with a specific domain & SSL certificate. we cannot accept to decrypt SSL and send unencrypted traffic to the backends as the LB might be located in another country etc. so we need to use passthrough.
Before anything, i just wanted to know if this is actually possible in HAProxy or not ?
I am talking about ~50 different applications. Our LB configuration would have to be HA so i guess we'll use something like keepalived with a shared VIP for HAProxy itself.
The setup would look like this i suppose :
domain-a.com-' '-> backend_dom_a -> 1.1.1.1 (app node1 dom a)
| | 1.1.1.2 (app node2 dom a)
domain-b.com-' '-> backend_dom_b -> 2.1.1.1 (app node1 dom b)
| | 2.1.1.2 (app node2 dom b)
domain-c.com-' '-> backend_dom_c -> 3.1.1.1 (app node1 dom c)
| | 3.1.1.2 (app node2 dom c)
domain-N.com-' '-> backend_dom_N -> 4.1.1.1 (app node1 dom N)
| | 4.1.1.2 (app node2 dom N)
+-> haproxy -+
Thanks for your support, best regards
FYI i'm using this configuration that works like a charm.
i have replaced the values in the files to hide our domains & hostnames, and limited the numbers of urls/backends but we have about 50 running now with the load balancer forwarding requests to many apache servers (and each apache forwards requests to tomcat servers behind)
feel free if you have any question
we use balance source to ensure session stickyness
#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global
daemon
user haproxy
group haproxy
log /dev/log local6 notice
log /dev/log local5 info
maxconn 50000
#chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
mode tcp
option tcplog
log global
option dontlognull
timeout connect 5000
timeout client 50000
timeout server 50000
#---------------------------------------------------------------------
# dedicated stats page
#---------------------------------------------------------------------
listen stats
mode http
bind :22222
stats enable
stats uri /haproxy?stats
stats realm Haproxy\ Statistics
stats auth <mylogin>:<mypass>
stats refresh 30s
#---------------------------------------------------------------------
# main frontend which proxys to the backends
#---------------------------------------------------------------------
frontend main_https_listen
bind <ip address>:443
mode tcp
option tcplog
log global
tcp-request inspect-delay 5s
tcp-request content accept if { req.ssl_hello_type 1 }
#---------------------------------------------------------------------
# Common HAProxy nodes configuration
#---------------------------------------------------------------------
# -------------------------------
# ACLs
# -------------------------------
acl acl_SIT_AT35073 req.ssl_sni -i <app_url1>.my.domain.net # SIT_AT35073 is just an internal code we use, but you can use any alias
acl acl_SIT_AT34305 req.ssl_sni -i <app_url2>.my.domain.net
acl acl_SIT_AT28548 req.ssl_sni -i <app_urlN>.my.domain.net
# -------------------------------
# Conditions
# -------------------------------
use_backend backend_SIT_AT35073 if acl_SIT_AT35073 # same here
use_backend backend_SIT_AT34305 if acl_SIT_AT34305
use_backend backend_SIT_AT28548 if acl_SIT_AT28548
#---------------------------------------------------------------------
# Backends
#---------------------------------------------------------------------
# APP 1
backend backend_SIT_AT35073
description APPNAME1
mode tcp
balance source
option ssl-hello-chk
server server_SIT_AT35073_1 <apache_server1>.my.domain.net:443 check
server server_SIT_AT35073_2 <apache_server2>.my.domain.net:443 check
# APP 2
backend backend_SIT_AT34305
description APPNAME2
mode tcp
balance source
option ssl-hello-chk
server server_SIT_AT34305_1 <apache_server3>.my.domain.net:443 check
server server_SIT_AT34305_2 <apache_server4>.my.domain.net:443 check
# APP N
backend backend_SIT_AT28548
description APPNAMEN
mode tcp
balance source
option ssl-hello-chk
server server_SIT_AT28548_1 <apache_server5>.my.domain.net:443 check
server server_SIT_AT28548_2 <apache_server6>.my.domain.net:443 check
I think you have two options:
pass the traffic through to the backend by using the TCP mode in haproxy frontend and backend. This has the benefit that your backend SSL certificate is passed through. Though you lose the possibility to have one SSL termination in your site. So I present you
Have one (usual) SSL certificate, acting as termination for your site and enable SSL between your backend and haproxy instance. This gives you the advantage that you still have only one entry point but different backends with unique certificates.
The second option might look like this:
frontend f_foo
bind :443 ssl crt /path/to/bundle
mode http
log global
use_backend b2_foo
backend be_foo
mode http
timeout connect 5s
server FOO address:port ssl check crt /path/to/client/bundle force-tlsv10 verify none
The drawback is that you need a client certificate for each backend server but that should be easily automatable.
AS more of an update answer for multi domain configs I use the below for routing different domains.
in the frontend is where you bind the port and add the certs which multiple have to be on the same line afaik.
frontend https_in
bind *:443 ssl crt /link/to/cert+key-file.pem crt /link/to/cert+key-file.pem
The acl host is where you specify the domain name and which backend to use based on that domain name.
acl host_example.com hdr(host) -i example.com
use_backend BACKEND_NAME if host_example.com
The backend where you specify the server that domain is running on.
backend BACKEND_NAME
mode http
option httpclose
option forwardfor
cookie JSESSIONID prefix
server server-name server-ip:443 check ssl verify none

How to specify ssl connection with Nginx stream?

Im trying to define reverse proxy with nginx.
I have a server which listens on port 943 (TCP with SSL). I use tekn0ir/nginx-stream docker. I have the following definitions in myotherservice.conf file:
upstream backend {
hash $remote_addr consistent;
server myserverip:943;
}
server {
listen localhost:943;
proxy_connect_timeout 300s;
proxy_timeout 300s;
proxy_pass backend;
}
When Im trying to connect loslhost:943, it refused. Im suspect its related to my SSL definitions. How should I define it?
Working with docker you must to bind port to all container interfaces in order to be able to expose ports:
...
server {
listen *:943;
...
Doc

Realm Object Server - SSL configuration causes ROS to fail start

Ubuntu 16.04 / ROS v1.3.0
I am attempting to configure my ROS to use secure SSL connections.
If I do not make any changes to the configuration.yml - ROS is fine. I can sync and use the dashboard as I would expect.
I have obtained an SSL cert from Letsencrypt. I used the CertBot in standalone mode so that I did not have to install or configure Nginx. (My preference is to not install yet another tech/layer - keep it clean!)
I have the following certificates/key stored in this folder:
/etc/letsencrypt/live/data.mydomain.net/cert.pem
/etc/letsencrypt/live/data.mydomain.net/chain.pem
/etc/letsencrypt/live/data.mydomain.net/fullchain.pem
/etc/letsencrypt/live/data.mydomain.net/privkey.pem
As soon as I enable HTTPS in the configuration.yml I am unable to launch ROS.
There are no error messages written to:
/var/log/realm-object-server.log
Here is a copy of the proxy section of configuration.yml.
http:
## Whether or not to enable the HTTP proxy module. It enables multiplexing requests
## by forwarding incoming requests on a single port to all services.
# enable: true
## The address/interface on which the HTTP proxy module should listen. This defaults
## to 127.0.0.1. If you wish to listen on all available interfaces,
## uncomment the following line.
# listen_address: '::'
## The port that the HTTP proxy module should bind to.
# listen_port: 9080
https:
## Whether or not to enable the HTTPS proxy module. It enables multiplexing requests
## by forwarding incoming requests on a single port to all services.
## Note that even if it enabled, the HTTPS proxy will only start if supplied
## with a valid pair of certificates through certificate_path and private_key_path below.
enable: true
## The path to the certificate and private keys (in PEM format) that will be used
## to set up the HTTPS server accepting connections.
## These configuration options are MANDATORY to start the HTTPS proxy module.
certificate_path: '/etc/letsencrypt/live/data.mydomain.net/fullchain.pem'
private_key_path: '/etc/letsencrypt/live/data.mydomain.net/privkey.pem'
## The address/interface on which the HTTPS proxy module should listen. This defaults
## to 127.0.0.1. If you wish to listen on all available interfaces,
## uncomment the following line.
# listen_address: '::'
## The port that the HTTPS proxy module should bind to.
listen_port: 9443
As I mention. The issue appears to be that as soon as I configure HTTPS the ROS server fails to start. If I disable the HTTPS then the ROS server starts without issue.
The reason I believe ROS is failing to start is - if I attempt curl 127.0.0.1:9080 or curl 127.0.0.1:9443 from the terminal I get the message curl: (7) Failed to connect to 127.0.0.1 port 9443: Connection refused
I'd love to hear your ideas/thoughts/suggestions on how I can get this to work. Cheers. Ian
Thanks to user #Radu - the answer was Permissions.
The realm user did not have permission to read the .pem files.
I picked up the answer from this answer.
Https Proxy for Realm Object Server not working
#Radu - is the man!

Yaws with SSL gives the error "SSL accept failed: timeout"

I used certbot to generate a Let's encrypt certificate for my website, but Yaws gives me an SSL accept failed: timeout error when I try to connect to it (after it times out of course). Interestingly it works when I redirect example.com to the local ip address of the server in the hosts file on my machine and connect to example.com:8080, but not when I connect to example.com without editing the hosts file or when I connect from my phone over 4G. Here's my webserver's configuration file (it is the only configuration file in conf.d):
<server www.example.com>
port = 8080
listen = 0.0.0.0
docroot = /usr/share/yaws
<ssl>
keyfile = /etc/letsencrypt/live/example.com/privkey.pem
certfile = /etc/letsencrypt/live/example.com/fullchain.pem
</ssl>
</server>
I made sure that the keyfile and the certificate are both readable by the yaws user. Next to the keyfiles is a README that contains the following:
`privkey.pem` : the private key for your certificate.
`fullchain.pem`: the certificate file used in most server software.
`chain.pem` : used for OCSP stapling in Nginx >=1.3.7.
`cert.pem` : will break many server configurations, and should not be used
without reading further documentation (see link below).
We recommend not moving these files. For more information, see the Certbot
User Guide at https://certbot.eff.org/docs/using.html#where-are-my-certificates.
So I'm relatively sure I've used the right file (the other ones gave me errors like badmatch and {tls_alert,"decrypt error"}). I also tried trivial things like writing https:// before the URL, but it didn't fix the issue, also, everything works fine when the server is running without SSL. The version of Erlang running on my server is Erlang/OTP 19. Also, if it's unclear, the domain isn't actually example.com.
Also, example.com is redirected via cname to examplecom.duckdns.org, if that matters.
UPDATE:
My server was listening on port 8080, that was forwarded from the external port 80, for https connections, when the default https port is port 443. My other mistake was connecting to http://example.com instead of https://example.com. Forwarding the external port 443 to the internal port 8443 and configuring yaws to listen on port 8443 fixed everything.
Just to be sure to understand, when you do something like curl -v https://example.com:8080, you get a timeout, that's it ? (here https protocol and port 8080 are mandatory of course)
SSL timeout during accept can be triggered when an unencrypted request is received on a SSL vhost.
Could you also provide the output of the following command:
echo -e "HEAD / HTTP/1.0\r\n\r\n" | openssl s_client -connect mysite.com:8080 -ign_eof
And finally, which version of Yaws are you running ? on which OS ?