I'm not even sure I asked the question right...
I have three servers running minio in distributed mode. I need all three servers to run with TLS enabled. It's easy enough to run certbot, generate a cert for each node, drop said certs into /etc/minio/certs/ and go! but here's where I start running into issues.
The servers are thus:
node1.files.example.com
node2.files.example.com
node3.files.example.com
I'm launching minio using the following command:
MINIO_ACCESS_KEY=minio \
MINIO_SECRET_KEY=secret \
/usr/local/bin/minio server \
-C /etc/minio --address ":443" \
https://node{1...3}.files.example.com:443/volume/{1...4}/
This works and I am able to connect to all three servers from a webbrowser using https with good certs. however, users will connect to the server using the parent domain "files.example.com" (using distributed DNS)
I already ran certbot and generated the certs for the parent domain... and I copied the certs into /etc/minio/certs/ as well as /etc/minio/certs/CAs/ (calling both files "files.example.com-public.crt" and "files.example.com-public.key" respectively)... this did not work. when I try to open the parent domain "files.example.com" I get a cert error (chich I can bypass) indicating the certificate is for the node in which I have connected and not for the parent domain.
I'm pretty sure this is just a matter of putting the cert in the right place and naming it correctly... right? does anyone know how to do that? I also have an idea there might be a way to issue a cert that covers multiple domains... is that how I'm supposed to do this? how?
I already hit up minio's slack channel and posted on their github, but no ones replying to me. not even, "this won't work."
any ideas?
I gave up and ran certbot in manual mode. it had to install apache on one of the nodes, then certbot had me jump through a couple of minor hoops (namely it had me create a new txt record with my DNS provider, and then create a file with a text string on the server for verification). I then copied the created certs into my minio config directory (/etc/minio/certs/) on all three nodes. that's it.
to be honest, I'd rather use the plugin as it allows for an automated cert renewal, but I'll live with this for now.
You could also run all of them behind a reverse proxy to handle the TLS termination using a wildcard domain certificate (ie. *.files.example.com). The reverse proxy would centralize the certificates, DNS, and certbot script if you prefer, etc to a single node, essentially load balancing the TLS and DNS for the minio nodes. The performance hit of "load-balancing" TLS like this may be acceptable depending on your workload, considering the simplification to your current DNS and TLS cert setup.
[Digital Ocean example using nginx and certbot plugins] https://www.digitalocean.com/community/tutorials/how-to-create-let-s-encrypt-wildcard-certificates-with-certbot
Our developers have recently said they added code for load balancing to work in our webapp. So I've spent the last 2 days and I'm at a loss on why its not working. First I found that the Site ID's where different on both IIS Servers so that is fixed. Anyways when balancing is setup the ASP.NET_SessionId keeps resetting as if its not getting stored properly in the Database. I have looked at the table and there is an entry in there.
Using HAproxy and have 2 Windows Servers 2016 IIS10. Looking at both servers the code below is what the devs said they added.
<sessionState timeout="15" mode="SQLServer" sqlConnectionString="Integrated Security=SSPI;data source=DB1.resource.local;initial catalog=prodAPP;" allowCustomSqlDatabase="true" />
<machineKey decryptionKey="REMOVEDFORSECURITY" validationKey="REMOVEDFORSECURITY" />
The backend for HAProxy
backend prodAPP
balance roundrobin
mode http
option httpchk GET https://prodAPP.com
http-check disable-on-404
option http-server-close
option forwardfor
option redispatch
http-request set-header X-Forwarded-Port %[dst_port]
server webprod01 10.10.10.10:443 check ssl verify none fall 3 rise 5 inter 60000 weight 10
server webprod02 10.10.10.11:443 check ssl verify none fall 3 rise 5 inter 60000 weight 10
Any ideas on what I can try to get this thing working?
EDIT/UPDATE
So I thought about taking the proxy/load balancer out of the mix and tested by just pointing my host file at webserver1 logging into the webapp and moving around. Then changing the host file to point to webserver2 and saved the host file. Then clicked a linked and continued to use the webapp without the ASP.NET_SessionId resetting.
So, what is going on with the proxy that is making the ASP.NET_SessionId reset?
EDIT/UPDATE #2
After figuring out my host file wasnt actually changing with properly the chrome session socket seem to stick to that original host. Only way to properly get it to change servers was to disconnect the virtual nic on the server.
Anyways its an issue with the web servers that I can tell now, but I'm still confused on why they are each thinking the session is different and generating a new key.
When testing Apache 2.4.16 on windows 7,8,2012 there is a severe limitation when serving static content. Apache can't serve more than 700 concurrent requests for static content with Keep Alive OFF.
When you attempt to do that one of two things will happen:
You will be able to server few thousand requests at first and then the remaining requests will take up to 10 seconds to complete.
OR
You will receive a connection refused error
Test method:
siege -b -c700 -t10s -v http://10.0.0.31/10k.txt (10KB file)
OR
ab -c 700 -n 40000 http://10.0.0.31/10k.txt
However, when testing with Apache bench on the localhost (bypassing the network) Apache works fine and can serve 1000 concurrent requests for 10K static file.
Apache has ThreadsPerChild 7000 (increasing it to 14000 didn't make any difference)
MaxConnectionsPerChild 0
Stack parameters:
MaxUserPort = 65534
TcpTimedWaitDelay = 30
Server has over 60,000 ephemeral ports available starting with port 5,000 to port 65534
My load testing server is Linux on a separate server and sends requests over the network to Apache Windows server over 10Gb/s network.
There are no errors in the Apache log and nothing in the system logs. The tasklist doesn't show anything unusual.
netstat shows few thousand (5,000) of open TCP connections and then Apache stops responding. However when testing with lower concurrency of 300 then the OS can open 60,000 of TCP connections and Apache works fine.
Potential Conclusions:
At first I thought this is OS stack tuning problem but serving php file with the same concurrency works fine.
ab -c 700 -n 10000 http://10.0.0.31/phpinfo.php
Then I tried Nginx for windows on the same machine and Nginx served this without a problem.
ab -c 700 -n 10000 http://10.0.0.31/10k.txt
Nginx was able to serve much higher concurrency up to 2000 requests per second (static content) and the OS opened about 40,000 TCP connections.
So this looks to me like a bug or a limitation in the way Apache communicates with the TCP/IP stack on windows.
When trying to duplicate this problem make sure Keep Alive is OFF and test it over the network (not on localhost).
Any answers or comments on this subject will be greatly appreciated.
Thanks to covener's suggestion here is the answer.
Keep Alive was intentionally disabled to simulate a large number of users connecting from different IP addresses and spawning new TCP connections.
Setting AcceptFilter http to "none" together with turning off MultiViews improved the performance on static content and allowed Apache on windows to serve with concurrency of 2000 and beyond untill all ephemeral ports get exhausted.
I have recently switched over to HAProxy from AWS ELB. I am terminating SSL at the load balancer (HAProxy 1.5dev19).
Since switching, I keep getting some SSL connection errors in the HAProxy log (5-10% of the total number of requests). There's three types of errors repeating:
Connection closed during SSL handshake
Timeout during SSL handshake
SSL handshake failure (this one happens rarely)
I'm using a free StartSSL certificate, so my first thought was that some hosts are having trouble accepting this certificate, and I didn't see these errors in the past because ELB offers no logging. The only issue is that some hosts have do have successful connections eventually.
I can connect to the servers without any errors, so I'm not sure how to replicate these errors on my end.
This sounds like clients who are going away mid-handshake (TCP RST or timeout). This would be normal at some rate, but 5-10% sounds too high. It's possible it's a certificate issue; I'm not certain exactly how that presents to
Things that occur to me:
If negotiation is very slow, you'll have more clients drop off.
You may have underlying TCP problems which you weren't aware of until your new SSL endpoint proxy started reporting them.
Do you see individual hosts that sometimes succeed and sometimes fail? If so, this is unlikely to be a certificate issue. I'm not sure how connections get torn down when a user rejects an untrusted certificate.
You can use Wireshark on the HAProxy machine to capture SSL handshakes and parse them (you won't need to decrypt the sessions for handshake analysis, although you could since you have the server private key).
I had this happen as well. The following appeared first SSL handshake failure then after switching off option dontlognull we also got Timeout during SSL handshake in the haproxy logs.
At first, I made sure all the defaults timeouts were correct.
timeout connect 30s
timeout client 30s
timeout server 60s
Unfortunately, the issue was in the frontend section
There was a line with timeout client 60 which I only assume means 60ms instead of 60s.
It seems certain clients were slow to connect and were getting kicked out during the SSL handshake. Check your frontend for client timeouts.
How is your haproxy ssl frontend configured ?
For example I use the following to mitigate BEAST attacks :
bind X.X.X.X:443 ssl crt /etc/haproxy/ssl/XXXX.pem no-sslv3 ciphers RC4-SHA:AES128-SHA:AES256-SHA
But some clients seem to generate the same "SSL handshake failure" errors. I think it's because the configuration is too restrictive.
At my workplace, the traffic blocker/firewall has been getting progressively worse. I can't connect to my home machine on port 22, and lack of ssh access makes me sad. I was previously able to use SSH by moving it to port 5050, but I think some recent filters now treat this traffic as IM and redirect it through another proxy, maybe. That's my best guess; in any case, my ssh connections now terminate before I get to log in.
These days I've been using Ajaxterm over HTTPS, as port 443 is still unmolested, but this is far from ideal. (Sucky terminal emulation, lack of port forwarding, my browser leaks memory at an amazing rate...) I tried setting up mod_proxy_connect on top of mod_ssl, with the idea that I could send a CONNECT localhost:22 HTTP/1.1 request through HTTPS, and then I'd be all set. Sadly, this seems to not work; the HTTPS connection works, up until I finish sending my request; then SSL craps out. It appears as though mod_proxy_connect takes over the whole connection instead of continuing to pipe through mod_ssl, confusing the heck out of the HTTPS client.
Is there a way to get this to work? I don't want to do this over plain HTTP, for several reasons:
Leaving a big fat open proxy like that just stinks
A big fat open proxy is not good over HTTPS either, but with authentication required it feels fine to me
HTTP goes through a proxy -- I'm not too concerned about my traffic being sniffed, as it's ssh that'll be going "plaintext" through the tunnel -- but it's a lot more likely to be mangled than HTTPS, which fundamentally cannot be proxied
Requirements:
Must work over port 443, without disturbing other HTTPS traffic (i.e. I can't just put the ssh server on port 443, because I would no longer be able to serve pages over HTTPS)
I have or can write a simple port forwarder client that runs under Windows (or Cygwin)
Edit
DAG: Tunnelling SSH over HTTP(S) has been pointed out to me, but it doesn't help: at the end of the article, they mention Bug 29744 - CONNECT does not work over existing SSL connection preventing tunnelling over HTTPS, exactly the problem I was running into. At this point, I am probably looking at some CGI script, but I don't want to list that as a requirement if there's better solutions available.
Find out why the company has such a restrictive policy. It might be for a good reason.
If you still find that you want to bypass the policy, you could write a small proxy that will listen on your server on port 443 and then, depending on the request, will forward the traffic either to your web server or to the SSH daemon. There are two catches though.
To determine whether it's an HTTPS request or an SSH request, you need to try to read some data with a (small) timeout, this is because TLS/SSL handshakes start with the client sending some data, whereas the SSH handshake starts with the server sending some data. The timeout has to be big enough to delays in delivering the initial data from the client in the TLS/SSL handshake, so it'll make establishing SSH connections slower.
If the HTTP proxy in your company is smart, it'll actually eavesdrop on the expected TLS/SSL "handshake" when you CONNECT to port 443, and, when it detects that it's not an TLS/SSL handshake, it might terminate the SSH connection attempt. To address that, you could wrap the SSH daemon into an TLS/SSL tunnel (e.g., stunnel), but then you'll need to differentiate requests based on the TLS/SSL version in your client request to determine whether to route the TLS/SSL connection to the web server or to the TLS/SSL-tunneled SSH daemon.
You should be able to use iptables to forward ssh traffic from your work machines to ssh while all other machines attaching to your home server on port 443 get the Apache server.
Try a rule like this:
iptables -t nat -A PREROUTING -p tcp -s 111.111.111.111 --dport 443 -j REDIRECT --to-port 22
Where 111.111.111.111 is your office computer's ip address.
That all assumes you're running Linux >= 2.4, which you should be by now. It's been out for almost a decade.
Documentation for iptables is at http://www.netfilter.org.
Set up OpenVPN 2.1 server at home, use port 443 (if you set up your home any HTTPS service at port 443, trigger OpenVPN's port-share option to handle both OpenVPN and HTTPS transactions at port 443; this feature is only available to non-Windows OS)
Then, set up your OpenVPN client on your laptop in road-warrior mode to access the OpenVPN server at home. You will be able to call home or anywhere you like within a secure VPN network you've created with OpenVPN. It is no longer required to use SSH for this purpose.
I'm really sorry for being the Devil's advocate here, but if they are blocking ports at your work, its likely because they don't want people breaching security.
Now if you get permission to open a tunnel from your boss, that's fine, but IF something happens, ANYTHING, and they figure out you have a tunnel, I can almost assure you, you'll become the scapegoat. So if I were you I'd not be opening tunnels at work if they are setting up firewalls against it.
How about using 2 IP adresses on your machine?
Bind apache/https on one IP_1:443 and your sshd on the other IP_2:443?
Could you set up a middle man?
Run a small/free/cheap instance in the cloud listening on 443 for SSH, then though that cloud instance tunnel to your home box on your favorite port - 22 or whatever.
It'll add some latency I'm sure, but it solves the problem of leaving the original home setup intact.
I think you'll have to find a port that you're not using currently that you can get out on, and listen on that. 443 is the obvious candidate, but you say that's not possible. What about mail (25, 110, 143), telnet (23), ftp (21), DNS (53), or even whois (43)?
Proxy tunnel may be your answer
http://proxytunnel.sourceforge.net/
lets say my ssh server is host.domain.tld and my works proxy server is 10.2.4.37
I would add this to my local ssh config
Host host.domain.tld
ProxyCommand /usr/local/bin/proxytunnel -q -p 10.2.4.37:3128 -d %h:%p
ProtocolKeepAlives 30
See:
SSH Through or Over Proxy
http://daniel.haxx.se/docs/sshproxy.html
http://www.agroman.net/corkscrew/
Since apache has no problem whatsoever with CONNECT when no SSL is involved, I turn off SSL features and I use stunnel to serve an https version of my site. This does not require any recompilation, and allows your site to serve https normally. So far, the cleanest workaround I know.
See http://chm.duquesne.free.fr/blog/?p=281 for details.
Must work over port 443, without disturbing other HTTPS traffic (i.e. I can't just put the ssh server on port 443, because I would no longer be able to serve pages over HTTPS)
Is it possible to bind your HTTPS server to a different port? Depending on what it's used for, you may even be able to get around the problem of not being able to directly access it from work by just SSHing home and then using lynx from there.
So, then, give proxifier a try (- it supports HTTP Proxy Server)!
http://www.proxifier.com/documentation/intro.htm
I managed to bypass my company's firewall using the following design via AjaxTerm, it works for me.
PC on company network --> company's proxy via https --> INTERNET --> My home Apache reverse proxy server on SSL + .htpasswd protection --> AjaxTerm Server(From here on ward, I can SSH to any other servers ).
Still not the perfect world... would be good if I can can tunneling to my home network via HTTPS.