I'm finding it pretty difficult to get reliable information on Google about how exactly to do load balancing for anything other than a web server. Here is my situation: I currently have a python/twisted SSL server running on one machine. This is not fast enough so I want to change this so that multiple instances of this server will run on multiple machines behind a load balancer. So suppose I have two copies of this server process: TWISTED1 and TWISTED2. TWISTED1 will run on MACHINE1 and TWISTED2 will run on MACHINE2. Both TWISTED1 and TWISTED2 are SSL server processes. A separate machine LOAD_BALANCER is used to load balance between the two machines.
Where do I put my existing SSL certificate? Do I put an identical copy on both MACHINE1 and MACHINE2? Do I also have an identical copy on LOAD_BALANCER? I do NOT want unencrypted traffic between LOAD_BALANCER and MACHINE1 or MACHINE2, and also the twisted processes are already set up as SSL servers, so it would be unnecessary work to remove SSL from the twisted process. Basically I want to set up load balancing for SSL traffic with minimal changes to the existing twisted scripts. What is the solution?
Regarding the load balancer, is it sufficient to use another machine MACHINE3 and put HAPROXY onto this machine as the load balancer or is it better to use a hardware load balancer like Baracuda?
Note also that most of the connections to the twisted process are persistent connections.
Could you have the certs on one machine and a mount from the other machine to the machine w/ the certs? Allowing you to only have one set of ssl certs.
The problem with load balancing a TLS server is that without the HTTP "forwarded for" header, there's no way to tell where the original connection came from. That's why so much documentation focuses on load-balancing HTTP(S).
You can configure TLS termination in more or less all of the ways that you've described; it sounds like you can simply have your load balancer act as a TCP load balancer. The only reason to have your load balancer have your certificate (with the implication being that it also would have your private key) would be for it to decrypt the traffic to figure out what to do with it, then re-encrypt it to the machines. If you don't need to do that, put the certificate only on the target machines and not on the LB.
Related
We have an Oracle application (Agile PLM) which is deployed in a clustered environment. We have one admin node and two managed nodes supporting our application, where admin and 1 managed nodes are on the same server. We also have Load balancer which manages the traffic between the cluster.
We want to configure SSL in our application so that the application URL will be accessible over https only. We have already configured SSL at Load Balancer level(by installing security certificates in weblogic server which is the admin server) but want to know if we have to configure SSL on the managed server as well or bringing Load Balancer on https is sufficient?
All the users access the application using the Load Balancer URL only but since I am from the development team, so is only aware of the fact that we can also connect to the application with Managed server URLs, which are still running on http. Is it must to bring Managed servers also on https or it is just a good practice but not necessary?
It's not necessary, though probably a good practice.
I believe I have read in Oracle's installation guide that the recommended way is HTTP on the managed servers and terminating SSL on the load balancer. This may have changed.
For what it's worth, I leave HTTP on the managed servers.
I have drawed the following chart to help explain our situation:
We have an API running on the KESTREL WebServer in the back end. This is running as several instances in docker containers. In front of this, we have a HAProxy Load Balancer which directs the traffic to the Kestrel server with the least connections.
All of this so far is only available internally on our network.
We have one server accessible from the internet, running IIS on port 80 and 443.
In IIS we have added a SSL Certificate which encrypts the communication between the end user and our first entry point.
Is there any further need to encrypt the communication between IIS which proxies the request to our HAProxy on the internal network, and from there on, necessary to encrypt communication from HAProxy to the backends?
As far as my knowledge goes, the only benefit of this, would be that nobody on the internal network at our office can listen to the connections going between the servers, but please correct me if I am wrong here.
I'm trying to configure load balancer to serve in HTTPS with certificates provide by Lets Encrypt, even though I couldn't do it yet, reading this article gives steps how to configure
Self-signed certs
Network Load-Balancer w/ TLS backend
HTTPS Load-Balancer w/ non-TLS backend
HTTPS Load-Balancer w/ TLS backend
As I'm intersting only in HTTPS, I wonder what is the difference between this two:
HTTPS Load-Balancer w/ non-TLS backend
HTTPS Load-Balancer w/ TLS backend
But I meant not the obvious reason that is the first one is not encrypted from load balancer to backend, I mean in performance and HTTP2 conection, for example will I continue to get all the benefits from http2 like multiplexing and streaming? or is the first option
HTTPS Load-Balancer w/ non-TLS backend
only an illusion but I won't get http2?
To talk HTTP/2 all web browsers require the use of HTTPS. And even without HTTP/2 it's still good to have HTTPS for various reasons.
So the point your web browser needs to talk to (often called the edge server), needs to be HTTPS enabled. This is often a load balancer, but could also be a CDN, or just a single web server (e.g. Apache) in front of an application server (e.g. Tomcat).
So then the question is does the connection from that edge server to any downstream servers need to be HTTPS enabled? Well, ultimately the browser will not know, so not for it. Then you're down to two reasons to encrypt this connection:
Because the traffic is still travelling across an insecure channel (e.g. CDN to origin server, across the Internet).
Many feel it's disingenuous to make the user think they are on a secure (with a green padlock) then in fact they are not for the full end to end connection.
This to me is less of an issue if your load balancer is in a segregated network area (or perhaps even on the same machine!) as the server it is connecting to. For example if the load balancer and the 2 (or more) web servers is is connecting to are both in a separate area in a DMZ segregated network or their own VPC.
Ultimately the traffic will be decrypted at some point and the question for server owners is where/when in your networking stack that happens and how comfortable you are with it.
Because you want HTTPS for some other reason (e.g. HTTP/2 all the way through).
On this one I think there is less of a good case to be made. HTTP/2 primarily helps high latency, low bandwidth connections (i.e. browser to edge node) and is less important for low latency, high bandwidth connections (as load balancer to web servers often are). My answer to this question discusses this more.
In both the above scenarios, if you are using HTTPS on your downstream servers, you can use self-signed certificates, or long lived self-signed certificates. This means you are not bound by the 30 days LetsEncrypt limitations, nor does it require you to purchase longer certificates from another CA. As the browser never sees these certificates you only need your load balancer to trust them, which is in your control to do for self-signed certificates. This is also useful if the downstream web servers cannot talk to LetsEncrypt to be able to get certificates from there.
The third option, if it really is important to have HTTPS and/or HTTP/2 all the way through, is to use a TCP load balancer (which is option 2 in your question so apologies for confusing the numbering here!). This just forwards TCP packets to the downstream servers. The packets may still be HTTPS encrypted but the load balancer does not care - it's just forwarding them on and if they are HTTPS encrypted then the downstream server is tasked with decrypting them. So you still can have HTTPS and HTTP/2 in this scenario, you just have the end user certificates (i.e. the LetsEncrypt ones) on the downstream web servers. This can be difficult to manage (should the same certificates be used on both? Or should they have different ones? Do we need to have sticky sessions so HTTPS traffic always hits the sae downstream server). It also means the load balancer cannot see or understand any HTTP traffic - they are all just TCP packets as far as it is concerned. So no filtering on HTTP headers, or adding new HTTP headers (e.g. X-FORWARDED_FOR with the orignal IP address.)
To be honest it is absolutely fine, and even quite common, to have HTTPS on the load balancer and HTTP traffic on downstream servers - if in a secure network between the two. It is usually the easiest to set up (one place to manage HTTPS certificates and renewals) and the easiest supported (e.g. some downstream servers may not easily support HTTPS or HTTP/2). Using HTTPS on this connection either by use of self-signed certificates or CA issued certificates is equally fine, though requires a bit more effort, and the TCP load balancer option is probably the most effort.
I'm trying to get RabbitMQ configured behind an F5 load balancer. I have a working RabbitMQ node with the default node name of rabbit#%computername%. It's set to listen on all network interfaces (all IP addresses 0.0.0.0:5671 which is the AMQP SSL port), and it's working fine. However, all client applications that connect to it are currently using the specific node name e.g. "%computername%". In order to take advantage of the fault tolerance of the load balancer, I want to update all my client applications to use the load-balanced name instead of the specific node name e.g. connect using HostName = "balancedname.mycompany.com" instead of "%computername%". However, when I update my client applications to connect to the load-balanced name, the connection fails. How can I get this to work?
I'm a novice at F5, and I did notice that the pool's members' addresses are IP addresses...should these be the node names instead of the IPs? Is that even possible seeing as the node name can be completely arbitrary and doesn't necessarily map to anything that's network-resolveable? I'm in a hosting situation where I don't have write access to the F5, so trying these things out is a bit tricky.
I haven't found very much information at all on load balancing a RabbitMQ setup. I do understand that all RabbitMQ queues only really exist on one node, and I've set up the F5 in an active-passive mode so that traffic will always route to the primary node unless it goes down.
Update 1: It seems that this issue came back to bite me here. I'm using EXTERNAL authentication using an SSL certificate, and since clients were connecting using the load balance name instead of the node name, and the load balance name was NOT used to create the certificate, it was rejecting the connection. I ended up re-generating the certificate and using the load balance name, but that wasn't enough - I also had to add an entry in the Windows hosts file to map 127.0.0.1 and ::1 to the load balance DNS address.
Update 2: Update 1 solves connection problems only for running client applications on the app server that is part of the load balancer, but remote clients don't work. Inner exception says "The certificate chain was issued by an authority that is not trusted". RabbitMQ + SSL is hard. And adding load balancing makes it even harder.
I'm answering my own question in the hopes that it will save folks some time. In my scenario, I needed for clients to connect to a load balanced address like myrabbithost.mycompany.com, and for the F5 to direct traffic to one node as long as it's up and failover to the secondary node if it's down. I had already configured security and was authenticating to RabbitMQ using self-signed certificates. Those certificates had common names specific to each host which was the problem. In order to work with .NET, the common name on the certificate must match the server name being connected to (myrabbithost.mycompany.com in my case). I had to do the following:
Generate new server and client certificates on the RabbitMQ servers with common names of myrabbithost.mycompany.com
Generate a new certificates for the clients to use while connecting in order to use SSL authentication
Still on the RabbitMQ servers, I had to concatenate the multiple cacert.pem files used for the certificate authority so that clients can authenticate to any node using a client certificate generated by any node. When I modified rabbit.config to use the "all.pem" instead of "cacert.pem", clients were able to connect, but it broke the management UI, so I modified the rabbitmq_management settings in rabbit.config to specific the host-specific cacert.pem file and it started working again.
In order to set up high availability, I set up a RabbitMQ cluster, but ran into some problems there as well. In addition to copying the Erlang cookie from the primary node to the secondary node at C:\Windows and C:\users\myusername, I had to kill the epmd.exe process via task manager as the rabbitmqctl join_cluster command was failing with a "node down" error. The epmd.exe process survives RabbitMQ stoppages and it can cause rabbitmqctl.bat to report erroneous errors like "node down" even when it's not down.
The past week or so I have been getting to grips with the world of AWS and more specifically Elastic Beanstalk and Load Balancing...
The application I'm developing enforces a SSL/HTTPS connection using a custom RequireHttps attribute I'm applying globally. I initially had problems configuring the load balancer with this set up, but it does appear to be working as expected.
My problem originates from a blog post I was glancing over around the time I was setting up the Load Balancer/RequireHttps attribute. Quoting this blog post:
When using Elastic Beanstalk ... the connection between the load balancer and application server is not secure. However, you don't need to be concerned with the security of the connection between the load balancer and the server but you do need to be concerned about the connection between the client and the load balancer.
As configuring load balancers is an entirely new area for me, I'm a little sceptical that the above is entirely true.
Is the connection between a load balancer and server truly none of my concern? Would it be better to not terminate SSL at the Load Balancer and pass a sercure connection straight through to the server?
After a little further research, I stumbled across the following post/discussion on security.stackexchange: Should SSL be terminated at a load balancer?
makerofthings7:
It seems to me the question is "do you trust your own datacenter". In other words, it seems like you're trying to finely draw the line where the untrusted networks lie, and the trust begins.
In my opinion, SSL/TLS trust should terminate at the SSL offloading device since the department that manages that device often also manages the networking and infrastructure. There is a certain amount of contractual trust there. There is no point of encrypting data at a downstream server since the same people who are supporting the network usually have access to this as well. (with the possible exception in multi-tenant environments, or unique business requirements that require deeper segmentation).
A second reason SSL should terminate at the load balancer is because it offers a centralized place to correct SSL attacks such as CRIME or BEAST. If SSL is terminated at a variety of web servers, running on different OS's you're more likely to run into problems due to the additional complexity . Keep it simple, and you'll have fewer problems in the long run.
I can see that what #makerofthings7 is saying makes sense. Whether SSL is terminated at the Load Balancer or Server should make little difference.