HAproxy SSL handshake failure - ssl

when i use HAproxy as load balancer, at HTTP termination mode and i tail log of it
(tail -f /var/log/haproxy.log). There are 2 types of log appearing
[time] frontend_name/1: SSL handshake failure
and
[time] frontend_name~ message
frontend_name is name follow frontend keyword config in /etc/haproxy/haproxy.cfg
I don't know what /1 and ~ in log message is, and why SSL handshake failure appearing at log has ~
Can someone help me explain and fix this error?
Thanks!

~ after frontend name means connection has been established using SSL/TLS
You can find reference to it in %ft entry in the table at: https://cbonte.github.io/haproxy-dconv/2.4/configuration.html#8.2.4
About /1 in frontend_name/1: SSL handshake failure:
I can't find it in the docs, but by experimenting i found it's the number of port in frontend, to which connection was attempted and SSL handshake failed.
For config:
frontend frontend_name
bind *:443,*:444 ssl crt <path_to_cert>
bind *:445 ssl crt <path_to_cert> no-tlsv13
If i make TLS1.3 connection to port 445 (e.g. openssl s_client -connect 127.0.0.1:445 -tls1_3), i will get:
frontend_name/3: SSL handshake failure
because 445 is 3. port listed in this frontend.
[UPDATE]
I found a bit more. Error log format explains that /1 in frontend_name/1 is bind_name and can be declared:
bind *:443,*:443 ssl crt <path_to_cert> no-tlsv13 name bind_ssl_foo
will result in frontend-name/bind_ssl_foo: SSL handshake failure.
Unfortunately we can't change error log format.
To learn more we have to make that connection successful and that most likely requires us to lower security (FOR DEBUGGING ONLY!). Normal clients will still negotiate highest security they can, TLS 1.2 or 1.3.
bind *:443 ssl crt <path_to_cert> ssl-min-ver TLSv1.0
Since haproxy 2.2 default for ssl-min-ver is TLSv1.2.
Second step is to log SSL version, negotiated cipher and maybe whole cipherlist send by client by appending %sslv %sslc and maybe %[ssl_fc_cipherlist_str] to your log-format:
log-format "your_log_format_here %sslv %sslc %[ssl_fc_cipherlist_str]"
If you don't have your own log format you can extend HTTP format:
log-format "%ci:%cp [%tr] %ft %b/%s %TR/%Tw/%Tc/%Tr/%Ta %ST %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %{+Q}r %sslv %sslc %[ssl_fc_cipherlist_str]"
To use ssl_fc_cipherlist_str we need to set tune.ssl.capture-cipherlist-size 800 in global section, because default is 0.
sslv is SSL/TLS version client connected with.
sslc is SSL/TLS cipher client connected with.
ssl_fc_cipherlist_str is cipher list client offered when negotiating SSL/TLS connection. It can be long. Use if you are extra curious.
That will append to your logs info like this:
TLSv1 ECDHE-RSA-AES256-SHA ECDHE-ECDSA-AES256-SHA,ECDHE-RSA-AES256-SHA,DHE-RSA-AES256-SHA,ECDHE-ECDSA-AES128-SHA,ECDHE-RSA-AES128-SHA,DHE-RSA-AES128-SHA,AES256-SHA,AES128-SHA,TLS_EMPTY_RENEGOTIATION_INFO_SCSV
Match by IP previous errors with current entries and you will know what TLS version and ciphers they were using. Then decide whether to adjust your ciphers or force this client to upgrade their SSL software.
So all required changes below:
global
log /dev/log daemon
tune.ssl.capture-cipherlist-size 800
frontend frontend_name
bind *:443 ssl crt <path_to_cert> ssl-min-ver TLSv1.0
log-format "%ci:%cp [%tr] %ft %b/%s %TR/%Tw/%Tc/%Tr/%Ta %ST %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %{+Q}r %sslv %sslc %[ssl_fc_cipherlist_str]"
mode http
(...)
Again, lower security only for debugging if this connection error really is a problem for you.

Related

Timeout execution at config file for disallowed protocols

I am working on a legacy code of WebRTC Gateway. I have updated a config file for disabling unsecure protocols like SSLv3, TLS1.1 and TLS1.0 which you can see from the code snippet below;
# Allowed cipher list.
cipher-list="TLSv1.2+HIGH:!TLSv1.1:!TLSv1:!SSLv3:!aNULL:!eNULL:!MD5:!DSS:!RC4:!PSK:!3DES:!DH:!EXPORT:!LOW:!SSLv2:#STRENGTH"
# Do not allow an SSL/TLS version of protocol
#
no-sslv2
no-sslv3
no-tlsv1
no-tlsv1_1
#no-tlsv1_2
Then I try to test protocols with this command;
openssl s_client -tls1_2 -connect www.example.com:443
When I execute the command with allowed protocols, there is no failure and handshake happens. But when I change the protocol with disallowed protocols like as TLS1.1 or TLS1.0, the system does not allow the handshake but waits until timeout and then throws an exception.
If I modify the config file's protocol list as below (comment out all the protocols), the handshake failure is thrown immediately;
# Allowed cipher list.
cipher-list="TLSv1.2+HIGH:!TLSv1.1:!TLSv1:!SSLv3:!aNULL:!eNULL:!MD5:!DSS:!RC4:!PSK:!3DES:!DH:!EXPORT:!LOW:!SSLv2:#STRENGTH"
# Do not allow an SSL/TLS version of protocol
#
#no-sslv2
#no-sslv3
#no-tlsv1
#no-tlsv1_1
#no-tlsv1_2
I will update the product description, so I am trying to figure out this execution in deep. Does anyone have any idea why this system behavior is so?
Thanks.

SSL in Bluemix and nginx configuration

I have added a certificate in Bluemix, following this post : https://www.ibm.com/blogs/bluemix/2014/09/ssl-certificates-bluemix-custom-domains/
I can see the certificate in the domain tab, and it's the one I have uploaded.
Now I have a container running nginx, because we use it as a reverse proxy. Previously it was handling the SSL configuration, but now that it's done in Bluemix directly, we just want to accept https request, without configuring certificate.
What we did was forwarding the http requests to https, like advised in the post (explaining how to do it for node.js though). We get something like this:
server {
listen 80;
server_name *hostname.domain*;
return 301 https://$http_host$request_uri;
}
And in the 443 part, we only listen, without the ssl part:
server {
listen 443;
server_name *host.domain*;
*other stuff for reverse proxy*
}
However, when trying to access it, I get a generic error in chrome: ERR_SSL_PROTOCOL_ERROR
Firefox gives a bit more information:
An error occurred during a connection to *host.domain*. SSL received a record that exceeded the maximum permissible length. Error code: SSL_ERROR_RX_RECORD_TOO_LONG
And when I try to check the certificate from command line, I don't get any.
openssl s_client -connect *host.domain*:443
CONNECTED(00000003)
140250419918480:error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol:s23_clnt.c:782:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 7 bytes and written 289 bytes
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
SSL-Session:
Protocol : TLSv1.2
Cipher : 0000
Session-ID:
Session-ID-ctx:
Master-Key:
Key-Arg : None
PSK identity: None
PSK identity hint: None
SRP username: None
Start Time: 1484673167
Timeout : 300 (sec)
Verify return code: 0 (ok)
---
There's no error in nginx logs, and I can't manage to tell if the issue in on Bluemix side, or in the configuration of nginx, or if nginx allows this kind of configuration where it has to handle https requests, without the certificate configuration...
Does someone have any idea?
Many thanks.
Regards.
If you want NGINX to pass-thru SSL, you have to use the stream module.
Thanks for your answers. I was not able to check your solutions, but I talked with a technical expert from IBM meanwhile, and here is what I learned.
About the SSL pass-thru, we would need to configure each component (behind the nginx) to handle the SSL, so it seems to be harder to manage. I'm not an expert though, so I'm just reporting what I had as an answer on that point.
First, what we want should be doable by removing the public IP adress of our nginx container.
Then, by creating a route from BM load balancer to our nginx container, we should solve the issue. Then the route would be configured to forward the port 443 to the nginx on port 80 (since the container is not publicly available, there no need to handle 80 AND 443).
However, Bluemix allows route only for container groups (for now?). Unfortunately, we use docker-compose that does not allow (for now?) to create container groups on BlueMix.
So the best solution was to put back the ssl configuration in nginx. The certificate being both on BlueMix domain and nginx container. And it's working fine, so we'll just improve the procedure to update the certificate, and wait until there's a need, or a new way to do it...
K.

nginx - log SSL handshake failures

I'm running an nginx server with SSL enabled.
My protocol / cipher settings are fairly secure, and I've checked them at ssllabs.com, but --
-- since this is a web service which is called by http clients that I have no control over, I have concerns about compatibility.
To the point:
Is there a way to log SSL handshake failures as they happen (if they happen) in my nginx logs?
For example, I've got SSLv3 disabled, and if I try to "curl -3" (forcing SSlv3) to my server, then I get this:
NSS error -12286 (SSL_ERROR_NO_CYPHER_OVERLAP)
Cannot communicate securely with peer: no common encryption algorithm(s).
Closing connection 0 curl: (35) Cannot communicate securely with peer: no common encryption algorithm(s).
I would like to log this type of error in server logs too, with the default nginx settings, there is nothing.
Enabling "debug" log level for the error log does what I want, will log SSL handshake errors -- but unfortunately it also logs too much other stuff, making the log too bloated, drowning out other potentially useful info.
You can use the info log level.

Debugging process for Kafka SSL security

I was able to set up the SSL in my Kafka brokers and client and I am also able to see that when we produce messages using the 9093 port that is the SSL port the messages are consumed by the consumer.
I tried to send a message via the 9093 port and the message is sent to the consumer from the producer.
Is there any way to verify if this is actually working, I mean how can I demonstrate that 9092 is not SSL and 9093 is SSL and secured?
Below two ways can verify the setup of SSL.
-Djavax.net.debug=all Add this property in
bin/kafka-run-class.sh at the same place as:
if [ -z "$KAFKA_JMX_OPTS" ]; then
KAFKA_JMX_OPTS=" <**add here**> -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false "
fi
then tail the kafka broker log file, you should see some encoded messages.
To verify if the server's keystore and truststore are setup correctly you can run the following command:
openssl s_client -debug -connect localhost:9093 -tls1
Note: TLSv1 should be listed under ssl.enabled.protocols.
In the output of this command you should see the server’s certificate:
-----BEGIN CERTIFICATE-----
{variable sized random bytes}
-----END CERTIFICATE-----
subject=/C=US/ST=CA/L=Santa Clara/O=org/OU=org/CN=Joe Smith
issuer=/C=US/ST=CA/L=Santa Clara/O=org/OU=org/CN=kafka/emailAddress=test#test.com
If the certificate does not show up or if there are any other error messages then your keystore is not setup correctly.
Reference :
http://docs.confluent.io/2.0.0/kafka/ssl.html
https://github.com/Symantec/kafka-security-0.9
We have tried setting up kafka with SSL and faced the same issue to crosscheck if its working on SSL I removed the listeners entry PLAINTEXT and kept the configuration which is required for SSL only and tested. We were able to send and receive message wirh SSL at secure port 9093 .
In logs it does gives a WARNING but it's for the client authentication so that's not an issue , once tested I added the PLAINTEXT entry again .
I know it's not the best way to test but it worked for me
When you mention security.protocol=SSL, there is no way it can use the other protocol. For more proofs, as mentioned above you can edit the kafka-run-class.sh to turn on debug all and verify the ssl handshakes happening and metadata being sent over ssl channel.
Thanks
Akash

handshake failure(40) and TLS_EMPTY_RENEGOTIATION_INFO_SCSV

A client installed on jBOSS is trying to access a secured website configured on DataPower xi50v6.0.0.2 appliance. The connection is getting failed at SSL handshake.
I have taken a packet capture at DataPower and observed that SSL Handshake is failing with the Description:Handshake failure(40).
However, at the Client Hello step, I have observed that, only one Cipher Suite is specified which is : TLS_EMPTY_RENEGOTIATION_INFO_SCSV.
The TLS protocol used ( as per packet capture) is TLS1.1. Can this Cipher Suite be a problem?
In the DataPower system logs I can see below error:
Request processing failed: Connection terminated before request headers read because of the connection error occurs
Update:
The client application is running on jBOSS7.I have asked our jBOSS administrator to check the configuration at jBOSS end. I somehow got the access to server where jBOSS instance is installed and checked domain.xml where the ssl is configured. Where exactly in domain.xml, ths configuration related to cipher suites can be found?
I have observed that, only one Cipher Suite is specified which is : TLS_EMPTY_RENEGOTIATION_INFO_SCSV
This is no real cipher. If no other ciphers are specified then the client does not offer any ciphers at all which means that no shared ciphers can be found and thus the handshake will fail. It looks like the client is buggy. Reason might be a failed attempt to fight POODLE attack by disabling all SSL3.0 ciphers, which in effect disables all ciphers for TLS1 1.0 and TLS 1.1.