Continuous TLS handshake error logs in vault nodes due to LB health check - ssl

Question

I am getting continuous TLS handshake errors every 5 sec due to my load balancer pinging vault nodes in every 5 seconds. Kube load balancer is pinging my vault nodes using
nc -vz podip podPort every 5 sec
I have already disabled client cert verification in my config.hcl but still see below logs in my kubectl logs for vault
kubectl logs pod-0 -n mynamespace
[INFO] http: TLS handshake error from 10.x.x.x:60056: EOF 2020-09-02T01:13:32.957Z
[INFO] http: TLS handshake error from 10.x.x.x:23995: EOF 2020-09-02T01:13:37.957Z
[INFO] http: TLS handshake error from 10.x.x.x:54165: EOF 2020-09-02T01:13:42.957Z
Below is my config.hcl which I am loading via kube config map
apiVersion: v1
kind: ConfigMap
metadata:
name: raft-config
labels:
name: raft-config
data:
config.hcl: |
storage "raft" {
path = "/vault-data"
tls_skip_verify = "true"
retry_join {
leader_api_addr = "https://vault-cluster-0:8200"
leader_ca_cert_file = "/opt/ca/vault.crt"
leader_client_cert_file = "/opt/ca/vault.crt"
leader_client_key_file = "/opt/ca/vault.key"
}
retry_join {
leader_api_addr = "https://vault-cluster-1:8200"
leader_ca_cert_file = "/opt/ca/vault.crt"
leader_client_cert_file = "/opt/ca/vault.crt"
leader_client_key_file = "/opt/ca/vault.key"
}
retry_join {
leader_api_addr = "https://vault-cluster-2:8200"
leader_ca_cert_file = "/opt/ca/vault.crt"
leader_client_cert_file = "/opt/ca/vault.crt"
leader_client_key_file = "/opt/ca/vault.key"
}
}
seal "transit" {
address = "https://vaulttransit:8200"
disable_renewal = "false"
key_name = "autounseal"
mount_path = "transit/"
tls_skip_verify = "true"
}
listener "tcp" {
address = "0.0.0.0:8200"
tls_cert_file = "/opt/ca/vault.crt"
tls_key_file = "/opt/ca/vault.key"
tls_skip_verify = "true"
tls_disable_client_certs = "true"
}
ui=true
disable_mlock = true
As I am using external open source vault image and my load balancer is an internal LB (which has internal CA cert). I am suspecting my vault pod is not able to recognize the CA cert provided by my load balancer when it tries to ping port 8200(TCP listener is started by vault on this port)
These logs are harmless and not causing any issue but they are unnecessary noise which I want to avoid. My vault nodes are working on https and there seems to be no issue in their functionality.
Can someone please help understand why vault TCP listener is trying to do TLS handshake even though I have explicitly specified tls_disable_client_certs = "true"
Again these logs are flooding my pods every 5 sec when my LB tries to do a health check on my pods using nc -vz podip podPort
My vault version is 1.5.3

Answer 1

The messages are not about client certs or CA certs, a TLS handshake happens whether the client presents a certificate or not.
Instead, it is because a TCP connection is created and established and the Go library now wants to start a TLS handshake. Instead, the other side (the health checker) just hangs up and the TLS handshake never happens. Go then logs this message.
You are correct in saying that it is harmless, this is purely a side effect of port-liveness health checking. It is however spammy and annoying.
You have two basic options to get around this:
filter the messages out of the logs when persisting them
change to a different type of health check
I would recommend the second option: switch to a different health check. Vault has a /sys/health endpoint that can be used with HTTPS health checks.
In addition to getting rid of the TLS warning messages, the health endpoint also allows to you check for active and unsealed nodes.