Filebeat: Certificate signed by unknown authority - ssl

Question

I am getting this error from filebeat:
Failed to connect to backoff(elasticsearch(https://elk.example.com:9200)): Get https://elk.example.com:9200: x509: certificate signed by unknown authority
INFO pipeline/output.go:93 Attempting to reconnect to backoff(elasticsearch(https://elk.example.com:9200)) with 1468 reconnect attempt(s)
INFO [publish] pipeline/retry.go:189 retryer: send unwait-signal to consumer
INFO [publish] pipeline/retry.go:191 done
INFO [publish] pipeline/retry.go:166 retryer: send wait signal to consumer
INFO [publish] pipeline/retry.go:168 done
However, elasticsearch is having valid SSL by letsencrypt. (This is not a self-signed certificate).
Filebeat kubernetes config:
output.elasticsearch:
hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}']
username: ${ELASTICSEARCH_USERNAME}
password: ${ELASTICSEARCH_PASSWORD}
I tried adding these parameters in config file and it worked. But, why do I need to bypass verification even if certificate is valid.
ssl.verification_mode: "none"

Answer 1

The reason is either an old default operating system truststore that does not feature your very much valid and well-known trusted CA chains or the Elasticsearch certificate is self-signed or signed by a private CA.
You can choose from a number of solutions:
Run your filebeat in an environment (server, container, etc) with an updated default truststore that knows the CA that signed your certificate - i.e: upgrade to a newer version of the operating system or updated container image.
Remove your ssl.verification_mode: "none" configuration and add a ssl.certificate_authorities point to one or more PEM files with the to-be trusted CA certificates.
output.elasticsearch:
hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}']
username: ${ELASTICSEARCH_USERNAME}
password: ${ELASTICSEARCH_PASSWORD}
ssl.certificate_authorities: ["/path/to/ca.pem"]
Remove your ssl.verification_mode: "none" configuration and add a ssl.certificate_authorities configuration with the embedded to-be trusted CA certificate directly in the YAML configuration.
Example from filebeat configuration documentation.
certificate_authorities:
- |
-----BEGIN CERTIFICATE-----
MIIDCjCCAfKgAwIBAgITJ706Mu2wJlKckpIvkWxEHvEyijANBgkqhkiG9w0BAQsF
ADAUMRIwEAYDVQQDDAlsb2NhbGhvc3QwIBcNMTkwNzIyMTkyOTA0WhgPMjExOTA2
MjgxOTI5MDRaMBQxEjAQBgNVBAMMCWxvY2FsaG9zdDCCASIwDQYJKoZIhvcNAQEB
BQADggEPADCCAQoCggEBANce58Y/JykI58iyOXpxGfw0/gMvF0hUQAcUrSMxEO6n
fZRA49b4OV4SwWmA3395uL2eB2NB8y8qdQ9muXUdPBWE4l9rMZ6gmfu90N5B5uEl
94NcfBfYOKi1fJQ9i7WKhTjlRkMCgBkWPkUokvBZFRt8RtF7zI77BSEorHGQCk9t
/D7BS0GJyfVEhftbWcFEAG3VRcoMhF7kUzYwp+qESoriFRYLeDWv68ZOvG7eoWnP
PsvZStEVEimjvK5NSESEQa9xWyJOmlOKXhkdymtcUd/nXnx6UTCFgnkgzSdTWV41
CI6B6aJ9svCTI2QuoIq2HxX/ix7OvW1huVmcyHVxyUECAwEAAaNTMFEwHQYDVR0O
BBYEFPwN1OceFGm9v6ux8G+DZ3TUDYxqMB8GA1UdIwQYMBaAFPwN1OceFGm9v6ux
8G+DZ3TUDYxqMA8GA1UdEwEB/wQFMAMBAf8wDQYJKoZIhvcNAQELBQADggEBAG5D
874A4YI7YUwOVsVAdbWtgp1d0zKcPRR+r2OdSbTAV5/gcS3jgBJ3i1BN34JuDVFw
3DeJSYT3nxy2Y56lLnxDeF8CUTUtVQx3CuGkRg1ouGAHpO/6OqOhwLLorEmxi7tA
H2O8mtT0poX5AnOAhzVy7QW0D/k4WaoLyckM5hUa6RtvgvLxOwA0U+VGurCDoctu
8F4QOgTAWyh8EZIwaKCliFRSynDpv3JTUwtfZkxo6K6nce1RhCWFAsMvDZL8Dgc0
yvgJ38BRsFOtkRuAGSf6ZUwTO8JJRRIFnpUzXflAnGivK9M13D5GEQMmIl6U9Pvk
sxSmbIUfc2SGJGCJD4I=
-----END CERTIFICATE-----
Disclaimer: You have not provided a filebeat version so I assumed the latest one. Nevertheless, this kind of configuration will probably be the same across filebeat versions.