syslog-ng unable to send logs via tls - handshake error - ssl

Question

Unable to send logs to a syslog-ng docker container using TLS (6514)
Logs are being transmitted successfully not using TLS on port 601.
305ef6ab4973 syslog-ng[1]: Syslog connection accepted; fd='14', client='AF_INET(172.17.0.3:35362)', local='AF_INET(0.0.0.0:6514)'
305ef6ab4973 syslog-ng[1]: SSL error while reading stream; tls_error='SSL routines:tls_process_client_certificate:peer did not return a certificate', location='/etc/syslog-ng/syslog-ng.conf:35:9'
305ef6ab4973 syslog-ng[1]: I/O error occurred while reading; fd='14', error='Connection reset by peer (104)'
305ef6ab4973 syslog-ng[1]: Syslog connection closed; fd='14', client='AF_INET(172.17.0.3:35362)', local='AF_INET(0.0.0.0:6514)'
environment setup: Debian 9 VM, docker, latest syslog-ng.
syslog-ng version
root#305ef6ab4973:/etc/syslog-ng# syslog-ng --version
syslog-ng 3 (3.21.1)
Config version: 3.21
Installer-Version: 3.21.1
Revision: 3.21.1-1
Compile-Date: May 3 2019 09:11:19
Module-Directory: /usr/lib/syslog-ng/3.21
Module-Path: /usr/lib/syslog-ng/3.21
Include-Path: /usr/share/syslog-ng/include
Available-Modules: cryptofuncs,kvformat,tfgetent,add-contextual-data,afsql,afuser,xml,riemann,json-plugin,geoip-plugin,redis,pacctformat,afamqp,pseudofile,hook-commands,examples,stardate,geoip2-plugin,tags-parser,system-source,graphite,date,kafka,snmptrapd-parser,confgen,afprog,basicfuncs,afsmtp,http,linux-kmsg-format,map-value-pairs,appmodel,disk-buffer,affile,afsocket,afstomp,afmongodb,csvparser,mod-java,syslogformat,cef,mod-python,sdjournal,dbparser
Enable-Debug: off
Enable-GProf: off
Enable-Memtrace: off
Enable-IPv6: on
Enable-Spoof-Source: on
Enable-TCP-Wrapper: on
Enable-Linux-Caps: on
Enable-Systemd: on
generated syslog msgs with loggen on port 601 non-tls
root#e41017b55dfa:loggen --stream 172.17.0.2 601
count=1816, rate = 915.72 msg/sec
count=2274, rate = 914.78 msg/sec
count=2732, rate = 914.93 msg/sec
logs are written in the log file for 601 conn
sudo tail -n 10 syslog-ng/logs/syslog-ng/tcp_601.log
Jul 21 10:35:11 ip-172-17-0-3 prg00000[1234]: seq: 0000004294, thread: 0000, runid: 1563705308, stamp: 2019-07-21T10:35:11 PADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADD
Jul 21 10:35:11 ip-172-17-0-3 prg00000[1234]: seq: 0000004295, thread: 0000, runid: 1563705308, stamp: 2019-07-21T10:35:11 PADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADD
Jul 21 10:35:11 ip-172-17-0-3 prg00000[1234]: seq: 0000004296, thread: 0000, runid: 1563705308, stamp: 2019-07-21T10:35:11 PADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADD
syslog messages log (601 conn)
Jul 21 10:39:14 305ef6ab4973 syslog-ng[1]: Syslog connection accepted; fd='18', client='AF_INET(****)', local='AF_INET(0.0.0.0:601)'
Jul 21 10:39:44 305ef6ab4973 syslog-ng[1]: Syslog connection closed; fd='18', client='AF_INET(****)', local='AF_INET(0.0.0.0:601)'
When I'm using TLS, I'm receiving the following error
client-side:
root#e41017b55dfa:#loggen --use-ssl 172.17.0.2 6514
error [loggen_helper.c:open_ssl_connection:247] SSL connect failed
139771316958976:error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:../ssl/record/rec_layer_s3.c:1407:SSL alert number 40
error [ssl_plugin.c:active_thread_func:313] can not connect to 172.17.0.2:6514 (0x5566c837e800)
Total runtime = 0.500195, count = 0
server-side:
305ef6ab4973 syslog-ng[1]: Syslog connection accepted; fd='14', client='AF_INET(172.17.0.3:35362)', local='AF_INET(0.0.0.0:6514)'
305ef6ab4973 syslog-ng[1]: SSL error while reading stream; tls_error='SSL routines:tls_process_client_certificate:peer did not return a certificate', location='/etc/syslog-ng/syslog-ng.conf:35:9'
305ef6ab4973 syslog-ng[1]: I/O error occurred while reading; fd='14', error='Connection reset by peer (104)'
305ef6ab4973 syslog-ng[1]: Syslog connection closed; fd='14', client='AF_INET(172.17.0.3:35362)', local='AF_INET(0.0.0.0:6514)'
conn test using openssl
root#e41017b55dfa:/etc/syslog-ng# openssl s_client -connect 172.17.0.2:6514
CONNECTED(00000003)
depth=1 C = IL, ST = ***, L = ***, O = ***, OU = IT, CN = *** Syslog CA, emailAddress = ***#***.com
verify return:1
depth=0 C = IL, ST = ***, L = ***, O = ***, OU = IT, CN = 172.17.0.2
verify return:1
140233519988800:error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:../ssl/record/rec_layer_s3.c:1407:SSL alert number 40
---
Certificate chain
0 s:/C=IL/ST=***/L=***/O=***/OU=IT/CN=172.17.0.2
i:/C=IL/ST=***/L=***/O=***/OU=IT/CN=*** Syslog CA/emailAddress=***#***.com
1 s:/C=IL/ST=***/L=***/O=***/OU=IT/CN=*** Syslog CA/emailAddress=***#***.com
i:/C=IL/ST=***/L=***/O=***/OU=IT/CN=*** Syslog CA/emailAddress=***#***.com
---
Server certificate
-----BEGIN CERTIFICATE-----
MIID7TCCAtWgAwIBAgIBATANBgkqhkiG9w0BAQsFADCBkTELMAkGA1UEBhMCSUwx
ETAPBgNVBAgMCFRlbCBBdml2MREwDwYDVQQHDAhUZWwgQXZpdjEOMAwGA1UECgwF
QXJtaXMxCzAJBgNVBAsMAklUMRgwFgYDVQQDDA9Bcm1pcyBTeXNsb2cgQ0ExJTAj
BgkqhkiG9w0BCQEWFm9tcmkudHNhYmFyaUBhcm1pcy5jb20wHhcNMTkwNzE4MTAx
MzQ3WhcNMjAwNzE3MTAxMzQ3WjBlMQswCQYDVQQGEwJJTDERMA8GA1UECAwIVGVs
IEF2aXYxETAPBgNVBAcMCFRlbCBBdml2MQ4wDAYDVQQKDAVBcm1pczELMAkGA1UE
CwwCSVQxEzARBgNVBAMMCjE3Mi4xNy4wLjIwggEiMA0GCSqGSIb3DQEBAQUAA4IB
DwAwggEKAoIBAQDSVTVKoNlgPk1q9MgbPF1ndDIhTFsXp62XPdNNWyP79GGunPlM
o+oqJJJh+SDP/0BUivyvYdH4gFdZ40RZ138CQz1L+i9sBK4alizRIqxWT379lnYd
nieMYP25uBQPw8TothegtHA30+PFg/qEVd/3bQQVFJ/z0Q6GsOkw/Qc4kS+hhP6B
dny2ul8yyS4oNeM4rMo/1/F8NKsdOlt/4St2aVo5kuuyosOdKaaXzzqeVI7QdqaJ
kuMwC5sGATDZ2qwr9TEgBVzZs5sFixOaA0vTb7FqVOfcBq1Crrf9qnNTzQXzjjjH
3eQ4tZXbVOTopxwR7zgqO/nR/3IAvVnirsjNAgMBAAGjezB5MAkGA1UdEwQCMAAw
LAYJYIZIAYb4QgENBB8WHU9wZW5TU0wgR2VuZXJhdGVkIENlcnRpZmljYXRlMB0G
A1UdDgQWBBSMTVONnqW+gof7SKD0V6uPZLoOdDAfBgNVHSMEGDAWgBTaK4jNVP3+
1V4wUSM+Gx7iYSjFKTANBgkqhkiG9w0BAQsFAAOCAQEAQqOJbvHcjG6pYbmtwexJ
C56a1qE0C9fjIlHY+EKuE1e/jTfIu1opggwTbov5BS9MHDK0As4JkwAn/36dbGKt
SS3K/JXvnM8Ag5tv09zVgSKwYNRpuVTi52shn4ELIktVCUc2H7XW1W9r1GsjkXCV
WhtJRP9lVJi77gxICTC5x39feA/p3BkRUIRwWPY2J8quJ37FTNBGMeX8lVAW4ipR
UbG3DQgj2r/HonjmZ5kWH8Bd46RZhpE7Nt4UGRutCnyi9jo3R7PDQW1D0rhRSByO
w/uTToHfaj7ZjGb9CXeV7LRuf6z5px881puqUsWYSeEh0Tm3AnTVNOzzvKE2Pp3*****
-----END CERTIFICATE-----
subject=/C=IL/ST=***/L=***/O=***/OU=IT/CN=172.17.0.2
issuer=/C=IL/ST=***/L=***/O=***/OU=IT/CN=*** Syslog CA/emailAddress=***#***.com
---
No client certificate CA names sent
Client Certificate Types: RSA sign, DSA sign, ECDSA sign
Requested Signature Algorithms: RSA+SHA512:DSA+SHA512:ECDSA+SHA512:RSA+SHA384:DSA+SHA384:ECDSA+SHA384:RSA+SHA256:DSA+SHA256:ECDSA+SHA256:RSA+SHA224:DSA+SHA224:ECDSA+SHA224:RSA+SHA1:DSA+SHA1:ECDSA+SHA1
Shared Requested Signature Algorithms: RSA+SHA512:DSA+SHA512:ECDSA+SHA512:RSA+SHA384:DSA+SHA384:ECDSA+SHA384:RSA+SHA256:DSA+SHA256:ECDSA+SHA256:RSA+SHA224:DSA+SHA224:ECDSA+SHA224:RSA+SHA1:DSA+SHA1:ECDSA+SHA1
Peer signing digest: SHA512
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 2487 bytes and written 281 bytes
Verification: OK
---
New, TLSv1.2, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol : TLSv1.2
Cipher : ECDHE-RSA-AES256-GCM-SHA384
Session-ID:
Session-ID-ctx:
Master-Key: 02FB22BADE731CF64439D69D1F1991F3FF3BD7C4E44AF531308DD021659B1220B8BEBE94C9934659734AB10D4AF25999
PSK identity: None
PSK identity hint: None
SRP username: None
Start Time: 1563704954
Timeout : 7200 (sec)
Verify return code: 0 (ok)
Extended master secret: yes
---
root#e41017b55dfa:/etc/syslog-ng#
syslog-ng client config
#version: 3.18
#include "scl.conf"
source s_local {
internal();
};
source s_network {
default-network-drivers( ); };
destination test2_d {
network("172.17.0.2" port(6514)
transport("tls")
tls( ca-dir("/etc/syslog-ng/ca.d")
key-file("/etc/syslog-ng/cert.d/clientkey.pem")
cert-file("/etc/syslog-ng/cert.d/clientcert.pem") )
); };
log { source(s_local); destination(test2_d); };
destination d_local {
file("/var/log/messages");
file("/var/log/messages-kv.log" template("$ISODATE $HOST $(format-welf --scope all-nv-pairs)\n") frac-digits(3));
};
log {
source(s_local);
source(s_network);
destination(d_local);
};
syslog-ng server config
#version: 3.18
#include "scl.conf"
source s_local {
internal();
};
source test1_s {
network(
transport("tcp")
port(601)
flags(syslog-protocol)
);
};
destination test1_d { file("/var/log/syslog-ng/tcp_601.log" dir_group(root) group(root) create_dirs(yes) dir_perm(0777) perm(0666) owner(root) dir_owner(root)); };
log { source("test1_s"); destination("test1_d"); };
source test2_s {
network(
ip(0.0.0.0) port(6514)
transport("tls")
tls( key-file("/etc/syslog-ng/cert.d/serverkey.pem")
cert-file("/etc/syslog-ng/cert.d/servercert.pem")
ca-dir("/etc/syslog-ng/ca.d"))
); };
destination test2_d { file("/var/log/syslog-ng/tls_6514.log" dir_group(root) group(root) create_dirs(yes) dir_perm(0777) perm(0666) owner(root) dir_owner(root)); };
log { source("test2_s"); destination("test2_d"); };
destination d_local {
file("/var/log/messages");
file("/var/log/messages-kv.log" template("$ISODATE $HOST $(format-welf --scope all-nv-pairs)\n") frac-digits(3));
};
log {
source(s_local);
destination(d_local);
};

Answer 1

You've tested your configuration using loggen --use-ssl and openssl s_client without specifying a client certificate (loggen does not support client certs, openssl s_client does).
The error message on the server side is about the missing client cert: peer did not return a certificate.
If you prefer not to use mutual authentication, you can make it optional by adding the peer-verify(optional-trusted) TLS option to the server config:
source test2_s {
network(
port(6514)
transport("tls")
tls( key-file("/etc/syslog-ng/cert.d/serverkey.pem")
cert-file("/etc/syslog-ng/cert.d/servercert.pem")
ca-dir("/etc/syslog-ng/ca.d")
peer-verify(optional-trusted)
)
);
};
syslog-ng Admin Guide - TLS options