WebSocketpp handshake issue with TLS - ssl

Question

I have been learning with WebSocket++ and built some of the server examples (Windows 10 Visual Studio 2019). The non-TLS examples work without issues, however, the TLS-enabled examples (echo_server_both.cpp and echo_server_tls.cpp) can't do the handshake. I am very new to web development in general so I know I must be doing something wrong with regards to the certificate and keys.
I am testing the servers with WebSocket King client, an extension of Google Chrome that connects correctly to other websocket servers like wss://echo.websocket.org and to my own localhost when I don't use TLS.
The echo_server_both example comes with a server.pem file, and the echo_server_tls example comes with server.pem and dh.pem. I have used the same files that come with the samples, and I have also tried generating and registering my own .pem files using openSSL. In both cases I get this when the client tries to connect:
[2021-06-29 20:51:21] [error] handle_transport_init received error: sslv3 alert certificate unknown
[2021-06-29 20:51:21] [fail] WebSocket Connection [::1]:63346 - "" - 0 asio.ssl:336151574 sslv3 alert certificate unknown
[2021-06-29 20:51:21] [info] asio async_shutdown error: asio.ssl:336462231 (shutdown while in init)
I discovered these errors after I edited handle_init() in tls.hpp, following a suggestion in another site, to look like this:
void handle_init(init_handler callback,lib::asio::error_code const & ec) {
if (ec) {
//m_ec = socket::make_error_code(socket::error::tls_handshake_failed);
m_ec = ec;
} else {
m_ec = lib::error_code();
}
callback(m_ec);
}
This change let the actual openSSL error to show in the console, otherwise it would show a generic "handshake failed" error.
I know I'm not doing what I should with the certificates, but I have no idea where else to look or what to do next. Can anyone here help please? Should I use the .pem files that come with the examples, or should I generate my own? in case I should generate my own, what would be the openSSL command to do that correctly and how do I tell my PC to recognize these as valid so that the server works?

Answer 1

Found the problem: WebSocket++ will not accept a self-signed certificate (the ones you can create directly in your own PC using OpenSSL or the Windows utilities). There is no way around it. You must have a valid, authority-validated and endorsed certificate. You can get such a certificate for free (valid only for 90 days) from https://zerossl.com/. The site has detailed instructions on how to request, obtain and install a certificate. After getting a valid certificate and installing it on my server, everything worked as it should.