Verifying my self-signed certificate with openSSL - authentication

Question

I own the server and I own the customer's executable. I'd like to establish a secure TLS connection between them.
I can embed whatever I want into the client executable but I'm not sure how to validate a self-assigned certificate that my client received from a connection to the server, i.e. from a SSL_get_peer_certificate call.
I read around that certificates are just public keys with metadata parts signed with the private key. Can I somehow verify that the certificate the server sent me has indeed all the metadata correctly signed by embedding the public key into my client application? Is this possible (and if it is, how?)

Answer 1

I'm not sure how to validate a self-assigned certificate that my client received from a connection to the server ...
Depending on the OpenSSL library you are using, you have to perform two or three steps for verification. The two versions bisect at OpenSSL 1.1.0. OpenSSL 1.1.0 and above performs hostname validation so it only takes two steps. OpenSSL 1.0.2 and below does not perform hostname validation so it requires three steps.
The steps detailed below are from SSL/TLS Client on the OpenSSL wiki.
Server Certificate
Both OpenSSL 1.0.2 and 1.1.0 require you to check for the presence of a certificate. If you use ADH (Anonymous Diffie-Hellman), TLS-PSK (Preshared Key), TLS_SRP (Secure Remote Password), then there may not be a server certificate to verify.
You get the server's certificate with SSL_get_peer_certificate. If it returns non-NULL, then a certificate is present. Lack of a certificate may or may not be a reason to fail.
Certificate Chain
Both OpenSSL 1.0.2 and 1.1.0 require you to check the result of chain validation. Chain validation is part of path building, and its detailed in RFC 4158, Certification Path Building.
You get the result of path validation with SSL_get_verify_result.
Certificate Names
OpenSSL 1.0.2 an below requires you to verify the hostname matches a name listed in the certificate. Its a big topic, but the short of it is: any hostname or dns name needs to be present in the certifcate's Subject Alternative Name (SAN), and not the Common Name (CN). Also see How do you sign Certificate Signing Request with your Certification Authority and How to create a self-signed certificate with openssl? It provides a lot of background information on X.509 server certificates, how to present names, and where the various rules come from.
Effectively, you fetch the SANs with X509_get_ext_d2i(cert, NID_subject_alt_name, ...). Then you loop over the list and extract each name with sk_GENERAL_NAME_num. Then, you extract a GENERAL_NAME entry and ASN1_STRING_to_UTF8, and see if it matches the name you tried to connect to.
Below are the routines for printing the Subject Alternative Name (SAN) and the Common Name (CN). It came from the example on the OpenSSL wiki page.
void print_san_name(const char* label, X509* const cert)
{
int success = 0;
GENERAL_NAMES* names = NULL;
unsigned char* utf8 = NULL;
do
{
if(!cert) break; /* failed */
names = X509_get_ext_d2i(cert, NID_subject_alt_name, 0, 0 );
if(!names) break;
int i = 0, count = sk_GENERAL_NAME_num(names);
if(!count) break; /* failed */
for( i = 0; i < count; ++i )
{
GENERAL_NAME* entry = sk_GENERAL_NAME_value(names, i);
if(!entry) continue;
if(GEN_DNS == entry->type)
{
int len1 = 0, len2 = -1;
len1 = ASN1_STRING_to_UTF8(&utf8, entry->d.dNSName);
if(utf8) {
len2 = (int)strlen((const char*)utf8);
}
if(len1 != len2) {
fprintf(stderr, " Strlen and ASN1_STRING size do not match (embedded null?): %d vs %d\n", len2, len1);
}
/* If there's a problem with string lengths, then */
/* we skip the candidate and move on to the next. */
/* Another policy would be to fails since it probably */
/* indicates the client is under attack. */
if(utf8 && len1 && len2 && (len1 == len2)) {
fprintf(stdout, " %s: %s\n", label, utf8);
success = 1;
}
if(utf8) {
OPENSSL_free(utf8), utf8 = NULL;
}
}
else
{
fprintf(stderr, " Unknown GENERAL_NAME type: %d\n", entry->type);
}
}
} while (0);
if(names)
GENERAL_NAMES_free(names);
if(utf8)
OPENSSL_free(utf8);
if(!success)
fprintf(stdout, " %s: <not available>\n", label);
}
void print_cn_name(const char* label, X509_NAME* const name)
{
int idx = -1, success = 0;
unsigned char *utf8 = NULL;
do
{
if(!name) break; /* failed */
idx = X509_NAME_get_index_by_NID(name, NID_commonName, -1);
if(!(idx > -1)) break; /* failed */
X509_NAME_ENTRY* entry = X509_NAME_get_entry(name, idx);
if(!entry) break; /* failed */
ASN1_STRING* data = X509_NAME_ENTRY_get_data(entry);
if(!data) break; /* failed */
int length = ASN1_STRING_to_UTF8(&utf8, data);
if(!utf8 || !(length > 0)) break; /* failed */
fprintf(stdout, " %s: %s\n", label, utf8);
success = 1;
} while (0);
if(utf8)
OPENSSL_free(utf8);
if(!success)
fprintf(stdout, " %s: <not available>\n", label);
}
Verifying my self-signed certificate with openSSL
Because its your self-signed certificate, you can do even better than above. You have a priori knowledge of the host's public key. You can pin the public key, and just use the certificate to deliever the public key or as a presentation detail.
To pin the public key, see Public Key Pinning over at OWASP.
You should also avoid the IETF's RFC 7469, Public Key Pinning Extension for HTTP with Overrides. The IETF's rendition allows the attacker to break a known good pinset so the attacker can MitM the connection. They also suppress reporting the problem, so the user agent becomes complicit in the coverup.