Does openssl refuse self signed certificates without basic constraints? - ssl

Question

I have two extremely similar self signed certificates, generated via two different methods.
To test them I have:
Added an entry in my hosts file for local.mydomain.com
Set up an nginx server to listen on that domain on port 443 with the certificate under test plus associated private key (I then switch the cert and restart nginx to compare)
Connected to nginx with openssl s_client -connect local.mydomain.com -CAfile /path/to/the/ca/cert.pem
One certificate fails:
CONNECTED(00000003)
depth=0 CN = local.mydomain.com
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 CN = local.mydomain.com
verify error:num=21:unable to verify the first certificate
verify return:1
---
Certificate chain
0 s:/CN=local.mydomain.com
i:/CN=local.mydomain.com
---
One certificate succeeds:
CONNECTED(00000003)
depth=0 CN = local.mydomain.com
verify return:1
---
Certificate chain
0 s:/CN = local.mydomain.com
i:/CN = local.mydomain.com
---
I compare the details of the certificates with openssl x509 -in /path/to/the/ca/cert.pem -text -noout
The failing cert:
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
47:dc:02:c7:11:fc:8e:96:45:22:aa:6b:23:79:32:ca
Signature Algorithm: sha256WithRSAEncryption
Issuer: CN=local.mydomain.com
Validity
Not Before: Nov 18 11:55:31 2016 GMT
Not After : Nov 18 12:15:31 2017 GMT
Subject: CN=local.mydomain.com
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
Public-Key: (2048 bit)
Modulus:
<stuff>
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Key Usage: critical
Digital Signature, Key Encipherment
X509v3 Extended Key Usage:
TLS Web Client Authentication, TLS Web Server Authentication
X509v3 Subject Alternative Name:
DNS:local.mydomain.com
X509v3 Subject Key Identifier:
6D:4F:AF:E4:60:23:72:E5:83:27:91:7D:1D:5F:E9:7C:D9:B6:00:2A
Signature Algorithm: sha256WithRSAEncryption
<stuff>
The working cert:
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
9b:6b:3d:a3:b9:a3:a4:b4
Signature Algorithm: sha256WithRSAEncryption
Issuer: CN=local.mydomain.com
Validity
Not Before: Nov 19 13:27:30 2016 GMT
Not After : Nov 19 13:27:30 2017 GMT
Subject: CN=local.mydomain.com
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
Public-Key: (2048 bit)
Modulus:
<stuff>
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Subject Key Identifier:
03:E7:DA:AA:2E:CC:23:ED:C5:07:3D:E1:33:86:F5:22:D4:76:EB:CB
X509v3 Authority Key Identifier:
keyid:03:E7:DA:AA:2E:CC:23:ED:C5:07:3D:E1:33:86:F5:22:D4:76:EB:CB
X509v3 Basic Constraints:
CA:TRUE
Signature Algorithm: sha256WithRSAEncryption
57<stuff>
Looking at this the most obvious difference is that the working cert has CA:TRUE under X509v3 Basic Constraints. However, from reading around the web I was under the impression that self signed certs weren't meant to be CAs, in particular this says they normally won't be:
https://security.stackexchange.com/questions/44340/basic-self-signed-certificate-questions
The answer there says that being self-signed there is no CA involved. But maybe openssl requires self signed certs to have that set anyway?

Answer 1

From my own experiments I can confirm what you see. My explanation of the behavior is that a self signed certificate is still a certificate which is signed by the issuer, even if the issuer's certificate is the certificate itself. But only CA certificates can be used to sign certificates, i.e. that's exactly the constraint CA:true allows. This means that a self-signed certificate needs also to be a CA certificate with the constraint CA:true.

Answer 2

RFC5280 says:
So, if your certificate does not have CA:TRUE flag, this certificate may not be used to verify the signature on any certificate, including itself. OpenSSL correctly follows the RFC.
It is incorrect to think that a certificate belongs to one of two types, either "CA certificate" or "end-entity certificate". A certificate with CA:TRUE can be used for authenticating the entity. This is exactly what you do when you authenticate with a self-signed certificate. It can also be a certificate with CA:TRUE, signed by someone else.