Is "X509v3 Key Usage: Certificate Sign" required for passing openssl verify? - ssl

Below are three self-signed certificates I've made and verified with openssl verify:
Certificate #1:
X509v3 Basic Constraints:
CA:FALSE
X509v3 Key Usage:
Digital Signature, Key Encipherment
This self-signed certificate is not a CA, it's missing the "Certificate Sign" value, and it fails verification:
$ openssl verify -CAfile ca_false_cert.crt ca_false_cert.crt
error 20 at 0 depth lookup: unable to get local issuer certificate
error ca_false_cert.crt: verification failed
Certificate #2:
X509v3 Basic Constraints:
CA:FALSE
X509v3 Key Usage:
Digital Signature, Key Encipherment, Certificate Sign
This self-signed certificate is not a CA, it includes the "Certificate Sign" value, and it passes verification:
$ openssl verify -CAfile ca_false_sign_cert.crt ca_false_sign_cert.crt
ca_false_sign_cert.crt: OK
Certificate #3:
X509v3 Basic Constraints:
CA:TRUE
X509v3 Key Usage:
Digital Signature, Key Encipherment
This self-signed certificate is a CA, it's missing the "Certificate Sign" value, and it fails verification:
$ openssl verify -CAfile ca_true_sign_cert.crt ca_true_sign_cert.crt
error 20 at 0 depth lookup: unable to get local issuer certificate
error ca_true_sign_cert.crt: verification failed
My two questions are:
Is X509v3 Key Usage: Certificate Sign necessary for a self-signed certificate (or for CA certificates in general)?
Are non-CA (i.e. without X509v3 Basic Constraints: CA:TRUE, or with CA:FALSE) self-signed certificates even valid? Or does openssl verify overlook this?

Related

Openssl different intermediates for client and server certs

I am trying to generate a certificate-(authority)-chain, where some intermediates have specific tasks.
There will be a root certificate (skipped for testing) that signs the actual CA, this CA on the one hand, signs a CA that should only be able to sign client-certificates, and on the other hand, signs one/multiple CA's that can only perform server-certificates-signing. The idea is, that this "server-ca" will be deployed into an embedded system in order to create a new server certificate with different sans if nescessary, but should not be able to sign client-certificates. (see my fine drawing)
I did not find much about constraining a ca. Here it says it would be possible, so I tried to create a test setup where I defined an extendedKeyUsage for the device/server-CA. The X509v3 extensions in the certs I created look like the following
Server/Device-CA:
X509v3 extensions:
X509v3 Basic Constraints: critical
CA:TRUE <-(yeah, forgot 'bout the path-length - hope this is not the problem)
X509v3 Subject Key Identifier:
51:0F:8C:55:88:4D:3E:25:DC:EC:6D:73:39:E4:7D:27:2E:AF:E4:2D
X509v3 Authority Key Identifier:
keyid:B3:6A:53:D9:9B:CF:74:69:B5:64:73:91:D7:18:92:30:E3:A7:7A:A6
X509v3 Key Usage: critical
Digital Signature, Certificate Sign, CRL Sign
X509v3 Extended Key Usage: critical
TLS Web Server Authentication <--
Server/Device-CA -> Server-Cert
X509v3 extensions:
X509v3 Basic Constraints:
CA:FALSE
Netscape Cert Type:
SSL Server
Netscape Comment:
OpenSSL Generated Server Certificate
X509v3 Subject Key Identifier:
AD:BC:E0:73:50:AB:7F:BE:3C:43:71:4F:07:06:D8:3F:1A:38:81:4C
X509v3 Authority Key Identifier:
keyid:51:0F:8C:55:88:4D:3E:25:DC:EC:6D:73:39:E4:7D:27:2E:AF:E4:2D
DirName:/C=XXX/ST=XXX/L=XX/O=XXX/OU=XXX/CN=XXX/emailAddress=XXX
serial:10:00
X509v3 Key Usage: critical
Digital Signature, Key Encipherment
X509v3 Extended Key Usage:
TLS Web Server Authentication <--
Server/Device-CA -> Client-Cert
X509v3 extensions:
X509v3 Basic Constraints:
CA:FALSE
Netscape Cert Type:
SSL Client, S/MIME
Netscape Comment:
OpenSSL Generated Client Certificate
X509v3 Subject Key Identifier:
AD:BC:E0:73:50:AB:7F:BE:3C:43:71:4F:07:06:D8:3F:1A:38:81:4C
X509v3 Authority Key Identifier:
keyid:51:0F:8C:55:88:4D:3E:25:DC:EC:6D:73:39:E4:7D:27:2E:AF:E4:2D
X509v3 Key Usage: critical
Digital Signature, Non Repudiation, Key Encipherment
X509v3 Extended Key Usage:
TLS Web Client Authentication, E-mail Protection <--
Anyway when I use openssl verify it says both certificates were valid. I hope you can help me.
Thank you
A simple openssl verify does not know that the leaf certificate should be used as a client certificate. That's why it will not take the purpose of the certificates into account when validating the trust chain:
$ openssl verify -CAfile root.pem -untrusted middle.pem client.pem
client.pem: OK
With an explicitly given purpose of sslclient it fails instead:
$ openssl verify -purpose sslclient -CAfile root.pem -untrusted middle.pem client.pem
CN = middle
error 26 at 1 depth lookup: unsupported certificate purpose
error client.pem: verification failed
Having a different intermediate certificate which has sslclient as purpose to it instead succeeds:
$ openssl verify -purpose sslclient -CAfile root.pem -untrusted alt-middle.pem client.pem
client.pem: OK

why does fabric-ca start as self-signed certificate?

Here is how I understand the flow.
version: '2'
services:
shop_ca:
image: hyperledger/fabric-ca
environment:
- FABRIC_CA_HOME=/etc/hyperledger/fabric-ca-server
- FABRIC_CA_SERVER_CA_NAME=shop_ca
- FABRIC_CA_SERVER_CA_CERTFILE=/etc/hyperledger/fabric-ca-server-config/ca-cert.pem
- FABRIC_CA_SERVER_CA_KEYFILE=/etc/hyperledger/fabric-ca-server-config/ca-key.pem
ports:
- "7054:7054"
command: sh -c 'fabric-ca-server start -b admin3:admin3'
volumes:
- ./conf.yaml:/etc/hyperledger/fabric-ca-server/fabric-ca-server-config.yaml
container_name: shop_ca
I passed my certfile and keyfile as options.
When the fabric-ca-server starts, what it should do is create ca-cert and ca-key pem files in /etc/hyperledger/fabric-ca-server folder. then use the /etc/hyperledger/fabric-ca-server/fabric-ca-server-config.yaml config file to generate the final certificate(path must be /etc/hyperledger/fabric-ca-server-config/ca-cert.pem) that will be used to issue other certificates.
What I don't understand is this generated certificate /etc/hyperledger/fabric-ca-server-config/ca-cert.pem is self-signed. Question is why? I think what it does is makes csr request to /etc/hyperledger/fabric-ca-server/ca-cert.pem and this ca-cert.pem issues another ca-cert.pem. This question happened on my mind because when I use openssl and print the final generated ca-cert certificate, issuer and subject are both the same. I think issuer has to be /etc/hyperledger/fabric-ca-server/ca-cert.pem and subject must be /etc/hyperledger/fabric-ca-server-config/ca-cert.pem. but both issuer and subject are /etc/hyperledger/fabric-ca-server-config/ca-cert.pem. Why?
The top root certificate for any Certificate Authority is always self-signed (check out the Verisign cert at the bottom of this post) ... that's why you explicitly trust root certificates.
Fabric CA allows you to either specify an existing root key pair or if the specified files do not exist it generates them for you. (if the cert file exists but a matching private key cannot be found you'll get an error and fabric-ca-server will not start).
When fabric-ca-server generates its own self-signed keypair, it actually generates the private key in the msp/keystore folder but it will store the self-signed X509 cert in the file specified via FABRIC_CA_SERVER_CA_CERTFILE if specified else it will use the location in fabric-ca-server-config.yaml. Note that if you use the FABRIC_CA_SERVER_CA_CERTFILE override, the value is not updated in the config file (perhaps this is causing some confusion).
Verisign Primary
Garis-MBP:tmp gsingh$ openssl x509 -noout -text -in verisign.pem
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
18:da:d1:9e:26:7d:e8:bb:4a:21:58:cd:cc:6b:3b:4a
Signature Algorithm: sha1WithRSAEncryption
Issuer: C=US, O=VeriSign, Inc., OU=VeriSign Trust Network, OU=(c) 2006 VeriSign, Inc. - For authorized use only, CN=VeriSign Class 3 Public Primary Certification Authority - G5
Validity
Not Before: Nov 8 00:00:00 2006 GMT
Not After : Jul 16 23:59:59 2036 GMT
Subject: C=US, O=VeriSign, Inc., OU=VeriSign Trust Network, OU=(c) 2006 VeriSign, Inc. - For authorized use only, CN=VeriSign Class 3 Public Primary Certification Authority - G5
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
Public-Key: (2048 bit)
Modulus:
00:af:24:08:08:29:7a:35:9e:60:0c:aa:e7:4b:3b:
4e:dc:7c:bc:3c:45:1c:bb:2b:e0:fe:29:02:f9:57:
08:a3:64:85:15:27:f5:f1:ad:c8:31:89:5d:22:e8:
2a:aa:a6:42:b3:8f:f8:b9:55:b7:b1:b7:4b:b3:fe:
8f:7e:07:57:ec:ef:43:db:66:62:15:61:cf:60:0d:
a4:d8:de:f8:e0:c3:62:08:3d:54:13:eb:49:ca:59:
54:85:26:e5:2b:8f:1b:9f:eb:f5:a1:91:c2:33:49:
d8:43:63:6a:52:4b:d2:8f:e8:70:51:4d:d1:89:69:
7b:c7:70:f6:b3:dc:12:74:db:7b:5d:4b:56:d3:96:
bf:15:77:a1:b0:f4:a2:25:f2:af:1c:92:67:18:e5:
f4:06:04:ef:90:b9:e4:00:e4:dd:3a:b5:19:ff:02:
ba:f4:3c:ee:e0:8b:eb:37:8b:ec:f4:d7:ac:f2:f6:
f0:3d:af:dd:75:91:33:19:1d:1c:40:cb:74:24:19:
21:93:d9:14:fe:ac:2a:52:c7:8f:d5:04:49:e4:8d:
63:47:88:3c:69:83:cb:fe:47:bd:2b:7e:4f:c5:95:
ae:0e:9d:d4:d1:43:c0:67:73:e3:14:08:7e:e5:3f:
9f:73:b8:33:0a:cf:5d:3f:34:87:96:8a:ee:53:e8:
25:15
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Basic Constraints: critical
CA:TRUE
X509v3 Key Usage: critical
Certificate Sign, CRL Sign
1.3.6.1.5.5.7.1.12:
0_.].[0Y0W0U..image/gif0!0.0...+..............k...j.H.,{..0%.#http://logo.verisign.com/vslogo.gif
X509v3 Subject Key Identifier:
7F:D3:65:A7:C2:DD:EC:BB:F0:30:09:F3:43:39:FA:02:AF:33:31:33
Signature Algorithm: sha1WithRSAEncryption
93:24:4a:30:5f:62:cf:d8:1a:98:2f:3d:ea:dc:99:2d:bd:77:
f6:a5:79:22:38:ec:c4:a7:a0:78:12:ad:62:0e:45:70:64:c5:
e7:97:66:2d:98:09:7e:5f:af:d6:cc:28:65:f2:01:aa:08:1a:
47:de:f9:f9:7c:92:5a:08:69:20:0d:d9:3e:6d:6e:3c:0d:6e:
d8:e6:06:91:40:18:b9:f8:c1:ed:df:db:41:aa:e0:96:20:c9:
cd:64:15:38:81:c9:94:ee:a2:84:29:0b:13:6f:8e:db:0c:dd:
25:02:db:a4:8b:19:44:d2:41:7a:05:69:4a:58:4f:60:ca:7e:
82:6a:0b:02:aa:25:17:39:b5:db:7f:e7:84:65:2a:95:8a:bd:
86:de:5e:81:16:83:2d:10:cc:de:fd:a8:82:2a:6d:28:1f:0d:
0b:c4:e5:e7:1a:26:19:e1:f4:11:6f:10:b5:95:fc:e7:42:05:
32:db:ce:9d:51:5e:28:b6:9e:85:d3:5b:ef:a5:7d:45:40:72:
8e:b7:0e:6b:0e:06:fb:33:35:48:71:b8:9d:27:8b:c4:65:5f:
0d:86:76:9c:44:7a:f6:95:5c:f6:5d:32:08:33:a4:54:b6:18:
3f:68:5c:f2:42:4a:85:38:54:83:5f:d1:e8:2c:f2:ac:11:d6:
a8:ed:63:6a

Does openssl refuse self signed certificates without basic constraints?

I have two extremely similar self signed certificates, generated via two different methods.
To test them I have:
Added an entry in my hosts file for local.mydomain.com
Set up an nginx server to listen on that domain on port 443 with the certificate under test plus associated private key (I then switch the cert and restart nginx to compare)
Connected to nginx with openssl s_client -connect local.mydomain.com -CAfile /path/to/the/ca/cert.pem
One certificate fails:
CONNECTED(00000003)
depth=0 CN = local.mydomain.com
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 CN = local.mydomain.com
verify error:num=21:unable to verify the first certificate
verify return:1
---
Certificate chain
0 s:/CN=local.mydomain.com
i:/CN=local.mydomain.com
---
One certificate succeeds:
CONNECTED(00000003)
depth=0 CN = local.mydomain.com
verify return:1
---
Certificate chain
0 s:/CN = local.mydomain.com
i:/CN = local.mydomain.com
---
I compare the details of the certificates with openssl x509 -in /path/to/the/ca/cert.pem -text -noout
The failing cert:
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
47:dc:02:c7:11:fc:8e:96:45:22:aa:6b:23:79:32:ca
Signature Algorithm: sha256WithRSAEncryption
Issuer: CN=local.mydomain.com
Validity
Not Before: Nov 18 11:55:31 2016 GMT
Not After : Nov 18 12:15:31 2017 GMT
Subject: CN=local.mydomain.com
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
Public-Key: (2048 bit)
Modulus:
<stuff>
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Key Usage: critical
Digital Signature, Key Encipherment
X509v3 Extended Key Usage:
TLS Web Client Authentication, TLS Web Server Authentication
X509v3 Subject Alternative Name:
DNS:local.mydomain.com
X509v3 Subject Key Identifier:
6D:4F:AF:E4:60:23:72:E5:83:27:91:7D:1D:5F:E9:7C:D9:B6:00:2A
Signature Algorithm: sha256WithRSAEncryption
<stuff>
The working cert:
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
9b:6b:3d:a3:b9:a3:a4:b4
Signature Algorithm: sha256WithRSAEncryption
Issuer: CN=local.mydomain.com
Validity
Not Before: Nov 19 13:27:30 2016 GMT
Not After : Nov 19 13:27:30 2017 GMT
Subject: CN=local.mydomain.com
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
Public-Key: (2048 bit)
Modulus:
<stuff>
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Subject Key Identifier:
03:E7:DA:AA:2E:CC:23:ED:C5:07:3D:E1:33:86:F5:22:D4:76:EB:CB
X509v3 Authority Key Identifier:
keyid:03:E7:DA:AA:2E:CC:23:ED:C5:07:3D:E1:33:86:F5:22:D4:76:EB:CB
X509v3 Basic Constraints:
CA:TRUE
Signature Algorithm: sha256WithRSAEncryption
57<stuff>
Looking at this the most obvious difference is that the working cert has CA:TRUE under X509v3 Basic Constraints. However, from reading around the web I was under the impression that self signed certs weren't meant to be CAs, in particular this says they normally won't be:
https://security.stackexchange.com/questions/44340/basic-self-signed-certificate-questions
The answer there says that being self-signed there is no CA involved. But maybe openssl requires self signed certs to have that set anyway?
From my own experiments I can confirm what you see. My explanation of the behavior is that a self signed certificate is still a certificate which is signed by the issuer, even if the issuer's certificate is the certificate itself. But only CA certificates can be used to sign certificates, i.e. that's exactly the constraint CA:true allows. This means that a self-signed certificate needs also to be a CA certificate with the constraint CA:true.
RFC5280 says:
So, if your certificate does not have CA:TRUE flag, this certificate may not be used to verify the signature on any certificate, including itself. OpenSSL correctly follows the RFC.
It is incorrect to think that a certificate belongs to one of two types, either "CA certificate" or "end-entity certificate". A certificate with CA:TRUE can be used for authenticating the entity. This is exactly what you do when you authenticate with a self-signed certificate. It can also be a certificate with CA:TRUE, signed by someone else.

SSL Certificate Cannot Be Trusted (COMODO)

I am getting the server ready for PCI DSS. There are no other problems but one which I can't solve. PCI scanner (https://www.hackerguardian.com/), says that SSL certificate can't be trusted:
SSL Certificate Cannot Be Trusted 443 / tcp / www
I have removed all other certificates from the chain, leaving only one that was purchased exactly for this server. It was signed by COMODO which is considered as trustworthy. Here is certificate dump:
openssl x509 -in /usr/local/psa/var/certificates/cert-f1nb7M -text -noout
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
e6:3c:e1:95:56:07:3c:f7:4c:5e:b3:bd:06:6d:37:f0
Signature Algorithm: sha256WithRSAEncryption
Issuer: C=GB, ST=Greater Manchester, L=Salford, O=COMODO CA Limited, CN=COMODO RSA Extended Validation Secure Server CA
Validity
Not Before: Nov 17 00:00:00 2015 GMT
Not After : Dec 3 23:59:59 2017 GMT
Subject: serialNumber=04045342/1.3.6.1.4.1.311.60.2.1.3=GB/businessCategory=Private Organization, C=GB/postalCode=BN27 2BY,
ST=East Sussex, L=Hailsham/street=Station Road/street=Unit 10 Swan Business Centre, O=Fuss 3 Solutions Ltd,
OU=COMODO EV SSL, CN=www.fuss3inkandtoner.co.uk
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
Public-Key: (2048 bit)
Modulus:
...................
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Authority Key Identifier:
keyid:39:DA:FF:CA:28:14:8A:A8:74:13:08:B9:E4:0E:A9:D2:FA:7E:9D:69
X509v3 Subject Key Identifier:
D1:C0:72:40:F1:A4:47:A6:FF:32:C4:56:6F:EF:F5:1E:40:6A:72:DC
X509v3 Key Usage: critical
Digital Signature, Key Encipherment
X509v3 Basic Constraints: critical
CA:FALSE
X509v3 Extended Key Usage:
TLS Web Server Authentication, TLS Web Client Authentication
X509v3 Certificate Policies:
Policy: 1.3.6.1.4.1.6449.1.2.1.5.1
CPS: https://secure.comodo.com/CPS
X509v3 CRL Distribution Points:
Full Name:
URI:http://crl.comodoca.com/COMODORSAExtendedValidationSecureServerCA.crl
Authority Information Access:
CA Issuers - URI:http://crt.comodoca.com/COMODORSAExtendedValidationSecureServerCA.crt
OCSP - URI:http://ocsp.comodoca.com
X509v3 Subject Alternative Name:
DNS:www.fuss3inkandtoner.co.uk, DNS:fuss3inkandtoner.co.uk
1.3.6.1.4.1.11129.2.4.2:
............
Signature Algorithm: sha256WithRSAEncryption
...............
Certificate is real, it is not expired and domain matches. I have tried other online diagnostic tools like https://www.ssllabs.com/ssltest/analyze.html?d=fuss3inkandtoner.co.uk and everyone says that certificate is good. Everyone but hackersguardian.com which I need to pass for PCI Compliance.
I am not a sysadmin and this certificate was installed by someone else (I think hosting support's sysadmin). I need your advise on how to solve this problem. Thank you in advance.
It was a false positive. Its a very strange thing when security scanner from COMODO (hackerguardian.com) reports a bad certificate issued by COMODO (!).
This tool will clarify the issue you have: https://decoder.link/sslchecker/?hostname=www.hackerguardian.com&port=443
CA bundle installed along with the certificate is malformed (incorrect order). The certificate itself is good and valid, however its validity cannot be verified against the CA bundle, thus it is expected.
here is the correct one bundle: http://helpdesk.ssls.com/hc/en-us/article_attachments/201576002/COMODO_OV_SHA-256_bundle.crt
You can pass that to your hosting so they can reinstall that for you. After that,all will be ok. trust me :)

OpenSSL verify fails, can't find root certificate

Important Note: The method of validating certificates in my question below is incorrect, and will result in both false positives and false negatives. See my answer for the correct method.
I'm in the process of testing a tool I wrote to test all of the certificates in our environment, and I've run into an issue where OpenSSL doesn't seem to recognize a particular GoDaddy root certificate.
The error text:
$ openssl verify -CAfile bundle.txt cert.txt
cert.txt: C = US, ST = Arizona, L = Scottsdale, O = "GoDaddy.com, Inc.", CN = Go Daddy Root Certificate Authority - G2
error 2 at 2 depth lookup:unable to get issuer certificate
But that certificate is definitely in both /etc/pki/tls/certs/ca-bundle.crt and ca-bundle.trusted.crt. I've manually verified that the Issuer and Subject keys in the x509v3 extensions match up back to KeyID D2:C4:B0:D2:91:D4:4C:11:71:B3:61:CB:3D:A1:FE:DD:A8:6A:D4:E3 in the trusted bundles.
Am I missing something?
Certificate details:
Certificate:
Issuer: C=US, ST=Arizona, L=Scottsdale, O=GoDaddy.com, Inc., OU=http://certs.godaddy.com/repository/, CN=Go Daddy Secure Certificate Authority - G2
X509v3 Subject Key Identifier:
28:3C:0E:1A:82:3E:7F:22:A6:DD:22:8C:45:78:BF:F6:40:47:4F:8A
X509v3 Authority Key Identifier:
keyid:40:C2:BD:27:8E:CC:34:83:30:A2:33:D7:FB:6C:B3:F0:B4:2C:80:CE
Bundle1:
Subject: C=US, ST=Arizona, L=Scottsdale, O=GoDaddy.com, Inc., OU=http://certs.godaddy.com/repository/, CN=Go Daddy Secure Certificate Authority - G2
Issuer: C=US, ST=Arizona, L=Scottsdale, O=GoDaddy.com, Inc., CN=Go Daddy Root Certificate Authority - G2
X509v3 Subject Key Identifier:
40:C2:BD:27:8E:CC:34:83:30:A2:33:D7:FB:6C:B3:F0:B4:2C:80:CE
X509v3 Authority Key Identifier:
keyid:3A:9A:85:07:10:67:28:B6:EF:F6:BD:05:41:6E:20:C1:94:DA:0F:DE
Bundle2:
Subject: C=US, ST=Arizona, L=Scottsdale, O=GoDaddy.com, Inc., CN=Go Daddy Root Certificate Authority - G2
Issuer: C=US, O=The Go Daddy Group, Inc., OU=Go Daddy Class 2 Certification Authority
X509v3 Subject Key Identifier:
3A:9A:85:07:10:67:28:B6:EF:F6:BD:05:41:6E:20:C1:94:DA:0F:DE
X509v3 Authority Key Identifier:
keyid:D2:C4:B0:D2:91:D4:4C:11:71:B3:61:CB:3D:A1:FE:DD:A8:6A:D4:E3
Trusted:
Subject: C=US, O=The Go Daddy Group, Inc., OU=Go Daddy Class 2 Certification Authority
X509v3 Subject Key Identifier:
D2:C4:B0:D2:91:D4:4C:11:71:B3:61:CB:3D:A1:FE:DD:A8:6A:D4:E3
X509v3 Authority Key Identifier:
keyid:D2:C4:B0:D2:91:D4:4C:11:71:B3:61:CB:3D:A1:FE:DD:A8:6A:D4:E3
DirName:/C=US/O=The Go Daddy Group, Inc./OU=Go Daddy Class 2 Certification Authority
serial:00
Edit
To add to the WTFery in play, opening a connection to the server in question using openssl s_client shows the certificate verifying just fine.
$ openssl s_client -servername www.foo.com -connect www.foo.com:443
Certificate chain
0 s:/OU=Domain Control Validated/CN=*.foo.com
i:/C=US/ST=Arizona/L=Scottsdale/O=GoDaddy.com, Inc./OU=http://certs.godaddy.com/repository//CN=Go Daddy Secure Certificate Authority - G2
1 s:/C=US/ST=Arizona/L=Scottsdale/O=GoDaddy.com, Inc./OU=http://certs.godaddy.com/repository//CN=Go Daddy Secure Certificate Authority - G2
i:/C=US/ST=Arizona/L=Scottsdale/O=GoDaddy.com, Inc./CN=Go Daddy Root Certificate Authority - G2
2 s:/C=US/ST=Arizona/L=Scottsdale/O=GoDaddy.com, Inc./CN=Go Daddy Root Certificate Authority - G2
i:/C=US/O=The Go Daddy Group, Inc./OU=Go Daddy Class 2 Certification Authority
...yadda yadda yadda...
Verify return code: 0 (ok)
The naming of the openssl verify flags can be a bit counter-intuitive, and none of the documentation I found does much to address that. As x539 touched on I was using the -CAfile option incorrectly, and additionally I was missing the -untrusted option to specify the intermediate certificates.
I've found that the reason that the validation passed for most certificates is that most CAs have begun including the root certificate in the CA bundle, and the -CAfile option essentially defines what should comprise the trusted portion of the chain, including the root certificate. It was in the case that the intermediate bundle did not contain the root that my initial, incorrect method of validation failed.
Now I was under the misconception that -untrusted implied something like "never use these certificates!", but is rather intended to specify a chain of non-trusted certificates that leads back to a root in the "trusted" -CAfile bundle.
So the correct way to validate a certificate by using its intermediate certificate[s] and a trusted root bundle is:
openssl verify -CAfile /etc/pki/tls/certs/ca-bundle.crt -untrusted bundle.crt certificate.crt
If you specify -CAfile openssl only checks the given file for issuers, which in your case probably only contains the intermediate Certificates.