I have a Debian 10 buster server, one of several identical hardware/software configs. I use a script that deploys them, they are all identical. However, one of them is throwing an error when using wget:
# wget https://mirrors.edge.kernel.org/pub/linux/kernel/v5.x/linux-5.6.15.tar.xz
--2020-05-30 12:49:20-- https://mirrors.edge.kernel.org/pub/linux/kernel/v5.x/linux-5.6.15.tar.xz
Resolving mirrors.edge.kernel.org (mirrors.edge.kernel.org)... 147.75.69.165, 2604:1380:1000:8100::1
Connecting to mirrors.edge.kernel.org (mirrors.edge.kernel.org)|147.75.69.165|:443... connected.
ERROR: The certificate of ‘mirrors.edge.kernel.org’ is not trusted.
ERROR: The certificate of ‘mirrors.edge.kernel.org’ has expired.
Things I've done:
Checked that the SSL certificate is valid
Confirmed the exact same command works on other servers
Checked the system date, confirmed correct
Checked resolv.conf, configuring for 8.8.8.8 8.8.4.4 identical to others
Checked name resolution manually, working fine
Reinstall apt-get install --reinstall ca-certificates
Ran update-ca-certificates --fresh
Ran c_rehash
Used curl, it works fine without errors, shows correct SSL expiration
The problem seems to only be wget from what I can tell. I don't need a workaround. I need a solution so I have confidence this server is operating as it should.
I have the same symptom for f-droid.org:
% wget 'https://f-droid.org/F-Droid.apk'
--2020-06-03 14:10:18-- https://f-droid.org/F-Droid.apk
Resolving f-droid.org (f-droid.org)... 217.160.165.113, 148.251.140.42
Connecting to f-droid.org (f-droid.org)|217.160.165.113|:443... connected.
ERROR: The certificate of ‘f-droid.org’ is not trusted.
ERROR: The certificate of ‘f-droid.org’ has expired.
In my case (Debian 9 stretch), curl doesn't work either:
% curl 'https://f-droid.org/F-Droid.apk'
curl: (60) SSL certificate problem: certificate has expired
More details here: https://curl.haxx.se/docs/sslcerts.html
Looking at wget source one can see the error comes from status flag returned by gnutls_certificate_verify_peers2. We can ask gnutls to show certificate verification details:
% gnutls-cli f-droid.org -p 443
Processed 151 CA certificate(s).
Resolving 'f-droid.org:443'...
Connecting to '217.160.165.113:443'...
- Certificate type: X.509
- Got a certificate list of 3 certificates.
- Certificate[0] info:
- subject `CN=f-droid.org,OU=PositiveSSL,OU=Domain Control Validated', issuer `CN=Sectigo RSA Domain Validation Secure Server CA,O=Sectigo Limited,L=Salford,ST=Greater Manchester,C=GB', serial 0x02a8508e042b9f065fafadd87cd7d103, RSA key 2048 bits, signed using RSA-SHA256, activated `2019-09-17 00:00:00 UTC', expires `2020-09-16 23:59:59 UTC', key-ID `sha256:e97ccbf3c188b5cf69a83ed0fc39b001ce1688a62b573193cef3f74984c7d703'
Public Key ID:
sha1:638f93856e1f5edfcbd40c46d4160cff21b0713a
sha256:e97ccbf3c188b5cf69a83ed0fc39b001ce1688a62b573193cef3f74984c7d703
Public key's random art:
+--[ RSA 2048]----+
| o o+o.|
| *...o|
| E ..+ |
| . ... o|
| S . o .|
| o * . o.|
| * o . .o|
| . + o .o. |
| o .oo|
+-----------------+
- Certificate[1] info:
- subject `CN=Sectigo RSA Domain Validation Secure Server CA,O=Sectigo Limited,L=Salford,ST=Greater Manchester,C=GB', issuer `CN=USERTrust RSA Certification Authority,O=The USERTRUST Network,L=Jersey City,ST=New Jersey,C=US', serial 0x7d5b5126b476ba11db74160bbc530da7, RSA key 2048 bits, signed using RSA-SHA384, activated `2018-11-02 00:00:00 UTC', expires `2030-12-31 23:59:59 UTC', key-ID `sha256:e1ae9c3de848ece1ba72e0d991ae4d0d9ec547c6bad1dddab9d6beb0a7e0e0d8'
- Certificate[2] info:
- subject `CN=USERTrust RSA Certification Authority,O=The USERTRUST Network,L=Jersey City,ST=New Jersey,C=US', issuer `CN=AddTrust External CA Root,OU=AddTrust External TTP Network,O=AddTrust AB,C=SE', serial 0x13ea28705bf4eced0c36630980614336, RSA key 4096 bits, signed using RSA-SHA384, activated `2000-05-30 10:48:38 UTC', expires `2020-05-30 10:48:38 UTC', key-ID `sha256:c784333d20bcd742b9fdc3236f4e509b8937070e73067e254dd3bf9c45bf4dde'
- Status: The certificate is NOT trusted. The certificate chain uses expired certificate.
*** PKI verification of server certificate failed...
So the problem is the USERTrust RSA Certification Authority certificate supplied by the server f-droid.org. This is an intermediate cert signed by AddTrust External CA Root that expired 4 days ago.
The Debian ca-certificates package has a self-signed (root) certificate for USERTrust RSA Certification Authority:
% openssl x509 -inform PEM -in /usr/share/ca-certificates/mozilla/USERTrust_RSA_Certification_Authority.crt -text | grep -A2 Validity
Validity
Not Before: Feb 1 00:00:00 2010 GMT
Not After : Jan 18 23:59:59 2038 GMT
...but gnutls is confused by the expired cert supplied by the server, hence the error.
In your case (mirrors.edge.kernel.org) the certificate chain is as follows:
% gnutls-cli mirrors.edge.kernel.org -p 443
Processed 151 CA certificate(s).
Resolving 'mirrors.edge.kernel.org:443'...
Connecting to '147.75.101.1:443'...
- Certificate type: X.509
- Got a certificate list of 3 certificates.
- Certificate[0] info:
- subject `CN=*.edge.kernel.org', issuer `CN=Sectigo RSA Domain Validation Secure Server CA,O=Sectigo Limited,L=Salford,ST=Greater Manchester,C=GB', serial 0x00a34d0ccfbf4ea450fe030fd3378f5d68, RSA key 2048 bits, signed using RSA-SHA256, activated `2020-03-16 00:00:00 UTC', expires `2021-03-16 23:59:59 UTC', key-ID `sha256:f3746cf281b6453def23289054e75e26e157bfd9eed5252b76ffdc828b802e41'
Public Key ID:
sha1:72d8fc84d0dea848189bc3ff8eb1d47629fd72c0
sha256:f3746cf281b6453def23289054e75e26e157bfd9eed5252b76ffdc828b802e41
Public key's random art:
+--[ RSA 2048]----+
| |
| . |
| . . . |
| . = * + |
| * . o.S o |
| + o =E+ |
| = = +.. |
| . * o... |
| o.o o. |
+-----------------+
- Certificate[1] info:
- subject `CN=Sectigo RSA Domain Validation Secure Server CA,O=Sectigo Limited,L=Salford,ST=Greater Manchester,C=GB', issuer `CN=USERTrust RSA Certification Authority,O=The USERTRUST Network,L=Jersey City,ST=New Jersey,C=US', serial 0x7d5b5126b476ba11db74160bbc530da7, RSA key 2048 bits, signed using RSA-SHA384, activated `2018-11-02 00:00:00 UTC', expires `2030-12-31 23:59:59 UTC', key-ID `sha256:e1ae9c3de848ece1ba72e0d991ae4d0d9ec547c6bad1dddab9d6beb0a7e0e0d8'
- Certificate[2] info:
- subject `CN=USERTrust RSA Certification Authority,O=The USERTRUST Network,L=Jersey City,ST=New Jersey,C=US', issuer `CN=USERTrust RSA Certification Authority,O=The USERTRUST Network,L=Jersey City,ST=New Jersey,C=US', serial 0x01fd6d30fca3ca51a81bbc640e35032d, RSA key 4096 bits, signed using RSA-SHA384, activated `2010-02-01 00:00:00 UTC', expires `2038-01-18 23:59:59 UTC', key-ID `sha256:c784333d20bcd742b9fdc3236f4e509b8937070e73067e254dd3bf9c45bf4dde'
- Status: The certificate is trusted.
Again we see USERTrust RSA Certification Authority, but here it's a self-signed (root) certificate, not expired. I guess kernel.org did supply expired intermediate certificate too, but corrected under your hands. I think --no-dns-cache is irrelevant, mere coincidence.
Incidentally, Firefox displays https://f-droid.org just fine. Show page info / Security reveals new self-signed (root) USERTrust cert: Not after: 18 January 2038, 23:59:59 GMT. That is, Firefox ignored the expired intermediate USERTrust cert supplied by the server, used the valid root USERTrust cert available locally.
Resolution? The server webmaster should remove the obsolete intermediate certificate from the server configuration. In the meantime, use Firefox (or curl, if it works) instead of wget.
Update: as f-droid people pointed out, the gnutls bug is fixed already, the fix should be in Debian-security soon.
My problem was with the Let's Encrypt cert, it said the same, on Debian 8.
My solution:
With the gnutls-cli command above I was able to look into the whole chain and it gave me a little more information, which eventually led me here: https://letsencrypt.org/docs/dst-root-ca-x3-expiration-september-2021/
With this information I issued
# dpkg-reconfigure ca-certificates
Unselected the already expired DST Root CA X3
And it started to work.
I was able to solve my own problem using --no-dns-cache
# wget --no-dns-cache --debug https://mirrors.edge.kernel.org/pub/linux/kernel/v5.x/linux-5.6.15.tar.xz
After this, it apparently updated whatever file was broken. It now works without the --no-dns-cache as well. So strange, but it seems stable.
Related
I am using the command ...
openssl s_client -showcerts -connect reds-cluster-01:443
And I get the output:
depth=2 C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert Global Root CA
verify return:1
depth=1 C = US, O = DigiCert Inc, CN = RapidSSL TLS DV RSA Mixed SHA256 2020 CA-1
verify return:1
depth=0 CN = *.my-co-example.com
verify return:1
CONNECTED(00000003)
---
Certificate chain
0 s:/CN=*.my-co-example.com
i:/C=US/O=DigiCert Inc/CN=RapidSSL TLS DV RSA Mixed SHA256 2020 CA-1
-----BEGIN CERTIFICATE-----
MIIGnxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxgkqhkiG9w0BAQsFADBZ
...
I assume that means somewhere on the filesystem of my server there would be a file somewhere that has the string ...
MIIGnxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxgkqhkiG9w0BAQsFADBZ
... in it. How can I find that file without having to execute something like?
sudo grep -sr MIIGnxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxgkqhkiG9w0BAQsFADBZ /
I assume that means somewhere on the filesystem of my server there would be a file somewhere that has the string ...
This assumption is wrong. What you see is part of the server certificate, which need to be checked against a CA certificate located in the local trust store. The server certificate is typically not in the local trust store. See SSL Certificate framework 101: How does the browser actually verify the validity of a given server certificate? for more on certificates are checked.
You can see more with "CURL -W CERTS" from Daniel Stenberg (also on Stack Overflow), which works on curl for the past 25+ years.
When a client connects to a TLS server it gets sent one or more certificates during the handshake.
Those certificates are verified by the client, to make sure that the server is indeed the right one: the server the client expects it to be; no impostor and no man in the middle etc.
When such a server certificate is signed by a Certificate Authority (CA), that CA’s certificate is normally not sent by the server but the client is expected to have it already in its CA store.
(So no file to check on the filesystem)
Ever since the day SSL and TLS first showed up in the 1990s user have occasionally wanted to be able to save the certificates provided by the server in a TLS handshake.
The openssl tool has offered this ability since along time and is actually one of my higher ranked stackoverflow answers.
Now (for curl 7.88.0,to be shipped in February 2023), Daniel proposes:
Using the –write-out (-w) option and the new variables %{certs} and %{num_certs}, curl can now do what you want.
Get the certificates from a server in PEM format:
$ curl https://curl.se -w "%{certs}" -o /dev/null > cacert.pem
$ curl --cacert cacert.pem https://curl.se/
That is easier to parse than the openssl s_client -showcerts -connect current alternative.
My builds started failing this afternoon, with this:
Notice: /Stage[main]/Server_init::Apt_get_update/Exec[apt-get-update]/returns:
Err:16 https://apt.puppetlabs.com bionic Release
Notice: /Stage[main]/Server_init::Apt_get_update/Exec[apt-get-update]/returns:
Certificate verification failed: The certificate is NOT trusted.
The certificate chain uses expired certificate. Could not handshake:
Error in the certificate verification. [IP: 13.33.171.122 443]
I assume the expired certificate expired this morning, since an earlier successful build.
How do I sort out which cert it is which has failed? And what I can do to get it updated. This says it is some cert in the chain, which I assume means that it is not necessarily the one for apt.puppetlabs.com.
Is there some way for me to tell apt to ignore the certs on this repo, for the time being, so I can resume work on my ticket, instead of this new bug?
There may be a more convenient way to do this, but here is how I found the broken link in the certificate chain:
openssl s_client </dev/null -showcerts -connect apt.puppetlabs.com:443 -servername apt.puppetlabs.com | awk 'BEGIN { pem_to_text = "openssl x509 -noout -text" }; /BEGIN CERTIFICATE/ { on = 1 }; on == 1 { print $0 | pem_to_text }; /END CERTIFICATE/ { on = 0; close(pem_to_text) }'
At the time of writing, one of the certificates in the chain (CN = USERTrust RSA Certification Authority) had expired as of May 30 10:48:38 2020 GMT.
Note, however, that on newer operating system versions (like Ubuntu 20.04) the problem does not cause an error, because the ca-certificates package contains an updated CA certificate for the same entity and public key, that is valid through Jan 18 23:59:59 2038 GMT, and newer versions of libssl (OpenSSL) take advantage of that fact (it seems) to verify the leaf certificate.
I have similar issue, with the same error from apt, however my certificates are valid for any other tool (curl, get, browsers). Checked the whole chain of certificates - and they are valid.
I'm generating SSL certificates with Let's Encrypt. It works fine, I have the CSR, the key and the certificate chain including the intermediate certificate for those browsers which don't know Let's Encrypt Certificate Authority yet. When creating a new certificate in Plesk (12), I copy-and-paste the given data, click on apply, apply the created certificate to the domain and it works as expected. Testing with openssl says
$ openssl s_client -crlf -connect mydomain.com:443 -servername mydomain.com
<snip>
Certificate chain
0 s:/CN=www.mydomain.com
i:/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
1 s:/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
i:/O=Digital Signature Trust Co./CN=DST Root CA X3
2 s:/O=Digital Signature Trust Co./CN=DST Root CA X3
i:/O=Digital Signature Trust Co./CN=DST Root CA X3
</snip>
Now, three month later, I have to re-issue those certificates. As I want to automate it, I want to use a CLI solution, like Plesk offers:
$ plesk bin certificate \
-u mycertificatename \
-domain mydomain.com \
-cert-file mynewcert.crt \
-key-file mykey.key
The problem: The certificate gets updated and is shown as expected in Plesk, but the intermediate certificate doesn't show up when testing with openssl:
Certificate chain
0 s:/CN=www.mydomain.com
i:/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
Any ideas? Did I forget to restart any service? Or any (un)documented feature?
Try to use option
-cacert-file <path> CA certificate file
with path to CA file which containts all intermediate certificates
Following steps of: https://devcenter.heroku.com/articles/ssl-endpoint
I have added SSL Endpoint add-on into my server hosted on Heroku.
Then, I've created a self signed certificate following:
https://devcenter.heroku.com/articles/ssl-certificate-self
At this point I have upload my self-signed cert and private key using:
heroku certs:add server.crt server.key
In fact it gaves me:
! Unable to parse certificate. Please ensure the certificate is in PEM format.
But I found a simple solution on Google:
openssl rsa -in server.key -out server.key.rsa
Then I 've uploaded all:
heroku certs:add server.crt server.key.rsa
Resolving trust chain... done
Adding SSL Endpoint to <myapp>... done
<myapp> now served by wakayama-xxxx.herokussl.com
Certificate details:
Common Name(s): <mydomain>
Expires At: 2016-09-29 11:24 UTC
Issuer: /C=AU/ST=Some-State/O=Internet Widgits Pty Ltd/CN=<mydomain>
Starts At: 2015-09-30 11:24 UTC
Subject: /C=AU/ST=Some-State/O=Internet Widgits Pty Ltd/CN=<mydomain>
SSL certificate is self signed.
So if I ask for my certificates into Heroku I get:
heroku certs:info
Fetching SSL Endpoint wakayama-xxxx.herokussl.com info for <myapp>... done
Certificate details:
Common Name(s): <mydomain>
Expires At: 2016-09-29 11:24 UTC
Issuer: /C=AU/ST=Some-State/O=Internet Widgits Pty Ltd/CN=<mydomain>
Starts At: 2015-09-30 11:24 UTC
Subject: /C=AU/ST=Some-State/O=Internet Widgits Pty Ltd/CN=<mydomain>
SSL certificate is self signed.
So seems that all is ok. Except... if I go to wakayama-xxxx.herokussl.com I see the message: "Heroku | No such app", hum, I've guess It should me redirect to my app, but not.
Any ideas?
Seems like: Heroku SSL Endpoint - "No Such App"
So I've closed my eyes and I've changed my DNS to add a CNAME pointing to wakayama-xxxx.herokussl.com (which doesn't redirect to app) and magically now HTTPS works. If someone can explain what's happening here I would appreciate.
Short version: I'm seeing an SSL certificate chain that's different based on how I access the https server. What's going on, and how do I verify the certificate under these circumstances?
Slightly longer version:
I'm trying to use libcurl to verify the certificate of an SSL connection. The server I'm connecting to is Amazon S3.
When I access Amazon S3 in Firefox, I get this certificate chain:
VeriSign Class 3 Public Primary Certification Authority - G5
Serial Number: 18:DA:D1:9E:26:7D:E8:BB:4A:21:58:CD:CC:6B:3B:4A
VeriSign Class 3 Secure Server CA - G3
Serial Number: 6E:CC:7A:A5:A7:03:20:09:B8:CE:BC:F4:E9:52:D4:91
*.s3.amazonaws.com
Serial Number: 43:FB:BA:C2:66:27:E0:97:1E:1C:11:E0:30:C3:6B:66
When I access the same URL via the OpenSSL command line tool, I get this certificate chain:
*.s3.amazonaws.com
Serial number: 43fbbac26627e0971e1c11e030c36b66
VeriSign Class 3 Secure Server CA - G3
Serial number: 6ecc7aa5a7032009b8cebcf4e952d491
VeriSign Class 3 Public Primary Certification Authority - G5
Serial number: 35973187f3873a07327ece580c9b7eda
The "*.s3.amazonaws.com" and "VeriSign Class 3 Secure Server CA - G3" certificates appear to be the same, but the one after that is different! It's named "VeriSign Class 3 Public Primary Certification Authority - G5" in both chains, but the certificate serial number is different. (Other information is different as well; let me know if you want a longer dump.)
I believe that this difference is the reason that I can't get libcurl to verify the SSL certificate. The certificate with serial number "18:DA:D1..." is in my CACERT.PEM file, but the certificate with serial number "35:97:31..." is not.
Obviously, the simple fix would be to add certificate "35:97:31..." to my CACERT.PEM file, but I want to make the right change here, not just a quick fix.
What does this difference in certificate chains signify?
Is it even possible for a SSL server to return different certificate chains based on the client (Firefox vs. OpenSSL/libcurl)?
How should I get libcurl to verify this SSL certificate?
Really long version and background information:
I'm using libcurl with OpenSSL to download from Amazon S3. Libcurl is returning "SSL certificate problem: unable to get local issuer certificate", which I know means that the root certificate isn't listed in my CACERT.PEM file. (I'm using the one downloaded from curl's website, which is converted from Mozilla's certificate store.) I am able to verify certificates on other SSL connections, so I know my libcurl setup is correct.
To see what was going on, and why the certificate didn't verify, I pulled up the same URL in Firefox. Firefox didn't show any SSL certificate warnings. Here's the certification path Firefox shows. The root certificate, "VeriSign Class 3 Public Primary Certification Authority - G5", is listed in my CACERT.PEM file, and the serial number of the certificate matches what's shown in the screenshot.
Here's the serial numbers of all three certificates in the chain:
VeriSign Class 3 Public Primary Certification Authority - G5
Serial Number: 18:DA:D1:9E:26:7D:E8:BB:4A:21:58:CD:CC:6B:3B:4A
VeriSign Class 3 Secure Server CA - G3
Serial Number: 6E:CC:7A:A5:A7:03:20:09:B8:CE:BC:F4:E9:52:D4:91
*.s3.amazonaws.com
Serial Number: 43:FB:BA:C2:66:27:E0:97:1E:1C:11:E0:30:C3:6B:66
On a different platform (different OS, different version of OpenSSL, etc.), I tried accessing the same URL using the OpenSSL command line tool, and I got a different certification path!
$ openssl s_client -showcerts -connect stackoverflowtest.s3.amazonaws.com:443
CONNECTED(00000003)
depth=3 C = US, O = "VeriSign, Inc.", OU = Class 3 Public Primary Certification Authority
verify return:1
depth=2 C = US, O = "VeriSign, Inc.", OU = VeriSign Trust Network, OU = "(c) 2006 VeriSign, Inc. - For authorized use only", CN = VeriSign Class 3 Public Primary Certification Authority - G5
verify return:1
depth=1 C = US, O = "VeriSign, Inc.", OU = VeriSign Trust Network, OU = Terms of use at https://www.verisign.com/rpa (c)10, CN = VeriSign Class 3 Secure Server CA - G3
verify return:1
depth=0 C = US, ST = Washington, L = Seattle, O = Amazon.com Inc., OU = S3-A, CN = *.s3.amazonaws.com
verify return:1
---
Certificate chain
0 s:/C=US/ST=Washington/L=Seattle/O=Amazon.com Inc./OU=S3-A/CN=*.s3.amazonaws.com
i:/C=US/O=VeriSign, Inc./OU=VeriSign Trust Network/OU=Terms of use at https://www.verisign.com/rpa (c)10/CN=VeriSign Class 3 Secure Server CA - G3
-----BEGIN CERTIFICATE-----
MIIFOTCCBCGgAwIBAgIQQ/u6wmYn4JceHBHgMMNrZjANBgkqhkiG9w0BAQUFADCB
tTELMAkGA1UEBhMCVVMxFzAVBgNVBAoTDlZlcmlTaWduLCBJbmMuMR8wHQYDVQQL
ExZWZXJpU2lnbiBUcnVzdCBOZXR3b3JrMTswOQYDVQQLEzJUZXJtcyBvZiB1c2Ug
YXQgaHR0cHM6Ly93d3cudmVyaXNpZ24uY29tL3JwYSAoYykxMDEvMC0GA1UEAxMm
VmVyaVNpZ24gQ2xhc3MgMyBTZWN1cmUgU2VydmVyIENBIC0gRzMwHhcNMTQwNjI1
MDAwMDAwWhcNMTUwNjA1MjM1OTU5WjB6MQswCQYDVQQGEwJVUzETMBEGA1UECBMK
V2FzaGluZ3RvbjEQMA4GA1UEBxQHU2VhdHRsZTEYMBYGA1UEChQPQW1hem9uLmNv
bSBJbmMuMQ0wCwYDVQQLFARTMy1BMRswGQYDVQQDFBIqLnMzLmFtYXpvbmF3cy5j
b20wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC+mjGH4PrN0TDNmTsF
my2fHpHpTuySRwBMe/nuGIkL3cvKIxFKLHvkK9kx1UpO0skdQTdCY55LywhubLNO
fD19IzJdoRGdlqgkDAYC8vz3LRYj8WWsGnROfS/YFtgj25YaHPnsNp6lWrff4/qi
ctbojJpMxm+9Q0A4nTzrZymHEUkRbx6AVVUBVKH3uZi/w0aV+i4cp2bs+CYIK3DL
Qp584DJ9bOImgUhDfz19+Wtv64zIezE0Uz9eOkqgQ1X//XumyZWyD6N6+h/XqTnc
YTvIer/s83T/IngGMbfPRqjpQCay6ySXCNbJ5izMgo+gwN84t7JhaI+EYcxf1dDN
w0mXAgMBAAGjggF9MIIBeTAJBgNVHRMEAjAAMA4GA1UdDwEB/wQEAwIFoDAdBgNV
HSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIwZQYDVR0gBF4wXDBaBgpghkgBhvhF
AQc2MEwwIwYIKwYBBQUHAgEWF2h0dHBzOi8vZC5zeW1jYi5jb20vY3BzMCUGCCsG
AQUFBwICMBkaF2h0dHBzOi8vZC5zeW1jYi5jb20vcnBhMB8GA1UdIwQYMBaAFA1E
XBZTRMGCfh0gqyX0AWPYvnmlMCsGA1UdHwQkMCIwIKAeoByGGmh0dHA6Ly9zZC5z
eW1jYi5jb20vc2QuY3JsMFcGCCsGAQUFBwEBBEswSTAfBggrBgEFBQcwAYYTaHR0
cDovL3NkLnN5bWNkLmNvbTAmBggrBgEFBQcwAoYaaHR0cDovL3NkLnN5bWNiLmNv
bS9zZC5jcnQwLwYDVR0RBCgwJoISKi5zMy5hbWF6b25hd3MuY29tghBzMy5hbWF6
b25hd3MuY29tMA0GCSqGSIb3DQEBBQUAA4IBAQBlXrn1FTPjVIFbcuQNbBesrAMI
NV4L7jS1mobEwFrb7UrqZ7kHvuvoR/BpDygATyqLvPihs7nUc2TUHsw/41EAHKoq
QBVfRTOH0yWaTC6SYSx7fiElL+k55Pvrz4+7gLRy5zUlVX3iUw93zr95ka/LPuCL
7PQOFPeQDOgveDjcSNVtLcTfQfvog/rMSu/4XPFHu7zaZwUEurt9CzLeVbdB6O25
bHuHTaZLP0wmjCIbwgXu8bFWqOTnAjG70EtYrbIiQhl/ISJU6HioFzLiy5Ibp07r
RbV4ir5EI2EPKxIy30YDvpCQ0WQvYLWFV0qQOuOXkMC2M2IsBmVn2/9GQ7eP
-----END CERTIFICATE-----
1 s:/C=US/O=VeriSign, Inc./OU=VeriSign Trust Network/OU=Terms of use at https://www.verisign.com/rpa (c)10/CN=VeriSign Class 3 Secure Server CA - G3
i:/C=US/O=VeriSign, Inc./OU=VeriSign Trust Network/OU=(c) 2006 VeriSign, Inc. - For authorized use only/CN=VeriSign Class 3 Public Primary Certification Authority - G5
-----BEGIN CERTIFICATE-----
MIIF7DCCBNSgAwIBAgIQbsx6pacDIAm4zrz06VLUkTANBgkqhkiG9w0BAQUFADCB
yjELMAkGA1UEBhMCVVMxFzAVBgNVBAoTDlZlcmlTaWduLCBJbmMuMR8wHQYDVQQL
ExZWZXJpU2lnbiBUcnVzdCBOZXR3b3JrMTowOAYDVQQLEzEoYykgMjAwNiBWZXJp
U2lnbiwgSW5jLiAtIEZvciBhdXRob3JpemVkIHVzZSBvbmx5MUUwQwYDVQQDEzxW
ZXJpU2lnbiBDbGFzcyAzIFB1YmxpYyBQcmltYXJ5IENlcnRpZmljYXRpb24gQXV0
aG9yaXR5IC0gRzUwHhcNMTAwMjA4MDAwMDAwWhcNMjAwMjA3MjM1OTU5WjCBtTEL
MAkGA1UEBhMCVVMxFzAVBgNVBAoTDlZlcmlTaWduLCBJbmMuMR8wHQYDVQQLExZW
ZXJpU2lnbiBUcnVzdCBOZXR3b3JrMTswOQYDVQQLEzJUZXJtcyBvZiB1c2UgYXQg
aHR0cHM6Ly93d3cudmVyaXNpZ24uY29tL3JwYSAoYykxMDEvMC0GA1UEAxMmVmVy
aVNpZ24gQ2xhc3MgMyBTZWN1cmUgU2VydmVyIENBIC0gRzMwggEiMA0GCSqGSIb3
DQEBAQUAA4IBDwAwggEKAoIBAQCxh4QfwgxF9byrJZenraI+nLr2wTm4i8rCrFbG
5btljkRPTc5v7QlK1K9OEJxoiy6Ve4mbE8riNDTB81vzSXtig0iBdNGIeGwCU/m8
f0MmV1gzgzszChew0E6RJK2GfWQS3HRKNKEdCuqWHQsV/KNLO85jiND4LQyUhhDK
tpo9yus3nABINYYpUHjoRWPNGUFP9ZXse5jUxHGzUL4os4+guVOc9cosI6n9FAbo
GLSa6Dxugf3kzTU2s1HTaewSulZub5tXxYsU5w7HnO1KVGrJTcW/EbGuHGeBy0RV
M5l/JJs/U0V/hhrzPPptf4H1uErT9YU3HLWm0AnkGHs4TvoPAgMBAAGjggHfMIIB
2zA0BggrBgEFBQcBAQQoMCYwJAYIKwYBBQUHMAGGGGh0dHA6Ly9vY3NwLnZlcmlz
aWduLmNvbTASBgNVHRMBAf8ECDAGAQH/AgEAMHAGA1UdIARpMGcwZQYLYIZIAYb4
RQEHFwMwVjAoBggrBgEFBQcCARYcaHR0cHM6Ly93d3cudmVyaXNpZ24uY29tL2Nw
czAqBggrBgEFBQcCAjAeGhxodHRwczovL3d3dy52ZXJpc2lnbi5jb20vcnBhMDQG
A1UdHwQtMCswKaAnoCWGI2h0dHA6Ly9jcmwudmVyaXNpZ24uY29tL3BjYTMtZzUu
Y3JsMA4GA1UdDwEB/wQEAwIBBjBtBggrBgEFBQcBDARhMF+hXaBbMFkwVzBVFglp
bWFnZS9naWYwITAfMAcGBSsOAwIaBBSP5dMahqyNjmvDz4Bq1EgYLHsZLjAlFiNo
dHRwOi8vbG9nby52ZXJpc2lnbi5jb20vdnNsb2dvLmdpZjAoBgNVHREEITAfpB0w
GzEZMBcGA1UEAxMQVmVyaVNpZ25NUEtJLTItNjAdBgNVHQ4EFgQUDURcFlNEwYJ+
HSCrJfQBY9i+eaUwHwYDVR0jBBgwFoAUf9Nlp8Ld7LvwMAnzQzn6Aq8zMTMwDQYJ
KoZIhvcNAQEFBQADggEBAAyDJO/dwwzZWJz+NrbrioBL0aP3nfPMU++CnqOh5pfB
WJ11bOAdG0z60cEtBcDqbrIicFXZIDNAMwfCZYP6j0M3m+oOmmxw7vacgDvZN/R6
bezQGH1JSsqZxxkoor7YdyT3hSaGbYcFQEFn0Sc67dxIHSLNCwuLvPSxe/20majp
dirhGi2HbnTTiN0eIsbfFrYrghQKlFzyUOyvzv9iNw2tZdMGQVPtAhTItVgooazg
W+yzf5VK+wPIrSbb5mZ4EkrZn0L74ZjmQoObj49nJOhhGbXdzbULJgWOw27EyHW4
Rs/iGAZeqa6ogZpHFt4MKGwlJ7net4RYxh84HqTEy2Y=
-----END CERTIFICATE-----
2 s:/C=US/O=VeriSign, Inc./OU=VeriSign Trust Network/OU=(c) 2006 VeriSign, Inc. - For authorized use only/CN=VeriSign Class 3 Public Primary Certification Authority - G5
i:/C=US/O=VeriSign, Inc./OU=Class 3 Public Primary Certification Authority
-----BEGIN CERTIFICATE-----
MIIExjCCBC+gAwIBAgIQNZcxh/OHOgcyfs5YDJt+2jANBgkqhkiG9w0BAQUFADBf
MQswCQYDVQQGEwJVUzEXMBUGA1UEChMOVmVyaVNpZ24sIEluYy4xNzA1BgNVBAsT
LkNsYXNzIDMgUHVibGljIFByaW1hcnkgQ2VydGlmaWNhdGlvbiBBdXRob3JpdHkw
HhcNMDYxMTA4MDAwMDAwWhcNMjExMTA3MjM1OTU5WjCByjELMAkGA1UEBhMCVVMx
FzAVBgNVBAoTDlZlcmlTaWduLCBJbmMuMR8wHQYDVQQLExZWZXJpU2lnbiBUcnVz
dCBOZXR3b3JrMTowOAYDVQQLEzEoYykgMjAwNiBWZXJpU2lnbiwgSW5jLiAtIEZv
ciBhdXRob3JpemVkIHVzZSBvbmx5MUUwQwYDVQQDEzxWZXJpU2lnbiBDbGFzcyAz
IFB1YmxpYyBQcmltYXJ5IENlcnRpZmljYXRpb24gQXV0aG9yaXR5IC0gRzUwggEi
MA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCvJAgIKXo1nmAMqudLO07cfLw8
RRy7K+D+KQL5VwijZIUVJ/XxrcgxiV0i6CqqpkKzj/i5Vbext0uz/o9+B1fs70Pb
ZmIVYc9gDaTY3vjgw2IIPVQT60nKWVSFJuUrjxuf6/WhkcIzSdhDY2pSS9KP6HBR
TdGJaXvHcPaz3BJ023tdS1bTlr8Vd6Gw9KIl8q8ckmcY5fQGBO+QueQA5N06tRn/
Arr0PO7gi+s3i+z016zy9vA9r911kTMZHRxAy3QkGSGT2RT+rCpSx4/VBEnkjWNH
iDxpg8v+R70rfk/Fla4OndTRQ8Bnc+MUCH7lP59zuDMKz10/NIeWiu5T6CUVAgMB
AAGjggGRMIIBjTAPBgNVHRMBAf8EBTADAQH/MDEGA1UdHwQqMCgwJqAkoCKGIGh0
dHA6Ly9jcmwudmVyaXNpZ24uY29tL3BjYTMuY3JsMA4GA1UdDwEB/wQEAwIBBjA9
BgNVHSAENjA0MDIGBFUdIAAwKjAoBggrBgEFBQcCARYcaHR0cHM6Ly93d3cudmVy
aXNpZ24uY29tL2NwczAdBgNVHQ4EFgQUf9Nlp8Ld7LvwMAnzQzn6Aq8zMTMwNAYD
VR0lBC0wKwYJYIZIAYb4QgQBBgpghkgBhvhFAQgBBggrBgEFBQcDAQYIKwYBBQUH
AwIwbQYIKwYBBQUHAQwEYTBfoV2gWzBZMFcwVRYJaW1hZ2UvZ2lmMCEwHzAHBgUr
DgMCGgQUj+XTGoasjY5rw8+AatRIGCx7GS4wJRYjaHR0cDovL2xvZ28udmVyaXNp
Z24uY29tL3ZzbG9nby5naWYwNAYIKwYBBQUHAQEEKDAmMCQGCCsGAQUFBzABhhho
dHRwOi8vb2NzcC52ZXJpc2lnbi5jb20wDQYJKoZIhvcNAQEFBQADgYEADyWuSO0b
M4VMDLXC1/5N1oMoTEFlYAALd0hxgv5/21oOIMzS6ke8ZEJhRDR0MIGBJopK90Rd
fjSAqLiD4gnXbSPdie0oCL1jWhFXCMSe2uJoKK/dUDzsgiHYAMJVRFBwQa2DF3m6
CPMr3u00HUSe0gST9MsFFy0JLS1j7/YmC3s=
-----END CERTIFICATE-----
---
Server certificate
subject=/C=US/ST=Washington/L=Seattle/O=Amazon.com Inc./OU=S3-A/CN=*.s3.amazonaws.com
issuer=/C=US/O=VeriSign, Inc./OU=VeriSign Trust Network/OU=Terms of use at https://www.verisign.com/rpa (c)10/CN=VeriSign Class 3 Secure Server CA - G3
---
No client certificate CA names sent
Server Temp Key: ECDH, prime256v1, 256 bits
---
SSL handshake has read 4624 bytes and written 399 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES128-SHA
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
Protocol : TLSv1.2
Cipher : ECDHE-RSA-AES128-SHA
Session-ID: B642ED1E6FE32F7A374B5A62847DEC63C2F37DCA7A18FD669B8F0FCDC98C49BF
Session-ID-ctx:
Master-Key: EE2D31F43D341A0895B36E0BCCE7557B221F1469AC1B7B0BA22D843C75F25F949822B0D0E22E967A1F373F034E9624E4
Key-Arg : None
Krb5 Principal: None
PSK identity: None
PSK identity hint: None
Start Time: 1430871883
Timeout : 300 (sec)
Verify return code: 0 (ok)
---
closed
When I decode the certificates it lists, I get a different certificate chain than Firefox gave me.
*.s3.amazonaws.com
Serial number: 43fbbac26627e0971e1c11e030c36b66
VeriSign Class 3 Secure Server CA - G3
Serial number: 6ecc7aa5a7032009b8cebcf4e952d491
VeriSign Class 3 Public Primary Certification Authority - G5
Serial number: 35973187f3873a07327ece580c9b7eda
The cert in use, and its immediate parent cert are the same, but the next one up has the same name, but a different serial number.
Here's my version information:
> curld --version
curl 7.40.0 (i386-pc-win32ce) libcurl/7.40.0 OpenSSL/1.0.1e
Protocols: dict ftp ftps gopher http https imap imaps pop3 pop3s rtsp smb smbs smtp smtps tftp
Features: NTLM SSL
> curld --version
curl 7.40.0 (i386-pc-win32) libcurl/7.40.0 OpenSSL/1.0.1c
Protocols: dict ftp ftps gopher http https imap imaps pop3 pop3s rtsp smb smbs smtp smtps tftp
Features: AsynchDNS Largefile NTLM SSL
$ cat /etc/redhat-release
CentOS release 6.6 (Final)
$ openssl version
OpenSSL 1.0.1e-fips 11 Feb 2013
$ yum list openssl
Installed Packages
openssl.x86_64 1.0.1e-30.el6.8 #updates
Firefox: 37.0.2, running on Windows 7 x64.
I did review this question: SSL Certificate - Certification Path in browser different from Certificate Chain File, but my problem appears to be different: In that case, the certificate chain in the OpenSSL command line tool went 1 - 2 - 3 - 4, and in IE it was 1 - 2 - 3. Because IE considered "3" to be a root certificate, the chain stopped early. In my case, Firefox is reporting 1 - 2 - 3, and OpenSSL is reporting 1 - 2 - 4; The chain is different.
The server sends the same chain certificates to firefox and s_client:
CN=.s3.amazonaws.com SAN=DNS:.s3.amazonaws.com,DNS:s3.amazonaws.com
CN=VeriSign Class 3 Secure Server CA - G3
CN=VeriSign Class 3 Public Primary Certification Authority - G5
But the way the certificates will be verified differs depending on the SSL stack and the trusted root certificates of the client. And in case of curl you run into an old OpenSSL validation problem. Details:
Firefox has a trusted root certificate similar to the certificate#3 send by the client. This means that it is a different certificate but it contains the same public key, so that the signature for certificate#2 is still valid. Since Firefox underlying TLS stack (NSS) thus has found a usable trust anchor it will consider the chain as verified and ignore the certificate#3 sent by the server.
But the version of curl you have uses OpenSSL as the TLS library. OpenSSL tries to get the longest match, that it it will try to find a trusted root certificate which signed certificate#3. If it fails it will not try with a shorter trust chain but simply fail. This is a long-standing bug which is the cause of many many strange problems like this and it looks like the issue is for now only fixed in the latest development branch (not released).
The solution is probably to use a CA store for curl which still includes the old deprecated 1024bit CA's so that it contains the trust anchor "/C=US/O=VeriSign, Inc./OU=Class 3 Public Primary Certification Authority"
For even more details about this problem see
more detailed writeup
OpenSSL bug #2732 from 2012
workarounds with patch
newer Bug reports which indicate a bug fix for current development branch (after 1.0.2)