My builds started failing this afternoon, with this:
Notice: /Stage[main]/Server_init::Apt_get_update/Exec[apt-get-update]/returns:
Err:16 https://apt.puppetlabs.com bionic Release
Notice: /Stage[main]/Server_init::Apt_get_update/Exec[apt-get-update]/returns:
Certificate verification failed: The certificate is NOT trusted.
The certificate chain uses expired certificate. Could not handshake:
Error in the certificate verification. [IP: 13.33.171.122 443]
I assume the expired certificate expired this morning, since an earlier successful build.
How do I sort out which cert it is which has failed? And what I can do to get it updated. This says it is some cert in the chain, which I assume means that it is not necessarily the one for apt.puppetlabs.com.
Is there some way for me to tell apt to ignore the certs on this repo, for the time being, so I can resume work on my ticket, instead of this new bug?
There may be a more convenient way to do this, but here is how I found the broken link in the certificate chain:
openssl s_client </dev/null -showcerts -connect apt.puppetlabs.com:443 -servername apt.puppetlabs.com | awk 'BEGIN { pem_to_text = "openssl x509 -noout -text" }; /BEGIN CERTIFICATE/ { on = 1 }; on == 1 { print $0 | pem_to_text }; /END CERTIFICATE/ { on = 0; close(pem_to_text) }'
At the time of writing, one of the certificates in the chain (CN = USERTrust RSA Certification Authority) had expired as of May 30 10:48:38 2020 GMT.
Note, however, that on newer operating system versions (like Ubuntu 20.04) the problem does not cause an error, because the ca-certificates package contains an updated CA certificate for the same entity and public key, that is valid through Jan 18 23:59:59 2038 GMT, and newer versions of libssl (OpenSSL) take advantage of that fact (it seems) to verify the leaf certificate.
I have similar issue, with the same error from apt, however my certificates are valid for any other tool (curl, get, browsers). Checked the whole chain of certificates - and they are valid.
Related
I am using the command ...
openssl s_client -showcerts -connect reds-cluster-01:443
And I get the output:
depth=2 C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert Global Root CA
verify return:1
depth=1 C = US, O = DigiCert Inc, CN = RapidSSL TLS DV RSA Mixed SHA256 2020 CA-1
verify return:1
depth=0 CN = *.my-co-example.com
verify return:1
CONNECTED(00000003)
---
Certificate chain
0 s:/CN=*.my-co-example.com
i:/C=US/O=DigiCert Inc/CN=RapidSSL TLS DV RSA Mixed SHA256 2020 CA-1
-----BEGIN CERTIFICATE-----
MIIGnxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxgkqhkiG9w0BAQsFADBZ
...
I assume that means somewhere on the filesystem of my server there would be a file somewhere that has the string ...
MIIGnxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxgkqhkiG9w0BAQsFADBZ
... in it. How can I find that file without having to execute something like?
sudo grep -sr MIIGnxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxgkqhkiG9w0BAQsFADBZ /
I assume that means somewhere on the filesystem of my server there would be a file somewhere that has the string ...
This assumption is wrong. What you see is part of the server certificate, which need to be checked against a CA certificate located in the local trust store. The server certificate is typically not in the local trust store. See SSL Certificate framework 101: How does the browser actually verify the validity of a given server certificate? for more on certificates are checked.
You can see more with "CURL -W CERTS" from Daniel Stenberg (also on Stack Overflow), which works on curl for the past 25+ years.
When a client connects to a TLS server it gets sent one or more certificates during the handshake.
Those certificates are verified by the client, to make sure that the server is indeed the right one: the server the client expects it to be; no impostor and no man in the middle etc.
When such a server certificate is signed by a Certificate Authority (CA), that CA’s certificate is normally not sent by the server but the client is expected to have it already in its CA store.
(So no file to check on the filesystem)
Ever since the day SSL and TLS first showed up in the 1990s user have occasionally wanted to be able to save the certificates provided by the server in a TLS handshake.
The openssl tool has offered this ability since along time and is actually one of my higher ranked stackoverflow answers.
Now (for curl 7.88.0,to be shipped in February 2023), Daniel proposes:
Using the –write-out (-w) option and the new variables %{certs} and %{num_certs}, curl can now do what you want.
Get the certificates from a server in PEM format:
$ curl https://curl.se -w "%{certs}" -o /dev/null > cacert.pem
$ curl --cacert cacert.pem https://curl.se/
That is easier to parse than the openssl s_client -showcerts -connect current alternative.
My application uses the Close API (https://developer.close.com/) to store user data. Our testing environment is now getting SSL errors when trying to write to it:
Faraday::SSLError (SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed (certificate has expired))
What could the cause be? My first assumption reading the error message is that the Close certificates might have expired. But that seems unlikely - the service is generally well-maintained, and we are not having any issues in production.
The next thing I considered was that perhaps the certificate store on our server is out of date. That would not surprise me much, because the server is running Ubuntu 14, and other people are experiencing problems there. But we have multiple machines with the same configuration, and only one of them is giving us trouble. Is it possible this is the root cause?
To reproduce:
$ openssl s_client -CApath /dev/null -showcerts -connect api.close.com:443 -servername api.close.com
CONNECTED(00000003)
depth=3 O = Digital Signature Trust Co., CN = DST Root CA X3
verify error:num=10:certificate has expired
notAfter=Sep 30 14:01:15 2021 GMT
verify return:0
The problem turned out to be the recent deprecation of the DST root certificate. An old root certificate (DST) was deprecated on Sep 30, and the modern version (ISRG) is now expected.
The testing server still had the DST and ISRG certificates installed, whereas all the other machines only had ISRG. I assume that something about this particular API was looking preferentially for the DST certificate and ignoring ISRG unless it was the only one.
To solve: Remove DST cert from /usr/share/ca-certificates/mozilla and leave ISRG one there - try the openssl command above, now it works.
Issue: users can't log into mobile app due to "unable to contact server"
debugging message: "TypeError: Network request failed"
Attempted fixes: restarted server, verified that db is running and nothing has changed, restarted VM that server is running on, I checked the api using postman. When I ran a simple POST request I got the following message:
There was an error connecting to
https://app.something.com/api/Accounts/5076/sometest?filter%5Bwhere%5D%xxxxx%5D=null&access_token=mwVfUBNxxxxxxx5x4A4Y5DktKnTZXeL6CB34MoP.
One of the suggestions I was given was:
Self-signed SSL certificates are being blocked: Fix this by turning
off 'SSL certificate verification' in Settings > General
As soon as I followed this step, I was able to make the POST request and everything seemed to work fine. I'm completely new to this type of error. Allso, I did not set up this app/db/certificates. So, other than unblocking self-signed SSL certificates(which seems like a really bad idea), I'm not sure how to proceed. What are my options?
here's what the result of examining the certificate:
depth=1 C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
verify error:num=20:unable to get local issuer certificate
verify return:0 poll
errornotBefore=Jan 28 11:54:38 2019 GMT
notAfter=Apr 28 11:54:38 2019 GMT
Either, purchase a signed certificate from a CA if you plan to expose this to the public.
Or the free option is to use Let's Encrypt, with this service, you are issues free certificates, however they expire in a relatively short period of time; most of the time however you can run an agent which will automatically rotate the certificates before they expire.
The third option is to install the CA certificate that was used to self sign this, into to your browser. i.e., like a large company might do.
edit
Seems like it might instead be an expired certificate? Check when it expires with this:
openssl s_client -showcerts -servername www.stackoverflow.com -connect www.stackoverflow.com:443 </dev/null | openssl x509 -noout -dates
change both instances of stackoverflow you your domain
One thing I could not find. I've just received a Comodo ssl certificate (.crt file and a key) from a client to install on the webserver. I did not receive an Intermediate though. The certificate CN is:
Extended Validation Secure Server CA
and i did find this Intermediate on Comodo website:
https://support.comodo.com/index.php?/Knowledgebase/Article/View/931/91/intermediate-2-comodo-ev-secure-server-ca
How can I check whether this particular certificate is validated by this Intermediate?
I was trying
openssl verify -verbose -purpose sslserver -CAfile comodoextendedvalidationsecureserverca.crt my_certificate.crt
but got this error:
error 20 at 0 depth lookup:unable to get local issuer certificate
Which I would expect if the validation fails. But surprisingly I got similar error (error 2 at 1 depth lookup:unable to get issuer certificate) while trying this command on a certificate/Intermediate pair I'm sure is correct.
I want to make sure, I'm out of options of finding a proper Intermediate, before i start nagging my client.
As Patrick suggested:
openssl verify -purpose sslserver -untrusted <Intermediate_file.crt> <cerificate_file.crt>
Is a good way to go. Thanks
I have a certificate chain as:
root CA -> intermediate CA -> org CA -> client Cert
When I verify the client cert with CA as root CA -> intermediate CA -> org CA, it works:
$ cat org_1_ca/ca_crt.pem intermediate_ca/ca_crt.pem root_ca/ca_crt.pem > /tmp/test123.pem
$ openssl verify -CAfile /tmp/test123.pem client/client_crt.pem
client_crt.pem: OK
But when I chained my client cert with org CA (org CA -> client Cert), and have the rest of the chain as CA (root CA -> intermediate CA), it doesn't:
$ cat intermediate_ca/ca_crt.pem root_ca/ca_crt.pem > /tmp/test12.pem
$ openssl verify -CAfile /tmp/test12.pem client/org1_client_crt.pem
client/org1_client_crt.pem: C = US, ST = CA, L = LA, O = PP, OU = TEST, CN = user
error 20 at 0 depth lookup:unable to get local issuer certificate
Is this something that fundamentally wrong or openssl verify doesn't like that? I tried the same thing with nginx and and openssl connect and there was no luck there. Any help is appreciated.
The openssl commandline verify operation reads only one certificate, the first one, from the file given as operand, or from each file if more than one is given. This differs from the files specified with the -CAfile -trusted -untrusted options which can (and typically do) contain multiple certs.
Your file client/org1_client_crt.pem presumably contains the client cert and the 'org CA' cert, in that order. Only the client cert is used, the 'org CA' cert is ignored, and as a result you do not have a valid chain to verify.
If you want to use commandline to mimic/test the validation that a receiver (for a client cert, the server) would do, supply the leaf cert as the operand and all other transmitted (chain) certs with -untrusted, and the anchor(s) plus any 'known' intermediates in the truststore either explicit or defaulted.
There is no openssl connect operation; I assume you mean openssl s_client with options including -connect since that's one place it would make sense to use a client cert chain. The -cert option to s_client similarly uses only the first cert in the file. There is no option on commandline to specify the client chain except in the most recent version, 1.1.0, and even there it isn't documented so you have to read the help message carefully or the code, although the API/library has long supported this for code you write yourself.
Through 1.0.2 if you want to send a client cert with full chain to the server (as you should per the RFCs), assuming the server requests client authentication which is not usual and not the default for nginx (among others), you have to use a trick: supply all the certs needed for the client chain in the truststore, in addition to the anchor(s) needed to verify the server, either using -CAfile and/or -CApath explicitly, or using (modifying if needed) the default truststore unless your openssl is an older non-RedHat version where the default truststore didn't work in s_client s_server s_time only.
And the same is true about the server cert/chain in s_server except that it is used almost always instead of very rarely.