How to use wget with ssl certificate - ssl

I am using wget in my program to get some file using HTTP protocol. Here i need to set security so we moved HTTP protocol to HTTPS.
After changing to HTTPS how to perform wget. I mean how to make trusted connection between two machines then perform wget.
I want to make sure that wget can be performed from certain system only.

Step 1: SSL Certificates
First things first, if this machine is on the internet and the SSL certificate is signed by a trusted source, there is no need to specify a certificate.
However, if there is a self signed certificate involved things get a little more interesting.
For example:
if this machine uses a self signed certificate, or
if you are on a network with a proxy that re-encrypts all https connections
Then you need to trust the public key of the self signed certificate. You will need to export the public key as a .CER file. How you got the SSL certificate will determine how you get the public key as a .CER
Once you have the .CER then...
Step 2: Trust the Certificate
I suggest two options:
option one
wget --ca-certificate={the_cert_file_path} https://www.google.com
option two
set the option on ~/.wgetrc
ca_certificate={the_cert_file_path}
Additional resources
Blog post about this wget and ssl certificates
wget manual

macOS users can use the cert.pem file:
wget --ca-certificate=/etc/ssl/cert.pem
or set in your ~/.wgetrc:
ca_certificate = /etc/ssl/cert.pem

On Linux (at least on my Debian and Ubuntu distributions), you can do the following to install your cert to be trusted system-wide.
Assuming your certificate is ~/tmp/foo.pem, do the following:
Install the ca-certificates package, if it is not already present, then do the following to install foo.pem:
$ cd ~/tmp
$ chmod 444 foo.pem
$ sudo cp foo.pem /usr/local/share/ca-certificates/foo.crt
$ sudo update-ca-certificates
Once this is done, most apps (including wget, Python and others) should automatically use it when it is required by the remote site.
The only exception to this I've found has been the Firefox web browser. It has its own private store of certificates, so you need to manually install the cert via its Settings interface if you require it there.
At least this has always worked for me (to install a corporate certificate needed for Internet access into the Linux VMs I create).

Related

cert in /etc/ssl/cert.pem expired today (MacOS Mojave)

The cert that is stored in /etc/ssl/cert.pem expired today on my Mojave computer, which has caused problems with my curl commands. Is there anyway to update it? I see that my Catalina computer has a newer certificate. Can I just copy it over to my Mojave computer?
Yesterday, the Let's Encrypt DST Root CA X3 root certificate expired, which is causing similar problems as you experienced: curl displays a SSL certificate problem: certificate has expired error when trying to access websites using the new Let's Encrypt ISRG Root X1 certificate.
On macOS Mojave (and maybe others, but this is what I use), the default curl uses certificates in /etc/ssl/cert.pem when verifying TLS connections (you can confirm this by by running curl -v https://example.com 2>&1 | grep CAfile).
The simplest fix is to delete the expired root certificate from the /etc/ssl/cert.pem file, assuming its replacement already exists in the file. This is enough to fix the expired DST Root CA X3, because its replacement, ISRG Root X1 already exists in the /etc/ssl/cert.pem file. Delete all lines from ### Digital Signature Trust Co. to -----END CERTIFICATE-----.
If you need to completely replace /etc/ssl/cert.pem with updated certificates, you can replace it with certificates exported from the macOS System Roots keychain:
Make a backup of the old /etc/ssl/cert.pem:
sudo cp /etc/ssl/cert.pem{,-orig}
Export system keychain and replace contents of /etc/ssl/cert.pem:
security find-certificate -a -p /System/Library/Keychains/SystemRootCertificates.keychain | sudo tee /etc/ssl/cert.pem >/dev/null
Alternatively, you can tell curl to use a different certificate file like this:
curl -I --cacert /path/to/certificates.pem https://example.com/
Or configure the location of this file using the CURL_CA_BUNDLE=/path/to/certificates.pem environment variable.
For Mojave in 2022/02, here is a simple solution:
Backup the current version of /etc/ssl/cert.pem in your macOS.
Download the new CA certificate( officially recommended by curl.se), renaming it to cert.pem.
Replace the original one with the new one: sudo mv cert.pem /etc/ssl/cert.pem.
I think you have problem with AddTrust CA
What You Need to Do
For most use cases, including certificates serving modern client or server systems, no action is required, whether or not you have issued certificates cross-chained to the AddTrust root.
As of April 30, 2020: For business processes that depend on very old systems, Sectigo has made available (by default in the certificate bundles) a new legacy root for cross-signing, the “AAA Certificate Services” root. However, please use extreme caution about any process that depends on very old legacy systems. Systems that have not received the updates necessary to support newer roots such as Sectigo’s COMODO root will inevitably be missing other essential security updates and should be considered insecure. If you would still like to cross-sign to the AAA Certificate Services root, please contact Sectigo directly.
https://support.sectigo.com/articles/Knowledge/Sectigo-AddTrust-External-CA-Root-Expiring-May-30-2020
I tried Quinn Comendant's solution, but it didn't work.
I ended up solving this by installing OpenSSL and linking the cert.pem file to the one under OpenSSL (in MacOS Mojave):
$ sudo cp /etc/ssl/cert.pem /etc/ssl/cert.pem.bak
$ brew install openssl
$ brew info openssl
openssl#3: stable 3.0.0 (bottled) [keg-only]
Cryptography and SSL/TLS Toolkit
https://openssl.org/
/usr/local/Cellar/openssl#3/3.0.0 (6,415 files, 28MB)
Poured from bottle on 2021-10-04 at 11:13:17
From: https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/openssl#3.rb
License: Apache-2.0
==> Caveats
A CA file has been bootstrapped using certificates from the system
keychain. To add additional certificates, place .pem files in
/usr/local/etc/openssl#3/certs
...
From this we'll know the config is in /usr/local/etc/openssl#3, then:
$ sudo rm /etc/ssl/cert.pem
$ sudo ln -s /usr/local/etc/openssl#3/cert.pem /etc/ssl/cert.pem
This solved my problem.
Thus far, the only working solution I can find is to just upgrade to Catalina. None of the above suggestions work because even Homebrew is broken because most of their mirrors use Let's Encrypt certificates. Adding a new ISRG Root X1 certificate file from http://x1.i.lencr.org/ to the system keychain and exporting to a new /etc/ssl/cert.pem doesn't even fix the problem.

Docker private registry | TLS certificate issue

I've tried to follow the following tutorial to setup our own private registry (v2) on an AWS Centos machine.
I've self signed a TLS certificate and placed it in /etc/docker/certs.d/MACHINE_STATIS_IP:5000/
When trying to login the registry (docker login MACHINE_IP:5000) or push a tagged repository (MACHINE_IP:5000/ubuntu:latest) i get the following error :
Error response from daemon: Get https://MACHINE_IP:5000/v1/users/: x509: cannot validate certificate for MACHINE_IP because it doesn't contain any IP SANs
Tried to search for an answer for 2 days, however I couldn't find any.
I've set the certificate CN (common name) to MACHINE_STATIC_IP:5000
When using a self signed TLS certificate docker daemon require you to add the certificate to it's known certificates.
Use the keytool command to grab the certificate :
keytool -printcert -sslserver ${NEXUS_DOMAIN}:${SSL_PORT} -rfc > ${NEXUS_DOMAIN}.crt
And copy it your client's machine SSL certificates directory (in my case - ubuntu):
sudo cp ${NEXUS_DOMAIN}.crt /usr/local/share/ca-certificates/${NEXUS_DOMAIN}.crt && sudo update-ca-certificates
Now reload docker daemon and you're good to go :
sudo systemctl restart docker
You can also use the following command to temporarily trust the certificate without adding it your system certificates.
docker --tlscert <the downloaded tls cert> pull <whatever you want to pull>

Using host machines CA cert within Docker container

For https access I need to add a CA cert file to /usr/local/share/ca-certificates on my Ubuntu host machine.
Currently my Dockerfile RUN wget https... is failing since the certificate verification is failing.
How can Docker use the host machine CA cert? Or is there an existing enhancement opened to allow this?
I've used CA and SSL certs via a passthrough mount, but this looks like you're trying to do it in the Dockerfile.
So my suggestion would be - copy the CA cert to the image as part of the Dockerfile, and then proceed as normal. Or drop to http, or run wget --no-check-certificate if you're happy with that.
There are a few open bugs in this area:
https://github.com/docker/machine/issues/1799
https://github.com/docker/docker/issues/4372
https://github.com/docker/machine/issues/1435
https://github.com/deis/deis/issues/2230

Using pip with 'insecure' mercurial

Is it possible to use pip with a requirements file to install from a Mercurial repository which needs to be accessed using --insecure in order to work.
The repository is accessed via SSL, and due to pip ignoring Server Name Indication (SNI), a valid SSL certificate is not being honoured (it uses the main certificate for the IP address). --insecure gets round this issue
ie:
hg clone https://username#domain.org/username/app_name --insecure
So where you can usually do something like this:
pip install -e hg+https://username#domain.org/username/app_name#egg=app_name
or have the following in a pip requirements file:
hg+https://username#domain.org/username/app_name#egg=app_name
Neither of these work due to the SSL SNI issue.
Does anyone have any good suggestions?
Why don't you try to use [hostfingerprints] in your ~/.hgrc?
For example:
[hostfingerprints]
example.com = 38:76:52:7c:87:26:9a:8f:4a:f8:d3:de:08:45:3b:ea:d6:4b:ee:cc
To obtain the self-signed certificate, follow the steps suggested in https://www.mercurial-scm.org/wiki/CACertificates
The root CA certificate for a server can for example be retrieved with
Firefox. Browse to https://example.com/repo and verify that this is the
repository you trust, click the lock symbol in the lower right corner,
View Certificate, Details, select the certificate at the top of the
Certificate Hierarchy, Export, "X.509 Certificate (PEM)" and save
somewhere for example as server.pem. With other browsers on Windows
you have to view the certificate at the top of the Certification Path
and "Copy to File" as "Base-64 encoded X.509 (.CER)". Several such
files can be concatenated into one cacerts file.

Getting error in Curl - Peer certificate cannot be authenticated with known CA certificates

I am getting the below error while making ssl connection with self signed certificate.
"Peer certificate cannot be authenticated with known CA certificates"
It is working fine with CA signed certificate.
I am setting the below using curl_easy_setopt().
curl_easy_setopt(MyContext, CURLOPT_CAPATH, CA_CERTIFICATE_PATH)
curl_easy_setopt(MyContext, CURLOPT_SSL_VERIFYPEER,TRUE);
The curl version:
libcurl-7.19.7-26
Openssl version is:
0_9_8u
Please let me know how to solve this issue.
By default CURL will generally verify the SSL certificate to see if its valid and issued by an accepted CA. To do this, curl uses a bundled set of CA certificates.
If you'd like to turn off curl's verification of the certificate, use the -k (or --insecure) option. Here's an example:
curl --noproxy -k \* -D - https://127.0.0.1:443/some-secure-endpoint
Security issue: This answer disables a security feature. Do not use this in production!
For php it is possible to switch off curl's verification of the certificate (see warning below) e.g. for curl_exec
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, FALSE);
http://php.net/manual/en/function.curl-setopt.php
(evaluate the security risk yourself, in my case it was on a partner company's server and the file required contained no secure information - just happened to be on a secure server)
We fixed a similar issue on CentOS 6 by updating curl to the latest version available in the standard repositories and installing the newest ca-certificates bundle:
yum update curl
yum install ca-certificates
libcurl performs peer SSL certificate verification by default. This is done
by using CA cert bundle that the SSL library can use to make sure the peer's
server certificate is valid.
If you communicate with HTTPS or FTPS servers using certificates that are
signed by CAs present in the bundle, you can be sure that the remote server
really is the one it claims to be.
Until 7.18.0, curl bundled a severely outdated ca bundle file that was
installed by default. These days, the curl archives include no ca certs at
all. You need to get them elsewhere. See below for example.
For more to know about Peer SSL Certificate Verification visit http://curl.haxx.se/docs/sslcerts.html
Though this error happened in the case of using git clone rather than with using curl, I've recently stumbled across an identical error message:
Peer certificate cannot be authenticated with known CA certificates
Similar to Arth's findings, something that worked for CentOS 6 (in order to successfully use HTTPS URLs with git clone for related GitLab repositories) involved updating the trusted certificates on the server (i.e., the server that is using HTTPS), using the following steps:
sudo yum install ca-certificates
sudo update-ca-trust enable
sudo cp /path/to/your_new_cert.crt /etc/pki/ca-trust/source/anchors/
sudo update-ca-trust extract
Perhaps the same certificate steps can be applied for the case of curl (or other similar scenarios) for users on CentOS in the future.
Security issue: This answer disables a security feature. Do not use this in production!
In 'C'
curl_easy_setopt(curl_handle, CURLOPT_SSL_VERIFYPEER, 0);
worked for me
As we checked and observed/ Found in Centos 8 .
Due to Proxy issue your packages not allowing you to get accessible to update or download any packages.
try to add sslverify=0 in file /etc/dnf/dnf.conf
Its worked for me.
Also make sure you must have proper internet acess on your server.