I was going through a Google tutorial to connect Google Sheets with python here. I am running this piece of code behind a corporate proxy, therefore the certificates received are already replaced by the proxy server.
I have created token.pickle file on a non-proxy machine and transferred it here. I get certificate verification failure at this line of the code given in the guide:
service = build('sheets', 'v4', credentials=creds)
The error that comes up is:
self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1076)
I dig out into the build method here. Upon searching through the source codes, I found out that creds and http object as parameters to this method are mutually exclusive, I.e. when I pass creds object, I can't pass a http object to this function.
If I could pass a httplib2.http object, I would have set it to disable SSL verification. But now with this wrapper around, I cannot find any way out to specify this.
I tried the following two hacks also:
Creating a unverified context:
ssl._create_default_https_context = ssl._create_unverified_context
Setting PYTHONHTTPSVERIFY=0 as a env variable
But none of them works.
I have my corporate proxy certificate installed in my ca-bundle, and the .cer file too.
Related
Problem Background:
As part of the Computer Networking course assignment, I have been given task of implementing a Proxy Server ( using python socket and ssl module ) that handles https communications between the browser and the origin server (The real server that my browser wants to talk to).
What I have done so far:
I have implemented the above requirement using ssl sockets and also generated self-signed 'cert.pem' 'key.pem' files.
What I need to do:
Now I just need to tell my browser (chrome 89 on kubuntu 20.04) to accept this self-signed certificate and then test the working of my proxy server.
Reading from this stackoverflow question, I can see that I have to:
(1) become my own CA (2) then sign my SSL certificate as a CA. (3) Then import the CA certificate (not the SSL certificate, which goes onto my server) into Chrome.
My confusion/question:
So if I do this, when eventually I am done with this assignment, how do I reverse all these steps to get my browser in the previous state before I had made all these changes. Also, how to reverse the "become your own CA" and also delete the SSL certificates signed by my CA.
Basically, I want my system to return to the previous state it was before I would have made all these changes.
UPDATE:
I have done the previously outlined steps but now I get an error.
Here is a snippet of my code:
serv_socket = socket(AF_INET, SOCK_STREAM)
serv_socket.bind(('', serv_port))
serv_socket.setsockopt(SOL_SOCKET, SO_REUSEADDR, 1)
context = ssl.create_default_context(ssl.Purpose.CLIENT_AUTH)
context = context.load_cert_chain('cert.pem', 'key.pem')
context.set_ciphers('EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH')
serv_socket.listen(10)
socket_to_browser, addr = serv_socket.accept()
conn_socket_to_browser = context.wrap_socket(socket_to_browser, server_side=True)
At the last line conn_socket_to_browser = context.wrap_socket(socket_to_browser, server_side=True) an exception is thrown: [SSL: HTTPS_PROXY_REQUEST] https proxy request (_ssl.c:1123)
What am I doing wrong ?
As glamorous as "becoming your own CA" sounds, with openssl it basically comes down to creating a self-signed certificate, and then creating a directory where some CA-specific configuration will be stored (I don't fully remember the specifics, but I think it was just some files related to CNs and serial numbers) so basically reversing the "become your own CA" step is something as mundane as deleting this directory along with the private key and self-signed certificate you were using for the CA. That's it, the CA is no more.
And for chrome returning to the previous state, you would just go the the CA list where you added the CA certificate, select it and delete it. Chrome will stop accepting certificates signed by your CA.
Regarding your new problem... In my opinion, you have developed some kind of reverse proxy (meaning that you expect normal HTTPS requests that you then redirect to the real server) but you have configured Chrome to use it as a forward proxy. In this case, Chrome does not send it a normal HTTPS request, it sends a special non-encrypted CONNECT command and only after receiving the non-encrypted response, it negotiates the TLS connection. That's why openssl says "https proxy request" because it has detected a "https proxy request" (a CONNECT command) instead of the normal TLS negotiation.
You can take a look at How can a Python proxy server (using SSL socket) pretend to be an HTTPS server and specify my own keys to get decrypted data?
It's python, but I think that you'll get the idea
I've always been an end consumer of HTTPS and have never really understood it that well but am looking to change that.
I am calling a RESTful web service over HTTPS. For example...
curl -X GET \
https://myCompanydns/rest/connect/v1.4/myEndpoint
With all my requests I send a basic authentication header i.e a username and password.
When I make these calls via my application I was expecting to have to add a certificate into like a jks (which I've had to do in the past) but on this occasion I've found that I can call the HTTPS web service without that.
For HTTPS to work I believe there is an SSL handshake? How is that happening successfully is this scenario without a jks?
Again, sorry for this beginner type question.
When doing a https://... request the client needs to verify that the servers certificate is the expected one - and not some man in the middle. This is done (among other things) by making sure that the servers certificate was issued by a trusted certificate authority (CA). Which CA is trusted is setup in the local trust store (i.e. local to the client). In the above call where no explicit trust store is given curl is using its default trust store. In the case where you've explicitly gave a jks you've provided the application with a specific trust store it should use.
For more on how the server certificates gets validated see SSL Certificate framework 101: How does the browser actually verify the validity of a given server certificate?.
I use resty.http module. But the data is used. For usual http or https without verification all works.
local http = require("resty.http").new()
local res, err = http:request_uri(url, {
method = method,
headers = headers,
body = body,
ssl_verify = false
})
But if I do not use ssl_verify it wouldn't work with the error:
lua ssl certificate verify error: (20: unable to get local issuer
certificate),
I found using Google that lua_ssl_trusted_certificate can help. But I don't know how it can help. I have tested such command: lua_ssl_trusted_certificate /etc/ssl/certs/GlobalSign_Root_CA.pem; but it did not help to me.
How to verify https in a proper way?
In your nginx.conf you need to configure
lua_ssl_verify_depth 2;
lua_ssl_trusted_certificate /pathto-ca-certs.pem;
In my case my server calls out to only one external HTTPS endpoint. So I exported the certificate with the full chain (via borwser ceritificate export in Firefox) and imported into a PEM file. This is the .pem file that I supplied above.
I use lua-resty-http to make the calls to https and it works fine. You can use tools like wireshark/fiddler to monitor the outgoing connections to see if the requests are being made the way you want.
I am trying to create an HTTPS-tunnel on my machine. My intention is having all requests to https://localhost:8888/<something> (the port where Fiddler is listening to) be directed to https://myserver.net/<something>. I am using the following script as per Fiddler doc:
static function OnBeforeRequest(oSession: Session) {
// <Fiddler 2 preexisting code>
// HTTPS redirect -----------------------
if (oSession.HTTPMethodIs("CONNECT") &&
(oSession.PathAndQuery == "localhost:8888"))
{
oSession.PathAndQuery = "myserver.net:443";
}
if (oSession.HostnameIs("localhost"))
oSession.hostname = "myserver.net";
// --------------------------------------
// <Fiddler 2 preexisting code>
}
Also in Fiddler settings I checked the decryption check and installed certificates as you can see in the image below:
I restart Fiddler, it prompts me to install its fake certificates, I agree. I can see the certificate in my Windows Certificate System Repository when using certmgr. It is a self-signed certificate.
So What I do is opening a browser and type: https://localhost:8888/mypage.html, and what I get is an error. Internet Explorer reports this:
Error: Mismatched Address. The security certificate presented by this
website was issued for a different website's address. This problem
might indicate an attempt to fool you or intercept any data...
When I get certificate info (basically the certificate presented by the contacted host is being rejected, the same certificate can be displayed), I can see that the rejected certificate was issued by Fiddler and the subject is myserver.net.
So the certificate is ok because it is certifying myserver.net, I see that the problem is that probably my browser was expecting a certificate whose subject is localhost. Is it true?
How to handle this situation?
Assumption
I can understand that the problem is a certificate being issued for a website which I did not ask for. So the solution would be using a certificate certifying localhost:8888?
A certificate is valid if it is directly or indirectly (via intermediate certificates) signed by a trusted CA and if the hostname matches the certificate. If the last condition would not be enforced anybody with a valid certificate from a trusted CA could incorporate any other site.
To make use of fiddler and not run into this problem you should configure your browser to use fiddler as a web proxy and then use the real URL inside the browser instead of ip:port of fiddler.
I have been trying to implement a very basic Boost SSL implementation to try and learn the basics. The server I want to communicate with had already given me their public key in plain text. I already put a lot of the other code (asynchronous connection, handshaking, etc) in.
I first tried to implement SSL without verification of their certificate using the following setup of the Boost SSL stream:
boost::asio::ssl::context ctxt(boost::asio::ssl::context::sslv23);
ctxt.set_verify_mode(boost::asio::ssl::verify_none);
This implementation worked fine and I was able to connect with the server. When I tried to implement the verification of the peer certificate, however, the handshaking fails. I tried using the following code:
boost::asio::ssl::context ctxt(boost::asio::ssl::context::sslv23);
ctxt.set_verify_mode(boost::asio::ssl::verify_peer);
ctxt.load_verify_file("peer.crt");
I put the "peer.crt" containing the public key (along with the -----BEGIN CERTIFICATE----- and -----END CERTIFICATE----- tags) in the directory where I am running my executable. For whatever reason the handshake now fails with the error code 336134278: certificate verify failed. I also tried putting the full path to the verify file in there but with no luck.
My questions are the following:
Where should I be specifying the file name for the verify file in load_verify_file? Is it simply in the directory where I am running my executable?
Am I not setting up the handshaking process with peer verification properly? I do not have my own verify callback as I assumed the peer verification would happen automatically if I specified it as such.
Should I be handling the certificate in a certain way by installing it or something like that?
Is there a better way of debugging this functionality? I am using VS10 and can only get to the ipp so I cannot actually view the verification taking place.
Any help is appreciated, thanks!
You should be able to use either a relative or absolute path.
Your use of set_verify_mode() and load_verify_file() looks fine. I have done exactly this in my own code. A default verify callback is used if you do not specify one.
You don't need to "install" the certificate.
I don't know of easy ways to debug boost::asio SSL connections, but you can use OpenSSL command line tools, such as s_client, to test connections. boost::asio uses OpenSSL under the hood.
I suspect that you don't have the entire certificate chain of certificates in your file. You can extract them from your server with (replace www.google.com:443 with your server and port):
openssl s_client -connect www.google.com:443 -showcerts
If you only wish to check some of the certificates, e.g. only the leaf certificate, you can use your own verify callback. An example of a custom callback, as well as a description of the verification modes and options are on this page.
A good place to start is the HTTP Client in asio examples.
Are you calling set_verify_callback on the socket with the callback function to verify the certificate? E.g.:
bool verify_certificate(bool preverified, boost::asio::ssl::verify_context& ctx)
{
char subject_name[256];
X509* cert = X509_STORE_CTX_get_current_cert(ctx.native_handle());
X509_NAME_oneline(X509_get_subject_name(cert), subject_name, 256);
return preverified;
}