This comes from the python documentation for Python "Requests" http library
"You can also specify a local cert to use as client side certificate, as a single file (containing the private key and the certificate) or as a tuple of both file’s path":
>>> requests.get('https://kennethreitz.com', cert=('/path/server.crt', '/path/key'))
<Response [200]>
http://docs.python-requests.org/en/latest/user/advanced/
What's a good way to do the same thing in Clojure ? I looked at clj-http and http-kit but did not see an example
Have you seen async-http-client?
It has specific tests for cert handling you can view here. The API docs are here, particularly relevant would be the namespace http.async.client.cert.
From that test, a typical example of loading keystore and certificate is:
(def ks-file "test-resources/keystore.jks")
(def cert-file "test-resources/certificate.crt")
(def password "secret")
(defn load-test-certificate [] (load-x509-cert cert-file))
(defn load-test-keystore [] (load-keystore (resource-stream ks-file) password))
Related
I have been learning with WebSocket++ and built some of the server examples (Windows 10 Visual Studio 2019). The non-TLS examples work without issues, however, the TLS-enabled examples (echo_server_both.cpp and echo_server_tls.cpp) can't do the handshake. I am very new to web development in general so I know I must be doing something wrong with regards to the certificate and keys.
I am testing the servers with WebSocket King client, an extension of Google Chrome that connects correctly to other websocket servers like wss://echo.websocket.org and to my own localhost when I don't use TLS.
The echo_server_both example comes with a server.pem file, and the echo_server_tls example comes with server.pem and dh.pem. I have used the same files that come with the samples, and I have also tried generating and registering my own .pem files using openSSL. In both cases I get this when the client tries to connect:
[2021-06-29 20:51:21] [error] handle_transport_init received error: sslv3 alert certificate unknown
[2021-06-29 20:51:21] [fail] WebSocket Connection [::1]:63346 - "" - 0 asio.ssl:336151574 sslv3 alert certificate unknown
[2021-06-29 20:51:21] [info] asio async_shutdown error: asio.ssl:336462231 (shutdown while in init)
I discovered these errors after I edited handle_init() in tls.hpp, following a suggestion in another site, to look like this:
void handle_init(init_handler callback,lib::asio::error_code const & ec) {
if (ec) {
//m_ec = socket::make_error_code(socket::error::tls_handshake_failed);
m_ec = ec;
} else {
m_ec = lib::error_code();
}
callback(m_ec);
}
This change let the actual openSSL error to show in the console, otherwise it would show a generic "handshake failed" error.
I know I'm not doing what I should with the certificates, but I have no idea where else to look or what to do next. Can anyone here help please? Should I use the .pem files that come with the examples, or should I generate my own? in case I should generate my own, what would be the openSSL command to do that correctly and how do I tell my PC to recognize these as valid so that the server works?
Found the problem: WebSocket++ will not accept a self-signed certificate (the ones you can create directly in your own PC using OpenSSL or the Windows utilities). There is no way around it. You must have a valid, authority-validated and endorsed certificate. You can get such a certificate for free (valid only for 90 days) from https://zerossl.com/. The site has detailed instructions on how to request, obtain and install a certificate. After getting a valid certificate and installing it on my server, everything worked as it should.
I have the below function in my code, which works perfectly fine when I'm not behind any proxy. In fact, without even mentioning the certifi default CA certificate, it works fine if I pass verify=TRUE, I guess, because it works in the same way.
def reverse_lookup(lat, long):
cafile=certifi.where()
params={'lat' : float(lat), 'lon' : float(long), 'format' : 'json',
'accept-language' : 'en', 'addressdetails' : 1}
response = requests.get("https://nominatim.openstreetmap.org/reverse", params=params, verify=cafile)
#response = requests.get("https://nominatim.openstreetmap.org/reverse", params=params, verify=True) <-- this works as well
result = json.loads(response.text)
return result['address']['country'], result['address']['state'], result['address']['city']
When I run the same code from within my enterprise infrastructure (where I'm behind proxy), I make some minor changes in the code mentioning the proxy as parameter in requests method:
def reverse_lookup(lat, long):
cafile=certifi.where()
proxies = {"https" : "https://myproxy.com"}
params={'lat' : float(lat), 'lon' : float(long), 'format' : 'json',
'accept-language' : 'en', 'addressdetails' : 1}
response = requests.get("https://nominatim.openstreetmap.org/reverse", params=params, verify=cafile, proxies=proxies)
result = json.loads(response.text)
return result['address']['country'], result['address']['state'], result['address']['city']
But it gives me one out of these 3 SSL errors at different times, if I set verify=True or verify=certifi.where():
CERTIFICATE_VERIFY_FAILED
UNKNOWN_PROTOCOL
WRONG_VERSION_NUMBER
Only time it works is when I completely bypass the SSL verification with verify=False
My questions are:
Since I'm sending the https request via proxy, is it ok if I bypass SSL verification ?
How to make the default context of SSL verification work in this case, when I'm behind proxy ?
Any help is appreciated. Code tested in both Python 2.7.15 and 3.9
Since I'm sending the https request via proxy, is it ok if I bypass SSL verification ?
Do you need the protection offered by HTTPS, i.e. encryption of the application data (like passwords, but also the full URL) to protect against sniffing or modifications by a malicious man in the middle? If you don't need the protection, then you can bypass certificate validation.
How to make the default context of SSL verification work in this case, when I'm behind proxy ?
The proxy is doing SSL interception and when doing this issues a new certificate for this site based on an internal CA. If this is expected (i.e. not an attack) then you need to import the CA from the proxy as trusted with verify='proxy-ca.pem'. Your IT department should be able to provide you with the proxy CA.
But it gives me one out of these 3 SSL errors at different times, if I
set verify=True or verify=certifi.where():
CERTIFICATE_VERIFY_FAILED
UNKNOWN_PROTOCOL
WRONG_VERSION_NUMBER
It should only give you CERTIFICATE_VERIFY_FAILED. The two other errors indicate wrong proxy settings, typically setting https_proxy to https://... instead of http://... (which also can be seen in your code).
I have a .pem file which will successfully connect to my website via the --cert parameter of curl. I then converted that to a der file:
openssl x509 -inform PEM -outform DER -in client.pem -out cert.der
Then I loaded that cert.der into my project and I'm now trying to use that with Alamofire, following the example on their homepage:
let serverTrustPolicy = ServerTrustPolicy.PinCertificates(
certificates: ServerTrustPolicy.certificatesInBundle(),
validateCertificateChain: true,
validateHost: true
)
let policyManager = ServerTrustPolicyManager(policies: ["my.domain.com" : serverTrustPolicy])
manager = Alamofire.Manager(configuration: configuration, serverTrustPolicyManager: policyManager)
manager.request(.GET, url, parameters: params, encoding: .URLEncodedInURL, headers: nil)
.authenticate(usingCredential: credential)
.validate()
.responseJSON {
When that runs though it just fails and I get a 'cancelled' as the error's localizedDescription, which is what Alamofire does when authentication fails.
What am I doing wrong?
The Alamofire cert pinning logic does not currently support this use case. It is only designed to handle cert and public key pinning, not client certificates used to authenticate with the server. This is something we could support in the future if this is a common use case.
With that said, I'm assuming in this case you are receiving a NSURLAuthenticationChallenge with a protection space that has an authentication method of type .NSURLAuthenticationMethodClientCertificate. In these cases, you need to evaluate the host of the challenge, then create an NSURLCredential using the credentialWithIdentity:certificates:persistence: API. By passing this credential off to the completion handler, the client certificate should be sent to the server to authenticate the connection. More info can be found here.
Client certificate authentication (NSURLAuthenticationMethodClientCertificate) requires the system identity and all certificates needed to authenticate with the server. Create an NSURLCredential object with credentialWithIdentity:certificates:persistence:.
I've never actually had a need to use this type of authentication before. You'll need to override the auth challenge SessionDelegate closure using the task override closure to get this working.
I have a problem verifiying a HTTPS endpoint when providing a specific certificate path to the 'verify' option; setting 'verify' to true DOES work however:
import requests
def run_tests():
url="https://www.google.com"
certfilename="google.crt"
generate_cert_file( certfilename )
response = requests.get( url, verify=False )
print "URL:%s, Verify=False. Result: %s"%(url, response.status_code )
response = requests.get( url, verify=True )
print "URL:%s, Verify=True. Result: %s"%(url, response.status_code )
response = requests.get( url, verify=certfilename )
print "URL:%s, Verify=%s. Result: %s"%(url, certfilename, response.status_code )
def generate_cert_file( filename ):
cert_text=('''\
-----BEGIN CERTIFICATE-----
MIIEgDCCA2igAwIBAgIIIxZ+YmNtB9AwDQYJKoZIhvcNAQELBQAwSTELMAkGA1UE
BhMCVVMxEzARBgNVBAoTCkdvb2dsZSBJbmMxJTAjBgNVBAMTHEdvb2dsZSBJbnRl
cm5ldCBBdXRob3JpdHkgRzIwHhcNMTUwOTA5MjI1MzM5WhcNMTUxMjA4MDAwMDAw
WjBoMQswCQYDVQQGEwJVUzETMBEGA1UECAwKQ2FsaWZvcm5pYTEWMBQGA1UEBwwN
TW91bnRhaW4gVmlldzETMBEGA1UECgwKR29vZ2xlIEluYzEXMBUGA1UEAwwOd3d3
Lmdvb2dsZS5jb20wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDqH1dz
jOr385g9DXVZowTLkhjCoYGQ/zOLzslSFowYqZnlMiE7iZ8Vm9/iJfhStSMoA6gZ
87uqLENCi3NK+pk4+n1SIPRZS0jorb1jsyenMMy6Fxb9sQe3ndqZpWBtsW5ezW9t
7q5HWA1cNm1KSi2fmwZpDrSaN0TYQdFbRAz6cPf6J/P66qyEC7OInFgLsy4FDdp8
itpjCo/gnk2gz1vpWjiRYLCkz/dlFY3M3kR/s7YcJa7UP7BZ+QmmDfN3y4mxmEfn
Cg8rB25ZbD+oQ+Hua3/oMrx4r3lliou3yrD08/ABEqs16EEPX5wFHtI4CQYAUt5E
rEDl36bpvFD0QkfLAgMBAAGjggFLMIIBRzAdBgNVHSUEFjAUBggrBgEFBQcDAQYI
KwYBBQUHAwIwGQYDVR0RBBIwEIIOd3d3Lmdvb2dsZS5jb20waAYIKwYBBQUHAQEE
XDBaMCsGCCsGAQUFBzAChh9odHRwOi8vcGtpLmdvb2dsZS5jb20vR0lBRzIuY3J0
MCsGCCsGAQUFBzABhh9odHRwOi8vY2xpZW50czEuZ29vZ2xlLmNvbS9vY3NwMB0G
A1UdDgQWBBQHiMS+8X3+WcfbMQymf9yX9XA1yDAMBgNVHRMBAf8EAjAAMB8GA1Ud
IwQYMBaAFErdBhYbvPZotXb1gba7Yhq6WoEvMCEGA1UdIAQaMBgwDAYKKwYBBAHW
eQIFATAIBgZngQwBAgIwMAYDVR0fBCkwJzAloCOgIYYfaHR0cDovL3BraS5nb29n
bGUuY29tL0dJQUcyLmNybDANBgkqhkiG9w0BAQsFAAOCAQEAUvY5/JbZLfahjC4F
IZU0jVtGRBDa6tXLhCHAdwLHYLQdHn74oQ8Y1vxL0AYd7V2SBAgrDjCoK9bDQbQi
UZ7xwG9K2O75bS2qYc/OlZLJr0Gdfs71ZpQzlv14SGBXvwPuD6noj+hiZjqICd6t
3Rd5oIFiZvkhNDdN6Uag28rIfR8HECdlkbZNZeLZnyoIaxsprANvlEkY0wEbn1K9
kww3/SYKbdPJ8VQSUOtWgGsO1RPFaE4PP7hCg8Q062+mmCs0ZMQSvrzzv7JQsO5J
EkdG6691HdVA6z/rRGGX8E6assTnJLmVMxRayV+Do07KvywLONTInUWue8heHUSX
EHJZbg==
-----END CERTIFICATE-----\
''')
with open(filename, "wb") as output:
output.write(cert_text)
if __name__=='__main__':
run_tests()
Am I doing something wrong here ? (I embedded the cert inline to make the code easier to run without having to provide a separate cert file)
I'm fetched 'requests' down from the git repository - the newest version TAG in the history is V2.7.0, and the latest commit is "46ff1a9a543cc4d33541aa64c94f50f0a698736e"
EDIT: I actually had the wrong certificate here (thanks Steffen Ullrich for pointing this out) : but I have now confirmed I have the correct cert/endpoint: and I get the same error.
I retrieved the cert like this:
openssl s_client -connect www.google.com:443
And just copied the cert details into the python program.
The issue is actually happening for my own in-house systems as well - using self-signed certs (which is my real use-case).
Alternatively : where does the 'verify=True' option actually look for trusted certs/CAs ? (On Java it would be 'cacerts' - not sure what the equivalent here is for Python/requests ?).
I'm on a Windows platform here.
You are using the certificate which is only valid for www.google.co.uk, but access www.google.com. Thus the certificate can not match at all. And I'm not sure if using the host certificate instead of an issuer certificate (i.e. root CA or intermediate CA) will work at all.
Can you try this:
s = Session()
req = Request('POST', 'https://www.google.com')
prepped = s.prepare_request(req)
resp = s.send(prepped, verify=False, cert=CERT_PATH)
if resp.status_code == 200:...
Currently using a really simple Twisted NameVirtualHost coupled with some JSON config files to serve really basic content in one Site object. The resources being served by Twisted are all WSGI objects built in flask.
I was wondering on how to go about wrapping the connections to these domains with an SSLContext, since reactor.listenSSL takes one and only one context, it isn't readily apparent how to give each domain/subdomain it's own crt/key pair. Is there any way to set up named virtual hosting with ssl for each domain that doesn't require proxying? I can't find any Twisted examples that use NameVirtualHost with SSL, and they only thing I could get to work is hook on the reactor listening on port 443 with only one domain's context?
I was wondering if anyone has attempted this?
My simple server without any SSL processing:
https://github.com/DeaconDesperado/twsrv/blob/master/service.py
TLS (the name for the modern protocol which replaces SSL) only very recently supports the feature you're looking for. The feature is called Server Name Indication (or SNI). It is supported by modern browsers on modern platforms, but not some older but still widely used platforms (see the wikipedia page for a list of browsers with support).
Twisted has no specific, built-in support for this. However, it doesn't need any. pyOpenSSL, upon which Twisted's SSL support is based, does support SNI.
The set_tlsext_servername_callback pyOpenSSL API gives you the basic mechanism to build the behavior you want. This lets you define a callback which is given access to the server name requested by the client. At this point, you can specify the key/certificate pair you want to use for the connection. You can find an example demonstrating the use of this API in pyOpenSSL's examples directory.
Here's an excerpt from that example to give you the gist:
def pick_certificate(connection):
try:
key, cert = certificates[connection.get_servername()]
except KeyError:
pass
else:
new_context = Context(TLSv1_METHOD)
new_context.use_privatekey(key)
new_context.use_certificate(cert)
connection.set_context(new_context)
server_context = Context(TLSv1_METHOD)
server_context.set_tlsext_servername_callback(pick_certificate)
You can incorporate this approach into a customized context factory and then supply that context factory to the listenSSL call.
Just to add some closure to this one, and for future searches, here is the example code for the echo server from the examples that prints the SNI:
from twisted.internet import ssl, reactor
from twisted.internet.protocol import Factory, Protocol
class Echo(Protocol):
def dataReceived(self, data):
self.transport.write(data)
def pick_cert(connection):
print('Received SNI: ', connection.get_servername())
if __name__ == '__main__':
factory = Factory()
factory.protocol = Echo
with open("keys/ca.pem") as certAuthCertFile:
certAuthCert = ssl.Certificate.loadPEM(certAuthCertFile.read())
with open("keys/server.key") as keyFile:
with open("keys/server.crt") as certFile:
serverCert = ssl.PrivateCertificate.loadPEM(
keyFile.read() + certFile.read())
contextFactory = serverCert.options(certAuthCert)
ctx = contextFactory.getContext()
ctx.set_tlsext_servername_callback(pick_cert)
reactor.listenSSL(8000, factory, contextFactory)
reactor.run()
And because getting OpenSSL to work can always be tricky, here is the OpenSSL statement you can use to connect to it:
openssl s_client -connect localhost:8000 -servername hello_world -cert keys/client.crt -key keys/client.key
Running the above python code against pyOpenSSL==0.13, and then running the s_client command above, will print this to the screen:
('Received SNI: ', 'hello_world')
There is now a txsni project that takes care of finding the right certificates per request. https://github.com/glyph/txsni