SSL certificate validation for a proxy connection using python requests - ssl

I'm trying to connect to a site, e.g www.yahoo.com via proxy using python requests library. So, I define proxy settings as:
HOST = 'host'
PORT = 1234 # random port I have used here
USER = 'user'
PASS = 'password'
PROXY = "%s:%s#%s:%d" % (USER, PASS, HOST, PORT)
PROXY_DICT = {
"http" : 'http://' + PROXY,
"https" : 'https://' + PROXY,
}
I use the following line of code:
requests.get('http://www.yahoo.com', proxies=proxy_dict)
This doesn't raise an exception but the response text is an error page from the proxy saying "Ensure you have installed the certificate". I have a certificate "certificate.crt", which runs fine when used with chrome browser. And the certificate is self-signed. I have tried a couple of things which raise errors.
When used the crt file as a verify param, following error:
SSLError: [Errno bad ca_certs: 'certificate.crt'] []
When used the crt file as a cert param, following error:
Error: [('PEM routines', 'PEM_read_bio', 'no start line'), ('SSL routines', 'SSL_CTX_use_certificate_file', 'PEM lib')]
I managed to get a .pem file(I'm not sure but, it might have been generated using a key and a crt file) as well. When using it with cert param, it doesn't throw error, but the response text is again having the text "...Ensure that the certificate is installed..."
When used .pem file with verify param, following error:
SSLError: [Errno bad handshake] [('SSL routines', 'SSL3_GET_SERVER_CERTIFICATE', 'certificate verify failed')]
Now when I refer to requests docs, I see I can use two parameters verify and cert. What shall I use here? And how?

Related

Python's default SSL certificate context not working in requests method when behind proxy, works fine otherwise

I have the below function in my code, which works perfectly fine when I'm not behind any proxy. In fact, without even mentioning the certifi default CA certificate, it works fine if I pass verify=TRUE, I guess, because it works in the same way.
def reverse_lookup(lat, long):
cafile=certifi.where()
params={'lat' : float(lat), 'lon' : float(long), 'format' : 'json',
'accept-language' : 'en', 'addressdetails' : 1}
response = requests.get("https://nominatim.openstreetmap.org/reverse", params=params, verify=cafile)
#response = requests.get("https://nominatim.openstreetmap.org/reverse", params=params, verify=True) <-- this works as well
result = json.loads(response.text)
return result['address']['country'], result['address']['state'], result['address']['city']
When I run the same code from within my enterprise infrastructure (where I'm behind proxy), I make some minor changes in the code mentioning the proxy as parameter in requests method:
def reverse_lookup(lat, long):
cafile=certifi.where()
proxies = {"https" : "https://myproxy.com"}
params={'lat' : float(lat), 'lon' : float(long), 'format' : 'json',
'accept-language' : 'en', 'addressdetails' : 1}
response = requests.get("https://nominatim.openstreetmap.org/reverse", params=params, verify=cafile, proxies=proxies)
result = json.loads(response.text)
return result['address']['country'], result['address']['state'], result['address']['city']
But it gives me one out of these 3 SSL errors at different times, if I set verify=True or verify=certifi.where():
CERTIFICATE_VERIFY_FAILED
UNKNOWN_PROTOCOL
WRONG_VERSION_NUMBER
Only time it works is when I completely bypass the SSL verification with verify=False
My questions are:
Since I'm sending the https request via proxy, is it ok if I bypass SSL verification ?
How to make the default context of SSL verification work in this case, when I'm behind proxy ?
Any help is appreciated. Code tested in both Python 2.7.15 and 3.9
Since I'm sending the https request via proxy, is it ok if I bypass SSL verification ?
Do you need the protection offered by HTTPS, i.e. encryption of the application data (like passwords, but also the full URL) to protect against sniffing or modifications by a malicious man in the middle? If you don't need the protection, then you can bypass certificate validation.
How to make the default context of SSL verification work in this case, when I'm behind proxy ?
The proxy is doing SSL interception and when doing this issues a new certificate for this site based on an internal CA. If this is expected (i.e. not an attack) then you need to import the CA from the proxy as trusted with verify='proxy-ca.pem'. Your IT department should be able to provide you with the proxy CA.
But it gives me one out of these 3 SSL errors at different times, if I
set verify=True or verify=certifi.where():
CERTIFICATE_VERIFY_FAILED
UNKNOWN_PROTOCOL
WRONG_VERSION_NUMBER
It should only give you CERTIFICATE_VERIFY_FAILED. The two other errors indicate wrong proxy settings, typically setting https_proxy to https://... instead of http://... (which also can be seen in your code).

Identity Server 4. Getting error when using self-signed certificate

I have an Identity Server, a Web API and a frontend app. Usually they're running on localhost and it works fine. Now I need to run the app on my local IP address. I changed all the settings from localhost:port to <my_ip>:port and I'm getting the following error in the Web API:
System.InvalidOperationException: IDX20803: Unable to obtain configuration from: '<my_ip>:5003/.well-known/openid-configuration'.
---> System.IO.IOException: IDX20804: Unable to retrieve document from: '<my_ip>:5003/.well-known/openid-configuration'.
---> System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception.
---> System.Security.Authentication.AuthenticationException: The remote certificate is invalid according to the validation procedure.
I tried to generate a self signed certificate and add to the certificate store, I tried many solutions on SO i.e. this one. All I achieved is switching from this error and Keyset does not exist
This is how I try to load the certificate:
var key = Configuration["certBase64"]; // exported from store as x509 base64 encoded
var pfxBytes = Convert.FromBase64String(key);
var cert = new X509Certificate2(pfxBytes, "<certificate-pwd>", X509KeyStorageFlags.MachineKeySet);
builder.AddSigningCredential(cert);
What am I missing?
UPDATE
I also tried to use a local DNS name for my IP. On my router I set up forwarding from myapp.local to <my_ip>. Then I created a self signed certificate with Subject and DNS name = myapp.local, but it didn't help.
The current error:
System.InvalidOperationException: IDX20803: Unable to obtain configuration from: 'myapp.local:5003/.well-known/openid-configuration'.
---> System.IO.IOException: IDX20804: Unable to retrieve document from: 'myapp.local:5003/.well-known/openid-configuration'.
---> System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception.
---> System.Security.Authentication.AuthenticationException: The remote certificate is invalid according to the validation procedure.
You can not use HTTPS and certificates when you try co connect to a service using an IP-address. You must always have a domain name and a valid certificate that your local machine trust.

JMeter load client-side certificate

I try to add a p12 file to JMeter 3.3 configuration to reach a site.
I added following lines to system.properties file:
javax.net.ssl.keyStoreType=pkcs12
javax.net.ssl.keyStore=C:\certs\mycert.p12
javax.net.ssl.keyStorePassword=mypassword
After that I restarted JMeter, but got the same error, javax.net.ssl.SSLHandshakeException. I converted p12 file with keytool to jks and added the following lines to same file by replacing previous ones.
javax.net.ssl.keyStore=C:\certs\mycert.jks
javax.net.ssl.keyStorePassword=mypassword
Error message is the same in this case too:
Response code: Non HTTP response code:
javax.net.ssl.SSLHandshakeException Response message: Non HTTP
response message: Received fatal alert: handshake_failure
I use HTTP Client 4 as implementation for HTTP Samplers. When I import same certificate to browser, it is working correctly.
I tried to follow this tutorial too: How to configure JMeter to use client side SSL
You can use Options -> SSL Manager option, where you can select your .p12 file to be used in current Test plan.

Jmeter testing integration with IBM dtapower

Need your help in setting the SSL manager in Jmeter for performance testing with IBM datapower.
I tried the below steps to Add cert.
• Added (* .jks /*.p12 ) file in the jmeter GUI > Options > SSL Manager.
• I tried the setting the jks file in system.properties file too.
Path : *\jMETER\apache-jmeter-3.0\apache-jmeter-3.0\bin\system.properties
# Truststore properties (trusted certificates)
#javax.net.ssl.trustStore=/path/to/[jsse]cacerts
#javax.net.ssl.trustStorePassword
#javax.net.ssl.trustStoreProvider
#javax.net.ssl.trustStoreType [default = KeyStore.getDefaultType()]
# Keystore properties (client certificates)
# Location
javax.net.ssl.keyStore=****.jks -- Added
#
#The password to your keystore
javax.net.ssl.keyStorePassword=****-- Added
#
#javax.net.ssl.keyStoreProvider
#javax.net.ssl.keyStoreType [default = KeyStore.getDefaultType()]
I dont see the SSL handshake jMETER and datapower even after i followed ablove steps. Getting below error from datapower.
12:47:26 AM ssl error 51751363 10.123.98.73 0x806000ca valcred (###_CVC_Reverse_Server): SSL Proxy Profile '###_SSLPP_Reverse_Server': connection error: peer did not send a certificate
12:47:26 AM mpgw error 51751363 10.123.98.73 0x80e00161 source-https (###_HTTPS_FSH_CON_****): Request processing failed: Connection terminated before request headers read because of the connection error occurs, from URL: 10.123.98.73:58394
12:47:26 AM ssl error 51751363 10.123.98.73 0x8120002f sslproxy (####_SSLPP_Reverse_Server): SSL library error: error:140890C7:SSL routines:SSL3_GET_CLIENT_CERTIFICATE:peer did not
Can you please advice how to send the cert(.jks/ .p12) file from jmeter.
Change "Implementation" of your HTTP Request sampler(s) to Java. The fastest and the easiest way of doing this is using HTTP Request Defaults.
If you're using .p12 keystores you will need an extra line in the system.properties file like:
javax.net.ssl.keyStoreType=pkcs12
JMeter restart is required to pick the properties up.
See How to Set Your JMeter Load Test to Use Client Side Certificates article for more information.

Roku https request failed

I have the below code to get data from my https website having godaddy certificate.but i unable to get the result from the url.
port = CreateObject("roMessagePort")
req = createObject("roUrlTransfer")
req.SetMessagePort(port)
req.setCertificatesFile("pkg:/source/domain.crt")
req.AddHeader("X-Roku-Reserved-Dev-Id", "")
req.InitClientCertificates()
req.setUrl("https://domain.com/index.php/roku/getdata")
response = req.getToString()
print "response: "; response
Note: i edited /etc/apache2/sites-available/default-ssl.conf file to add godaddy certificate.in my website,https is working.i alwaise getting error code -77.i exported the crt file from mozilla firefox.is this a correct way?
No idea about the correct format of certificate export. But have you tried:
req.SetCertificatesFile("common:/certs/ca-bundle.crt")