OIDC SSO in flask SSL certificate verify failed - ssl

I have one flask powered app, I'm trying to enable OIDC SSO for this app. I opted for wso2 as the identity server. I have created a callback URL and added the needful things in both the Identity Server and the flask app as shown below. The app is able to flow through the credential logging page and after that, I'm getting an SSL certificate verification error.
My try:
I have tried by using self signed certificates and app.run(ssl_context='adhoc') didn't worked.
Code Snippet:
from flask import Flask, g
from flask_oidc import OpenIDConnect
# import ssl
logging.basicConfig(level=logging.DEBUG)
app = Flask(__name__)
app.config.update({
'SECRET_KEY': 'SomethingNotEntirelySecret',
'TESTING': True,
'DEBUG': True,
'OIDC_CLIENT_SECRETS': 'client_secrets.json',
'OIDC_ID_TOKEN_COOKIE_SECURE': False,
'OIDC_REQUIRE_VERIFIED_EMAIL': False,
})
oidc = OpenIDConnect(app)
#app.route('/private')
#oidc.require_login
def hello_me():
# import pdb;pdb.set_trace()
info = oidc.user_getinfo(['email', 'openid_id'])
return ('Hello, %s (%s)! Return' %
(info.get('email'), info.get('openid_id')))
if __name__ == '__main__':
# app.run(host='sanudev', debug=True)
# app.run(debug=True)
# app.run(ssl_context='adhoc')
app.run(ssl_context=('cert.pem', 'key.pem'))
# app.run(ssl_context=('cert.pem', 'key.pem'))
Client Info:
{
"web": {
"auth_uri": "https://localhost:9443/oauth2/authorize",
"client_id": "hXCcX_N75aIygBIY7IwnWRtRpGwa",
"client_secret": "8uMLQ92Pm8_dPEjmGSoGF7Y6fn8a",
"redirect_uris": [
"https://sanudev:5000/oidc_callback"
],
"userinfo_uri": "https://localhost:9443/oauth2/userinfo",
"token_uri": "https://localhost:9443/oauth2/token",
"token_introspection_uri": "https://localhost:9443/oauth2/introspect"
}
}
App Info:
python 3.8
Flask 1.1.2

Hi Answering my own question just to reach the community effectively, here I can express where did I stuck and all the stories behind the fix.
TLDR:
The SSL issue was appearing because in OIDC flow wso2 server has to communicate or transfer secure-auth token only through the SSL tunnel. This is a mandatory standard need to keep for security purposes.
Yes carbon server has SSL certificate (self signed one) to make the secure token transfer through SSL Tunnel client also has to make at least self-signed certificate configuration.
Since I was using the flask-oidc library there is a provision to allow that, please refer to the configuration here.
{
"web": {
"auth_uri": "https://localhost:9443/oauth2/authorize",
"client_id": "someid",
"client_secret": "somesecret",
"redirect_uris": [
"https://localhost:5000/oidc_callback"
],
"userinfo_uri": "http://localhost:9763/oauth2/userinfo",
"token_uri": "http://localhost:9763/oauth2/token",
"token_introspection_uri": "http://localhost:9763/oauth2/introspect",
"issuer": "https://localhost:9443/oauth2/token" # This can solve your issue
}
}
For quick development purpose you can enable Secure connection in HTTPS by adding ad-hoc config in flask app run settings.
if __name__ == '__main__':
# app.run(ssl_context=('cert.pem', 'key.pem')) # for self signed cert
app.run(debug=True, ssl_context='adhoc') # Adhoc way of making https

Let me preface this answer with this one Caveat:
DO NOT DO THIS IN PRODUCTION ENVIRONMENTS
No, serously, do not do this in production, this should only be done for development purposes.
Anyways, open the oauth2client\transport.py file.
You're going to see this file location in your error that is spit out. for me it was in my anaconda env
AppData\Local\Continuum\anaconda3\envs\conda_env\lib\site-packages\oauth2client\transport.py
Find this line (line 73 for me)
def get_http_object(*args, **kwargs):
"""Return a new HTTP object.
Args:
*args: tuple, The positional arguments to be passed when
contructing a new HTTP object.
**kwargs: dict, The keyword arguments to be passed when
contructing a new HTTP object.
Returns:
httplib2.Http, an HTTP object.
"""
return httplib2.Http(*args, **kwargs)
change the return to
return httplib2.Http(*args, **kwargs, disable_ssl_certificate_validation=True)
You may need to do the same thing to line 445 of flask_oidc/__init__.py
credentials.refresh(httplib2.Http(disable_ssl_certificate_validation=True))

Upgrading certifi package should solve the problem.
pip install --upgrade certifi
It worked for me when I faced the exactly the same issue.

Related

Python's default SSL certificate context not working in requests method when behind proxy, works fine otherwise

I have the below function in my code, which works perfectly fine when I'm not behind any proxy. In fact, without even mentioning the certifi default CA certificate, it works fine if I pass verify=TRUE, I guess, because it works in the same way.
def reverse_lookup(lat, long):
cafile=certifi.where()
params={'lat' : float(lat), 'lon' : float(long), 'format' : 'json',
'accept-language' : 'en', 'addressdetails' : 1}
response = requests.get("https://nominatim.openstreetmap.org/reverse", params=params, verify=cafile)
#response = requests.get("https://nominatim.openstreetmap.org/reverse", params=params, verify=True) <-- this works as well
result = json.loads(response.text)
return result['address']['country'], result['address']['state'], result['address']['city']
When I run the same code from within my enterprise infrastructure (where I'm behind proxy), I make some minor changes in the code mentioning the proxy as parameter in requests method:
def reverse_lookup(lat, long):
cafile=certifi.where()
proxies = {"https" : "https://myproxy.com"}
params={'lat' : float(lat), 'lon' : float(long), 'format' : 'json',
'accept-language' : 'en', 'addressdetails' : 1}
response = requests.get("https://nominatim.openstreetmap.org/reverse", params=params, verify=cafile, proxies=proxies)
result = json.loads(response.text)
return result['address']['country'], result['address']['state'], result['address']['city']
But it gives me one out of these 3 SSL errors at different times, if I set verify=True or verify=certifi.where():
CERTIFICATE_VERIFY_FAILED
UNKNOWN_PROTOCOL
WRONG_VERSION_NUMBER
Only time it works is when I completely bypass the SSL verification with verify=False
My questions are:
Since I'm sending the https request via proxy, is it ok if I bypass SSL verification ?
How to make the default context of SSL verification work in this case, when I'm behind proxy ?
Any help is appreciated. Code tested in both Python 2.7.15 and 3.9
Since I'm sending the https request via proxy, is it ok if I bypass SSL verification ?
Do you need the protection offered by HTTPS, i.e. encryption of the application data (like passwords, but also the full URL) to protect against sniffing or modifications by a malicious man in the middle? If you don't need the protection, then you can bypass certificate validation.
How to make the default context of SSL verification work in this case, when I'm behind proxy ?
The proxy is doing SSL interception and when doing this issues a new certificate for this site based on an internal CA. If this is expected (i.e. not an attack) then you need to import the CA from the proxy as trusted with verify='proxy-ca.pem'. Your IT department should be able to provide you with the proxy CA.
But it gives me one out of these 3 SSL errors at different times, if I
set verify=True or verify=certifi.where():
CERTIFICATE_VERIFY_FAILED
UNKNOWN_PROTOCOL
WRONG_VERSION_NUMBER
It should only give you CERTIFICATE_VERIFY_FAILED. The two other errors indicate wrong proxy settings, typically setting https_proxy to https://... instead of http://... (which also can be seen in your code).

SSL Certification Verify Failed on Heroku Redis

I'm deploying a flask app on Heroku using a Redis premium plan. I get the following error: 'SSL Certification Verify Failed'. Attempted fixes:
Downgrading to Redis 5
Passing ssl_cert_reqs=None to the Redis constructor in redis-py
A solution to this problem could be:
Explain how to disable TLS certification on heroku redis premium plans
Explain how to make TLS certification work on heroku redis premium plans
From Heroku's docs, this may be a hint: 'you must enable TLS in your Redis client’s configuration in order to connect to a Redis 6 database'. I don't understand what this means.
I solved my problem by adding ?ssl_cert_reqs=CERT_NONE to the end of REDIS_URL in my Heroku config.
You can disable TLS certification on Heroku by downgrading to Redis 5 and passing ssl_cert_reqs=None to the Redis constructor.
$ heroku addons:create heroku-redis:premium-0 --version 5
from redis import ConnectionPool, Redis
import os
connection_pool = ConnectionPool.from_url(os.environ.get('REDIS_URL'))
app.redis = Redis(connection_pool=connection_pool, ssl_cert_reqs=None)
My mistake was not doing both at the same time.
An ideal solution would explain how to configure TLS certification for Redis 6.
The docs are actually incorrect, you have to set SSL to verify_none because TLS happens automatically.
From Heroku support:
"Our data infrastructure uses self-signed certificates so certificates
can be cycled regularly... you need to set the verify_mode
configuration variable to OpenSSL::SSL::VERIFY_NONE"
I solved this by setting the ssl_params to verify_none:
ssl_params: { verify_mode: OpenSSL::SSL::VERIFY_NONE } }
For me it was where I config redis (in a sidekiq initializer):
# config/initializers/sidekiq.rb
Sidekiq.configure_client do |config|
config.redis = { url: ENV['REDIS_URL'], size: 1, network_timeout: 5,
ssl_params: { verify_mode: OpenSSL::SSL::VERIFY_NONE } }
end
Sidekiq.configure_server do |config|
config.redis = { url: ENV['REDIS_URL'], size: 7, network_timeout: 5,
ssl_params: { verify_mode: OpenSSL::SSL::VERIFY_NONE } }
end
On Heroku (assuming Heroku Redis addon), the redis TLS route already has the ssl_cert_reqs param sorted out. A common oversight that can cause errors in cases like this on heroku is: using REDIS_URL over REDIS_TLS_URL.
Solution:
redis_url = os.environ.get('REDIS_TLS_URL')
This solution works with redis 6 and python on Heroku
import os, redis
redis_url = os.getenv('REDIS_URL')
redis_store = redis.from_url(redis_url, ssl_cert_reqs=None)
In my local development environment I do not use redis with the rediss scheme, so I use a function like this to allow work in both cases:
def get_redis_store():
'''
Get a connection pool to redis based on the url configured
on env variable REDIS_URL
Returns
-------
redis.ConnectionPool
'''
redis_url = os.getenv('REDIS_URL')
if redis_url.startswith('rediss://'):
redis_store = redis.from_url(
redis_url, ssl_cert_reqs=None)
else:
redis_store = redis.from_url(redis_url)
return redis_store
If using the django-rq wrapper and trying to deal with this, be sure to not use the URL parameter with SSL_CERTS_REQS. There is an outstanding issue that describes this all, but basically you need to specify each connection param instead of using the URL.

Unable to access unsecured https with selenium using browsermob proxy

I'm trying to embed browsermob proxy with my selenium (chrome) framework for UI automated testing in order to intercept responses and other networking.
Description :
Selenium webdriver using browsermob proxy and it works just fine - HTTP and secured HTTPS URL's are OK. When I'm trying to navigate to unsecured HTTPS URL I get this chrome error:
ERR_TUNNEL_CONNECTION_FAILED
Here's my python code:
class Browser(object):
display = None
browser = None
def __init__(self, implicitly_wait_seconds=10, is_visible=True, display_size=None, browser_name='chrome'):
if not is_visible:
self.display = Display(display_size)
self.server = Server('/home/erez/Downloads/browsermob-proxy-2.1.4/bin/browsermob-proxy')
self.server.start()
self.proxy = self.server.create_proxy()
self.capabilities = DesiredCapabilities.CHROME
self.proxy.add_to_capabilities(self.capabilities)
self.proxy.new_har("test", options={'captureHeaders': True, 'captureContent': True})
self.start_browser(display_size, implicitly_wait_seconds, browser_name)
def __enter__(self):
return self
def __exit__(self, _type, value, trace):
self.close()
def start_browser(self, display_size, implicitly_wait_seconds=10, browser_name='chrome'):
if browser_name == 'chrome':
chrome_options = Options()
# chrome_options.add_argument("--disable-extensions")
chrome_options.add_experimental_option("excludeSwitches", ["ignore-certificate-errors"])
chrome_options.add_argument("--ssl-version-max")
chrome_options.add_argument("--start-maximized")
chrome_options.add_argument('--proxy-server=%s' % self.proxy.proxy)
chrome_options.add_argument('--ignore-certificate-errors')
chrome_options.add_argument('--allow-insecure-localhost')
chrome_options.add_argument('--ignore-urlfetcher-cert-requests')
self.browser = webdriver.Chrome(os.path.dirname(os.path.realpath(__file__)) + "/chromedriver",
chrome_options=chrome_options, desired_capabilities=self.capabilities)
self.browser.implicitly_wait(implicitly_wait_seconds)
You can also call create_proxy with trustAllServers as argument:
self.proxy = self.server.create_proxy(params={'trustAllServers':'true'})
I faced the same problem for SSL proxying using BrowserMob Proxy. For this you have to install a certificate in your browser that has been defined in this link
Go to the bottom of the document, and you will see a section called "SSL support". Install the ca-certificate-rsa.cer in your browser and there would be no issue in SSL proxying anymore.
If installing Browsermobs/test servers certificates won't do the job, like in my case, not the most elegant way, but gets the job done:
You are able to bypass ERR_TUNNEL_CONNECTION_FAILED error by passing a trustAllServers-parameter to the proxy instance, which will disable the certificate verification for upstream servers. Unfortunately, for as far as I know, this functionality has not been implemented in the Browsermobs Python wrapper.
However, you can start the proxy with the parameter via Browsermobs REST API (see section REST API # https://github.com/lightbody/browsermob-proxy/blob/master/README.md). In case of Python, Requests might be the way to go.
Here's a snippet:
import json
import requests
from browsermobproxy import Server
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
# Start the proxy via python
server = Server('/path_to/bin/browsermob-proxy')
server.start()
# Start proxy instance with trustAllServers set to true, store returned port number
r = requests.post('http://localhost:8080/proxy', data = {'trustAllServers':'true'})
port = json.loads(r.text)['port']
# Start Chromedriver, pass on the proxy
chrome_options = Options()
chrome_options.add_argument('--proxy-server=localhost:%s' % port)
driver = webdriver.Chrome('/path_to/selenium/chromedriver', chrome_options=chrome_options)
# Get a site with untrusted cert
driver.get('https://site_with_untrusted_cert')
Also, if you need to access HAR-data, you'll need to do that also trough the REST API:
requests.put('http://localhost:8080/proxy/%s/har' % port, data = {'captureContent':'true'})
r = requests.get('http://localhost:8080/proxy/%s/har' % port)
Of course, since you're disabling safety features, this parameter should be used for limited testing purposes only.
Try use this
self.capabilities['acceptSslCerts'] = True

Download file over HTTPS and SSO in groovy avoiding server certificate validation

I want to download a file in groovy over a connection that is both using single sign on (SSO) over HTTPS (SSL) is there an easy way to do this. I'm not intending to build a full blown application so security is not as much of a concern.
def data = new URL("https://server/context/servlet?param1=value1").getText()
print data
I currently do the download using curl but would ideally not have to call curl. current used call below.
curl --negotiate -u user:pass -L --insecure -o filename.txt "https://server/context/servlet?param1=value1"
Two key points to the solution i'm looking for
- It does not involve making a system call to curl
- It does not include manually setting up a certificate.
Would consider libraries.
To avoid the SSL PKIX validation check, in Groovy, you can implement a X509TrustManager in the same way that you do it in Java.
Note that this disable the validation server certificate validation, therefore it's a security risk:
import javax.net.ssl.*
// create a TrustManager to avoid PKIX path validation
def trustManager = [
checkClientTrusted: { chain, authType -> },
checkServerTrusted: { chain, authType -> },
getAcceptedIssuers: { null }
] as X509TrustManager
// creat a sslCtx to use "lax" trustManager
def context = SSLContext.getInstance("TLS")
context.init(null, [trustManager] as TrustManager[], null)
// set as default ssl context
HttpsURLConnection.setDefaultSSLSocketFactory(context.getSocketFactory())
// finally you can connect to the server "insecurely"
def data = new URL("https://server/context/servlet?param1=value1").getText()
print data
About your second question, to provide a basic authentication like curl does with --user argument, you can set a default user/password for your connections using Authenticator class:
Authenticator.setDefault (new Authenticator() {
protected PasswordAuthentication getPasswordAuthentication() {
return new PasswordAuthentication ("user", "pass".toCharArray())
}
})
Note that is possible to do so on other ways in Groovy using some libraries, but this is a possible way using standard Java classes.

2-way SSL with CherryPy

From CherryPy 3.0 and onwards, one-way SSL can be turned on simply by pointing to the server certificate and private key, like this:
import cherrypy
class HelloWorld(object):
def index(self):
return "Hello SSL World!"
index.exposed = True
cherrypy.server.ssl_certificate = "keys/server.crt"
cherrypy.server.ssl_private_key = "keys/server.crtkey"
cherrypy.quickstart(HelloWorld())
This enables clients to validate the server's authenticity. Does anyone know whether CherryPy supports 2-way ssl, e.g. where the server can also check client authenticity by validating a client certificate?
If yes, could anyone give an example how is that done? Or post a reference to an example?
It doesn't out of the box. You'd have to patch the wsgiserver to provide that feature. There is a ticket (and patches) in progress at http://www.cherrypy.org/ticket/1001.
I have been looking for the same thing. I know there are some patches on the CherryPy site.
I also found the following at CherryPy SSL Client Authentication. I haven't compared this vs the CherryPy patches but maybe the info will be helpful.
We recently needed to develop a quick
but resilient REST application and
found that CherryPy suited our needs
better than other Python networking
frameworks, like Twisted.
Unfortunately, its simplicity lacked a
key feature we needed, Server/Client
SSL certificate validation. Therefore
we spent a few hours writing a few
quick modifications to the current
release, 3.1.2. The following code
snippets are the modifications we
made:
cherrypy/_cpserver.py
## -55,7 +55,6 ## instance = None ssl_certificate = None ssl_private_key
= None
+ ssl_ca_certificate = None nodelay = True
def __init__(self):
cherrypy/wsgiserver/__init__.py
## -1480,6 +1480,7 ##
# Paths to certificate and private key files ssl_certificate = None ssl_private_key = None
+ ssl_ca_certificate = None
def __init__(self, bind_addr, wsgi_app, numthreads=10, server_name=None, max=-1, request_queue_size=5, timeout=10, shutdown_timeout=5):
## -1619,7 +1620,9 ##
self.socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) if self.nodelay: self.socket.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)
- if self.ssl_certificate and self.ssl_private_key:
+ if self.ssl_certificate and self.ssl_private_key and \
+ self.ssl_ca_certificate:
+ if SSL is None: raise ImportError("You must install pyOpenSSL to use HTTPS.")
## -1627,6 +1630,11 ## ctx = SSL.Context(SSL.SSLv23_METHOD) ctx.use_privatekey_file(self.ssl_private_key) ctx.use_certificate_file(self.ssl_certificate)
+ x509 = crypto.load_certificate(crypto.FILETYPE_PEM,
+ open(self.ssl_ca_certificate).read())
+ store = ctx.get_cert_store()
+ store.add_cert(x509)
+ ctx.set_verify(SSL.VERIFY_PEER | SSL.VERIFY_FAIL_IF_NO_PEER_CERT, lambda *x:True) self.socket = SSLConnection(ctx, self.socket) self.populate_ssl_environ()
The above patches require the
inclusion of a new configuration
option inside of the CherryPy server
configuration,
server.ssl_ca_certificate. This
option identifies the certificate
authority file that connecting clients
will be validated against, if the
client does not present a valid client
certificate it will close the
connection immediately.
Our solution has advantages and
disadvantages, the primary advantage
being if the connecting client doesn’t
present a valid certificate it’s
connection is immediately closed.
This is good for security concerns as
it does not permit the client any
access into the CherryPy application
stack. However, since the restriction
is done at the socket level the
CherryPy application can never see the
client connecting and hence the
solution is somewhat inflexible.
An optimal solution would allow the
client to connect to the CherryPy
socket and send the client certificate
up into the application stack. Then a
custom CherryPy Tool would validate
the certificate inside of the
application stack and close the
connection if necessary; unfortunately
because of the structure of CherryPy’s
pyOpenSSL implementation it is
difficult to retrieve the client
certificate inside of the application
stack.
Of course the patches above should
only be used at your own risk. If you
come up with a better solution please
let us know.
If the current version of CherryPy does not support client certificate verification, it is possible to configure CherryPy to listen to 127.0.0.1:80, install HAProxy to listen to 443 and verify client side certificates and to forward traffic to 127.0.0.1:80
HAProxy is simple, light, fast and reliable.
An example of HAProxy configuration