I'm trying to embed browsermob proxy with my selenium (chrome) framework for UI automated testing in order to intercept responses and other networking.
Description :
Selenium webdriver using browsermob proxy and it works just fine - HTTP and secured HTTPS URL's are OK. When I'm trying to navigate to unsecured HTTPS URL I get this chrome error:
ERR_TUNNEL_CONNECTION_FAILED
Here's my python code:
class Browser(object):
display = None
browser = None
def __init__(self, implicitly_wait_seconds=10, is_visible=True, display_size=None, browser_name='chrome'):
if not is_visible:
self.display = Display(display_size)
self.server = Server('/home/erez/Downloads/browsermob-proxy-2.1.4/bin/browsermob-proxy')
self.server.start()
self.proxy = self.server.create_proxy()
self.capabilities = DesiredCapabilities.CHROME
self.proxy.add_to_capabilities(self.capabilities)
self.proxy.new_har("test", options={'captureHeaders': True, 'captureContent': True})
self.start_browser(display_size, implicitly_wait_seconds, browser_name)
def __enter__(self):
return self
def __exit__(self, _type, value, trace):
self.close()
def start_browser(self, display_size, implicitly_wait_seconds=10, browser_name='chrome'):
if browser_name == 'chrome':
chrome_options = Options()
# chrome_options.add_argument("--disable-extensions")
chrome_options.add_experimental_option("excludeSwitches", ["ignore-certificate-errors"])
chrome_options.add_argument("--ssl-version-max")
chrome_options.add_argument("--start-maximized")
chrome_options.add_argument('--proxy-server=%s' % self.proxy.proxy)
chrome_options.add_argument('--ignore-certificate-errors')
chrome_options.add_argument('--allow-insecure-localhost')
chrome_options.add_argument('--ignore-urlfetcher-cert-requests')
self.browser = webdriver.Chrome(os.path.dirname(os.path.realpath(__file__)) + "/chromedriver",
chrome_options=chrome_options, desired_capabilities=self.capabilities)
self.browser.implicitly_wait(implicitly_wait_seconds)
You can also call create_proxy with trustAllServers as argument:
self.proxy = self.server.create_proxy(params={'trustAllServers':'true'})
I faced the same problem for SSL proxying using BrowserMob Proxy. For this you have to install a certificate in your browser that has been defined in this link
Go to the bottom of the document, and you will see a section called "SSL support". Install the ca-certificate-rsa.cer in your browser and there would be no issue in SSL proxying anymore.
If installing Browsermobs/test servers certificates won't do the job, like in my case, not the most elegant way, but gets the job done:
You are able to bypass ERR_TUNNEL_CONNECTION_FAILED error by passing a trustAllServers-parameter to the proxy instance, which will disable the certificate verification for upstream servers. Unfortunately, for as far as I know, this functionality has not been implemented in the Browsermobs Python wrapper.
However, you can start the proxy with the parameter via Browsermobs REST API (see section REST API # https://github.com/lightbody/browsermob-proxy/blob/master/README.md). In case of Python, Requests might be the way to go.
Here's a snippet:
import json
import requests
from browsermobproxy import Server
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
# Start the proxy via python
server = Server('/path_to/bin/browsermob-proxy')
server.start()
# Start proxy instance with trustAllServers set to true, store returned port number
r = requests.post('http://localhost:8080/proxy', data = {'trustAllServers':'true'})
port = json.loads(r.text)['port']
# Start Chromedriver, pass on the proxy
chrome_options = Options()
chrome_options.add_argument('--proxy-server=localhost:%s' % port)
driver = webdriver.Chrome('/path_to/selenium/chromedriver', chrome_options=chrome_options)
# Get a site with untrusted cert
driver.get('https://site_with_untrusted_cert')
Also, if you need to access HAR-data, you'll need to do that also trough the REST API:
requests.put('http://localhost:8080/proxy/%s/har' % port, data = {'captureContent':'true'})
r = requests.get('http://localhost:8080/proxy/%s/har' % port)
Of course, since you're disabling safety features, this parameter should be used for limited testing purposes only.
Try use this
self.capabilities['acceptSslCerts'] = True
Related
I have the below function in my code, which works perfectly fine when I'm not behind any proxy. In fact, without even mentioning the certifi default CA certificate, it works fine if I pass verify=TRUE, I guess, because it works in the same way.
def reverse_lookup(lat, long):
cafile=certifi.where()
params={'lat' : float(lat), 'lon' : float(long), 'format' : 'json',
'accept-language' : 'en', 'addressdetails' : 1}
response = requests.get("https://nominatim.openstreetmap.org/reverse", params=params, verify=cafile)
#response = requests.get("https://nominatim.openstreetmap.org/reverse", params=params, verify=True) <-- this works as well
result = json.loads(response.text)
return result['address']['country'], result['address']['state'], result['address']['city']
When I run the same code from within my enterprise infrastructure (where I'm behind proxy), I make some minor changes in the code mentioning the proxy as parameter in requests method:
def reverse_lookup(lat, long):
cafile=certifi.where()
proxies = {"https" : "https://myproxy.com"}
params={'lat' : float(lat), 'lon' : float(long), 'format' : 'json',
'accept-language' : 'en', 'addressdetails' : 1}
response = requests.get("https://nominatim.openstreetmap.org/reverse", params=params, verify=cafile, proxies=proxies)
result = json.loads(response.text)
return result['address']['country'], result['address']['state'], result['address']['city']
But it gives me one out of these 3 SSL errors at different times, if I set verify=True or verify=certifi.where():
CERTIFICATE_VERIFY_FAILED
UNKNOWN_PROTOCOL
WRONG_VERSION_NUMBER
Only time it works is when I completely bypass the SSL verification with verify=False
My questions are:
Since I'm sending the https request via proxy, is it ok if I bypass SSL verification ?
How to make the default context of SSL verification work in this case, when I'm behind proxy ?
Any help is appreciated. Code tested in both Python 2.7.15 and 3.9
Since I'm sending the https request via proxy, is it ok if I bypass SSL verification ?
Do you need the protection offered by HTTPS, i.e. encryption of the application data (like passwords, but also the full URL) to protect against sniffing or modifications by a malicious man in the middle? If you don't need the protection, then you can bypass certificate validation.
How to make the default context of SSL verification work in this case, when I'm behind proxy ?
The proxy is doing SSL interception and when doing this issues a new certificate for this site based on an internal CA. If this is expected (i.e. not an attack) then you need to import the CA from the proxy as trusted with verify='proxy-ca.pem'. Your IT department should be able to provide you with the proxy CA.
But it gives me one out of these 3 SSL errors at different times, if I
set verify=True or verify=certifi.where():
CERTIFICATE_VERIFY_FAILED
UNKNOWN_PROTOCOL
WRONG_VERSION_NUMBER
It should only give you CERTIFICATE_VERIFY_FAILED. The two other errors indicate wrong proxy settings, typically setting https_proxy to https://... instead of http://... (which also can be seen in your code).
I have one flask powered app, I'm trying to enable OIDC SSO for this app. I opted for wso2 as the identity server. I have created a callback URL and added the needful things in both the Identity Server and the flask app as shown below. The app is able to flow through the credential logging page and after that, I'm getting an SSL certificate verification error.
My try:
I have tried by using self signed certificates and app.run(ssl_context='adhoc') didn't worked.
Code Snippet:
from flask import Flask, g
from flask_oidc import OpenIDConnect
# import ssl
logging.basicConfig(level=logging.DEBUG)
app = Flask(__name__)
app.config.update({
'SECRET_KEY': 'SomethingNotEntirelySecret',
'TESTING': True,
'DEBUG': True,
'OIDC_CLIENT_SECRETS': 'client_secrets.json',
'OIDC_ID_TOKEN_COOKIE_SECURE': False,
'OIDC_REQUIRE_VERIFIED_EMAIL': False,
})
oidc = OpenIDConnect(app)
#app.route('/private')
#oidc.require_login
def hello_me():
# import pdb;pdb.set_trace()
info = oidc.user_getinfo(['email', 'openid_id'])
return ('Hello, %s (%s)! Return' %
(info.get('email'), info.get('openid_id')))
if __name__ == '__main__':
# app.run(host='sanudev', debug=True)
# app.run(debug=True)
# app.run(ssl_context='adhoc')
app.run(ssl_context=('cert.pem', 'key.pem'))
# app.run(ssl_context=('cert.pem', 'key.pem'))
Client Info:
{
"web": {
"auth_uri": "https://localhost:9443/oauth2/authorize",
"client_id": "hXCcX_N75aIygBIY7IwnWRtRpGwa",
"client_secret": "8uMLQ92Pm8_dPEjmGSoGF7Y6fn8a",
"redirect_uris": [
"https://sanudev:5000/oidc_callback"
],
"userinfo_uri": "https://localhost:9443/oauth2/userinfo",
"token_uri": "https://localhost:9443/oauth2/token",
"token_introspection_uri": "https://localhost:9443/oauth2/introspect"
}
}
App Info:
python 3.8
Flask 1.1.2
Hi Answering my own question just to reach the community effectively, here I can express where did I stuck and all the stories behind the fix.
TLDR:
The SSL issue was appearing because in OIDC flow wso2 server has to communicate or transfer secure-auth token only through the SSL tunnel. This is a mandatory standard need to keep for security purposes.
Yes carbon server has SSL certificate (self signed one) to make the secure token transfer through SSL Tunnel client also has to make at least self-signed certificate configuration.
Since I was using the flask-oidc library there is a provision to allow that, please refer to the configuration here.
{
"web": {
"auth_uri": "https://localhost:9443/oauth2/authorize",
"client_id": "someid",
"client_secret": "somesecret",
"redirect_uris": [
"https://localhost:5000/oidc_callback"
],
"userinfo_uri": "http://localhost:9763/oauth2/userinfo",
"token_uri": "http://localhost:9763/oauth2/token",
"token_introspection_uri": "http://localhost:9763/oauth2/introspect",
"issuer": "https://localhost:9443/oauth2/token" # This can solve your issue
}
}
For quick development purpose you can enable Secure connection in HTTPS by adding ad-hoc config in flask app run settings.
if __name__ == '__main__':
# app.run(ssl_context=('cert.pem', 'key.pem')) # for self signed cert
app.run(debug=True, ssl_context='adhoc') # Adhoc way of making https
Let me preface this answer with this one Caveat:
DO NOT DO THIS IN PRODUCTION ENVIRONMENTS
No, serously, do not do this in production, this should only be done for development purposes.
Anyways, open the oauth2client\transport.py file.
You're going to see this file location in your error that is spit out. for me it was in my anaconda env
AppData\Local\Continuum\anaconda3\envs\conda_env\lib\site-packages\oauth2client\transport.py
Find this line (line 73 for me)
def get_http_object(*args, **kwargs):
"""Return a new HTTP object.
Args:
*args: tuple, The positional arguments to be passed when
contructing a new HTTP object.
**kwargs: dict, The keyword arguments to be passed when
contructing a new HTTP object.
Returns:
httplib2.Http, an HTTP object.
"""
return httplib2.Http(*args, **kwargs)
change the return to
return httplib2.Http(*args, **kwargs, disable_ssl_certificate_validation=True)
You may need to do the same thing to line 445 of flask_oidc/__init__.py
credentials.refresh(httplib2.Http(disable_ssl_certificate_validation=True))
Upgrading certifi package should solve the problem.
pip install --upgrade certifi
It worked for me when I faced the exactly the same issue.
I'm more than a day look how to deploy telegram bot with webhook instead of polling..
even in the official doc it's not work for me https://github.com/python-telegram-bot/python-telegram-bot/wiki/Webhooks
someone can explain me oh give me some link to work tutorial how to deploy the most basic bot like this one
#!/usr/bin/env python
# -*- coding: utf-8 -*-
import os
from telegram.ext import Updater, CommandHandler, MessageHandler, Filters
def start(update, context):
"""Send a message when the command /start is issued."""
update.message.reply_text('Hi!')
def help_command(update, context):
"""Send a message when the command /help is issued."""
update.message.reply_text('Help!')
def echo(update, context):
"""Echo the user message."""
update.message.reply_text(update.message.text)
def main():
TOKEN = "telegramToken"
updater = Updater(TOKEN, use_context=True)
# Get the dispatcher to register handlers
dp = updater.dispatcher
# on different commands - answer in Telegram
dp.add_handler(CommandHandler("start", start))
dp.add_handler(CommandHandler("help", help_command))
dp.add_handler(MessageHandler(Filters.text & ~Filters.command, echo))
PORT = int(os.environ.get('PORT', '5000'))
SERVER = 'myipserver/'
CERT = 'cert.pem'
updater.bot.setWebhook(SERVER + TOKEN, certificate=open(CERT, 'rb'))
# updater.bot.setWebhook(SERVER + TOKEN) also dont working
updater.start_webhook(listen="0.0.0.0", port=PORT, url_path=TOKEN)
# updater.start_polling()
updater.idle()
if __name__ == '__main__':
main()
The problem is that the default port in App Engine is 8080
And the webhook in telegram does not support this port (just ports 443, 80, 88, or 8443)
https://core.telegram.org/bots/webhooks
I run my Telegram BOTs on Heroku and normally start the webhook then is set it:
updater.start_webhook(listen="0.0.0.0",
port=int(os.environ.get("PORT", 5000)),
url_path='token'
updater.bot.setWebhook("https://myapp.com/token")
updater.idle()
When deploying on Heroku you get a URL for your application using SSL (https://myapp.herokuapp.com), which you then configure via the BotFather.
Providing the BotFather with a HTTPS url is a must, I wonder if you don't do that or maybe there is an issue with the SSL certificate you are using (self-signed?)
I want to implement a proxy server that intercepts both http and https requests. I came across libmproxy (http://mitmproxy.org/doc/scripting/libmproxy.html) that it is SSL-capable. I start with this simplest proxy that just prints the headers of all requests and responses, and forwards them to clients and servers normally.
#!/usr/bin/env python
from libmproxy import controller, proxy
import os
class Master(controller.Master):
def __init__(self, server):
controller.Master.__init__(self, server)
self.stickyhosts = {}
def run(self):
try:
return controller.Master.run(self)
except KeyboardInterrupt:
self.shutdown()
def handle_request(self, msg):
print "handle request.................................................."
print msg.headers
msg.reply()
def handle_response(self, msg):
print "handle response................................................."
print msg.headers
msg.reply()
config = proxy.ProxyConfig(
cacert = os.path.expanduser("~/.mitmproxy/mitmproxy-ca.pem")
)
server = proxy.ProxyServer(config, 1234)
m = Master(server)
m.run()
Then I configure http and ssl proxy in firefox to 127.0.0.1 port 1234. http seems to work fine as I can see all the headers are printed out. However, when the browser sends https requests, the proxy server does not print anything at all, and the browser displays "the connect was interrupted" error.
Further investigation reveals that the https requests go though the proxy server but not controller.Master. I see that proxy.ProxyHandler.establish_ssl() is being called when there is an https request, but the request does not go though controller.Master.handle_request(). Despite that establish_ssl() is called, the browser does not seem to get any response back. I test this with https://www.google.com.
First, how can I make proxy.ProxyHandler works properly with https requests/responses? Second, how can I modify controller.Master so that it can intercept https requests? I'm also open to other tools that I can build a custom http/https proxy server on top of.
You need to install the mitmproxy CA in the browser you are testing with.
Please see details here ("Installing the mitmproxy CA" section):
http://mitmproxy.org/doc/ssl.html
This solved the problem for me.
Currently using a really simple Twisted NameVirtualHost coupled with some JSON config files to serve really basic content in one Site object. The resources being served by Twisted are all WSGI objects built in flask.
I was wondering on how to go about wrapping the connections to these domains with an SSLContext, since reactor.listenSSL takes one and only one context, it isn't readily apparent how to give each domain/subdomain it's own crt/key pair. Is there any way to set up named virtual hosting with ssl for each domain that doesn't require proxying? I can't find any Twisted examples that use NameVirtualHost with SSL, and they only thing I could get to work is hook on the reactor listening on port 443 with only one domain's context?
I was wondering if anyone has attempted this?
My simple server without any SSL processing:
https://github.com/DeaconDesperado/twsrv/blob/master/service.py
TLS (the name for the modern protocol which replaces SSL) only very recently supports the feature you're looking for. The feature is called Server Name Indication (or SNI). It is supported by modern browsers on modern platforms, but not some older but still widely used platforms (see the wikipedia page for a list of browsers with support).
Twisted has no specific, built-in support for this. However, it doesn't need any. pyOpenSSL, upon which Twisted's SSL support is based, does support SNI.
The set_tlsext_servername_callback pyOpenSSL API gives you the basic mechanism to build the behavior you want. This lets you define a callback which is given access to the server name requested by the client. At this point, you can specify the key/certificate pair you want to use for the connection. You can find an example demonstrating the use of this API in pyOpenSSL's examples directory.
Here's an excerpt from that example to give you the gist:
def pick_certificate(connection):
try:
key, cert = certificates[connection.get_servername()]
except KeyError:
pass
else:
new_context = Context(TLSv1_METHOD)
new_context.use_privatekey(key)
new_context.use_certificate(cert)
connection.set_context(new_context)
server_context = Context(TLSv1_METHOD)
server_context.set_tlsext_servername_callback(pick_certificate)
You can incorporate this approach into a customized context factory and then supply that context factory to the listenSSL call.
Just to add some closure to this one, and for future searches, here is the example code for the echo server from the examples that prints the SNI:
from twisted.internet import ssl, reactor
from twisted.internet.protocol import Factory, Protocol
class Echo(Protocol):
def dataReceived(self, data):
self.transport.write(data)
def pick_cert(connection):
print('Received SNI: ', connection.get_servername())
if __name__ == '__main__':
factory = Factory()
factory.protocol = Echo
with open("keys/ca.pem") as certAuthCertFile:
certAuthCert = ssl.Certificate.loadPEM(certAuthCertFile.read())
with open("keys/server.key") as keyFile:
with open("keys/server.crt") as certFile:
serverCert = ssl.PrivateCertificate.loadPEM(
keyFile.read() + certFile.read())
contextFactory = serverCert.options(certAuthCert)
ctx = contextFactory.getContext()
ctx.set_tlsext_servername_callback(pick_cert)
reactor.listenSSL(8000, factory, contextFactory)
reactor.run()
And because getting OpenSSL to work can always be tricky, here is the OpenSSL statement you can use to connect to it:
openssl s_client -connect localhost:8000 -servername hello_world -cert keys/client.crt -key keys/client.key
Running the above python code against pyOpenSSL==0.13, and then running the s_client command above, will print this to the screen:
('Received SNI: ', 'hello_world')
There is now a txsni project that takes care of finding the right certificates per request. https://github.com/glyph/txsni