I'm writing a little code in python3 using the requests module.
I send https requests to a server but i need to dump the exechanges beetwin the server and my client.
I know how dump the ssl key with firefox and use it to decrypt the data with whireshark but how can do the same with a python program ?
I need a thirth program like ssldump for exemple or i can do it in the python code ?
def get_city_id(city_name):
result = requests.get(url+'/json/ac-geo', params='q=' + city_name, headers=header)
return(result)
def search_annonce(criteres):
result = requests.post(url+'/annonce', headers=header, data=criteres)
print(get_city_id('draveil').json())
city_id = get_city_id('versailles').json()
criteres = {'produit': 'vente', 'geo_objets_ids': city_id[0]['id'], 'typesbien[]': 'appartement', 'surface[min]': 40, 'prix[max]': '', 'nb_pieces[list][]': 2, 'recherche': 1, 'reference_courte': ''}
You will need to use a TLS proxy if you wanna decrypt HTTPS traffic from a client that cannot export an SSLKEYLOGFILE. We have a free tool called PolarProxy, which acts as a transparent proxy end dumps the decrypted traffic as plain HTTP to a PCAP file that you can open in Wireshark.
Related
I have the below function in my code, which works perfectly fine when I'm not behind any proxy. In fact, without even mentioning the certifi default CA certificate, it works fine if I pass verify=TRUE, I guess, because it works in the same way.
def reverse_lookup(lat, long):
cafile=certifi.where()
params={'lat' : float(lat), 'lon' : float(long), 'format' : 'json',
'accept-language' : 'en', 'addressdetails' : 1}
response = requests.get("https://nominatim.openstreetmap.org/reverse", params=params, verify=cafile)
#response = requests.get("https://nominatim.openstreetmap.org/reverse", params=params, verify=True) <-- this works as well
result = json.loads(response.text)
return result['address']['country'], result['address']['state'], result['address']['city']
When I run the same code from within my enterprise infrastructure (where I'm behind proxy), I make some minor changes in the code mentioning the proxy as parameter in requests method:
def reverse_lookup(lat, long):
cafile=certifi.where()
proxies = {"https" : "https://myproxy.com"}
params={'lat' : float(lat), 'lon' : float(long), 'format' : 'json',
'accept-language' : 'en', 'addressdetails' : 1}
response = requests.get("https://nominatim.openstreetmap.org/reverse", params=params, verify=cafile, proxies=proxies)
result = json.loads(response.text)
return result['address']['country'], result['address']['state'], result['address']['city']
But it gives me one out of these 3 SSL errors at different times, if I set verify=True or verify=certifi.where():
CERTIFICATE_VERIFY_FAILED
UNKNOWN_PROTOCOL
WRONG_VERSION_NUMBER
Only time it works is when I completely bypass the SSL verification with verify=False
My questions are:
Since I'm sending the https request via proxy, is it ok if I bypass SSL verification ?
How to make the default context of SSL verification work in this case, when I'm behind proxy ?
Any help is appreciated. Code tested in both Python 2.7.15 and 3.9
Since I'm sending the https request via proxy, is it ok if I bypass SSL verification ?
Do you need the protection offered by HTTPS, i.e. encryption of the application data (like passwords, but also the full URL) to protect against sniffing or modifications by a malicious man in the middle? If you don't need the protection, then you can bypass certificate validation.
How to make the default context of SSL verification work in this case, when I'm behind proxy ?
The proxy is doing SSL interception and when doing this issues a new certificate for this site based on an internal CA. If this is expected (i.e. not an attack) then you need to import the CA from the proxy as trusted with verify='proxy-ca.pem'. Your IT department should be able to provide you with the proxy CA.
But it gives me one out of these 3 SSL errors at different times, if I
set verify=True or verify=certifi.where():
CERTIFICATE_VERIFY_FAILED
UNKNOWN_PROTOCOL
WRONG_VERSION_NUMBER
It should only give you CERTIFICATE_VERIFY_FAILED. The two other errors indicate wrong proxy settings, typically setting https_proxy to https://... instead of http://... (which also can be seen in your code).
I've implemented a SOAP client in python with the zeep library. Some of the endpoints require client-side certificate authentication, thus the need to attach the certificate to the python requests's session.
After googling around, I've come up with:
from zeep import Client
from zeep.transports import Transport
from django.utils import timezone
import requests
......
session = requests.Session()
session.verify = False
session.cert= ('pat_to_cert.pem','path_to_privKey.pem')
transport = Transport(session=session)
....
client = Client(wsdl=wsdl, transport=transport)
send = getattr(service, service_name)
result = send(**data)
Debbugging the TLS handshake, the server issues a Certificate Request, but the client replies with an empty certificate. I've already checked the .pem files with openssl with no errors.
Is it possible that python's requests is not attaching the certificate because it does not recognize the server name? How can I enforce to use this certificate for every request?
In your code, you are setting session.verify = False that's why TLS verification is disabled.
what you need to do is to set:
session.verify = 'path/to/my/certificate.pem'
Also, Alternatively, instead of using session.verify you can use session.cert if you just want to use an TLS client certificate:
session.verify = True
session.cert = ('my_cert', 'my_key')
please check the documentation for more details: https://docs.python-zeep.org/en/master/transport.html
hope this helps
I want to implement a proxy server that intercepts both http and https requests. I came across libmproxy (http://mitmproxy.org/doc/scripting/libmproxy.html) that it is SSL-capable. I start with this simplest proxy that just prints the headers of all requests and responses, and forwards them to clients and servers normally.
#!/usr/bin/env python
from libmproxy import controller, proxy
import os
class Master(controller.Master):
def __init__(self, server):
controller.Master.__init__(self, server)
self.stickyhosts = {}
def run(self):
try:
return controller.Master.run(self)
except KeyboardInterrupt:
self.shutdown()
def handle_request(self, msg):
print "handle request.................................................."
print msg.headers
msg.reply()
def handle_response(self, msg):
print "handle response................................................."
print msg.headers
msg.reply()
config = proxy.ProxyConfig(
cacert = os.path.expanduser("~/.mitmproxy/mitmproxy-ca.pem")
)
server = proxy.ProxyServer(config, 1234)
m = Master(server)
m.run()
Then I configure http and ssl proxy in firefox to 127.0.0.1 port 1234. http seems to work fine as I can see all the headers are printed out. However, when the browser sends https requests, the proxy server does not print anything at all, and the browser displays "the connect was interrupted" error.
Further investigation reveals that the https requests go though the proxy server but not controller.Master. I see that proxy.ProxyHandler.establish_ssl() is being called when there is an https request, but the request does not go though controller.Master.handle_request(). Despite that establish_ssl() is called, the browser does not seem to get any response back. I test this with https://www.google.com.
First, how can I make proxy.ProxyHandler works properly with https requests/responses? Second, how can I modify controller.Master so that it can intercept https requests? I'm also open to other tools that I can build a custom http/https proxy server on top of.
You need to install the mitmproxy CA in the browser you are testing with.
Please see details here ("Installing the mitmproxy CA" section):
http://mitmproxy.org/doc/ssl.html
This solved the problem for me.
Currently using a really simple Twisted NameVirtualHost coupled with some JSON config files to serve really basic content in one Site object. The resources being served by Twisted are all WSGI objects built in flask.
I was wondering on how to go about wrapping the connections to these domains with an SSLContext, since reactor.listenSSL takes one and only one context, it isn't readily apparent how to give each domain/subdomain it's own crt/key pair. Is there any way to set up named virtual hosting with ssl for each domain that doesn't require proxying? I can't find any Twisted examples that use NameVirtualHost with SSL, and they only thing I could get to work is hook on the reactor listening on port 443 with only one domain's context?
I was wondering if anyone has attempted this?
My simple server without any SSL processing:
https://github.com/DeaconDesperado/twsrv/blob/master/service.py
TLS (the name for the modern protocol which replaces SSL) only very recently supports the feature you're looking for. The feature is called Server Name Indication (or SNI). It is supported by modern browsers on modern platforms, but not some older but still widely used platforms (see the wikipedia page for a list of browsers with support).
Twisted has no specific, built-in support for this. However, it doesn't need any. pyOpenSSL, upon which Twisted's SSL support is based, does support SNI.
The set_tlsext_servername_callback pyOpenSSL API gives you the basic mechanism to build the behavior you want. This lets you define a callback which is given access to the server name requested by the client. At this point, you can specify the key/certificate pair you want to use for the connection. You can find an example demonstrating the use of this API in pyOpenSSL's examples directory.
Here's an excerpt from that example to give you the gist:
def pick_certificate(connection):
try:
key, cert = certificates[connection.get_servername()]
except KeyError:
pass
else:
new_context = Context(TLSv1_METHOD)
new_context.use_privatekey(key)
new_context.use_certificate(cert)
connection.set_context(new_context)
server_context = Context(TLSv1_METHOD)
server_context.set_tlsext_servername_callback(pick_certificate)
You can incorporate this approach into a customized context factory and then supply that context factory to the listenSSL call.
Just to add some closure to this one, and for future searches, here is the example code for the echo server from the examples that prints the SNI:
from twisted.internet import ssl, reactor
from twisted.internet.protocol import Factory, Protocol
class Echo(Protocol):
def dataReceived(self, data):
self.transport.write(data)
def pick_cert(connection):
print('Received SNI: ', connection.get_servername())
if __name__ == '__main__':
factory = Factory()
factory.protocol = Echo
with open("keys/ca.pem") as certAuthCertFile:
certAuthCert = ssl.Certificate.loadPEM(certAuthCertFile.read())
with open("keys/server.key") as keyFile:
with open("keys/server.crt") as certFile:
serverCert = ssl.PrivateCertificate.loadPEM(
keyFile.read() + certFile.read())
contextFactory = serverCert.options(certAuthCert)
ctx = contextFactory.getContext()
ctx.set_tlsext_servername_callback(pick_cert)
reactor.listenSSL(8000, factory, contextFactory)
reactor.run()
And because getting OpenSSL to work can always be tricky, here is the OpenSSL statement you can use to connect to it:
openssl s_client -connect localhost:8000 -servername hello_world -cert keys/client.crt -key keys/client.key
Running the above python code against pyOpenSSL==0.13, and then running the s_client command above, will print this to the screen:
('Received SNI: ', 'hello_world')
There is now a txsni project that takes care of finding the right certificates per request. https://github.com/glyph/txsni
From CherryPy 3.0 and onwards, one-way SSL can be turned on simply by pointing to the server certificate and private key, like this:
import cherrypy
class HelloWorld(object):
def index(self):
return "Hello SSL World!"
index.exposed = True
cherrypy.server.ssl_certificate = "keys/server.crt"
cherrypy.server.ssl_private_key = "keys/server.crtkey"
cherrypy.quickstart(HelloWorld())
This enables clients to validate the server's authenticity. Does anyone know whether CherryPy supports 2-way ssl, e.g. where the server can also check client authenticity by validating a client certificate?
If yes, could anyone give an example how is that done? Or post a reference to an example?
It doesn't out of the box. You'd have to patch the wsgiserver to provide that feature. There is a ticket (and patches) in progress at http://www.cherrypy.org/ticket/1001.
I have been looking for the same thing. I know there are some patches on the CherryPy site.
I also found the following at CherryPy SSL Client Authentication. I haven't compared this vs the CherryPy patches but maybe the info will be helpful.
We recently needed to develop a quick
but resilient REST application and
found that CherryPy suited our needs
better than other Python networking
frameworks, like Twisted.
Unfortunately, its simplicity lacked a
key feature we needed, Server/Client
SSL certificate validation. Therefore
we spent a few hours writing a few
quick modifications to the current
release, 3.1.2. The following code
snippets are the modifications we
made:
cherrypy/_cpserver.py
## -55,7 +55,6 ## instance = None ssl_certificate = None ssl_private_key
= None
+ ssl_ca_certificate = None nodelay = True
def __init__(self):
cherrypy/wsgiserver/__init__.py
## -1480,6 +1480,7 ##
# Paths to certificate and private key files ssl_certificate = None ssl_private_key = None
+ ssl_ca_certificate = None
def __init__(self, bind_addr, wsgi_app, numthreads=10, server_name=None, max=-1, request_queue_size=5, timeout=10, shutdown_timeout=5):
## -1619,7 +1620,9 ##
self.socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) if self.nodelay: self.socket.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)
- if self.ssl_certificate and self.ssl_private_key:
+ if self.ssl_certificate and self.ssl_private_key and \
+ self.ssl_ca_certificate:
+ if SSL is None: raise ImportError("You must install pyOpenSSL to use HTTPS.")
## -1627,6 +1630,11 ## ctx = SSL.Context(SSL.SSLv23_METHOD) ctx.use_privatekey_file(self.ssl_private_key) ctx.use_certificate_file(self.ssl_certificate)
+ x509 = crypto.load_certificate(crypto.FILETYPE_PEM,
+ open(self.ssl_ca_certificate).read())
+ store = ctx.get_cert_store()
+ store.add_cert(x509)
+ ctx.set_verify(SSL.VERIFY_PEER | SSL.VERIFY_FAIL_IF_NO_PEER_CERT, lambda *x:True) self.socket = SSLConnection(ctx, self.socket) self.populate_ssl_environ()
The above patches require the
inclusion of a new configuration
option inside of the CherryPy server
configuration,
server.ssl_ca_certificate. This
option identifies the certificate
authority file that connecting clients
will be validated against, if the
client does not present a valid client
certificate it will close the
connection immediately.
Our solution has advantages and
disadvantages, the primary advantage
being if the connecting client doesn’t
present a valid certificate it’s
connection is immediately closed.
This is good for security concerns as
it does not permit the client any
access into the CherryPy application
stack. However, since the restriction
is done at the socket level the
CherryPy application can never see the
client connecting and hence the
solution is somewhat inflexible.
An optimal solution would allow the
client to connect to the CherryPy
socket and send the client certificate
up into the application stack. Then a
custom CherryPy Tool would validate
the certificate inside of the
application stack and close the
connection if necessary; unfortunately
because of the structure of CherryPy’s
pyOpenSSL implementation it is
difficult to retrieve the client
certificate inside of the application
stack.
Of course the patches above should
only be used at your own risk. If you
come up with a better solution please
let us know.
If the current version of CherryPy does not support client certificate verification, it is possible to configure CherryPy to listen to 127.0.0.1:80, install HAProxy to listen to 443 and verify client side certificates and to forward traffic to 127.0.0.1:80
HAProxy is simple, light, fast and reliable.
An example of HAProxy configuration