Is it possible to identify TLS info. in requests response? - ssl

Question

I am using python's requests module. I can get the server's response headers and application layer data as:
import requests
r = requests.get('https://yahoo.com')
print(r.url)
My question: Does requests allow retrieving Transport layer data (the server's TLS selected version, ciphersuite, etc. ?).

Answer 1

Here is a quick ugly monkey patching version that works:
import requests
from requests.packages.urllib3.connection import VerifiedHTTPSConnection
SOCK = None
_orig_connect = requests.packages.urllib3.connection.VerifiedHTTPSConnection.connect
def _connect(self):
global SOCK
_orig_connect(self)
SOCK = self.sock
requests.packages.urllib3.connection.VerifiedHTTPSConnection.connect = _connect
requests.get('https://yahoo.com')
tlscon = SOCK.connection
print 'Cipher is %s/%s' % (tlscon.get_cipher_name(), tlscon.get_cipher_version())
print 'Remote certificates: %s' % (tlscon.get_peer_certificate())
print 'Protocol version: %s' % tlscon.get_protocol_version_name()
This yields:
Cipher is ECDHE-RSA-AES128-GCM-SHA256/TLSv1.2
Remote certificates: <OpenSSL.crypto.X509 object at 0x10c60e310>
Protocol version: TLSv1.2
However it is bad because monkey patching and relying on a unique global variable, which also means you can not inspect what happens at redirect steps, and so on.
Maybe with some work that can be turned out as a Transport Adapter, to get the underlying connection as a property of the request (probably of the session or something). That may create leaks though, because in the current implementation the underlying socket is thrown away as quickly as possible (see How to get the underlying socket when using Python requests).
Update, now using a Transport Adapter
This works, and is in line with the framework (no global variable, should handle redirects, etc. there may something to do for proxies though, like adding an override for proxy_manager_for too), but it is a lot more code.
import requests
from requests.adapters import HTTPAdapter
from requests.packages.urllib3.connectionpool import HTTPSConnectionPool
from requests.packages.urllib3.poolmanager import PoolManager
class InspectedHTTPSConnectionPool(HTTPSConnectionPool):
#property
def inspector(self):
return self._inspector
#inspector.setter
def inspector(self, inspector):
self._inspector = inspector
def _validate_conn(self, conn):
r = super(InspectedHTTPSConnectionPool, self)._validate_conn(conn)
if self.inspector:
self.inspector(self.host, self.port, conn)
return r
class InspectedPoolManager(PoolManager):
#property
def inspector(self):
return self._inspector
#inspector.setter
def inspector(self, inspector):
self._inspector = inspector
def _new_pool(self, scheme, host, port):
if scheme != 'https':
return super(InspectedPoolManager, self)._new_pool(scheme, host, port)
kwargs = self.connection_pool_kw
if scheme == 'http':
kwargs = self.connection_pool_kw.copy()
for kw in SSL_KEYWORDS:
kwargs.pop(kw, None)
pool = InspectedHTTPSConnectionPool(host, port, **kwargs)
pool.inspector = self.inspector
return pool
class TLSInspectorAdapter(HTTPAdapter):
def __init__(self, inspector):
self._inspector = inspector
super(TLSInspectorAdapter, self).__init__()
def init_poolmanager(self, connections, maxsize, block=False, **pool_kwargs):
self.poolmanager = InspectedPoolManager(num_pools=connections, maxsize=maxsize, block=block, strict=True, **pool_kwargs)
self.poolmanager.inspector = self._inspector
def connection_inspector(host, port, connection):
print 'host is %s' % host
print 'port is %s' % port
print 'connection is %s' % connection
sock = connection.sock
sock_connection = sock.connection
print 'socket is %s' % sock
print 'Protocol version: %s' % sock_connection.get_protocol_version_name()
print 'Cipher is %s/%s' % (sock_connection.get_cipher_name(), sock_connection.get_cipher_version())
print 'Remote certificate: %s' % sock.getpeercert()
url = 'https://yahoo.com'
s = requests.Session()
s.mount(url, TLSInspectorAdapter(connection_inspector))
r = s.get(url)
Yes, there is a lot of confusion in naming between socket and connection: requests uses a "connection pool" that has a set of connections, which are in fact, for HTTPS, a PyOpenSSL WrappedSocket, which has itself an underlying real TLS connection (that is a PyOpenSSL Connection object). Hence the strange forms in connection_inspector.
But this returns the expected:
host is yahoo.com
port is 443
connection is <requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x10bb372d0>
socket is <requests.packages.urllib3.contrib.pyopenssl.WrappedSocket object at 0x10bb37410>
Protocol version: TLSv1.2
Cipher is ECDHE-RSA-AES128-GCM-SHA256/TLSv1.2
Remote certificate: {'subjectAltName': [('DNS', '*.www.yahoo.com'), ('DNS', 'add.my.yahoo.com'), ('DNS', '*.amp.yimg.com'), ('DNS', 'au.yahoo.com'), ('DNS', 'be.yahoo.com'), ('DNS', 'br.yahoo.com'), ('DNS', 'ca.my.yahoo.com'), ('DNS', 'ca.rogers.yahoo.com'), ('DNS', 'ca.yahoo.com'), ('DNS', 'ddl.fp.yahoo.com'), ('DNS', 'de.yahoo.com'), ('DNS', 'en-maktoob.yahoo.com'), ('DNS', 'espanol.yahoo.com'), ('DNS', 'es.yahoo.com'), ('DNS', 'fr-be.yahoo.com'), ('DNS', 'fr-ca.rogers.yahoo.com'), ('DNS', 'frontier.yahoo.com'), ('DNS', 'fr.yahoo.com'), ('DNS', 'gr.yahoo.com'), ('DNS', 'hk.yahoo.com'), ('DNS', 'hsrd.yahoo.com'), ('DNS', 'ideanetsetter.yahoo.com'), ('DNS', 'id.yahoo.com'), ('DNS', 'ie.yahoo.com'), ('DNS', 'in.yahoo.com'), ('DNS', 'it.yahoo.com'), ('DNS', 'maktoob.yahoo.com'), ('DNS', 'malaysia.yahoo.com'), ('DNS', 'mbp.yimg.com'), ('DNS', 'my.yahoo.com'), ('DNS', 'nz.yahoo.com'), ('DNS', 'ph.yahoo.com'), ('DNS', 'qc.yahoo.com'), ('DNS', 'ro.yahoo.com'), ('DNS', 'se.yahoo.com'), ('DNS', 'sg.yahoo.com'), ('DNS', 'tw.yahoo.com'), ('DNS', 'uk.yahoo.com'), ('DNS', 'us.yahoo.com'), ('DNS', 'verizon.yahoo.com'), ('DNS', 'vn.yahoo.com'), ('DNS', 'www.yahoo.com'), ('DNS', 'yahoo.com'), ('DNS', 'za.yahoo.com')], 'subject': ((('commonName', u'*.www.yahoo.com'),),)}
Other ideas:
You may remove a lot of code if you do monkey patching like in https://stackoverflow.com/a/22253656/6368697 with basically poolmanager.pool_classes_by_scheme['http'] = MyHTTPConnectionPool; but this is still monkey patching, and it is sad that PoolManager does not give a nice API for the pool_classes_by_scheme variable to be able to easily override it
PyOpenSSL ssl_context may be able to hold callbacks that will be called during the TLS handshake and get the underlying data; then in init_poolmanager you would just need to setup the ssl_context in kwargs before calling superclass; this example in https://gist.github.com/aiguofer/1eb881ccf199d4aaa2097d87f93ace6a <= or maybe not because in fact the structure comes from ssl.create_default_context and ssl is far less powerful than PyOpenSSL and I see no way to add callbacks using ssl, where they exist for PyOpenSSL. YMMV.
PS:
Once you find out you have _validate_conn that you can override as it gets the proper connection object, life is easier
And especially if you do the import on top correctly, you need to use the urllib3 packages that are distributed inside requests, not the "real" urllib3 otherwise you get a lot of strange errors, because the same methods in both do not have the same signatures...