Currently using a really simple Twisted NameVirtualHost coupled with some JSON config files to serve really basic content in one Site object. The resources being served by Twisted are all WSGI objects built in flask.
I was wondering on how to go about wrapping the connections to these domains with an SSLContext, since reactor.listenSSL takes one and only one context, it isn't readily apparent how to give each domain/subdomain it's own crt/key pair. Is there any way to set up named virtual hosting with ssl for each domain that doesn't require proxying? I can't find any Twisted examples that use NameVirtualHost with SSL, and they only thing I could get to work is hook on the reactor listening on port 443 with only one domain's context?
I was wondering if anyone has attempted this?
My simple server without any SSL processing:
https://github.com/DeaconDesperado/twsrv/blob/master/service.py
TLS (the name for the modern protocol which replaces SSL) only very recently supports the feature you're looking for. The feature is called Server Name Indication (or SNI). It is supported by modern browsers on modern platforms, but not some older but still widely used platforms (see the wikipedia page for a list of browsers with support).
Twisted has no specific, built-in support for this. However, it doesn't need any. pyOpenSSL, upon which Twisted's SSL support is based, does support SNI.
The set_tlsext_servername_callback pyOpenSSL API gives you the basic mechanism to build the behavior you want. This lets you define a callback which is given access to the server name requested by the client. At this point, you can specify the key/certificate pair you want to use for the connection. You can find an example demonstrating the use of this API in pyOpenSSL's examples directory.
Here's an excerpt from that example to give you the gist:
def pick_certificate(connection):
try:
key, cert = certificates[connection.get_servername()]
except KeyError:
pass
else:
new_context = Context(TLSv1_METHOD)
new_context.use_privatekey(key)
new_context.use_certificate(cert)
connection.set_context(new_context)
server_context = Context(TLSv1_METHOD)
server_context.set_tlsext_servername_callback(pick_certificate)
You can incorporate this approach into a customized context factory and then supply that context factory to the listenSSL call.
Just to add some closure to this one, and for future searches, here is the example code for the echo server from the examples that prints the SNI:
from twisted.internet import ssl, reactor
from twisted.internet.protocol import Factory, Protocol
class Echo(Protocol):
def dataReceived(self, data):
self.transport.write(data)
def pick_cert(connection):
print('Received SNI: ', connection.get_servername())
if __name__ == '__main__':
factory = Factory()
factory.protocol = Echo
with open("keys/ca.pem") as certAuthCertFile:
certAuthCert = ssl.Certificate.loadPEM(certAuthCertFile.read())
with open("keys/server.key") as keyFile:
with open("keys/server.crt") as certFile:
serverCert = ssl.PrivateCertificate.loadPEM(
keyFile.read() + certFile.read())
contextFactory = serverCert.options(certAuthCert)
ctx = contextFactory.getContext()
ctx.set_tlsext_servername_callback(pick_cert)
reactor.listenSSL(8000, factory, contextFactory)
reactor.run()
And because getting OpenSSL to work can always be tricky, here is the OpenSSL statement you can use to connect to it:
openssl s_client -connect localhost:8000 -servername hello_world -cert keys/client.crt -key keys/client.key
Running the above python code against pyOpenSSL==0.13, and then running the s_client command above, will print this to the screen:
('Received SNI: ', 'hello_world')
There is now a txsni project that takes care of finding the right certificates per request. https://github.com/glyph/txsni
Related
I have been learning with WebSocket++ and built some of the server examples (Windows 10 Visual Studio 2019). The non-TLS examples work without issues, however, the TLS-enabled examples (echo_server_both.cpp and echo_server_tls.cpp) can't do the handshake. I am very new to web development in general so I know I must be doing something wrong with regards to the certificate and keys.
I am testing the servers with WebSocket King client, an extension of Google Chrome that connects correctly to other websocket servers like wss://echo.websocket.org and to my own localhost when I don't use TLS.
The echo_server_both example comes with a server.pem file, and the echo_server_tls example comes with server.pem and dh.pem. I have used the same files that come with the samples, and I have also tried generating and registering my own .pem files using openSSL. In both cases I get this when the client tries to connect:
[2021-06-29 20:51:21] [error] handle_transport_init received error: sslv3 alert certificate unknown
[2021-06-29 20:51:21] [fail] WebSocket Connection [::1]:63346 - "" - 0 asio.ssl:336151574 sslv3 alert certificate unknown
[2021-06-29 20:51:21] [info] asio async_shutdown error: asio.ssl:336462231 (shutdown while in init)
I discovered these errors after I edited handle_init() in tls.hpp, following a suggestion in another site, to look like this:
void handle_init(init_handler callback,lib::asio::error_code const & ec) {
if (ec) {
//m_ec = socket::make_error_code(socket::error::tls_handshake_failed);
m_ec = ec;
} else {
m_ec = lib::error_code();
}
callback(m_ec);
}
This change let the actual openSSL error to show in the console, otherwise it would show a generic "handshake failed" error.
I know I'm not doing what I should with the certificates, but I have no idea where else to look or what to do next. Can anyone here help please? Should I use the .pem files that come with the examples, or should I generate my own? in case I should generate my own, what would be the openSSL command to do that correctly and how do I tell my PC to recognize these as valid so that the server works?
Found the problem: WebSocket++ will not accept a self-signed certificate (the ones you can create directly in your own PC using OpenSSL or the Windows utilities). There is no way around it. You must have a valid, authority-validated and endorsed certificate. You can get such a certificate for free (valid only for 90 days) from https://zerossl.com/. The site has detailed instructions on how to request, obtain and install a certificate. After getting a valid certificate and installing it on my server, everything worked as it should.
I have the below function in my code, which works perfectly fine when I'm not behind any proxy. In fact, without even mentioning the certifi default CA certificate, it works fine if I pass verify=TRUE, I guess, because it works in the same way.
def reverse_lookup(lat, long):
cafile=certifi.where()
params={'lat' : float(lat), 'lon' : float(long), 'format' : 'json',
'accept-language' : 'en', 'addressdetails' : 1}
response = requests.get("https://nominatim.openstreetmap.org/reverse", params=params, verify=cafile)
#response = requests.get("https://nominatim.openstreetmap.org/reverse", params=params, verify=True) <-- this works as well
result = json.loads(response.text)
return result['address']['country'], result['address']['state'], result['address']['city']
When I run the same code from within my enterprise infrastructure (where I'm behind proxy), I make some minor changes in the code mentioning the proxy as parameter in requests method:
def reverse_lookup(lat, long):
cafile=certifi.where()
proxies = {"https" : "https://myproxy.com"}
params={'lat' : float(lat), 'lon' : float(long), 'format' : 'json',
'accept-language' : 'en', 'addressdetails' : 1}
response = requests.get("https://nominatim.openstreetmap.org/reverse", params=params, verify=cafile, proxies=proxies)
result = json.loads(response.text)
return result['address']['country'], result['address']['state'], result['address']['city']
But it gives me one out of these 3 SSL errors at different times, if I set verify=True or verify=certifi.where():
CERTIFICATE_VERIFY_FAILED
UNKNOWN_PROTOCOL
WRONG_VERSION_NUMBER
Only time it works is when I completely bypass the SSL verification with verify=False
My questions are:
Since I'm sending the https request via proxy, is it ok if I bypass SSL verification ?
How to make the default context of SSL verification work in this case, when I'm behind proxy ?
Any help is appreciated. Code tested in both Python 2.7.15 and 3.9
Since I'm sending the https request via proxy, is it ok if I bypass SSL verification ?
Do you need the protection offered by HTTPS, i.e. encryption of the application data (like passwords, but also the full URL) to protect against sniffing or modifications by a malicious man in the middle? If you don't need the protection, then you can bypass certificate validation.
How to make the default context of SSL verification work in this case, when I'm behind proxy ?
The proxy is doing SSL interception and when doing this issues a new certificate for this site based on an internal CA. If this is expected (i.e. not an attack) then you need to import the CA from the proxy as trusted with verify='proxy-ca.pem'. Your IT department should be able to provide you with the proxy CA.
But it gives me one out of these 3 SSL errors at different times, if I
set verify=True or verify=certifi.where():
CERTIFICATE_VERIFY_FAILED
UNKNOWN_PROTOCOL
WRONG_VERSION_NUMBER
It should only give you CERTIFICATE_VERIFY_FAILED. The two other errors indicate wrong proxy settings, typically setting https_proxy to https://... instead of http://... (which also can be seen in your code).
Is there a way to prevent Thin from accepting requests using SSLv3?
I cannot find any resources on how to deal with Poodle for a Thin server running with SSL. I don't want to move thin behind nginx if I don't have to so any resources would be helpful. I took a look at the source code but couldn't find a way to monkey patch it, can't even find a reference to it.
Thin uses eventmachine, so the solution involves making eventmachine not use SSLv2 or v3.
This discussion has some insights on a generic patch
https://github.com/eventmachine/eventmachine/issues/359
Another option is to build eventmachine with a patch to disable
(https://github.com/eventmachine/eventmachine/wiki/Building-EventMachine)
and then a patch of
--- a/ext/ssl.cpp
+++ b/ext/ssl.cpp
## -145,7 +145,7 ## SslContext_t::SslContext_t (bool is_server, const string &privkeyfile, const str
}
bIsServer = is_server;
- pCtx = SSL_CTX_new (is_server ? SSLv23_server_method() : SSLv23_client_method());
+ pCtx = SSL_CTX_new (is_server ? TLSv1_server_method() : TLSv1_client_method());
if (!pCtx)
throw std::runtime_error ("no SSL context");
I haven't been able to get it fully tested, but this command should fail:
openssl s_client -connect 127.0.0.1:3000 -ssl3
I would like to mask the version or remove the header altogether.
To change the 'Server:' http header, in your conf.py file:
import gunicorn
gunicorn.SERVER_SOFTWARE = 'Microsoft-IIS/6.0'
And use an invocation along the lines of gunicorn -c conf.py wsgi:app
To remove the header altogether, you can monkey-patch gunicorn by replacing its http response class with a subclass that filters out the header. This might be harmless, but is probably not recommended. Put the following in conf.py:
from gunicorn.http import wsgi
class Response(wsgi.Response):
def default_headers(self, *args, **kwargs):
headers = super(Response, self).default_headers(*args, **kwargs)
return [h for h in headers if not h.startswith('Server:')]
wsgi.Response = Response
Tested with gunicorn 18
This hasn't been clearly written here so I'm gonna confirm that the easiest way for the latest version of Gunicorn (20.1.x) is to add following lines into configuration file:
import gunicorn
gunicorn.SERVER = 'undisclosed'
For newer releases (20.0.4): Create a gunicorn.conf.py file with the content below in the directory from where you will run the gunicorn command:
import gunicorn
gunicorn.SERVER_SOFTWARE = 'My WebServer'
It's better to change it to something unique than remove it. You don't want to risk, e.g., spiders thinking you're noncompliant. Changing it to the name of software you aren't using can cause similar problems. Making it unique will prevent the same kind of assumptions ever being made. I recommend something like this:
import gunicorn
gunicorn.SERVER_SOFTWARE = 'intentionally-undisclosed-gensym384763'
You can edit __init__.py to set SERVER_SOFTWARE to whatever you want. But I'd really like the ability to disable this with a flag so I didn't need to reapply the patch when I upgrade.
My mocky-patch free solution, involves wrapping the default_headers method:
import gunicorn.http.wsgi
from six import wraps
def wrap_default_headers(func):
#wraps(func)
def default_headers(*args, **kwargs):
return [header for header in func(*args, **kwargs) if not header.startswith('Server: ')]
return default_headers
gunicorn.http.wsgi.Response.default_headers = wrap_default_headers(gunicorn.http.wsgi.Response.default_headers)
This doesn't directly answer to the question but could address the issue as well and without monkey patching gunicorn.
If you are using gunicorn behind a reverse proxy, as it usually happens, you can set, add, remove or perform a replacement in a response header coming downstream from the backend. In our case the Server header.
I guess every Webserver should have an equivalent feature.
For example, in Caddy 2 (currently in beta) it would be something as simple as:
https://localhost {
reverse_proxy unix//tmp/foo.sock {
header_down Server intentionally-undisclosed-12345678
}
}
For completeness I still add a minimal (but fully working) Caddyfile to handle Server header modification even in manual http->https redirect process (Caddy 2 does it automatically, if you don't override it), which could a bit tricky to figure it out correctly.
http://localhost {
# Fact: the `header` directive has less priority than `redir` (which means
# it's evaluated later), so the header wouldn't be changed (and Caddy would
# shown instead of the faked value).
#
# To override the directive ordering only for this server, instead of
# change the "order" option globally, put the configuration inside a
# route directive.
# ref.
# https://caddyserver.com/docs/caddyfile/options
# https://caddyserver.com/docs/caddyfile/directives/route
# https://caddyserver.com/docs/caddyfile/directives#directive-order
route {
header Server intentionally-undisclosed-12345678
redir https://{host}{uri}
}
}
https://localhost {
reverse_proxy unix//tmp/foo.sock {
header_down Server intentionally-undisclosed-12345678
}
}
To check if it works just use curl as curl --insecure -I http://localhost and curl --insecure -I http://localhost (--insecure because localhost certs are automatically generated as self signed).
It's so simple to setup that you could also think to use it in development (with gunicorn --reload), especially if it resembles your staging/production environment.
From CherryPy 3.0 and onwards, one-way SSL can be turned on simply by pointing to the server certificate and private key, like this:
import cherrypy
class HelloWorld(object):
def index(self):
return "Hello SSL World!"
index.exposed = True
cherrypy.server.ssl_certificate = "keys/server.crt"
cherrypy.server.ssl_private_key = "keys/server.crtkey"
cherrypy.quickstart(HelloWorld())
This enables clients to validate the server's authenticity. Does anyone know whether CherryPy supports 2-way ssl, e.g. where the server can also check client authenticity by validating a client certificate?
If yes, could anyone give an example how is that done? Or post a reference to an example?
It doesn't out of the box. You'd have to patch the wsgiserver to provide that feature. There is a ticket (and patches) in progress at http://www.cherrypy.org/ticket/1001.
I have been looking for the same thing. I know there are some patches on the CherryPy site.
I also found the following at CherryPy SSL Client Authentication. I haven't compared this vs the CherryPy patches but maybe the info will be helpful.
We recently needed to develop a quick
but resilient REST application and
found that CherryPy suited our needs
better than other Python networking
frameworks, like Twisted.
Unfortunately, its simplicity lacked a
key feature we needed, Server/Client
SSL certificate validation. Therefore
we spent a few hours writing a few
quick modifications to the current
release, 3.1.2. The following code
snippets are the modifications we
made:
cherrypy/_cpserver.py
## -55,7 +55,6 ## instance = None ssl_certificate = None ssl_private_key
= None
+ ssl_ca_certificate = None nodelay = True
def __init__(self):
cherrypy/wsgiserver/__init__.py
## -1480,6 +1480,7 ##
# Paths to certificate and private key files ssl_certificate = None ssl_private_key = None
+ ssl_ca_certificate = None
def __init__(self, bind_addr, wsgi_app, numthreads=10, server_name=None, max=-1, request_queue_size=5, timeout=10, shutdown_timeout=5):
## -1619,7 +1620,9 ##
self.socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) if self.nodelay: self.socket.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)
- if self.ssl_certificate and self.ssl_private_key:
+ if self.ssl_certificate and self.ssl_private_key and \
+ self.ssl_ca_certificate:
+ if SSL is None: raise ImportError("You must install pyOpenSSL to use HTTPS.")
## -1627,6 +1630,11 ## ctx = SSL.Context(SSL.SSLv23_METHOD) ctx.use_privatekey_file(self.ssl_private_key) ctx.use_certificate_file(self.ssl_certificate)
+ x509 = crypto.load_certificate(crypto.FILETYPE_PEM,
+ open(self.ssl_ca_certificate).read())
+ store = ctx.get_cert_store()
+ store.add_cert(x509)
+ ctx.set_verify(SSL.VERIFY_PEER | SSL.VERIFY_FAIL_IF_NO_PEER_CERT, lambda *x:True) self.socket = SSLConnection(ctx, self.socket) self.populate_ssl_environ()
The above patches require the
inclusion of a new configuration
option inside of the CherryPy server
configuration,
server.ssl_ca_certificate. This
option identifies the certificate
authority file that connecting clients
will be validated against, if the
client does not present a valid client
certificate it will close the
connection immediately.
Our solution has advantages and
disadvantages, the primary advantage
being if the connecting client doesn’t
present a valid certificate it’s
connection is immediately closed.
This is good for security concerns as
it does not permit the client any
access into the CherryPy application
stack. However, since the restriction
is done at the socket level the
CherryPy application can never see the
client connecting and hence the
solution is somewhat inflexible.
An optimal solution would allow the
client to connect to the CherryPy
socket and send the client certificate
up into the application stack. Then a
custom CherryPy Tool would validate
the certificate inside of the
application stack and close the
connection if necessary; unfortunately
because of the structure of CherryPy’s
pyOpenSSL implementation it is
difficult to retrieve the client
certificate inside of the application
stack.
Of course the patches above should
only be used at your own risk. If you
come up with a better solution please
let us know.
If the current version of CherryPy does not support client certificate verification, it is possible to configure CherryPy to listen to 127.0.0.1:80, install HAProxy to listen to 443 and verify client side certificates and to forward traffic to 127.0.0.1:80
HAProxy is simple, light, fast and reliable.
An example of HAProxy configuration