How to enter SSL PEM pass phrase only once in gevent streamServer? - ssl

Every time a client connects to the server I get a prompt to enter the PEM pass phrase. I want to only enter it on startup of the server and not have to enter it again. The Twisted Matrix framework only requires the pass phrase on startup, why not gevent? Or am I using gevent wrong?
The code below works fine if I use a certificate that require no PEM pass phrase, but I want to use a certificate with a pass phrase.
from gevent.server import StreamServer
from gevent.pool import Pool
from gevent import monkey
class SocketPool(object):
def __init__(self): self.pool = Pool(1000)
def listen(self, socket):
while True:
line = socket.recv(1024)
if not line: break
print line
socket.close()
break
def add_handler(self, socket, address):
print "connection made:", address
if self.pool.full(): raise Exception("At maximum pool size")
else: self.pool.spawn(self.listen, socket)
def shutdown(self): self.pool.kill()
monkey.patch_all()
sockPool = SocketPool()
server = StreamServer(('', 5000), sockPool.add_handler, keyfile='key.pem', certfile='cert.pem')
server.serve_forever()

There doesn't appear to be an easy way to fix this on python 2.7. Internally, gevent.server uses the python ssl module which calls the OpenSSL library from C. The _ssl.sslwrap() function creates a new SSLContext for each call and this is why you are prompted for the key passphrase each time a connection is made.
Twisted uses pyOpenSSL instead of the ssl module. In pyOpenSSL you create your SSLContext first and that single context is used each time a connection is made. Thus you are only prompted for the passphrase once when using Twisted.
Python 3.2 adds an ssl.SSLContext class with a wrap_socket() method. If you were on this version or later then you could patch the gevent code to work more like Twisted, i.e. use a single SSLContext and replace the call to _ssl.sslwrap() with a call to the wrap_socket() method. It doesn't appear that gevent is ported to python3, though.
You could offload SSL handling by gevent by placing an SSL proxy in front so incoming SSL connections to a public port are proxied to your gevent server on an unencrypted internal port. You could write your own proxy with Twisted or use a dedicated proxy like stunnel. However, this might negate some or all of the reasons you wanted to use gevent in the first place.
You could patch your _ssl module either to cache SSLContexts or to use another way of accepting the passphrase. _ssl in python 2.7 uses the default passphrase callback, which interactively prompts to enter the passphrase. This wouldn't be that difficult, but it requires that you know at least a bit of C, the python C API, and the OpenSSL API, plus you would have to build and override the module everywhere you deploy.
You could patch gevent with your own code to wrap a socket with SSL. This can be done entirely in python with a library like pyOpenSSL, but getting it right is probably a fair amount of work.
Finally, you could just use an unencrypted key. Perhaps putting the key on a RAM disk or on an encrypted file system (or both) would address whatever security requirements you have.

Related

Apache VFS SFTP Connection hangs

I am using Apache VFS to upload a file to an SFTP server, if the file is newer than the file on the server or doesn't exist there yet. The server connection uses SSH Keys for Authentication.
I am using the following java code (plus error handling etc.) to connect to the server and check the file modification date-time:
DefaultFileSystemManager manager = new DefaultFileSystemManager();
manager.addProvider("sftp", new SftpFileProvider());
manager.init();
FileSystemOptions opts = createDefaultOptions();
BytesIdentityInfo identityInfo = new BytesIdentityInfo(server.sshKey.getBytes(), null);
SftpFileSystemConfigBuilder.getInstance().setIdentityProvider(opts, identityInfo);
remoteFileObject = manager.resolveFile(new URI("sftp",server.UserName,server.HostName,server.Port,remoteFilePath,null,null).toString(), createDefaultOptions(server.Key));
FileContent content = remoteFileObject.getContent();
return content.getLastModifiedTime();
The SSH key is in the format -----BEGIN RSA PRIVATE KEY----- etc.; as exported by puttyGen under Conversions -> Export OpenSSH Key (i.e. the old format of OpenSSH key, not the new one).
I have tested this code on Windows, with a locally hosted SFTP server (i.e. also on the same Windows machine), and it works successfully.
I am now wanting to use this in a Linux environment (RHEL), connecting to an AWS Transfer SFTP server, secured using SSH keys as described.
I can connect successfully using the SFTP command from the Linux OS shell:
sftp -oIdentityFile=/path/to/test.ppk USER#xxx.xxx.xxx.xxx
But, when I try to run the java code, the code hangs on the call to manager.resolveFile.
After half an hour (I think - this might not be related), I get the following in /var/log/messages:
systemd-logind[1297]: Session 115360 logged out. Waiting for processes to exit.
systemd[1]: session-115360.scope: Succeeded.
systemd-logind[1297]: Removed session 115360.
I don't have SELinux enabled, so I don't think that's interfering in any way.
Can anyone help suggest what might be causing this?
There were a couple of things, as it turns out:
Timeout
The timeout can be set when you configure the SftpFileSystemConfigBuilder, by using the .setSessionTimeout(FileSystemOptions, Duration) method call. This reduces the timeout which, if nothing else, makes the issue easier to debug.
The Session comments in the messages log were not related to the issue. Instead, the issue happened because the exec channel is disabled on the SFTP server, but VFS is trying to use it. At a simple level, this can be disabled using setDisableDetectExecChannel on the SftpFileSystemConfigBuilder object - but you should know the implications of this before doing so.

What are the hidden configurations that enable TLS communication with only CA info from Kafka cluster?

I have this code, it works with my Kafka cluster (I got it from my colleague and no idea how it is configured). I copied the content of cluster-ca-cert file into the file myca.ca.pem. My issue here is that I don't get why it works without many parameters necessary to establish an SSL connection.
from time import sleep
from json import dumps
from kafka import KafkaProducer
producer = KafkaProducer( bootstrap_servers='test-kafka.com:443',value_serializer=lambda x: dumps(x).encode('utf-8'), security_protocol='SSL', ssl_cafile = 'myca.ca.pem', ssl_check_hostname=False)
for e in range(1000):
data = {'number' : e}
producer.send('hello', value=data)
sleep(5)
I understand what and how pub/priv key or CA are used for, but when putting all these things in SSL and Kafka context, I have difficulties understanding. Many questions come up in my mind, e.g.
Where are pub/priv keys generated by the client? In the code, I don't expose either pub key or CA anywhere, how the server can trust the client?
Why does it work with only CA from the cluster, how about pub key generated at the servers side? Same, in the code, I only copy the CA from the server, not the public key.
Does it mean there is a configuration that allows using a default value from somewhere? I am sure there is, just want to know which part of the SSL handshake is configured.
Thanks in advance,

Twisted SNI with deferreds

In our system, virtual hosts configuration is stored in redis. During connection setup, when the SNI is received, we would like to query redis for the correct certificate and key pair to use for the TLS connection and create a new Context instance with that attached.
The bulk of the code is similar to the accepted answer here: Twisted listenSSL virtualhosts
The issue we are facing is that, since accessing the certificates involves an additional network operation, we would like to make the set_tlsext_servername_callback function return a deferred.
Is there a way to tell Twisted/pyOpenSSL to wait until the deferred fires?
Edit: I found this link which seems promising, but falls short of providing a solution: https://mta.openssl.org/pipermail/openssl-dev/2015-January/000480.html
You can find an example of Twisted and SNI here: https://pypi.python.org/pypi/txsni. I would really, really like that callback to be able to take a Deferred. I think that the way to do this would be to pause the underlying transport from delivering any further bytes either in or out (stopReading/stopWriting) and then resume when the Deferred fires, after doing the rest of the SNI dance. However, I'm not even sure if this is possible with OpenSSL, because the SNI is received with the rest of ClientHello and you may need to be able to react immediately to serve the correct certificate. In this worst-possible-case scenario, you could feed the first chunk of bytes you receive into a dummy memory-BIO, wait for the TLS handshake, throw it away and never deliver any generated responses, and then don't initialize your "real" sub-transport until you've decided on which context object to use.
Hope this helps - and if you figure it out, please contribute a patch to TxSNI or Twisted!

Io: Protocol 'https' unsupported

I am trying to fetch a file over HTTPS in Io language:
url := URL with("https://api.example.com")
url fetch println
And I get this:
Error_0x7f97e1509a80:
location = "/opt/local/lib/io/addons/Socket/io/URL.io:232"
message = "Protocol 'https' unsupported"
I was trying to find something on the net, but, as everybody knows, it's not easy because of the name. I only found this thread http://tech.groups.yahoo.com/group/iolanguage/message/10898 but that's quite old.
How can I get the HTTPS support in Io?
EDIT
I've found that there is a SecureSocket addon, a wrapper over OpenSSL, in Io's source. It wasn't installed when I did sudo port io install on my MacBook with Mountain Lion, though. I tried building it from source, but no luck. It didn't build for me on a Linux machine, either.
EDIT2
I just tried to build Io from source (git clone https://github.com/stevedekorte/io.git) again (using the included script build.sh) and it turned out that cmake did detect OpenSSL:
-- Found OpenSSL: /usr/lib/libssl.dylib;/usr/lib/libcrypto.dylib
But then the SecureSocket addon is not built. Its readme file: https://github.com/stevedekorte/io/tree/master/addons/SecureSocket says:
The DTLS1 bindings are not usable unless the patches in this file are
applied to OpenSSL 0.9.8e. However, this patch includes a
deactivation of the handshake retransmission code in d1_both.c,
making it unsuitable for production environments. I take no
responsibility, etc, etc. If you want to use it anyway, apply the
patches(gathered from various newsgroups and my own experimentation)
and uncomment the commented-out block of build.io. For what it's
worth, DTLS support in OpenSSL is new as of 0.9.8 and is pretty buggy
to begin with. It's a nice idea, but it doesn't seem to be
production ready at all yet. These bindings are no exception.
If you can't get io to do it your best option would be calling an external tool like wget or curl which can and then loading the file/result locally or returning it via a pipe.
For anybody else interested in another workaround, it should be possible to put stud in front of an Io program which will do the SSL stuff. I have not tested that myself yet.
stud - The Scalable TLS Unwrapping Daemon stud is a network proxy that
terminates TLS/SSL connections and forwards the unencrypted traffic to
some backend. It's designed to handle 10s of thousands of connections
efficiently on multicore machines.

Opening an SSL web-browser connection in HtmlUnit library

I've searched through web for couple hours on this issue, and none of the answers I found didn't really fit into my problem, so here's me, asking my first-ever question in SOF.
So, I'm trying to open a web-browser from a java program using the htmlunit library. The web site I need to connect requires SSL connection, and the certificate is stored in a USB key. Its iKey2023 product.
The system used to work(I did not write it), but one of the certificates in the USB key expired, so it automatically moved on to the next one (there were 4 certificates in total), and it suddenly stopped working.
It is giving me javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated error.
I'm back home now and I forgot the exact name of the method, but I remember the following.
Browser instance is created, using IE8
browser.setWebConnection method was called. This method, according to the API, is an internal API.
Make connection to the website by passing the URL as parameter
It's throwing the exception at step 3.
Some more details. The little details might be incorrect but I'm trying to describe a big picture.
At step 2, the method requites WebConnection object as a parameter, and there is a implementation of that interface. Within this implementation, a keystore is created using sun.security.pkcs11.SunPKCS11(configFileInputStream) (did I spell that correctly?)
It was sth like this.
Provider p = new sun.security.pkcs11.SunPKCS11(configFileInputStream);
Security.addProvider(p);
And create a keystore from this provider.
Using this keystore, within the WebConnection implementation, it creates a SSLSocket.
So, after the certificate has been switched to a new one, it's not picking up the certificate correctly.
Here's what I've tried.
I've tried to use different methods in the htmlunit library, something like setSecurityProvider, and I tried to put the Provider object created in above code snippet. I got class cast exception.
I tried to manually set the system properties(trustStore, trustStorePassword, keyStore, etc). In order to do this, I wanted to export the certificate out of the USB key, but it did not let me export the private key out from it, so I could not really create a valid PKCS12 file out of it (openSSL wanted a private key file along with .pem file for conversion, and I did not have that key file).
They did not work, and I'm so stuck right now.
I have a similar issue. In my case, an admin changed the certificate and I began encountering the same SSLPeerUnverifiedException.
I found that I can set the WebClient to use insecureSSL (prior to calling getPage())and I will no longer get the exception.
webClient.setUseInsecureSSL(true);
This however, doesn't resolve the issue as the server basically doesn't authenticate the client.
Its as if the WebClient is storing something that doesn't work with the new certificate.