TwistedWeb on multicore/multiprocessor - twisted

Question

What techniques are people using to utilize multiple processors/cores when running a TwistedWeb server? Is there a recommended way of doing it?
My twisted.web based web service is running on Amazon EC2 instances, which often have multiple CPU cores (8, 16), and the type of work that the service is doing benefits from extra processing power, so i would very much like to use that.
I understand that it is possible to use haproxy, squid or a web server, configured as a reverse proxy, in front of multiple instances of Twisted. In fact, we are currently using such a setup, with nginx serving as a reverse proxy to several upstream twisted.web services running on the same host, but each on different port.
This works fine, but what i'm really interested in, is a solution where there is no "front-facing" server, but all twistd processes somehow bind to the same socket and accept requests. Is such thing even possible... or am i being crazy? The operating system is Linux (CentOS).
Thanks.
Anton.

Answer 1

There are a number of ways to support multiprocess operation for a Twisted application. One important question to answer at the start, though, is what you expect your concurrency model to be, and how your application deals with shared state.
In a single process Twisted application, concurrency is all cooperative (with help from Twisted's asynchronous I/O APIs) and shared state can be kept anywhere a Python object would go. Your application code runs knowing that, until it gives up control, nothing else will run. Additionally, any part of your application that wants to access some piece of shared state can probably do so quite easily, since that state is probably kept in a boring old Python object that is easy to access.
When you have multiple processes, even if they're all running Twisted-based applications, then you have two forms of concurrency. One is the same as for the previous case - within a particular process, the concurrency is cooperative. However, you have a new kind, where multiple processes are running. Your platform's process scheduler might switch execution between these processes at any time, and you have very little control over this (as well as very little visibility into when it happens). It might even schedule two of your processes to run simultaneously on different cores (this is probably even what you're hoping for). This means that you lose some guarantees about consistency, since one process doesn't know when a second process might come along and try to operate on some shared state. This leads in to the other important area of consideration, how you will actually share state between the processes.
Unlike the single process model, you no longer have any convenient, easily accessed places to store your state where all your code can reach it. If you put it in one process, all the code in that process can access it easily as a normal Python object, but any code running in any of your other processes no longer has easy access to it. You might need to find an RPC system to let your processes communicate with each other. Or, you might architect your process divide so that each process only receives requests which require state stored in that process. An example of this might be a web site with sessions, where all state about a user is stored in their session, and their sessions are identified by cookies. A front-end process could receive web requests, inspect the cookie, look up which back-end process is responsible for that session, and then forward the request on to that back-end process. This scheme means that back-ends typically don't need to communicate (as long as your web application is sufficiently simple - ie, as long as users don't interact with each other, or operate on shared data).
Note that in that example, a pre-forking model is not appropriate. The front-end process must exclusively own the listening port so that it can inspect all incoming requests before they are handled by a back-end process.
Of course, there are many types of application, with many other models for managing state. Selecting the right model for multi-processing requires first understanding what kind of concurrency makes sense for your application, and how you can manage your application's state.
That being said, with very new versions of Twisted (unreleased as of this point), it's quite easy to share a listening TCP port amongst multiple processes. Here is a code snippet which demonstrates one way you might use some new APIs to accomplish this:
from os import environ
from sys import argv, executable
from socket import AF_INET
from twisted.internet import reactor
from twisted.web.server import Site
from twisted.web.static import File
def main(fd=None):
root = File("/var/www")
factory = Site(root)
if fd is None:
# Create a new listening port and several other processes to help out.
port = reactor.listenTCP(8080, factory)
for i in range(3):
reactor.spawnProcess(
None, executable, [executable, __file__, str(port.fileno())],
childFDs={0: 0, 1: 1, 2: 2, port.fileno(): port.fileno()},
env=environ)
else:
# Another process created the port, just start listening on it.
port = reactor.adoptStreamPort(fd, AF_INET, factory)
reactor.run()
if __name__ == '__main__':
if len(argv) == 1:
main()
else:
main(int(argv[1]))
With older versions, you can sometimes get away with using fork to share the port. However, this is rather error prone, fails on some platforms, and isn't a supported way to use Twisted:
from os import fork
from twisted.internet import reactor
from twisted.web.server import Site
from twisted.web.static import File
def main():
root = File("/var/www")
factory = Site(root)
# Create a new listening port
port = reactor.listenTCP(8080, factory)
# Create a few more processes to also service that port
for i in range(3):
if fork() == 0:
# Proceed immediately onward in the children.
# The parent will continue the for loop.
break
reactor.run()
if __name__ == '__main__':
main()
This works because of the normal behavior of fork, where the newly created process (the child) inherits all of the memory and file descriptors from the original process (the parent). Since processes are otherwise isolated, the two processes don't interfere with each other, at least as far as the Python code they are executing goes. Since the file descriptors are inherited, either the parent or any of the children can accept connections on the port.
Since forwarding HTTP requests is such an easy task, I doubt you'll notice much of a performance improvement using either of these techniques. The former is a bit nicer than proxying, because it simplifies your deployment and works for non-HTTP applications more easily. The latter is probably more of a liability than it's worth accepting.

Answer 2

The recommended way IMO is to use haproxy (or another load balancer) like you already are, the bottleneck shouldn't be the load balancer if configured correctly. Besides, you'll want to have some fallover method which haproxy provides in case one of your processes goes down.
It isn't possible to bind multiple processes to the same TCP socket, but it is possible with UDP.

Answer 3

If you wish to serve your web content over HTTPS as well, this is what you will need to do on top of #Jean-Paul's snippet.
from twisted.internet.ssl import PrivateCertificate
from twisted.protocols.tls import TLSMemoryBIOFactory
'''
Original snippet goes here
..........
...............
'''
privateCert = PrivateCertificate.loadPEM(open('./server.cer').read() + open('./server.key').read())
tlsFactory = TLSMemoryBIOFactory(privateCert.options(), False, factory)
reactor.adoptStreamPort(fd, AF_INET, tlsFactory)
By using fd, you will serve either HTTP or HTTPS but not both.
If you wish to have both, listenSSL on the parent process and include the ssl fd you get from the ssl port as the second argument when spawning the child process.
Complete snipper is here:
from os import environ
from sys import argv, executable
from socket import AF_INET
from twisted.internet import reactor
from twisted.web.server import Site
from twisted.web.static import File
from twisted.internet import reactor, ssl
from twisted.internet.ssl import PrivateCertificate
from twisted.protocols.tls import TLSMemoryBIOFactory
def main(fd=None, fd_ssl=None):
root = File("/var/www")
factory = Site(root)
spawned = []
if fd is None:
# Create a new listening port and several other processes to help out.
port = reactor.listenTCP(8080, factory)
port_ssl = reactor.listenSSL(8443, factory, ssl.DefaultOpenSSLContextFactory('./server.key', './server.cer'))
for i in range(3):
child = reactor.spawnProcess(
None, executable, [executable, __file__, str(port.fileno()), str(port_ssl.fileno())],
childFDs={0: 0, 1: 1, 2: 2, port.fileno(): port.fileno(), port_ssl.fileno(): port_ssl.fileno()},
env=environ)
spawned.append(child)
else:
# Another process created the port, just start listening on it.
port = reactor.adoptStreamPort(fd, AF_INET, factory)
cer = open('./server.cer')
key = open('./server.key')
pem_data = cer.read() + key.read()
cer.close()
pem.close()
privateCert = PrivateCertificate.loadPEM(pem_data )
tlsFactory = TLSMemoryBIOFactory(privateCert.options(), False, factory)
reactor.adoptStreamPort(fd_ssl, AF_INET, tlsFactory)
reactor.run()
for p in spawned:
p.signalProcess('INT')
if __name__ == '__main__':
if len(argv) == 1:
main()
else:
main(int(argv[1:]))