twisted threadpool import fail - twisted

Question

Context:
I am trying to launch graphite with pypy interpreter.
Error:
launching graphite(*) leads to
ImportError: cannot import name 'threadpool',
even though launching a python (pypy) interpreter and typing from twisted.python import threadpool works.
Full stacktrace:
15/09/2014 13:29:09 :: File "app_main.py", line 75, in run_toplevel
15/09/2014 13:29:09 :: File "/opt/graphite/bin/carbon-cache.py", line 30, in <module>
15/09/2014 13:29:09 :: run_twistd_plugin(__file__)
15/09/2014 13:29:09 :: File "/opt/graphite/lib/carbon/util.py", line 93, in run_twistd_plugin
15/09/2014 13:29:09 :: runApp(config)
15/09/2014 13:29:09 :: File "/home/vagrant/test2/site-packages/twisted/scripts/twistd.py", line 23, in runApp
15/09/2014 13:29:09 :: File "/home/vagrant/test2/site-packages/twisted/application/app.py", line 380, in run
15/09/2014 13:29:09 :: File "/home/vagrant/test2/site-packages/twisted/scripts/_twistd_unix.py", line 193, in postApplication
15/09/2014 13:29:09 :: File "/home/vagrant/test2/site-packages/twisted/scripts/_twistd_unix.py", line 390, in startApplication
15/09/2014 13:29:09 :: File "/home/vagrant/test2/site-packages/twisted/application/app.py", line 658, in startApplication
15/09/2014 13:29:09 :: File "/home/vagrant/test2/site-packages/twisted/application/service.py", line 282, in startService
15/09/2014 13:29:09 :: File "/home/vagrant/test2/site-packages/twisted/application/service.py", line 282, in startService
15/09/2014 13:29:09 :: File "/opt/graphite/lib/carbon/writer.py", line 191, in startService
15/09/2014 13:29:09 :: reactor.callInThread(writeForever)
15/09/2014 13:29:09 :: File "/home/vagrant/test2/site-packages/twisted/internet/base.py", line 997, in callInThread
15/09/2014 13:29:09 :: File "/home/vagrant/test2/site-packages/twisted/internet/base.py", line 989, in getThreadPool
15/09/2014 13:29:09 :: File "/home/vagrant/test2/site-packages/twisted/internet/base.py", line 954, in _initThreadPool
15/09/2014 13:29:09 :: ImportError: cannot import name 'threadpool'
I am using pypy (2.3.1-linux_x86_64-portable) on centos 6.5
and have run pip to install twisted, whisper (and applied an additional patch)
(*) test2/bin/python /opt/graphite/bin/carbon-cache.py --instance=a start
Edit:
the virtualenv is called test2.
test2/site-packages/twisted/python/threadpool.py{,c} shows twisted has threadpool
test2/bin/pypy --info shows [usemodules] thread = True
'twisted.python.threadpool' in sys.modules returns false just before the failing import
Edit2:
adding from twisted.python import threadpool earlier in the call stack (in /opt/graphite/lib/carbon/util.py for instance) works and make graphite work.

Answer 1

Does your installation of Twisted have the threadpool module? Look for a file named threadpool.py in the twisted/python/ directory. If this is missing, your installation of Twisted has been corrupted somehow. You might be able to fix it by re-installing Twisted (perhaps destroying your ... virtualenv? and creating a new one).
Does it have the stdlib threading module? Twisted's threading support only works if the underlying Python runtime supports threads. If this is missing you may need a different Python runtime. PyPy supports threads but perhaps you got a build with threads disabled somehow.
If neither of these is the problem, you might learn more by running Python with import debugging enabled. python -v enables minimal import debugging and python -vv enables more verbose import debugging. I'm not sure whether these behave the same way on PyPy as they do on CPython. Hopefully they do or things are a bit more difficult.
If that doesn't help then you might also try adding a breakpoint with pdb just before the import of the twisted.python.threadpool module and then stepping carefully through, inspecting state as you go. One thing to check could be whether sys.modules has a 'twisted.python.threadpool' item in it already, and if so, what value is associated with it. None will prevent the import system from even looking for a module implementation on disk and just fail.

Answer 2

I had a similar problem. The problem was that in carbon.conf's USER setting, I specified my own user "graphite". When you do this, the carbon-cache daemon will run as this user. The graphite user did not have permissions to the twisted modules. Solution was
chown -R graphite /usr/lib64/python2.6/site-packages/twisted