hang in google/protobuf/pyext/_message.so at exit - tensorflow

Question

This is TensorFlow 1.0.1 installed via pip.
It runs via an embedded CPython (libpython).
Sometimes (maybe 30% of my runs) it hangs in Py_Finalize(), and I see this backtrace:
/work/asr2/zeyer/sprint-executables/20160902.235443.fad8965.linux-x86_64-standard/Flf/flf-tool.linux-intel-standard(_ZN17AssertionsPrivate15safe_stackTraceEi+0x21)[0xc5b891]
/work/asr2/zeyer/sprint-executables/20160902.235443.fad8965.linux-x86_64-standard/Flf/flf-tool.linux-intel-standard[0xc5b8ef]
/u/zeyer/tools/glibc217/libpthread.so.0(+0x113d0)[0x2b6d89bad3d0]
/u/zeyer/tools/glibc217/libpthread.so.0(raise+0x29)[0x2b6d89bad2a9]
/u/zeyer/py-envs/py2-ubuntu16/local/lib/python2.7/site-packages/faulthandler.so(+0x3198)[0x2b6dc2372198]
/u/zeyer/tools/glibc217/libpthread.so.0(+0x113d0)[0x2b6d89bad3d0]
/u/zeyer/py-envs/py2-ubuntu16/local/lib/python2.7/site-packages/google/protobuf/pyext/_message.so(+0xaa943)[0x2b6dc14f0943]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(+0x160f6b)[0x2b6d8b23af6b]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(+0xc8f0e)[0x2b6d8b1a2f0e]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(+0x15d747)[0x2b6d8b237747]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyDict_SetItem+0x7b)[0x2b6d8b23becb]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(_PyModule_Clear+0xb5)[0x2b6d8b278565]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyImport_Cleanup+0x437)[0x2b6d8b2280e7]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(Py_Finalize+0xfe)[0x2b6d8b1fed9e]
/work/asr2/zeyer/sprint-executables/20160902.235443.fad8965.linux-x86_64-standard/Flf/flf-tool.linux-intel-standard(_ZN6Python11Initializer19AtExitUninitHandlerEv+0x2e)[0xff80de]
/u/zeyer/tools/glibc217/libc.so.6(+0x39fe8)[0x2b6d8bc39fe8]
/u/zeyer/tools/glibc217/libc.so.6(+0x3a035)[0x2b6d8bc3a035]
/u/zeyer/tools/glibc217/libc.so.6(__libc_start_main+0xf7)[0x2b6d8bc20837]
/work/asr2/zeyer/sprint-executables/20160902.235443.fad8965.linux-x86_64-standard/Flf/flf-tool.linux-intel-standard[0x7d6991]
or with GDB:
(gdb) bt full
#0 0x00002b6dc14f0943 in std::tr1::_Hashtable<google::protobuf::DescriptorPool const*, std::pair<google::protobuf::DescriptorPool const* const, google::protobuf::python::PyDescriptorPool*>, std::allocator<std::pair<google::protobuf::DescriptorPool const* const, google::protobuf::python::PyDescriptorPool*> >, std::_Select1st<std::pair<google::protobuf::DescriptorPool const* const, google::protobuf::python::PyDescriptorPool*> >, std::equal_to<google::protobuf::DescriptorPool const*>, google::protobuf::hash<google::protobuf::DescriptorPool const*>, std::tr1::__detail::_Mod_range_hashing, std::tr1::__detail::_Default_ranged_hash, std::tr1::__detail::_Prime_rehash_policy, false, false, true>::erase (
__k=#0x7ffd1bbea740: 0x8269780, this=0x2b6dc1826e40 <google::protobuf::python::descriptor_pool_map>)
at /opt/rh/devtoolset-2/root/usr/include/c++/4.8.2/tr1/hashtable.h:1041
__slot = <optimized out>
__saved_slot = <optimized out>
__code = 136746880
__n = 0
__result = 0
#1 google::protobuf::python::cdescriptor_pool::Dealloc (self=0x2b6dc0d86880)
at google/protobuf/pyext/descriptor_pool.cc:152
No locals.
#2 0x00002b6d8b23af6b in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
No symbol table info available.
#3 0x00002b6d8b1a2f0e in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
No symbol table info available.
#4 0x00002b6d8b237747 in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
No symbol table info available.
#5 0x00002b6d8b23becb in PyDict_SetItem () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
No symbol table info available.
#6 0x00002b6d8b278565 in _PyModule_Clear () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
No symbol table info available.
#7 0x00002b6d8b2280e7 in PyImport_Cleanup () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
No symbol table info available.
#8 0x00002b6d8b1fed9e in Py_Finalize () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
No symbol table info available.
#9 0x0000000000ff80de in Python::Initializer::AtExitUninitHandler() ()
No symbol table info available.
#10 0x00002b6d8bc39fe8 in ?? () from /u/zeyer/tools/glibc217/libc.so.6
No symbol table info available.
#11 0x00002b6d8bc3a035 in exit () from /u/zeyer/tools/glibc217/libc.so.6
No symbol table info available.
#12 0x00002b6d8bc20837 in __libc_start_main () from /u/zeyer/tools/glibc217/libc.so.6
No symbol table info available.
#13 0x00000000007d6991 in _start ()
No symbol table info available.
I.e. it happens in _PyModule_Clear, and then inside google/protobuf/pyext/_message.so, that's why I think this is TF related.
In the case when it does not hang, I see this output:
Exception AttributeError: AttributeError("'NoneType' object has no attribute 'raise_exception_on_not_ok_status'",) in <bound method Session.__del__ of <tensorflow.python.client.session.Session object at 0x2afd625b12d0>> ignored
I also asked upstream on TF but they suggested to post it here.
Any idea why it might hang and how to resolve this?

Answer 1

Note that this crash was happening inside a callback via std::atexit. I guess the problem is that some stuff from Google or std is cleaned up before I call Py_Finalize from my atexit-handler which leads to this crash. I think this should not happen though.
Anyway, I kind of worked around the problem by not using std::atexit now but by using my own exit handler logic instead (which however would not work if I directly use exit() anywhere).