Pybind11 multiprocessing hangs - python-multiprocessing

Question

I'm writing an application that use Pybind11 to embed Python interpreter (Windows, 64 bit, Visual C++ 2017). From Python, I need to spawn multiple processes, but it seems doesn't works. I try the following code as test:
import multiprocessing
import os
import sys
import time
print("This is the name of the script: ", sys.argv[0])
print("Number of arguments: ", len(sys.argv))
print("The arguments are: " , str(sys.argv))
prefix=str(os.getpid())+"-"
if len(sys.argv) > 1:
__name__ = "__mp_main__"
def print_cube(num):
"""
function to print cube of given num
"""
print("Cube: {}".format(num * num * num))
def print_square(num):
"""
function to print square of given num
"""
print("Square: {}".format(num * num))
print(__name__)
if __name__ == "__main__":
print(prefix, "checkpoint 1")
# creating processes
p1 = multiprocessing.Process(target=print_square, args=(10, ))
p1.daemon = True
p2 = multiprocessing.Process(target=print_cube, args=(10, ))
# starting process 1
p1.start()
print(prefix, "checkpoint 2")
# starting process 2
p2.start()
print(prefix, "checkpoint 3")
# wait until process 1 is finished
print(prefix, "checkpoint 4")
p1.join()
print(prefix, "checkpoint 5")
# wait until process 2 is finished
p2.join()
print(prefix, "checkpoint 6")
# both processes finished
print("Done!")
print(prefix, "checkpoint 7")
time.sleep(10)
Running it with the Python from command prompt, I obtain:
This is the name of the script: mp.py
Number of arguments: 1
The arguments are: ['mp.py']
__main__
12872- checkpoint 1
12872- checkpoint 2
This is the name of the script: C:\tmp\mp.py
Number of arguments: 1
The arguments are: ['C:\\tmp\\mp.py']
__mp_main__
7744- checkpoint 7
Square: 100
12872- checkpoint 3
12872- checkpoint 4
12872- checkpoint 5
This is the name of the script: C:\tmp\mp.py
Number of arguments: 1
The arguments are: ['C:\\tmp\\mp.py']
__mp_main__
15020- checkpoint 7
Cube: 1000
12872- checkpoint 6
Done!
12872- checkpoint 7
which is correct. If I try the same from a C++ project with Pybind11, the output is:
This is the name of the script: C:\AGPX\Documenti\TestPyBind\x64\Debug\TestPyBind.exe
Number of arguments: 1
The arguments are: ['C:\\AGPX\\Documenti\\TestPyBind\\x64\\Debug\\TestPyBind.exe']
__main__
4440- checkpoint 1
This is the name of the script: C:\AGPX\Documenti\TestPyBind\x64\Debug\TestPyBind.exe
Number of arguments: 4
The arguments are: ['C:\\AGPX\\Documenti\\TestPyBind\\x64\\Debug\\TestPyBind.exe', '-c', 'from multiprocessing.spawn import spawn_main; spawn_main(parent_pid=4440, pipe_handle=128)', '--multiprocessing-fork']
__mp_main__
10176- checkpoint 7
Note that, in this case, the variable __name__ is always set to '__main__', so I have to change it manually (for the spawned processes) to '__mp_main__' (I can detect the child processes thanks to the sys.argv). This is the first strange behaviour.
The parent process have pid 4440 and I can see the process in process explorer.
The first child process have pid 10176 and it reach the end 'checkpoint 7' and process disappears from process explorer. However, the main process doesn't print 'checkpoint 2', that is looks like it hangs on 'p1.start()' and I cannot understand why.
The complete C++ code is:
#include <pybind11/pybind11.h>
#include <pybind11/stl.h>
#include <pybind11/stl_bind.h>
#include <pybind11/embed.h>
#include <iostream>
namespace py = pybind11;
using namespace py::literals;
int wmain(int argc, wchar_t **argv)
{
py::initialize_interpreter();
PySys_SetArgv(argc, argv);
std::string pyCode = std::string(R"(
import multiprocessing
import os
import sys
import time
print("This is the name of the script: ", sys.argv[0])
print("Number of arguments: ", len(sys.argv))
print("The arguments are: " , str(sys.argv))
prefix=str(os.getpid())+"-"
if len(sys.argv) > 1:
__name__ = "__mp_main__"
def print_cube(num):
"""
function to print cube of given num
"""
print("Cube: {}".format(num * num * num))
def print_square(num):
"""
function to print square of given num
"""
print("Square: {}".format(num * num))
print(__name__)
if __name__ == "__main__":
print(prefix, "checkpoint 1")
# creating processes
p1 = multiprocessing.Process(target=print_square, args=(10, ))
p1.daemon = True
p2 = multiprocessing.Process(target=print_cube, args=(10, ))
# starting process 1
p1.start()
print(prefix, "checkpoint 2")
# starting process 2
p2.start()
print(prefix, "checkpoint 3")
# wait until process 1 is finished
print(prefix, "checkpoint 4")
p1.join()
print(prefix, "checkpoint 5")
# wait until process 2 is finished
p2.join()
print(prefix, "checkpoint 6")
# both processes finished
print("Done!")
print(prefix, "checkpoint 7")
time.sleep(10)
)");
try
{
py::exec(pyCode);
} catch (const std::exception &e) {
std::cout << e.what();
}
py::finalize_interpreter();
}
Can anyone explain to me how to overcome this problem, please?
Thanks in advance (and I apologize for my english).

Answer 1

Ok, thanks to this link: https://blender.stackexchange.com/questions/8530/how-to-get-python-multiprocessing-module-working-on-windows, I solved this strange issue (that seems to be Windows related).
It's not a Pybind11 issue, but a Python C API itself.
You can solve the issue by setting sys.executable equals to the path of the python interpreter executable (python.exe) and by writing the python code to a file and setting the path to the __file__ variable. That is, I have to add:
import sys
sys.executable = "C:\\Users\\MyUserName\\Miniconda3\\python.exe"
__file__ = "C:\\tmp\\run.py"
and I need to write the python code to the file specified by __file__, that is:
FILE *f = nullptr;
fopen_s(&f, "c:\\tmp\\run.py", "wt");
fprintf(f, "%s", pyCode.c_str());
fclose(f);
just before execute py::exec(pyCode).
In addition the code:
if len(sys.argv) > 1:
__name__ = "__mp_main__"
is no longer necessary. However, note that in this way the runned processes are not embedded anymore and, unfortunately, if you want to directly pass a C++ module to them, you cannot do it.
Hope this can help someone else.