How do you load a module #everywhere inside a function in Julia - module

Question

I am trying to load a module in my workers after creating them with addprocs. When addprocs is called at the top level, everything is fine. However, I am not able to do the same thing when I wrap the code in a function.
In my case, I am adding workers dynamically, so it is not feasible to call #everywhere using XXX always on the top level, I would need to do this inside a function.
In short, this works:
addprocs(1)
#everywhere using XXX
and this doesn't:
function myaddprocs()
addprocs(1)
#everywhere using XXX
end
Any ideas?

Answer 1

After a bit more investigation, I have pinpointed some of the issues which were making my code not work.
Imports must happen after addprocs. If an import has happened before, the import must be prefixed by an #everywhere.
Top-level expressions (such as using) inside functions do not work, unless wrapped in an eval statement.
A fix to my code would be:
function myaddprocs()
addprocs(1)
eval(macroexpand(quote #everywhere using XXX end))
end
Examples
I have tested the following snippets on Julia 0.6.1. I have also tested them using the same version on a SGE cluster (OGS/GE 2011.11p1), by substituting all addprocs by addprocs_sge, and importing ClusterManagers.jl. The following snippets work:
using after addprocs:
addprocs(1)
using SpecialFunctions
pmap(x->SpecialFunctions.sinint(1), workers())
using before and after addprocs, the second with #everywhere:
using SpecialFunctions
addprocs(1)
#everywhere using SpecialFunctions
pmap(x->sinint(1), workers())
using wrapped in eval after addprocs within function
function getprocs()
addprocs(1)
eval(Expr(:using,:SpecialFunctions))
pmap(x->SpecialFunctions.sinint(1), workers())
end
getprocs()
Same as before with with #everywhere applied to eval
function getprocs()
addprocs(1)
#everywhere eval(Expr(:using,:SpecialFunctions))
pmap(x->sinint(1), workers())
end
getprocs()
Same as before with with #everywhere within eval instead
function getprocs()
addprocs(1)
eval(macroexpand(quote #everywhere using SpecialFunctions end))
pmap(x->sinint(1), workers())
end
getprocs()
These snippets, on the other hand, do not work:
using before addprocs
using SpecialFunctions
addprocs(1)
pmap(x->SpecialFunctions.sinint(1), workers())
using before and after addprocs
using SpecialFunctions
addprocs(1)
using SpecialFunctions
pmap(x->SpecialFunctions.sinint(1), workers())
using within function
using SpecialFunctions
function getprocs()
addprocs(1)
#everywhere using SpecialFunctions
pmap(x->sinint(1), workers())
end
getprocs()
using within function
function getprocs()
addprocs(1)
using SpecialFunctions
pmap(x->SpecialFunctions.sinint(1), workers())
end
getprocs()

Answer 2

you don't need #everywhere. this works for me:
addprocs()
using mymodule # loads code on all procs but brings in to scope only on master process.
this is what you want if you want to then do pmap(x->fun(x),workers()), and fun is exported from mymodule. you can read this up here: https://docs.julialang.org/en/release-0.6/manual/parallel-computing/#Code-Availability-and-Loading-Packages-1

Answer 3

Cako's solution is helpful but I had to add the module as first argument to macroexpand :
eval(macroexpand(Distributed,quote #everywhere using DistributedArrays end))