Is there an easy way to run DataFrames::by in parallel? - dataframe

Question

I have a large dataframe I want to compute on in parallel. The call I want to parallelize is
df = by(df, [:Chromosome], some_func)
Is there a way to easily parallelize this? Preferably without any copying.
Also, I guess the kinds of parallelization used should be different depending on the size of the groups created by by.
Minimal reproducible example to use in answers:
using DataFrames, CSV, Pkg
iris = CSV.read(joinpath(Pkg.dir("DataFrames"), "test/data/iris.csv"))
iris_count = by(iris, [:Species], nrow)

Answer 1

On Windows run in console (adjust to the number of cores/threads you have):
$ set JULIA_NUM_THREADS=4
$ julia
On Linux run in console:
$ export JULIA_NUM_THREADS=4
$ julia
Now check if it works:
julia> Threads.nthreads()
4
Run the code below (I update your code to match Julia 1.0):
using CSV, DataFrames, BenchmarkTools
iris = CSV.read(joinpath(dirname(pathof(DataFrames)),"..","test/data/iris.csv"))
iris.PetalType = iris.PetalWidth .> 2; #add an additional column for testing
Let us define some function that operates on a part of a DataFrame
function nrow2(df::AbstractDataFrame)
val = nrow(df)
#do something much more complicated...
val
end
Now the most complicated part of the puzzle comes:
function par_by(df::AbstractDataFrame,f::Function,cols::Symbol...;block_size=40)
#f needs to be precompiled - we precompile using the first row of the DataFrame.
#If try to do it within #thread macro
#Julia will crash in most ugly and unexpected ways
#if you comment out this line you can observe a different crash with every run
by(view(df,1:1),[cols...],f);
nr = nrow(df)
local dfs = DataFrame()
blocks = Int(ceil(nr/block_size))
s = Threads.SpinLock()
Threads.#threads for block in 1:blocks
startix = (block-1)*block_size+1
endix = min(block*block_size,nr)
rv= by(view(df,startix:endix), [cols...], f)
Threads.lock(s)
if nrow(dfs) == 0
dfs = rv
else
append!(dfs,rv)
end
Threads.unlock(s)
end
dfs
end
Let's test it and aggregate the results
julia> res = par_by(iris,nrow2,:Species)
6×2 DataFrame
│ Row │ Species │ x1 │
│ │ String │ Int64 │
├─────┼────────────┼───────┤
│ 1 │ versicolor │ 20 │
│ 2 │ virginica │ 20 │
│ 3 │ setosa │ 10 │
│ 4 │ versicolor │ 30 │
│ 5 │ virginica │ 30 │
│ 6 │ setosa │ 40 │
julia> by(res, :Species) do df;DataFrame(x1=sum(df.x1));end
3×2 DataFrame
│ Row │ Species │ x1 │
│ │ String │ Int64 │
├─────┼────────────┼───────┤
│ 1 │ setosa │ 50 │
│ 2 │ versicolor │ 50 │
│ 3 │ virginica │ 50 │
The par_by also supports multiple columns
julia> res = par_by(iris,nrow2,:Species,:PetalType)
8×3 DataFrame
│ Row │ Species │ PetalType │ x1 │
│ │ String │ Bool │ Int64 │
├─────┼───────────┼───────────┼───────┤
│ 1 │ setosa │ false │ 40 │
⋮
│ 7 │ virginica │ true │ 13 │
│ 8 │ virginica │ false │ 17 │
#Bogumił Kamiński commented that it is reasonable to use groupby() before threading. Unless for some reason groupby cost is to high (requires full scan) this is the recommended way - makes the aggregation simpler.
ress = DataFrame(Species=String[],count=Int[])
for group in groupby(iris,:Species)
r = par_by(group,nrow2,:Species,block_size=15)
push!(ress,[r.Species[1],sum(r.x1)])
end
julia> ress
3×2 DataFrame
│ Row │ Species │ count │
│ │ String │ Int64 │
├─────┼────────────┼───────┤
│ 1 │ setosa │ 50 │
│ 2 │ versicolor │ 50 │
│ 3 │ virginica │ 50 │
Note that in the example above are only three groups so we parallelize over each group. However, if you have big number of groups you could consider running:
function par_by2(df::AbstractDataFrame,f::Function,cols::Symbol...)
res = NamedTuple[]
s = Threads.SpinLock()
groups = groupby(df,[cols...])
f(view(groups[1],1:1));
Threads.#threads for g in 1:length(groups)
rv= f(groups[g])
Threads.lock(s)
key=tuple([groups[g][cc][1] for cc in cols]...)
push!(res,(key=key,val=rv))
Threads.unlock(s)
end
res
end
julia> iris.PetalType = iris.PetalWidth .> 2;
julia> par_by2(iris,nrow2,:Species,:PetalType)
4-element Array{NamedTuple,1}:
(key = ("setosa", false), val = 50)
(key = ("versicolor", false), val = 50)
(key = ("virginica", true), val = 23)
(key = ("virginica", false), val = 27)
Let me know if it worked for you.
Since more people might have similar problem I will make this code into a Julia package (and that is why I kept this code to be very general)

Answer 2

Start julia with julia -p 4 then run
using CSV, DataFrames
iris = CSV.read(joinpath(dirname(pathof(DataFrames)),"..","test/data/iris.csv"))
g = groupby(iris, :Species)
pmap(nrow, [i for i in g])
This will run groupby in parallel.