Juila Dataframe rollup by groups (aka subtotals) - dataframe

Question

What is a concise way to express rollup aggregations in DataFrames.jl?
Example dataset:
+---+----------+-----+---------+------+
| id| date_col|group| item|amount|
+---+----------+-----+---------+------+
| 1|2020-03-11| A|BOO00OXXX| 1.0|
| 2|2020-03-11| A|BOO00OXXY| 2.0|
| 3|2020-03-11| B|BOO00OXXZ| 17.0|
| 4|2020-03-12| B|BOO00OXXA| 9.0|
| 5|2020-03-12| B|BOO00OXXB| 1.0|
| 6|2020-03-12| B|BOO00OXXY| 5.0|
| 7|2020-03-13| C|BOO00OXXY| 2.0|
| 8|2020-03-13| C|BOO00OXXX| 1.0|
| 9|2020-03-13| C|BOO00OXXY| 2.0|
+---+----------+-----+---------+------+
# desired output
+------+---------+
|group |total_amt|
+------+---------+
|ROLLUP| 40.0|
| A | 3.0|
| B | 32.0|
| C | 5.0|
+------+---------+
I commonly need to summarize a dataset, sometimes for sharing reports, which aggregates values over certain columns with subtotals and grand totals. These are called 'rollups' or 'subtotals'/'grand totals' in Excel.
In Spark these are conveniently generated with rollup or cube aggregations. The above result is generated with the following spark api call.
How can I produce a similar table with Julia DataFrames.jl?
// scala spark
df.rollup("group")
.agg(sum("amount").as("total_amt"))
.orderBy("group")
.show()
+-----+---------+
|group|total_amt|
+-----+---------+
| null| 40.0|
| A| 3.0|
| B| 32.0|
| C| 5.0|
+-----+---------+
// note the aggregated column label is null for the subtotal (aka rollup)
NOTE: I am able to produce the result with multiple julia groupby() and combine() operations, and then union or vcat the result into a single dataframe. I need and want a concise and readable idiom.
EDIT: adding a specific julia implementation to show why I want something more concise.
using DataFrames, Dates
df = DataFrame(id = [1,2,3,4,5,6,7,8,9]
, date_col = Date.(["2020-03-11","2020-03-11","2020-03-11","2020-03-12","2020-03-12","2020-03-12","2020-03-13","2020-03-13","2020-03-13"])
, group = ["A","A","B","B","B","B","C","C","C"]
, amount = [1.0,2.0,17.0,9.0,1.0,5.0,2.0,1.0,2.0]
)
# replicate the spark.rollup example
df1 = combine(groupby(_, :group), :amount => sum => :total_amt);
df2 = combine(df, :amount => sum => :total_amt);
df2[:, :group] = [missing];
df_result = sort(vcat(df1, df2, cols = :setequal), rev = true)
4×2 DataFrame
Row │ group total_amt
│ String? Float64
─────┼────────────────────
1 │ missing 40.0
2 │ C 5.0
3 │ B 32.0
4 │ A 3.0
Adding a version of #bkamins answer, sticking with combine()
I think I prefer this answer so far, as it maintains a bit of symmetry and if made into a function is easier to see where the arguments would go.
using Chain
#chain df begin
groupby(:group)
combine(:amount => sum => :total_amt)
append!(insertcols!(combine(df, :amount => sum => :total_amt), :group => "ROLLUP"))
sort(:total_amt, rev = true)
end

Answer 1

This is how I would do it:
julia> using DataFrames, Chain
julia> df = DataFrame(group=["A", "A", "B", "B", "C", "C"], amount=1:6)
6×2 DataFrame
Row │ group amount
│ String Int64
─────┼────────────────
1 │ A 1
2 │ A 2
3 │ B 3
4 │ B 4
5 │ C 5
6 │ C 6
julia> #chain df begin
groupby(:group)
combine(:amount => sum => :total_amount)
push!(_, (missing, sum(_.total_amount)), promote=true)
sort(:total_amount, rev=true)
end
4×2 DataFrame
Row │ group total_amount
│ String? Int64
─────┼───────────────────────
1 │ missing 21
2 │ C 11
3 │ B 7
4 │ A 3
This will be efficient and hopefully you find it readable.
As #jling commented we do not have in-built rollup.

Answer 2

Here is an answer with DataFramesMeta.jl
julia> using DataFramesMeta;
julia> #chain df begin
groupby(:group)
#combine :total_amount = sum(:amount)
#aside df2 = #combine df :total_amount = sum(:amount)
vcat(df2; cols = :union)
end
4×2 DataFrame
Row │ group total_amount
│ String? Int64
─────┼───────────────────────
1 │ A 3
2 │ B 7
3 │ C 11
4 │ missing 21

Answer 3

julia> df
5×2 DataFrame
Row │ g amt
│ Int64 Int64
─────┼──────────────
1 │ 0 2
2 │ 1 1
3 │ 1 1
4 │ 0 1
5 │ 1 1
julia> combine(groupby(df, :g), :amt => sum => :total_amt)
2×2 DataFrame
Row │ g total_amt
│ Int64 Int64
─────┼─────────────────────
1 │ 0 3
2 │ 1 3
#alternative do-block syntax:
julia> combine(groupby(df, :g)) do sub_df
(total_amt = sum(sub_df.amt),)
end
2×2 DataFrame
Row │ g total_amt
│ Int64 Int64
─────┼──────────────────
1 │ 0 3
2 │ 1 3
does this more or less do what you want? btw the relevant docs: https://dataframes.juliadata.org/stable/man/split_apply_combine/
I feel like we would need a feel iteration to solve all the things you might want to do in Spark, SO is hard to do those kind of back and forth.