I need to do something quite specific and i'm trying to do it the good way , especially i want it to be optimized .
So i have a DataFrame that look like this :
v = ["x","y","z"][rand(1:3, 10)]
df = DataFrame(Any[collect(1:10), v, rand(10)], [:USER_ID, :GENRE_MAIN, :TOTAL_LISTENED])
Row │ USER_ID GENRE_MAIN TOTAL_LISTENED
│ Int64 String Float64
─────┼─────────────────────────────────────
1 │ 1 x 0.237186
12 │ 1 y 0.237186
13 │ 1 x 0.254486
2 │ 2 z 0.920804
3 │ 3 y 0.140626
4 │ 4 x 0.653306
5 │ 5 x 0.83126
6 │ 6 x 0.928973
7 │ 7 y 0.519728
8 │ 8 x 0.409969
9 │ 9 z 0.798064
10 │ 10 x 0.701332
I want to aggregate it by user (i have many rows per user_id ) and do many calculations
I need to calculate the top 1 ,2 ,3 ,4 ,5 genre, album name, artist name per user_id and its respective values (the total_listened that correspond) and it has to be like this :
USER_ID │ ALBUM1_NAME │ ALBUM2_NAME | ALBUM1_NAME_VALUE | ALBUM2_NAME_VALUES | ......│ GENRE1 │ GENRE2
One line per user_id .
I got this solution that fits 90% of what i wanted but i can't modify it to also include the values of total_listened:
using DataFrames, Pipe, Random, Pkg
Pkg.activate(".")
Pkg.add("DataFrames")
Pkg.add("Pipe")
Random.seed!(1234)
df = DataFrame(USER_ID=rand(1:10, 80),
GENRE_MAIN=rand(string.("genre_", 1:6), 80),
ALBUM_NAME=rand(string.("album_", 1:6), 80),
ALBUM_ARTIST_NAME=rand(string.("artist_", 1:6), 80))
function top5(sdf, col, prefix)
return #pipe groupby(sdf, col) |>
combine(_, nrow) |>
sort!(_, :nrow, rev=true) |>
first(_, 5) |>
vcat(_[!, 1], fill(missing, 5 - nrow(_))) |>
DataFrame([string(prefix, i) for i in 1:5] .=> _)
end
#pipe groupby(df, :USER_ID) |>
combine(_,
x -> top5(x, :GENRE_MAIN, "genre"),
x -> top5(x, :ALBUM_NAME, "album"),
x -> top5(x, :ALBUM_ARTIST_NAME, "artist"))
An example :
for the user 1 of the DataFrame just up i want the result to be :
Row │ USER_ID GENRE1 GENRE2 GENRE1_VALUE GENRE2_VALUE ......
│ Int64 String String Float64 Float64
─────┼─────────────────────────────────────────────────────
1 │ 1 x y 0.491672 0.237186. ......
I took only GENRE here , but i also want it for ALBUM_NAME, ALBUM_ARTIST_NAME
I also want after to do a top rank % ,
Order the users by total_listened and calculate their percentile.
to rank them by top5% , top10%, top20% of the total
I can calculate the tagetted quantile i want with
x = .05
quantile(df.TOTAL_LISTENED, x)
and then just put all the users's total_listened that is superior to this quantile
but i don't know how to calculate it properly in the combine...
Thank you
As commented in the previous post - I would recommend you to ask a specific question not to redo your whole project on StackOverflow (if you need such help https://discourse.julialang.org/ is a good place to discuss, especially that you need many steps of the analysis and they require a precise definition of what you want exactly - also it would be best if on https://discourse.julialang.org/ you shared your full data set, as the sampler you provide here is not enough to do a proper analysis later since it is too small).
Here is an example how to add totals columns (I assume that you want data to be ordered by the totals):
julia> using Random, DataFrames, Pipe
julia> Random.seed!(1234);
julia> df = DataFrame([rand(1:10, 100), rand('a':'k', 100), rand(100)],
[:USER_ID, :GENRE_MAIN, :TOTAL_LISTENED]);
julia> function top5(sdf, col, prefix)
#pipe groupby(sdf, col) |>
combine(_, :TOTAL_LISTENED => sum => :SUM) |>
sort!(_, :SUM, rev=true) |>
first(_, 5) |>
vcat(_[!, 1], fill(missing, 5 - nrow(_)),
_[!, 2], fill(missing, 5 - nrow(_))) |>
DataFrame([[string(prefix, i) for i in 1:5];
[string(prefix, i, "_VALUE") for i in 1:5]] .=> _)
end;
julia> #pipe groupby(df, :USER_ID) |>
combine(_, x -> top5(x, :GENRE_MAIN, "genre"))
10×11 DataFrame
Row │ USER_ID genre1 genre2 genre3 genre4 genre5 genre1_VALUE genre2_VALUE genre3_VALUE genre4_VALUE genre5_VALUE
│ Int64 Char Char Char Char Char? Float64 Float64 Float64 Float64 Float64?
─────┼───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
1 │ 1 d b j e i 2.34715 2.014 1.68587 0.693472 0.377869
2 │ 4 b e d c missing 0.90263 0.589418 0.263121 0.107839 missing
3 │ 8 c d i k j 1.55335 1.40416 0.977785 0.779468 0.118024
4 │ 2 a e f g k 1.34841 0.901507 0.87146 0.797606 0.669002
5 │ 10 a e f i d 1.60554 1.07311 0.820425 0.757363 0.678598
6 │ 7 f i g c a 2.59654 1.49654 1.15944 0.670488 0.258173
7 │ 9 i b e a g 1.57373 0.954117 0.603848 0.338918 0.133201
8 │ 5 f g c k d 1.33899 0.722283 0.664457 0.54016 0.507337
9 │ 3 d c f h e 1.63695 0.919088 0.544296 0.531262 0.0540101
10 │ 6 d g f j i 1.68768 0.97688 0.333207 0.259212 0.0636912
Related
I have a dataframe with two columns a and b and at the moment both are looking like column a, but I want to add separators so that column b looks like below. I have tried using the package format.jl. But I haven't gotten the result I'm afte. Maybe worth mentioning is that both columns is Int64 and the column names a and b is of type symbol.
a | b
150000 | 1500,00
27 | 27,00
16614 | 166,14
Is there some other way to solve this than using format.jl? Or is format.jl the way to go?
Assuming you want the commas in their typical positions rather than how you wrote them, this is one way:
julia> using DataFrames, Format
julia> f(x) = format(x, commas=true)
f (generic function with 1 method)
julia> df = DataFrame(a = [1000000, 200000, 30000])
3×1 DataFrame
Row │ a
│ Int64
─────┼─────────
1 │ 1000000
2 │ 200000
3 │ 30000
julia> transform(df, :a => ByRow(f) => :a_string)
3×2 DataFrame
Row │ a a_string
│ Int64 String
─────┼────────────────────
1 │ 1000000 1,000,000
2 │ 200000 200,000
3 │ 30000 30,000
If you instead want the row replaced, use transform(df, :a => ByRow(f), renamecols=false).
If you just want the output vector rather than changing the DataFrame, you can use format.(df.a, commas=true)
You could write your own function f to achieve the same behavior, but you might as well use the one someone already wrote inside the Format.jl package.
However, once you transform you data to Strings as above, you won't be able to filter/sort/analyze the numerical data in the DataFrame. I would suggest that you apply the formatting in the printing step (rather than modifying the DataFrame itself to contain strings) by using the PrettyTables package. This can format the entire DataFrame at once.
julia> using DataFrames, PrettyTables
julia> df = DataFrame(a = [1000000, 200000, 30000], b = [500, 6000, 70000])
3×2 DataFrame
Row │ a b
│ Int64 Int64
─────┼────────────────
1 │ 1000000 500
2 │ 200000 6000
3 │ 30000 70000
julia> pretty_table(df, formatters = ft_printf("%'d"))
┌───────────┬────────┐
│ a │ b │
│ Int64 │ Int64 │
├───────────┼────────┤
│ 1,000,000 │ 500 │
│ 200,000 │ 6,000 │
│ 30,000 │ 70,000 │
└───────────┴────────┘
(Edited to reflect the updated specs in the question)
julia> df = DataFrame(a = [150000, 27, 16614]);
julia> function insertdecimalcomma(n)
if n < 100
return string(n) * ",00"
else
return replace(string(n), r"(..)$" => s",\1")
end
end
insertdecimalcomma (generic function with 1 method)
julia> df.b = insertdecimalcomma.(df.a)
julia> df
3×2 DataFrame
Row │ a b
│ Int64 String
─────┼─────────────────
1 │ 150000 1500,00
2 │ 27 27,00
3 │ 16614 166,14
Note that the b column will necessarily be a String after this change, as integer types cannot store formatting information in them.
If you have a lot of data and find that you need better performance, you may also want to use the InlineStrings package:
julia> #same as before upto the function definition
julia> using InlineStrings
julia> df.b = inlinestrings(insertdecimalcomma.(df.a))
3-element Vector{String7}:
"1500,00"
"27,00"
"166,14"
This stores the b column's data as fixed-size strings (String7 type here), which are generally treated like normal Strings, but can be significantly better for performance.
is there a way to double quote all fields when outputting a DataFrame to a csv in Julia? I am having trouble find an answer with Google.
In python I would add quoting=csv.QUOTE_ALL to df.to_csv(file)
I am having trouble finding something similar with CSV.write(file,df)
You can do the following:
julia> using CSV, DataFrames
julia> io = IOBuffer()
IOBuffer(data=UInt8[...], readable=true, writable=true, seekable=true, append=false, size=0, maxsize=Inf,ptr=1, mark=-1)
julia> df = DataFrame(rand(1:10, 3, 5), :auto)
3×5 DataFrame
Row │ x1 x2 x3 x4 x5
│ Int64 Int64 Int64 Int64 Int64
─────┼───────────────────────────────────
1 │ 6 10 5 4 4
2 │ 1 9 6 5 3
3 │ 5 4 5 8 4
julia> CSV.write(io, df; quotestrings=true, transform=(col,val)->string(val)) |> take! |> String |> println
"x1","x2","x3","x4","x5"
"6","10","5","4","4"
"1","9","6","5","3"
"5","4","5","8","4"
The trouble is that quotestrings only forces quoting strings (so that when you read back the file numbers are not quoted and correctly parsed) and therefore you need also transform argument to force every value to be written as string.
I was able to extract the column of a DataFrame that I want using a regular expression, but now I want to extract from that DataFrame column a String with the column name and a Vector with the data. How can I construct f and g below? Alternate approaches also welcome.
julia> df = DataFrame("x (in)" => 1:3, "y (°C)" => 4:6)
3×2 DataFrame
Row │ x (in) y (°C)
│ Int64 Int64
─────┼────────────────
1 │ 1 4
2 │ 2 5
3 │ 3 6
julia> y = df[:, r"y "]
3×1 DataFrame
Row │ y (°C)
│ Int64
─────┼────────
1 │ 4
2 │ 5
3 │ 6
julia> y_units = f(y)
"°C"
julia> y_data = g(y)
3-element Vector{Int64}:
4
5
6
f(df) = only(names(df))
g(df) = only(eachcol(df)) # or df[!, 1] if you do not need to check that this is the only column
(only is used to check that the data frame actually has only one column)
An alternate approach to get the column name without creating an intermediate data frame is just writing:
julia> names(df, r"y ")
1-element Vector{String}:
"y (°C)"
to extract out the column name (you need to get the first element of this vector)
I am diving into Julia, hence my "novice"-question.
Coming from R and Python, I am used to apply simple functions (arithmetic or otherwise) to entire pandas.DataFrames and data.frames, respectively.
#both R and Python
df - 1 # returns all values -1, given all values are numeric
df == "someString" # returns a boolean df
a bit more complex
#python
df = df.applymap(lambda v: v - 1 if v > 1 else v)
#R
df[] <- lapply(df, function(x) ifelse(x>1,x-1,x))
The thing is, I don't know how to do this in Julia, I don't find analogue solutions easily on the web. And Stackoverflow helps a lot when using Google. So here it is. How do I do it in Julia?
Thanks for your help!
PS:
So far I have come up with the following solutions, where I loos my column names.
DataFrame(colwise(x -> x .-1, df))
# seems like to much code for only subtracting 1 and loosing col names
Please update your DataFrames.jl installation to version 1.4.2.
You can do all you want using broadcasting like this:
julia> df = DataFrame(rand(2,3), :auto)
2×3 DataFrame
Row │ x1 x2 x3
│ Float64 Float64 Float64
─────┼──────────────────────────────
1 │ 0.720264 0.759493 0.998702
2 │ 0.726994 0.560153 0.243982
julia> df .+ 1
2×3 DataFrame
Row │ x1 x2 x3
│ Float64 Float64 Float64
─────┼───────────────────────────
1 │ 1.72026 1.75949 1.9987
2 │ 1.72699 1.56015 1.24398
julia> df .< 0.5
2×3 DataFrame
Row │ x1 x2 x3
│ Bool Bool Bool
─────┼─────────────────────
1 │ false false false
2 │ false false true
julia> df2 = string.(df)
2×3 DataFrame
Row │ x1 x2 x3
│ String String String
─────┼────────────────────────────────────────────────────────────
1 │ 0.7202642575401104 0.7594928463144177 0.9987024771396766
2 │ 0.7269944483236035 0.5601527006649413 0.2439815742224939
julia> parse.(Float64, df2)
2×3 DataFrame
Row │ x1 x2 x3
│ Float64 Float64 Float64
─────┼──────────────────────────────
1 │ 0.720264 0.759493 0.998702
2 │ 0.726994 0.560153 0.243982
Is this what you wanted?
I am trying out the Julia DataFrames module. I am interested in it so I can use it to plot simple simulations in Gadfly. I want to be able to iteratively add rows to the dataframe and I want to initialize it as empty.
The tutorials/documentation on how to do this is sparse (most documentation describes how to analyse imported data).
To append to a nonempty dataframe is straightforward:
df = DataFrame(A = [1, 2], B = [4, 5])
push!(df, [3 6])
This returns.
3x2 DataFrame
| Row | A | B |
|-----|---|---|
| 1 | 1 | 4 |
| 2 | 2 | 5 |
| 3 | 3 | 6 |
But for an empty init I get errors.
df = DataFrame(A = [], B = [])
push!(df, [3, 6])
Error message:
ArgumentError("Error adding 3 to column :A. Possible type mis-match.")
while loading In[220], in expression starting on line 2
What is the best way to initialize an empty Julia DataFrame such that you can iteratively add items to it later in a for loop?
A zero length array defined using only [] will lack sufficient type information.
julia> typeof([])
Array{None,1}
So to avoid that problem is to simply indicate the type.
julia> typeof(Int64[])
Array{Int64,1}
And you can apply that to your DataFrame problem
julia> df = DataFrame(A = Int64[], B = Int64[])
0x2 DataFrame
julia> push!(df, [3 6])
julia> df
1x2 DataFrame
| Row | A | B |
|-----|---|---|
| 1 | 3 | 6 |
using Pkg, CSV, DataFrames
iris = CSV.read(joinpath(Pkg.dir("DataFrames"), "test/data/iris.csv"))
new_iris = similar(iris, nrow(iris))
head(new_iris, 2)
# 2×5 DataFrame
# │ Row │ SepalLength │ SepalWidth │ PetalLength │ PetalWidth │ Species │
# ├─────┼─────────────┼────────────┼─────────────┼────────────┼─────────┤
# │ 1 │ missing │ missing │ missing │ missing │ missing │
# │ 2 │ missing │ missing │ missing │ missing │ missing │
for (i, row) in enumerate(eachrow(iris))
new_iris[i, :] = row[:]
end
head(new_iris, 2)
# 2×5 DataFrame
# │ Row │ SepalLength │ SepalWidth │ PetalLength │ PetalWidth │ Species │
# ├─────┼─────────────┼────────────┼─────────────┼────────────┼─────────┤
# │ 1 │ 5.1 │ 3.5 │ 1.4 │ 0.2 │ setosa │
# │ 2 │ 4.9 │ 3.0 │ 1.4 │ 0.2 │ setosa │
The answer from #waTeim already answers the initial question. But what if I want to dynamically create an empty DataFrame and append rows to it. E.g. what if I don't want hard-coded column names?
In this case, df = DataFrame(A = Int64[], B = Int64[]) is not sufficient.
The NamedTuple A = Int64[], B = Int64[] needs to be create dynamically.
Let's assume we have a vector of column names col_names and a vector of column types colum_types from which to create an emptyDataFrame.
col_names = [:A, :B] # needs to be a vector Symbols
col_types = [Int64, Float64]
# Create a NamedTuple (A=Int64[], ....) by doing
named_tuple = (; zip(col_names, type[] for type in col_types )...)
df = DataFrame(named_tuple) # 0×2 DataFrame
Alternatively, the NameTuple could be created with
# or by doing
named_tuple = NamedTuple{Tuple(col_names)}(type[] for type in col_types )
I think at least in the latest version of Julia you can achieve this by creating a pair object without specifying type
df = DataFrame("A" => [], "B" => [])
push!(df, [5,'f'])
1×2 DataFrame
Row │ A B
│ Any Any
─────┼──────────
1 │ 5 f
as seen in this post by #Bogumił Kamiński where multiple columns are needed, something like this can be done:
entries = ["A", "B", "C", "D"]
df = DataFrame([ name =>[] for name in entries])
julia> push!(df,[4,5,'r','p'])
1×4 DataFrame
Row │ A B C D
│ Any Any Any Any
─────┼────────────────────
1 │ 4 5 r p
Or as pointed out by #Antonello below if you know that type you can do.
df = DataFrame([name => Int[] for name in entries])
which is also in #Bogumil Kaminski's original post.