I have a DataFrame variable like this:
julia> data
11×7 DataFrames.DataFrame
│ Row │ Time │ Wind1VelX │ Wind1VelY │ Wind1VelZ │ TwrBsFxt │ TwrBsFyt │ TwrBsFzt │
│ 1 │ 0.0 │ 25.17 │ 0.944 │ -0.1424 │ 325.4 │ -123.2 │ -6726.0 │
│ 2 │ 0.01 │ 25.62 │ 0.592 │ -0.335 │ 338.7 │ -131.0 │ -6749.0 │
│ 3 │ 0.02 │ 26.07 │ 0.24 │ -0.5275 │ 345.7 │ -141.7 │ -6754.0 │
I would like to know if there is a method to get the column names in an array of String like:
julia> header=["Time", "Wind1VelX", "Wind1VelY", "Wind1VelZ", "TwrBsFxt", "TwrBsFyt", "TwrBsFzt"]
Thanks in advance
You can write:
String.(names(data))
Without String., like this:
names(data)
you will get a vector of Symbol.
Note that calling String, in this case, converts a single Symbol to String and by adding a dot . after it you broadcast it over all elements of a vector returned by names(data).
Get a Vector of Strings:
names(data)
Get a Vector of Symbols:
propertynames(data)
Related
I am looking for the opposite of the dropmissing function in DataFrames.jl so that the user knows where to look to fix their bad data. It seems like this should be easy, but the filter function expects a column to be specified and I cannot get it to iterate over all columns.
julia> df=DataFrame(a=[1, missing, 3], b=[4, 5, missing])
3×2 DataFrame
│ Row │ a │ b │
│ │ Int64? │ Int64? │
├─────┼─────────┼─────────┤
│ 1 │ 1 │ 4 │
│ 2 │ missing │ 5 │
│ 3 │ 3 │ missing │
julia> filter(x -> ismissing(eachcol(x)), df)
ERROR: MethodError: no method matching eachcol(::DataFrameRow{DataFrame,DataFrames.Index})
julia> filter(x -> ismissing.(x), df)
ERROR: ArgumentError: broadcasting over `DataFrameRow`s is reserved
I am basically trying to recreate the disallowmissing function, but with a more useful error message.
Here are two ways to do it:
julia> df = DataFrame(a=[1, missing, 3], b=[4, 5, missing])
3×2 DataFrame
│ Row │ a │ b │
│ │ Int64? │ Int64? │
├─────┼─────────┼─────────┤
│ 1 │ 1 │ 4 │
│ 2 │ missing │ 5 │
│ 3 │ 3 │ missing │
julia> df[.!completecases(df), :] # this will be faster
2×2 DataFrame
│ Row │ a │ b │
│ │ Int64? │ Int64? │
├─────┼─────────┼─────────┤
│ 1 │ missing │ 5 │
│ 2 │ 3 │ missing │
julia> #view df[.!completecases(df), :]
2×2 SubDataFrame
│ Row │ a │ b │
│ │ Int64? │ Int64? │
├─────┼─────────┼─────────┤
│ 1 │ missing │ 5 │
│ 2 │ 3 │ missing │
julia> filter(row -> any(ismissing, row), df)
2×2 DataFrame
│ Row │ a │ b │
│ │ Int64? │ Int64? │
├─────┼─────────┼─────────┤
│ 1 │ missing │ 5 │
│ 2 │ 3 │ missing │
julia> filter(row -> any(ismissing, row), df, view=true) # requires DataFrames.jl 0.22
2×2 SubDataFrame
Row │ a b
│ Int64? Int64?
─────┼──────────────────
1 │ missing 5
2 │ 3 missing
I have the following Dataframe
using DataFrames, Statistics
df = DataFrame(name=["John", "Sally", "Kirk"],
age=[23., 42., 59.],
children=[3,5,2], height = [180, 150, 170])
print(df)
3×4 DataFrame
│ Row │ name │ age │ children │ height │
│ │ String │ Float64 │ Int64 │ Int64 │
├─────┼────────┼─────────┼──────────┼────────┤
│ 1 │ John │ 23.0 │ 3 │ 180 │
│ 2 │ Sally │ 42.0 │ 5 │ 150 │
│ 3 │ Kirk │ 59.0 │ 2 │ 170 │
I can compute the mean of a column as follow:
println(mean(df[:4]))
166.66666666666666
Now I want to get the mean of all the numeric column and tried this code:
x = [2,3,4]
for i in x
print(mean(df[:x[i]]))
end
But got the following error message:
MethodError: no method matching getindex(::Symbol, ::Int64)
Stacktrace:
[1] top-level scope at ./In[64]:3
How can I solve the problem?
You are trying to access the DataFrame's column using an integer index specifying the column's position. You should just use the integer value without any : before i, which would create the symbol :i but you do not a have column named i.
x = [2,3,4]
for i in x
println(mean(df[i])) # no need for `x[i]`
end
You can also index a DataFrame using a Symbol denoting the column's name.
x = [:age, :children, :height];
for c in x
println(mean(df[c]))
end
You get the following error in your attempt because you are trying to access the ith index of the symbol :x, which is an undefined operation.
MethodError: no method matching getindex(::Symbol, ::Int64)
Note that :4 is just 4.
julia> :4
4
julia> typeof(:4)
Int64
Here is a one-liner that actually selects all Number columns:
julia> mean.(eachcol(df[findall(x-> x<:Number, eltypes(df))]))
3-element Array{Float64,1}:
41.333333333333336
3.3333333333333335
166.66666666666666
For many scenarios describe is actually more convenient:
julia> describe(df)
4×8 DataFrame
│ Row │ variable │ mean │ min │ median │ max │ nunique │ nmissing │ eltype │
│ │ Symbol │ Union… │ Any │ Union… │ Any │ Union… │ Nothing │ DataType │
├─────┼──────────┼─────────┼──────┼────────┼───────┼─────────┼──────────┼──────────┤
│ 1 │ name │ │ John │ │ Sally │ 3 │ │ String │
│ 2 │ age │ 41.3333 │ 23.0 │ 42.0 │ 59.0 │ │ │ Float64 │
│ 3 │ children │ 3.33333 │ 2 │ 3.0 │ 5 │ │ │ Int64 │
│ 4 │ height │ 166.667 │ 150 │ 170.0 │ 180 │ │ │ Int64 │
In the question println(mean(df[4])) works as well (instead of println(mean(df[:4]))).
Hence we can write
x = [2,3,4]
for i in x
println(mean(df[i]))
end
which works
I need to convert the following DataFrame
julia> df = DataFrame(:A=>["", "2", "3"], :B=>[1.1, 2.2, 3.3])
which looks like
3×2 DataFrame
│ Row │ A │ B │
│ │ String │ Float64 │
├─────┼────────┼─────────┤
│ 1 │ │ 1.1 │
│ 2 │ 2 │ 2.2 │
│ 3 │ 3 │ 3.3 │
I would like to convert A column from Array{String,1} to array of Int with missing values.
I tried
julia> df.A = tryparse.(Int, df.A)
3-element Array{Union{Nothing, Int64},1}:
nothing
2
3
julia> df
3×2 DataFrame
│ Row │ A │ B │
│ │ Union… │ Float64 │
├─────┼────────┼─────────┤
│ 1 │ │ 1.1 │
│ 2 │ 2 │ 2.2 │
│ 3 │ 3 │ 3.3 │
julia> eltype(df.A)
Union{Nothing, Int64}
but I'm getting A column with elements of type Union{Nothing, Int64}.
nothing (of type Nothing) and missing (of type Missing) seems to be 2 differents kind of value.
So I wonder how I can A columns with missing values instead?
I also wonder if missing and nothing leads to different performance.
I would have done the following:
julia> df.A = map(x->begin val = tryparse(Int, x)
ifelse(typeof(val) == Nothing, missing, val)
end, df.A)
3-element Array{Union{Missing, Int64},1}:
missing
2
3
julia> df
3×2 DataFrame
│ Row │ A │ B │
│ │ Int64⍰ │ Float64 │
├─────┼─────────┼─────────┤
│ 1 │ missing │ 1.1 │
│ 2 │ 2 │ 2.2 │
│ 3 │ 3 │ 3.3 │
I think missing is more suitable for dataframes which indeed have missing values, instead of nothing, because the latter is more considered as a void in C, or None in Python, see here.
As a side note, Missing type has some Julia functionalities.
Replacing nothing by missing can simply be done using replace:
julia> df.A = replace(df.A, nothing=>missing)
3-element Array{Union{Missing, Int64},1}:
missing
2
3
julia> df
3×2 DataFrame
│ Row │ A │ B │
│ │ Int64⍰ │ Float64 │
├─────┼─────────┼─────────┤
│ 1 │ missing │ 1.1 │
│ 2 │ 2 │ 2.2 │
│ 3 │ 3 │ 3.3 │
an other solution is to use tryparsem function defined as following
tryparsem(T, str) = something(tryparse(T, str), missing)
and use it like
julia> df = DataFrame(:A=>["", "2", "3"], :B=>[1.1, 2.2, 3.3])
julia> df.A = tryparsem.(Int, df.A)
df3[10, :A] = missing
df3[15, :B] = missing
df3[15, :C] = missing
Even NA is not working.
I am getting an error
MethodError: Cannot convert an object of type Missings.Missing to an object of type Int64
This may have arisen from a call to the constructor Int64(...),
since type constructors fall back to convert methods.
Stacktrace:
[1] setindex!(::Array{Int64,1}, ::Missings.Missing, ::Int64) at ./array.jl:583
[2] insert_single_entry!(::DataFrames.DataFrame, ::Missings.Missing, ::Int64, ::Symbol) at /home/jrun/.julia/v0.6/DataFrames/src/dataframe/dataframe.jl:361
[3] setindex!(::DataFrames.DataFrame, ::Missings.Missing, ::Int64, ::Symbol) at /home/jrun/.julia/v0.6/DataFrames/src/dataframe/dataframe.jl:448
[4] include_string(::String, ::String) at ./loading.jl:522
Use allowmissing! function.
julia> using DataFrames
julia> df = DataFrame(a=[1,2,3])
3×1 DataFrame
│ Row │ a │
│ │ Int64 │
├─────┼───────┤
│ 1 │ 1 │
│ 2 │ 2 │
│ 3 │ 3 │
julia> df.a[1] = missing
ERROR: MethodError: Cannot `convert` an object of type Missing to an object of type Int64
julia> allowmissing!(df)
3×1 DataFrame
│ Row │ a │
│ │ Int64⍰ │
├─────┼────────┤
│ 1 │ 1 │
│ 2 │ 2 │
│ 3 │ 3 │
julia> df.a[1] = missing
missing
julia> df
3×1 DataFrame
│ Row │ a │
│ │ Int64⍰ │
├─────┼─────────┤
│ 1 │ missing │
│ 2 │ 2 │
│ 3 │ 3 │
You can see which columns in a DataFrame allow missing because they are highlighted with ⍰ after type name under column name.
You can also use allowmissing function to create a new DataFrame.
Both functions optionally accept columns that are to be converted.
Finally there is a disallowmissing/disallowmissing! pair that does the reverse (i.e. strips optional Missing union from eltype if a vector actually contains no missing values).
In 0.6 I was using:
colnames = ["Date_Time","Date_index","Time_index"]
names!(data1_date_time_index.colindex, map(parse, colnames))
What is the syntax for v1.0 - right now .colindex is not found.
Per DataFrames docs:
rename!(data1_date_time_index, f => t for (f, t) =
zip([:x1, :x1_1, :x1_2],
[:Date_Time, :Date_index, :Time_index]))
Assuming data1_date_time_index is a DataFrame that has three columns use:
colnames = ["Date_Time","Date_index","Time_index"]
names!(data1_date_time_index, Symbol.(colnames))
I am not 100% sure if this is what you want, as your example was not fully reproducible (so if actually you needed something else can you please submit full code that can be run).
The problem with data1_date_time_index.colindex is that currently . is used to access columns of a DataFrame by their name (and not fields of DataFrame type). In general you are not recommended to use colindex as it is not part of exposed API and might change in the future. If you really need to reach it use getfield(data_frame_name, :colindex).
EDIT
In DataFrames 0.20 you should write:
rename!(data1_date_time_index, Symbol.(colnames))
and in DataFrames 0.21 (which will be released before summer 2020) also passing strings directly will most probably be allowed like this:
rename!(data1_date_time_index, colnames)
(see here for a related discussion)
You can rename column through select also
For Ex:
df = DataFrame(col1 = 1:4, col2 = ["John", "James", "Finch", "May"])
│ Row │ col1 │ col2 │
│ │ Int64 │ String │
├─────┼───────┼────────┤
│ 1 │ 1 │ John │
│ 2 │ 2 │ James │
│ 3 │ 3 │ Finch │
│ 4 │ 4 │ May │
select(df, "col1" => "Id", "col2" => "Name")
│ Row │ Id │ Name │
│ │ Int64 │ String │
├─────┼───────┼────────┤
│ 1 │ 1 │ John │
│ 2 │ 2 │ James │
│ 3 │ 3 │ Finch │
│ 4 │ 4 │ May │
Rename columns:
names!(df, [:c1,:c2,:c3]) #(all)
rename!(df, Dict(:oldCol => :newCol)) # (a selection)
(from: https://syl1.gitbook.io/julia-language-a-concise-tutorial/useful-packages/dataframes )