After reading the context, if you felt the title could be enhanced to fit the question and you had an idea, feel free to update it.
Suppose I have the following DataFrame:
using DataFrames
df = DataFrame(
g=["a","b","a","c",missing,missing,missing,missing],
a=[1,2,3,4,missing,missing,missing,missing],
Column1=[missing,missing,missing,missing,false,false,false,true],
Column2=[missing,missing,missing,missing,false,true,true,true],
Column3=[missing,missing,missing,missing,true,true,false,false],
)
# 8×5 DataFrame
# Row │ g a Column1 Column2 Column3
# │ String? Int64? Bool? Bool? Bool?
# ─────┼─────────────────────────────────────────────
# 1 │ a 1 missing missing missing
# 2 │ b 2 missing missing missing
# 3 │ a 3 missing missing missing
# 4 │ c 4 missing missing missing
# 5 │ missing missing false false true
# 6 │ missing missing false true true
# 7 │ missing missing false true false
# 8 │ missing missing true true false
I want to convert it to this:
# 8×5 DataFrame
# Row │ g a Column1 Column2 Column3
# │ String? Int64? Bool? Bool? Bool?
# ─────┼─────────────────────────────────────────────
# 1 │ a 1 false false true
# 2 │ b 2 false true true
# 3 │ a 3 false true false
# 4 │ c 4 true true false
I tried:
DataFrame(collect.(skipmissing.(eachcol(df))), names(df))
But I think this is not an optimal way since I'm using the collect function. Is there any better way to do it?
For me a natural way to do it would be:
julia> mapcols(x -> filter(!ismissing, x), df)
4×5 DataFrame
Row │ g a Column1 Column2 Column3
│ String? Int64? Bool? Bool? Bool?
─────┼────────────────────────────────────────────
1 │ a 1 false false true
2 │ b 2 false true true
3 │ a 3 false true false
4 │ c 4 true true false
However, this assumes that number of missing values in every column is the same (but I guess this is what you have in this exercise - right?).
skipmissing is designed for cases when user wants a non-copying iterable skipping missing values (which is not the case here).
Related
I have a dataframe with two columns a and b and at the moment both are looking like column a, but I want to add separators so that column b looks like below. I have tried using the package format.jl. But I haven't gotten the result I'm afte. Maybe worth mentioning is that both columns is Int64 and the column names a and b is of type symbol.
a | b
150000 | 1500,00
27 | 27,00
16614 | 166,14
Is there some other way to solve this than using format.jl? Or is format.jl the way to go?
Assuming you want the commas in their typical positions rather than how you wrote them, this is one way:
julia> using DataFrames, Format
julia> f(x) = format(x, commas=true)
f (generic function with 1 method)
julia> df = DataFrame(a = [1000000, 200000, 30000])
3×1 DataFrame
Row │ a
│ Int64
─────┼─────────
1 │ 1000000
2 │ 200000
3 │ 30000
julia> transform(df, :a => ByRow(f) => :a_string)
3×2 DataFrame
Row │ a a_string
│ Int64 String
─────┼────────────────────
1 │ 1000000 1,000,000
2 │ 200000 200,000
3 │ 30000 30,000
If you instead want the row replaced, use transform(df, :a => ByRow(f), renamecols=false).
If you just want the output vector rather than changing the DataFrame, you can use format.(df.a, commas=true)
You could write your own function f to achieve the same behavior, but you might as well use the one someone already wrote inside the Format.jl package.
However, once you transform you data to Strings as above, you won't be able to filter/sort/analyze the numerical data in the DataFrame. I would suggest that you apply the formatting in the printing step (rather than modifying the DataFrame itself to contain strings) by using the PrettyTables package. This can format the entire DataFrame at once.
julia> using DataFrames, PrettyTables
julia> df = DataFrame(a = [1000000, 200000, 30000], b = [500, 6000, 70000])
3×2 DataFrame
Row │ a b
│ Int64 Int64
─────┼────────────────
1 │ 1000000 500
2 │ 200000 6000
3 │ 30000 70000
julia> pretty_table(df, formatters = ft_printf("%'d"))
┌───────────┬────────┐
│ a │ b │
│ Int64 │ Int64 │
├───────────┼────────┤
│ 1,000,000 │ 500 │
│ 200,000 │ 6,000 │
│ 30,000 │ 70,000 │
└───────────┴────────┘
(Edited to reflect the updated specs in the question)
julia> df = DataFrame(a = [150000, 27, 16614]);
julia> function insertdecimalcomma(n)
if n < 100
return string(n) * ",00"
else
return replace(string(n), r"(..)$" => s",\1")
end
end
insertdecimalcomma (generic function with 1 method)
julia> df.b = insertdecimalcomma.(df.a)
julia> df
3×2 DataFrame
Row │ a b
│ Int64 String
─────┼─────────────────
1 │ 150000 1500,00
2 │ 27 27,00
3 │ 16614 166,14
Note that the b column will necessarily be a String after this change, as integer types cannot store formatting information in them.
If you have a lot of data and find that you need better performance, you may also want to use the InlineStrings package:
julia> #same as before upto the function definition
julia> using InlineStrings
julia> df.b = inlinestrings(insertdecimalcomma.(df.a))
3-element Vector{String7}:
"1500,00"
"27,00"
"166,14"
This stores the b column's data as fixed-size strings (String7 type here), which are generally treated like normal Strings, but can be significantly better for performance.
I am diving into Julia, hence my "novice"-question.
Coming from R and Python, I am used to apply simple functions (arithmetic or otherwise) to entire pandas.DataFrames and data.frames, respectively.
#both R and Python
df - 1 # returns all values -1, given all values are numeric
df == "someString" # returns a boolean df
a bit more complex
#python
df = df.applymap(lambda v: v - 1 if v > 1 else v)
#R
df[] <- lapply(df, function(x) ifelse(x>1,x-1,x))
The thing is, I don't know how to do this in Julia, I don't find analogue solutions easily on the web. And Stackoverflow helps a lot when using Google. So here it is. How do I do it in Julia?
Thanks for your help!
PS:
So far I have come up with the following solutions, where I loos my column names.
DataFrame(colwise(x -> x .-1, df))
# seems like to much code for only subtracting 1 and loosing col names
Please update your DataFrames.jl installation to version 1.4.2.
You can do all you want using broadcasting like this:
julia> df = DataFrame(rand(2,3), :auto)
2×3 DataFrame
Row │ x1 x2 x3
│ Float64 Float64 Float64
─────┼──────────────────────────────
1 │ 0.720264 0.759493 0.998702
2 │ 0.726994 0.560153 0.243982
julia> df .+ 1
2×3 DataFrame
Row │ x1 x2 x3
│ Float64 Float64 Float64
─────┼───────────────────────────
1 │ 1.72026 1.75949 1.9987
2 │ 1.72699 1.56015 1.24398
julia> df .< 0.5
2×3 DataFrame
Row │ x1 x2 x3
│ Bool Bool Bool
─────┼─────────────────────
1 │ false false false
2 │ false false true
julia> df2 = string.(df)
2×3 DataFrame
Row │ x1 x2 x3
│ String String String
─────┼────────────────────────────────────────────────────────────
1 │ 0.7202642575401104 0.7594928463144177 0.9987024771396766
2 │ 0.7269944483236035 0.5601527006649413 0.2439815742224939
julia> parse.(Float64, df2)
2×3 DataFrame
Row │ x1 x2 x3
│ Float64 Float64 Float64
─────┼──────────────────────────────
1 │ 0.720264 0.759493 0.998702
2 │ 0.726994 0.560153 0.243982
Is this what you wanted?
For example, I have a DataFrame like the one seen below, let's call it df.
╔═════╦══════╦══════╦══════╗
║ Row ║ a ║ b ║ c ║
╠═════╬══════╬══════╬══════╣
║ 1 ║ 0.66 ║ 0.55 ║ 0.44 ║
╠═════╬══════╬══════╬══════╣
║ 2 ║ 0.11 ║ 0.22 ║ 0.33 ║
╠═════╬══════╬══════╬══════╣
║ 3 ║ 1.00 ║ 2.00 ║ 3.00 ║
╚═════╩══════╩══════╩══════╝
When I use writetable("output.txt",df) I receive the following type of output for the numerical data in the text file.
"Nullable{Float64}(0.66)"
instead of
0.66
Any thoughts on how to get writetable to export just the data?
EDIT:
I should note that this only occurs after importing data using the ReadStat package. Is it possible to convert an entire data set to just an Array that can be exported properly? That may solve the problem.
EDIT #2:
I just tried running the following code (utilizing a created function converter) but am receiving errors (posted below).
f(a,n)=
if typeof(a[n])==NullableArrays.NullableArray{String,1}
convert(Array{String},a[n])
elseif typeof(a[n])==NullableArrays.NullableArray{Float64,1}
convert(Array{Float64},a[n])
elseif typeof(a[n])==NullableArrays.NullableArray{Int64,1}
convert(Array{Float64},a[n])
end
converter(a)=hcat([f(a,n) for n=1:length(a)]...)
The errors received are as follows:
julia> converter(af)
ERROR: NullException()
in convert at /home/ale/.julia/v0.5/NullableArrays/src/primitives.jl:248 [inlined]
in convert(::Type{Array{Float64,N}}, ::NullableArrays.NullableArray{Float64,1}) at /home/ale/.julia/v0.5/NullableArrays/src/primitives.jl:256
in f(::DataFrames.DataFrame, ::Int64) at ./REPL[6]:5
in collect_to!(::Array{Array{T,1},1}, ::Base.Generator{UnitRange{Int64},##1#2{DataFrames.DataFrame}}, ::Int64, ::Int64) at ./array.jl:340
in collect_to!(::Array{Array{Float64,1},1}, ::Base.Generator{UnitRange{Int64},##1#2{DataFrames.DataFrame}}, ::Int64, ::Int64) at ./array.jl:350
in collect(::Base.Generator{UnitRange{Int64},##1#2{DataFrames.DataFrame}}) at ./array.jl:308
in converter(::DataFrames.DataFrame) at ./REPL[7]:1
Have a look / play a bit with the following:
julia> using DataFrames
julia> a = [Nullable(0.1), Nullable{Float64}(), Nullable(0.3)];
julia> b = [Nullable{Float64}(), Nullable(2.), Nullable(3.)];
julia> df = DataFrame(Any[a,b], [:a,:b])
3×2 DataFrames.DataFrame
│ Row │ a │ b │
├─────┼───────┼───────┤
│ 1 │ 0.1 │ #NULL │
│ 2 │ #NULL │ 2.0 │
│ 3 │ 0.3 │ 3.0 │
julia> c = [df[x] for x in names(df)];
julia> f(x) = [get(y, "Missing") for y in x];
julia> d = Any[f(x) for x in c]; # "Any" required for dataframes (I think)
julia> df2 = DataFrame(d, names(df))
│ Row │ a │ b │
├─────┼───────────┼───────────┤
│ 1 │ 0.1 │ "Missing" │
│ 2 │ "Missing" │ 2.0 │
│ 3 │ 0.3 │ 3.0 │
julia> writetable("/home/tasos/Desktop/output.txt", df2)
Note that for each column, if there is even one missing value, your numbers will also be reported inside quotes, because of the mixed array. If you want it to be all integers, you'll have to choose a different default value to "Missing" to denote your missing values (e.g. -1 if you only expect positive numbers).
If you don't like that, then you're probably better off writing your own "writetable" function; it's not that difficult, it's just a case of opening a file and printing what you want per column.
Also, to address some of our discussion in the comments:
The nullable type has two fields:
julia> fieldnames(Nullable)
2-element Array{Symbol,1}:
:hasvalue
:value
Let's create two instances to show what they mean:
julia> a = Nullable(1, true); b = Nullable(2, false);
julia> a.hasvalue, a.value
(true,1)
julia> b.hasvalue, b.value
(false,2)
You can test for nullity explicitly:
julia> isnull(a)
false
julia> isnull(b)
true
julia> isnull(0), isnull("")
(false, false) # isnull returns false by default if input is not a Nullable Type
Or you can use the "get" function to get a Nullable's value. If you don't define an alternative in the case of null, you get a NullException:
julia> get(a)
1
julia> get(b)
ERROR: NullException()
Stacktrace:
[1] get(::Nullable{Int64}) at ./nullable.jl:92
julia> get(b, "Null Detected")
"Null Detected"
A Nullable instance defined as Nullable(1, false) has a .value of 1, but this is superfluous as it is declared as .hasvalue=false and is therefore effectively null (though you can query the .value if you really want to).
A Nullable instance defined as n = Nullable{Float64}() will give you a nullable instance with .hasvalue=false and a meaningless value, presumably whatever was in memory at that location during instantiation, though interpreted as whatever Type of Nullable you declared (i.e. Float64):
julia> n = Nullable{Float64}()
Nullable{Float64}()
julia> n.hasvalue, n.value
(false, 6.9015724352651e-310)
Is there something like R's table function in Julia? I've read about xtab, but do not know how to use it.
Suppose we have R's data.frame rdata which col6 is of the Factor type.
R sample code:
rdata <- read.csv("mycsv.csv") #1
table(rdata$col6) #2
In order to read data and make factors in Julia I do it like this:
using DataFrames
jldata = readtable("mycsv.csv", makefactors=true) #1 :col6 will be now pooled.
..., but how to build R's table like in julia (how to achieve #2)?
You can use the countmap function from StatsBase.jl to count the entries of a single variable. General cross tabulation and statistical tests for contingency tables are lacking at this point. As Ismael points out, this has been discussed in the issue tracker for StatsBase.jl.
I came to the conclusion that a similar effect can be achieved using by:
Let jldata consists of :gender column.
julia> by(jldata, :gender, nrow)
3x2 DataFrames.DataFrame
| Row | gender | x1 |
|-----|----------|-------|
| 1 | NA | 175 |
| 2 | "female" | 40254 |
| 3 | "male" | 58574 |
Of course it's not a table but at least I get the same data type as the datasource. Surprisingly by seems to be faster than countmap.
I believe, "by" is depreciated in Julia as of 1.5.3 (It says: ERROR: ArgumentError: by function was removed from DataFrames.jl).
So here are some alternatives, we can use split apply combine to do a cross tabs as well or use FreqTables.
Using Split Combine:
Example 1 - SingleColumn:
using RDatasets
using DataFrames
mtcars = dataset("datasets", "mtcars")
## To do a table on cyl column
gdf = groupby(mtcars, :Cyl)
combine(gdf, nrow)
Output:
# 3×2 DataFrame
# Row │ Cyl nrow
# │ Int64 Int64
# ─────┼──────────────
# 1 │ 6 7
# 2 │ 4 11
# 3 │ 8 14
Example 2 - CrossTabs Between 2 columns:
## we have to just change the groupby code a little bit and rest is same
gdf = groupby(mtcars, [:Cyl, :AM])
combine(gdf, nrow)
Output:
#6×3 DataFrame
# Row │ Cyl AM nrow
# │ Int64 Int64 Int64
#─────┼─────────────────────
# 1 │ 6 1 3
# 2 │ 4 1 8
# 3 │ 6 0 4
# 4 │ 8 0 12
# 5 │ 4 0 3
# 6 │ 8 1 2
Also on a side note if you don't like the name as nrow on top, you can use :
combine(gdf, nrow => :Count)
to change the name to Count
Alternate way: Using FreqTables
You can use package, FreqTables like below to do count and proportion very easily, to add it you can use Pkg.add("FreqTables") :
## Cross tab between cyl and am
freqtable(mtcars.Cyl, mtcars.AM)
## Proportion between cyl and am
prop(freqtable(mtcars.Cyl, mtcars.AM))
## with margin like R you can use it too in this (columnwise proportion: margin=2)
prop(freqtable(mtcars.Cyl, mtcars.AM), margins=2)
## with margin for rowwise proportion: margin = 1
prop(freqtable(mtcars.Cyl, mtcars.AM), margins=1)
Outputs:
## count cross tabs
#3×2 Named Array{Int64,2}
#Dim1 ╲ Dim2 │ 0 1
#────────────┼───────
#4 │ 3 8
#6 │ 4 3
#8 │ 12 2
## proportion wise (overall)
#3×2 Named Array{Float64,2}
#Dim1 ╲ Dim2 │ 0 1
#────────────┼─────────────────
#4 │ 0.09375 0.25
#6 │ 0.125 0.09375
#8 │ 0.375 0.0625
## Column wise proportion
#3×2 Named Array{Float64,2}
#Dim1 ╲ Dim2 │ 0 1
#────────────┼───────────────────
#4 │ 0.157895 0.615385
#6 │ 0.210526 0.230769
#8 │ 0.631579 0.153846
## Row wise proportion
#3×2 Named Array{Float64,2}
#Dim1 ╲ Dim2 │ 0 1
#────────────┼───────────────────
#4 │ 0.272727 0.727273
#6 │ 0.571429 0.428571
#8 │ 0.857143 0.142857
I am trying out the Julia DataFrames module. I am interested in it so I can use it to plot simple simulations in Gadfly. I want to be able to iteratively add rows to the dataframe and I want to initialize it as empty.
The tutorials/documentation on how to do this is sparse (most documentation describes how to analyse imported data).
To append to a nonempty dataframe is straightforward:
df = DataFrame(A = [1, 2], B = [4, 5])
push!(df, [3 6])
This returns.
3x2 DataFrame
| Row | A | B |
|-----|---|---|
| 1 | 1 | 4 |
| 2 | 2 | 5 |
| 3 | 3 | 6 |
But for an empty init I get errors.
df = DataFrame(A = [], B = [])
push!(df, [3, 6])
Error message:
ArgumentError("Error adding 3 to column :A. Possible type mis-match.")
while loading In[220], in expression starting on line 2
What is the best way to initialize an empty Julia DataFrame such that you can iteratively add items to it later in a for loop?
A zero length array defined using only [] will lack sufficient type information.
julia> typeof([])
Array{None,1}
So to avoid that problem is to simply indicate the type.
julia> typeof(Int64[])
Array{Int64,1}
And you can apply that to your DataFrame problem
julia> df = DataFrame(A = Int64[], B = Int64[])
0x2 DataFrame
julia> push!(df, [3 6])
julia> df
1x2 DataFrame
| Row | A | B |
|-----|---|---|
| 1 | 3 | 6 |
using Pkg, CSV, DataFrames
iris = CSV.read(joinpath(Pkg.dir("DataFrames"), "test/data/iris.csv"))
new_iris = similar(iris, nrow(iris))
head(new_iris, 2)
# 2×5 DataFrame
# │ Row │ SepalLength │ SepalWidth │ PetalLength │ PetalWidth │ Species │
# ├─────┼─────────────┼────────────┼─────────────┼────────────┼─────────┤
# │ 1 │ missing │ missing │ missing │ missing │ missing │
# │ 2 │ missing │ missing │ missing │ missing │ missing │
for (i, row) in enumerate(eachrow(iris))
new_iris[i, :] = row[:]
end
head(new_iris, 2)
# 2×5 DataFrame
# │ Row │ SepalLength │ SepalWidth │ PetalLength │ PetalWidth │ Species │
# ├─────┼─────────────┼────────────┼─────────────┼────────────┼─────────┤
# │ 1 │ 5.1 │ 3.5 │ 1.4 │ 0.2 │ setosa │
# │ 2 │ 4.9 │ 3.0 │ 1.4 │ 0.2 │ setosa │
The answer from #waTeim already answers the initial question. But what if I want to dynamically create an empty DataFrame and append rows to it. E.g. what if I don't want hard-coded column names?
In this case, df = DataFrame(A = Int64[], B = Int64[]) is not sufficient.
The NamedTuple A = Int64[], B = Int64[] needs to be create dynamically.
Let's assume we have a vector of column names col_names and a vector of column types colum_types from which to create an emptyDataFrame.
col_names = [:A, :B] # needs to be a vector Symbols
col_types = [Int64, Float64]
# Create a NamedTuple (A=Int64[], ....) by doing
named_tuple = (; zip(col_names, type[] for type in col_types )...)
df = DataFrame(named_tuple) # 0×2 DataFrame
Alternatively, the NameTuple could be created with
# or by doing
named_tuple = NamedTuple{Tuple(col_names)}(type[] for type in col_types )
I think at least in the latest version of Julia you can achieve this by creating a pair object without specifying type
df = DataFrame("A" => [], "B" => [])
push!(df, [5,'f'])
1×2 DataFrame
Row │ A B
│ Any Any
─────┼──────────
1 │ 5 f
as seen in this post by #Bogumił Kamiński where multiple columns are needed, something like this can be done:
entries = ["A", "B", "C", "D"]
df = DataFrame([ name =>[] for name in entries])
julia> push!(df,[4,5,'r','p'])
1×4 DataFrame
Row │ A B C D
│ Any Any Any Any
─────┼────────────────────
1 │ 4 5 r p
Or as pointed out by #Antonello below if you know that type you can do.
df = DataFrame([name => Int[] for name in entries])
which is also in #Bogumil Kaminski's original post.