Writetable is exporting data with "Nullable{Type}(data)" instead of just data in Julia - dataframe

Question

For example, I have a DataFrame like the one seen below, let's call it df.
╔═════╦══════╦══════╦══════╗
║ Row ║ a ║ b ║ c ║
╠═════╬══════╬══════╬══════╣
║ 1 ║ 0.66 ║ 0.55 ║ 0.44 ║
╠═════╬══════╬══════╬══════╣
║ 2 ║ 0.11 ║ 0.22 ║ 0.33 ║
╠═════╬══════╬══════╬══════╣
║ 3 ║ 1.00 ║ 2.00 ║ 3.00 ║
╚═════╩══════╩══════╩══════╝
When I use writetable("output.txt",df) I receive the following type of output for the numerical data in the text file.
"Nullable{Float64}(0.66)"
instead of
0.66
Any thoughts on how to get writetable to export just the data?
EDIT:
I should note that this only occurs after importing data using the ReadStat package. Is it possible to convert an entire data set to just an Array that can be exported properly? That may solve the problem.
EDIT #2:
I just tried running the following code (utilizing a created function converter) but am receiving errors (posted below).
f(a,n)=
if typeof(a[n])==NullableArrays.NullableArray{String,1}
convert(Array{String},a[n])
elseif typeof(a[n])==NullableArrays.NullableArray{Float64,1}
convert(Array{Float64},a[n])
elseif typeof(a[n])==NullableArrays.NullableArray{Int64,1}
convert(Array{Float64},a[n])
end
converter(a)=hcat([f(a,n) for n=1:length(a)]...)
The errors received are as follows:
julia> converter(af)
ERROR: NullException()
in convert at /home/ale/.julia/v0.5/NullableArrays/src/primitives.jl:248 [inlined]
in convert(::Type{Array{Float64,N}}, ::NullableArrays.NullableArray{Float64,1}) at /home/ale/.julia/v0.5/NullableArrays/src/primitives.jl:256
in f(::DataFrames.DataFrame, ::Int64) at ./REPL[6]:5
in collect_to!(::Array{Array{T,1},1}, ::Base.Generator{UnitRange{Int64},##1#2{DataFrames.DataFrame}}, ::Int64, ::Int64) at ./array.jl:340
in collect_to!(::Array{Array{Float64,1},1}, ::Base.Generator{UnitRange{Int64},##1#2{DataFrames.DataFrame}}, ::Int64, ::Int64) at ./array.jl:350
in collect(::Base.Generator{UnitRange{Int64},##1#2{DataFrames.DataFrame}}) at ./array.jl:308
in converter(::DataFrames.DataFrame) at ./REPL[7]:1

Answer 1

Have a look / play a bit with the following:
julia> using DataFrames
julia> a = [Nullable(0.1), Nullable{Float64}(), Nullable(0.3)];
julia> b = [Nullable{Float64}(), Nullable(2.), Nullable(3.)];
julia> df = DataFrame(Any[a,b], [:a,:b])
3×2 DataFrames.DataFrame
│ Row │ a │ b │
├─────┼───────┼───────┤
│ 1 │ 0.1 │ #NULL │
│ 2 │ #NULL │ 2.0 │
│ 3 │ 0.3 │ 3.0 │
julia> c = [df[x] for x in names(df)];
julia> f(x) = [get(y, "Missing") for y in x];
julia> d = Any[f(x) for x in c]; # "Any" required for dataframes (I think)
julia> df2 = DataFrame(d, names(df))
│ Row │ a │ b │
├─────┼───────────┼───────────┤
│ 1 │ 0.1 │ "Missing" │
│ 2 │ "Missing" │ 2.0 │
│ 3 │ 0.3 │ 3.0 │
julia> writetable("/home/tasos/Desktop/output.txt", df2)
Note that for each column, if there is even one missing value, your numbers will also be reported inside quotes, because of the mixed array. If you want it to be all integers, you'll have to choose a different default value to "Missing" to denote your missing values (e.g. -1 if you only expect positive numbers).
If you don't like that, then you're probably better off writing your own "writetable" function; it's not that difficult, it's just a case of opening a file and printing what you want per column.
Also, to address some of our discussion in the comments:
The nullable type has two fields:
julia> fieldnames(Nullable)
2-element Array{Symbol,1}:
:hasvalue
:value
Let's create two instances to show what they mean:
julia> a = Nullable(1, true); b = Nullable(2, false);
julia> a.hasvalue, a.value
(true,1)
julia> b.hasvalue, b.value
(false,2)
You can test for nullity explicitly:
julia> isnull(a)
false
julia> isnull(b)
true
julia> isnull(0), isnull("")
(false, false) # isnull returns false by default if input is not a Nullable Type
Or you can use the "get" function to get a Nullable's value. If you don't define an alternative in the case of null, you get a NullException:
julia> get(a)
1
julia> get(b)
ERROR: NullException()
Stacktrace:
[1] get(::Nullable{Int64}) at ./nullable.jl:92
julia> get(b, "Null Detected")
"Null Detected"
A Nullable instance defined as Nullable(1, false) has a .value of 1, but this is superfluous as it is declared as .hasvalue=false and is therefore effectively null (though you can query the .value if you really want to).
A Nullable instance defined as n = Nullable{Float64}() will give you a nullable instance with .hasvalue=false and a meaningless value, presumably whatever was in memory at that location during instantiation, though interpreted as whatever Type of Nullable you declared (i.e. Float64):
julia> n = Nullable{Float64}()
Nullable{Float64}()
julia> n.hasvalue, n.value
(false, 6.9015724352651e-310)