how to read a .js file into a DataFrame - dataframe

Question

i have the following non standard file and want to read it efficiently into a Dataframe, but do not know how to efficiently handle it.
m[mi++]="18.06.16 22:00:00|0;3;1;1667;49;49;12|0;17;3;2153;328;153;57|0;18;0;2165;284;156;53"
m[mi++]="18.06.16 21:55:00|0;24;7;1667;306;306;61|0;21;4;2153;384;166;62|0;19;1;2165;368;185;62"
m[mi++]="18.06.16 21:50:00|0;31;6;1667;394;349;62|0;32;6;2153;402;164;63|0;33;4;2165;380;171;63"
m[mi++]="18.06.16 21:45:00|11;50;19;1667;390;312;63|34;61;9;2153;410;166;63|19;60;8;2165;391;185;63"
m[mi++]="18.06.16 21:40:00|37;63;24;1666;387;313;63|55;80;12;2150;418;169;64|46;79;11;2163;398;186;63"
If I have saved this data in a file a.js the following code works:
b = readtable("a.js")
dat_5m = DataFrame(t=DateTime[], Pac_1=Float64[], Pdc_A1=Float64[], Pdc_B1=Float64[], Etot1=Float64[],
U_A1=Float64[], U_B1=Float64[], T_1=Float64[], Pac_2=Float64[], Pdc_A2=Float64[], Pdc_B2=Float64[],
Etot2=Float64[], U_A2=Float64[], U_B2=Float64[], T_2=Float64[], Pac_3=Float64[], Pdc_A3=Float64[],
Pdc_B3=Float64[], Etot3=Float64[], U_A3=Float64[], U_B3=Float64[], T_3=Float64[])
for r in nrow(b):-1:1
c = split(b[r,1], [';','|','=']);
d = float(c[3:end])'
t = DateTime(c[2], DateFormat("dd.mm.yy HH:MM:SS"))
push!(dat_5m, [t d])
end
But I think it is quite unelegant. Is there any more efficient way?

Answer 1

If possible (IO permitting), I'd recommend cleaning the file first, removing | and any funky " and prefixes. Then you can use CSV.jl to make your life easier.
The process is: read line by line, clean each line and then use CSV to read it back in:
julia> using CSV
julia> open("b.js", "a") do new_file
for line in readlines("a.js")
line = replace(line, "m[mi++]=\"", "")
# remove only trailing quote
line = replace(line, r"\"$", "")
line = replace(line, "|", ";")
write(new_file, string(line, "\n"))
end
end
julia> b = CSV.read("b.js",
delim = ";",
header = false, # change this to a string vector to provide column names
dateformat = "dd.mm.yy HH:MM:SS");
julia> # correct times to be in 2016
b[:Column1] += Dates.Year(2000);
julia> head(b)
6×22 DataFrames.DataFrame. Omitted printing of 15 columns
│ Row │ Column1 │ Column2 │ Column3 │ Column4 │ Column5 │ Column6 │ Column7 │
├─────┼─────────────────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┤
│ 1 │ 2016-06-18T22:00:00 │ 0 │ 3 │ 1 │ 1667 │ 49 │ 49 │
│ 2 │ 2016-06-18T21:55:00 │ 0 │ 24 │ 7 │ 1667 │ 306 │ 306 │
│ 3 │ 2016-06-18T21:50:00 │ 0 │ 31 │ 6 │ 1667 │ 394 │ 349 │
│ 4 │ 2016-06-18T21:45:00 │ 11 │ 50 │ 19 │ 1667 │ 390 │ 312 │
│ 5 │ 2016-06-18T21:40:00 │ 37 │ 63 │ 24 │ 1666 │ 387 │ 313 │
│ 6 │ 2016-06-18T22:00:00 │ 0 │ 3 │ 1 │ 1667 │ 49 │ 49 │
I cleaned the lines here step by step to make the process easier to understand. If you're a regex master you might want to do that in one line.
There might be a better way of handling the 2 digit years, so if someone knows how to fix that please edit this answer or add it as a comment. Thanks!
Edit:
If you want to do the same without writing to another file, here's a hack to apply the cleaning function, and reuse CSV.read() by converting the readlines array to a IOBuffer again:
function cleaner(line)
line = replace(line, "m[mi++]=\"", "")
line = replace(line, r"\"$", "")
line = replace(line, "|", ";")
println(line)
string(line, "\n")
end
c = CSV.read(
Base.IOBuffer(
string(cleaner.(readlines("a.js"))...)),
delim = ";",
header = false,
dateformat = "dd.mm.yy HH:MM:SS");
c[:Column1] += Dates.Year(2000);
This gives the same result as the other solution.