Pandas load from_records non-sparsely - pandas

Question

I am trying to load a list of dictionaries into pandas as efficiently as possible. Here is a minimal example for constructing my data, which I call below, mylist:
import pandas as pd
import random
from string import ascii_lowercase
random.seed(100)
mylist = []
for i in range(100):
random_string_variable = "".join(random.sample("DINOSAUR", len("DINOSAUR")))
random_string = "".join(random.sample("DINOSAUR", len("DINOSAUR")))
for j in range(10):
myrecord = {"i": i,
"identifier" : random_string,
f"var_{ascii_lowercase[j].upper()}_xx" : random.random(),
f"var_{ascii_lowercase[j].upper()}_yy" : random.random()*10,
f"var_{ascii_lowercase[j].upper()}_zz" : random.random()*100
}
mylist.append(myrecord)
pprint(mylist[0:5])
[{'i': 0,
'identifier': 'NROUIDSA',
'var_A_xx': 0.03694960304368877,
'var_A_yy': 4.4615792434297585,
'var_A_zz': 68.37385464983947},
{'i': 0,
'identifier': 'NROUIDSA',
'var_B_xx': 0.7476846773635049,
'var_B_yy': 3.2014779786116643,
'var_B_zz': 58.91595571819701},
{'i': 0,
'identifier': 'NROUIDSA',
'var_C_xx': 0.3502573960649995,
'var_C_yy': 6.713087131908023,
'var_C_zz': 74.36827046647622},
{'i': 0,
'identifier': 'NROUIDSA',
'var_D_xx': 0.23513409285324904,
'var_D_yy': 3.894932754840866,
'var_D_zz': 65.35552900764706},
{'i': 0,
'identifier': 'NROUIDSA',
'var_E_xx': 0.6660170004345193,
'var_E_yy': 1.9094479278081555,
'var_E_zz': 36.84983796653053}]
When I try to load this into pandas, it makes the data frame very non-sparse, with a lot of NaN repetition:
df = pd.DataFrame.from_records(mylist)
df
produces:
df
i identifier var_A_xx var_A_yy var_A_zz var_B_xx var_B_yy ... var_H_zz var_I_xx var_I_yy var_I_zz var_J_xx var_J_yy var_J_zz
0 0 NROUIDSA 0.03695 4.461579 68.373855 NaN NaN ... NaN NaN NaN NaN NaN NaN NaN
1 0 NROUIDSA NaN NaN NaN 0.747685 3.201478 ... NaN NaN NaN NaN NaN NaN NaN
2 0 NROUIDSA NaN NaN NaN NaN NaN ... NaN NaN NaN NaN NaN NaN NaN
3 0 NROUIDSA NaN NaN NaN NaN NaN ... NaN NaN NaN NaN NaN NaN NaN
4 0 NROUIDSA NaN NaN NaN NaN NaN ... NaN NaN NaN NaN NaN NaN NaN
.. .. ... ... ... ... ... ... ... ... ... ... ... ... ... ...
995 99 SORIUDAN NaN NaN NaN NaN NaN ... NaN NaN NaN NaN NaN NaN NaN
996 99 SORIUDAN NaN NaN NaN NaN NaN ... NaN NaN NaN NaN NaN NaN NaN
997 99 SORIUDAN NaN NaN NaN NaN NaN ... 63.72333 NaN NaN NaN NaN NaN NaN
998 99 SORIUDAN NaN NaN NaN NaN NaN ... NaN 0.367797 4.162167 84.699542 NaN NaN NaN
999 99 SORIUDAN NaN NaN NaN NaN NaN ... NaN NaN NaN NaN 0.634893 7.628154 75.903316
[1000 rows x 32 columns]
What I would like it to look like is:
var_A_xx var_A_yy var_A_zz var_B_xx var_B_yy var_B_zz ... var_I_xx var_I_yy var_I_zz var_J_xx var_J_yy var_J_zz
i identifier ...
0 NROUIDSA 0.036950 4.461579 68.373855 0.747685 3.201478 58.915956 ... 0.962999 7.332500 13.216899 0.847280 6.504308 8.552283
1 NURDASOI 0.814194 9.570388 21.239626 0.468727 6.180384 24.260818 ... 0.346681 9.865105 82.261586 0.221160 8.481875 92.645263
2 OARNDUIS 0.813418 1.103359 1.198749 0.646912 2.409214 76.037434 ... 0.404528 2.112085 8.461932 0.621124 5.372169 36.500880
3 DISORNAU 0.533450 1.094177 44.053734 0.804385 5.947438 28.360524 ... 0.121844 5.806337 85.657067 0.735207 4.011567 38.368097
4 SIONUDRA 0.672725 3.724022 58.280713 0.346717 7.432624 49.726532 ... 0.238869 0.769056 58.188641 0.415537 6.828866 38.802765
... ... ... ... ... ... ... ... ... ... ... ... ... ...
95 URIADNSO 0.231775 3.114448 65.241238 0.116461 4.330002 12.864624 ... 0.516712 5.589706 87.261427 0.572551 4.060943 80.102004
96 ISDONRAU 0.295684 8.406004 22.817404 0.160434 8.415922 47.288958 ... 0.050647 8.720049 44.407892 0.038166 5.027924 73.852513
97 OIAUSDNR 0.331393 9.480417 90.311381 0.985708 6.384429 55.459062 ... 0.947673 4.406426 68.098531 0.377523 5.258620 61.035638
98 DIONAURS 0.690593 4.316975 9.866558 0.822896 3.822044 68.863371 ... 0.994493 3.550660 22.769721 0.199187 7.254650 91.232969
99 SORIUDAN 0.960168 6.769579 49.488535 0.671168 1.577146 78.835216 ... 0.367797 4.162167 84.699542 0.634893 7.628154 75.903316
[100 rows x 30 columns]
You can see it is a 10x waste of memory to have the first representation. Obviously, there are a variety of ways to get from A to B. How can I tell pandas to /read in/ this list of records as non-sparse, as I assume this would be the most performant? You can see extra records are inserted with NaN values. I'm expecting 100 rows, where the index is given by ["i", "identifier"] and 30 columns.
My preference is to do this at load time with the correct keywords and data load method, rather than relying after the fact on a pivot operations in pandas as they are comparatively slow. I'm asking this question largely for performance, for example with much larger i and somewhat larger j.
df = pd.DataFrame.from_records(mylist, index=["i", "identifier"])
df
Did not do the job.
pd.DataFrame.from_records(mylist, index=["i", "identifier"]).unstack()
ValueError: Index contains duplicate entries, cannot reshape
Also fails.
If there do not exist arguments to ingest the list of dictionaries non-sparsely into a dataframe---this is the focus of my question---which of the .agg, pivot_table, reshape, long_to_wide, and unstack methods would be the fastest at getting from A to B for larger data sets?

Answer 1

There’s a number of ways to load the data as it is:
>>> df_idx = pd.DataFrame.from_records(mylist, index=['i', 'identifier'])
>>> df = pd.DataFrame.from_records(mylist)
>>> df = pd.DataFrame.from_dict(mylist)
>>> df = pd.DataFrame(mylist)
Then we could group by columns or levels, and take the first non-NA value:
>>> df_idx.groupby(level=[0, 1]).first()
>>> df.groupby(['i', 'identifier']).first()
Those are both pretty much improved versions of .agg() as you don’t have to specify a lambda function.
Then with the same loading, we can try to reshape the data, with stack/unstack or melt/pivot
>>> df_idx.stack().unstack()
>>> pd.melt(pd.DataFrame(mylist), id_vars=['i', 'identifier']).dropna().pivot(index=['i', 'identifier'], columns='variable', values='value')
If that’s not satisfactory, there’s also reshaping before loading, which could be done through list comprehensions, and either ChainMap or dict comprehensions, or with numpy. This relies on the fact that there’s always 10 dictionaries with the same keys in a row, and chaining the same iterator the appropriate number of times with zip():
>>> pd.DataFrame({k: v for d in tup for k, v in d.items()} for tup in zip(*[iter(mylist)] * 10))
>>> pd.DataFrame(ChainMap(*tup) for tup in zip(*[iter(mylist)] * 10))
>>> df = pd.concat([pd.DataFrame(mylist[n::10]) for n in range(10)], axis='columns')
>>> df.groupby(df.columns, axis='columns').first()
>>> reshaped = np.reshape(mylist, (100, 10))
>>> df = pd.concat([pd.json_normalize(reshaped[:,n]) for n in range(10)], axis='columns')
>>> df.groupby(df.columns, axis='columns').first()
I’ve measured with i=100 and j=10 as in your example and with i=1000 and j=100.
You can see which way you get the data into a dataframe does not matter: all groupby variants have the same results. As you suspect loading the data and then “fixing” it performs pretty bad. pd.concat does not work too well on 100x10 but scales better on the 1000x100 data, and what seems the best is pure-python dict iteration (maybe because it’ a comprehension and not a list? Not sure). The reshaping techniques, stack/unstack and melt/pivot, are always the worst.
Of course these results may change with different data sizes, and you probably know better what the right sizes to test are, based on your real data. Here’s the full script I used to run the tests so you can run some yourself:
#!/usr/bin/python3
import numpy as np
import pandas as pd
from collections import ChainMap
from matplotlib import pyplot as plt
import timeit
def gen(imax, jmax):
mylist = []
l = ord('A')
for i in range(imax):
random_string = ''.join(np.random.permutation(list('DINOSAUR')))
for j in range(jmax):
var = f'var_{chr(l + (j // 26)) if j >= 26 else ""}{chr(l + (j % 26))}'
mylist.append({
'i': i,
'identifier' : random_string,
var + '_xx' : np.random.random(),
var + '_yy' : np.random.random()*10,
var + '_zz' : np.random.random()*100
})
return mylist
def load_rec_idx_groupby(mylist, j):
return pd.DataFrame.from_records(mylist, index=['i', 'identifier']).groupby(level=[0, 1]).first()
def load_rec_groupby(mylist, j):
return pd.DataFrame.from_records(mylist).groupby(['i', 'identifier']).first()
def load_dict_groupby(mylist, j):
return pd.DataFrame.from_dict(mylist).groupby(['i', 'identifier']).first()
def load_constr_groupby(mylist, j):
return pd.DataFrame(mylist).groupby(['i', 'identifier']).first()
def load_constr_stack(mylist, j):
return pd.DataFrame(mylist).set_index(['i', 'identifier']).stack().unstack()
def load_constr_melt_pivot(mylist, j):
return pd.melt(pd.DataFrame(mylist), id_vars=['i', 'identifier']).dropna().pivot(index=['i', 'identifier'], columns='variable', values='value')
def load_zip_iter_dict(mylist, j):
return pd.DataFrame({k: v for d in tup for k, v in d.items()} for tup in zip(*[iter(mylist)] * j))
def load_zip_iter_chainmap(mylist, j):
return pd.DataFrame(ChainMap(*tup) for tup in zip(*[iter(mylist)] * 10))
def load_concat_step(mylist, j):
return pd.concat([pd.DataFrame(mylist[n::10]).drop(columns=['i', 'identifier'] if n else []) for n in range(10)], axis='columns')
def load_concat_reshape(mylist, j):
reshaped = np.reshape(mylist, (len(mylist) // j, j))
return pd.concat([pd.json_normalize(reshaped[:,n]).drop(columns=['i', 'identifier'] if n else []) for n in range(j)], axis='columns')
def plot_results(df):
mins = df.groupby(level=0).median().min(axis='columns')
rel = df.unstack().T.div(mins)
ax = rel.groupby(level=0).median().plot.barh()
ax.set_xlabel('slowdown over fastest')
ax.axvline(1, color='black', lw=1)
ax.set_xticks([1, *ax.get_xticks()[1:]])
ax.set_xticklabels([f'{n:.0f}×' for n in ax.get_xticks()])
plt.subplots_adjust(left=.4, bottom=.15)
plt.show()
def run():
candidates = {n: f for n, f in globals().items() if n.startswith('load_') and callable(f)}
df = {}
for tup in [(100, 10), (1000, 100)]:
glob = {'mylist': gen(*tup), **candidates}
dat = pd.DataFrame({name:
timeit.Timer(f'{name}(mylist, {10})', globals=glob).repeat(5, int(100000 / np.multiply(*tup)))
for name in candidates
})
print(dat)
df['{}×{}'.format(*tup)] = dat
df = pd.concat(df).rename(columns=lambda s: s.replace('load_', '').replace('_', ' '))
print(df)
plot_results(df)
if __name__ == '__main__':
run()

Answer 2

I don't think this is possible with a pandas argument at load time, but you can use a comprehension to collapse your list of dicts into a single dict for each row
Data:
a = [
{'i': 0,
'identifier': 'NROUIDSA',
'var_A_xx': 0.03694960304368877,
'var_A_yy': 4.4615792434297585,
'var_A_zz': 68.37385464983947},
{'i': 0,
'identifier': 'NROUIDSA',
'var_B_xx': 0.7476846773635049,
'var_B_yy': 3.2014779786116643,
'var_B_zz': 58.91595571819701},
{'i': 0,
'identifier': 'NROUIDSA',
'var_C_xx': 0.3502573960649995,
'var_C_yy': 6.713087131908023,
'var_C_zz': 74.36827046647622},
{'i': 0,
'identifier': 'NROUIDSA',
'var_D_xx': 0.23513409285324904,
'var_D_yy': 3.894932754840866,
'var_D_zz': 65.35552900764706},
{'i': 0,
'identifier': 'NROUIDSA',
'var_E_xx': 0.6660170004345193,
'var_E_yy': 1.9094479278081555,
'var_E_zz': 36.84983796653053},
{'i': 1,
'identifier': 'SORIUDAN',
'var_A_xx': 0.03694960304368877,
'var_A_yy': 4.4615792434297585,
'var_A_zz': 68.37385464983947},
{'i': 1,
'identifier': 'SORIUDAN',
'var_B_xx': 0.7476846773635049,
'var_B_yy': 3.2014779786116643,
'var_B_zz': 58.91595571819701},
{'i': 1,
'identifier': 'SORIUDAN',
'var_C_xx': 0.3502573960649995,
'var_C_yy': 6.713087131908023,
'var_C_zz': 74.36827046647622},
{'i': 1,
'identifier': 'SORIUDAN',
'var_D_xx': 0.23513409285324904,
'var_D_yy': 3.894932754840866,
'var_D_zz': 65.35552900764706},
{'i': 1,
'identifier': 'SORIUDAN',
'var_E_xx': 0.6660170004345193,
'var_E_yy': 1.9094479278081555,
'var_E_zz': 36.84983796653053}
]
Cleaning the list of dicts:
# get list of keys--assumed here to be the identifier dict value
l_key = list(dict.fromkeys([l.get('identifier') for l in a]))
# a data dict we'll append the properly parsed dict to
data = list()
# iterate through original dict and append.
for i in l_key:
l_data = [l for l in a if l.get('identifier') == i]
data.append({k: v for d in l_data for k, v in d.items()})
adding data to the pandas df.
import pandas as pd
df = pd.DataFrame.from_records(data)
print(df)
print(df.dtypes)
I don't know that this would be faster than dealing with the data after you've loaded it into the DataFrame but it is another approach.

Answer 3

I think your issue is not from pandas, you can create the records according to what you want as a result. I edit your implementation as below,
import random
from string import ascii_lowercase
import pandas as pd
random.seed(100)
mylist = []
for i in range(100):
random_string_variable = "".join(random.sample("DINOSAUR", len("DINOSAUR")))
random_string = "".join(random.sample("DINOSAUR", len("DINOSAUR")))
record = {
"i": i,
"identifier": random_string
}
for j in range(10):
record[f"var_{ascii_lowercase[j].upper()}_xx"] = random.random()
record[f"var_{ascii_lowercase[j].upper()}_yy"] = random.random() * 10,
record[f"var_{ascii_lowercase[j].upper()}_zz"] = random.random() * 100
mylist.append(record)
print(len(mylist))
df = pd.DataFrame.from_records(mylist)
df
As it is about loading the records into Pandas, maybe it can be easier to process the list before passing into pandas such as
from itertools import groupby
from collections import ChainMap
records = []
for k, v in groupby(mylist, key=lambda x: (x['i'], x['identifier'])):
record = dict(ChainMap(*v))
records.append(record)
df = pd.DataFrame.from_records(records)
print(df)