I am trying to load a list of dictionaries into pandas as efficiently as possible. Here is a minimal example for constructing my data, which I call below, mylist:
import pandas as pd
import random
from string import ascii_lowercase
random.seed(100)
mylist = []
for i in range(100):
random_string_variable = "".join(random.sample("DINOSAUR", len("DINOSAUR")))
random_string = "".join(random.sample("DINOSAUR", len("DINOSAUR")))
for j in range(10):
myrecord = {"i": i,
"identifier" : random_string,
f"var_{ascii_lowercase[j].upper()}_xx" : random.random(),
f"var_{ascii_lowercase[j].upper()}_yy" : random.random()*10,
f"var_{ascii_lowercase[j].upper()}_zz" : random.random()*100
}
mylist.append(myrecord)
pprint(mylist[0:5])
[{'i': 0,
'identifier': 'NROUIDSA',
'var_A_xx': 0.03694960304368877,
'var_A_yy': 4.4615792434297585,
'var_A_zz': 68.37385464983947},
{'i': 0,
'identifier': 'NROUIDSA',
'var_B_xx': 0.7476846773635049,
'var_B_yy': 3.2014779786116643,
'var_B_zz': 58.91595571819701},
{'i': 0,
'identifier': 'NROUIDSA',
'var_C_xx': 0.3502573960649995,
'var_C_yy': 6.713087131908023,
'var_C_zz': 74.36827046647622},
{'i': 0,
'identifier': 'NROUIDSA',
'var_D_xx': 0.23513409285324904,
'var_D_yy': 3.894932754840866,
'var_D_zz': 65.35552900764706},
{'i': 0,
'identifier': 'NROUIDSA',
'var_E_xx': 0.6660170004345193,
'var_E_yy': 1.9094479278081555,
'var_E_zz': 36.84983796653053}]
When I try to load this into pandas, it makes the data frame very non-sparse, with a lot of NaN repetition:
df = pd.DataFrame.from_records(mylist)
df
produces:
df
i identifier var_A_xx var_A_yy var_A_zz var_B_xx var_B_yy ... var_H_zz var_I_xx var_I_yy var_I_zz var_J_xx var_J_yy var_J_zz
0 0 NROUIDSA 0.03695 4.461579 68.373855 NaN NaN ... NaN NaN NaN NaN NaN NaN NaN
1 0 NROUIDSA NaN NaN NaN 0.747685 3.201478 ... NaN NaN NaN NaN NaN NaN NaN
2 0 NROUIDSA NaN NaN NaN NaN NaN ... NaN NaN NaN NaN NaN NaN NaN
3 0 NROUIDSA NaN NaN NaN NaN NaN ... NaN NaN NaN NaN NaN NaN NaN
4 0 NROUIDSA NaN NaN NaN NaN NaN ... NaN NaN NaN NaN NaN NaN NaN
.. .. ... ... ... ... ... ... ... ... ... ... ... ... ... ...
995 99 SORIUDAN NaN NaN NaN NaN NaN ... NaN NaN NaN NaN NaN NaN NaN
996 99 SORIUDAN NaN NaN NaN NaN NaN ... NaN NaN NaN NaN NaN NaN NaN
997 99 SORIUDAN NaN NaN NaN NaN NaN ... 63.72333 NaN NaN NaN NaN NaN NaN
998 99 SORIUDAN NaN NaN NaN NaN NaN ... NaN 0.367797 4.162167 84.699542 NaN NaN NaN
999 99 SORIUDAN NaN NaN NaN NaN NaN ... NaN NaN NaN NaN 0.634893 7.628154 75.903316
[1000 rows x 32 columns]
What I would like it to look like is:
var_A_xx var_A_yy var_A_zz var_B_xx var_B_yy var_B_zz ... var_I_xx var_I_yy var_I_zz var_J_xx var_J_yy var_J_zz
i identifier ...
0 NROUIDSA 0.036950 4.461579 68.373855 0.747685 3.201478 58.915956 ... 0.962999 7.332500 13.216899 0.847280 6.504308 8.552283
1 NURDASOI 0.814194 9.570388 21.239626 0.468727 6.180384 24.260818 ... 0.346681 9.865105 82.261586 0.221160 8.481875 92.645263
2 OARNDUIS 0.813418 1.103359 1.198749 0.646912 2.409214 76.037434 ... 0.404528 2.112085 8.461932 0.621124 5.372169 36.500880
3 DISORNAU 0.533450 1.094177 44.053734 0.804385 5.947438 28.360524 ... 0.121844 5.806337 85.657067 0.735207 4.011567 38.368097
4 SIONUDRA 0.672725 3.724022 58.280713 0.346717 7.432624 49.726532 ... 0.238869 0.769056 58.188641 0.415537 6.828866 38.802765
... ... ... ... ... ... ... ... ... ... ... ... ... ...
95 URIADNSO 0.231775 3.114448 65.241238 0.116461 4.330002 12.864624 ... 0.516712 5.589706 87.261427 0.572551 4.060943 80.102004
96 ISDONRAU 0.295684 8.406004 22.817404 0.160434 8.415922 47.288958 ... 0.050647 8.720049 44.407892 0.038166 5.027924 73.852513
97 OIAUSDNR 0.331393 9.480417 90.311381 0.985708 6.384429 55.459062 ... 0.947673 4.406426 68.098531 0.377523 5.258620 61.035638
98 DIONAURS 0.690593 4.316975 9.866558 0.822896 3.822044 68.863371 ... 0.994493 3.550660 22.769721 0.199187 7.254650 91.232969
99 SORIUDAN 0.960168 6.769579 49.488535 0.671168 1.577146 78.835216 ... 0.367797 4.162167 84.699542 0.634893 7.628154 75.903316
[100 rows x 30 columns]
You can see it is a 10x waste of memory to have the first representation. Obviously, there are a variety of ways to get from A to B. How can I tell pandas to /read in/ this list of records as non-sparse, as I assume this would be the most performant? You can see extra records are inserted with NaN values. I'm expecting 100 rows, where the index is given by ["i", "identifier"] and 30 columns.
My preference is to do this at load time with the correct keywords and data load method, rather than relying after the fact on a pivot operations in pandas as they are comparatively slow. I'm asking this question largely for performance, for example with much larger i and somewhat larger j.
df = pd.DataFrame.from_records(mylist, index=["i", "identifier"])
df
Did not do the job.
pd.DataFrame.from_records(mylist, index=["i", "identifier"]).unstack()
ValueError: Index contains duplicate entries, cannot reshape
Also fails.
If there do not exist arguments to ingest the list of dictionaries non-sparsely into a dataframe---this is the focus of my question---which of the .agg, pivot_table, reshape, long_to_wide, and unstack methods would be the fastest at getting from A to B for larger data sets?
There’s a number of ways to load the data as it is:
>>> df_idx = pd.DataFrame.from_records(mylist, index=['i', 'identifier'])
>>> df = pd.DataFrame.from_records(mylist)
>>> df = pd.DataFrame.from_dict(mylist)
>>> df = pd.DataFrame(mylist)
Then we could group by columns or levels, and take the first non-NA value:
>>> df_idx.groupby(level=[0, 1]).first()
>>> df.groupby(['i', 'identifier']).first()
Those are both pretty much improved versions of .agg() as you don’t have to specify a lambda function.
Then with the same loading, we can try to reshape the data, with stack/unstack or melt/pivot
>>> df_idx.stack().unstack()
>>> pd.melt(pd.DataFrame(mylist), id_vars=['i', 'identifier']).dropna().pivot(index=['i', 'identifier'], columns='variable', values='value')
If that’s not satisfactory, there’s also reshaping before loading, which could be done through list comprehensions, and either ChainMap or dict comprehensions, or with numpy. This relies on the fact that there’s always 10 dictionaries with the same keys in a row, and chaining the same iterator the appropriate number of times with zip():
>>> pd.DataFrame({k: v for d in tup for k, v in d.items()} for tup in zip(*[iter(mylist)] * 10))
>>> pd.DataFrame(ChainMap(*tup) for tup in zip(*[iter(mylist)] * 10))
>>> df = pd.concat([pd.DataFrame(mylist[n::10]) for n in range(10)], axis='columns')
>>> df.groupby(df.columns, axis='columns').first()
>>> reshaped = np.reshape(mylist, (100, 10))
>>> df = pd.concat([pd.json_normalize(reshaped[:,n]) for n in range(10)], axis='columns')
>>> df.groupby(df.columns, axis='columns').first()
I’ve measured with i=100 and j=10 as in your example and with i=1000 and j=100.
You can see which way you get the data into a dataframe does not matter: all groupby variants have the same results. As you suspect loading the data and then “fixing” it performs pretty bad. pd.concat does not work too well on 100x10 but scales better on the 1000x100 data, and what seems the best is pure-python dict iteration (maybe because it’ a comprehension and not a list? Not sure). The reshaping techniques, stack/unstack and melt/pivot, are always the worst.
Of course these results may change with different data sizes, and you probably know better what the right sizes to test are, based on your real data. Here’s the full script I used to run the tests so you can run some yourself:
#!/usr/bin/python3
import numpy as np
import pandas as pd
from collections import ChainMap
from matplotlib import pyplot as plt
import timeit
def gen(imax, jmax):
mylist = []
l = ord('A')
for i in range(imax):
random_string = ''.join(np.random.permutation(list('DINOSAUR')))
for j in range(jmax):
var = f'var_{chr(l + (j // 26)) if j >= 26 else ""}{chr(l + (j % 26))}'
mylist.append({
'i': i,
'identifier' : random_string,
var + '_xx' : np.random.random(),
var + '_yy' : np.random.random()*10,
var + '_zz' : np.random.random()*100
})
return mylist
def load_rec_idx_groupby(mylist, j):
return pd.DataFrame.from_records(mylist, index=['i', 'identifier']).groupby(level=[0, 1]).first()
def load_rec_groupby(mylist, j):
return pd.DataFrame.from_records(mylist).groupby(['i', 'identifier']).first()
def load_dict_groupby(mylist, j):
return pd.DataFrame.from_dict(mylist).groupby(['i', 'identifier']).first()
def load_constr_groupby(mylist, j):
return pd.DataFrame(mylist).groupby(['i', 'identifier']).first()
def load_constr_stack(mylist, j):
return pd.DataFrame(mylist).set_index(['i', 'identifier']).stack().unstack()
def load_constr_melt_pivot(mylist, j):
return pd.melt(pd.DataFrame(mylist), id_vars=['i', 'identifier']).dropna().pivot(index=['i', 'identifier'], columns='variable', values='value')
def load_zip_iter_dict(mylist, j):
return pd.DataFrame({k: v for d in tup for k, v in d.items()} for tup in zip(*[iter(mylist)] * j))
def load_zip_iter_chainmap(mylist, j):
return pd.DataFrame(ChainMap(*tup) for tup in zip(*[iter(mylist)] * 10))
def load_concat_step(mylist, j):
return pd.concat([pd.DataFrame(mylist[n::10]).drop(columns=['i', 'identifier'] if n else []) for n in range(10)], axis='columns')
def load_concat_reshape(mylist, j):
reshaped = np.reshape(mylist, (len(mylist) // j, j))
return pd.concat([pd.json_normalize(reshaped[:,n]).drop(columns=['i', 'identifier'] if n else []) for n in range(j)], axis='columns')
def plot_results(df):
mins = df.groupby(level=0).median().min(axis='columns')
rel = df.unstack().T.div(mins)
ax = rel.groupby(level=0).median().plot.barh()
ax.set_xlabel('slowdown over fastest')
ax.axvline(1, color='black', lw=1)
ax.set_xticks([1, *ax.get_xticks()[1:]])
ax.set_xticklabels([f'{n:.0f}×' for n in ax.get_xticks()])
plt.subplots_adjust(left=.4, bottom=.15)
plt.show()
def run():
candidates = {n: f for n, f in globals().items() if n.startswith('load_') and callable(f)}
df = {}
for tup in [(100, 10), (1000, 100)]:
glob = {'mylist': gen(*tup), **candidates}
dat = pd.DataFrame({name:
timeit.Timer(f'{name}(mylist, {10})', globals=glob).repeat(5, int(100000 / np.multiply(*tup)))
for name in candidates
})
print(dat)
df['{}×{}'.format(*tup)] = dat
df = pd.concat(df).rename(columns=lambda s: s.replace('load_', '').replace('_', ' '))
print(df)
plot_results(df)
if __name__ == '__main__':
run()
I don't think this is possible with a pandas argument at load time, but you can use a comprehension to collapse your list of dicts into a single dict for each row
Data:
a = [
{'i': 0,
'identifier': 'NROUIDSA',
'var_A_xx': 0.03694960304368877,
'var_A_yy': 4.4615792434297585,
'var_A_zz': 68.37385464983947},
{'i': 0,
'identifier': 'NROUIDSA',
'var_B_xx': 0.7476846773635049,
'var_B_yy': 3.2014779786116643,
'var_B_zz': 58.91595571819701},
{'i': 0,
'identifier': 'NROUIDSA',
'var_C_xx': 0.3502573960649995,
'var_C_yy': 6.713087131908023,
'var_C_zz': 74.36827046647622},
{'i': 0,
'identifier': 'NROUIDSA',
'var_D_xx': 0.23513409285324904,
'var_D_yy': 3.894932754840866,
'var_D_zz': 65.35552900764706},
{'i': 0,
'identifier': 'NROUIDSA',
'var_E_xx': 0.6660170004345193,
'var_E_yy': 1.9094479278081555,
'var_E_zz': 36.84983796653053},
{'i': 1,
'identifier': 'SORIUDAN',
'var_A_xx': 0.03694960304368877,
'var_A_yy': 4.4615792434297585,
'var_A_zz': 68.37385464983947},
{'i': 1,
'identifier': 'SORIUDAN',
'var_B_xx': 0.7476846773635049,
'var_B_yy': 3.2014779786116643,
'var_B_zz': 58.91595571819701},
{'i': 1,
'identifier': 'SORIUDAN',
'var_C_xx': 0.3502573960649995,
'var_C_yy': 6.713087131908023,
'var_C_zz': 74.36827046647622},
{'i': 1,
'identifier': 'SORIUDAN',
'var_D_xx': 0.23513409285324904,
'var_D_yy': 3.894932754840866,
'var_D_zz': 65.35552900764706},
{'i': 1,
'identifier': 'SORIUDAN',
'var_E_xx': 0.6660170004345193,
'var_E_yy': 1.9094479278081555,
'var_E_zz': 36.84983796653053}
]
Cleaning the list of dicts:
# get list of keys--assumed here to be the identifier dict value
l_key = list(dict.fromkeys([l.get('identifier') for l in a]))
# a data dict we'll append the properly parsed dict to
data = list()
# iterate through original dict and append.
for i in l_key:
l_data = [l for l in a if l.get('identifier') == i]
data.append({k: v for d in l_data for k, v in d.items()})
adding data to the pandas df.
import pandas as pd
df = pd.DataFrame.from_records(data)
print(df)
print(df.dtypes)
I don't know that this would be faster than dealing with the data after you've loaded it into the DataFrame but it is another approach.
I think your issue is not from pandas, you can create the records according to what you want as a result. I edit your implementation as below,
import random
from string import ascii_lowercase
import pandas as pd
random.seed(100)
mylist = []
for i in range(100):
random_string_variable = "".join(random.sample("DINOSAUR", len("DINOSAUR")))
random_string = "".join(random.sample("DINOSAUR", len("DINOSAUR")))
record = {
"i": i,
"identifier": random_string
}
for j in range(10):
record[f"var_{ascii_lowercase[j].upper()}_xx"] = random.random()
record[f"var_{ascii_lowercase[j].upper()}_yy"] = random.random() * 10,
record[f"var_{ascii_lowercase[j].upper()}_zz"] = random.random() * 100
mylist.append(record)
print(len(mylist))
df = pd.DataFrame.from_records(mylist)
df
As it is about loading the records into Pandas, maybe it can be easier to process the list before passing into pandas such as
from itertools import groupby
from collections import ChainMap
records = []
for k, v in groupby(mylist, key=lambda x: (x['i'], x['identifier'])):
record = dict(ChainMap(*v))
records.append(record)
df = pd.DataFrame.from_records(records)
print(df)
When I download data from yfinance, I get 8 columns (Open, High, Low, etc...) per ticker. Since I am downloading 15 tickers, I have 120 columns and 1 index column (date). They add up horizontally. See image 1
Instead of having that many columns, in 2 levels, I want just the 8 unique columns. Plus creating one new column that identifies the ticker. See Image 2.
Image 1: Current Form
Image 1 but in raw text:
Adj Close ... Volume
DANHOS13.MX FCFE18.MX FHIPO14.MX FIBRAHD15.MX FIBRAMQ12.MX FIBRAPL14.MX FIHO12.MX FINN13.MX FMTY14.MX FNOVA17.MX ... FIBRAPL14.MX FIHO12.MX FINN13.MX FMTY14.MX FNOVA17.MX FPLUS16.MX FSHOP13.MX FUNO11.MX FVIA16.MX TERRA13.MX
Date
2015-01-02 26.065336 NaN 18.526043 NaN 16.337654 18.520781 14.683501 11.301384 9.247743 NaN ... 338697 189552 148064 57 NaN NaN 212451 2649823 NaN 1111343
2015-01-05 24.670488 NaN 18.436762 NaN 15.857328 17.859756 13.795850 11.071105 9.209846 NaN ... 449555 364819 244594 19330 NaN NaN 491587 3317923 NaN 1255128
Image 2: Desired outcome
The code Im applying is:
start = dt.datetime(2015,1,1)
end = dt.datetime.now()
df = yf.download("FUNO11.MX FIBRAMQ12.MX FIHO12.MX DANHOS13.MX FINN13.MX FSHOP13.MX TERRA13.MX FMTY14.MX FIBRAPL14.MX FHIPO14.MX FIBRAHD15.MX FPLUS16.MX FVIA16.MX FNOVA17.MX FCFE18.MX",
start = start,
end = end,
group_by = 'Ticker',
actions = True)
I will download the data a little differently:
import yfinance as yf
from datetime import datetime as dt
from dateutil.relativedelta import relativedelta
start = dt(2015,1,1)
end = dt.now()
symbols = ["FUNO11.MX", "FIBRAMQ12.MX", "FIHO12.MX", "DANHOS13.MX", "FINN13.MX", "FSHOP13.MX", "TERRA13.MX", "FMTY14.MX",
"FIBRAPL14.MX", "FHIPO14.MX", "FIBRAHD15.MX", "FPLUS16.MX", "FVIA16.MX", "FNOVA17.MX", "FCFE18.MX"]
data = yf.download(symbols, start=start, end=end, actions=True)
And then
Option 1:
def reshaper(symb, dframe):
df = dframe.unstack().reset_index()
df.columns = ['variable','symbol','Date','Value']
df = df.loc[df.symbol==symb,['Date','variable','Value']].pivot_table(index='Date', columns='variable', values='Value').reset_index()
df.columns.name = ''
df['Ticker'] = symb
return df
h = pd.DataFrame()
for s in symbols:
h = h.append(reshaper(s, data), ignore_index=True)
h
Option 2: For a one-liner, you could do this:
data.stack().reset_index().rename(columns={'level_1':'Ticker'})
A slightly simpler version relies on stacking first the two column index levels (measure and ticker) to get long form tidy data, and then stack on the measure level, keeping ticker and date as indices:
import yfinance as yf
symbols = ["FUNO11.MX", "FIBRAMQ12.MX", "FIHO12.MX", "DANHOS13.MX",
"FINN13.MX", "FSHOP13.MX", "TERRA13.MX", "FMTY14.MX",
"FIBRAPL14.MX", "FHIPO14.MX", "FIBRAHD15.MX", "FPLUS16.MX",
"FVIA16.MX", "FNOVA17.MX", "FCFE18.MX"]
data = yf.download(symbols, start='2015-01-01', end='2020-11-15', actions=True)
data_reshape=data.stack(level=[0,1]).unstack(1)
data_reshape.index=data_reshape.index.set_names(['ticker'],level=[1])
data_reshape.head()
data_reshape.head()
Adj Close Close Dividends High \
Date ticker
2015-01-02 DANHOS13.MX 26.065336 37.000000 0.0 37.400002
FHIPO14.MX 18.526043 24.900000 0.0 24.900000
FIBRAMQ12.MX 16.337654 24.490000 0.0 25.110001
FIBRAPL14.MX 18.520781 26.740801 0.0 27.118500
FIHO12.MX 14.683501 21.670000 0.0 22.190001
Low Open Stock Splits Volume
Date ticker
2015-01-02 DANHOS13.MX 36.330002 36.330002 0.0 82849.0
FHIPO14.MX 24.900000 24.900000 0.0 94007.0
FIBRAMQ12.MX 24.350000 24.990000 0.0 1172917.0
FIBRAPL14.MX 26.343100 26.750700 0.0 338697.0
FIHO12.MX 21.209999 22.120001 0.0 189552.0
Assume running pandas' dataframe['prod_code'].value_counts() and storing result as 'df'. The operation outputs:
125011 90300
762 72816
None 55512
7156 14892
75162 8825
How would I extract the count for None? I'd expect the result to be 55512.
I've tried
>>> df.loc[df.index.isin(['None'])]
>>> Series([], Name: prod_code, dtype: int64)
and also
>>> df.loc['None']
>>> KeyError: 'the label [None] is not in the [index]'
It seems you need None, not string 'None':
df.loc[df.index.isin([None])]
df.loc[None]
EDIT:
If need check where NaN in index:
print (s1.loc[np.nan])
#or
print (df[pd.isnull(df.index)])
Sample:
s = pd.Series(['90300', '90300', '8825', '8825', '8825', None, np.nan])
s1 = s.value_counts(dropna=False)
print (s1)
8825 3
90300 2
NaN 2
dtype: int64
print (s1[pd.isnull(s1.index)])
NaN 2
dtype: int64
print (s1.loc[np.nan])
2
print (s1.loc[None])
2
EDIT1:
For stripping whitespaces:
s = pd.Series(['90300', '90300', '8825', '8825', '8825', 'None ', np.nan])
print (s)
0 90300
1 90300
2 8825
3 8825
4 8825
5 None
6 NaN
dtype: object
s1 = s.value_counts()
print (s1)
8825 3
90300 2
None 1
dtype: int64
s1.index = s1.index.str.strip()
print (s1.loc['None'])
1
Couple of things
pd.Series([None] * 2 + [1] * 3).value_counts() automatically drops the None.
pd.Series([None] * 2 + [1] * 3).value_counts(dropna=False) converts the None to np.NaN
That tells me that your None is a string. But since df.loc['None'] didn't work, I suspect your string has white space around it.
Try:
df.filter(regex='None', axis=0)
Or:
df.index = df.index.to_series().str.strip().combine_first(df.index.to_series())
df.loc['None']
All that said, I was curious how to reference np.NaN in the index
s = pd.Series([1, 2], [0, np.nan])
s.iloc[s.index.get_loc(np.nan)]
2