data manipulation with na values - pandas

Question

My code are like that df2 DataFrame (toy data) :
import pandas as pd
import numpy as np
end = '20201117'
np.random.seed(107)
df = pd.DataFrame()
for i in range(10):
start = np.random.choice(['20000101', '20100101', '20160101', '20121010'])
df_tempo = pd.DataFrame({'product': 'p'+str(i),
'category': 'cat'+str(np.random.choice(4))
},
index=pd.bdate_range(start=pd.to_datetime(start),
end=end,
freq=np.random.choice(['D', 'W'])))
df_tempo['valeur1'] = np.random.randint(50, 100, df_tempo.shape[0])
df_tempo['valeur2'] = np.random.randint(100, 200, df_tempo.shape[0])
df = pd.concat([df, df_tempo])
df = df.reset_index().rename(columns={'index': 'date'})
df2 = df.pivot(index='date', columns=['category', 'product'], values=['valeur1', 'valeur2'])
from this I would like to calculate the growth rate for 2020, 2019,..years
I tried :
df2.apply([
lambda x: 100*x.valeur1[x.valeur1['2020'].last_valid_index()]/x.valeur1[x.valeur1['2019'].last_valid_index()]-100,
])
but get an error message with is (I would like to calculate different stat for valeur1 and valeur2) :
AttributeError: 'Series' object has no attribute 'valeur1'
so I did (it's not what I would like to do because as I said I would like to apply different stat on valeur1 and valeur2) :
df2['valeur1'].apply([
lambda x: 100*x[x['2020'].last_valid_index()]/x[x['2019'].last_valid_index()]-100,
lambda x: 100*x[x['2010'].last_valid_index()]/x[x['2009'].last_valid_index()]-100,
])
It works except that
lambda x: 100*x[x['2010'].last_valid_index()]/x[x['2009'].last_valid_index()]-100
because I have data series that have no data for 2010 and so return an error.
I tried then a dropna()but not help
Any ideas to help on these 2 problems ?

Answer 1

Since date is already the index of df2, pd.Grouper(freq="Y").last() can be used to retrieve the value on the last date. Is this what you are expecting?
Note: df2 = df2.groupby(pd.Grouper(freq="Y")).ffill() will fill the last date of each year with previous valid value(s). For this sample dataset, however, no difference was introduced.
Code
df_last = df2.groupby(pd.Grouper(freq="Y")).last()
out = df_last.diff() / df_last.shift() * 100
Result
print(out)
valeur1 ... valeur2
category cat0 cat2 ... cat0 cat2
product p0 p1 p2 ... p7 p8 p9
date ...
2000-12-31 NaN NaN NaN ... NaN NaN NaN
2001-12-31 NaN 62.000000 -35.869565 ... NaN -37.113402 NaN
2002-12-31 NaN 8.641975 28.813559 ... NaN 13.934426 NaN
2003-12-31 NaN -40.909091 19.736842 ... NaN 30.935252 NaN
2004-12-31 NaN 25.000000 -30.769231 ... NaN -18.131868 NaN
2005-12-31 NaN -20.000000 57.142857 ... NaN -18.791946 NaN
2006-12-31 NaN 19.230769 -15.151515 ... NaN 14.049587 NaN
2007-12-31 NaN 25.806452 -40.476190 ... NaN -5.072464 NaN
2008-12-31 NaN -1.282051 4.000000 ... NaN 32.824427 NaN
2009-12-31 NaN 7.792208 38.461538 ... NaN -9.770115 NaN
2010-12-31 NaN -33.734940 -26.388889 ... NaN 12.738854 NaN
2011-12-31 NaN 29.090909 83.018868 ... NaN -12.994350 43.442623
2012-12-31 NaN -19.718310 -40.206186 ... NaN -5.194805 13.142857
2013-12-31 NaN 17.543860 34.482759 ... NaN 12.328767 -19.696970
2014-12-31 NaN 8.955224 23.076923 ... NaN -22.560976 -18.867925
2015-12-31 NaN -6.849315 -14.583333 ... NaN 43.307087 23.255814
2016-12-31 NaN 33.823529 -7.317073 ... NaN -22.527473 -6.289308
2017-12-31 -22.580645 -35.164835 -2.631579 ... -8.176101 10.638298 -22.818792
2018-12-31 -6.944444 55.932203 5.405405 ... -30.821918 1.923077 35.652174
2019-12-31 14.925373 -8.695652 -1.282051 ... 70.297030 -6.918239 19.230769
2020-12-31 2.597403 8.333333 -20.779221 ... -28.488372 -6.081081 -17.204301
[21 rows x 20 columns]