Parse Excel file with multi-index in Pandas - pandas

Question

What would be the best way to parse the above Excel file in a Pandas dataframe? The idea would be to be able to update data easily, adding columns, dropping lines. For example, for every origin, I would like to keep only output3. Then for every column (2000, ....,2013) divide it by 2 given a condition (say value > 6000) .
Below is what I tried: first to parse and drop the unnecessary lines, but it's not satisfactory, as I had to rename columns manually. So this doesn't look very optimal as a solution. Any better idea?
df = pd.read_excel("myExcel.xlsx", skiprows=2, sheet_name='1')
cols1 = list(df.columns)
cols1 = [str(x)[:4] for x in cols1]
cols2 = list(df.iloc[0,:])
cols2 = [str(x) for x in cols2]
cols = [x + "_" + y for x,y in zip(cols1,cols2)]
df.columns = cols
df = df.drop(["Unna_nan"], axis =1).rename(columns ={'Time_Origine':'Country','Unna_Output' : 'Series','Unna_Ccy' : 'Unit','2000_nan' : '2000','2001_nan': '2001','2002_nan':'2002','2003_nan' : '2003','2004_nan': '2004','2005_nan' : '2005','2006_nan' : '2006','2007_nan' : '2007','2008_nan' : '2008','2009_nan' : '2009','2010_nan' : '2010','2011_nan': '2011','2012_nan' : '2012','2013_nan':'2013','2014_nan':'2014','2015_nan':'2015','2016_nan':'2016','2017_nan':'2017'})
df.drop(0,inplace=True)
df.drop(df.tail(1).index, inplace=True)
idx = ['Country', 'Series', 'Unit']
df = df.set_index(idx)
df = df.query('Series == "Output3"')

Answer 1

Without having an excel like this I think something like this might work.
In order to only get the rows from output3, you can use the following:
df = pd.read_excel("myExcel.xlsx", skiprows=2, sheet_name='1')
df = df.loc[df['Output'] == 'output3']
Now dividing every cell by 2 if the cell value is greater than 6000 by using pandas apply:
def foo(bar):
if bar > 6000:
return bar / 2
return bar
for col in df.columns:
try:
int(col) # to check if this column is a year
df[col] = df[col].apply(foo)
except ValueError:
pass

Answer 2

#read first 2 rows to MultiIndex nad remove last one
df = pd.read_excel("Excel1.xlsx", skiprows=2, header=[0,1], skipfooter=1)
print (df)
#create helper DataFrame
cols = df.columns.to_frame().reset_index(drop=True)
cols.columns=['a','b']
cols['a'] = pd.to_numeric(cols['a'], errors='ignore')
cols['b'] = cols['b'].replace('Unit.1','tmp', regex=False)
#create new column by condition
cols['c'] = np.where(cols['b'].str.startswith('Unnamed'), cols['a'], cols['b'])
print (cols)
a b c
0 Time Country Country
1 Time Series Series
2 Time Unit Unit
3 Time tmp tmp
4 2000 Unnamed: 4_level_1 2000
5 2001 Unnamed: 5_level_1 2001
6 2002 Unnamed: 6_level_1 2002
7 2003 Unnamed: 7_level_1 2003
8 2004 Unnamed: 8_level_1 2004
9 2005 Unnamed: 9_level_1 2005
10 2006 Unnamed: 10_level_1 2006
11 2007 Unnamed: 11_level_1 2007
12 2008 Unnamed: 12_level_1 2008
13 2009 Unnamed: 13_level_1 2009
14 2010 Unnamed: 14_level_1 2010
15 2011 Unnamed: 15_level_1 2011
16 2012 Unnamed: 16_level_1 2012
17 2013 Unnamed: 17_level_1 2013
18 2014 Unnamed: 18_level_1 2014
19 2015 Unnamed: 19_level_1 2015
20 2016 Unnamed: 20_level_1 2016
21 2017 Unnamed: 21_level_1 2017
#overwrite columns by column c
df.columns = cols['c'].tolist()
#forward filling missing values
df['Country'] = df['Country'].ffill()
df = df.drop('tmp', axis=1).set_index(['Country','Series','Unit'])
print (df)