How to align several different dataframes with different shapes on common column? - pandas

Question

I have a few different data frames like this below
df1
idx col1 col2 col3
2020-11-20 01:00:00 1 5 9
2020-11-20 02:00:00 2 6 10
2020-11-20 03:00:00 3 7 11
2020-11-20 04:00:00 4 8 12
df2
idx col4 col5 col6
2020-11-20 02:00:00 13 15 17
2020-11-20 03:00:00 14 16 18
df3
idx col7 col8 col9
2020-11-20 01:00:00 19 20 21
and essentially I need to keep all the columns from all DF's but align the values on the timestamp that is the index for each dataframe. My expected output is this
df_merged
idx col1 col2 col3 col4 col5 col6 col7 col8 col9
2020-11-20 01:00:00 1 5 9 NaN NaN NaN 19 20 21
2020-11-20 02:00:00 2 6 10 13 15 17 NaN NaN NaN
2020-11-20 03:00:00 3 7 11 14 16 18 NaN NaN NaN
2020-11-20 04:00:00 4 8 12 NaN NaN NaN NaN NaN NaN
I have tried various things like merge, concat, join, manually doing it for hours now and I am stumped why it wont work. These df's are simplified versions, but my issue with these approaches are that my df1 has a length of 1619, df2 has a length of 1619, df3 has a length of 1617, and df4 (not here but follows same idea) has a length of 1613. When I try this
df_merged = reduce(lambda left,right: pd.merge(left,right,how='left'), [df1,df2,df3,df4]) what happens is that the df_merged size is now 12k rows (not 1619 like the original df). I tried dropping duplicates as well on the final df_merged and that only left me with like 600 rows. I also have tried manually combining them with loc, iloc and isin() but still no luck.
Really any help would be greatly appreciated!

Answer 1

Use merge with how = 'outer'.
Demonstration:
# data preparation
string = """idx col1 col2 col3
2020-11-20 01:00:00 1 5 9
2020-11-20 02:00:00 2 6 10
2020-11-20 03:00:00 3 7 11
2020-11-20 04:00:00 4 8 12"""
data = [x.split(' ') for x in string.split('\n')]
df = pd.DataFrame(data[1:], columns = data[0])
string = """idx col4 col5 col6
2020-11-20 02:00:00 13 15 17
2020-11-20 03:00:00 14 16 18"""
data = [x.split(' ') for x in string.split('\n')]
df2 = pd.DataFrame(data[1:], columns = data[0])
string = """idx col7 col8 col9
2020-11-20 01:00:00 19 20 21"""
data = [x.split(' ') for x in string.split('\n')]
df3 = pd.DataFrame(data[1:], columns = data[0])
#solution
df.merge(df2, on = 'idx', how = 'outer').merge(df3, on='idx', how='outer')
Output:
idx col1 col2 col3 col4 col5 col6 col7 col8 col9
0 2020-11-20 01:00:00 1 5 9 NaN NaN NaN 19 20 21
1 2020-11-20 02:00:00 2 6 10 13 15 17 NaN NaN NaN
2 2020-11-20 03:00:00 3 7 11 14 16 18 NaN NaN NaN
3 2020-11-20 04:00:00 4 8 12 NaN NaN NaN NaN NaN NaN