Concatenating two tables in pandas, giving preference to one for identical indices - pandas

Question

I'm trying to combine two data sets df1 and df2. Rows with unique indices are always copied, rows with duplicate indices should always be picked from df1. Imagine two time series, and df2 has additional data but is of lesser quality than df1, so ideally data comes from df1, but I'm willing to backfill from df2
df1:
date value v2
2020/01/01 df1-1 x
2020/01/03 df1-3 y
df2:
date value v2
2020/01/02 df2-2 a
2020/01/03 df2-3 b
2020/01/04 df2-4 c
are combined into
date value v2
2020/01/01 df1-1 x
2020/01/02 df2-2 a
2020/01/03 df1-3 y
2020/01/04 df2-4 c
The best I've got so far is
df = df1.merge(df2, how="outer",left_index=True, right_index=True, suffixes=('','_y'))
df['value'] = result_df['value'].combine_first(result_df['value_y'])
df['v2'] = result_df['v2'].combine_first(result_df['v2'])
df=df[['value', 'v2']]
That gets the job done, but it seems unnecessarily clunky. Is there a more idiomatic way to achieve this?

Answer 1

Your wrote rows with unique indices but you didn't show them,
so I assume that date column should be treated as these indices.
Furthermore, I noticed that all values in your DataFrames are not NaN.
If you guaratee this, you can run:
df1.set_index('date').combine_first(df2.set_index('date'))\
.reset_index()
Steps:
combine_first - combine both DataFrames based on values
in their date columns.
reset_index - change date column (for now the index) into
a "regular" column.
Another possible approach
If both your DataFrames have "standard" index (consecutive numbers starting
from 0) and you want to keep only rows for just these unique indices,
you can run:
df = pd.concat([df1, df2]).reset_index().drop_duplicates(subset='index')\
.set_index('index')
df.index.name = None
But then the result is:
date value v2
0 2020-01-01 df1-1 x
1 2020-01-03 df1-3 y
2 2020-01-04 df2-4 c
so it is different from what you presented as are combined into
(as I assume - your expected result). This time you lost the row
with v2 == 'a'.
Yet another approach
Based also on the assumption that all values in your DataFrames are not NaN:
df1.combine_first(df2)
The result will be just as the previous one.