Merging unequal dataframes with partial string match - pandas

Question

I want to merge two data frames. df1 has 115 rows and df2 has 600,000 rows.
f1 = pd.DataFrame({'Invoice': ['20561', '20562', '20563', '20564'],
'Currency': ['EUR', 'EUR', 'EUR', 'USD']})
df2 = pd.DataFrame({'Ref': ['20561', 'INV20562', 'INV20563BG', '20564'],
'Type': ['01', '03', '04', '02'],
'Amount': ['150', '175', '160', '180'],
'Comment': ['bla', 'bla', 'bla', 'bla']})
print(df1)
Invoice Currency
0 20561 EUR
1 20562 EUR
2 20563 EUR
3 20564 USD
print(df2)
Ref Type Amount Comment
0 20561 01 150 bla
1 INV20562 03 175 bla
2 INV20563BG 04 160 bla
3 20564 02 180 bla
I applied following code:
compList = '|'.join(df1['Invoice'].tolist())
df2['compMatch'] = df2.Ref.str.contains(compList)
# drop unmatched articles
df2 = df2[df2['compMatch']==True]
df2['content'] = df2['Ref'].str.lower().str.split()
df2['matchedName'] = df2['content'].apply(lambda x: [item for item in x if item in df1['Invoice'].tolist()])
df1['Invoice'].tolist()])
print (df2)
Ref Type Amount Comment compMatch content matchedName
0 20561 01 150 bla True [20561] [20561]
1 INV20562 03 175 bla True [inv20562] []
2 INV20563BG 04 160 bla True [inv20563bg] []
3 20564 02 180 bla True [20564] [20564]
here you see, a few MatchedNames are missing for Ref INV20562 and Ref INV20563BG.
What's wrong with this code? Is there any other solution?

Answer 1

Looks like you want to merge on the digits of the ref:
df2.merge(df1,
left_on=df2['Ref'].str.extract(r'(\d+)', expand=False),
right_on='Invoice', how='left')
Output:
Ref Type Amount Comment Invoice Currency
0 20561 01 150 bla 20561 EUR
1 INV20562 03 175 bla 20562 EUR
2 INV20563BG 04 160 bla 20563 EUR
3 20564 02 180 bla 20564 USD