Remove the identical values, and leave only different - pandas

Question

I would like to know if there is more optimal solution to leave the different value (to easily catch them) and to remove identical values under some columns.
merged = pd.merge(us_df, gb_df, how='outer', indicator=True)
res = pd.merge(merged[merged['_merge'] == 'left_only'].drop('_merge', axis=1),
merged[merged['_merge'] == 'right_only'].drop('_merge', axis=1),
on=us_df.columns.tolist()[0:col_range],
how='outer',
suffixes=('_US', '_GB')).fillna(' ')
cols = [col for col in res.columns.tolist() if '_US' in col or '_GB' in col]
sorted_cols = [col for col in res.columns.tolist() if '_US' not in col and '_GB' not in col] + sorted(cols)
I get this table (res):
Id
ages_GB
ages_US
salary_GB
salary_US
6
45
45
34
67
43
12
11
65
65
So far, I used this iteration:
cols = [ages_US, salary_US, ages_GB, salary_GB]
for i, row in res.iterrows():
for us, gb in zip(cols[:len(cols) // 2], cols[len(cols) // 2:]):
if row[us] == row[gb]:
res.at[i, us] = res.at[i, gb] = ' '
to get the result (where identical values under columns in cols are replaced with " " (space)):
Id
ages_GB
ages_US
salary_GB
salary_US
6
34
67
43
12
11
Is there another method to get the similar result?

Answer 1

Given your example I think loc offers a simpler solution assuming you want to compare two sets of columns.
I will first recreate a reproducible example of your dataset (I would recommend you create this in future questions as it makes it easier to understand and answer you question: How to create a Minimal, Reproducible Example)
d = {
'ages_GB': [45, 12],
'ages_US': [45, 11],
'salary_GB': [34, 65],
'salary_US': [67, 65]
}
df = pd.DataFrame(data=d)
print(df)
Initial DataFrame
ages_GB ages_US salary_GB salary_US
0 45 45 34 67
1 12 11 65 65
The simplest solution I can think of is to use loc to just reassign records to "" or NaN where ages_GB == ages_US & salary_GB == salary_US.
df.loc[df.ages_GB == df.ages_US, ['ages_GB', 'ages_US']] = ["", ""]
df.loc[df.salary_GB == df.salary_US, ['salary_GB', 'salary_US']] = ["", ""]
Output
ages_GB ages_US salary_GB salary_US
0 34 67
1 12 11

Answer 2

For a generic method, you can groupby on axis=1 using the columns prefixes, and get the duplicated values to use with mask:
prefix = df.columns.str.extract('^([^_]+)', expand=False)
# ['Id', 'ages', 'ages', 'salary', 'salary']
m = df.groupby(prefix, axis=1).transform(lambda s: s.duplicated(keep=False))
out = df.mask(m, '')
Output:
Id ages_GB ages_US salary_GB salary_US
0 6 34 67
1 43 12 11
Intermediate m:
Id ages_GB ages_US salary_GB salary_US
0 False True True False False
1 False False False True True