how to pick single row from multiple rows in pandas - sql

Question

if I have two rows for the same ID, then I have to check for col2 and pick rows with values N and Q and skip the row with U. If there is single record with col2=U, then let it be. so for ID 123 and 555, output is with col2 N and Q resp.
ID Col1 Col2 Col3
123 AAA N true
123 BBB U true
000 AAA N true
222 CCC U false
555 FIC Q false
555 VAN U true
expected output is:
Expected output:
ID Col1 Col2 Col3
123 AAA N true
000 AAA N true
222 CCC U false
555 FIC Q false
how can I do this in pandas ?
in sql, I tried with having count(*)>1, and then picked these columns.

Answer 1

You can use this code:
df.drop_duplicates('ID')
Above code keep always first record. You can change this with last instead of first record.
df.drop_duplicates(subset='ID', keep="first")
df.drop_duplicates(subset='ID', keep="last")
or you may sort for any column and then using of drop_duplicates method. In this way, (by order Ascending or Descending) you may use keep="first" for Min or Max.

Answer 2

One simple approach is to sort your dataframe by Col2 ensuring that 'U' will end up last. There are several possibilities:
pandas.Categorical
This sets an ordered categorical type on Col2
categories = np.append(np.setdiff1d(df['Col2'], ['U']), ['U'])
df['Col2'] = pd.Categorical(df['Col2'], categories=categories, ordered=True)
df.sort_values(by='Col2').groupby('ID').first()
Split dataframe
This splits the dataframe in two based on the values of Col2 (not-U and U), and concatenates the two parts to ensure the U are at the end
pd.concat([df.query('Col2 != "U"'), df.query('Col2 == "U"')]).groupby('ID').first()
Custom sort order
This manually defines the sorting order from a list
custom_order = ['N', 'Q', 'Z', 'U']
custom_order_dict = dict(zip(custom_order, range(len(custom_order))))
df.sort_values(by='Col2', key=lambda x: x.map(custom_order_dict)).groupby('ID').first()
input
ID Col1 Col2 Col3
0 123 AAA N True
1 123 BBB U True
2 0 AAA N True
3 222 CCC U False
4 555 FIC Q False
5 555 VAN U True
6 777 UUU U False
7 777 ZZZ Z True
8 999 UUU U False
9 999 NNN N True
output
Col1 Col2 Col3
ID
000 AAA N True
123 AAA N True
222 CCC U False
555 FIC Q False
777 ZZZ Z True
999 NNN N True

Answer 3

I tried a solution with multiple steps. this might not be the best way to do it but I did not find any other solution.
First step:
Separate records/rows for multiple ID's
df_multiple_record=pd.concat(g for _, g in df.groupby("ID") if len(g) > 1)
Output:
ID Col1 Col2 Col3
123 AAA N true
123 BBB U true
555 FIC Q false
555 VAN U true
Second Step:
drop the record with col2='U'
df_drop_U=df_multiple_record[df_multiple_record['Col2']!='U']
output:
ID Col1 Col2 Col3
123 AAA N true
555 FIC Q false
third Step:
drop the duplicates on ID from main extract to get the records for single occurance of ID
df_single_record=df.drop_duplicates(subset=['ID'],keep=False)
output:
ID Col1 Col2 Col3
000 AAA N true
222 CCC U false
fourth step:
concatenate single record df with the df where we drop U
df_final=pd.concat([df_single_record,df_drop_U],ignore_index=True)
output:
ID Col1 Col2 Col3
000 AAA N true
222 CCC U false
123 AAA N true
555 FIC Q false