Finding duplicate entries - pandas

Question

I am working with the 515k Hotel Reviews dataset from Kaggle. There are 1492 unique hotel names and 1493 unique addresses. So at first it would appear that one (or possibly more) hotel has more than one address. But, if I do a groupby.count on the data, I get 1494 whether I groupby HotelName followed by Address or if I reverse the order.
In order to make this reproducible, hopefully this simplification will suffice:
data = {
'HotelName': ['A', 'B', 'C', 'D', 'A', 'B', 'C', 'D', 'A', 'B', 'B', 'C', 'C'],
'Address': [1, 2, 3, 4, 1, 2, 3, 4, 2, 2, 2, 3, 5]
}
df = pd.DataFrame(data, columns = ['HotelName', 'Address'])
df['HotelName'].unique().shape[0] # Returns 4
df['Address'].unique().shape[0] # Returns 5
df.groupby(['Address', 'HotelName']).count().shape[0] # Returns 6
df.groupby(['Address', 'HotelName']).count().shape[0] # Returns 6
I would like to find the hotel names that have different addresses. So in my example, I would like to find the A and C along with their addresses (1,2 and 3,5 respectively). That code should be enough for me to also find the addresses that have duplicate hotel names.

Answer 1

Use the nunique groupby aggregator:
>>> n_uniq = df.groupby('HotelName')['Address'].nunique()
>>> n_uniq
HotelName
A 2
B 1
C 2
D 1
Name: Address, dtype: int64
If you want to look at the distinct hotels with more than one address in the original dataframe,
>>> hotels_with_mult_addr = n_uniq.index[n_uniq > 1]
>>> df[df['HotelName'].isin(hotels_with_mult_addr)].drop_duplicates()
HotelName Address
0 A 1
2 C 3
8 A 2
12 C 5

Answer 2

If I understand your correctly, we can check which hotel has more then 1 unique adress with groupby.transform(nunqiue):
m = df.groupby('HotelName')['Address'].transform('nunique').ne(1)
print(df.loc[m])
HotelName Address
0 A 1
2 C 3
4 A 1
6 C 3
8 A 2
11 C 3
12 C 5
If you want to get a more concise view on which the duplicates are, use groupby.agg(set):
df.loc[m].groupby('HotelName')['Address'].agg(set).reset_index(name='addresses')
HotelName addresses
0 A {1, 2}
1 C {3, 5}
Step by step:
transform(nunique) gives us the amount of unique adresses next to each row
df.groupby('HotelName')['Address'].transform('nunique')
0 2
1 1
2 2
3 1
4 2
5 1
6 2
7 1
8 2
9 1
10 1
11 2
12 2
Name: Address, dtype: int64
Then we check which rows are not equal (ne) to 1 and filter those:
df.groupby('HotelName')['Address'].transform('nunique').ne(1)
0 True
1 False
2 True
3 False
4 True
5 False
6 True
7 False
8 True
9 False
10 False
11 True
12 True
Name: Address, dtype: bool

Answer 3

Groupby didn't do what you were expected. After you did the groupby here is what you got
HotelName Address
0 A 1
4 A 1
HotelName Address
8 A 2
HotelName Address
1 B 2
5 B 2
9 B 2
10 B 2
HotelName Address
2 C 3
6 C 3
11 C 3
HotelName Address
3 D 4
7 D 4
HotelName Address
12 C 5
There are indeed 6 combinations!
If you want to know the duplication in each group, you should check the row index.

Answer 4

Here is the long way to do it, where in newdf['count'] == 1 is unique
df = pd.DataFrame(data, columns = ['HotelName', 'Address'])
df = df.sort_values(by = ['HotelName','Address']).reset_index(drop = True)
count = df.groupby(['HotelName','Address'])['Address'].count().reset_index(drop = True)
df['rownum'] = df.groupby(['HotelName','Address']).cumcount()+1
dfnew = df[df['rownum']==1].reset_index(drop = True).drop(columns = 'rownum')
dfnew['count'] = count
dfnew