Identify duplicates based on multiple columns - sql

Question

I want to identify duplicates in a db based on multiple columns from various tables. In the example below, 1&5 and 2&4 are duplicates - as all four columns have same values. How do I identify such records using a sql? I have used group by having count>1 when I had to identify duplicates based on a single column, but I am unsure how to identify them based on multiple columns. However, I see that when I do group by having count>1 based on all 4 columns, #3 and 6 are showing up, they are technically not duplicates per my requirement.
T1
ID | Col1 | Col2
---| --- | ---
1 | A | US
2 | B | FR
3 | C | AU
4 | B | FR
5 | A | US
6 | D | UK
T2
ID | Col1
---| ---
1 | Apple
1 | Kiwi
2 | Pear
3 | Banana
3 | Banana
4 | Pear
5 | Apple
T3
ID | Col1
---| ---
1 | Spinach
1 | Beets
2 | Celery
3 | Radish
4 | Celery
5 | Spinach
6 | Celery
6 | Celery
My expected result would be:
1 A US Apple Spinach
5 A US Apple Spinach
2 B FR Pear Celery
4 B FR Pear Celery

Answer 1

For your sample data, you can achieve this using inner join-ing all three tables and using just group by tA.Col1 having count(tA.Col1)>1 in where clause sub-query as below to obtain your desired result.
SELECT t1.ID,
t1.Col1,
t1.Col2,
t2.Col1,
t3.Col1
FROM table1 t1
JOIN table2 t2 ON t1.ID = t2.ID
JOIN table3 t3 ON t1.ID = t3.ID
WHERE t1.Col1 IN
( SELECT tA.Col1
FROM table1 tA
GROUP BY tA.Col1
HAVING count(tA.Col1)>1)
ORDER BY t1.ID;
Result
ID Col1 Col2 Col1 Col1
-----------------------------------
1 A US Apple Spinach
2 B FR Pear Celery
4 B FR Pear Celery
5 A US Apple Spinach
You can check the demo here
Hope this will help.

Answer 2

The problem is your result set needs to include the ID column which is unique. So a straightforward GROUP BY ... HAVING won't cut it. This would work.
with cte as
( select t1.id
, t1.col1 as t1_col1
, t1.col2 as t1_col2
, t2.col1 as t2_col1
, t3.col1 as t3_col1
from t1
join t2 on t1.id = t2.id
join t3 on t1.id = t3.id
)
select cte.*
from cte
where (t1_col1, t1_col2, t2_col1, t3_col1) in
( select t1_col1, t1_col2, t2_col1, t3_col1
from cte
group by t1_col1, t1_col2, t2_col1, t3_col1 having count(*) > 1)
/
The use of the sub-query factoring syntax is optional, but I find it useful to signal that the subquery is used more than one in the query.
"I have encountered another scenario in the data, some of the IDs have same values in T2 and T3 and they are showing up as dups."
The duplicated IDs in the child tables cause Cartesian products in the joined subquery, which causes false positives in the main result set. Ideally you should be able to handle this by introducing additional filters on those tables to remove the unwanted rows. However, if the data quality is so poor that there are no valid rules you will have to fall back on distinct:
with cte as (
select t1.id
, t1.col1 as t1_col1
, t1.col2 as t1_col2
, t2.col1 as t2_col1
, t3.col1 as t3_col1
from t1
join ( select distinct id, col1 from t2) t2 on t1.id = t2.id
join ( select distinct id, col1 from t3) t3 on t1.id = t3.id
) ...

Answer 3

You can add all columns in the group by clause for which you want to find the duplicate and then write the count condition in having claus
select t1.id,t1.col1,t2.col2,t2.col3,t3.col4 from t1 join t2 on t1.id=t2.id join t3 on t3.id=t1.id where (t1.col1,t2.col2,t2.col3,t3.col4) in (
select t1.col1,t2.col2,t2.col3,t3.col4
from t1 join t2 on t1.id=t2.id join t3 on t3.id=t1.id
group by t1.col1,t2.col2,t2.col3,t3.col4
having count(*) >1 )