Identifying rows for deletion/update based on criteria from matching rows - sql

Question

I have a data set that contains rows considered duplicates based on certain fields. I need to match the duplicate rows, evaluate the non-matching fields, and flag one of them for deletion. A sample table is:
ID Col1 Col2 Col3
1 A B CC
2 A B DD
3 E F GG
4 E F HH
So I need to identify rows 1 & 2 as duplicates based on Col1 and Col2 matching, and compare the Col3 fields, ultimately flagging either row 1 or 2 for deletion. And the same for rows 3 & 4. This table consists entirely of rows that match at least one other row across Col1 and Col2.
My first thought was to join onto itself to flatten the rows into this format:
t1.ID t2.ID t1.Col1 t1.Col2 TableOneCol3 TableTwoCol3
1 2 A B CC DD
3 4 E F GG HH
Then it would be simple to evaluate TableOneCol3 and TableTwoCol3 for each row.
I tried to do this with a self join:
select t1.ID, t2.ID, t1.Col1, t1.Col2, t1.Col3 as TableOneCol3, t2.col3 as TableTwoCol3
into #temptable
from tableOne t1
join tableTwo t2
where t1.Col1 = t2.Col2
and t1.Col2 = t2.Col2
and t1.ID <> t2.ID
But of course this doesn't remove duplicates - just adds the duplicate field information to each row.
I went down the path of pivoting the data - but I end up with a similar result -I pivot the duplicates as well.
I dug through SO but not sure if I have the specific words for what I need to do (the admittedly vague title might be a giveaway - apologies for that). I found many examples of flattening data into single columns and pivots, but nothing that would flatten paired rows and remove one of them from the resultset.
Not sure if I'm going down the wrong road for this or not. It seems I need to evaluate each row in the context of what has been evaluated prior - but I'm not sure how to do this without resorting to a cursor.

Answer 1

It is extremely unclear what you are trying to do. I tossed together a couple of quick ideas that might be what you are trying to do.
if OBJECT_ID('tempdb..#Something') is not null
drop table #Something
create table #Something
(
ID int
, Col1 char(1)
, Col2 char(1)
, Col3 char(2)
)
insert #Something
(
ID
, Col1
, Col2
, Col3
)
VALUES
(1, 'A', 'B', 'CC'),
(2, 'A', 'B', 'DD'),
(3, 'E', 'F', 'GG'),
(4, 'E', 'F', 'HH');
with SortedResults as
(
select *
, ROW_NUMBER() over(partition by Col1, Col2 order by Col3) as RowNum
from #Something
)
delete SortedResults
where RowNum > 1
select *
from #Something;
--OR maybe you want to cross tab the data???
drop table #Something
GO
create table #Something
(
ID int
, Col1 char(1)
, Col2 char(1)
, Col3 char(2)
)
insert #Something
(
ID
, Col1
, Col2
, Col3
)
VALUES
(1, 'A', 'B', 'CC'),
(2, 'A', 'B', 'DD'),
(3, 'E', 'F', 'GG'),
(4, 'E', 'F', 'HH');
with SortedResults as
(
select *
, ROW_NUMBER() over(partition by Col1, Col2 order by Col3) as RowNum
from #Something
)
select
MAX(case when RowNum = 1 then ID end) as ID_1
, MAX(case when RowNum = 2 then ID end) as ID_2
, Col1
, Col2
, MAX(case when RowNum = 1 then Col3 end) as Col3_1
, MAX(case when RowNum = 2 then Col3 end) as Col3_2
from SortedResults
group by
Col1
, Col2

Answer 2

You could obtain a table in a form similar to the one you describe with use of the LEAD() analytic function. This will have the benefit that it works reasonably well when your dupes come in groups larger than two. For example:
select
ID,
lead(ID) over (partition by col1, col2 order by col3) as nextId,
Col1,
Col2,
Col3,
lead(Col3) over (partition by col1, col2 order by col3) as nextCol3
into #temptable
from tableOne
Results would be of the form
ID nextId Col1 Col2 Col3 nextCol3
1 2 A B CC DD
2 NULL A B DD NULL
3 4 E F GG HH
4 NULL E F HH NULL
If you are confident that you don't need to deal with groups larger than two then you could get the exact table you wanted by afterward filtering out, say, the rows having nextId IS NULL.