removing duplicates based on a number of specific criteria in oracle - sql

Question

I am working on a project where I must examine duplicate records and discern which of the records I must keep. There is a general criteria to be met for the record based on the attributes we are looking at. The following table examines the relationships between the criteria.
Table1
+----------+-----+-------+-------+-------+-------+
| dup_id | idm | ucode | great | good |yo2005 |
+----------+-----+-------+-------+-------+-------+
| a | 1 | 6 | yes | yes | yes |
| a | 2 | 1 | no | yes | yes |
| a | 3 | 1 | no | no | yes |
| b | 4 | 1 | yes | yes | no |
| b | 5 | 1 | no | no | no |
| c | 6 | 7 | no | no | yes |
| c | 7 | 1 | yes | no |no |
| d | 8 | 6 | no | yes |no |
| d | 9 | 1 | yes | no |no |
| e | 10 | 3 | yes | no |yes |
| e | 11 | 4 | no | yes |no |
| f | 12 | 1 | yes | yes | yes |
| f | 13 | 1 | yes | no |yes |
| g | 14 | 1 | no | no |yes |
| g | 15 | 1 | yes | no |no |
+----------+----+--------+-------+-------+-------+
Table 2
+-----+-------+
| ido | yo1998|
+-----+-------+
| 1 | yes |
| 2 | no |
| 3 | no |
| 4 | no |
| 5 | no |
| 6 | no |
| 7 | no |
| 8 | yes |
| 9 | yes |
| 10 | yes |
| 11 | yes |
| 12 | yes |
| 13 | no |
| 14 | yes |
| 15 | no |
+----+-------+
The tables have other records we would like to keep, but these are the main ones that fit the criteria
Table1
• dup_id- this is the id of the collection of all duplicates that are associated with it. This can have 2 or more records associated with it
• idm-the id of records in table 1, matches the ido in table 2
• ucode-this attribute has a duplicate signifier from a previous classification. If it is a value of 6, then it is considered a duplicate (but for some reason the new algorithm accepted it as non duplicate)
• great-this is a field that is preferred because it was verified at some point
• good-this is a field that is preferred, but has not been verified
• yo2005- data that was collected in 2005
Table2
• ido-the id of records in table 2; matches the idm in table 1
• yo1998-data that was collected in 1998
The issue is, we have so many records to sift through. What I have been attempting to do is to develop a query for each criteria to attempt to filter the data we need to look at down.
The criteria
The order of importance of the criteria is as follows:
• ucode- if one of the records in a dupid has a ucode =6, that means it is already known as a duplicate record, so the other ucodes take precedence. For example, dupid d has 2 records, so we know that the correct one is idm=8. For example, if our table has 10,000 records, this may pick up 2000 of them, which leaves us with 8000 to be manually examined.
• great- this is the 2nd level of importance for us. If great = yes, then we would like this record to be selected from any records that were not resolved by the first query. For example, of the 8000 left from the query above, this might pick up another 1000, leaving us with 7000 to be manually examined.
• good-this is 3rd level of importance to us. If great = no, but good = yes, then this would be our choice for anything not earlier resolved. For example, of the 7000 left from the query above, this might pick up another 500, leaving us with 6500 to be manually examined.
• At this point we have 2 tables involved; our 4th priority is that both yo2005 and yo1998 = yes. For example, of the 6500 left in the query above, this might pick up another 1000, leaving us with 5500 to be manually examined.
• If both are not equal to yes, yo2005 is our 5th priority. For example, of the 5500 left in the query above, this might pick up another 2000, leaving us with 3500 to be manually examined.
• yo1998 = ‘yes’ is our final priority. For example, of the 3500 left in the query above, this might pick up another 1000, leaving us with 2500 to be manually examined.
As you can see, this would cut out a great deal of the manual examination of the records.
Ideally, there would be 2 output tables; one for all the records that fit the critera (which is 7500 records). Maybe even a new field can be created with the justifications for it, to be populated by which criteria it was based on. We would also need another table that contains the records that did not meet any of the criteria, so that we can further investigate those records to decide which is the duplicate. Unfortunately, I am not very well versed in sql, so I don’t even know if something like this is possible.
Thank you for your time.

Answer 1

You can write all of these in SQL. Below is the ucode one. It selects all the dupids that have two records, with one of them having ucode = 6. Then picks the other record:
SELECT *
FROM t1
WHERE ucode <> 6
AND dupid IN
(SELECT dupid
FROM t1
INNER JOIN t2
ON t1.idm = t2.ido
GROUP BY dupid
HAVING COUNT(*) = 2
AND EXISTS
(SELECT 1
FROM t1 sub
WHERE ucode = 6
AND sub.dupid = t1.dupid))
This one will give you all the records marked as great and ucode does not = 6:
SELECT *
FROM t1
WHERE great = 'yes'
AND ucode <> 6
This one will give you all the records marked as good that do not have a record in the same dupid marked as great, excluding those with ucode = 6:
SELECT *
FROM t1
WHERE good = 'yes'
AND ucode <> 6
AND NOT EXISTS
(SELECT 1
FROM t1 sub
WHERE great = 'yes'
AND sub.dupid = t1.dupid)
This one finds all records where yo2005 = yes and great = no and good = no and unicode is not equal 6:
SELECT *
FROM t1
WHERE yo2005 = 'yes'
AND ucode <> 6
AND NOT EXISTS
(SELECT 1
FROM t1 sub
WHERE (great = 'yes'
OR good = 'yes')
AND sub.dupid = t1.dupid)
Finally, this one shows the records where yo1998 = yes and all other conditions fail:
SELECT *
FROM t1
INNER JOIN t2
ON t1.idm = t2.ido
WHERE yo1998 = 'yes'
AND ucode <> 6
AND NOT EXISTS
(SELECT 1
FROM t1 sub
WHERE (great = 'yes'
OR good = 'yes'
OR yo2005 = 'yes')
AND sub.dupid = t1.dupid)
Hopefully these will be useful to you!

Answer 2

I am not sure how you are going to use this, but it may help. I believe it gives you the two tables in your last paragraph (combined in one); the "priority" is a number, corresponding to the six criteria you have.
with
table_1 ( dup_id, idm, ucode, great, good, yo2005 ) as (
select 'a', 1, 6, 'yes', 'yes', 'yes' from dual union all
select 'a', 2, 1, 'no' , 'yes', 'yes' from dual union all
select 'a', 3, 1, 'no' , 'no' , 'yes' from dual union all
select 'b', 4, 1, 'yes', 'yes', 'no' from dual union all
select 'b', 5, 1, 'no' , 'no' , 'no' from dual union all
select 'c', 6, 7, 'no' , 'no' , 'yes' from dual union all
select 'c', 7, 1, 'yes', 'no' , 'no' from dual union all
select 'd', 8, 6, 'no' , 'yes', 'no' from dual union all
select 'd', 9, 1, 'yes', 'no' , 'no' from dual union all
select 'e', 10, 3, 'yes', 'no' , 'yes' from dual union all
select 'e', 11, 4, 'no' , 'yes', 'no' from dual union all
select 'f', 12, 1, 'yes', 'yes', 'yes' from dual union all
select 'f', 13, 1, 'yes', 'no' , 'yes' from dual union all
select 'g', 14, 1, 'no' , 'no' , 'yes' from dual union all
select 'g', 15, 1, 'yes', 'no' , 'no' from dual
),
table_2 ( ido, yo1998 ) as (
select 1, 'yes' from dual union all
select 2, 'no' from dual union all
select 3, 'no' from dual union all
select 4, 'no' from dual union all
select 5, 'no' from dual union all
select 6, 'no' from dual union all
select 7, 'no' from dual union all
select 8, 'yes' from dual union all
select 9, 'yes' from dual union all
select 10, 'yes' from dual union all
select 11, 'yes' from dual union all
select 12, 'yes' from dual union all
select 13, 'no' from dual union all
select 14, 'yes' from dual union all
select 15, 'no' from dual
)
select t1.dup_id, t1.idm, t1.ucode, t1.great, t1.good, t1.yo2005, t2.yo1998,
case when ucode = 6 then 1
when great = 'yes' then 2
when good = 'yes' then 3
when yo2005 = 'yes' then case when yo1998 = 'yes' then 4
else 5
end
when yo1998 = 'yes' then 6
end as priority
from table_1 t1 left outer join table_2 t2 on t1.idm = t2.ido
order by dup_id, priority
;
Output:
DUP_ID IDM UCODE GREAT GOOD YO2005 YO1998 PRIORITY
------ ---- ----- ----- ---- ------ ------ --------
a 1 6 yes yes yes yes 1
a 2 1 no yes yes no 3
a 3 1 no no yes no 5
b 4 1 yes yes no no 2
b 5 1 no no no no
c 7 1 yes no no no 2
c 6 7 no no yes no 5
d 8 6 no yes no yes 1
d 9 1 yes no no yes 2
e 10 3 yes no yes yes 2
e 11 4 no yes no yes 3
f 12 1 yes yes yes yes 2
f 13 1 yes no yes no 2
g 15 1 yes no no no 2
g 14 1 no no yes yes 4
15 rows selected
ADDED: Here is one way to use this (as a subquery) to further analyze the results. See OP's comments below. DUP_ID = a and d don't appear in the output at all, since each has a row with UCODE=6; for every other DUP_ID the row with the highest PRIORITY is selected (if there are ties, one random row for that DUP_ID, from those with highest PRIORITY, is selected).
with
table_1 ( dup_id, idm, ucode, great, good, yo2005 ) as (
....
),
table_2 ( ido, yo1998 ) as (
....
),
final ( dup_id, idm, ucode, great, good, yo2005, yo1998, priority ) as (
select t1.dup_id, t1.idm, t1.ucode, t1.great, t1.good, t1.yo2005, t2.yo1998,
case when ucode = 6 then 1
when great = 'yes' then 2
when good = 'yes' then 3
when yo2005 = 'yes' then case when yo1998 = 'yes' then 4
else 5
end
when yo1998 = 'yes' then 6
end as priority
from table_1 t1 left outer join table_2 t2 on t1.idm = t2.ido
),
o ( dup_id, idm, ucode, great, good, yo2005, yo1998, priority, rn ) as (
select dup_id, idm, ucode, great, good, yo2005, yo1998, priority,
row_number() over (partition by dup_id order by priority)
from final
)
select dup_id, idm, ucode, great, good, yo2005, yo1998, priority
from o
where rn = 1 and priority > 1;
DUP_ID IDM UCODE GREAT GOOD YO2005 YO1998 PRIORITY
------ --- ----- ----- ----- ------ ------ --------
b 4 1 yes yes no no 2
c 7 1 yes no no no 2
e 10 3 yes no yes yes 2
f 12 1 yes yes yes yes 2
g 15 1 yes no no no 2