nested for loop in oracle to find similarity optimize - sql

Question

I have two tables both has same value but bot are from different source.
Table 1
------------
ID Title
1 Introduction to Science
2 Introduction to C
3 Let is C
4 C
5 Java
Table 2
------------------------
ID Title
a Intro to Science
b Intro to C
c Let is C
d C
e Java
I want to compare all the title in table 1 with title in table 2 and find the similarity match.
I Have used the built-in function in orcale "UTL_MATCH.edit_distance_similarity (LS_Title, LSO_Title);"
Script:
DECLARE
LS_count NUMBER;
LSO_count NUMBER;
percentage NUMBER;
LS_Title VARCHAR2 (4000);
LSO_Title VARCHAR2 (4000);
LS_CPNT_ID VARCHAR2 (64);
LSO_CPNT_ID VARCHAR2 (64);
BEGIN
SELECT COUNT (*) INTO LS_count FROM tbl_zim_item;
SELECT COUNT (*) INTO LSO_count FROM tbl_zim_lso_item;
DBMS_OUTPUT.put_line ('value of a: ' || LS_count);
DBMS_OUTPUT.put_line ('value of a: ' || LSO_count);
FOR i IN 1 .. LS_count
LOOP
SELECT cpnt_title
INTO LS_Title
FROM tbl_zim_item
WHERE iden = i;
SELECT cpnt_id
INTO LS_CPNT_ID
FROM tbl_zim_item
WHERE iden = i;
FOR j IN 1 .. lso_count
LOOP
SELECT cpnt_title
INTO LSO_Title
FROM tbl_zim_lso_item
WHERE iden = j;
SELECT cpnt_id
INTO LSO_CPNT_ID
FROM tbl_zim_lso_item
WHERE iden = j;
percentage :=
UTL_MATCH.edit_distance_similarity (LS_Title, LSO_Title);
IF percentage > 50
THEN
INSERT INTO title_sim
VALUES (ls_cpnt_id,
ls_title,
lso_cpnt_id,
lso_title,
percentage);
END IF;
END LOOP;
END LOOP;
END;
This is running for more than 15 hours. Kindly provide a better solution.
Note : My table 1 has 20000 records and table 2 has 10000 records.

Answer 1

Unless I'm missing something, you don't need all of the looping and row-by-row lookups since SQL can do cross joins. Therefore my first try would be just:
insert into title_sim
( columns... )
select ls_cpnt_id
, ls_title
, lso_cpnt_id
, lso_title
, percentage
from ( select i.cpnt_id as ls_cpnt_id
, i.cpnt_title as ls_title
, li.cpnt_id as lso_cpnt_id
, li.cpnt_title as lso_title
, case -- Using Boneist's suggestion:
when i.cpnt_title = li.cpnt_title then 100
else utl_match.edit_distance_similarity(i.cpnt_title, li.cpnt_title)
end as percentage
from tbl_zim_item i
cross join tbl_zim_lso_item li )
where percentage > 50;
If there is much repetition in the titles, you might benefit from some scalar subquery caching by wrapping the utl_match.edit_distance_similarity function in a ( select ... from dual ).
If the titles are often exactly the same and assuming in those cases percentage should be 100%, you might avoid calling the function when the titles are an exact match:
begin
select count(*) into ls_count from tbl_zim_item;
select count(*) into lso_count from tbl_zim_lso_item;
dbms_output.put_line('tbl_zim_item contains ' || ls_count || ' rows.');
dbms_output.put_line('tbl_zim_lso_item contains ' || lso_count || ' rows.');
for r in (
select i.cpnt_id as ls_cpnt_id
, i.cpnt_title as ls_title
, li.cpnt_id as lso_cpnt_id
, li.cpnt_title as lso_title
, case
when i.cpnt_title = li.cpnt_title then 100 else 0
end as percentage
from tbl_zim_item i
cross join tbl_zim_lso_item li
)
loop
if r.percentage < 100 then
r.percentage := utl_match.edit_distance_similarity(r.ls_title, r.lso_title);
end if;
if r.percentage > 50 then
insert into title_sim (columns...)
values
( ls_cpnt_id
, ls_title
, lso_cpnt_id
, lso_title
, percentage );
end if;
end loop;
end;

Answer 2

Rather than looping through all the data, I'd merely join the two tables together, eg:
WITH t1 AS (SELECT 1 ID, 'Introduction to Science' title FROM dual UNION ALL
SELECT 2 ID, 'Introduction to C' title FROM dual UNION ALL
SELECT 3 ID, 'Let is C' title FROM dual UNION ALL
SELECT 4 ID, 'C' title FROM dual UNION ALL
SELECT 5 ID, 'Java' title FROM dual UNION ALL
SELECT 6 ID, 'Oracle for Newbies' title FROM dual),
t2 AS (SELECT 'a' ID, 'Intro to Science' title FROM dual UNION ALL
SELECT 'b' ID, 'Intro to C' title FROM dual UNION ALL
SELECT 'c' ID, 'Let is C' title FROM dual UNION ALL
SELECT 'd' ID, 'C' title FROM dual UNION ALL
SELECT 'e' ID, 'Java' title FROM dual UNION ALL
SELECT 'f' ID, 'PL/SQL rocks!' title FROM dual)
SELECT t1.title t1_title,
t2.title t2_title,
UTL_MATCH.edit_distance_similarity(t1.title, t2.title)
FROM t1
INNER JOIN t2 ON UTL_MATCH.edit_distance_similarity(t1.title, t2.title) > 50;
T1_TITLE T2_TITLE UTL_MATCH.EDIT_DISTANCE_SIMILA
----------------------- ---------------- ------------------------------
Introduction to Science Intro to Science 70
Introduction to C Intro to C 59
Let is C Let is C 100
C C 100
Java Java 100
By doing that, you can then reduce the whole thing to a single DML statement, something like:
INSERT INTO title_sim (t1_id,
t1_title,
t2_id,
t2_title,
percentage)
SELECT t1.id t1_id,
t1.title t1_title,
t2.id t2_id,
t2.title t2_title,
UTL_MATCH.edit_distance_similarity(t1.title, t2.title) percentage
FROM t1
INNER JOIN t2 ON UTL_MATCH.edit_distance_similarity(t1.title, t2.title) > 50;
which ought to be a good deal faster than your row-by-row attempt, particularly as you are unnecessarily selecting from each table twice.
As an aside, you know that you can select multiple columns into multiple variables in the same query, right?
So instead of having:
SELECT cpnt_title
INTO LS_Title
FROM tbl_zim_item
WHERE iden = i;
SELECT cpnt_id
INTO LS_CPNT_ID
FROM tbl_zim_item
WHERE iden = i;
you could instead do:
SELECT cpnt_title, cpnt_id
INTO LS_Title, LS_CPNT_ID
FROM tbl_zim_item
WHERE iden = i;

Answer 3

https://www.techonthenet.com/oracle/intersect.php
this will give you data which is similar in both queries
select title from table_1
intersect
select title from table_2