Fastest way to find distinct matching records - sql

Question

I have two tables A and B. Both have same structure. We find matching records between these two. Here are the scripts
CREATE TABLE HRS.A
(
F_1 NUMBER(5,0),
F_2 NUMBER(5,0),
F_3 NUMBER(5,0)
);
CREATE TABLE HRS.B
(
F_1 NUMBER(5,0),
F_2 NUMBER(5,0),
F_3 NUMBER(5,0)
);
INSERT INTO hrs.a VALUES (1,1000,2000);
INSERT INTO hrs.a VALUES (2,1100,8000);
INSERT INTO hrs.a VALUES (3,4000,3000);
INSERT INTO hrs.a VALUES (4,2000,5000);
INSERT INTO hrs.a VALUES (5,5000,3000);
INSERT INTO hrs.a VALUES (6,6000,6000);
INSERT INTO hrs.a VALUES (7,3000,7000);
INSERT INTO hrs.a VALUES (8,1100,9000);
INSERT INTO hrs.b VALUES (1,4000,2000);
INSERT INTO hrs.b VALUES (2,6000,8000);
INSERT INTO hrs.b VALUES (3,1000,3000);
INSERT INTO hrs.b VALUES (4,2000,5000);
INSERT INTO hrs.b VALUES (5,8000,3000);
INSERT INTO hrs.b VALUES (6,1100,6000);
INSERT INTO hrs.b VALUES (7,5000,7000);
INSERT INTO hrs.b VALUES (8,1000,9000);
To find matching records
SELECT a.F_1 A_F1, b.F_1 B_F1 FROM HRS.A, HRS.B WHERE A.F_2 = B.F_2
results
A_F1 B_F1
3 1
6 2
1 3
4 4
8 6
2 6
5 7
1 8
Now i want to remove duplicate entries in both columns separately e.g. 1 is repeating in A_F1 (regardless of B_F1) so row # 3(1-3) and 8(1-8) will be removed. Now 6 is repeating in B_F1 (regardless of A_F1) so row # 5(8-6) and 6(2-6) will be removed. Final result should be
A_F1 B_F1
3 1
6 2
4 4
5 7
Now most important part, These two tables contain 500,000 records each. I was first finding and inserting these matching records into a temp table, then removing duplicate from first column then from second column and then selecting all from temp table. This is too too slow. How can i achieve this as faster as possible?
Edit # 1
I executed following statements multiple times to generate 4096 records in each table
INSERT INTO hrs.a SELECT F_1 + 1, F_2 + 1, 0 FROM hrs.a;
INSERT INTO hrs.b SELECT F_1 + 1, F_2 + 1, 0 FROM hrs.b;
Now i executed all answers and found these
Rachcha 9.11 secs OK
techdo 1.14 secs OK
Gentlezerg 577 msecs WRONG RESULTS
Justin 218 msecs OK
Even #Justin took 37.69 secs for 65,536 records in each (total = 131,072)
Waiting for more optimized answers as actual number of records are 1,000,000 :)
Here is the execution plan of the query based on Justin's answer

Answer 1

Please try:
select A_F1, B_F1 From(
SELECT a.F_1 A_F1, b.F_1 B_F1,
count(*) over (partition by a.F_1 order by a.F_1) C1,
count(*) over (partition by b.F_1 order by b.F_1) C2
FROM HRS.A A, HRS.B B WHERE A.F_2 = B.F_2
)x
where C1=1 and C2=1;
How about an INNER JOIN instead? Please check with this query.
select A_F1, B_F1 From(
SELECT a.F_1 A_F1, b.F_1 B_F1,
count(*) over (partition by a.F_1 order by a.F_1) C1,
count(*) over (partition by b.F_1 order by b.F_1) C2
FROM HRS.A A INNER JOIN HRS.B B ON A.F_2 = B.F_2
)x
where C1=1 and C2=1;

Answer 2

Query:
SQLFIDDLEExample
SELECT a.f_1 AS a_f_1,
b.f_1 AS b_f_1
FROM a JOIN b ON a.f_2 = b.f_2
WHERE 1 = (SELECT COUNT(*)
FROM a aa JOIN b bb ON aa.f_2 = bb.f_2
WHERE aa.f_1 = a.f_1 )
AND 1 = (SELECT COUNT(*)
FROM a aa JOIN b bb ON aa.f_2 = bb.f_2
WHERE bb.f_1 = b.f_1 )
Result:
| A_F_1 | B_F_1 |
-----------------
| 3 | 1 |
| 6 | 2 |
| 4 | 4 |
| 5 | 7 |

Answer 3

According to #techdo 's answer, I think this can be better:
select A_F1, B_F1 From(
SELECT a.F_1 A_F1, b.F_1 B_F1,a.F_2,
count(*) OVER(PARTITION BY A.F_2) C
FROM HRS.A A, HRS.B B WHERE A.F_2 = B.F_2
)x
where C=1 ;
The existence of multi rows is due to the same f_2. This SQL has only one count..over,so you said you have vast data, I think this would be a little faster.

Answer 4

I have the answer.
See this fiddle here.
I used the following code:
WITH x AS (SELECT a.f_1 AS a_f_1, b.f_1 AS b_f_1
FROM a JOIN b ON a.f_2 = b.f_2)
SELECT *
FROM x x1
WHERE NOT EXISTS (SELECT 1
FROM x x2
WHERE (x2.a_f_1 = x1.a_f_1
AND x2.b_f_1 != x1.b_f_1)
OR (x2.a_f_1 != x1.a_f_1
AND x2.b_f_1 = x1.b_f_1)
)
;
EDIT
I used to following code that runs within 14 ms on SQL fiddle. I removed the common table expression and observed that the query performance improved.
SELECT a1.f_1 AS a_f1, b1.f_1 AS b_f1
FROM a a1 JOIN b b1 ON a1.f_2 = b1.f_2
WHERE NOT EXISTS (SELECT 1
FROM a a2 JOIN b b2 ON a2.f_2 = b2.f_2
WHERE (a2.f_1 = a1.f_1
AND b2.f_1 != b1.f_1)
OR (a2.f_1 != a1.f_1
AND b2.f_1 = b1.f_1))
;
Output:
A_F_1 B_F_1
3 1
6 2
4 4
5 7

Answer 5

Each one of these solutions are taking time, the best one (Justin) took almost 45 mins without even returning for 2 million records. I ended up with inserting matching records in a temp table and then removing duplicates and i found it much faster than these solutions with this data set.