Oracle SQL - Efficient Join N Rows Per Match - sql

Question

The basic idea is to join two tables, let's call them MYTABLE1 and MYTABLE2 on a field JOINID. There will be lots of matches per JOINID (one row from MYTABLE1 corresponding to many rows in MYTABLE2, and for the purposes of testing, MYTABLE1 has 50 rows), but we want to only select up to N matches per JOINID value. I have seen lots of inefficient solutions, for example:
select t1.*, t2.*
from MYTABLE1 t1 inner join
(select MYTABLE2.*,
row_number() over (partition by MYTABLE2.JOINKEY order by 1) as seqnum
from MYTABLE2) t2
on t1.joinkey = t2.joinkey and seqnum <= 2;
which takes over 5 minutes for me to run and returns less than 100 results, whereas something like
select t1.*, t2.*
from MYTABLE1 t1 inner join MYTABLE2 t2 on t1.JOINKEY = t2.JOINKEY
where rownum <= 100;
returns 100 results in ~60 milliseconds.
(To be sure of the validity of these results, I selected a different test table and performed the second query above on a specific single JOINKEY until I got a result set with less than 100 results, meaning it did in fact search through all of MYTABLE2. The total query time was 30 milliseconds. Afterwards, I started the original query, but this time getting 50 joins per row of MYTABLE1, which again took over 5 minutes to complete.)
How can I approach this in a not-so-terribly-inefficient manner?
It seems so simple, all we need to do is go through the rows of MYTABLE1 and matching the JOINKEY field to that of rows of MYTABLE2, moving on to the next row of MYTABLE1 once we have matched the desired number for that row.
In the worst case scenario for my second example, we should have to spend 30ms searching through the full TABLE2 per row of TABLE1, of which there are 50 rows, for a total execution time of 1.5 seconds.

Answer 1

I wouldn't call the below approach efficient by any means and it cheats a little and has some clunkiness, but it comes in under the 1500ms limit you provided so I'll add as something to consider.
This example cheats in that it compiles a TYPE, so it can table an anonymous function.
This approach just iteratively probes MYTABLE2 with each JOINKEY from MYTABLE1 using an anonymous subquery-factoring-clause function and accumulates the results as it goes.
I don't know the real structure of the tables involved, so this example pretends MYTABLE2 has one additional CHAR attribute called OTHER_DATA that is the target of the SELECT.
First, setup the test tables:
CREATE TABLE MYTABLE1 (
JOINKEY NUMBER NOT NULL
);
CREATE TABLE MYTABLE2 (
JOINKEY NUMBER NOT NULL,
OTHER_DATA CHAR(1) NOT NULL
);
CREATE INDEX MYTABLE2_I
ON MYTABLE2 (JOINKEY);
Then add the test data. 50 rows to MYTABLE1 and 100M rows to MYTABLE2:
INSERT INTO MYTABLE1
SELECT LEVEL
FROM DUAL
CONNECT BY LEVEL < 51;
BEGIN
<<COMMIT_LOOP>>
FOR OUTER_POINTER IN 1..4000 LOOP
<<DATA_LOOP>>
FOR POINTER IN 1..10 LOOP
INSERT INTO MYTABLE2
SELECT
JOINKEY, OTHER_DATA
FROM
(SELECT LEVEL AS JOINKEY FROM DUAL CONNECT BY LEVEL < 51)
CROSS JOIN
(SELECT CHR(64 + LEVEL) AS OTHER_DATA FROM DUAL CONNECT BY LEVEL < 51);
END LOOP DATA_LOOP;
COMMIT;
END LOOP COMMIT_LOOP;
END;
/
Then gather stats...
Verify the table counts:
SELECT COUNT(*) FROM MYTABLE1;
50
SELECT COUNT(*) FROM MYTABLE2;
100000000
Then create a TYPE that includes the desired data:
CREATE OR REPLACE TYPE JOINKEY_OTHER_DATA IS OBJECT (JOINKEY1 NUMBER, OTHER_DATA CHAR(1));
/
CREATE OR REPLACE TYPE JOINKEY_OTHER_DATA_LIST IS TABLE OF JOINKEY_OTHER_DATA;
/
And then run a query that uses an anonymous subquery-factoring-block function that imposes a rowcount per JOINKEY to be returned. In this first example, it fetches two MYTABLE2 rows per JOINKEY:
SELECT SYSTIMESTAMP FROM DUAL;
WITH FUNCTION FETCH_N_ROWS
(P_MATCHES_LIMIT IN NUMBER)
RETURN JOINKEY_OTHER_DATA_LIST
AS
V_JOINKEY_OTHER_DATAS JOINKEY_OTHER_DATA_LIST;
BEGIN
V_JOINKEY_OTHER_DATAS := JOINKEY_OTHER_DATA_LIST();
FOR JOINKEY_POINTER IN (SELECT MYTABLE1.JOINKEY
FROM MYTABLE1)
LOOP
DECLARE
V_MYTABLE2_JOINKEYS JOINKEY_OTHER_DATA_LIST;
BEGIN
SELECT JOINKEY_OTHER_DATA(MYTABLE2.JOINKEY, MYTABLE2.OTHER_DATA)
BULK COLLECT INTO V_MYTABLE2_JOINKEYS
FROM MYTABLE2 WHERE MYTABLE2.JOINKEY = JOINKEY_POINTER.JOINKEY
FETCH FIRST P_MATCHES_LIMIT ROWS ONLY;
V_JOINKEY_OTHER_DATAS := V_JOINKEY_OTHER_DATAS MULTISET UNION ALL V_MYTABLE2_JOINKEYS;
END;
END LOOP;
RETURN V_JOINKEY_OTHER_DATAS;
END;
SELECT *
FROM TABLE (FETCH_N_ROWS(2));
/
SELECT SYSTIMESTAMP FROM DUAL;
Results in:
SYSTIMESTAMP
18-APR-17 03.32.10.623056000 PM -06:00
JOINKEY1 OTHER_DATA
1 A
1 B
2 A
2 B
3 A
3 B
...
49 A
49 B
50 A
50 B
100 rows selected.
SYSTIMESTAMP
18-APR-17 03.32.11.014554000 PM -06:00
By changing the number passed to FETCH_N_ROWS, different data volumens can be fetched with fairly consistent performance.
...
SELECT * FROM TABLE (FETCH_N_ROWS(13));
Returns:
...
50 K
50 L
50 M
650 rows selected.

Answer 2

You cannot compare the two queries. The second query is simply returning whatever rows come first. The second has to go through all the data to return anything. A more apt comparison would use:
select . . .
from (select t1.*, t2.*
from MYTABLE1 t1 inner join
MYTABLE2 t2
on t1.JOINKEY = t2.JOINKEY
order by t1.JOINKEY
) t1
where rownum <= 100;
This has to read all the data before returning anything so it is more analogous to using row number.
But, start with this query:
select t1.*, t2.*
from MYTABLE1 t1 inner join
(select MYTABLE2.*,
row_number() over (partition by t2.JOINKEY order by 1) as seqnum
from MYTABLE2 t2
) t2
on t1.joinkey = t2.joinkey and seqnum <= 2;
For this query, you want an index on MYTABLE2(JOINKEY). If the ORDER BY has another key, that should be in the query as well.