Performance improvement of plsql using Bulk collect - sql

Question

I am using bulk collect to improve execution time. When I do not use bulk collect, it executes in 4 mins.
But when I use bulk collect there is no output, neither the error message is shown in console. I can see a blank spool file created.
Please let me know if I have utilized bulk collect incorrectly, also can we use this clause in select statement with limit?
Table consists of maximum 1 million records.
SET SERVEROUTPUT ON FORMAT WRAPPED
SET VERIFY OFF
SET FEEDBACK OFF
SET TERMOUT OFF
SPOOL C:\Temp\spool_1.txt
DECLARE
cursor c2 is (
select count(distinct e.cdb_pref_event_id)
,e.supp_cd
from (select distinct eh.cdb_customer_id cdb_customer_id
,eh.cdb_pref_event_id cdb_pref_event_id
,eh.supp_cd supp_cd
from (select *
from cdb_stg.cpm_pref_event_stg_arc
where trunc(load_date) = trunc(sysdate - 1)) eh
Left outer join cdb_admin.cpm_pref_result er on (eh.cdb_customer_id =
er.cdb_customer_id and
eh.cdb_pref_event_id =
er.cdb_pref_event_id)
where er.cdb_pref_event_id is null
and er.cdb_customer_id is null) r
join cdb_admin.cpm_pref_event_exception e on (r.cdb_customer_id =
e.cdb_customer_id and
r.cdb_pref_event_id =
e.cdb_pref_event_id)
group by e.supp_cd);
TYPE totalprefresults is table of NUMBER(20);
TYPE supcd_1 is table of cdb_admin.cpm_pref_event_stg.supp_cd%TYPE;
total_prefresults totalprefresults;
supcd1 supcd_1;
--Total_prefresults NUMBER(20);
--SUPCD1 CDB_ADMIN.CPM_PREF_EVENT_STG.supp_cd%TYPE;
profile_counts NUMBER(20);
iter Integer := 0;
BEGIN
select count(distinct cdb_customer_id)
into profile_counts
from cdb_admin.cpm_pref_event_exception h
where cdb_customer_id in
(Select distinct e.cdb_customer_id
from (Select distinct eh.cdb_customer_id cdb_customer_id
,eh.cdb_pref_event_id cdb_pref_event_id
,eh.supp_cd supp_cd
from (select *
from cdb_stg.cpm_pref_event_stg_arc
where trunc(load_date) = trunc(sysdate - 1)) eh
Left outer join cdb_admin.cpm_pref_result er on (eh.cdb_customer_id =
er.cdb_customer_id and
eh.cdb_pref_event_id =
er.cdb_pref_event_id)
where er.cdb_pref_event_id is null
and er.cdb_customer_id is null) r
join cdb_admin.cpm_pref_event_exception e on (r.cdb_customer_id =
e.cdb_customer_id and
r.cdb_pref_event_id =
e.cdb_pref_event_id)
where e.supp_cd = 'PROFILE-NOT-FOUND')
and h.supp_cd != 'PROFILE-NOT-FOUND';
dbms_output.put_line('TOTAL EVENTS VALIDATION');
dbms_output.put_line('-------------------------------------------------------------');
dbms_output.put_line('');
dbms_output.put_line(rpad('Pref_Counts', 25) || rpad('Supp_CD', 25));
OPEN c2;
LOOP
FETCH c2 BULK COLLECT
INTO total_prefresults
,supcd1 limit 100;
EXIT WHEN c2%NOTFOUND;
dbms_output.put_line(rpad(total_prefresults, 25) || rpad(supcd1, 25));
IF (supcd1 = 'PROFILE-NOT-FOUND')
then
dbms_output.put_line('');
dbms_output.put_line('Profile not found records count : ' ||
total_prefresults);
dbms_output.put_line(profile_counts ||
' : counts moved to other exceptions ');
dbms_output.put_line((total_prefresults - profile_counts) ||
' : are still in Profile_not_found exception');
END IF;
iter := iter + 1;
END LOOP;
CLOSE c2;
dbms_output.put_line('');
dbms_output.put_line('Number of missing Records: ' || iter);
END;
/
SPOOL OFF

Answer 1

I think the bottleneck is this condition: where trunc(load_date) = trunc(sysdate - 1)
Do you haven an index on trunc(load_date)? Either create a function-based index on trunc(load_date) or if you already have an index on load_date then try
WHERE load_date >= trunc(sysdate - 1) AND load_date < trunc(sysdate)
Also check your queries whether distinct is really needed. Remove them, if possible.

Answer 2

I have reframed your code starting from OPEN c2; to CLOSE c2;
BULK COLLECT should be executed to store all the data in the collection only once(in one go) and then this collection can be used using the index(i.e. I in the following case) in FOR loop as follows:
OPEN C2;
FETCH C2 BULK COLLECT INTO
TOTAL_PREFRESULTS,
SUPCD1;
--EXIT WHEN C2%NOTFOUND;
CLOSE C2;
-- To list down all the values before processing the logic
FOR I IN TOTAL_PREFRESULTS.FIRST..TOTAL_PREFRESULTS.LAST LOOP
DBMS_OUTPUT.PUT_LINE(RPAD(TOTAL_PREFRESULTS(I), 25)
|| RPAD(SUPCD1(I), 25));
END LOOP;
FOR I IN TOTAL_PREFRESULTS.FIRST..TOTAL_PREFRESULTS.LAST LOOP
IF ( SUPCD1(I) = 'PROFILE-NOT-FOUND' ) THEN
DBMS_OUTPUT.PUT_LINE('');
DBMS_OUTPUT.PUT_LINE('Profile not found records count : ' || TOTAL_PREFRESULTS(I));
DBMS_OUTPUT.PUT_LINE(PROFILE_COUNTS || ' : counts moved to other exceptions ');
DBMS_OUTPUT.PUT_LINE((TOTAL_PREFRESULTS(I) - PROFILE_COUNTS)
|| ' : are still in Profile_not_found exception');
END IF;
ITER := ITER + 1;
END LOOP;
replace above code snippet in your code and try to execute.
Refer the guide to use BULK COLLECT
Cheers!!

Answer 3

Bulk collect can provide considerable performance gain. However there are a couple gotchas involved.
First off there is the difference in the meaning of %notfound.
On a standard cursor %notfound means all rows have already been fetched and there is no more. With bulk collect this changes to 'there were insufficient rows to reach the LIMIT specified (if present).
That does NOT mean there no rows fetched just the limit specified was not reached. For example if your limit is 100 and the fetch retrieved only 50 then %notfound would return True. This is where the referenced guide fails.
The second is what happens without the limit clause: All rows from the cursor are returned into shared memory(PGA?). So what's the problem with that.
If there 100 rows or 1000 then most likely you're k, but suppose there are 100,000 or 1M rows, they are still all loaded into memory. Finally (at least for now) when the limit clause is used then the entire Fetch+Process must itself be enclosed within a loop or you process only the first fetch - meaning only the specified limit number of rows - no matter how many actually exist. Another point where the referenced guide fails.
The following skeleton accommodates the above.
declare
max_bulk_rows constant integer := 1000; -- define the max number of rows for each fetch ...
cursor c_bulk is(
Select ... ;
type bulk_row_t is table of c_bulk%rowtype;
bulk_row bulk_row_t;
Begin
open c_bulk;
loop
fetch c_bulk -- fill buffer
bulk collect into bulk_row
limit max_bulk_row;
for i in bulk_row.first .. bulk_row.last -- process each row in buffer
loop
"process individual row here"
end loop;
foreach ... -- bulk output of rows here is needed.
exit when bulk_row.count < max_bulk_row; -- exit process loop if all rows processed
end loop ; -- loop back and fetch next buffer if needed
close c_bulk;
...
end;