Performance Tuning Oracle SQL involving huge tables - sql

Question

I have an Oracle SQL query that involves joins with 4 large tables along with few small tables.
Large tables are TBL_1, TBL_2, TBL_3, TBL4 each having approx 8M records. Rest are small tables having less than 10K records.
The problem: Query takes more than 3 minutes even if there is no data to return.
The table and index stats are up-to-date. There are no stale stats on these tables.
I've tried using hints but it didn't work.
Please see my observations below:
Query:
SELECT a.*, ROWNUM AS rnm
FROM ( SELECT c.idntfr,
pr.program_name AS "Program",
e.case_number,
(SELECT DECODE (s.status_name,
'EA', 'A',
'ED', 'D',
'EU', 'U',
s.status_name)
FROM TBL_5 ms, status s
WHERE ms.status_type_cid = 7
AND mbr_sid = c.mbr_sid
AND ms.status_type_cid = s.status_type_cid
AND s.status_cid = ms.status_cid
AND ms.oprtnl_flag = 'A'
AND SYSDATE BETWEEN ms.from_date AND ms.TO_DATE),
DECODE (
LENGTH (TRIM (e.social_security_nmbr)),
NULL, 'Not Available',
( SUBSTR (e.social_security_nmbr, 1, 3)
|| '-'
|| SUBSTR (e.social_security_nmbr, 4, 2)
|| '-'
|| SUBSTR (e.social_security_nmbr, 6, 4)))
AS "SSN",
e.last_name || ',' || e.first_name || ' ' || e.middle_name,
TO_CHAR (e.injury_date, 'MM/dd/yyyy'),
DECODE (e.gender_lkpcd,
'M', 'Male',
'F', 'Female',
'U', 'Unknown'),
e.mbr_sid,
pr.program_cid,
e.last_name,
e.social_security_nmbr,
e.first_name AS
FROM TBL_1 c,
program pr,
TBL_2 e,
TBL_3 mai,
TBL_4 uaxou
WHERE c.mbr_sid = e.mbr_sid
AND c.mbr_sid = mai.mbr_sid
AND c.oprtnl_flag = 'A'
AND c.idntfr_type_cid = 423
AND TRUNC (SYSDATE) BETWEEN c.from_date AND c.TO_DATE
AND TRUNC (SYSDATE) BETWEEN e.from_date AND e.TO_DATE
AND e.oprtnl_flag = 'A'
AND e.status_cid = 2
AND mai.oprtnl_flag = 'A'
AND mai.status_cid = 2
AND TRUNC (SYSDATE) BETWEEN mai.from_date AND mai.TO_DATE
AND e.program_code = pr.program_code
AND pr.oprtnl_flag = 'A'
AND uaxou.user_acct_sid = 1
AND uaxou.oprtnl_flag = 'A'
AND SYSDATE BETWEEN uaxou.from_date AND uaxou.TO_DATE
AND uaxou.org_unit_sid = mai.org_unit_sid
ORDER BY "Program" ASC) a
WHERE ROWNUM < 102;
There is no data for the following condition
AND uaxou.user_acct_sid = 1
Expected Result: Response time should be under 4 seconds if NO data is returned.
Explain Plan:
Plan hash value: 2272581586
---------------------------------------------------------------------------------------------------------------------
| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time |
---------------------------------------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 101 | 22220 | 1361 (1)| 00:00:01 |
| 1 | NESTED LOOPS | | 1 | 58 | 7 (0)| 00:00:01 |
| 2 | NESTED LOOPS | | 1 | 58 | 7 (0)| 00:00:01 |
|* 3 | TABLE ACCESS BY INDEX ROWID BATCHED| TBL_5 | 1 | 31 | 6 (0)| 00:00:01 |
|* 4 | INDEX RANGE SCAN | XIF1TBL_5 | 8 | | 3 (0)| 00:00:01 |
|* 5 | INDEX UNIQUE SCAN | XPKSTATUS | 1 | | 0 (0)| 00:00:01 |
| 6 | TABLE ACCESS BY INDEX ROWID | STATUS | 1 | 27 | 1 (0)| 00:00:01 |
|* 7 | COUNT STOPKEY | | | | | |
| 8 | VIEW | | 169 | 37180 | 1361 (1)| 00:00:01 |
| 9 | NESTED LOOPS | | 169 | 36166 | 767 (0)| 00:00:01 |
| 10 | NESTED LOOPS | | 11904 | 36166 | 767 (0)| 00:00:01 |
| 11 | NESTED LOOPS | | 62 | 11284 | 333 (0)| 00:00:01 |
| 12 | NESTED LOOPS | | 45 | 6660 | 108 (0)| 00:00:01 |
| 13 | NESTED LOOPS | | 33 | 3564 | 9 (0)| 00:00:01 |
|* 14 | TABLE ACCESS BY INDEX ROWID | PROGRAM | 5 | 70 | 2 (0)| 00:00:01 |
| 15 | INDEX FULL SCAN | XAK1OWCP_PROGRAM | 2 | | 1 (0)| 00:00:01 |
|* 16 | TABLE ACCESS FULL | TBL_2 | 20 | 1880 | 4 (0)| 00:00:01 |
|* 17 | TABLE ACCESS BY INDEX ROWID | TBL_1 | 1 | 40 | 3 (0)| 00:00:01 |
|* 18 | INDEX RANGE SCAN | TUNE_WS_19NOV10_X2 | 1 | | 2 (0)| 00:00:01 |
|* 19 | TABLE ACCESS BY INDEX ROWID | TBL_3 | 1 | 34 | 5 (0)| 00:00:01 |
|* 20 | INDEX RANGE SCAN | XIE2_TBL_3 | 3 | | 2 (0)| 00:00:01 |
|* 21 | INDEX RANGE SCAN | XIF3TBL_4 | 192 | | 1 (0)| 00:00:01 |
|* 22 | TABLE ACCESS BY INDEX ROWID | TBL_4 | 3 | 96 | 7 (0)| 00:00:01 |
---------------------------------------------------------------------------------------------------------------------
Predicate Information (identified by operation id):
---------------------------------------------------
3 - filter("MS"."STATUS_TYPE_CID"=7 AND "MS"."OPRTNL_FLAG"='A' AND "MS"."TO_DATE">=SYSDATE#! AND
"MS"."FROM_DATE"<=SYSDATE#!)
4 - access("MBR_SID"=:B1)
5 - access("S"."STATUS_TYPE_CID"=7 AND "S"."STATUS_CID"="MS"."STATUS_CID")
7 - filter(ROWNUM<102)
14 - filter("PR"."OPRTNL_FLAG"='A')
16 - filter("E"."PROGRAM_CODE"="PR"."PROGRAM_CODE" AND "E"."OPRTNL_FLAG"='A' AND "E"."STATUS_CID"=2 AND
"E"."FROM_DATE"<=TRUNC(SYSDATE#!) AND TRUNC(INTERNAL_FUNCTION("FROM_DATE"))<=TRUNC(TRUNC(SYSDATE#!)) AND
"E"."TO_DATE">=TRUNC(SYSDATE#!) AND TRUNC(INTERNAL_FUNCTION("TO_DATE"))>=TRUNC(TRUNC(SYSDATE#!)))
17 - filter("C"."FROM_DATE"<=TRUNC(SYSDATE#!) AND "C"."TO_DATE">=TRUNC(SYSDATE#!))
18 - access("C"."MBR_SID"="E"."MBR_SID" AND "C"."IDNTFR_TYPE_CID"=423 AND "C"."OPRTNL_FLAG"='A')
19 - filter("MAI"."OPRTNL_FLAG"='A' AND "MAI"."STATUS_CID"=2 AND "MAI"."FROM_DATE"<=TRUNC(SYSDATE#!) AND
"MAI"."TO_DATE">=TRUNC(SYSDATE#!))
20 - access("C"."MBR_SID"="MAI"."MBR_SID")
21 - access("UAXOU"."USER_ACCT_SID"=1)
22 - filter("UAXOU"."ORG_UNIT_SID"="MAI"."ORG_UNIT_SID" AND "UAXOU"."OPRTNL_FLAG"='A' AND
"UAXOU"."FROM_DATE"<=SYSDATE#! AND "UAXOU"."TO_DATE">=SYSDATE#!)
This is output of query from v$parameter
NAME | VALUE
compatible | 12.2.0
optimizer_adaptive_plans | TRUE
optimizer_adaptive_reporting_only | FALSE
optimizer_features_enable | 12.2.0.1
This is the explain plan after adding GATHER_PLAN_STATISTICS showing the actual cardinality values:
Plan hash value: 2272581586
-------------------------------------------------------------------------------------------------------------------------------
| Id | Operation | Name | Starts | E-Rows | A-Rows | A-Time | Buffers |
-------------------------------------------------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 1 | | 0 |00:00:00.01 | 0 |
| 1 | NESTED LOOPS | | 0 | 1 | 0 |00:00:00.01 | 0 |
| 2 | NESTED LOOPS | | 0 | 1 | 0 |00:00:00.01 | 0 |
|* 3 | TABLE ACCESS BY INDEX ROWID BATCHED| TBL_5 | 0 | 1 | 0 |00:00:00.01 | 0 |
|* 4 | INDEX RANGE SCAN | XIF1TBL_5 | 0 | 8 | 0 |00:00:00.01 | 0 |
|* 5 | INDEX UNIQUE SCAN | XPKSTATUS | 0 | 1 | 0 |00:00:00.01 | 0 |
| 6 | TABLE ACCESS BY INDEX ROWID | STATUS | 0 | 1 | 0 |00:00:00.01 | 0 |
|* 7 | COUNT STOPKEY | | 1 | | 0 |00:00:00.01 | 0 |
| 8 | VIEW | | 1 | 169 | 0 |00:00:00.01 | 0 |
| 9 | NESTED LOOPS | | 1 | 169 | 0 |00:00:00.01 | 0 |
| 10 | NESTED LOOPS | | 1 | 11904 | 0 |00:00:00.01 | 0 |
| 11 | NESTED LOOPS | | 1 | 62 | 0 |00:00:00.01 | 0 |
| 12 | NESTED LOOPS | | 1 | 45 | 0 |00:00:00.01 | 0 |
| 13 | NESTED LOOPS | | 1 | 33 | 0 |00:00:00.01 | 0 |
|* 14 | TABLE ACCESS BY INDEX ROWID | PROGRAM | 1 | 5 | 1 |00:00:00.01 | 2 |
| 15 | INDEX FULL SCAN | XAK1OWCP_PROGRAM | 1 | 2 | 2 |00:00:00.01 | 1 |
|* 16 | TABLE ACCESS FULL | TBL_2 | 1 | 20 | 0 |00:00:00.01 | 0 |
|* 17 | TABLE ACCESS BY INDEX ROWID | TBL_1 | 0 | 1 | 0 |00:00:00.01 | 0 |
|* 18 | INDEX RANGE SCAN | TUNE_WS_19NOV10_X2 | 0 | 1 | 0 |00:00:00.01 | 0 |
|* 19 | TABLE ACCESS BY INDEX ROWID | TBL_3 | 0 | 1 | 0 |00:00:00.01 | 0 |
|* 20 | INDEX RANGE SCAN | XIE2_TBL_3 | 0 | 3 | 0 |00:00:00.01 | 0 |
|* 21 | INDEX RANGE SCAN | XIF3TBL_4 | 0 | 192 | 0 |00:00:00.01 | 0 |
|* 22 | TABLE ACCESS BY INDEX ROWID | TBL_4 | 0 | 3 | 0 |00:00:00.01 | 0 |
-------------------------------------------------------------------------------------------------------------------------------
Predicate Information (identified by operation id):
---------------------------------------------------
3 - filter(("MS"."STATUS_TYPE_CID"=7 AND "MS"."OPRTNL_FLAG"='A' AND "MS"."TO_DATE">=SYSDATE#! AND
"MS"."FROM_DATE"<=SYSDATE#!))
4 - access("MBR_SID"=:B1)
5 - access("S"."STATUS_TYPE_CID"=7 AND "S"."STATUS_CID"="MS"."STATUS_CID")
7 - filter(ROWNUM<102)
14 - filter("PR"."OPRTNL_FLAG"='A')
16 - filter(("E"."PROGRAM_CODE"="PR"."PROGRAM_CODE" AND "E"."OPRTNL_FLAG"='A' AND "E"."STATUS_CID"=2 AND
"E"."FROM_DATE"<=TRUNC(SYSDATE#!) AND "E"."TO_DATE">=TRUNC(SYSDATE#!)))
17 - filter(("C"."FROM_DATE"<=TRUNC(SYSDATE#!) AND "C"."TO_DATE">=TRUNC(SYSDATE#!)))
18 - access("C"."MBR_SID"="E"."MBR_SID" AND "C"."IDNTFR_TYPE_CID"=423 AND "C"."OPRTNL_FLAG"='A')
19 - filter(("MAI"."OPRTNL_FLAG"='A' AND "MAI"."STATUS_CID"=2 AND "MAI"."FROM_DATE"<=TRUNC(SYSDATE#!) AND
"MAI"."TO_DATE">=TRUNC(SYSDATE#!)))
20 - access("C"."MBR_SID"="MAI"."MBR_SID")
21 - access("UAXOU"."USER_ACCT_SID"=1)
22 - filter(("UAXOU"."ORG_UNIT_SID"="MAI"."ORG_UNIT_SID" AND "UAXOU"."OPRTNL_FLAG"='A' AND
"UAXOU"."FROM_DATE"<=SYSDATE#! AND "UAXOU"."TO_DATE">=SYSDATE#!))
I've tried various hints USE_HASH(c e) and various other combinations, none worked.
One interesting observation, if I comment the condition:
--AND uaxou.user_acct_sid = 1
The results came in 7 seconds. (Obviously, data is returned in this case).
So, what's causing the query to take so long when no data is returned? (i.e. this condition is not commented AND uaxou.user_acct_sid = 1)
I let the slow query to finish, it took 10 min 46 seconds. No data is returned
Here is the explain plan. I don't know why A-Time doesn't match with actual execution time.
Plan hash value: 2272581586
-------------------------------------------------------------------------------------------------------------------------------
| Id | Operation | Name | Starts | E-Rows | A-Rows | A-Time | Buffers |
-------------------------------------------------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 1 | | 0 |00:00:00.01 | 0 |
| 1 | NESTED LOOPS | | 0 | 1 | 0 |00:00:00.01 | 0 |
| 2 | NESTED LOOPS | | 0 | 1 | 0 |00:00:00.01 | 0 |
|* 3 | TABLE ACCESS BY INDEX ROWID BATCHED| TBL_5 | 0 | 1 | 0 |00:00:00.01 | 0 |
|* 4 | INDEX RANGE SCAN | XIF1TBL_5 | 0 | 8 | 0 |00:00:00.01 | 0 |
|* 5 | INDEX UNIQUE SCAN | XPKSTATUS | 0 | 1 | 0 |00:00:00.01 | 0 |
| 6 | TABLE ACCESS BY INDEX ROWID | STATUS | 0 | 1 | 0 |00:00:00.01 | 0 |
|* 7 | COUNT STOPKEY | | 1 | | 0 |00:00:00.01 | 0 |
| 8 | VIEW | | 1 | 169 | 0 |00:00:00.01 | 0 |
| 9 | NESTED LOOPS | | 1 | 169 | 0 |00:00:00.01 | 0 |
| 10 | NESTED LOOPS | | 1 | 11904 | 0 |00:00:00.01 | 0 |
| 11 | NESTED LOOPS | | 1 | 62 | 0 |00:00:00.01 | 0 |
| 12 | NESTED LOOPS | | 1 | 45 | 0 |00:00:00.01 | 0 |
| 13 | NESTED LOOPS | | 1 | 33 | 0 |00:00:00.01 | 0 |
|* 14 | TABLE ACCESS BY INDEX ROWID | PROGRAM | 1 | 5 | 1 |00:00:00.01 | 2 |
| 15 | INDEX FULL SCAN | XAK1OWCP_PROGRAM | 1 | 2 | 2 |00:00:00.01 | 1 |
|* 16 | TABLE ACCESS FULL | TBL_2 | 1 | 20 | 0 |00:00:00.01 | 0 |
|* 17 | TABLE ACCESS BY INDEX ROWID | TBL_1 | 0 | 1 | 0 |00:00:00.01 | 0 |
|* 18 | INDEX RANGE SCAN | TUNE_WS_19NOV10_X2 | 0 | 1 | 0 |00:00:00.01 | 0 |
|* 19 | TABLE ACCESS BY INDEX ROWID | TBL_3 | 0 | 1 | 0 |00:00:00.01 | 0 |
|* 20 | INDEX RANGE SCAN | XIE2_TBL_3 | 0 | 3 | 0 |00:00:00.01 | 0 |
|* 21 | INDEX RANGE SCAN | XIF3TBL_4 | 0 | 192 | 0 |00:00:00.01 | 0 |
|* 22 | TABLE ACCESS BY INDEX ROWID | TBL_4 | 0 | 3 | 0 |00:00:00.01 | 0 |
-------------------------------------------------------------------------------------------------------------------------------
Predicate Information (identified by operation id):
---------------------------------------------------
3 - filter(("MS"."STATUS_TYPE_CID"=7 AND "MS"."OPRTNL_FLAG"='A' AND "MS"."TO_DATE">=SYSDATE#! AND
"MS"."FROM_DATE"<=SYSDATE#!))
4 - access("MBR_SID"=:B1)
5 - access("S"."STATUS_TYPE_CID"=7 AND "S"."STATUS_CID"="MS"."STATUS_CID")
7 - filter(ROWNUM<102)
14 - filter("PR"."OPRTNL_FLAG"='A')
16 - filter(("E"."PROGRAM_CODE"="PR"."PROGRAM_CODE" AND "E"."OPRTNL_FLAG"='A' AND "E"."STATUS_CID"=2 AND
"E"."FROM_DATE"<=TRUNC(SYSDATE#!) AND "E"."TO_DATE">=TRUNC(SYSDATE#!)))
17 - filter(("C"."FROM_DATE"<=TRUNC(SYSDATE#!) AND "C"."TO_DATE">=TRUNC(SYSDATE#!)))
18 - access("C"."MBR_SID"="E"."MBR_SID" AND "C"."IDNTFR_TYPE_CID"=423 AND "C"."OPRTNL_FLAG"='A')
19 - filter(("MAI"."OPRTNL_FLAG"='A' AND "MAI"."STATUS_CID"=2 AND "MAI"."FROM_DATE"<=TRUNC(SYSDATE#!) AND
"MAI"."TO_DATE">=TRUNC(SYSDATE#!)))
20 - access("C"."MBR_SID"="MAI"."MBR_SID")
21 - access("UAXOU"."USER_ACCT_SID"=1)
22 - filter(("UAXOU"."ORG_UNIT_SID"="MAI"."ORG_UNIT_SID" AND "UAXOU"."OPRTNL_FLAG"='A' AND
"UAXOU"."FROM_DATE"<=SYSDATE#! AND "UAXOU"."TO_DATE">=SYSDATE#!))

Answer 1

Adaptive plans may improve the execution plan.
You tagged the question as 12c but it looks like the execution plan is not using adaptive plans for some reason. Adaptive plans let Oracle change operations at run time, such as switching between NESTED LOOPS and HASH JOINS.
NESTED LOOPS are good for a small percentage of rows, and HASH JOINS are good for a large percentage of rows. Since the ROWS estimates are all small, but the query runs for three minutes, I would guess the optimizer is significantly underestimating the cardinality of the expressions and joins, and is using too many NESTED LOOPs.
If adaptive plans were enabled the execution plan should have this at the bottom:
Note
-----
- this is an adaptive plan
Since that Note is missing, I would guess there is a parameter issue on your database preventing adaptive plans. Run the below query and see if one of the features is turned off, or if the features are set to a version lower than 12:
select name, value
from v$parameter
where name in (
'optimizer_adaptive_features', --12.1 only
'optimizer_adaptive_plans', --12.2+
'optimizer_adaptive_reporting_only',
'optimizer_features_enable',
'compatible'
)
order by 1;
EDIT 1
I'm not sure why adaptive plans aren't working for you. If nobody can figure that out, then we'll need to investigate the execution plan actual values to find out precisely which operations are slow.
There are at least two ways to get the actual numbers. If you can change and re-run the offending query, you can use the hint GATHER_PLAN_STATISTICS.
--Run slow query and wait for it to finish:
select /*+ gather_plan_statistics */ * from dual;
--Find the SQL_ID of the query using some distinctive text:
select *
from v$sql
where lower(sql_fulltext) like '%gather_plan_statistics%';
--Generate execution plan with actual values.
select *
from table(dbms_xplan.display_cursor(sql_id => 'SQL_ID from above', format=>'allstats last'));
If you cannot alter the query, you can use SQL Monitor reports to find the actual values. (This feature requires Enterprise Edition and a tuning pack license.)
--Generate SQL Monitoring Report:
select dbms_sqltune.report_sql_monitor(sql_id => 'SQL_ID from above') from dual;
EDIT 2
Are you 100% sure you've found the right SQL_ID? You might want to double check GV$SQL. Sometimes the case is switched, if the SQL is submitted from an application or PL/SQL block. And rarely the real SQL statement will age out of GV$SQL, if someone runs alter system flush shared_pool;, or stats are gathered, or if you wait too long.
If that is truly the right execution plan, no time is spent on the query. Normally this means that time must be spent by the network sending the results, or the application processing the results. But since there are no rows returned a network or application issue doesn't sound likely.
If the time is spent on the database, but not on that query, then I would guess it's either a parsing issue or a recursive query issue. Parsing issues can be found by tracing, but those are caused by extremely unusual problems or queries, and likely isn't the case here.
Perhaps one of the queries Oracle uses to gather metadata is taking too long. For many queries, Oracle needs to run other queries that check privileges, dynamic sampling, etc. You may need to tune one of those other queries, and the below statements can help with that painful process:
--Clear existing run times (be careful running this on production).
--(This won't flush queries that are actively running.)
alter system flush shared_pool;
--Run your slow SQL statement here.
--...
--Now look for anything "weird" that has taken up most of the time.
select elapsed_time/1000000 seconds, gv$sql.*
from gv$sql
order by seconds desc;