Related
My Oracle query takes over 1.5 min and I do not know if it's because of inefficient query writing, bad choice of indexes or some other database issue that I cannot control.
Some tables and data were changed to protect IP.
SELECT /*+ PARALLEL (AUTO) */ COUNT(DISTINCT SUD_USERID)
FROM (
SELECT /*+ PARALLEL (AUTO) */
SUD_USERID ,
CASE WHEN SCH_PAGETYPE = 'Page' AND SUD_EVENTTYPE = 'S'
THEN 'EVENTTYPE1'
WHEN SCH_PAGETYPE = 'Page' AND SUD_EVENTTYPE = 'V'
THEN 'EVENTTYPE2'
WHEN SCH_PAGETYPE = 'Hub' AND SUD_EVENTTYPE = 'S'
THEN 'EVENTTYPE3'
END AS CALC_EVENT_SOURCE,
SUD_EVENT_SOURCE
FROM
(
SELECT /*+ PARALLEL (AUTO) */
UPPER(PAGETYPE)|| '-' || SCH.ID PAGETYPE_ID ,
SCH.PAGETYPE SCH_PAGETYPE
FROM TABLE1 SCH
WHERE SCH.PAGETYPE IN ('Page', 'Hub')
AND SCH.CATEGORY_NAME NOT IN ('archive', 'testcategory')
)
INNER JOIN (
SELECT /*+ PARALLEL (AUTO) */
DISTINCT SUD.TRACEID TRACEID ,
SUD.EVENTTYPE SUD_EVENTTYPE ,
SUD.USERID SUD_USERID,
SUD.EVENT_SOURCE SUD_EVENT_SOURCE
FROM
SOMESCHEMA.USAGE_DETAILS SUD
WHERE
SUD.EVENTTYPE IN ('S', 'V')
)
ON TRACEID = PAGETYPE_ID
INNER JOIN USER_JOB_FAMILY_MAPPING SFD
ON SUD_USERID = SFD.USERID
)
WHERE CALC_EVENT_SOURCE = SUD_EVENT_SOURCE
I could not copy the text of the explain plan (generated via DBeaver)
but here is a screenshot:
USAGE_DETAILS table has 3941810 rows
TABLE1 has 5908 rows
USER_JOB_FAMILY_MAPPING has 578233 rows
There are no keys on any of these tables.
USAGE_DETAILS.TRACEID is VARCHAR2(500) NOT NULL has function index=SUBSTR("TRACEID",1,4) and
another index declared as default but on that column.
USAGE_DETAILS.USERID is VARCHAR2(50) NOT NULL
USAGE_DETAILS.EVENTTYPE is VARCHAR2(10) NOT NULL and has default index
USAGE_DETAILS.EVENT_SOURCE is VARCHAR2(200) NOT NULL and has default index
I have tried doing inner joins on the full tables rather than the parenthetically generated (subselect?) tables, but that did not perform better and also limited my ability to use an alias in the WHERE clause.
I do not know what kind of machine this is running on, just that it's set up for development. I'd like this query to give me accurate answers in under 10s. Some times the query above though still does not return even after 10+ minutes.
Plan hash value: 2784166315
-----------------------------------------------------------------------------------------------------------------------------------
| Id | Operation | Name | Rows | Bytes |TempSpc| Cost (%CPU)| Time |
-----------------------------------------------------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 1 | 27 | | 258K (1)| 00:00:11 |
| 1 | SORT AGGREGATE | | 1 | 27 | | | |
| 2 | VIEW | VM_NWVW_1 | 1809K| 46M| | 258K (1)| 00:00:11 |
| 3 | HASH GROUP BY | | 1809K| 745M| | 258K (1)| 00:00:11 |
|* 4 | HASH JOIN | | 1809K| 745M| | 258K (1)| 00:00:11 |
|* 5 | TABLE ACCESS FULL | TABLE1S | 5875 | 172K| | 309 (0)| 00:00:01 |
| 6 | MERGE JOIN SEMI | | 3079K| 1180M| | 257K (1)| 00:00:11 |
| 7 | SORT JOIN | | 3079K| 1139M| | 254K (1)| 00:00:10 |
| 8 | VIEW | | 3079K| 1139M| | 254K (1)| 00:00:10 |
| 9 | HASH UNIQUE | | 3079K| 1139M| 1202M| 254K (1)| 00:00:10 |
| 10 | INLIST ITERATOR | | | | | | |
| 11 | TABLE ACCESS BY INDEX ROWID BATCHED| USAGE_DETAILS | 3079K| 1139M| | 46 (0)| 00:00:01 |
|* 12 | INDEX RANGE SCAN | IDX_UUD_EVENTTYPE | 13704 | | | 46 (0)| 00:00:01 |
|* 13 | SORT UNIQUE | | 578K| 7905K| 22M| 3558 (1)| 00:00:01 |
| 14 | INDEX FAST FULL SCAN | USERID_IDX | 578K| 7905K| | 704 (1)| 00:00:01 |
-----------------------------------------------------------------------------------------------------------------------------------
Predicate Information (identified by operation id):
---------------------------------------------------
4 - access("TRACEID"=UPPER("PAGETYPE")||'-'||TO_CHAR("SCH"."ID"))
filter("from$_subquery$_004"."SUD_EVENT_SOURCE"=CASE WHEN (("SCH"."PAGETYPE"='Page') AND
("from$_subquery$_004"."SUD_EVENTTYPE"='S')) THEN 'EVENTTYPE1' WHEN (("SCH"."PAGETYPE"='Page') AND
("from$_subquery$_004"."SUD_EVENTTYPE"='V')) THEN 'EVENTTYPE2' WHEN (("SCH"."PAGETYPE"='Hub') AND
("from$_subquery$_004"."SUD_EVENTTYPE"='S')) THEN 'EVENTTYPE3' END )
5 - filter("SCH"."CATEGORY_NAME"<>'archive' AND "SCH"."CATEGORY_NAME"<>'testcategory' AND ("SCH"."PAGETYPE"='Hub' OR
"SCH"."PAGETYPE"='Page'))
12 - access("SUD"."EVENTTYPE"='S' OR "SUD"."EVENTTYPE"='V')
13 - access("from$_subquery$_004"."SUD_USERID"="SFD"."USERID")
filter("from$_subquery$_004"."SUD_USERID"="SFD"."USERID")
Note
-----
- dynamic statistics used: dynamic sampling (level=2)
- automatic DOP: Computed Degree of Parallelism is 1 because of no expensive parallel operation
After running
exec dbms_stats.gather_table_stats(ownname=>'SCHEMA1',tabname=>'USAGE_DETAILS');
exec dbms_stats.gather_table_stats(ownname=>'SCHEMA1',tabname=>'TABLE1');
I have this new plan:
SQL> select plan_table_output from table(dbms_xplan.display());
Plan hash value: 3419946982
----------------------------------------------------------------------------------------------------------------
| Id | Operation | Name | Rows | Bytes |TempSpc| Cost (%CPU)| Time |
----------------------------------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 1 | 27 | | 70152 (1)| 00:00:03 |
| 1 | SORT AGGREGATE | | 1 | 27 | | | |
| 2 | VIEW | VM_NWVW_1 | 53144 | 1401K| | 70152 (1)| 00:00:03 |
| 3 | HASH GROUP BY | | 53144 | 21M| 21M| 70152 (1)| 00:00:03 |
|* 4 | HASH JOIN RIGHT SEMI | | 53144 | 21M| 14M| 65453 (1)| 00:00:03 |
| 5 | INDEX FAST FULL SCAN | USERID_IDX | 578K| 7905K| | 704 (1)| 00:00:01 |
|* 6 | HASH JOIN | | 53144 | 20M| | 62995 (1)| 00:00:03 |
| 7 | JOIN FILTER CREATE | :BF0000 | 5503 | 161K| | 309 (0)| 00:00:01 |
|* 8 | TABLE ACCESS FULL | TABLE1 | 5503 | 161K| | 309 (0)| 00:00:01 |
| 9 | VIEW | | 3549K| 1259M| | 62677 (1)| 00:00:03 |
| 10 | HASH UNIQUE | | 3549K| 159M| 203M| 62677 (1)| 00:00:03 |
| 11 | JOIN FILTER USE | :BF0000 | 3549K| 159M| | 21035 (1)| 00:00:01 |
|* 12 | TABLE ACCESS FULL| USAGE_DETAILS | 3549K| 159M| | 21035 (1)| 00:00:01 |
----------------------------------------------------------------------------------------------------------------
Predicate Information (identified by operation id):
---------------------------------------------------
4 - access("from$_subquery$_004"."SUD_USERID"="SFD"."USERID")
6 - access("TRACEID"=UPPER("PAGETYPE")||'-'||TO_CHAR("SCH"."ID"))
filter("from$_subquery$_004"."SUD_EVENT_SOURCE"=CASE WHEN (("SCH"."PAGETYPE"='Page') AND
("from$_subquery$_004"."SUD_EVENTTYPE"='S')) THEN 'EVENTTYPE1' WHEN (("SCH"."PAGETYPE"='Page') AND
("from$_subquery$_004"."SUD_EVENTTYPE"='V')) THEN 'EVENTTYPE2' WHEN (("SCH"."PAGETYPE"='Hub') AND
("from$_subquery$_004"."SUD_EVENTTYPE"='S')) THEN 'EVENTTYPE3' END )
8 - filter("SCH"."CATEGORY_NAME"<>'archive' AND "SCH"."CATEGORY_NAME"<>'testcategory' AND
("SCH"."PAGETYPE"='Hub' OR "SCH"."PAGETYPE"='Page'))
12 - filter(("SUD"."EVENTTYPE"='S' OR "SUD"."EVENTTYPE"='V') AND
SYS_OP_BLOOM_FILTER(:BF0000,"SUD"."TRACEID"))
Note
-----
- automatic DOP: Computed Degree of Parallelism is 1 because of no expensive parallel operation
37 rows selected.
Is it more likely that the compute statistics massively helped this query or that someone did something else that I was not aware of? Yes, the query ran much better, but I'd feel better too if I knew why.
Hy,
after reviewing your SQL I noticed your statements are full of string comparisons and searches. For example
SELECT /*+ PARALLEL (AUTO) */
UPPER(PAGETYPE)|| '-' || SCH.ID PAGETYPE_ID ,
SCH.PAGETYPE SCH_PAGETYPE
FROM TABLE1 SCH
WHERE SCH.PAGETYPE IN ('Page', 'Hub')
AND SCH.CATEGORY_NAME NOT IN ('archive', 'testcategory')
This can be indexed int 2 ways.
First: Create table that has 'Page', 'Hub', and other types that you need, create for the a column Index and then "replace" basically adapt your query to resolve those indexes instead of string compare.
Tables can have multiple indices on columns those have to be treated with caution because they create problems in regards of database size.
Also I would check if what are the biggest tables and reorder their selections to the last. Meaning:
if one table has 12 rows and the other 100. First put the 12 row table then the 100 row. This will multiply in your case since the tables and nested and chained.
I made 1 more review and realized I made an oversight.
USAGE_DETAILS table has 3941810 rows
TABLE1 has 5908 rows
USER_JOB_FAMILY_MAPPING has 578233 rows
First filter the table 1, this is costly already, then Inner join raw USAGE_DETAILS and then select the join ID-s.
Then inner join USER_JOB_FAMILY_MAPPING, and select after that. The reason is that the joins are done on the ID which is probably int type.
Gather statistics on the relevant objects like this:
begin
dbms_stats.gather_table_stats(ownname => user, tabname => 'TABLE1');
dbms_stats.gather_table_stats(ownname => 'SOMESCHEMA', tabname => 'USAGE_DETAILS');
end;
/
This line in the execution plan implies that one of the tables is missing statistics:
- dynamic statistics used: dynamic sampling (level=2)
Not all uses of dynamic sampling imply missing statistics, but level 2 is highly suspicious. That sampling level is usually intended to "Apply dynamic sampling to all unanalyzed tables."
Optimizer statistics are necessary for Oracle to make good execution plans. The algorithms and access paths for joining small amounts of data are different than the algorithms and access paths for joining large amounts of data. The optimizer statistics help Oracle estimate the size of the results and build good plans.
If this solves your problem, you should also investigate the root cause. Optimizer statistics should always be gathered manually after a large change, and automatically by the system every night. If you have a large ETL process that significantly changes a table, it should include a call to DBMS_STATS at the end. The database by default gathers stats at 10PM every night, unless a DBA foolishly disabled the autotask.
If that doesn't solve the problem, then regenerate the execution plan with actual numbers using DBMS_SQLTUNE or the GATHER_PLAN_STATISTICS_HINT. SQL tuning is about optimizing the operations. Your SQL statement has 14 operations, each of which is like a miniature program. We need to know which one of the operations is causing the problem. Finding actual cardinalities and actual run times, and comparing them to estimates, helps tremendously with diagnosing SQL problems.
How do we know that gathering stats was what fixed the performance?
We can't be 100% sure. But it's a safe bet that gathering statistics was responsible for the improvement, for several reasons.
Bad or missing statistics are responsible for a large percentage of all Oracle performance problems. Ask any DBA and they'll have plenty of stories about missing statistics.
The Note section changes strongly imply there are no other weird things happening behind the scenes. There are lots of tricks to silently fix queries, like SQL profiles, baselines, adaptive reoptimization, dynamic sampling (shows up in the first plan, but not the second one, because stats are better), etc. But if those tricks were used they would show up in the Note section.
My requirement is to find the idle period for the each customer.To find the idle customer first i have to fetch the
registration table and it has 1 million records. To find out the last transaction time for each customer i have to
join the transaction log table it has 60 million records.Below is my query for that.
SELECT CUSTOMERNAME,MOBILENUMBER,ACCOUNTNUMBER,
CUSTOMERID,LASTTXNDATE,
FLOOR(SYSDATE - to_date(TO_CHAR(LASTTXNDATE, 'DD/MM/YYYY'),'DD/MM/YYYY')) AS "IDLE DAYS"
FROM REGN_MAST
LEFT JOIN
( SELECT TXNMOBILENUMBER,MAX(TXNDT) AS LASTTXNDATE
FROM TXN_DETL
GROUP BY TXNMOBILENUMBER
)
ON MOBILENUMBER=TXNMOBILENUMBER;
explain plan for
SELECT CUSTOMERNAME,MOBILENUMBER,ACCOUNTNUMBER,
CUSTOMERID,LASTTXNDATE,
FLOOR(SYSDATE - to_date(TO_CHAR(LASTTXNDATE, 'DD/MM/YYYY'),'DD/MM/YYYY')) AS "IDLE DAYS"
FROM REGN_MAST
LEFT JOIN
( SELECT TXNMOBILENUMBER,MAX(TXNDT) AS LASTTXNDATE
FROM TXN_DETL
GROUP BY TXNMOBILENUMBER
)
ON MOBILENUMBER=TXNMOBILENUMBER;
Plan hash value: 403296370
------------------------------------------------------------------------------------------------------------------------------------------
| Id | Operation | Name | Rows | Bytes |TempSpc| Cost (%CPU)| Time | Pstart| Pstop |
------------------------------------------------------------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 1231K| 102M| | 1554K (1)| 05:10:59 | | |
|* 1 | HASH JOIN RIGHT OUTER | | 1231K| 102M| 58M| 1554K (1)| 05:10:59 | | |
| 2 | VIEW | | 1565K| 40M| | 1535K (1)| 05:07:07 | | |
| 3 | HASH GROUP BY | | 1565K| 37M| 2792M| 1535K (1)| 05:07:07 | | |
| 4 | PARTITION RANGE ALL | | 80M| 1926M| | 1321K (1)| 04:24:24 | 1 |1048575|
| 5 | PARTITION HASH ALL | | 80M| 1926M| | 1321K (1)| 04:24:24 | 1 | 4 |
| 6 | TABLE ACCESS FULL | TXN_DETL | 80M| 1926M| | 1321K (1)| 04:24:24 | 1 |1048575|
| 7 | PARTITION RANGE ALL | | 1231K| 70M| | 12237 (1)| 00:02:27 | 1 |1048575|
| 8 | PARTITION HASH ALL | | 1231K| 70M| | 12237 (1)| 00:02:27 | 1 | 4 |
| 9 | TABLE ACCESS BY LOCAL INDEX ROWID| REGN_MAST | 1231K| 70M| | 12237 (1)| 00:02:27 | 1 |1048575|
| 10 | BITMAP CONVERSION TO ROWIDS | | | | | | | | |
| 11 | BITMAP INDEX FULL SCAN | IDX_REGN_MAST_7 | | | | | | 1 |1048575|
------------------------------------------------------------------------------------------------------------------------------------------
Predicate Information (identified by operation id):
---------------------------------------------------
1 - access("MOBILENUMBER"="TXNMOBILENUMBER"(+))
Note
-----
- dynamic sampling used for this statement (level=11)
------------------------------------------------------------------------------------------------------------------------------------------------
This query takes more than 25 minutes.How to improve the performance of this query.
Any help will be greatly appreciated!!!!!!
Your query uses all data from both tables, so the first choice is to chect the execution plan using the FULL TABLE SCAN.
Remember FULL TABLE SCAN is slow, but selecting all rows from a table with an INDEX is much slower...
So you should approach an execotion plan as follows:
------------------------------------------------------------------------------------------
| Id | Operation | Name | Rows | Bytes |TempSpc| Cost (%CPU)| Time |
------------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 1000K| 60M| | 176K (2)| 00:00:07 |
|* 1 | HASH JOIN OUTER | | 1000K| 60M| 41M| 176K (2)| 00:00:07 |
| 2 | TABLE ACCESS FULL | REGN_MAST | 1000K| 29M| | 1370 (1)| 00:00:01 |
| 3 | VIEW | | 1014K| 30M| | 170K (2)| 00:00:07 |
| 4 | HASH GROUP BY | | 1014K| 16M| 1610M| 170K (2)| 00:00:07 |
| 5 | TABLE ACCESS FULL| TXN_DETL | 60M| 972M| | 49771 (1)| 00:00:02 |
------------------------------------------------------------------------------------------
Predicate Information (identified by operation id):
---------------------------------------------------
1 - access("MOBILENUMBER"="TXNMOBILENUMBER"(+))
Depending on your HW and memory configuration the time will vary, but on a recent HW I'd expect elapces time below 10 minutes.
You may further limit it using
a) parallel query
b) keep a materialized view holding the last transaction date
Here my test with generated data leding to 5+ minutes (see below).
So my advice either remove all indexes or hint the FULL and retry.
SQL> set timi on
SQL> set autotrace traceonly
SQL> SELECT CUSTOMERNAME,MOBILENUMBER,ACCOUNTNUMBER,
2 CUSTOMERID,LASTTXNDATE,
3 FLOOR(SYSDATE - to_date(TO_CHAR(LASTTXNDATE, 'DD/MM/YYYY'),'DD/MM/YYYY')
) AS "IDLE DAYS"
4 FROM REGN_MAST
5 LEFT JOIN
6 ( SELECT TXNMOBILENUMBER,MAX(TXNDT) AS LASTTXNDATE
7 FROM TXN_DETL
8 GROUP BY TXNMOBILENUMBER
9 )
10 ON MOBILENUMBER=TXNMOBILENUMBER;
1000000 rows selected.
Elapsed: 00:05:42.23
Sample Data
create table REGN_MAST
as
select
'Name'||rownum CUSTOMERNAME,'00'||rownum MOBILENUMBER, 99*rownum ACCOUNTNUMBER, rownum CUSTOMERID
from dual connect by level <= 1000000;
create table TXN_DETL
as
with cust as (
select
'00'||rownum TXNMOBILENUMBER
from dual connect by level <= 1000000),
trans as (
select DATE'2018-01-01' + rownum TXNDT
from dual connect by level <= 60)
select TXNMOBILENUMBER, TXNDT
from cust CROSS join trans;
I would try rewriting the query as:
SELECT m.CUSTOMERNAME, m.MOBILENUMBER, m.ACCOUNTNUMBER,
m.CUSTOMERID, t.TXNDT,
FLOOR(SYSDATE - TRUNC(TXNDT)) AS IDLE_DAYS
FROM REGN_MAST m JOIN
TXN_DETL t
ON m.MOBILENUMBER = t.TXNMOBILENUMBER
WHERE t.TXNDT = (SELECT MAX(t2.TXNDT) FROM TXN_DETL t2 WHERE m.MOBILENUMBER = t2.TXNMOBILENUMBER);
Then, be sure that you have an index on TXN_DETL(TXNMOBILENUMBER, TXNDT) for performance.
I changed the LEFT JOIN to an INNER JOIN under the assumption that all customers have transactions.
This also simplifies the date arithmetic. That has less to do with performance than readability.
Create a covering index on TXN_DETL(TXNMOBILENUMBER,TXNDT).
According to the execution plan 86% of the cost is for the full table scan on TXN_DETL. If there is an index on all the relevant columns Oracle can use that index as a skinny table. An INDEX FAST FULL SCAN operation might run significantly faster than TABLE ACCESS FULL.
We have a partioned table in our Oracle database using this syntaxe:
...
PARTITION BY RANGE(saledate)
(PARTITION sal99q1 VALUES LESS THAN (TO_DATE('01-APR-1999', 'DD-MON-YYYY')),
PARTITION sal99q2 VALUES LESS THAN (TO_DATE('01-JUL-1999', 'DD-MON-YYYY')),
...
We usually use partition key in select statement like this:
Select * from table where saledate >= trunc(sysdate-3) and saledate < trunc(sysdate-2)
To have same result using less code, I usually use this query instead :
Select * from table where trunc(saledate) = trunc(sysdate-3)
My question is, by using partition key in a function, in this case trunc(), do we loose partioning performance ?
You are misinterpreting the plan shown by Toad (in your answer). For the two queries that shows, respectively:
Partition #: 2 Partitions determined by key values
and
Partition #: 1 Partitions accessed #1 - #17
The first query is accessing only the partitions it needs, based on the key value, which is the date; so it only has to do a full scan of the partition(s) that could contain your date.
The second query has to access all partitions because you are manipulating the key value with a function, meaning you are no longer really using the partition key. The key is saledate, not trunc(saledate). This is similar to what happens when you use a function on an indexed column; the index is no longer used in that case, and the partition key is no longer used here. And as you seem to suspect from your question, yes, you do lose efficiency.
You can also see that the cardinality has been guessed as 50 because of the function call, instead of the stats-provided value of 4966.
You can see the same thing from a dummy table using dbms_xplan; from your first query:
--------------------------------------------------------------------------------------------------
| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time | Pstart| Pstop |
--------------------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 4996 | 39968 | 14 (0)| 00:00:01 | | |
|* 1 | FILTER | | | | | | | |
| 2 | PARTITION RANGE ITERATOR| | 4996 | 39968 | 14 (0)| 00:00:01 | KEY | KEY |
|* 3 | TABLE ACCESS FULL | T42 | 4996 | 39968 | 14 (0)| 00:00:01 | KEY | KEY |
--------------------------------------------------------------------------------------------------
Predicate Information (identified by operation id):
---------------------------------------------------
1 - filter(TRUNC(SYSDATE#!-3)<TRUNC(SYSDATE#!-2))
3 - filter("SALEDATE">=TRUNC(SYSDATE#!-3) AND "SALEDATE"<TRUNC(SYSDATE#!-2))
And from your second query:
--------------------------------------------------------------------------------------------
| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time | Pstart| Pstop |
--------------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 50 | 400 | 14 (0)| 00:00:01 | | |
| 1 | PARTITION RANGE ALL| | 50 | 400 | 14 (0)| 00:00:01 | 1 | 17 |
|* 2 | TABLE ACCESS FULL | T42 | 50 | 400 | 14 (0)| 00:00:01 | 1 | 17 |
--------------------------------------------------------------------------------------------
Predicate Information (identified by operation id):
---------------------------------------------------
2 - filter(TRUNC(INTERNAL_FUNCTION("SALEDATE"))=TRUNC(SYSDATE#!-3))
Notice the pstart/pstop values and the cardinality in each query.
Your first query is going to be more efficient because it can use the partition key to be selective about which partitions it does a full scan of, while the second cannot and has to scan them all.
I have a query which is running fine and giving me output, Here the problem is, same query is taking different elapsed times to get comeplete it's run. The avereage elapased time is 10 mins, but some times it is taking more than a hour and query using sql_profile to get the best execution plan and it is forced every time by DBA.
INSERT INTO DATA_UPDATE_EVENT
(
structure_definition_id,
eod_run_id,
publish_group_name,
JMSDBUS_DESTINATION,
dbaxbuild_location,
LOCATION,
original_data_type)
SELECT DISTINCT eod_structure_definition_id,
p_eod_run_id,
p_publish_group,
pg.JMSDBUS_DESTINATION,
pg.dbaxbuild_location,
pg.LOCATION,
sd.DATA_TYPE
FROM
PUBLISH_GROUP pg,
STRUCTURE_EOD_MAPPING sem,
WATCH_LIST_STRUCTURE wls,
STRUCTURE_DEFINITION sd
WHERE pg.publish_group_name = sem.publish_group_name
AND sem.publish_group_name = p_publish_group
AND wls.structure_definition_id = sem.structure_definition_id
AND wls.watch_list_id IN (SELECT watch_list_id
FROM TMP_WATCHLIST)
AND sd.structure_definition_id = sem.structure_definition_id
AND (sd.defcurve_name IS NULL
OR sd.defcurve_name IN (SELECT curve_shortname
FROM
DEFCURVE_CURRENT
WHERE CURVE_STATUS = 'live')
)
AND (sd.generic_class_name is null
or sd.generic_class_name <> 'CREDIT'
or (
sd.generic_class_name = 'CREDIT'
and generic_name in
(
select generic_name
from
analytic_object ao,
analytic_object_instance aoi,
analytic_object_property aop,
defcurve_current dc
where ao.analytic_object_id = aoi.analytic_object_id
and aop.analytic_object_instance_id =aoi.analytic_object_instance_id
and AOP.PROPERTY_NAME = 'CreditObjectName'
and aop.prop_value1 = dc.curve_shortname
and aop.effective_to > systimestamp
and aop.effective_from < systimestamp
and dc.curve_status = 'live'
and aoi.analytic_object_instance_id in
(select analytic_object_instance_id
from
analytic_object_property
where property_name = 'CreditObjectType'
)
)
)
);
Please take the execution plan for the above query
Execution Plan
------------------------------------------------------------------------------------------------------------------------
| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time
------------------------------------------------------------------------------------------------------------------------
| 0 | INSERT STATEMENT | | | | 232 (100)|
| 1 | VIEW | VW_DIS_1 | 1 | 4052 | 232 (1)| 00:00:
| 2 | SORT UNIQUE | | 1 | 186 | 232 (1)| 00:00:
| 3 | FILTER | | | | |
| 4 | NESTED LOOPS | | 1 | 186 | 231 (0)| 00:00:
| 5 | NESTED LOOPS | | 1 | 158 | 229 (0)| 00:00:
| 6 | NESTED LOOPS | | 1 | 132 | 228 (0)| 00:00:
| 7 | NESTED LOOPS | | 15 | 1830 | 3 (0)| 00:00:
| 8 | TABLE ACCESS BY INDEX ROWID | PUBLISH_GROUP | 1 | 109 | 1 (0)| 00:00:
| 9 | INDEX UNIQUE SCAN | PK$PUBLISHGROUP | 1 | | 0 (0)|
| 10 | TABLE ACCESS FULL | TMP_WATCHLIST | 15 | 195 | 2 (0)| 00:00:
| 11 | INDEX RANGE SCAN | PK$WATCHLISTSTRUCTURE | 1322 | 13220 | 15 (0)| 00:00:
| 12 | INDEX UNIQUE SCAN | PK$STRUCTURE_EOD_MAPPING | 1 | 26 | 1 (0)| 00:00:
| 13 | TABLE ACCESS BY INDEX ROWID | STRUCTURE_DEFINITION | 1 | 28 | 2 (0)| 00:00:
| 14 | INDEX UNIQUE SCAN | PK$STRUCTURE_DEFINITION | 1 | | 1 (0)| 00:00:
| 15 | TABLE ACCESS BY INDEX ROWID | DEFCURVE_CURRENT | 1 | 22 | 2 (0)| 00:00:
| 16 | INDEX UNIQUE SCAN | PK$DEFCURVE_CURRENT | 1 | | 1 (0)| 00:00:
| 17 | PX COORDINATOR | | | | |
| 18 | PX SEND QC (RANDOM) | :TQ10000 | 1 | 118 | 5023 (1)| 00:01:
| 19 | NESTED LOOPS | | 1 | 118 | 5023 (1)| 00:01:
| 20 | NESTED LOOPS | | 1 | 96 | 5023 (1)| 00:01:
| 21 | NESTED LOOPS | | 135 | 6480 | 4939 (1)| 00:01:
| 22 | NESTED LOOPS | | 8343 | 236K| 1228 (1)| 00:00:
| 23 | PX BLOCK ITERATOR | | | | |
| 24 | TABLE ACCESS FULL | ANALYTIC_OBJECT | 28 | 504 | 231 (1)| 00:00:
| 25 | TABLE ACCESS BY GLOBAL INDEX ROWID| ANALYTIC_OBJECT_INSTANCE | 300 | 3300 | 298 (0)| 00:00:
| 26 | INDEX RANGE SCAN | UQ$ANALYTIC_OBJECT_INSTANCE | 300 | | 3 (0)| 00:00:
| 27 | INDEX UNIQUE SCAN | PK$ANALYTIC_OBJECT_PROPERTY | 1 | 19 | 2 (0)| 00:00:
| 28 | TABLE ACCESS BY GLOBAL INDEX ROWID | ANALYTIC_OBJECT_PROPERTY | 1 | 48 | 3 (0)| 00:00:
| 29 | INDEX UNIQUE SCAN | PK$ANALYTIC_OBJECT_PROPERTY | 1 | | 2 (0)| 00:00:
| 30 | TABLE ACCESS BY INDEX ROWID | DEFCURVE_CURRENT | 1 | 22 | 1 (0)| 00:00:
| 31 | INDEX UNIQUE SCAN | PK$DEFCURVE_CURRENT | 1 | | 0 (0)|
------------------------------------------------------------------------------------------------------------------------
Note
-----
- dynamic sampling used for this statement
- SQL profile "SYS_SQLPROF_01505a8ce6144000" used for this statement
Can some one please suggest how to achive this with good steps and what we need to ask DBA to provide the information.
Either you or your DBA need to understand your data and your system. This is the most basic principle of tuning.
Queries will perform predictably, provided the environment is stable. If run times vary wildly then you need to find what is different. Erratically poor performance may be due to lots of other users contending for system resource at particular times or it might be due to variations in the volume or nature of the data. There are other possibilities too, but those are the ones to start with.
Your DBA should already monitor the database usage. But if they aren't they need to start right now. As it doesn't seem likely your organization is paying for the Diagnostics option you can use Statspack for this. Find out more.
As for data variation, there is one clue in the posted code:
AND wls.watch_list_id IN (SELECT watch_list_id
FROM TMP_WATCHLIST)
Assuming you adhere to a sensible naming convention (and years of SO have convinced me to be wary of such assumptions) then TMP_WATCHLIST is a temporary table. Which suggests it could hold different data and different volumes of data each time you run the query. If that is the case that would be a good place to start. Depending on the precise problem possible solutions include dynamic sampling, fixed stats or a cardinality hint.
Here are some ideas:
Bind variable? Is p_publish_group a bind variable? If so, is there a histogram on sem.publish_group_name? It looks like this query returns vastly different amounts depending on the input. Adaptive cursor sharing might help, but that would require a histogram.
Bad profile? The cardinality estimates are horrible. If this statement ran for an hour then I would assume there are many millions of rows. But Oracle only expects 1 rows, even with the SQL Profile. Was the profile created on a very small data set and applied on a much larger data set?
Full hints? Even if your optimizer statistics are accurate I bet the cardinality would still be far off. This may be one of those difficult queries that is so weird it requires a large amount of hints. For example, something like /*+ full(pg) full(sem) use_hash(pg sem) ... */. Indexes and nested loops work fine for small amount of data. But if this query runs for an hour then it likely needs full table scans and hash joins.
SQL Monitoring. Run select dbms_sqltune.report_sql_monitor(sql_id => '<your sql id>', type => 'text') from dual; to find out what the query is actually doing and how much time is spent on each operation. The explain plans are only estimates, you need to know what's really happening. I bet when you run this you'll see a few long-running steps where Estimated Rows = 1 and Actual Rows = 1000000.
I need advice on the attached Query. The query executes for over an hour and has full table scan as per the Explain Plan. I am fairly new to query tuning and would appriciate some advice.
Firstly why would I get a full table scan even though all the columns I use have index created on them.
Secondly, is there any possibility where in I can reduce the execution time, all tables accessed are huge and contain millions of records, even then I would like to scope out some options. Appriciate your help.
Query:
select
distinct rtrim(a.cod_acct_no)||'|'||
a.cod_prod||'|'||
to_char(a.dat_acct_open,'Mon DD YYYY HH:MMAM')||'|'||
a.cod_acct_title||'|'||
a.cod_acct_stat||'|'||
ltrim(to_char(a.amt_od_limit,'99999999999999999990.999999'))||'|'||
ltrim(to_char(a.bal_book,'99999999999999999990.999999'))||'|'||
a.flg_idd_auth||'|'||
a.flg_mnt_status||'|'||
rtrim(c.cod_acct_no)||'|'||
c.cod_10||'|'||
d.nam_branch||'|'||
d.nam_cc_city||'|'||
d.nam_cc_state||'|'||
c.cod_1||'|'||
c.cod_14||'|'||
num_14||'|'||
a.cod_cust||'|'||
c.cod_last_mnt_chkrid||'|'||
c.dat_last_mnt||'|'||
c.ctr_updat_srlno||'|'||
c.cod_20||'|'||
c.num_16||'|'||
c.cod_14||'|'||
c.num_10 ||'|'||
a.flg_classif_reqd||'|'||
(select g.cod_classif_plan_id||'|'||
g.cod_classif_plan_id
from
ac_acct_preferences g
where
a.cod_acct_no=g.cod_acct_no AND g.FLG_MNT_STATUS = 'A' )||'|'||
(select e.dat_cam_expiry from flexprod_host.AC_ACCT_PLAN_CRITERIA e where a.cod_acct_no=e.cod_acct_no and e.FLG_MNT_STATUS ='A')||'|'||
c.cod_23||'|'||
lpad(trim(a.cod_cc_brn),4,0)||'|'||
(select min( o.dat_eff) from ch_acct_od_hist o where a.cod_acct_no=o.cod_acct_no )
from
ch_acct_mast a,
ch_acct_cbr_codes c,
ba_cc_brn_mast d
where
a.flg_mnt_status ='A'
and c.flg_mnt_status ='A'
and a.cod_acct_no= c.cod_acct_no(+)
and a.cod_cc_brn=d.cod_cc_brn
and a.cod_prod in (
299,200,804,863,202,256,814,232,182,844,279,830,802,833,864,
813,862,178,205,801,235,897,231,187,229,847,164,868,805,207,
250,837,274,253,831,893,201,809,846,819,820,845,811,843,285,
894,284,817,832,278,818,810,181,826,867,825,848,871,866,895,
770,806,827,835,838,881,853,188,816,293,298)
Query Plan:
PLAN_TABLE_OUTPUT
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Plan hash value: 4253465430
------------------------------------------------------------------------------------------------------------------------------------------
| Id | Operation | Name | Rows | Bytes |TempSpc| Cost (%CPU)| Time | Pstart| Pstop |
------------------------------------------------------------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 733K| 125M| | 468K (1)|999:59:59 | | |
| 1 | TABLE ACCESS BY INDEX ROWID | AC_ACCT_PREFERENCES | 1 | 26 | | 3 (0)| 00:01:05 | | |
|* 2 | INDEX UNIQUE SCAN | IN_AC_ACCT_PREFERENCES_1 | 1 | | | 2 (0)| 00:00:43 | | |
| 3 | PARTITION HASH SINGLE | | 1 | 31 | | 3 (0)| 00:01:05 | KEY | KEY |
| 4 | TABLE ACCESS BY LOCAL INDEX ROWID| AC_ACCT_PLAN_CRITERIA | 1 | 31 | | 3 (0)| 00:01:05 | KEY | KEY |
|* 5 | INDEX UNIQUE SCAN | IN_AC_ACCT_PLAN_CRITERIA_1 | 1 | | | 2 (0)| 00:00:43 | KEY | KEY |
| 6 | SORT AGGREGATE | | 1 | 29 | | | | | |
| 7 | FIRST ROW | | 1 | 29 | | 3 (0)| 00:01:05 | | |
|* 8 | INDEX RANGE SCAN (MIN/MAX) | IN_CH_ACCT_OD_HIST_1 | 1 | 29 | | 3 (0)| 00:01:05 | | |
| 9 | HASH UNIQUE | | 733K| 125M| 139M| 468K (1)|999:59:59 | | |
|* 10 | HASH JOIN | | 733K| 125M| | 439K (1)|999:59:59 | | |
|* 11 | TABLE ACCESS FULL | BA_CC_BRN_MAST | 3259 | 136K| | 31 (0)| 00:11:04 | | |
|* 12 | HASH JOIN | | 747K| 97M| 61M| 439K (1)|999:59:59 | | |
| 13 | PARTITION HASH ALL | | 740K| 52M| | 286K (1)|999:59:59 | 1 | 64 |
|* 14 | TABLE ACCESS FULL | CH_ACCT_MAST | 740K| 52M| | 286K (1)|999:59:59 | 1 | 64 |
|* 15 | TABLE ACCESS FULL | CH_ACCT_CBR_CODES | 9154K| 541M| | 117K (1)|699:41:01 | | |
------------------------------------------------------------------------------------------------------------------------------------------
Predicate Information (identified by operation id):
---------------------------------------------------
2 - access("COD_ACCT_NO"=:B1 AND "FLG_MNT_STATUS"='A' AND "COD_ENTITY_VPD"=TO_NUMBER(NVL(SYS_CONTEXT('CLIENTCONTEXT','entity_co
de'),'0')))
5 - access("COD_ACCT_NO"=:B1 AND "FLG_MNT_STATUS"='A' AND "COD_ENTITY_VPD"=TO_NUMBER(NVL(SYS_CONTEXT('CLIENTCONTEXT','entity_co
de'),'0')))
8 - access("COD_ACCT_NO"=:B1)
filter("COD_ENTITY_VPD"=TO_NUMBER(NVL(SYS_CONTEXT('CLIENTCONTEXT','entity_code'),'0')))
10 - access("COD_CC_BRN"="COD_CC_BRN")
11 - filter("COD_ENTITY_VPD"=TO_NUMBER(NVL(SYS_CONTEXT('CLIENTCONTEXT','entity_code'),'0')))
12 - access("COD_ACCT_NO"="COD_ACCT_NO")
14 - filter(("COD_PROD"=164 OR "COD_PROD"=178 OR "COD_PROD"=181 OR "COD_PROD"=182 OR "COD_PROD"=187 OR "COD_PROD"=188 OR
"COD_PROD"=200 OR "COD_PROD"=201 OR "COD_PROD"=202 OR "COD_PROD"=205 OR "COD_PROD"=207 OR "COD_PROD"=229 OR "COD_PROD"=231 OR
"COD_PROD"=232 OR "COD_PROD"=235 OR "COD_PROD"=250 OR "COD_PROD"=253 OR "COD_PROD"=256 OR "COD_PROD"=274 OR "COD_PROD"=278 OR
"COD_PROD"=279 OR "COD_PROD"=284 OR "COD_PROD"=285 OR "COD_PROD"=293 OR "COD_PROD"=298 OR "COD_PROD"=299 OR "COD_PROD"=770 OR
"COD_PROD"=801 OR "COD_PROD"=802 OR "COD_PROD"=804 OR "COD_PROD"=805 OR "COD_PROD"=806 OR "COD_PROD"=809 OR "COD_PROD"=810 OR
"COD_PROD"=811 OR "COD_PROD"=813 OR "COD_PROD"=814 OR "COD_PROD"=816 OR "COD_PROD"=817 OR "COD_PROD"=818 OR "COD_PROD"=819 OR
"COD_PROD"=820 OR "COD_PROD"=825 OR "COD_PROD"=826 OR "COD_PROD"=827 OR "COD_PROD"=830 OR "COD_PROD"=831 OR "COD_PROD"=832 OR
"COD_PROD"=833 OR "COD_PROD"=835 OR "COD_PROD"=837 OR "COD_PROD"=838 OR "COD_PROD"=843 OR "COD_PROD"=844 OR "COD_PROD"=845 OR
"COD_PROD"=846 OR "COD_PROD"=847 OR "COD_PROD"=848 OR "COD_PROD"=853 OR "COD_PROD"=862 OR "COD_PROD"=863 OR "COD_PROD"=864 OR
"COD_PROD"=866 OR "COD_PROD"=867 OR "COD_PROD"=868 OR "COD_PROD"=871 OR "COD_PROD"=881 OR "COD_PROD"=893 OR "COD_PROD"=894 OR
"COD_PROD"=895 OR "COD_PROD"=897) AND "FLG_MNT_STATUS"='A' AND "COD_ENTITY_VPD"=TO_NUMBER(NVL(SYS_CONTEXT('CLIENTCONTEXT','entity_
code'),'0')))
15 - filter("FLG_MNT_STATUS"='A' AND "COD_ENTITY_VPD"=TO_NUMBER(NVL(SYS_CONTEXT('CLIENTCONTEXT','entity_code'),'0')))
Considering each table contains over 100 columns I am limited while uploading the entire table definition. however please find the below details for the columns accessed in the where clause. Hope this helps.
Columns Type Nullable
cod_acct_no CHAR(16) N
FLG_MNT_STATUS CHAR(1) N
cod_23 VARCHAR2(360) Y
cod_cc_brn NUMBER(5) N
cod_prod NUMBER N
I Hope this can bring the cost down.
select
distinct rtrim(a.cod_acct_no)||'|'||
a.cod_prod||'|'||
to_char(a.dat_acct_open,'Mon DD YYYY HH:MMAM')||'|'||
a.cod_acct_title||'|'||
a.cod_acct_stat||'|'||
ltrim(to_char(a.amt_od_limit,'99999999999999999990.999999'))||'|'||
ltrim(to_char(a.bal_book,'99999999999999999990.999999'))||'|'||
a.flg_idd_auth||'|'||
a.flg_mnt_status||'|'||
rtrim(c.cod_acct_no)||'|'||
c.cod_10||'|'||
d.nam_branch||'|'||
d.nam_cc_city||'|'||
d.nam_cc_state||'|'||
c.cod_1||'|'||
c.cod_14||'|'||
num_14||'|'||
a.cod_cust||'|'||
c.cod_last_mnt_chkrid||'|'||
c.dat_last_mnt||'|'||
c.ctr_updat_srlno||'|'||
c.cod_20||'|'||
c.num_16||'|'||
c.cod_14||'|'||
c.num_10 ||'|'||
a.flg_classif_reqd||'|'||
g.cod_classif_plan_id||'|'||g.cod_classif_plan_id
||'|'||
e.dat_cam_expiry ||'|'||
c.cod_23||'|'||
lpad(trim(a.cod_cc_brn),4,0)||'|'||
(select min( o.dat_eff) from ch_acct_od_hist o where a.cod_acct_no=o.cod_acct_no )
from
ch_acct_mast a
JOIN ch_acct_cbr_codes c
ON a.flg_mnt_status ='A'
and c.flg_mnt_status ='A'
and a.cod_acct_no= c.cod_acct_no(+)
JOIN ba_cc_brn_mast d
a.cod_cc_brn=d.cod_cc_brn
JOIN ac_acct_preferences g
ON a.cod_acct_no=g.cod_acct_no AND g.FLG_MNT_STATUS = 'A'
INNER JOIN flexprod_host.AC_ACCT_PLAN_CRITERIA e
ON a.cod_acct_no=e.cod_acct_no and e.FLG_MNT_STATUS ='A'
WHERE a.cod_prod in (
299,200,804,863,202,256,814,232,182,844,279,830,802,833,864,
813,862,178,205,801,235,897,231,187,229,847,164,868,805,207,
250,837,274,253,831,893,201,809,846,819,820,845,811,843,285,
894,284,817,832,278,818,810,181,826,867,825,848,871,866,895,
770,806,827,835,838,881,853,188,816,293,298)
1. Don't fear full table scans. If a large percent of the rows in a table are being accessed it is more efficient to use a hash join/full table scan than a nested loop/index scan.
2. Fix statistics and re-analyze objects. 999 hours to read a table? That's probably an optimizer bug, have a dba look at select * from sys.aux_stats$; for some ridiculous values.
The time isn't very useful, but if one of your forecasted values is so significantly off then you need to check all of them. You should probably re-gather stats on all the relevant tables. Use default settings unless there is a good reason. For example, exec dbms_stats.gather_table_stats('your_schema_name','CH_ACCT_MAST');.
3. Look at cardinalities. Are the Rows estimates in the ballpark? They'll almost never be perfect, but if they are off by more than
an order of magnitude or two it can cause problems. Look for the first significant difference and try to correct it.
4. Code change. #Santhosh had a good idea to re-write using ANSI joins and manually unnest a subquery. Although I think you should
try to unnest the other subquery instead. Oracle can automatically unnest subqueries, but not if subqueries "contain aggregate functions".
5. Disable VPD Looks like this query is being transformed. Make sure you understand exactly what it's doing and why. You may want to disable VPD temporarily, for yourself, while you debug this problem.
6. Parallelism. Since some of these tables are large, you may want to add a parallel hint. But be careful, it is easy to use up a lot
of resources. Try to get the plan right before you do this.