SELECT col, (SELECT COUNT(*) FROM table) as total_count FROM table
This query executes subquery for every row, right?
Now if we have
;WITH CTE(total_count) AS (
SELECT COUNT(*) FROM table
)
SELECT col, (SELECT total_count FROM CTE) FROM table;
Will be second method more efficient? Will CTE execute COUNT(*) only once and then SELECT uses it as prepared value? or in second case also executed COUNT(*) for each row?
For Oracle the surest way is to observe the behaviour of the statements with extended statistics.
Do do so first increase the statistics level to ALL
alter session set statistics_level=all;
Then run both statements (fetching all rows) and find the SQL_ID of those statements
Finally display the statistics using following statement (passing the proper SQL_ID):
select * from table(dbms_xplan.display_cursor('your SQL_ID here',null,'ALLSTATS LAST'));
This gives for my test table
SQL_ID 5n0sdcu8347j9, child number 0
-------------------------------------
SELECT col, (SELECT COUNT(*) FROM t1) as total_count FROM t1
Plan hash value: 1306093980
-------------------------------------------------------------------------------------
| Id | Operation | Name | Starts | E-Rows | A-Rows | A-Time | Buffers |
-------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 1 | | 1000 |00:00:00.01 | 351 |
| 1 | SORT AGGREGATE | | 1 | 1 | 1 |00:00:00.01 | 338 |
| 2 | TABLE ACCESS FULL| T1 | 1 | 1061 | 1000 |00:00:00.01 | 338 |
| 3 | TABLE ACCESS FULL | T1 | 1 | 1061 | 1000 |00:00:00.01 | 351 |
-------------------------------------------------------------------------------------
and
SQL_ID fs0h660f08bj6, child number 0
-------------------------------------
WITH CTE(total_count) AS ( SELECT COUNT(*) FROM t1 ) SELECT col,
(SELECT total_count FROM CTE) FROM t1
Plan hash value: 1223456497
--------------------------------------------------------------------------------------
| Id | Operation | Name | Starts | E-Rows | A-Rows | A-Time | Buffers |
--------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 1 | | 1000 |00:00:00.01 | 351 |
| 1 | VIEW | | 1 | 1 | 1 |00:00:00.01 | 338 |
| 2 | SORT AGGREGATE | | 1 | 1 | 1 |00:00:00.01 | 338 |
| 3 | TABLE ACCESS FULL| T1 | 1 | 1061 | 1000 |00:00:00.01 | 338 |
| 4 | TABLE ACCESS FULL | T1 | 1 | 1061 | 1000 |00:00:00.01 | 351 |
--------------------------------------------------------------------------------------
So the plans are slightly different, but in both cases the FULL TABLE SCAN is started only once (column Starts = 1). Which gives no real difference.
For purpose of camparison I run also a correlated subquery, which gives a complete different picture with high number of Starts (of FTS)
SQL_ID cbvwd6pm6699m, child number 0
-------------------------------------
SELECT col, (SELECT COUNT(*) FROM t1 where col = a.col) as total_count
FROM t1 a
Plan hash value: 1306093980
-------------------------------------------------------------------------------------
| Id | Operation | Name | Starts | E-Rows | A-Rows | A-Time | Buffers |
-------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 1 | | 1000 |00:00:00.01 | 351 |
| 1 | SORT AGGREGATE | | 1000 | 1 | 1000 |00:00:00.31 | 338K|
|* 2 | TABLE ACCESS FULL| T1 | 1000 | 11 | 1000 |00:00:00.31 | 338K|
| 3 | TABLE ACCESS FULL | T1 | 1 | 1061 | 1000 |00:00:00.01 | 351 |
-------------------------------------------------------------------------------------
Predicate Information (identified by operation id):
---------------------------------------------------
2 - filter("COL"=:B1)
I believe that the query optimizer in both Oracle and SQL Server will recognize that the count query is not correlated, compute it once, and then use the cached result throughout the execution of the outer query.
Also, the CTE won't change anything as far as I know, since at execution time the code inside it will basically just be inlined into the actual outer query.
Here is a reference for Oracle which mentions that a non correlated subquery will be executed once and cached, except in cases where the outer query only has a few rows. In that case, it might not be cached because there isn't much of a penalty in executing the count subquery multiple times.
Related
I have the below query, but when I execute it runs forever.
WITH aux AS (
SELECT
contract,
contract_account,
business_partner,
payment_plan,
installation,
contract_status
FROM
reta.mv_integrated_md a
WHERE
contract_status IN (
'LIVE',
'FINAL'
)
), aux1 AS (
SELECT
a.*,
CASE
WHEN EXISTS (
SELECT
NULL
FROM
aux b
WHERE
b.business_partner = a.business_partner
AND b.installation = a.installation
AND b.payment_plan = 'BMW'
) THEN
'X'
END h
FROM
aux a
)
SELECT
*
FROM
aux1;
My execution plan shows a huge cost which I cannot locate. How could I optimize this query? I have tried some hints but none of them have worked :(
Plan hash value: 1662974027
----------------------------------------------------------------------------------------------------------------------------
| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time | Pstart| Pstop |
----------------------------------------------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 19M| 2000M| 825G (1)|999:59:59 | | |
|* 1 | VIEW | | 19M| 990M| 41331 (1)| 00:00:02 | | |
| 2 | TABLE ACCESS STORAGE FULL | SYS_TEMP_0FDA49C92_9A7BE8DE | 19M| 1066M| 41331 (1)| 00:00:02 | | |
| 3 | TEMP TABLE TRANSFORMATION | | | | | | | |
| 4 | LOAD AS SELECT | SYS_TEMP_0FDA49C92_9A7BE8DE | | | | | | |
| 5 | PARTITION RANGE SINGLE | | 18M| 974M| 759K (1)| 00:00:30 | 1 | 1 |
|* 6 | TABLE ACCESS STORAGE FULL| MV_INTEGRATED_MD | 18M| 974M| 759K (1)| 00:00:30 | 1 | 1 |
| 7 | VIEW | | 19M| 2000M| 41331 (1)| 00:00:02 | | |
| 8 | TABLE ACCESS STORAGE FULL | SYS_TEMP_0FDA49C92_9A7BE8DE | 19M| 1066M| 41331 (1)| 00:00:02 | | |
----------------------------------------------------------------------------------------------------------------------------
Kindly let me know if any additional information needed.
Use window functions:
SELECT r.contract, r.contract_account, r.business_partner,
r.payment_plan, r.installation, r.contract_status,
MAX(CASE WHEN r.payment_plan = 'BMW' THEN 'X' END) OVER (PARTITION BY business_partner, installation) as h
FROM reta.mv_integrated_md#rbip r
WHERE r.contract_status IN ('LIVE', 'FINAL');
Not only is the query much simpler to write and read, but it should perform much better too.
Highest cost is due to FTS(Full table scan) on table/MV MV_INTEGRATED_MD.
Try to create index on contract_status and check if it reduces the cost and also, what is size of this mv/table in terms of block and it is 10 percent or more than total buffer cache size ?
TABLE ACCESS STORAGE FULL| MV_INTEGRATED_MD | 18M| 974M| 759K (1)| 00:00:30 | 1 | 1
If you run your query with the /*+ gather_plan_statistics */ hint (I'm simulating it with a 1000 row table) you imediately see the problem :
select * from table(dbms_xplan.display_cursor(null,null,'ALLSTATS LAST'));
-------------------------------------------------------------------------------------------------------------------------------------------
| Id | Operation | Name | Starts | E-Rows | A-Rows | A-Time | Buffers | Reads |
-------------------------------------------------------------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 1 | | 1000 |00:00:00.01 | 9 | 5 |
|* 1 | VIEW | | 1000 | 1000 | 1000 |00:00:00.09 | 0 | 0 |
| 2 | TABLE ACCESS FULL | SYS_TEMP_0FD9D6737_1A17DE13 | 1000 | 1000 | 500K|00:00:00.08 | 0 | 0 |
| 3 | TEMP TABLE TRANSFORMATION | | 1 | | 1000 |00:00:00.01 | 9 | 5 |
| 4 | LOAD AS SELECT (CURSOR DURATION MEMORY)| SYS_TEMP_0FD9D6737_1A17DE13 | 1 | | 0 |00:00:00.01 | 8 | 5 |
|* 5 | TABLE ACCESS FULL | MV_INTEGRATED_MD | 1 | 1000 | 1000 |00:00:00.01 | 7 | 5 |
| 6 | VIEW | | 1 | 1000 | 1000 |00:00:00.01 | 0 | 0 |
| 7 | TABLE ACCESS FULL | SYS_TEMP_0FD9D6737_1A17DE13 | 1 | 1000 | 1000 |00:00:00.01 | 0 | 0 |
-------------------------------------------------------------------------------------------------------------------------------------------
Predicate Information (identified by operation id):
---------------------------------------------------
1 - filter(("B"."BUSINESS_PARTNER"=:B1 AND "B"."INSTALLATION"=:B2 AND "B"."PAYMENT_PLAN"='BMW'))
5 - filter("CONTRACT_STATUS"='LIVE')
It is in the line 2 where a full scan is activated in a loop for each line of the main table (see starts = 1000)
Typically you want to resolve the EXISTS with a semi join to preserve good performance, but here it seems that Oracle can not rewrite it.
So you'll need to rewrite the query yourself.
Despite the excelent proposal of #GordonLinoff (that I'll start with) you may try to use an outer join as follows
with bmw as (
select distinct business_partner, installation
from mv_integrated_md
where payment_plan = 'BMW')
SELECT
a.contract,
a.contract_account,
a.business_partner,
a.payment_plan,
a.installation,
a.contract_status,
case when b.business_partner is not null then 'X' end as h
FROM mv_integrated_md a
left outer join bmw b
on b.business_partner = a.business_partner and
b.installation = a.installation
WHERE a.contract_status IN ( 'LIVE', 'FINAL')
This will lead to two fulls scans, one deduplication and outer join.
i am doubting about this case, but not clear why.
consider the following sql :
create table t1(tid int not null, t1 int not null);
create table t2(t2 int not null, tname varchar(30) null);
create unique index i_t2 on t2(t2);
create or replace view v_1 as
select t1.tid,t1.t1,max(t2.tname) as tname
from t1 left join t2
on t1.t1 = t2.t2
group by t1.tid,t1.t1;
then check the execution plan for select count(1) from v_1, the t2 is eliminated by the optimizer:
SQL> select count(1) from v_1;
Execution Plan
----------------------------------------------------------
Plan hash value: 3243658773
----------------------------------------------------------------------------------
| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time |
----------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 1 | | 3 (34)| 00:00:01 |
| 1 | SORT AGGREGATE | | 1 | | | |
| 2 | VIEW | VM_NWVW_0 | 1 | | 3 (34)| 00:00:01 |
| 3 | HASH GROUP BY | | 1 | 26 | 3 (34)| 00:00:01 |
| 4 | TABLE ACCESS FULL| T1 | 1 | 26 | 2 (0)| 00:00:01 |
----------------------------------------------------------------------------------
but if the index i_t2 is dropped or recreated without unique attribute,
the table t2 is not eliminated in execution plan:
SQL> drop index i_t2;
Index dropped.
SQL> select count(1) from v_1;
Execution Plan
----------------------------------------------------------
Plan hash value: 2710188186
-----------------------------------------------------------------------------------
| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time |
-----------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 1 | | 5 (20)| 00:00:01 |
| 1 | SORT AGGREGATE | | 1 | | | |
| 2 | VIEW | VM_NWVW_0 | 1 | | 5 (20)| 00:00:01 |
| 3 | HASH GROUP BY | | 1 | 39 | 5 (20)| 00:00:01 |
|* 4 | HASH JOIN OUTER | | 1 | 39 | 4 (0)| 00:00:01 |
| 5 | TABLE ACCESS FULL| T1 | 1 | 26 | 2 (0)| 00:00:01 |
| 6 | TABLE ACCESS FULL| T2 | 1 | 13 | 2 (0)| 00:00:01 |
-----------------------------------------------------------------------------------
it seems even if the index is removed,
the result of select count(1) from v_1 also equal to
select count(1) from (select tid,t1 from t1 group by tid,t1)
why the optimizer does not eliminate t2 in the second case?
is there any principle or actual data example discribing this?
thanks :)
This is an optimization called join elimination. Because t2.t2 us unique, the optimizer knows that every row retrieved from t1 can only ever retrieve one row from t2. Since there is nothing projected from t2, there is no need to perform the join.
If you do
select tid, t1 from v_1;
you will see that we do not perform the join. However, if we project from t2, then the join is needed.
I know that there are the following ways to select values present in one table but not in other.
LEFT JOIN, NOT IN and NOT EXISTS
Which is the recommended option to use?
There probably isn't an universal answer - so would appreciate the used-case where each is advisable.
(I am not looking for the syntax of the above options - just a comparison of the approaches)
In short, LEFT JOIN takes slightly more time that other two. But NOT EXISTS and NOT IN took almost same time.
I prefer left join when I need to utilize the values of other table in select clause. Else I prefer not exists.
I suggest you to replicate the below test on your machine as mine is a home machine with Oracle 12c and hardly anything else running. May be in a bigger environment, the test will give more accurate result.
Test in detail:
To practically test it I will create 2 tables and insert first one with 10 Million rows and second one with some other condition from first one, so some rows are not inserted to second table.
--Create first table
create table test_data_left (empno integer, ename varchar2(10),CONSTRAINT tdl_pk primary key(empno));
--PLSQL Block to enter 10 Million rows in test_data_left
declare v_max_empno integer;
BEGIN
select coalesce(max(EMPNO),0) into v_max_empno from emp_data;
FOR i IN 1..1000000 LOOP -- add 10 Million rows
insert into test_data_left(empno,ename) values (
i+v_max_empno,
DBMS_RANDOM.string('U',TRUNC(DBMS_RANDOM.value(10,11)))
);
END LOOP;
END;
/
commit;
--Create second table and populate with some condition to block some rows from first table
create table test_data_right (empno integer, ename varchar2(10),CONSTRAINT tdr_pk primary key(empno));
insert into test_data_right (empno,ename)
select empno,ename from test_data_left
where ename not like 'JK%';
These are the queries I am using to get the data.
NOTE: I am not using t1.* in select statements, as SQL Developer only displays first 50 rows and you cannot run explain plan on it. Hence I am using count(*)
select count(*) from test_data_left t1 left join test_data_right t2 on
t1.empno=t2.empno where t2.empno is nulll
select count(*) from test_data_left t1
where t1.empno not in (select empno from test_data_right);
select count(*) from test_data_left t1
where not exists (select 1 from test_data_right t2 where t1.empno=t2.empno);
To gather status of last run query, I used this command.
SELECT * FROM TABLE(DBMS_XPLAN.DISPLAY_CURSOR(null,null,'ALLSTATS LAST')) ;
Just to be careful, that Oracle is not doing any funny business while calculating, I had reset the database connection before running every query.
Below are the status after each query. I have repeated it in reverse order to give a fair chance to LEFT JOIN.
As far as I can see, the LEFT JOIN is the slowest but NOT IN and
NOT EXISTS are almost same. (based on couple of more iterations which I wasn't able to capture)
Iteration 1
LEFT JOIN
SQL_ID 0qz2qtza4yrr0, child number 0
-------------------------------------
select count(*) from test_data_left t1 left join test_data_right t2 on
t1.empno=t2.empno where t2.empno is null
Plan hash value: 2082679279
-------------------------------------------------------------------------------------------
| Id | Operation | Name | Starts | E-Rows | A-Rows | A-Time | Buffers |
-------------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 1 | | 1 |00:00:01.41 | 5012 |
| 1 | SORT AGGREGATE | | 1 | 1 | 1 |00:00:01.41 | 5012 |
| 2 | NESTED LOOPS ANTI | | 1 | 1206K| 900K|00:00:01.32 | 5012 |
| 3 | INDEX FAST FULL SCAN| TDL_PK | 1 | 1206K| 1000K|00:00:00.22 | 1891 |
|* 4 | INDEX UNIQUE SCAN | TDR_PK | 1000K| 1 | 99865 |00:00:00.54 | 3121 |
-------------------------------------------------------------------------------------------
Predicate Information (identified by operation id):
---------------------------------------------------
4 - access("T1"."EMPNO"="T2"."EMPNO")
Note
-----
- dynamic statistics used: dynamic sampling (level=2)
NOT EXISTS
SQL_ID c498qdbzw5dxv, child number 0
-------------------------------------
select count(*) from test_data_left t1 where not exists (select 1 from
test_data_right t2 where t1.empno=t2.empno)
Plan hash value: 2082679279
-------------------------------------------------------------------------------------------
| Id | Operation | Name | Starts | E-Rows | A-Rows | A-Time | Buffers |
-------------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 1 | | 1 |00:00:01.27 | 5012 |
| 1 | SORT AGGREGATE | | 1 | 1 | 1 |00:00:01.27 | 5012 |
| 2 | NESTED LOOPS ANTI | | 1 | 1206K| 900K|00:00:01.19 | 5012 |
| 3 | INDEX FAST FULL SCAN| TDL_PK | 1 | 1206K| 1000K|00:00:00.21 | 1891 |
|* 4 | INDEX UNIQUE SCAN | TDR_PK | 1000K| 1 | 99865 |00:00:00.49 | 3121 |
-------------------------------------------------------------------------------------------
Predicate Information (identified by operation id):
---------------------------------------------------
4 - access("T1"."EMPNO"="T2"."EMPNO")
Note
-----
- dynamic statistics used: dynamic sampling (level=2)
NOT IN
SQL_ID gwm775xqnufgm, child number 0
-------------------------------------
select count(*) from test_data_left t1 where t1.empno not in (select
empno from test_data_right)
Plan hash value: 2082679279
-------------------------------------------------------------------------------------------
| Id | Operation | Name | Starts | E-Rows | A-Rows | A-Time | Buffers |
-------------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 1 | | 1 |00:00:01.23 | 5012 |
| 1 | SORT AGGREGATE | | 1 | 1 | 1 |00:00:01.23 | 5012 |
| 2 | NESTED LOOPS ANTI | | 1 | 1206K| 900K|00:00:01.15 | 5012 |
| 3 | INDEX FAST FULL SCAN| TDL_PK | 1 | 1206K| 1000K|00:00:00.19 | 1891 |
|* 4 | INDEX UNIQUE SCAN | TDR_PK | 1000K| 1 | 99865 |00:00:00.47 | 3121 |
-------------------------------------------------------------------------------------------
Predicate Information (identified by operation id):
---------------------------------------------------
4 - access("T1"."EMPNO"="EMPNO")
Note
-----
- dynamic statistics used: dynamic sampling (level=2)
ITERATION 2
NOT IN
SQL_ID gwm775xqnufgm, child number 0
-------------------------------------
select count(*) from test_data_left t1 where t1.empno not in (select
empno from test_data_right)
Plan hash value: 2082679279
-------------------------------------------------------------------------------------------
| Id | Operation | Name | Starts | E-Rows | A-Rows | A-Time | Buffers |
-------------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 1 | | 1 |00:00:01.19 | 5012 |
| 1 | SORT AGGREGATE | | 1 | 1 | 1 |00:00:01.19 | 5012 |
| 2 | NESTED LOOPS ANTI | | 1 | 1206K| 900K|00:00:01.11 | 5012 |
| 3 | INDEX FAST FULL SCAN| TDL_PK | 1 | 1206K| 1000K|00:00:00.19 | 1891 |
|* 4 | INDEX UNIQUE SCAN | TDR_PK | 1000K| 1 | 99865 |00:00:00.46 | 3121 |
-------------------------------------------------------------------------------------------
Predicate Information (identified by operation id):
---------------------------------------------------
4 - access("T1"."EMPNO"="EMPNO")
Note
-----
- dynamic statistics used: dynamic sampling (level=2)
NOT EXISTS
SQL_ID c498qdbzw5dxv, child number 0
-------------------------------------
select count(*) from test_data_left t1 where not exists (select 1 from
test_data_right t2 where t1.empno=t2.empno)
Plan hash value: 2082679279
-------------------------------------------------------------------------------------------
| Id | Operation | Name | Starts | E-Rows | A-Rows | A-Time | Buffers |
-------------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 1 | | 1 |00:00:01.19 | 5012 |
| 1 | SORT AGGREGATE | | 1 | 1 | 1 |00:00:01.19 | 5012 |
| 2 | NESTED LOOPS ANTI | | 1 | 1206K| 900K|00:00:01.12 | 5012 |
| 3 | INDEX FAST FULL SCAN| TDL_PK | 1 | 1206K| 1000K|00:00:00.19 | 1891 |
|* 4 | INDEX UNIQUE SCAN | TDR_PK | 1000K| 1 | 99865 |00:00:00.46 | 3121 |
-------------------------------------------------------------------------------------------
Predicate Information (identified by operation id):
---------------------------------------------------
4 - access("T1"."EMPNO"="T2"."EMPNO")
Note
-----
- dynamic statistics used: dynamic sampling (level=2)
LEFT JOIN
SQL_ID 0qz2qtza4yrr0, child number 0
-------------------------------------
select count(*) from test_data_left t1 left join test_data_right t2 on
t1.empno=t2.empno where t2.empno is null
Plan hash value: 2082679279
-------------------------------------------------------------------------------------------
| Id | Operation | Name | Starts | E-Rows | A-Rows | A-Time | Buffers |
-------------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 1 | | 1 |00:00:01.33 | 5012 |
| 1 | SORT AGGREGATE | | 1 | 1 | 1 |00:00:01.33 | 5012 |
| 2 | NESTED LOOPS ANTI | | 1 | 1206K| 900K|00:00:01.24 | 5012 |
| 3 | INDEX FAST FULL SCAN| TDL_PK | 1 | 1206K| 1000K|00:00:00.22 | 1891 |
|* 4 | INDEX UNIQUE SCAN | TDR_PK | 1000K| 1 | 99865 |00:00:00.50 | 3121 |
-------------------------------------------------------------------------------------------
Predicate Information (identified by operation id):
---------------------------------------------------
4 - access("T1"."EMPNO"="T2"."EMPNO")
Note
-----
- dynamic statistics used: dynamic sampling (level=2)
this will return everything from table a where there is not corresponding record in table b
SELECT a.col FROM a WHERE a.col NOT IN (SELECT b.col from b)
Try on below query
select tabA.* from tabA left join tabB on tabA.id = tabB.tabA_id
where tabB.tabA_id is null
Hope it can help.
I have a set of complex optimized selects that suffer from physical reads. Without them they would be even faster!
These physical reads occur due to the WITH clause, one physical_read_request per WITH sub-query. They seem totally unnecessary to me, I'd prefer Oracle keeping the sub-query results in memory instead of writing them down to disk and reading them again.
I'm looking for a way how to get rid of these phys reads.
A simple sample having the same problems is this:
Edit: Example replaced with simpler one that does not use dictionary views.
alter session set STATISTICS_LEVEL=ALL;
create table T as
select level NUM from dual
connect by level <= 1000;
with /*a2*/ TT as (
select NUM from T
where NUM between 100 and 110
)
select * from TT
union all
select * from TT
;
SELECT * FROM TABLE(dbms_xplan.display_cursor(
(select sql_id from v$sql
where sql_fulltext like 'with /*a2*/ TT%'
and sql_fulltext not like '%v$sql%'
and sql_fulltext not like 'explain%'),
NULL, format=>'allstats last'));
and the corresponding execution plan is
SQL_ID bpqnhfdmxnqvp, child number 0
-------------------------------------
with /*a2*/ TT as ( select NUM from T where NUM between 100 and
110 ) select * from TT union all select * from TT
Plan hash value: 4255080040
---------------------------------------------------------------------------------------------------------------------------------------------------------------
| Id | Operation | Name | Starts | E-Rows | A-Rows | A-Time | Buffers | Reads | Writes | OMem | 1Mem | Used-Mem |
---------------------------------------------------------------------------------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 1 | | 22 |00:00:00.01 | 20 | 1 | 1 | | | |
| 1 | TEMP TABLE TRANSFORMATION | | 1 | | 22 |00:00:00.01 | 20 | 1 | 1 | | | |
| 2 | LOAD AS SELECT | | 1 | | 0 |00:00:00.01 | 8 | 0 | 1 | 266K| 266K| 266K (0)|
|* 3 | TABLE ACCESS FULL | T | 1 | 11 | 11 |00:00:00.01 | 4 | 0 | 0 | | | |
| 4 | UNION-ALL | | 1 | | 22 |00:00:00.01 | 9 | 1 | 0 | | | |
| 5 | VIEW | | 1 | 11 | 11 |00:00:00.01 | 6 | 1 | 0 | | | |
| 6 | TABLE ACCESS FULL | SYS_TEMP_0FD9D6646_63A776 | 1 | 11 | 11 |00:00:00.01 | 6 | 1 | 0 | | | |
| 7 | VIEW | | 1 | 11 | 11 |00:00:00.01 | 3 | 0 | 0 | | | |
| 8 | TABLE ACCESS FULL | SYS_TEMP_0FD9D6646_63A776 | 1 | 11 | 11 |00:00:00.01 | 3 | 0 | 0 | | | |
---------------------------------------------------------------------------------------------------------------------------------------------------------------
Predicate Information (identified by operation id):
---------------------------------------------------
3 - filter(("NUM">=100 AND "NUM"<=110))
Note
-----
- dynamic sampling used for this statement (level=2)
See the (phys) Write upon each WITH view creation, and the (phys) Read upon the first view usage. I also tried the RESULT_CACHE hint (which is not reflected in this sample select, but was reflected in my original queries), but it doesn't remove the disk accesses either (which is understandable).
How can I get rid of the phys writes/reads?
I'm attempting to build an infrastructure for quickly running regressions on demand, pulling apache requests from a database that contains all historic activity on our webservers. To improve coverage by making sure that we still regress requests from our smaller clients, I'd like to ensure a distribution of requests by retrieving at most n (for the sake of this question, say 10) requests for each client.
I found a number of similar questions answered here, the closest of which seemed to be SQL query to return top N rows per ID across a range of IDs, but the answers were for the most part performance-agnostic solutions
that I had already tried. For instance, the row_number() analytic function gets us exactly the data we're looking for:
SELECT
*
FROM
(
SELECT
dailylogdata.*,
row_number() over (partition by dailylogdata.contextid order by occurrencedate) rn
FROM
dailylogdata
WHERE
shorturl in (?)
)
WHERE
rn <= 10;
However, given that this table contains millions of entries for a given day and this approach necessitates reading all rows from the index that match our selection criteria in order to apply the row_number analytic function, performance is terrible. We end up selecting nearly a million rows, only to throw out the vast majority of them because their row_number exceeded 10. Stats from executing the above query:
|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
|| Id | Operation | Name | Starts | E-Rows | A-Rows | A-Time | Buffers | Reads | Writes | OMem | 1Mem | Used-Mem | Used-Tmp||
|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
|| 0 | SELECT STATEMENT | | 1 | | 12222 |00:09:08.94 | 895K| 584K| 301 | | | | ||
||* 1 | VIEW | | 1 | 4427K| 12222 |00:09:08.94 | 895K| 584K| 301 | | | | ||
||* 2 | WINDOW SORT PUSHED RANK | | 1 | 4427K| 13536 |00:09:08.94 | 895K| 584K| 301 | 2709K| 743K| 97M (1)| 4096 ||
|| 3 | PARTITION RANGE SINGLE | | 1 | 4427K| 932K|00:22:27.90 | 895K| 584K| 0 | | | | ||
|| 4 | TABLE ACCESS BY LOCAL INDEX ROWID| DAILYLOGDATA | 1 | 4427K| 932K|00:22:27.61 | 895K| 584K| 0 | | | | ||
||* 5 | INDEX RANGE SCAN | DAILYLOGDATA_URLCONTEXT | 1 | 17345 | 932K|00:00:00.75 | 1448 | 0 | 0 | | | | ||
|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| |
|Predicate Information (identified by operation id): |
|--------------------------------------------------- |
| |
| 1 - filter("RN"<=:SYS_B_2) |
| 2 - filter(ROW_NUMBER() OVER ( PARTITION BY "DAILYLOGDATA"."CONTEXTID" ORDER BY "OCCURRENCEDATE")<=:SYS_B_2) |
| 5 - access("SHORTURL"=:P1) |
| |
+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
However, if instead we only query for the first 10 results for a specific contextid, we can execute this dramatically faster:
SELECT
*
FROM
(
SELECT
dailylogdata.*
FROM
dailylogdata
WHERE
shorturl in (?)
and contextid = ?
)
WHERE
rownum <= 10;
Stats from running this query:
|-------------------------------------------------------------------------------------------------------------------------|
|| Id | Operation | Name | Starts | E-Rows | A-Rows | A-Time | Buffers ||
|-------------------------------------------------------------------------------------------------------------------------|
|| 0 | SELECT STATEMENT | | 1 | | 10 |00:00:00.01 | 14 ||
||* 1 | COUNT STOPKEY | | 1 | | 10 |00:00:00.01 | 14 ||
|| 2 | PARTITION RANGE SINGLE | | 1 | 10 | 10 |00:00:00.01 | 14 ||
|| 3 | TABLE ACCESS BY LOCAL INDEX ROWID| DAILYLOGDATA | 1 | 10 | 10 |00:00:00.01 | 14 ||
||* 4 | INDEX RANGE SCAN | DAILYLOGDATA_URLCONTEXT | 1 | 1 | 10 |00:00:00.01 | 5 ||
|-------------------------------------------------------------------------------------------------------------------------|
| |
|Predicate Information (identified by operation id): |
|--------------------------------------------------- |
| |
| 1 - filter(ROWNUM<=10) |
| 4 - access("SHORTURL"=:P1 AND "CONTEXTID"=TO_NUMBER(:P2)) |
| |
+-------------------------------------------------------------------------------------------------------------------------+
In this instance, Oracle is smart enough to stop retrieving data after getting 10 results. I could gather a complete set of contextids and programatically generate a query consisting of one instance of this query for each contextid and union all the whole mess together, but given the sheer number of contextids, we might run into an internal Oracle limitation, and even if not, this approach reeks of kludge.
Does anyone know of an approach that maintains the simplicity of the first query, while retaining performance commensurate with the second query? Also note that I don't actually care about retrieving a stable set of rows; so long as they satisfy my criteria, they're fine for the purposes of a regression.
Edit: Adam Musch's suggestion did the trick. I'm appending performance results with his changes up here since I can't fit them in a comment response to his answer. I'm also using a larger data set for testing this time, here are the (cached) stats from my original row_number approach for comparison:
|-------------------------------------------------------------------------------------------------------------------------------------------------|
|| Id | Operation | Name | Starts | E-Rows | A-Rows | A-Time | Buffers | Reads | OMem | 1Mem | Used-Mem ||
|-------------------------------------------------------------------------------------------------------------------------------------------------|
|| 0 | SELECT STATEMENT | | 1 | | 12624 |00:00:22.34 | 1186K| 931K| | | ||
||* 1 | VIEW | | 1 | 1163K| 12624 |00:00:22.34 | 1186K| 931K| | | ||
||* 2 | WINDOW NOSORT | | 1 | 1163K| 1213K|00:00:21.82 | 1186K| 931K| 3036M| 17M| ||
|| 3 | TABLE ACCESS BY INDEX ROWID| TWTEST | 1 | 1163K| 1213K|00:00:20.41 | 1186K| 931K| | | ||
||* 4 | INDEX RANGE SCAN | TWTEST_URLCONTEXT | 1 | 1163K| 1213K|00:00:00.81 | 8568 | 0 | | | ||
|-------------------------------------------------------------------------------------------------------------------------------------------------|
| |
|Predicate Information (identified by operation id): |
|--------------------------------------------------- |
| |
| 1 - filter("RN"<=10) |
| 2 - filter(ROW_NUMBER() OVER ( PARTITION BY "CONTEXTID" ORDER BY NULL )<=10) |
| 4 - access("SHORTURL"=:P1) |
+-------------------------------------------------------------------------------------------------------------------------------------------------+
I've taken the liberty of boiling down Adam's suggestion a bit; here's the modified query...
select
*
from
twtest
where
rowid in (
select
rowid
from (
select
rowid,
shorturl,
row_number() over (partition by shorturl, contextid
order by null) rn
from
twtest
)
where rn <= 10
and shorturl in (?)
);
...and stats from its (cached) evaluation:
|--------------------------------------------------------------------------------------------------------------------------------------|
|| Id | Operation | Name | Starts | E-Rows | A-Rows | A-Time | Buffers | OMem | 1Mem | Used-Mem ||
|--------------------------------------------------------------------------------------------------------------------------------------|
|| 0 | SELECT STATEMENT | | 1 | | 12624 |00:00:01.33 | 19391 | | | ||
|| 1 | NESTED LOOPS | | 1 | 1 | 12624 |00:00:01.33 | 19391 | | | ||
|| 2 | VIEW | VW_NSO_1 | 1 | 1163K| 12624 |00:00:01.27 | 6770 | | | ||
|| 3 | HASH UNIQUE | | 1 | 1 | 12624 |00:00:01.27 | 6770 | 1377K| 1377K| 5065K (0)||
||* 4 | VIEW | | 1 | 1163K| 12624 |00:00:01.25 | 6770 | | | ||
||* 5 | WINDOW NOSORT | | 1 | 1163K| 1213K|00:00:01.09 | 6770 | 283M| 5598K| ||
||* 6 | INDEX RANGE SCAN | TWTEST_URLCONTEXT | 1 | 1163K| 1213K|00:00:00.40 | 6770 | | | ||
|| 7 | TABLE ACCESS BY USER ROWID| TWTEST | 12624 | 1 | 12624 |00:00:00.04 | 12621 | | | ||
|--------------------------------------------------------------------------------------------------------------------------------------|
| |
|Predicate Information (identified by operation id): |
|--------------------------------------------------- |
| |
| 4 - filter("RN"<=10) |
| 5 - filter(ROW_NUMBER() OVER ( PARTITION BY "SHORTURL","CONTEXTID" ORDER BY NULL NULL )<=10) |
| 6 - access("SHORTURL"=:P1) |
| |
|Note |
|----- |
| - dynamic sampling used for this statement (level=2) |
| |
+--------------------------------------------------------------------------------------------------------------------------------------+
As advertised, we're only accessing the dailylogdata table for the fully-filtered rows. I'm concerned that it appears to still be doing a full scan of the urlcontext index based on the number of rows it claims to be selecting (1213K), but given that it's using only using 6770 buffers (this number remains constant even if I increase the number of context-specific results) this may be misleading.
This is kind of a janky solution, but it seems to do what you want: shortcut the index scan as soon as possible, and not read data until it's been qualified both by filtering conditioning and top-n query criteria.
Note that it was tested with a shorturl = condition, not a shorturl IN condition.
with rowid_list as
(select rowid
from (select *
from (select rowid,
row_number() over (partition by shorturl, contextid
order by null) rn
from dailylogdata
)
where rn <= 10
)
where shorturl = ?
)
select *
from dailylogdata
where rowid in (select rowid from rowid_list)
The with clause grabs the first 10 rowids filtering a WINDOW NOSORT for each unique combination of shorturl and contextid that meets your criteria. Then it loops over that set of rowids, fetching each by rowid.
----------------------------------------------------------------------------------------------------
| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time |
----------------------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 1 | 286 | 1536 (1)| 00:00:19 |
| 1 | NESTED LOOPS | | 1 | 286 | 1536 (1)| 00:00:19 |
| 2 | VIEW | VW_NSO_1 | 136K| 1596K| 910 (1)| 00:00:11 |
| 3 | HASH UNIQUE | | 1 | 3326K| | |
|* 4 | VIEW | | 136K| 3326K| 910 (1)| 00:00:11 |
|* 5 | WINDOW NOSORT | | 136K| 2794K| 910 (1)| 00:00:11 |
|* 6 | INDEX RANGE SCAN | TABLE_REDACTED_INDEX | 136K| 2794K| 910 (1)| 00:00:11 |
| 7 | TABLE ACCESS BY USER ROWID| TABLE_REDACTED | 1 | 274 | 1 (0)| 00:00:01 |
----------------------------------------------------------------------------------------------------
Predicate Information (identified by operation id):
---------------------------------------------------
4 - filter("RN"<=10)
5 - filter(ROW_NUMBER() OVER ( PARTITION BY "CLIENT_ID","SCE_ID" ORDER BY NULL NULL
)<=10)
6 - access("TABLE_REDACTED"."SHORTURL"=:b1)
It seems to be the sort taking all the time. Is occurrenceDate your clustered index, and if not, is it much quicker if you change to order by your clustered index? I.e. if it's clustered by a sequential id, then order by that.
Last time I simply cached last most interesting rows in a smaller table. with my data distribution it was cheaper to update the cache table on every insert rather than query the bulk table.
I think you should also check other ways/queries to achieve the same result set.
Self-JOIN / GROUP BY
SELECT
d.*
, COUNT(*) AS rn
FROM
dailylogdata AS d
LEFT OUTER JOIN
dailylogdata AS d2
ON d.contextid = d2.contextid
AND d.occurrencedate >= d2.occurrencedate)
AND d2.shorturl IN (?)
WHERE
d.shorturl IN (?)
GROUP BY
d.*
HAVING
COUNT(*) <= 10
And another one which I have no idea if it works correctly:
SELECT
d.*
, COUNT(*) AS rn
FROM
( SELECT DISTINCT
contextid
FROM
dailylogdata
WHERE
shorturl IN (?)
) AS dd
JOIN
dailylogdata AS d
ON d.PK IN
( SELECT
d10.PK
FROM
dailylogdata AS d10
WHERE
d10.contextid = dd.contextid
AND
d10.shorturl IN (?)
AND
rownum <= 10
ORDER BY
d10.occurrencedate
)