I have a table with list partitions on two columns in order of MY_ID (integer with values 1,2,3,5,8...1100), RUN_DATE (past some days).
My query is
select * from my_partitioned_table
where run_date = '10-sep-2014'
and my_id in (select my_id from mapping_table where category = 1)
;
It is going for full table scan, with following explain plan.
PX RECEIVE 115K 4M 600 1,01 PCWP
PX SEND BROADCAST :TQ10000 115K 4M 600 1,00 P->P BROADCAST
PX BLOCK ITERATOR 115K 4M 600 1,00 PCWC
TABLE ACCESS FULL MAPPING_TABLE 115K 4M 600 1,00 PCWP
PX BLOCK ITERATOR 1G 412G 34849 1,01 PCWC 1 16
TABLE ACCESS FULL MY_PARTITIONED_TABLE 1G 412G 34849 1,01 PCWP KEY KEY
How can I force it to access only certain partitions rather than going for full table scan?
Sorry I am little new to Oracle hints and couldn't find specific question before.
That query plan indicates that it is going after one (or more) partitions of my_partitioned_table. So partition pruning is happening already.
You've cut off the column headers when you posted your explain plan (it would also be helpful to get a fixed width version). But the last two columns are almost assuredly the start and end partitions. When you see KEY for a start or an end partition, that means that Oracle is determining the set of partitions that it actually needs to scan at runtime. In this case, it needs to determine the set of my_id values that your subquery will return before it can determine which partitions from your table need to be accessed. The TABLE ACCESS FULL bit merely indicates that it is going to do a full scan of the partition(s) that it needs to access.
Related
Introduction
Hi Everyone, I am a bit of an Oracle SQL novice coming mostly from Python. I have a large procedure that I will outline below with an example. This procedure is taking upwards of 5 minutes to process 500 records and essentially hangs after 750 records. So the runtime is essentially exponentially increasing.
SQL
The general overview of the procedure is made up of two select blocks selecting data from two different sources. These blocks are wrapped inside of a larger select statement that filters and matches records and selects the remainder:
For example:
SELECT DISINCT
*matched sales*
FROM
(SELECT
*direct sales info from db1*
FROM
DB1
WHERE
sales_code = 'DIRECT') a
db2.prod,
db2.cont,
db2.cust, --etc
(SELECT *qualified customer information
FROM *a few DB2 tables*
WHERE code = 'DIR') qual
--A few more of the above inline views to get eligible cust and price
WHERE
*DB2 product numbers, customer numbers and contract numbers are matched to eachother & above
views* This is where the most time is being taken up.
--ex
cust.cont_num = cont.cont_num
*DB1 records matched to DB2 records*
--ex
a.cont_num = cont.cont_num
Question
Ok so my issue here is the performance of the DB2 block essentially, selecting all of the different tables necessary, creating the inline views and matching all of them together. This is taking upwards of 10 minutes by itself.
As a novice, how can I tune this? Would using a temp table to store this block work so it doesn't have to do it over and over? Or should I use more inline views? Nest another select block like the first one?
Explain Plan
OPERATION OBJECT_NAME OPTIONS CARDINALITY COST
SELECT STATEMENT
639039097 31298
HASH JOIN
639039097 31298
INDEX
CARSNG.IE_PRODID_IDX_4 FAST FULL SCAN 9184 13
HASH JOIN
639039097 29585
TABLE ACCESS
CARSNG.UOM FULL 6 3
HASH JOIN
639039097 27881
VIEW
CARSNG.index$_join$_011 8236 77
HASH JOIN
HASH JOIN
INDEX
CARSNG.FK_PROD_IDX_4 FAST FULL SCAN 8236 20
INDEX
CARSNG.IE_PROD_IDX_1 FAST FULL SCAN 8236 33
INDEX
CARSNG.PK_PROD FAST FULL SCAN 8236 24
HASH JOIN
639094333 26104
INDEX
CARSNG.IE_CPPT_IDX_3 FAST FULL SCAN 1254629 2473
NESTED LOOPS
634106 17709
HASH JOIN
2580 2212
VIEW
CARSNG.index$_join$_014 24 2
HASH JOIN
INDEX
CARSNG.AK_WHOAMI_IDX_1 FAST FULL SCAN 24 1
INDEX
CARSNG.PK_WHOAMI FAST FULL SCAN 24 1
HASH JOIN
2580 2210
HASH JOIN
2589 2161
VIEW
2589 1690
HASH
GROUP BY 2589 1690
NESTED LOOPS
2589 1689
NESTED LOOPS
5874 1689
VIEW
SYS.VW_GBF_18 89 626
HASH
GROUP BY 89 626
HASH JOIN
SEMI 1963 625
TABLE ACCESS
CARSNG.CPGRP FULL 1970 591
VIEW
CARSNG.index$_join$_003 6415 34
HASH JOIN
INDEX
CARSNG.FK_CONT_IDX_3 FAST FULL SCAN 6415 18
INDEX
CARSNG.AK_CONT_IDX_1 FAST FULL SCAN 6415 25
INDEX
CARSNG.IE_CPPT_IDX_2 RANGE SCAN 66 2
TABLE ACCESS
CARSNG.CPPT BY INDEX ROWID 29 12
VIEW
CARSNG.index$_join$_013 43365 471
HASH JOIN
HASH JOIN
INDEX
CARSNG.PK_CPGRP FAST FULL SCAN 43365 114
INDEX
CARSNG.AK_CPGRP_IDX_4 FAST FULL SCAN 43365 192
INDEX
CARSNG.IE_CPGRP_IDX_3 FAST FULL SCAN 43365 168
VIEW
CARSNG.index$_join$_012 6415 49
HASH JOIN
INDEX
CARSNG.FK_CONT_IDX_3 FAST FULL SCAN 6415 18
INDEX
CARSNG.AK_CONT_IDX_3 FAST FULL SCAN 6415 44
INDEX
CARSNG.IE_ELIG_IDX_1 RANGE SCAN 246 6
I actually just figured this out but maybe my answer can help someone debugging in the future. I converted the DB2 selections into an inline view and that helped some. But where I was going wrong was through SELECT DISTINCT rather than SELECT, changing this saved an unreal amount of time. There was also one table that was being referenced but not matched, so I removed that.
Use a select ... from ... group by <column_names>
instead of using distinct for selecting columns use group by for them...
In oracle Is there any way to determine howlong the sql query will take to fetch the entire records and what will be the size of it, Without actually executing and waiting for entire result.
I am getting repeatedly to download and provide the data to the users using normal oracle SQL select (not datapump/import etc) . Some times rows will be in millions.
Actual run time will not known unless you run it, but you can try to estimate it..
first you can do explain plan explain only, this will NOT run query -- based on your current stats it will show you more or less how it will be executed
this will not have actual time and efforts to read the data from datablocks..
do you have large blocksize
is this schema normalized/de-normalized for query/reporting?
how large is row does it fit in same block so only 1 fetch is needed?
of rows you are expecting
based on amount of data * your network latency
Based on this you can try estimate time
This requires good statistics, explain plan for ..., adjusting sys.aux_stats, and then adjusting your expectations.
Good statistics The explain plan estimates are based on optimizer statistics. Make sure that tables and indexes have up-to-date statistics. On 11g this usually means sticking with the default settings and tasks, and only manually gathering statistics after large data loads.
Explain plan for ... Use a statement like this to create and store the explain plan for any SQL statement. This even works for creating indexes and tables.
explain plan set statement_id = 'SOME_UNIQUE_STRING' for
select * from dba_tables cross join dba_tables;
This is usually the best way to visualize an explain plan:
select * from table(dbms_xplan.display);
Plan hash value: 2788227900
-------------------------------------------------------------------
| Id | Operation | Name | Rows | Bytes | Time |
-------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 12M| 5452M| 00:00:19 |
|* 1 | HASH JOIN RIGHT OUTER | | 12M| 5452M| 00:00:19 |
| 2 | TABLE ACCESS FULL | SEG$ | 7116 | 319K| 00:00:01 |
...
The raw data is stored in PLAN_TABLE. The first row of the plan usually sums up the estimates for the other steps:
select cardinality, bytes, time
from plan_table
where statement_id = 'SOME_UNIQUE_STRING'
and id = 0;
CARDINALITY BYTES TIME
12934699 5717136958 19
Adjust sys.aux_stats$ The time estimate is based on system statistics stored in sys.aux_stats. These are numbers for metrics like CPU speed, single-block I/O read time, etc. For example, on my system:
select * from sys.aux_stats$ order by sname
SNAME PNAME PVAL1 PVAL2
SYSSTATS_INFO DSTART 09-11-2014 11:18
SYSSTATS_INFO DSTOP 09-11-2014 11:18
SYSSTATS_INFO FLAGS 1
SYSSTATS_INFO STATUS COMPLETED
SYSSTATS_MAIN CPUSPEED
SYSSTATS_MAIN CPUSPEEDNW 3201.10192837466
SYSSTATS_MAIN IOSEEKTIM 10
SYSSTATS_MAIN IOTFRSPEED 4096
SYSSTATS_MAIN MAXTHR
SYSSTATS_MAIN MBRC
SYSSTATS_MAIN MREADTIM
SYSSTATS_MAIN SLAVETHR
SYSSTATS_MAIN SREADTIM
The numbers can be are automatically gathered by dbms_stats.gather_system_stats. They can also be manually modified. It's a SYS table but relatively safe to modify. Create some sample queries, compare the estimated time with the actual time, and adjust the numbers until they match.
Discover you probably wasted a lot of time
Predicting run time is theoretically impossible to get right in all cases, and in practice it is horribly difficult to forecast for non-trivial queries. Jonathan Lewis wrote a whole book about those predictions, and that book only covers the "basics".
Complex explain plans are typically "good enough" if the estimates are off by one or two orders of magnitude. But that kind of difference is typically not good enough to show to a user, or use for making any important decisions.
I want to know the difference of those operators, mainly their performance difference.
I have had a look at Difference between <> and != in SQL, it has no performance related information.
Then I found this on dba-oracle.com,
it suggests that in 10.2 onwards the performance can be quite different.
I wonder why? does != always perform better then <>?
NOTE: Our tests, and performance on the live system shows, changing from <> to != has a big impact on the time the queries return in. I am here to ask WHY this is happening, not whether they are same or not. I know semantically they are, but in reality they are different.
I have tested the performance of the different syntax for the not equal operator in Oracle. I have tried to eliminate all outside influence to the test.
I am using an 11.2.0.3 database. No other sessions are connected and the database was restarted before commencing the tests.
A schema was created with a single table and a sequence for the primary key
CREATE TABLE loadtest.load_test (
id NUMBER NOT NULL,
a VARCHAR2(1) NOT NULL,
n NUMBER(2) NOT NULL,
t TIMESTAMP NOT NULL
);
CREATE SEQUENCE loadtest.load_test_seq
START WITH 0
MINVALUE 0;
The table was indexed to improve the performance of the query.
ALTER TABLE loadtest.load_test
ADD CONSTRAINT pk_load_test
PRIMARY KEY (id)
USING INDEX;
CREATE INDEX loadtest.load_test_i1
ON loadtest.load_test (a, n);
Ten million rows were added to the table using the sequence, SYSDATE for the timestamp and random data via DBMS_RANDOM (A-Z) and (0-99) for the other two fields.
SELECT COUNT(*) FROM load_test;
COUNT(*)
----------
10000000
1 row selected.
The schema was analysed to provide good statistics.
EXEC DBMS_STATS.GATHER_SCHEMA_STATS(ownname => 'LOADTEST', estimate_percent => NULL, cascade => TRUE);
The three simple queries are:-
SELECT a, COUNT(*) FROM load_test WHERE n <> 5 GROUP BY a ORDER BY a;
SELECT a, COUNT(*) FROM load_test WHERE n != 5 GROUP BY a ORDER BY a;
SELECT a, COUNT(*) FROM load_test WHERE n ^= 5 GROUP BY a ORDER BY a;
These are exactly the same with the exception of the syntax for the not equals operator (not just <> and != but also ^= )
First each query is run without collecting the result in order to eliminate the effect of caching.
Next timing and autotrace were switched on to gather both the actual run time of the query and the execution plan.
SET TIMING ON
SET AUTOTRACE TRACE
Now the queries are run in turn. First up is <>
> SELECT a, COUNT(*) FROM load_test WHERE n <> 5 GROUP BY a ORDER BY a;
26 rows selected.
Elapsed: 00:00:02.12
Execution Plan
----------------------------------------------------------
Plan hash value: 2978325580
--------------------------------------------------------------------------------------
| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time |
--------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 26 | 130 | 6626 (9)| 00:01:20 |
| 1 | SORT GROUP BY | | 26 | 130 | 6626 (9)| 00:01:20 |
|* 2 | INDEX FAST FULL SCAN| LOAD_TEST_I1 | 9898K| 47M| 6132 (2)| 00:01:14 |
--------------------------------------------------------------------------------------
Predicate Information (identified by operation id):
---------------------------------------------------
2 - filter("N"<>5)
Statistics
----------------------------------------------------------
0 recursive calls
0 db block gets
22376 consistent gets
22353 physical reads
0 redo size
751 bytes sent via SQL*Net to client
459 bytes received via SQL*Net from client
3 SQL*Net roundtrips to/from client
1 sorts (memory)
0 sorts (disk)
26 rows processed
Next !=
> SELECT a, COUNT(*) FROM load_test WHERE n != 5 GROUP BY a ORDER BY a;
26 rows selected.
Elapsed: 00:00:02.13
Execution Plan
----------------------------------------------------------
Plan hash value: 2978325580
--------------------------------------------------------------------------------------
| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time |
--------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 26 | 130 | 6626 (9)| 00:01:20 |
| 1 | SORT GROUP BY | | 26 | 130 | 6626 (9)| 00:01:20 |
|* 2 | INDEX FAST FULL SCAN| LOAD_TEST_I1 | 9898K| 47M| 6132 (2)| 00:01:14 |
--------------------------------------------------------------------------------------
Predicate Information (identified by operation id):
---------------------------------------------------
2 - filter("N"<>5)
Statistics
----------------------------------------------------------
0 recursive calls
0 db block gets
22376 consistent gets
22353 physical reads
0 redo size
751 bytes sent via SQL*Net to client
459 bytes received via SQL*Net from client
3 SQL*Net roundtrips to/from client
1 sorts (memory)
0 sorts (disk)
26 rows processed
Lastly ^=
> SELECT a, COUNT(*) FROM load_test WHERE n ^= 5 GROUP BY a ORDER BY a;
26 rows selected.
Elapsed: 00:00:02.10
Execution Plan
----------------------------------------------------------
Plan hash value: 2978325580
--------------------------------------------------------------------------------------
| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time |
--------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 26 | 130 | 6626 (9)| 00:01:20 |
| 1 | SORT GROUP BY | | 26 | 130 | 6626 (9)| 00:01:20 |
|* 2 | INDEX FAST FULL SCAN| LOAD_TEST_I1 | 9898K| 47M| 6132 (2)| 00:01:14 |
--------------------------------------------------------------------------------------
Predicate Information (identified by operation id):
---------------------------------------------------
2 - filter("N"<>5)
Statistics
----------------------------------------------------------
0 recursive calls
0 db block gets
22376 consistent gets
22353 physical reads
0 redo size
751 bytes sent via SQL*Net to client
459 bytes received via SQL*Net from client
3 SQL*Net roundtrips to/from client
1 sorts (memory)
0 sorts (disk)
26 rows processed
The execution plan for the three queries is identical and the timings 2.12, 2.13 and 2.10 seconds.
It should be noted that whichever syntax is used in the query the execution plan always displays <>
The tests were repeated ten times for each operator syntax. These are the timings:-
<>
2.09
2.13
2.12
2.10
2.07
2.09
2.10
2.13
2.13
2.10
!=
2.09
2.10
2.12
2.10
2.15
2.10
2.12
2.10
2.10
2.12
^=
2.09
2.16
2.10
2.09
2.07
2.16
2.12
2.12
2.09
2.07
Whilst there is some variance of a few hundredths of the second it is not significant. The results for each of the three syntax choices are the same.
The syntax choices are parsed, optimised and are returned with the same effort in the same time. There is therefore no perceivable benefit from using one over another in this test.
"Ah BC", you say, "in my tests I believe there is a real difference and you can not prove it otherwise".
Yes, I say, that is perfectly true. You have not shown your tests, query, data or results. So I have nothing to say about your results. I have shown that, with all other things being equal, it doesn't matter which syntax you use.
"So why do I see that one is better in my tests?"
Good question. There a several possibilities:-
Your testing is flawed (you did not eliminate outside factors -
other workload, caching etc You have given no information about
which we can make an informed decision)
Your query is a special case (show me the query and we can discuss it).
Your data is a special case (Perhaps - but how - we don't see that either).
There is some other outside influence.
I have shown via a documented and repeatable process that there is no benefit to using one syntax over another. I believe that <> != and ^= are synonymous.
If you believe otherwise fine, so
a) show a documented example that I can try myself
and
b) use the syntax which you think is best. If I am correct and there is no difference it won't matter. If you are correct then cool, you have an improvement for very little work.
"But Burleson said it was better and I trust him more than you, Faroult, Lewis, Kyte and all those other bums."
Did he say it was better? I don't think so. He didn't provide any definitive example, test or result but only linked to someone saying that != was better and then quoted some of their post.
Show don't tell.
You reference the article on the Burleson site. Did you follow the link to the Oracle-L archive? And did you read the other emails replying to the email Burleson cites?
I don't think you did, otherwise you wouldn't have asked this question. Because there is no fundamental difference between != and <>. The original observation was almost certainly a fluke brought about by ambient conditions in the database. Read the responses from Jonathan Lewis and Stephane Faroult to understand more.
" Respect is not something a programmer need to have, its the basic
attitude any human being should have"
Up to a point. When we meet a stranger in the street then of course we should be courteous and treat them with respect.
But if that stranger wants me to design my database application in a specific way to "improve performance" then they should have a convincing explanation and some bulletproof test cases to back it up. An isolated anecdote from some random individual is not enough.
The writer of the article, although a book author and the purveyor of some useful information, does not have a good reputation for accuracy. In this case the article was merely a mention of one persons observations on a well known Oracle mailing list. If you read through the responses you will see the assumptions of the post challenged, but no presumption of accuracy. Here are some excerpts:
Try running your query through explain plan (or autotrace) and see
what that says...
According to this, "!=" is considered to be the same as "<>"...
Jonathan Lewis
Jonathan Lewis is a well respected expert in the Oracle community.
Just out of curiosity... Does the query optimizer generate a different
execution plan for the two queries? Regards, Chris
.
Might it be bind variable peeking in action? The certain effect of
writing != instead of <> is to force a re-parse. If at the first
execution the values for :id were different and if you have an
histogram on claws_doc_id it could be a reason. And if you tell me
that claws_doc_id is the primary key, then I'll ask you what is the
purpose of counting, in particular when the query in the EXISTS clause
is uncorrelated with the outer query and will return the same result
whatever :id is. Looks like a polling query. The code surrounding it
must be interesting.
Stéphane Faroult
.
I'm pretty sure the lexical parse converts either != to <> or <> to
!=, but I'm not sure whether that affects whether the sql text will
match a stored outline.
.
Do the explain plans look the same? Same costs?
The following response is from the original poster.
Jonathan, Thank you for your answer. We did do an explain plan on
both versions of the statement and they were identical, which is what
is so puzzling about this. According to the documentation, the two
forms of not equal are the same (along with ^= and one other that I
can't type), so it makes no sense to me why there is any difference in
performance.
Scott Canaan
.
Not an all inclusive little test but it appears at least in 10.1.0.2
it gets pared into a "<>" for either (notice the filter line for each
plan)
.
Do you have any Stored Outline ? Stored Outlines do exact (literal)
matches so if you have one Stored Outline for, say, the SQL with a
"!=" and none for the SQL with a "<>" (or a vice versa), the Stored
Outline might be using hints ? (although, come to think of it, your
EXPLAIN PLAN should have shown the hints if executing a Stored Outline
?)
.
Have you tried going beyond just explain & autotrace and running a
full 10046 level 12 trace to see where the slower version is spending
its time? This might shed some light on the subject, plus - be sure
to verify that the explain plans are exactly the same in the 10046
trace file (not the ones generated with the EXPLAIN= option), and in
v$sqlplan. There are some "features" of autotrace and explain that
can cause it to not give you an accurate explain plan.
Regards, Brandon
.
Is the phenomenon totally reproducible ?
Did you check the filter_predicates and access_predicates of the plan,
or just the structure. I don't expect any difference, but a change in
predicate order can result in a significant change in CPU usage if you
are unlucky.
If there is no difference there, then enable rowsource statistics
(alter session set "_rowsource_execution_statistics"=true) and run the
queries, then grab the execution plan from V$sql_plan and join to
v$sql_plan_statistics to see if any of the figures about last_starts,
last_XXX_buffer_gets, last_disk_reads, last_elapsed_time give you a
clue about where the time went.
If you are on 10gR2 there is a /*+ gather_plan_statistics */ hint you
can use instead of the "alter session".
Regards Jonathan Lewis
At this point the thread dies and we see no further posts from the original poster, which leads me to believe that either the OP discovered an assumption they had made that was not true or did no further investigation.
I will also point out that if you do an explain plan or autotrace, you will see that the comparison is always displayed as <>.
Here is some test code. Increase the number of loop iterations if you like. You may see one side or the other get a higher number depending on the other activity on the server activity, but in no way will you see one operator come out consistently better than the other.
DROP TABLE t1;
DROP TABLE t2;
CREATE TABLE t1 AS (SELECT level c1 FROM dual CONNECT BY level <=144000);
CREATE TABLE t2 AS (SELECT level c1 FROM dual CONNECT BY level <=144000);
SET SERVEROUTPUT ON FORMAT WRAPPED
DECLARE
vStart Date;
vTotalA Number(10) := 0;
vTotalB Number(10) := 0;
vResult Number(10);
BEGIN
For vLoop In 1..10 Loop
vStart := sysdate;
For vLoop2 In 1..2000 Loop
SELECT count(*) INTO vResult FROM t1 WHERE t1.c1 = 777 AND EXISTS
(SELECT 1 FROM t2 WHERE t2.c1 <> 0);
End Loop;
vTotalA := vTotalA + ((sysdate - vStart)*24*60*60);
vStart := sysdate;
For vLoop2 In 1..2000 Loop
SELECT count(*) INTO vResult FROM t1 WHERE t1.c1 = 777 AND EXISTS
(SELECT 1 FROM t2 WHERE t2.c1 != 0);
End Loop;
vTotalB := vTotalB + ((sysdate - vStart)*24*60*60);
DBMS_Output.Put_Line('Total <>: ' || RPAD(vTotalA,8) || '!=: ' || vTotalB);
vTotalA := 0;
vTotalB := 0;
End Loop;
END;
A Programmer will use !=
A DBA will use <>
If there is a different execution plan it may be that there are differences in the query cache or statistics for each notation. But I don't really think it is so.
Edit:
What I mean above. In complex databases there can be some strange side effects. I don't know oracle good enough, but I think there is an Query Compilation Cache like in SQL Server 2008 R2.
If a query is compiled as new query, the database optimiser calculates a new execution plan depending on the current statistics. If the statistics has changed it will result in a other, may be a worse plan.
Guys i have the following oracle sql query that gives me the monthwise report between the dates.Basically for nov month i want sum of values between the dates 01nov to 30 nov.
The table that is being queried is residing in another database and accesssed using dblink. The DT columns is of NUMBER type (for ex 20101201).
SELECT /*+ PARALLEL (A 8) */ /*+ DRIVING_STATE(A) */
TO_CHAR(TRUNC(TRUNC(SYSDATE,'MM')- 1,'MM'),'MONYYYY') "MONTH",
TYPE AS "TYPE", COLUMN, COUNT (DISTINCT A) AS "A_COUNT",
COUNT (COLUMN) AS NO_OF_COLS, SUM (DURATION) AS "SUM_DURATION",
SUM (COST) AS "COST" FROM **A#LN_PROD A**
WHERE DT >= TO_NUMBER(TO_CHAR(add_months(SYSDATE,-1),'YYYYMM"01"'))
AND DT < TO_NUMBER(TO_CHAR(SYSDATE,'YYYYMM"01"'))
GROUP BY TYPE, COLUMN
The execution of the query is taking a day long and not completed. kindly suggest me , if their is any optimisation that can be suggested to my DBA on the dblink, or any tuning that can be done on the query , or rewriting the same.
UPDATES ON THE TABLE
The table is partiontioned on the date column and almost 1 billion records.
Below i have given the EXPLAIN PLAN from TOAD
**Plan**
SELECT STATEMENT REMOTE ALL_ROWSCost: 1,208,299 Bytes: 34,760 Cardinality: 790
12 PX COORDINATOR
11 PX SEND QC (RANDOM) SYS.:TQ10002 Cost: 1,208,299 Bytes: 34,760 Cardinality: 790
10 SORT GROUP BY Cost: 1,208,299 Bytes: 34,760 Cardinality: 790
9 PX RECEIVE Cost: 1,208,299 Bytes: 34,760 Cardinality: 790
8 PX SEND HASH SYS.:TQ10001 Cost: 1,208,299 Bytes: 34,760 Cardinality: 790
7 SORT GROUP BY Cost: 1,208,299 Bytes: 34,760 Cardinality: 790
6 PX RECEIVE Cost: 1,208,299 Bytes: 34,760 Cardinality: 790
5 PX SEND HASH SYS.:TQ10000 Cost: 1,208,299 Bytes: 34,760 Cardinality: 790
4 SORT GROUP BY Cost: 1,208,299 Bytes: 34,760 Cardinality: 790
3 FILTER
2 PX BLOCK ITERATOR Cost: 1,203,067 Bytes: 15,066,833,144 Cardinality: 342,428,026 Partition #: 11 Partitions accessed #1 - #5
1 TABLE ACCESS FULL TABLE CDRR.FRD_CDF_DATA_INTL_IN_P Cost: 1,203,067 Bytes: 15,066,833,144 Cardinality: 342,428,026 Partition #: 11
The following things i am going to do today ,any additional tips would be helpful.
I am going to gather the tablewise statistics for this table, which may give optimal
execution plan.
Check whether an local index is created for the partition .
using BETWEEN instead of >= and <.
As usual for this type of question, an explain plan would be useful. It would help us work out what is actually going on in the database.
Ideally you want to make sure the query is running on the remote database the sending the result set back, rather than sending the data across the link and running the query locally. This ensures that less data is sent across the link. The DRIVING_SITE hint can help with this, although Oracle is usually fairly smart about it so it might not help at all.
Oracle seems to have got better at running remote queries but there still can be problems.
Also, it might pay to simplify some of your date conversions.
For example, replace this:
TO_CHAR(TRUNC(TRUNC(SYSDATE,'MM')- 1,'MM'),'MONYYYY')
with this:
TO_CHAR(add_months(TRUNC(SYSDATE,'MM'), -1),'MONYYYY')
It is probably slightly more efficient but also is easier to read.
Likewise replace this:
WHERE DT >=TO_NUMBER(TO_CHAR(TRUNC(TRUNC(SYSDATE,'MM')-1,'MM'),'YYYYMMDD'))
AND DT < TO_NUMBER(TO_CHAR(TRUNC(TRUNC(SYSDATE,'MM'),'MM'),'YYYYMMDD'))
with
WHERE DT >=TO_NUMBER(TO_CHAR(add_months(TRUNC(SYSDATE,'MM'), -1),'YYYYMMDD'))
AND DT < TO_NUMBER(TO_CHAR(TRUNC(SYSDATE,'MM'),'YYYYMMDD'))
or even
WHERE DT >=TO_NUMBER(TO_CHAR(add_months(SYSDATE,-1),'YYYYMM"01"'))
AND DT < TO_NUMBER(TO_CHAR(SYSDATE,'YYYYMM"01"'))
It may be because several issues:
1.Network speed because the database may be residing on different hardware.
However you can refer this link
http://www.experts-exchange.com/Database/Oracle/Q_21799513.html.
There is a similar issue.
Impossible to answer without knowing the table structure, constraints, indexes, data volume, resultset size, network speed, level of concurrency, execution plans etcetera.
Some things I would investigate:
If the table is partitioned, does statistics exist for the partition the query is hitting? A common problem is that statistics are gathered on an empty partition before data has been inserted. Then when you query it (before the statistics are refreshed) Oracle chooses an index scan, when in fact it should use an FTS on that partition.
Also related to statistics: Make sure that
WHERE DT >=TO_NUMBER(TO_CHAR(TRUNC(TRUNC(SYSDATE,'MM')-1,'MM'),'YYYYMMDD'))
AND DT < TO_NUMBER(TO_CHAR(TRUNC(TRUNC(SYSDATE,'MM'),'MM'),'YYYYMMDD'))
generates the same execution plan as:
WHERE DT >= 20101201
AND DT < 20110101
Updated
What version of Oracle are you on? The reason I'm asking is that on Oracle 10g and later, there is another implementation of group by that should have been selected in this case (hashing rather than sorting). It looks like you are basically sorting the 342 million rows returned from the date filter (14 gigabytes). Do you have the RAM to back that up? Otherwise you will be doing a multipass sort, spilling to disk. This is likely what is happening.
According to the plan, about 790 rows will be returned. Is that in the right ballpark?
If so, you can rule out network issues :)
Also, I'm not entirely familiar with the format on that plan. Is the table sub partitioned? Otherwise I don't get the partition #11 reference.
First of all, I am new to optimizing mysql. The fact is that I have in my web application (around 400 queries per second), a query that uses a GROUP BY that i can´t avoid and that is the cause of creating temporary tables. My configuration was:
max_heap_table_size = 16M
tmp_table_size = 32M
The result: temp table to disk percent + - 12.5%
Then I changed my settings, according to this post
max_heap_table_size = 128M
tmp_table_size = 128M
The result: temp table to disk percent + - 18%
The results were not expected, do not understand why.
It is wrong tmp_table_size = max_heap_table_size?
Should not increase the size?
Query
SELECT images, id
FROM classifieds_ads
WHERE parent_category = '1' AND published='1' AND outdated='0'
GROUP BY aux_order
ORDER BY date_lastmodified DESC
LIMIT 0, 100;
EXPLAIN
| 1 |SIMPLE|classifieds_ads | ref |parent_category, published, combined_parent_oudated_published, oudated | combined_parent_oudated_published | 7 | const,const,const | 67552 | Using where; Using temporary; Using filesort |
"Using temporary" in the EXPLAIN report does not tell us that the temp table was on disk. It only tells us that the query expects to create a temp table.
The temp table will stay in memory if its size is less than tmp_table_size and less than max_heap_table_size.
Max_heap_table_size is the largest a table can be in the MEMORY storage engine, whether that table is a temp table or non-temp table.
Tmp_table_size is the largest a table can be in memory when it is created automatically by a query. But this can't be larger than max_heap_table_size anyway. So there's no benefit to setting tmp_table_size greater than max_heap_table_size. It's common to set these two config variables to the same value.
You can monitor how many temp tables were created, and how many on disk like this:
mysql> show global status like 'Created%';
+-------------------------+-------+
| Variable_name | Value |
+-------------------------+-------+
| Created_tmp_disk_tables | 20 |
| Created_tmp_files | 6 |
| Created_tmp_tables | 43 |
+-------------------------+-------+
Note in this example, 43 temp tables were created, but only 20 of those were on disk.
When you increase the limits of tmp_table_size and max_heap_table_size, you allow larger temp tables to exist in memory.
You may ask, how large do you need to make it? You don't necessarily need to make it large enough for every single temp table to fit in memory. You might want 95% of your temp tables to fit in memory and only the remaining rare tables go on disk. Those last 5% might be very large -- a lot larger than the amount of memory you want to use for that.
So my practice is to increase tmp_table_size and max_heap_table_size conservatively. Then watch the ratio of Created_tmp_disk_tables to Created_tmp_tables to see if I have met my goal of making 95% of them stay in memory (or whatever ratio I want to see).
Unfortunately, MySQL doesn't have a good way to tell you exactly how large the temp tables were. That will vary per query, so the status variables can't show that, they can only show you a count of how many times it has occurred. And EXPLAIN doesn't actually execute the query so it can't predict exactly how much data it will match.
An alternative is Percona Server, which is a distribution of MySQL with improvements. One of these is to log extra information in the slow-query log. Included in the extra fields is the size of any temp tables created by a given query.