I have a large SQL query for a report with many optional parameters and 20-30 branches in the WHERE clause. The problem is its running very slow. The execution time varies from 6 secs to 5 mins. If I remove one or two conditions it works in 6 secs but with all conditions it takes over 5 mins. The problem is not with any specific indexes or a condition because if I remove ANY ONE condition from the query it improves. There are about 20 tables in the join and about 100 fields in the output. Problem area of the query looks something like this :
AND Abc_ID IN (SELECT * FROM fnSplit(#Abc_IDs, ','))
AND Xyx_ID IN (SELECT * FROM fnSplit(#Xyz_Ids, ','))
AND Aaa_ID IN (SELECT * FROM fnSplit(#Aaa_Ids, ','))
AND Zzz_ID IN (SELECT * FROM fnSplit(#Zzz_Ids, ','))
where fnSplit is a udf. Also note if I remove fnsplit with hardcoded value (1,2,3,4,5) it also improves. But my guess is the issue is to do with the server memory or some configuration rather than in an specific clause in the WHERE. As I said earlier there are 20-30 branches in the WHERE clause and removing any one condition improves the performance (a lot).
Any thoughts?
Few initial things / ideas that you could try:
Check the "Reason for Early Termination" from the leftmost object in the execution plan. If it is timeout, that could be the reason for a (horribly) bad plan
Test if replacing the (SELECT * FROM fnSplit..) with a temp. tables would help the optimizer to understand the cardinality better
Look at "statistics io", the execution plans and plan cache what is consuming the most I/O and CPU, that might help to understand where the problem is
Change the whole thing to be a dynamic SQL
Include the execution plan, statistics io and table & indexing structure etc into the question for further analysis
Bonus idea: Compare your split function to DelimitedSplit8k, maybe that would be better.
Related
If we look at a simple query like this one:
SELECT * FROM CUSTOMER2;
We can tell by looking at it that it simply does one thing, retrieves everything from CUSTOMER2.
Now my question is, why is it that when we run it like this:
SELECT/*+ PARALLEL(CUSTOMER2, 8) */ * FROM CUSTOMER2;
The cost of it (according to execution plan) goes from 581 to 81? Since its only one task, isn't it just performed on the same thread anyway?
I can understand if there were two full table scans needing to be done as you can run those two in parallel threads so they execute at the same time. But in our case, there is only one full table scan.
So how does running it in parallel make it faster when there is nothing to run it "in parallel" with?
Lastly, when I altered my personal cluster and the one table to run in parallel when anything is performed on it I did not see any change in cost like I did with the small statement.
This is my personal one:
SELECT AVG(s.sellprice), s.qty, s.custid
FROM CUSTOMER_saracl c, sale_saracl s
WHERE c.custid = s.custid
GROUP BY (s.qty, s.custid)
HAVING AVG(s.sellprice) >
(SELECT MIN(AVG(price))
FROM product_saracl
WHERE pname
LIKE 'FA%'
GROUP BY price);
Why would that be?
Thank you for any help, I just today learnt about parallel execution so go easy on me haha!
One very important point about relational databases is that tables represent unordered sets. That means that the pages that are scanned for a table can be scanned in any order.
Oracle actually takes advantage of this for parallel scans of a single table. There is additional overhead to bring the results back together, which is why the estimated cost is 81 and not 73 (581 / 8).
I think this documentation has good examples that explain this. Some are quite close to your query.
Note that parallelism does not just apply to reading tables. In fact, it is more commonly associated with other operations, such as joins, aggregation, and sorting.
I have a select query with some complex joins and where conditions and it takes ~9seconds to execute.
Now, the strange thing is if I wrap the query with select count(1) the execution time will increase dramatically.
SELECT COUNT(1) FROM
(
SELECT .... -- initial query, executes ~9s
)
-- executes 1min
That's very strange to me, since I would expect an opposite result - the sql-server engine should be smart enough to optimize the inner query execution (for instance, do not execute nested queries in the select clause, etc).
And that's what execution plans comparison shows! It says it should be 74% to 26% (the former is initial query and latter is wrapped with select count(1)).
But that's not what really happens.
Idk if I should post the query itself, since it's rather large (if you need it then just let me know in comments).
Thaks you!)
When you use count(1) you no longer need all the columns.
This means that SQL Server can consider different execution plans using narrower indexes that do not cover all the columns used in the SELECT list of the original query.
Generally this should of course lead to a leaner, faster, execution plan however looks like in this case you were unlucky and it didn't.
Probably you will find a node with a large discrepancy between actual and estimated rows - this kind of thing will propagate up in the plan and can lead to sub optimal choices of strategy for other sub trees (e.g. sub optimal join orderings or algorithms)
Below is my query, I use four joins to access data from three different tables, now when searching for 1000 records it takes around 5.5 seconds, but when I amp it up to 100,000 it takes what seems like an infinite amount of time, (last cancelled at 7 hours..)
Does anyone have any idea of what I am doing wrong? Or what could be done to speed up the query?
This query will proabably end up having to be run to return millions of records, I've only limited it to 100,000 for the purpose of testing the query and it seems to fall over at even this small amount.
For the record im on oracle 8
CREATE TABLE co_tenancyind_batch01 AS
SELECT /*+ CHOOSE */ ou_num,
x_addr_relat,
x_mastership_flag,
x_ten_3rd_party_source
FROM s_org_ext,
s_con_addr,
s_per_org_unit,
s_contact
WHERE s_org_ext.row_id = s_con_addr.accnt_id
AND s_org_ext.row_id = s_per_org_unit.ou_id
AND s_per_org_unit.per_id = s_contact.row_id
AND x_addr_relat IS NOT NULL
AND rownum < 100000
Explain Plan in Picture : http://imgur.com/Xw9x4BA (easy to read)
Your test based on 100,000 rows is not meaningful if you are then going to run it for many millions. The optimiser knows that it can satisfy the query faster when it has a stopkey by using nested loop joins.
When you run it for a very large data set you're likely to need a different plan, with hash joins most likely. Covering indexes might help with that, but we can't tell because the selected columns are missing column aliases that tell us which table they come from. You're most likely to hit memory problems with large hash joins, which could be ameliorated with hash partitioning but there's no way the Siebel people would go for that -- you'll have to use manual memory management and monitor v$sql_workarea to see how much you really need.
(Hate the visual explain plan, by the way).
First of all, can you make sure there is an index on S_CONTACT table and it is enabled ?
If it is so, try the select statement with /*+ CHOOSE */ hint and have another look at the explain plan to see if optimizer mode is still RULE. I believe cost based optimizer would result better in this query.
If still rule try updating database statistics and try again. You can use DBMS_STATS package for that purpose, if i am not wrong it was introduced with version 8i. Are you using 8i ?
And at last, i don't know the record numbers, the cardinality between tables. I might have been more helpful if i knew the design.
Your dataset, looking at the last execution plan appear to be huge, you could limit access to the base table instead of limiting the number of returned row, like this:
CREATE TABLE co_tenancyind_batch01 AS
SELECT /*+ CHOOSE */ ou_num,
x_addr_relat,
x_mastership_flag,
x_ten_3rd_party_source
FROM s_org_ext,
s_con_addr,
s_per_org_unit,
(select * from s_contact where rownum <= 100000) cont
WHERE s_org_ext.row_id = s_con_addr.accnt_id
AND s_org_ext.row_id = s_per_org_unit.ou_id
AND s_per_org_unit.per_id = cont.row_id
AND x_addr_relat IS NOT NULL
should improve but not be extremely quick.
We have an interesting problem that I was hoping someone could help to shed some light on. At a high level the problem is as below:
The following query executes quickly (1 second):
SELECT SA.*
FROM cg.SEARCHSERVER_ACTYS AS SA
JOIN CONTAINSTABLE(CG.SEARCHSERVER_ACTYS, NOTE, 'reports') AS T1 ON T1.[Key]=SA.UNIQUE_ID
but if we add a filter to the query, then it takes approximately 2 minutes to return:
SELECT SA.*
FROM cg.SEARCHSERVER_ACTYS AS SA
JOIN CONTAINSTABLE(CG.SEARCHSERVER_ACTYS, NOTE, 'reports') AS T1 ON T1.[Key]=SA.UNIQUE_ID
WHERE SA.CHG_DATE>'19 Feb 2010'
Looking at the execution plan for the two queries, I can see that in the second case there are two places where there are huge differences between the actual and estimated number of rows, these being:
1) For the FulltextMatch table valued function where the estimate is approx 22,000 rows and the actual is 29 million rows (which are then filtered down to 1670 rows before the join) and
2) For the index seek on the full text index, where the estimate is 1 row and the actual is 13,000 rows
As a result of the estimates, the optimiser is choosing to use a nested loops join (since it assumes a small number of rows) hence the plan is inefficient.
We can work around the problem by either (a) parameterising the query and adding an OPTION (OPTIMIZE FOR UNKNOWN) to the query or (b) by forcing a HASH JOIN to be used. In both of these cases the query returns in sub 1 second and the estimates appear reasonable.
My question really is 'why are the estimates being used in the poorly performing case so wildly inaccurate and what can be done to improve them'?
Statistics are up to date on the indexes on the indexed view being used here.
Any help greatly appreciated.
The problem here turned out to be with the version of SQL Server. The problem manifested itself with SQL Server 2008 (no service pack) and was resolved by upgrading to SQL Server 2008 SP1 (and adding CU5). Since we did not test without CU5 installed I cannot determine if the fix came with SP1 or CU5. No matter, the issue is resolved. Morale? Keep your server up to date.
Perhaps you could add some statistics on the column in question - that will help SQL Server make better estimates about both the number of rows and their contents.
What statistics or indexes are currently involved?
I was reading over the documentation for query hints:
http://msdn.microsoft.com/en-us/library/ms181714(SQL.90).aspx
And noticed this:
FAST number_rows
Specifies that the query is optimized for fast retrieval of the first number_rows. This is a nonnegative integer. After the first number_rows are returned, the query continues execution and produces its full result set.
So when I'm doing a query like:
Select Name from Students where ID = 444
Should I bother with a hint like this? Assuming SQL Server 2005, when should I?
-- edit --
Also should one bother when limiting results:
Select top 10 * from Students OPTION (FAST 10)
The FAST hint only makes sense on complex queries where there are multiple alternatives the optimizer could choose from. For a simple query like your example it doesn't help with anything, the query optimizer will immediately determine that there is a trivial plan (seek in ID index, lookup Name if not covering) to satisfy the query and go for it. Even if no index exists on ID, the plan is still trivial (probably clustered scan).
To give an example where FAST would be useful consider a join between A and B, with an ORDER BY constraint. Say evaluating the join B first and nested loops A honors the ORDER BY constraint, so will produce fast results (no SORT necessary), but is more costly because of cardinality (B has many records that match the WHERE, while A has few). On the other hand evaluating B first and nested loop A would produce a query that does less IO hence is faster overall, but the result would have to be sorted first and SORT can only start after the join is evaluated, so the first result will come very late. The optimizer would normally pick the second plan because is more efficient overall. The FAST hint would cause the optimizer to pick the first plan, because it produces results faster.
When using TOP x, there's no benefit of also using OPTION FAST x. The query optimizer already makes its decisions based on how many rows you are retrieving. Same goes for trivial queries, such as querying for a particular value from a unique index.
Other than that, OPTION FAST x could help when you know the number of results is likely below x, but the query optimizer does not. Of course, if the query optimizer is choosing poor paths for complex queries with few results, your statistics may need to be updated. And if you guess wrong on x, the query may end up taking longer--almost always a risk when giving hints.
The above statement has not been tested--it may be that all queries take just as long to fully execute, if not longer. Getting the first 10 rows fast is great if there are only 8 rows, but theoretically the query still has to execute fully before finishing. The benefit I'm thinking may be there because the query execution takes a different path expecting fewer total records, when in fact it's really trying to get the first x faster. Those two types of optimizations may not be in alignment.
For that particular query, certainly not! It's only going to return one row — the row with ID = 444. SQL Server will select that row as efficiently as it can.
FAST 10 might be used in a situation where you could make use of the first 10 rows immediately, even as you continue to wait for further results.