I have a query that runs slow (around 5 minutes), but if I add this commented out lines, it runs very fast (about 2 seconds), which doesn't make sense at all.
What could be the explanation ?
FROM .....
--CREF.SYSTEM_PARAMETER sp,
...
WHERE ....
--t.LEDGER_EFFECTIVE_DATE BETWEEN NVL(adhoc.START_DATE, TRUNC(SYSDATE) - sp.value) and nvl(ec.END_DATE, TRUNC(SYSDATE)) -- use t.LEDGER_EFFECTIVE_DATE
Thank you.
One possibility is that the original query is cached with a poor execution plan, so every time the database sees that query it uses the cached plan. The amended version gets a fresh parse and a new execution plan.
Perhaps if you invalidated the slow version in some way (gather stats with noinvalidate => false, add an empty comment to the table, flush the shared pool, restart the database etc) it would get the new plan.
You mentioned "The explain plan for both are exactly the same" - the explain plan utility shows the plan likely to be used at the next execution. You need to check the actual runtime execution plan, not the explain plan.
Oracle creates a cache when the query is executed multiple times, which can result in reduced execution time. Are you executing the query multiple times?
Related
I am using postgres db version 13.5
From experience I am aware that EXPLAIN ANALYZE actually runs the query and gives the actual cost and the actual rows output.
And from postgres doc on ANALYZE command - I read below,
ANALYZE collects statistics about the contents of tables in the
database, and stores the results in the pg_statistic system catalog.
So I want to know if EXPLAIN ANALYZE also collects statistics about the contents of tables in the database, and stores the results in the pg_statistic system catalog.'
The manual for the EXPLAIN command states:
The ANALYZE option causes the statement to be actually executed, not only planned. Then actual run time statistics are added to the display, including the total elapsed time expended within each plan node (in milliseconds) and the total number of rows it actually returned. This is useful for seeing whether the planner's estimates are close to reality.
The manual only lists the things are are done by the commands. Typically things not listed there, are not done (which makes sense because there is whole lot more that this command does not do, then it does).
The part "whether the planner's estimates are close to reality" also indicates that EXPLAIN uses the information from pg_statistic (and other statistics tables in the system catalog), not that it updates them.
So it's pretty safe to say that explain (analyze) will not update the statistics of the involved tables.
This can only be achieved by running the analyze command or vacuum analyze (or waiting for the autovacuum daemon to analyze the tables).
Execution Plan Download Link: https://www.dropbox.com/s/3spvo46541bf6p1/Execution%20plan.xml?dl=0
I'm using SQL Server 2008 R2
I have a pretty complex stored procedure that's requesting way too much memory upon execution. Here's a screenshot of the execution plan:
http://s15.postimg.org/58ycuhyob/image.png
The underlying query probably needs a lot of tuning as indicated by massive number of estimated rows, but that's besides the point. Regardless of the complexity of the query, it should not be requesting 3 gigabytes of memory upon execution.
How do I prevent this behavior? I've tried the following:
DBCC FREEPROCCACHE to clear plan cache. This accomplished nothing.
Setting RECOMPILE option on both SP and SQL level. Again, this does nothing.
Messing around with MAXDOP option, from 0 to 8. Same issue.
The query returns about ~1k rows on average, and it does look into a table with more than 3 million rows with about 4 tables being joined. Executing the query returns the result in less than 3 seconds in majority of the cases.
Edit:
One more thing, using query hints is not really viable for this case since the parameters vary greatly for our case.
Edit2:
Uploaded execution plan upon request
Edit3:
I've tried rebuilding/reorganizing fragmented indices. Apparently, there were few but nothing too serious. Anyhow, this didn't reduce the amount of memory granted and didn't reduce the number of estimated rows (If this is somehow related).
You say optimizing the query is besides the point, but actually it's actually just the point. When a query is executed, SQL Server will -after generating the execution plan- reserve the memory needed for executing the query. The more rows intermediate results are estimated to hold the more memory is estimated to be required.
So, rewrite your query and/or create new indexes to get a decent query plan. A quick glance at the query plan shows some nested loops without join predicates and a number of table scans of which probably only a few records are used.
I have stored procedure which is building dynamic SQL statement depending on its input parameters and then executed it.
One of the queries is causing time outs, so I have decided to check it. The first time (and only the first time) the issue statement is executed it is slow (30 secs - 45 secs) and every next execute takes 1-2 seconds.
In order to reproduce the issue, I am using
DBCC FREEPROCCACHE
DBCC DROPCLEANBUFFERS
I am really confused where the problem is, because ordinary if SQL statement is slow, it is always slow. Now, it has long execution time only the first time.
Is is possible, the itself to be slow and needs optimization or the problem can be caused by something else?
The execution plan is below, but for me there is nothing strange with it:
From your reply to my comment, it would appear that the first time this query runs it is performing a lot of physical reads or read-ahead reads, meaning that a lot of IO is required to get the right pages into the buffer pool to satisfy this query.
Once pages are read into the buffer pool (memory) they generally stay there so that physical IO is not required to read them again (you can see this is happening as you indicated that the physical reads are converted to logical reads the second time the query is run). Memory is orders of magnitude faster than disk IO, hence the difference in speed for this query.
Looking at the plan, I can just about see that every read operation is being done against the clustered index of the table. As the clustered index contains every column for the row it is potentially fetching more data per row than is actually required for the query.
Unless you are selecting every column from every table, I would suggest that creating non-clustered covering indexes that satisfy this query (that are as narrow as possible), this will reduce the IO requirement for the query and make it less expensive the first time round.
Of course this may not be possible/viable for you to do, in which case you should either just take the hit on the first run and not empty the caches, or rewrite the query itself to be more efficient and perform less reads.
Its very simple the reason 1st and very 1st time it takes longer and then all later executions are done fairly quickly. the reason behind this mystery is "CACHED EXECUTION PLANS".
While working with Stored Procedures, Sql server takes the following
steps.
1) Parse Syntax of command. 2) Translate to Query Tree. 3) Develop
Execution Plan. 4) Execute.
The 1st two steps only take place when you create a Stored Procedure.
3rd step only takes place on very 1st Execution or if the CACHED PLAN has been flushed from the CACHE MEMORY.
Fourth Step takes place on every execution, and this is the only step that takes place after the very 1st execution if the Plan is still in cache memory.
In your case its quite understandable that very 1st execution took long and then later it gets executed fairly quickly.
To reproduce the "issue" you executed DBCC FREEPROCCACHE AND DBCC DROPCLEANBUFFERS commanda which basically Flushes the BUFFER CACHE MEMORY and causes your stored procedure to create a new Execution plan on it next execution. Hope this will clear the fog a little bit :)
Generally, when a Stored procedure is first created, or its statistics etc reset, it will take the first value passed into the Stored Procedure as the 'default' value for the stored procedure. It will then try to optimize itself based off of that.
To stop that from happening, there are a couple of things you can do.
You could potentially use the Query hints feature to mark certain variables as Unknown. So, as an example, at the end of the stored procedure you could put something along the lines of:
select * from foo where foo.bar = #myParam option (optimize for #myParam unknown)
As another approach, you could force the SQL plan to be re-compiled each time - which might be a good idea if your stored procedure is highly variable in the type of SQL it generates. The way you'd do that is:
select * from foo where foo.bar = #myParam option (optimize recompile)
Hope this helps.
I am troubleshooting a slow query, it runs in less than 100 ms 99% of the time, but once in an hr (or two no pattern, i guess), goes bad and does 6 million reads and takes 11 seconds! I saw the query plan, it does do a clustered index scan, I noticed the cached_plans dynamic management view use counts column keeps increasing every time the query executes, so i am thinking its the same plan, just wondering why at one point it goes out-of-whack! any pointers will be helpful. I haven't tried anything as it runs pretty fast most of the time.
First something could easily be blocking the query to make it run slow. Otr there could be other things happening onthe server at the same time that are consuming most of its resources.
Next, the parameters of the query might be bad for the saved execution plan.
Or the statistics might be out of date
Or if the query is an action query as opposed to a select, the particular parameters may be causing a problem in a trigger that makes it take longer.
Or teh query might be returning significanlty more results at times. If you run it at 10 and return 10 results and an import puts more records inteh table that meet the query conditions, at 10:30 you might return a million results which would clearly be slower.
One thing I like to do in such circumstances is set up logging so that the exact query is logged with the time at the time of execution. Then you can see what the query that ran sloweractaully was if you have varaible , than might be differnt from run to run.
When I run a certain stored procedure for the first time it takes about 2 minutes to finish. When I run it for the second time it finished in about 15 seconds. I'm assuming that this is because everything is cached after the first run. Is it possible for me to "warm the cache" before I run this procedure for the first time? Is the cached information only used when I call the same stored procedure with the same parameters again or will it be used if I call the same stored procedure with different params?
When you peform your query, the data is read into memory in blocks. These blocks remain in memory but they get "aged". This means the blocks are tagged with the last access and when Sql Server requires another block for a new query and the memory cache is full, the least recently used block (the oldest) is kicked out of memory. (In most cases - full tables scan blocks are instantly aged to prevent full table scans overrunning memory and choking the server).
What is happening here is that the data blocks in memory from the first query haven't been kicked out of memory yet so can be used for your second query, meaning disk access is avoided and performance is improved.
So what your question is really asking is "can I get the data blocks I need into memory without reading them into memory (actually doing a query)?". The answer is no, unless you want to cache the entire tables and have them reside in memory permanently which, from the query time (and thus data size) you are describing, probably isn't a good idea.
Your best bet for performance improvement is looking at your query execution plans and seeing whether changing your indexes might give a better result. There are two major areas that can improve performance here:
creating an index where the query could use one to avoid inefficient queries and full table scans
adding more columns to an index to avoid a second disk read. For example, you have a query that returns columns A, and B with a where clause on A and C and you have an index on column A. Your query will use the index for column A requiring one disk read but then require a second disk hit to get columns B and C. If the index had all columns A, B and C in it the second disk hit to get the data can be avoided.
I don't think that generating the execution plan will cost more that 1 second.
I believe that the difference between first and second run is caused by caching the data in memory.
The data in the cache can be reused by any further query (stored procedure or simple select).
You can 'warm' the cache by reading the data through any select that reads the same data. But that will even cost about 90 seconds as well.
You can check the execution plan to find out which tables and indexes your query uses. You can then execute some SQL to get the data into the cache, depending on what you see.
If you see a clustered index seek, you can simply do SELECT * FROM my_big_table to force all the table's data pages into the cache.
If you see a non-clustered index seek, you could try SELECT first_column_in_index FROM my_big_table.
To force a load of a specific index, you can also use the WITH(INDEX(index)) table hint in your cache warmup queries.
SQL server cache data read from disc.
Consecutive reads will do less IO.
This is of great help since disk IO is usually the bottleneck.
More at:
http://blog.sqlauthority.com/2014/03/18/sql-server-performance-do-it-yourself-caching-with-memcached-vs-automated-caching-with-safepeak/
The execution plan (the cached info for your procedure) is reused every time, even with different parameters. It is one of the benefits of using stored procs.
The very first time a stored procedure is executed, SQL Server generates an execution plan and puts it in the procedure cache.
Certain changes to the database can trigger an automatic update of the execution plan (and you can also explicitly demand a recompile).
Execution plans are dropped from the procedure cache based an their "age". (from MSDN: Objects infrequently referenced are soon eligible for deallocation, but are not actually deallocated unless memory is required for other objects.)
I don't think there is any way to "warm the cache", except to perform the stored proc once. This will guarantee that there is an execution plan in the cache and any subsequent calls will reuse it.
more detailed information is available in the MSDN documentation: http://msdn.microsoft.com/en-us/library/ms181055(SQL.90).aspx