I have an application that calls an Azure SQL stored procedure "report.GetReportCell". To call this procedure my app uses SqlCommand.ExecuteReader() method.
Sometimes calls are too slow. In Dependency telemetry I see a lot of records with:
my proc name ("report.GetReportCell") in the Data field
long (> 25,000) Duration
I suspect the stored procedure is too slow but I can't confirm it:
sys.dm_exec_procedure_stats.max_elapsed_time is short (less than 8 sec)
sys.query_store_runtime_stats doesn't contain any related long-term query
when I call this proc in SSMS it works fast
What could be causing the slowdown when calling a stored procedure with a SqlCommand.ExecuteReader() method?
UPD 2022-07-21
I have read an excellent article about "Parameter Sniffing Problems" and other other possible reasons for slow executions - https://www.sommarskog.se/query-plan-mysteries.html
But I think the reason for the slowdown is something else. Because:
Query Store doesn't contain any related slow query
sys.dm_exec_procedure_stats.max_elapsed_time for report.GetReportCell is small
I had the exact same issue and #DeepDave-MT had the answer. The procedure was taking 30 seconds in the application but 0 seconds via SSMS. When I ran it in SSMS with SET ARITHABORT OFF, I found that it ran for 22 seconds. Adding SET ARITHABORT ON directly to the stored procedure made my application run in <1 second.
Related
I have run into an enigma of sorts while researching a performance issue with a specific stored procedure. I did not create that stored procedure, and the logic is fairly ugly with nested selects in join statements, etc...
However, when I copy the logic of the stored procedure directly into a new query window, add the parameters at the top and execute it, this runs in under 400 milliseconds. Yet, when I call the stored procedure and execute it with the exact same parameter values, it takes 23 seconds to run!
This makes absolutely no sense at all to me!
Are there some server-level settings that I should check which could potentially be impacting this?
Thanks
Recompile your stored procedure.
The Microsoft documentation says
When a procedure is compiled for the first time or recompiled, the procedure's query plan is optimized for the current state of the database and its objects. If a database undergoes significant changes to its data or structure, recompiling a procedure updates and optimizes the procedure's query plan for those changes. This can improve the procedure's processing performance.
and
Another reason to force a procedure to recompile is to counteract the "parameter sniffing" behavior of procedure compilation. When SQL Server executes procedures, any parameter values that are used by the procedure when it compiles are included as part of generating the query plan. If these values represent the typical ones with which the procedure is subsequently called, then the procedure benefits from the query plan every time that it compiles and executes. If parameter values on the procedure are frequently atypical, forcing a recompile of the procedure and a new plan based on different parameter values can improve performance.
If you run the SP's queries yourself (in SSMS maybe) they get compiled and run.
How to recompile your SP? (See the doc page linked above.)
You can rerun its definition to recompile it once. That may be enough if the procedure was first defined long ago in the life of your database and your tables have grown a lot.
You can put CREATE PROCEDURE ... WITH RECOMPILE AS into its definition so it will recompile every time you run it.
You can EXEC procedure WITH RECOMPILE; when you run it.
You can restart your SQL Server. (This can be a nuisance; booting the server magically makes bad things stop happening and nobody's quite sure why.)
Recompiling takes some time. But it takes far less time than a complex query with a bad execution plan would.
So... I ended up turning the nested selects on the joins to table variables, and now the sproc is executing in 60-milliseconds, while the in-line sql is taking 250=ms.
However, I still do not understand why the sproc was performing so much slower than the in-line sql version with the original nested sql logic?
I mean, both were using the exact same sql logic, so why was the sproc taking 23-seconds while the in-line was 400-ms?
I'm working on updated procedures that I inherited from someone who is no longer at the company.
I have found procedures that include WITH RECOMPILE option after the header. In the notes it says that it was added "to mitigate timeouts from blocking"
ALTER PROC ups_SomeProc (#pMyParameter INT) WITH RECOMPILE
AS
BEGIN
In all my experience I never heard of WITH RECOMPILE option helping with blocking or even be related to blocking in any way.
Am I missing some understanding about this option or the person who implemented it was the one who got confused on what it does? Have anyone heard of this before as a solution for blocking?
Note: This was done when server was still running SQL Sever 2008 edition.
OPTION WITH RECOMPILE forces Sql Server to Recompile an execution plan even if there is an existing plan cached in the memory.
If the underlying data changes dramaticly and very quickly the cached execution plan becomes less efficient. Therefore in such situation using WITH RECOMPILE option executes the procedure much faster then it would execute if it had used the already compiled execution plan.
My guess the developer/person experienced some long delays when he executed the stored procedure. and when he used the WITH RECOMPILE option it got executed faster. So maybe he thought executing this procedure without recompile option causes blocking. :) funny story but I think this is what happened.
I can think of one way it could help (which I actually experienced myself). Each stored proc has a query plan cached. There could be a problem sometimes if the way the stored is executed varies wildly based on some values as the cached query plan may be completely unsuitable.
Let's say you have a stored proc that looks like
create procedure SomeProc
as
begin
declare #value int
select #value = somevalue from atable where date = getdate() -- getting different value every time
if #value = 1 then -- do one thing
if #value = 2 then -- do something different
if #value = 3 then -- do something completely different
end
The query plan might have been cached when the value was 1. When you run it again and the value is now 2 or 3, the information is not suitable and may result in a query execution that takes very long time. You can sometimes spot such queries by having wildly varied number of reads, etc.
When you use WITH RECOMPILE, it can mitigate those problems by forcing SQL Server to come up with a new execution plan.
I am breaking my head on this issue since long. I have stored procedure in MS SQl and when I try to execute that procedure by providing all the parameters in SQL Query, it takes long time to execute but when I try to directly run the query which is there in SP it executes in no time. This is affecting my application performance also as we are using stored procedures to fetch the data from DB Server.
Please help.
Regards,
Vikram
Looks like parameter sniffing.
Here is a nice explanation: I Smell a Parameter!
Basically, sql server has cached query execution plan for the parameters it was first run with so the plan is not optimal for the new values you are passing. When you run the query directly the plan is generated at that moment so that's why it's fast.
You can mark the procedure for recompilation manually using sp_recompile or use With Recompile option in its definition so it is compiled on every run.
So basically I have this relatively long stored procedure. The basic execution flow is that it SELECTS INTO some data into temp tables declared with the # sign and then runs a cursor through these tables to generate a 'running total' into a third temp table which is created using CREATE. Then this resulting temp table is joined with other tables in the DB to generated the result after some grouping etc. The problem is, this SP had been running fine until now returning results in 1-2 minutes. And now, suddenly, its taking 12-15 minutes. If I extract the query from the SP and executed it in management studio by manually setting the same parameters, it returns results in 1-2 minutes but the SP takes very long. Any idea what could be happening? I tried to generate the Actual Execution plans of both the query and the SP but it couldn't generate it because of the cursor. Any idea why the SP takes so long while the query doesn't?
This is the footprint of parameter-sniffing. See here for another discussion about it; SQL poor stored procedure execution plan performance - parameter sniffing
There are several possible fixes, including adding WITH RECOMPILE to your stored procedure which works about half the time.
The recommended fix for most situations (though it depends on the structure of your query and sproc) is to NOT use your parameters directly in your queries, but rather store them into local variables and then use those variables in your queries.
its due to parameter sniffing. first of all declare temporary variable and set the incoming variable value to temp variable and use temp variable in whole application here is an example below.
ALTER PROCEDURE [dbo].[Sp_GetAllCustomerRecords]
#customerId INT
AS
declare #customerIdTemp INT
set #customerIdTemp = #customerId
BEGIN
SELECT *
FROM Customers e Where
CustomerId = #customerIdTemp
End
try this approach
Try recompiling the sproc to ditch any stored query plan
exec sp_recompile 'YourSproc'
Then run your sproc taking care to use sensible paramters.
Also compare the actual execution plans between the two methods of executing the query.
It might also be worth recomputing any statistics.
I'd also look into parameter sniffing. Could be the proc needs to handle the parameters slighlty differently.
I usually start troubleshooting issues like that by using
"print getdate() + ' - step '". This helps me narrow down what's taking the most time. You can compare from where you run it from query analyzer and narrow down where the problem is at.
I would guess it could possible be down to caching. If you run the stored procedure twice is it faster the second time?
To investigate further you could run them both from management studio the stored procedure and the query version with the show query plan option turned on in management studio, then compare what area is taking longer in the stored procedure then when run as a query.
Alternativly you could post the stored procedure here for people to suggest optimizations.
For a start it doesn't sound like the SQL is going to perform too well anyway based on the use of a number of temp tables (could be held in memory, or persisted to tempdb - whatever SQL Server decides is best), and the use of cursors.
My suggestion would be to see if you can rewrite the sproc as a set-based query instead of a cursor-approach which will give better performance and be a lot easier to tune and optimise. Obviously I don't know exactly what your sproc does, to give an indication as to how easy/viable this is for you.
As to why the SP is taking longer than the query - difficult to say. Is there the same load on the system when you try each approach? If you run the query itself when there's a light load, it will be better than when you run the SP during a heavy load.
Also, to ensure the query truly is quicker than the SP, you need to rule out data/execution plan caching which makes a query faster for subsequent runs. You can clear the cache out using:
DBCC FREEPROCCACHE
DBCC DROPCLEANBUFFERS
But only do this on a dev/test db server, not on production.
Then run the query, record the stats (e.g. from profiler). Clear the cache again. Run the SP and compare stats.
1) When you run the query for the first time it may take more time. One more point is if you are using any corellated sub query and if you are hardcoding the values it will be executed for only one time. When you are not hardcoding it and run it through the procedure and if you are trying to derive the value from the input value then it might take more time.
2) In rare cases it can be due to network traffic, also where we will not have consistency in the query execution time for the same input data.
I too faced a problem where we had to create some temp tables and then manipulating them had to calculate some values based on rules and finally insert the calculated values in a third table. This all if put in single SP was taking around 20-25 min. So to optimize it further we broke the sp into 3 different sp's and the total time now taken was around 6-8 mins. Just identify the steps that are involved in the whole process and how to break them up in different sp's. Surely by using this approach the overall time taken by the entire process will reduce.
This is because of parameter snipping. But how can you confirm it?
Whenever we supposed to optimize SP we look for execution plan. But in your case, you will see an optimized plan from SSMS because it's taking more time only when it called through Code.
For every SP and Function, the SQL server generates two estimated plans because of ARITHABORT option. One for SSMS and second is for the external entities(ADO Net).
ARITHABORT is by default OFF in SSMS. So if you want to check what exact query plan your SP is using when it calls from Code.
Just enable the option in SSMS and execute your SP you will see that SP will also take 12-13 minutes from SSMS.
SET ARITHABORT ON
EXEC YourSpName
SET ARITHABORT OFF
To solve this problem you just need to update the estimate query plan.
There are a couple of ways to update the estimate query plan.
1. Update table statistics.
2. recompile SP
3. SET ARITHABORT OFF in SP so it will always use query plan created for SSMS (this option is not recommended)
For more options please refer to this awesome article -
http://www.sommarskog.se/query-plan-mysteries.html
I would suggest the issue is related to the type of temp table (the # prefix). This temp table holds the data for that database session. When you run it through your app the temp table is deleted and recreated.
You might find when running in SSMS it keeps the session data and updates the table instead of creating it.
Hope that helps :)
Our application executes a long, hairy stored procedure with multiple result sets. The users are experiencing long wait times for this query, so I set out to determine what is causing the delay.
I put a Stopwatch on executing and reading the data, and it takes 6-7 seconds each time. I timed the execution of the stored procedure, expecting that this would be taking all the time. It wasn't - it took 30ms or so.
So I put timers around each of the ~20 result sets. Each "block" took very little time ( < 10ms) except for one in the middle of the processing, which took 5-6 seconds. Upon further research, I discovered it was the "reader.NextResult()" call that took all the time. This long delay happens in the same spot each time.
If I just exec the stored procedure, it seems to run real snappy, so it doesn't APPEAR to be a problem with the query - but I don't know...
How do I interpret this? Is SQL shipping me the result sets as it gets them, and is the result set in question likely to be a problem area in my SQL query? Or is something else possibly causing the delay?
EDIT:
Thanks for the answer and the comments - I am using SQL Server and .NET
What I was most curious about was WHY my delay happens on the "NextResult()" call. Being new to SQL development, I assumed that a delay due to a long stored procedure execution would show up in my application while waiting for the "ExecuteReader()" call to return. It now seems that SQL will start returning data BEFORE the query is complete, and if there is a delay it will delay on the NextResult() call.
I started out thinking my delay was in the stored procedure. When the ExecuteReader() call came back quickly, I thought my delay was in my code's handling of the reader. When the delay ended up being on the NextResult() call, I was confused. I am now back to reviewing the stored procedure.
Thanks to those of you who took the time to review my problem and offered your help.
When you execute a stored proc from a .Net command, the results will start streaming as soon as SQL has them ready.
This means that you may start seeing results in your .Net app before the entire stored proc has been executed.
Your bottleneck is probably in the stored procedure, run a sql server trace, and trace all the statements running inside the stored procedure (get the durations). You will be able to track down the exact statement in the proc that is slow and you also will be able to pick up on the params that are being passed to the proc so you can test this in Query Analyzer and look at the plan.
Another point that is missing from the question seems to be the amount of data you are moving, though unlikely, it may be that you have some really large chunks of data (like blobs) that are being sent and the time is being spent on the wire. You really need to expand the question a bit to help with the diagnosis.
The answer will be dependent on what RDBMS you are using.
If its SQL Server and .NET then from my experience:
Check other open transactions on the same connection which is used to invoke the sproc. They may have row locks on the table one of your selects is executing against. You can try adding "MultipleActiveResultSets=false" to your SQL Server connection string and see if you get an improvement, or more likely an exception (and you can hunt down the problem from an exception). This can also be an effect from an unreset connection returned to the connection pool (something I've ran into since I've started to use MARS).
You may need to specify the NOLOCK (or READUNCOMMITTED, same thing) table hint in your SELECT query if dirty reads are acceptible.
SELECT * FROM [table] WITH NOLOCK