Sporadically Slow Calls From .NET Application To SQL Server - sql-server-2005

Question

I have a table in SQL Server that I inherited from a legacy system thats still in production that is structured according to the code below. I created a SP to query the table as described in the code below the table create statement. My issue is that, sporadically, calls from .NET to this SP both through the Enterprise Library 4 and through a DataReader object are slow. The SP is called through a loop structure in the Data Layer that specifies the params that go into the SP for the purpose of populating user objects. It's also important to mention that a slow call will not take place on every pass the loop structure. It will generally be fine for most of a day or more, and then start presenting which makes it extremely hard to debug.
The table in question contains about 5 million rows. The calls that are slow, for instance, will take as long as 10 seconds, while the calls that are fast will take 0 to 10 milliseconds on average. I checked for locking/blocking transactions during the slow calls, none were found. I created some custom performance counters in the data layer to monitor call times. Essentially, when performance is bad, it's really bad for that one call. But when it's good, it's really good. I've been able to recreate the issue on a few different developer machines, but not on our development and staging database servers, which of course have beefier hardware. Generally, the problem is resolved through restarting the SQL server services, but not always. There are indexes on the table for the fields I'm querying, but there are more indexes than I would like. However, I'm hesitant to remove any or toy with the indexes due to the impact it may have on the legacy system. Has anyone experienced a problem like this before, or do you have a recommendation to remedy it?
CREATE TABLE [dbo].[product_performance_quarterly](
[performance_id] [int] IDENTITY(1,1) NOT FOR REPLICATION NOT NULL,
[product_id] [int] NULL,
[month] [int] NULL,
[year] [int] NULL,
[performance] [decimal](18, 6) NULL,
[gross_or_net] [char](15) NULL,
[vehicle_type] [char](30) NULL,
[quarterly_or_monthly] [char](1) NULL,
[stamp] [datetime] NULL CONSTRAINT [DF_product_performance_quarterly_stamp] DEFAULT (getdate()),
[eA_loaded] [nchar](10) NULL,
[vehicle_type_id] [int] NULL,
[yearmonth] [char](6) NULL,
[gross_or_net_id] [tinyint] NULL,
CONSTRAINT [PK_product_performance_quarterly_4_19_04] PRIMARY KEY CLUSTERED
(
[performance_id] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON, FILLFACTOR = 80) ON [PRIMARY]
) ON [PRIMARY]
GO
SET ANSI_PADDING OFF
GO
ALTER TABLE [dbo].[product_performance_quarterly] WITH NOCHECK ADD CONSTRAINT [FK_product_performance_quarterlyProduct_id] FOREIGN KEY([product_id])
REFERENCES [dbo].[products] ([product_id])
GO
ALTER TABLE [dbo].[product_performance_quarterly] CHECK CONSTRAINT [FK_product_performance_quarterlyProduct_id]
CREATE PROCEDURE [eA.Analytics.Calculations].[USP.GetCalculationData]
(
#PRODUCTID INT, --products.product_id
#BEGINYEAR INT, --year to begin retrieving performance data
#BEGINMONTH INT, --month to begin retrieving performance data
#ENDYEAR INT, --year to end retrieving performance data
#ENDMONTH INT, --month to end retrieving performance data
#QUARTERLYORMONTHLY VARCHAR(1), --do you want quarterly or monthly data?
#VEHICLETYPEID INT, --what product vehicle type are you looking for?
#GROSSORNETID INT --are your looking gross of fees data or net of fees data?
)
AS
BEGIN
SET NOCOUNT ON
DECLARE #STARTDATE VARCHAR(6),
#ENDDATE VARCHAR(6),
#vBEGINMONTH VARCHAR(2),
#vENDMONTH VARCHAR(2)
IF LEN(#BEGINMONTH) = 1
SET #vBEGINMONTH = '0' + CAST(#BEGINMONTH AS VARCHAR(1))
ELSE
SET #vBEGINMONTH = #BEGINMONTH
IF LEN(#ENDMONTH) = 1
SET #vENDMONTH = '0' + CAST(#ENDMONTH AS VARCHAR(1))
ELSE
SET #vENDMONTH = #ENDMONTH
SET #STARTDATE = CAST(#BEGINYEAR AS VARCHAR(4)) + #vBEGINMONTH
SET #ENDDATE = CAST(#ENDYEAR AS VARCHAR(4)) + #vENDMONTH
--because null values for gross_or_net_id and vehicle_type_id are represented in
--multiple ways (true null, empty string, or 0) in the PPQ table, need to account for all possible variations if
--a -1 is passed in from the .NET code, which represents an enumerated value that
--indicates that the value(s) should be true null.
IF #VEHICLETYPEID = '-1' AND #GROSSORNETID = '-1'
SELECT
PPQ.YEARMONTH, PPQ.PERFORMANCE
FROM PRODUCT_PERFORMANCE_QUARTERLY PPQ
WITH (NOLOCK)
WHERE
(PPQ.PRODUCT_ID = #PRODUCTID)
AND (PPQ.YEARMONTH BETWEEN #STARTDATE AND #ENDDATE)
AND (PPQ.QUARTERLY_OR_MONTHLY = #QUARTERLYORMONTHLY)
AND (PPQ.VEHICLE_TYPE_ID IS NULL OR PPQ.VEHICLE_TYPE_ID = '0' OR PPQ.VEHICLE_TYPE_ID = '')
AND (PPQ.GROSS_OR_NET_ID IS NULL OR PPQ.GROSS_OR_NET_ID = '0' OR PPQ.GROSS_OR_NET_ID = '')
ORDER BY PPQ.YEARMONTH ASC
IF #VEHICLETYPEID <> '-1' AND #GROSSORNETID <> '-1'
SELECT
PPQ.YEARMONTH, PPQ.PERFORMANCE
FROM PRODUCT_PERFORMANCE_QUARTERLY PPQ
WITH (NOLOCK)
WHERE
(PPQ.PRODUCT_ID = #PRODUCTID)
AND (PPQ.YEARMONTH BETWEEN #STARTDATE AND #ENDDATE)
AND (PPQ.QUARTERLY_OR_MONTHLY = #QUARTERLYORMONTHLY)
AND (PPQ.VEHICLE_TYPE_ID = #VEHICLETYPEID )
AND (PPQ.GROSS_OR_NET_ID = #GROSSORNETID)
ORDER BY PPQ.YEARMONTH ASC
IF #VEHICLETYPEID = '-1' AND #GROSSORNETID <> '-1'
SELECT
PPQ.YEARMONTH, PPQ.PERFORMANCE
FROM PRODUCT_PERFORMANCE_QUARTERLY PPQ
WITH (NOLOCK)
WHERE
(PPQ.PRODUCT_ID = #PRODUCTID)
AND (PPQ.YEARMONTH BETWEEN #STARTDATE AND #ENDDATE)
AND (PPQ.QUARTERLY_OR_MONTHLY = #QUARTERLYORMONTHLY)
AND (PPQ.VEHICLE_TYPE_ID IS NULL OR PPQ.VEHICLE_TYPE_ID = '0' OR PPQ.VEHICLE_TYPE_ID = '')
AND (PPQ.GROSS_OR_NET_ID = #GROSSORNETID)
ORDER BY PPQ.YEARMONTH ASC
IF #VEHICLETYPEID <> '-1' AND #GROSSORNETID = '-1'
SELECT
PPQ.YEARMONTH, PPQ.PERFORMANCE
FROM PRODUCT_PERFORMANCE_QUARTERLY PPQ
WITH (NOLOCK)
WHERE
(PPQ.PRODUCT_ID = #PRODUCTID)
AND (PPQ.YEARMONTH BETWEEN #STARTDATE AND #ENDDATE)
AND (PPQ.QUARTERLY_OR_MONTHLY = #QUARTERLYORMONTHLY)
AND (PPQ.VEHICLE_TYPE_ID = #VEHICLETYPEID)
AND (PPQ.GROSS_OR_NET_ID IS NULL OR PPQ.GROSS_OR_NET_ID = '0' OR PPQ.GROSS_OR_NET_ID = '')
ORDER BY PPQ.YEARMONTH ASC
END

Answer 1

I have seen this happen with indexes that were out of date. It could also be a parameter sniffing problem, where a different query plan is being used for different parameters that come in to the stored procedure.
You should capture the parameters of the slow calls and see if they are the same ones each time it runs slow.
You might also try running the tuning wizard and see if it recommends any indexes.
You don't want to worry about having too many indexes until you can prove that updates and inserts are happening too slow (time needed to modify the index plus locking/contention), or you are running out of disk space for them.

Answer 2

Sounds like another query is running in the background that has locked the table and your innocent query is simply waiting for it to finish

Answer 3

A strange, edge case but I encountered it recently.
If the queries run longer in the application than they do when run from within Management Studio, you may want to check to make sure that Arithabort is set off. The connection parameters used by Management Studio are different from the ones used by .NET.

Answer 4

It seems like it's one of two things - either the parameters on the slow calls are different in some way than on the fast calls, and they're not able to use the indexes as well, or there's some type of locking contention that's holding you up. You say you've checked for blocking locks while a particular process is hung, and saw none - that would suggest that it's the first one. However - are you sure that your staging server (that you can't reproduce this error on) and the development servers (that you can reproduce it on) have the same database configuration? For example, maybe "READ COMMITTED SNAPSHOT" is enabled in production, but not in development, which would cause read contention issues to disappear in production.
If it's a difference in parameters, I'd suggest using SQL Profiler to watch the transactions and capture a few - some slow ones and some faster ones, and then, in a Management Studio window, replace the variables in that SP above with the parameter values and then get an execution plan by pressing "Control-L". This will tell you exactly how SQL Server expects to process your query, and you can compare the execution plan for different parameter combination to see if there's a difference with one set, and work from there to optimize it.
Good luck!