Azure SQL DB causing connection time out for stored procedures - sql

Question

We have hosted our database in Azure and are running stored procedures on this DB. The stored procedures had been running fine till last week but suddenly started giving error connection timeout.
Our database size is 14 GB and the stored procedures in general return 2k to 20k records and we are using the S3 pricing tier (50 DTU) of Azure DB.
What I found interesting was the first time the stored procedure is executed, it takes a lot of time 2 - 3 mins and this is causing the timeout. The later executions are fast (maybe it caches the execution plan).
Also when I run on the same DB with the same number of records on a machine with the config of 8gb ram, Win10 it runs in 15 seconds.
This is my stored procedure:
CREATE PROCEDURE [dbo].[PRSP]
#CompanyID INT,
#fromDate DATETIME,
#toDate DATETIME,
#ListMailboxId as MailboxIds Readonly,
#ListConversationType as ConversationTypes Readonly
AS
BEGIN
SET NOCOUNT ON;
SELECT
C.ID,
C.MailboxID,
C.Status,
C.CustomerID,
Cust.FName,
Cust.LName,
C.ArrivalDate as ConversationArrivalDate,
C.[ClosureDate],
C.[ConversationType],
M.[From],
M.ArrivalDate as MessageArrivalDate,
M.ID as MessageID
FROM
[Conversation] as C
INNER JOIN
[ConversationHistory] AS CHis ON (CHis.ConversationID = C.ID)
INNER JOIN
[Message] AS M ON (M.ConversationID = C.ID)
INNER JOIN
[Mailbox] AS Mb ON (Mb.ID = C.MailboxID)
INNER JOIN
[Customer] AS Cust ON (Cust.ID = C.CustomerID)
JOIN
#ListConversationType AS convType ON convType.ID = C.[ConversationType]
JOIN
#ListMailboxId AS mailboxIds ON mailboxIds.ID = Mb.ID
WHERE
Mb.CompanyID = #CompanyID
AND ((CHis.CreatedOn > #fromDate
AND CHis.CreatedOn < #toDate
AND CHis.Activity = 1
AND CHis.TagData = '3')
OR (M.ArrivalDate > #fromDate
AND M.ArrivalDate < #toDate))
END
This is the execution plan :
Execution Plan
Please do give your suggestions as to what improvement is needed? Also do we need to upgrade my pricing tier?
Ideally for a 14GB DB what should be the Azure Pricing tier?

Answer 1

That query should take 1 to 3 seconds to complete on your Windows 10 8Gb RAM machine. It takes 15 seconds because SQL Server choose a poor execution plan. In this case, the root cause of poor execution plan is bad estimates, several operators in the plan show big difference between estimated rows and actual rows. For example, SQL Server estimated it only need to perform one seek into pk_customer clustered index, but it performed 16,522 seeks. The same thing occurs with [ConversationHistory].[IX_ConversationID_CreatedOn_Activity_ByWhom] and with [Message].[IX_ConversationID_ID_ArrivalDt_From_RStatus_Type.
Here you have some hints you could follow to improve the performance of the query:
Update statistics
Try OPTION (HASH JOIN) at the end of the query. It might improve the
performance or it might slow it down, it even can cause the query to
error.
Store table variable data in temporal tables and use them in the query. (SELECT * INTO #temp_table FROM #table_variable). Table variables don't have statistics causing bad estimates.
Identify the first operator where the difference between estimated rows and actual rows are big enough. Split the query. Query1: SELECT * INTO #operator_result FROM (query equivalent to operator). Query2: write the query using #operator_result. Because #operator_result is a temporal table SQL Server is forced to reevaluate estimates. In this case, the offending operator is the hash match (inner join)
There are other things you can do to improve the performance of this query:
Avoid key lookups. There are 16,522 key lookups into Conversation.PK_dbo.Conversation clusterd index. It can ve avoided by creating the appropriate covering index. In this case, the covering index is the following:
DROP INDEX [IX_MailboxID] ON [dbo].[Conversation]
GO
CREATE INDEX IX_MailboxID ON [dbo].[Conversation](MailboxID)
INCLUDE (ArrivalDate, Status, ClosureDate, CustomerID, ConversationType)
Split OR predicate into UNION or UNION ALL. For example:
instead of:
SELECT *
FROM table
WHERE <predicate1> OR <predicate2>
use:
SELECT *
FROM table
WHERE <predicate1>
UNION
SELECT *
FROM table
WHERE <predicate2>
Sometimes it improves performance.
Apply each hint individually and measure performance.
EDIT: You can try the following and see if it improves performance:
SELECT
C.ID,
C.MailboxID,
C.Status,
C.CustomerID,
Cust.FName,
Cust.LName,
C.ArrivalDate as ConversationArrivalDate,
C.[ClosureDate],
C.[ConversationType],
M.[From],
M.ArrivalDate as MessageArrivalDate,
M.ID as MessageID
FROM
#ListConversationType AS convType
INNER JOIN (
#ListMailboxId AS mailboxIds
INNER JOIN
[Mailbox] AS Mb ON (Mb.ID = mailboxIds.MailboxID)
INNER JOIN
[Conversation] as C
ON C.ID = Mb.ID
) ON convType.ID = C.[ConversationType]
INNER HASH JOIN
[Customer] AS Cust ON (Cust.ID = C.CustomerID)
INNER HASH JOIN
[ConversationHistory] AS CHis ON (CHis.ConversationID = C.ID)
INNER HASH JOIN
[Message] AS M ON (M.ConversationID = C.ID)
WHERE
Mb.CompanyID = #CompanyID
AND ((CHis.CreatedOn > #fromDate
AND CHis.CreatedOn < #toDate
AND CHis.Activity = 1
AND CHis.TagData = '3')
OR (M.ArrivalDate > #fromDate
AND M.ArrivalDate < #toDate))
And this:
SELECT
C.ID,
C.MailboxID,
C.Status,
C.CustomerID,
Cust.FName,
Cust.LName,
C.ArrivalDate as ConversationArrivalDate,
C.[ClosureDate],
C.[ConversationType],
M.[From],
M.ArrivalDate as MessageArrivalDate,
M.ID as MessageID
FROM
#ListConversationType AS convType
INNER JOIN (
#ListMailboxId AS mailboxIds
INNER JOIN
[Mailbox] AS Mb ON (Mb.ID = mailboxIds.MailboxID)
INNER JOIN
[Conversation] as C
ON C.ID = Mb.ID
) ON convType.ID = C.[ConversationType]
INNER MERGE JOIN
[Customer] AS Cust ON (Cust.ID = C.CustomerID)
INNER MERGE JOIN
[ConversationHistory] AS CHis ON (CHis.ConversationID = C.ID)
INNER MERGE JOIN
[Message] AS M ON (M.ConversationID = C.ID)
WHERE
Mb.CompanyID = #CompanyID
AND ((CHis.CreatedOn > #fromDate
AND CHis.CreatedOn < #toDate
AND CHis.Activity = 1
AND CHis.TagData = '3')
OR (M.ArrivalDate > #fromDate
AND M.ArrivalDate < #toDate))

Answer 2

50 DTU is equivalent to 1/2 logical core.
See more: Using the Azure SQL Database DTU Calculator

Answer 3

I had the same issue this week and the final users claimed slowness in using the application connected to the VM hosted in Azure. Also, I have almost the same VM (4CPUs, 14GB of RAM and S3 but with 100DTUs).
In my case, I had a lot of indexes with avg_fragmentation_in_percent greater than 30 and this caused poor performance in executing stored procedures.
Run this in SSMS and if the indexes of the tables you are running your stored procedure against are there, then you might take care of it:
SELECT dbschemas.[name] as 'Schema',
dbtables.[name] as 'Table',
dbindexes.[name] as 'Index',
indexstats.avg_fragmentation_in_percent,
indexstats.page_count
FROM sys.dm_db_index_physical_stats (DB_ID(), NULL, NULL, NULL, NULL) AS indexstats
INNER JOIN sys.tables dbtables on dbtables.[object_id] = indexstats.[object_id]
INNER JOIN sys.schemas dbschemas on dbtables.[schema_id] = dbschemas.[schema_id]
INNER JOIN sys.indexes AS dbindexes ON dbindexes.[object_id] = indexstats.[object_id]
WHERE indexstats.database_id = DB_ID()
AND indexstats.index_id = dbindexes.index_id
AND indexstats.avg_fragmentation_in_percent >30
--AND dbindexes.[name] like '%CLUSTER%'
ORDER BY indexstats.avg_fragmentation_in_percent DESC
More info here.
Edit:
Check also how old the statistics are:
SELECT
sys.objects.name AS table_name,
sys.indexes.name as index_name,
sys.indexes.type_desc as index_type,
stats_date(sys.indexes.object_id,sys.indexes.index_id)
as last_update_stats_date,
DATEDIFF(d,stats_date(sys.indexes.object_id,sys.indexes.index_id),getdate())
as stats_age_in_days
FROM
sys.indexes
INNER JOIN sys.objects on sys.indexes.object_id=sys.objects.object_id
WHERE
sys.objects.type = 'U'
AND
sys.indexes.index_id > 0
--AND sys.indexes.name Like '%CLUSTER%'
ORDER BY
stats_age_in_days DESC;
GO