SQL Query Optimization taking 20 - 30 secs to run - sql

Below query takes 20 secs to execute and i need to optimize it as much as i can. Please help me on this.
SELECT Distinct
qh.QuoteHeaderId, [dbo].[mpx2_Get_PhoneGrade](qh.QuoteHeaderId)
FROM
t_QuoteHeader QH
INNER JOIN
t_HandsetQuote h ON Qh.QuoteHeaderId = h.QuoteHeaderId
INNER JOIN
t_phoneAudit P ON ISNULL(h.InspectionPhoneAuditId, h.QuotePhoneAuditId) = p.PhoneAuditId
INNER JOIN
mpx2_vw_customers C ON qh.CustomerId = C.CustomerId
INNER JOIN
#ContactChannels CC ON C.ContactChannelId = CC.ContactChannelId
LEFT OUTER JOIN
t_HandsetQuoteAdditionalInfo_TRNX hqa ON hqa.hqid = h.HandsetQuoteId
WHERE
((#VirtualStatusId = 0 OR #VirtualStatusId = -2 OR
C.ContactChannelId NOT IN (1, 2, 13, 80)))
AND ((#VirtualStatusId = -2) OR
('Q'+ CAST(Qh.QuoteStatusId AS VARCHAR(3)) + 'S' + CAST(h.StockStatusId AS VARCHAR(3)) IN
(SELECT 'Q'+ CAST(QuoteStatusId AS VARCHAR(3)) + 'S' + CAST(StockStatusId AS VARCHAR(3)) FROM t_VirtualStatusMap WHERE (#VirtualStatusId IS NULL OR #VirtualStatusId IN (0,-1) OR VirtualStatusId = #VirtualStatusId))
)
)
AND ((qh.IsCancelled = 0 and #onlyOpenOrders = 1) OR #onlyOpenOrders = 0)
AND ((h.IsCancelled = 0 and #onlyOpenOrders = 1) OR #onlyOpenOrders = 0)
AND (qh.ConfirmedDate <= #CutOff)
Please help me to optimize it. This query is used in a stored procedure.

This is too long for a comment.
ORs in WHERE and ON clauses are very hard to optimize. Often with a query like this, it is better to construct the query based on the components and use dynamic SQL.
For instance, the condition on #OnlyOpenOrders would be included like this:
declare #sql varchar(max);
set #sql = ' . . .';
declare #where varchar(max);
set #where = 'where . . .';
if (#OnlyOpenOrders = 0) begin
set #where = #where + ' and qh.IsCancelled = 0 and h.IsCancelled = 0'
end;
set #sql = #sql + ' ' + #where;
exec sp_executesql #sql;
You need to have similar logic for all the variables you are using.

There are a couple of things, although as others have said without all the required information such as a full execution plan, and schemas of the tables involved it is mostly guidelines/guesswork;
1.) In this part, it would appear you build a string from QuoteStatusId and StockStatusId in order to compare them;
('Q'+ CAST(Qh.QuoteStatusId AS VARCHAR(3)) + 'S' + CAST(h.StockStatusId AS VARCHAR(3))
IN
(SELECT 'Q'+ CAST(QuoteStatusId AS VARCHAR(3)) + 'S' + CAST(StockStatusId AS VARCHAR(3))
FROM t_VirtualStatusMap
WHERE (#VirtualStatusId IS NULL
OR #VirtualStatusId IN (0,-1) OR VirtualStatusId = #VirtualStatusId)))
If you skipped building the strings, since they are comprised of the same two columns and just compared the two columns directly that may speed things up.
2.) Have you tried adding the index which it suggests in the picture you attached? Without seeing your schema and an execution plan it is hard to suggest appropriate ones but it might be worth adding the one suggested (right click the green writing and it will generate the code to add the suggested index). I would read up on indexes and ensure there is an appropriate index for the query to use. ConfirmedDate seems like an obvious one, as well as all the join keys.
3.) As Gordon suggested using dynamic sql or if you are not comfortable with that - maybe splitting the query out into a few queries and switching between each using an IF statement, could help SQL generate a good plan for each scenario, instead of trying to find a generic plan to work for all cases.

Related

Is there a faster way to run an SQL Where Case

I have the following stored procedure (In MS SQL):
ALTER PROCEDURE [dbo].[proc_GetWorksWithEngineerVisits3]
#sTextSearch nvarchar(255) = NULL,
#bCompleteFlag bit = NULL,
#dExpectedStartDateTime datetime = NULL,
#dExpectedEndDateTime datetime = NULL,
#sResponsible_UserIDs nvarchar(255) = NULL,
#bEnableTextSearchFilter bit = false,
#bEnableCompleteFlagFilter bit = false,
#bEnableExpectedDateTimeRangeFilter bit = false,
#bEnableResponsible_UserIDFilter bit = false
AS
SELECT *
FROM dbo.vwWorksWithEngineerVisits
WHERE
--TextSearch Filter Start
(sCustomer LIKE CASE
WHEN #bEnableTextSearchFilter = 1
THEN '%' + #sTextSearch + '%'
ELSE sCustomer
END
OR
sSite LIKE CASE
WHEN #bEnableTextSearchFilter = 1
THEN '%' + #sTextSearch + '%'
ELSE sSite
END
OR
sCallID LIKE CASE
WHEN #bEnableTextSearchFilter = 1
THEN '%' + #sTextSearch + '%'
ELSE sCallID
END)
--TextSearch Filter End
AND
--Complete Filter Start
bIsComplete = CASE
WHEN #bEnableCompleteFlagFilter = 1
THEN #bCompleteFlag
ELSE bIsComplete
END
--Complete Filter End
AND
--Expected DateTime Range Filter Start
dExpectedStartDateTime >= CASE
WHEN #bEnableExpectedDateTimeRangeFilter = 1
THEN #dExpectedStartDateTime
ELSE dExpectedStartDateTime
END
AND
dExpectedEndDateTime <=
CASE
WHEN #bEnableExpectedDateTimeRangeFilter = 1
THEN #dExpectedEndDateTime
ELSE dExpectedEndDateTime
END
----Expected DateTime Range Filter End
AND
--Responsible_UserID Filter Start
lResponsible_UserID in (
CASE
WHEN #bEnableResponsible_UserIDFilter = 0
THEN lResponsible_UserID
ELSE (SELECT Value FROM dbo.CSVToList(#sResponsible_UserIDs) AS CSVToList_1)
END
)
--Responsible_UserID Filter End
ORDER BY dExpectedEndDateTime
The output is fine, but it is very slow (15 sec for only 5000 rows) Executing dbo.vwWorksWithEngineerVisits directly takes 1sec for the same number. When executing the SP, I am setting all enable flags = 0.
DECLARE #return_value int
EXEC #return_value = [dbo].[proc_GetWorksWithEngineerVisits3]
#sTextSearch = NULL,
#bCompleteFlag = False,
#dExpectedStartDateTime = N'01/01/1969',
#dExpectedEndDateTime = N'01/01/2021',
#sResponsible_UserIDs = NULL,
#bEnableTextSearchFilter = 0,
#bEnableCompleteFlagFilter = 0,
#bEnableExpectedDateTimeRangeFilter = 0,
#bEnableResponsible_UserIDFilter = 0
SELECT 'Return Value' = #return_value
I want to be able to only filter a column, if the corresponding flag is set. I probably could just check for NULL in the primary parameters and reduce the parameters, but I don't think it changes the problem I am having.
The first 4 Case filters are very basic, and when I comment the remaining last 3 out, the performance/result is instantaneous. As soon as I add one of last 3 back into the mix, things slow down as above. What makes these different is that they do ">=" or "in", rather than just an "=" or "like". The other thing that I noticed is that when I changed the following:
lResponsible_UserID in (
CASE
WHEN #bEnableResponsible_UserIDFilter = 0
THEN lResponsible_UserID
ELSE (SELECT Value FROM dbo.CSVToList(#sResponsible_UserIDs) AS CSVToList_1)
END
to
lResponsible_UserID in (
CASE
WHEN #bEnableResponsible_UserIDFilter = 0
THEN lResponsible_UserID
ELSE lResponsible_UserID
END
This also speed things up to 1 sec. How is this the case that changing the else part of the statement makes any difference whatsoever, when the flag is always 0, so should never run?
I need these filters, and I need them dynamic. There are a mix of operator types (including an IN that targets a function). Is there a way to refactor this stored procedure to have the same result (it does work), but in a much more optional way?
Apologies if I have missed something in my post, and I will edit if this pointed out.
Thanks
That's a big query!
SQL Server runs a compiler against the queries in your sp when you define it. Then it uses that compiled procedure, blithely ignoring any optimizations that might come from your specific parameter values. This page explains:
When SQL Server executes procedures, any parameter values that are used by the procedure when it compiles are included as part of generating the query plan. If these values represent the typical ones with which the procedure is subsequently called, then the procedure benefits from the query plan every time that it compiles and executes. If parameter values on the procedure are frequently atypical, forcing a recompile of the procedure and a new plan based on different parameter values can improve performance.
In your situation, your parameter settings dramatically simplify the search you want. But the compiled sp doesn't know that so it uses an excessively generalized search plan.
Try appending this to the query in your SP (after your ORDER BY clause) to force the generation of a new, hopefully more specific, execution plan.
OPTION (RECOMPILE)
Also, you can tidy up your filter clauses and make them a little less gnarly.
Try this for your text-search cases: Change
sCustomer LIKE CASE
WHEN #bEnableTextSearchFilter = 1
THEN '%' + #sTextSearch + '%'
ELSE sCustomer
END
to
(#bEnableTextSearchFilter <> 1 OR sCustomer LIKE '%' + #sTextSearch + '%')
This will refrain from saying column LIKE column when your filter is disabled, and may save some time.
You can apply the same principle to the rest of your CASE statements too.
Note: the filter pattern column LIKE '%value%' is inherently slow; it can't use an index range scan on column because the text-matching isn't anchored at the beginning of the pattern. Rather it must scan all the values.

How to select into table as string builder instead of using cursor in SQL

I am wondering if there is a faster way to handle the following code in SQL. Currently I am using SQL cursor to do a select and build a string of delimiter values, as a dynamic value of suggestion items?
Here is the snippet of SQL:
begin
set #cursor = cursor for
select top 5 Manufacturer,ManufacturerPartNumber,Description as ManufacturerDescription, CONVERT(money,Price) as Price,fms.Score
from Products_OurProducts_Products_View
open #cursor
fetch next from #cursor
into #CURSOR_Mfr,#CURSOR_Model,#CURSOR_Desc,#CURSOR_Price,#CURSOR_Score
while ##FETCH_STATUS = 0
begin
set #suggestionsStringBuilder += #CURSOR_Mfr + ',' + #CURSOR_Model + ',' + #CURSOR_Desc + ',' + convert(varchar(20),#CURSOR_Price) + ',' + convert(varchar(4),#CURSOR_Score) + '^'
fetch next from #SuggestionsListCursor
into #CURSOR_Mfr,#CURSOR_Model,#CURSOR_Desc,#CURSOR_Price,#CURSOR_Score
end
insert into BASE (Manufacturer, ManufacturerOrig, ManufacturerPartNumber,ManufacturerPArtNumberOrig,ManufacturerDescription, QWDescription, Serial,AssetID,Price,Score,ItemType,MfrFound,ModelFound,trained, SuggestionList,LineNumberIn)
values(#objectORIGMfr,#objectORIGMfr, #objectORIGModel, #objectORIGModel, #objectDescription, #objectDescription, '',#objectAssetID,'0.00',#topMaxScore,'NA','1','0',#trained,#suggestionsStringBuilder,#objectLineNumber)
close #cursor
deallocate #cursor
end
The code above is trying to build a dynamic column of delimiter values such as shown below:
Object Example:
Mfr,
Model,
Price,
Score,
Description,
Suggestions = 'Mfr,Model,Desc,Price^Mfr,Model,Description,Price^
A return model would truly be as follows:
BaseMfr:Fluke,
BaseModel:Tb1,
BaseDescription:'Multi meter item',
BasePrice:120.00,
Suggestions: "Fluke, Tc1, 'Desc', '120.00' ^ 'Fluke', 'T11', 'Desc', 220.00"
Can I do the string builder / cursor section without having to use a looping cursor? The idea behind this is we send in items to be priced. If the item is not found, we then build a list of suggestions to bring back to the user of what they may use in the system or so they can see if there is a typo in the data.
The suggestion list is just the rows found, separating the columns by a "," and separating entities by a "^".
Thanks very much in advance!
Thanks you all for the feedback and I appreciate the help even though I know I had a rough time explaining the question correctly. Thanks to the suggestion from Sean Lange, I was able to be directed in the correct direction and came up with this. Now I will test the performance of it to see if it is better or not. Here is the code:
select
SUBSTRING(
(select top 5
Manufacturer + ',' + ManufacturerPartNumber + ',' + Description +',' + CONVERT(VARCHAR,Price) +',' + CONVERT(varchar,fms.Score) +'^' as [text()]
from Products_OurProducts_Products_View
CROSS APPLY (
select
dbo.FuzzyControlMatch('Flooke', Manufacturer) AS score
) AS fms
order by fms.score desc
FOR XML PATH ('')
), 2, 1000) [Suggestions]
The above code produces the following string:
ARD BROOKE,WB808 10UNF,TORQUE SCREWDRIVER,70.00,50^WARD BROOKE,WB808 1146,TORQUE SCREWDRIVER,70.00,50^WARD BROOKE,WB808 1246,TORQUE SCREWDRIVER,70.00,50^WARD BROOKE,WB808 6UNC,TORQUE SCREWDRIVER,70.00,50^ROKEM TECHNOLOGIES,FIRESET,RC STANDARD,105.00,50^
Now I am not sure if I am handling this the best way, but this is what I was searching for. I will post a comment update to let the feed know if the performance is better or worse.
-Thanks-

How to get rid of Cursor and use UPDATE with SELECT

I believe that the cursor used in this code is the reason for some major performance issues, however I am new to TSQL.
Following script runs on SQL SERVER 2008. I am trying to redo it so I use JOIN statements instead, however I have not been able to do so successfully.
DECLARE AIRAMSDET CURSOR FOR
SELECT BILL, RECIEPT, NAME
FROM Client_Table
WHERE IsProcessed = 1
AND TYPE IN ('Sub','First_Time','Old') AND LEN(BILL) > 1
OPEN AIRAMSDET
FETCH AIRAMSDET into #VARBILL, #VARRECIEPT, #VARNAME
WHILE ##Fetch_Status = 0
BEGIN
UPDATE archieve
SET entry = left(#VARBILL + '- '+ #VARNAME)
WHERE archiveID = #VARBILL
END
It should be something like following
UPDATE ARCHIEVE
SET ENTRY = CT.BILL + '-' + CT.NAME
FROM CLIENT_TABLE CT
WHERE
ARCHIEVE.ARCHIVEID = CT.BILL
AND CT.ISPROCESSED = 1
AND CT.TYPE IN ('Sub','First_Time','Old') AND LEN(BILL) > 1
I have not included LEFT() as its use in your query wasn't very clear. Left takes an integer_expression as its second parameter while you are passing ##VARNAME which most likely is a VARCHAR. Please add that as you deem fit.

SQL HTML email showing blank when one table has no results

The following code sends 2 diferent tables based on an sql query through the function sp_send_dbmail , the catch is , if both tables return results , the email shows up without any problem , perfectly. If one of the tables has NO results, the email comes up completly blank.
How can i fix this?
Thanks
declare #tableHTML NVARCHAR(MAX);
set #tableHTML = N'Este foi o resultado de Faturas Emitidas: <br><br><table border ="1">' +
N'<tr><th>Documento</th></tr>' +
cast (( select td = cc.tx
from cc
for xml path ('tr'),type) as nvarchar(max)) +
N' </table><table border ="1"><tr><th>Valor Total Vencido</th></tr>'
+
cast (( select td = fx.tc
from fx
for xml path ('tr'),type) as nvarchar(max)) +
N'</table>';
EXEC sp_send_dbmail
#profile_name ='xx_SqlMail',
#recipients ='ccccc#hotmail.com',
#subject ='Resumo',
#body =#tableHTML,
#body_format='HTML';
I would suspect that part of your query is returning a NULL value. Concatenating any value with a NULL will always result in NULL.
SELECT 'A' + NULL + 'B' will return NULL.
As you are doing multiple concatenations it would mean that if any value is NULL then #tableHTML will be NULL. Try wrapping your selects in an ISNULL().
select ISNULL(td, '') = cc.tx ...
Any table in your concatenation that returns a NULL will make the entire concatenation NULL.
To resolve this, just wrap each section that could potentially be NULL with an ISNULL().
I had a similar issue where I was running two separate queries and building two tables that I wanted to include in the body of the email. One would occasionally return no values and the email would come back blank. Using ISNULL fixed it for me.
See the code below for an example of what I did:
set #tablesHTML = **ISNULL**(#tableOneHTML,'NO RESULTS')
+ **ISNULL**(#tableTwoHTML,'NO RESULTS')
exec XXXXXX.[XXX].[sp_send_dbmail]
#profile_name='Mail'
,#recipients = #EmailRecipients
,#copy_recipients= #EmailCopyRecipients
,#subject = 'Email Subject Here'
,#body = #tablesHTML
,#body_format = 'HTML'

Logic to prepare SQL statements dynamically

So i have a requirement where I need to read through records of all records of a file and insert them into another file if they meet a set of rules which are described in another table as shown below..
A record after it has been read from the first file has to meet all the sequences of at least one Rule to make it eligible to be written into the Second table.
For example once a record is read from CAR file, the rules below have to be checked till all sequences of atleast one rule set is satisfied. For this I was planning to Create a dynamic SQL program something of this sort. But this does not work as Prepared SQL does not support host variables.
If any body can suggest or provide any guidance on how to create SQL statemtns dynamically and check if records satisfy the required rules for them to be entered into the second file, it would be great
So basically what I am looking for is once I select a field from a table, how do I store it somehere to do further validation and checking.
Update
:
Based on the intelligent advice from Danny117, I have come up with the below code:
H Option(*NoDebugIO:*SrcStmt)
D RULEDS E DS EXTNAME(RULESTABLE)
D MAXRUL S 1 0
D MAXSEQ S 1 0
D STMT S 512
D WHERESTMT S 512 INZ('')
D FullSqlStmt S 512 INZ('')
D RULINDEX S 1 0 INZ(1)
D SEQINDEX S 1 0 INZ(1)
D APOS C CONST('''')
/Free
Exec SQL SELECT MAX(RULENO)INTO :MAXRUL FROM RULESTABLE;
Exec SQL DECLARE RULCRS CURSOR FOR SELECT * FROM RULESTABLE;
Exec SQL OPEN RULCRS;
Exec SQL FETCH RULCRS INTO :RULEDS;
DoW (Sqlcod = 0 AND RULINDEX <= MAXRUL);
Exec SQL SELECT MAX(SEQNO) INTO :MAXSEQ FROM RULESTABLE
WHERE RULENO=:RULINDEX ;
DoW (SEQINDEX <= MAXSEQ);
If (Position <> '');
Field = 'SUBSTR('+%Trim(Field)+','+%Trim(Position)+','
+'1'+')';
EndIf;
WhereStmt = %Trim(WhereStmt) + ' ' + %Trim(field)+ ' ' +
%Trim(condition) + ' ' + APOS + %Trim(Value) + APOS;
If (SeqIndex < MaxSeq);
WhereStmt = %Trim(WhereStmt) + ' AND ';
EndIf;
Exec SQL FETCH NEXT FROM RULCRS INTO :RULEDS;
SeqIndex = SeqIndex + 1;
EndDo;
FullSqlStmt = %Trim('INSERT INTO ITMRVAT SELECT * +
FROM ITMRVA WHERE '+ %Trim(WhereStmt));
Exec SQL Prepare InsertStmt from :FullSqlStmt;
Exec SQL EXECUTE InsertStmt;
RulIndex = RulIndex + 1;
EndDo;
This produces SQL statement as shown below which is what I want. Now let me go ahead and look at the other parts of the code.
> EVAL FullSqlStmt
FULLSQLSTMT =
....5...10...15...20...25...30...35...40...45...50...55...60
1 'INSERT INTO ITMRVAT SELECT * FROM ITMRVA WHERE STID = 'PLD' '
61 'AND ENGNO LIKE '%415015%' AND SUBSTR(ENGNO,1,1) = 'R' AND SU'
121 'BSTR(ENGNO,5,1) = 'Y' '
181 ' '
241 ' '
301 ' '
361 ' '
421 ' '
481 ' '
But the issue is now as I mentioned in my comment to Danny, how to handle if a new rule involving second table is specified..
Embedded SQL does allow for 'dynamic statements' in ILE languages. You are able to have a query within a character field and then pass it into the Embedded SQL.
Dcl-S lQuery Varchar(100);
lQuery = 'SELECT * FROM CUST';
EXEC SQL
PREPARE SCust FROM :lQuery;
EXEC SQL
DECLARE SearchCust CURSOR FOR SCust;
//Continue working with cursor..
You may want to just prepare, execute and return a result set:
lQuery = 'SELECT * FROM CUST WHERE ID = ' + %Char(CustID);
EXEC SQL
PREPARE SCust FROM :lQuery;
DECLARE c1 CURSOR FOR SCust;
OPEN c1;
FETCH c1 INTO :CustDS;
CLOSE c1;
Optional extra: You may also want to use field markers (?) in your query.
//'SELECT * FROM CUST WHERE CUSTID = ?';
EXEC SQL OPEN SearchCust USING :CustID;
//'INSERT INTO CUST VALUES(?,?)';
EXEC SQL EXECUTE CUST USING :CustID;
You have to translate the rules into a join statement or a where clause. The join statement is more complex so go that route.
If you were smart (and you are) consider saving the rules as a SQL clause that you can join or use in a where clause. Its infinitely flexible this way a more modern design.
rule 1 / car.year = 1990 and car.engno like '%43243%' and substring(car.vin,12,1) = 'X'
eval statement =
insert into sometable
Select car.* from car
join sysibm.sysdummy1
on car.year = 1990
and car.engno lile '%43243%'
...etc on to rule 2 starting with "OR"
or car.year = PLD
and car.engno like '%1234%'
...etc other rules starting with "OR"
exec immediate statement