I am working on an old auditing stored procedure that is taking a long time to get data causing timeouts on the system. We have managed to get the time down from over 20 minutes to running just over a minute, this is still too long.
I am running on SQL Server 2008 R2.
My question is is there anything I could do to improve the speed of the query? In particular the temporary tables and the a bulk insert statement.
SELECT
dbo.[Audit Result Entry Detail].PK_ID,
+ every other value in the table (all values required)
INTO
#temp5
FROM
dbo.[Audit Result Entry Detail]
An INNER JOIN then occurs on dbo.[Audit Register] another select occurs and added to another temporary table #result is created.
Result temporary table gets the old and new result values for comparison. Below I have provided what event occurs on #temp5. Note these are just snippets; the stored procedure is way too big to post everything.
SELECT
RED.[FK_RegisterID],
total of a 106 rows are selected here :(
FROM
#temp5 AS RED
LEFT JOIN
#temp5 AS OLD ON RED.IdentityColumn = OLD.IdentityColumn
LEFT JOIN
[Audit Register] AS REG ON REG.PK_ID = RED.FK_RegisterID
SELECT MAX(PK_ID)
OVER (PARTITION BY FK_RegisterID) AS MAX_PK_ID
FROM #temp5
DROP TABLE #temp5
The second part of my question is a bulk insert into this table created at the very top of the stored procedure
DECLARE #AuditOverView TABLE
(
FK_RegisterID INT,
Audit_Date DATETIME,
ContextUser NVARCHAR(30),
Original_Value NVARCHAR(255),
New_Value NVARCHAR(255),
Assay NVARCHAR(200),
Inst NVARCHAR(40),
LotLevel NVARCHAR(3),
Lot_ExpiryDate NVARCHAR(10),
Lot_Number NCHAR(50),
Audit_Type NVARCHAR(10),
Partnumber INT,
[Type] NVARCHAR(50),
SubType NVARCHAR(50)
)
The insert statement below:
INSERT INTO #AuditOverView
SELECT DISTINCT
t.FK_RegisterID,
t.Audit_Date,
t.ContextUser,
CONVERT(NVARCHAR, Original_Date_Run, 127) AS Original_Value,
CONVERT(NVARCHAR, New_Date_Run, 127) AS New_Value,
t.Assay AS 'Assay',
Instrument AS 'Inst',
'1' AS 'LotLevel',
Lot1_Expiry_Date AS 'Lot_ExpiryDate',
Lot1Number AS 'Lot_Number',
t.Audit_Type,
part_number AS Partnumber,
'QC Result' AS [Type],
'Result Date' AS SubType
FROM
#Result AS t
INNER JOIN
(SELECT MAX(Audit_Date) AS DATE, FK_RegisterID
FROM #Result
GROUP BY FK_RegisterID) dt ON t.FK_RegisterID = dt.FK_RegisterID
AND t.Audit_Date = dt.DATE
WHERE
RTRIM(Lot1Number) != ''
AND (CONVERT(NVARCHAR, Original_Date_Run, 120) != CONVERT(NVARCHAR, New_Date_Run, 120)
OR t.Audit_Type = 'i'
OR t.Audit_Type = 'd')
This insert statement occurs about 50 times and the only line changes occur are:
'Result Date' AS SubType
CONVERT(NVARCHAR, Original_Date_Run, 120) != CONVERT(NVARCHAR, New_Date_Run, 120)
I can provide more information if needed. Any help would be much appreciated to improve performance.
Related
As I understand it, you cannot create a view with variables. The dilemma is I have to do some datediff on values. Our third party reporting software can only do basic Selects on Tables or views. I cannot call up any stored procedures or set variables there.
My original source data looks select * from tblquotestatuschangelog
results in this I would like to see these results
Quotenumber UpdatedOn Status UpdatedBy
----------------------------------------------
100001 04102019 Open domain/user
100001 04132019 Closed domain/user
I have done a pivot on this data to get the results in the desired format with this query. (There are more status types than in this example, and more can be added via the CRM we use) (Q1)
DECLARE #cols AS NVARCHAR(MAX),
#query AS NVARCHAR(MAX);
select #cols = STUFF((SELECT distinct ',' + QUOTENAME(T1.STATUS)
FROM tblCglQuoteStatusChangeLog as T1
FOR XML PATH(''), TYPE
).value('.', 'NVARCHAR(MAX)')
,1,1,'')
set #query = 'SELECT quotenumber, ' + #cols + ' from
(
select quotenumber, status, updatedon, updatedby
from tblcglquotestatuschangelog
where updatedon > 2019-04-01
) x
pivot
(
max(updatedon)
for status in (' + #cols + ')
) p '
execute #query
To get these results
Quotenumber Open Closed
----------------------------------------------
100001 04102019 04132019
I would like to call (Q1) as a View to perform
Select
QuoteNumber,
case when datediff(day,Open, Closed) = 0 then Cast('SAMEDAY' AS NVARCHAR(20)) else cast(datediff(day,openedon,updatedon) as NVarCHAR(20)) end as TotalAge,
datediff(day,Open,SentToCustomer) as Stage1,
datediff(day,SentToCustomer,) as Stage2,
From V1
Any help or alternate direction to achieve results would be much appreciated.
You can use conditional aggregation to achieve your results - as long as you want to rely on the assumptions about the shape of your data. Those are your answers to my questions - that the tuple is unique and that there are only 2 status values. If you have more status values (and it smells like you do) you can extend the logic
use tempdb;
set nocount on;
go
if object_id('v1') is not null
drop view v1;
if object_id('chglog') is not null
drop table chglog;
go
create table chglog (IdNumber int not null, UpdatedOn date not null, Status varchar(10) not null, UpdatedBy varchar(20) not null
constraint chg_unique unique (IdNumber, Status),
constraint chg_check check (Status in ('Open', 'Closed'))
);
insert chglog(IdNumber, UpdatedOn, Status, UpdatedBy)
values (100001, '20190410', 'Open', 'domain/user'), (100001, '20190413', 'Closed', 'domain/user'),
(9999, '20190401', 'Open', 'zork'),
(99001, '20190402', 'Open', 'bob'),
(99001, '20190402', 'Closed', 'alice')
;
go
-- setup complete, now create a view to do a hard-coded pivot
create view v1 as
select IdNumber,
max(case Status when 'Open' then UpdatedOn else null end) as 'Open',
max(case Status when 'Closed' then UpdatedOn else null end) as 'Closed'
from chglog
group by IdNumber;
go
-- test the view
select * from v1
order by IdNumber;
-- now the query that you wanted to write/use
Select
IdNumber,
case when datediff(day, [Open], Closed) = 0 then N'SAMEDAY' else cast(datediff(day, [Open], Closed) as nvarchar(20)) end as TotalAge
-- datediff(day,Open,SentToCustomer) as Stage1,
-- datediff(day,SentToCustomer,) as Stage2,
from v1
order by IdNumber;
I'll point out that your "table" contained IdNumber but your last query referenced QuoteNumber (as well as other columns you did not mention). I just ignored some and guesssed some. This is just old-style pivoting based on hard-coded values. So long as you know ahead of time the values you need to consider when pivoting and you know that the domain is unchanging (or don't care about other values), this will what you asked.
The question you should consider is whether you should care about other status values that are added via your CRM after you create this view. But you're kinda stuck with the 3rd party reporting software anyways. A reporting tool that can't use anything but a table or view as a source of data seems primitive (to say the least).
I have A stored procedure that is giving me an unexpected Token; ORDER expecting semicolon when I have this statement near the end when I try to execute it.
select year from #temp where year is not null ORDER BY year DESC;
if I remove the ORDER BY year DESC; the procedure works correctly.
I've tried every way possible to sort the resulting table in descending order. I'm fairly new to SQL so I'm sure its something simple. TIA.
// --------- full stored procedure ------ //
ALTER PROCEDURE GetYearForExhaustCatalog
(
CatCodeString Memo,
Year CHAR ( 4 ) OUTPUT
)
BEGIN
/*
EXECUTE PROCEDURE GetYearForExhaustCatalog('(e.catalogcode= ''2182'')');
EXECUTE PROCEDURE GetYearForExhaustCatalog('');
*/
DECLARE #CatCodeString string;
DECLARE #SQL string;
#CatCodeString = (SELECT CatCodeString FROM __input);
if #CatCodeString IS NULL or #CatCodeString = '' then
select e2.year,
(SELECT top 1 e2.year
FROM eenginecatalog e LEFT JOIN exhaustengine e2 ON e2.app_no=e.app_no)
as year
into #temp
from Exhaustengine e2;
select year from #temp where year is not null
GROUP BY year
ORDER BY year DESC;
else
#SQL =
'select e2.year, '+
'(SELECT top 1 e2.year '+
'FROM eenginecatalog e LEFT JOIN exhaustengine e2 ON e2.app_no=e.app_no and '+
#CatCodeString +' ) '+
'as year '+
'into #temp '+
'from Exhaustengine e2; '+
'select year from #temp where year is not null '+
'GROUP BY year '+
'ORDER BY year DESC ';
execute immediate #SQL;
end;
insert into __output
select year from #temp where year is not null ORDER BY year;
drop table #temp;
END;
It seems that ADS does not like the ORDER BY clause when inserting into the special __output table.
This does not work as well:
CREATE PROCEDURE MyProcedure(Year INTEGER OUTPUT)
BEGIN
CREATE TABLE #tmp ("Year" INTEGER);
INSERT INTO #tmp ("Year") VALUES (2019);
INSERT INTO #tmp ("Year") VALUES (2017);
INSERT INTO #tmp ("Year") VALUES (2018);
INSERT INTO
__output
SELECT
"Year"
FROM #tmp
ORDER BY
"Year";
DROP TABLE #tmp;
END;
It fails with the same error message you got:
poQuery: Error 7200: AQE Error: State = 42000; NativeError = 2117; [SAP][Advantage SQL Engine]Unexpected token: ORDER -- Expecting semicolon. -- Location of error in the SQL statement is: 269 (line: 15 column: 1)
As a workaround you can create another temporary table that has the result sorted:
CREATE PROCEDURE MyProcedure(Year INTEGER OUTPUT)
BEGIN
CREATE TABLE #tmp ("Year" INTEGER);
INSERT INTO #tmp ("Year") VALUES (2019);
INSERT INTO #tmp ("Year") VALUES (2017);
INSERT INTO #tmp ("Year") VALUES (2018);
SELECT
*
INTO #sorted
FROM #tmp
ORDER BY
"Year"
;
INSERT INTO
__output
SELECT
"Year"
FROM #sorted;
DROP TABLE #sorted;
DROP TABLE #tmp;
END;
This works without errors and the data is sorted.
The __output table is not the culprit. In SQL standard, ORDER BY is not allowed in sub-queries in general. The reason is that sorting a set of rows should have no effect during the internal resolution of a query. It is only useful in the final result. In pure SQL sense, the only action that guarantees the result is sorted is the ORDER BY on the final result.
If you follow this logic, the possible alternative is not to try to put the data into the __output table in sorted order but to make final output sorted with the following:
SELECT * FROM (EXECUTE PROCEDURE MyProcedure(inParam)) t ORDER BY t.year
Very confused as to why the final select * from #tempTbl is empty.
When I run this, everything else works fine, but I need the results in a temp table to continue with the next step of the stored procedure. I've been searching every forum possible including this one, and cannot fix this issue. Any help would be greatly appreciated.
CREATE TABLE #tempTbl
(
POID VARCHAR(15),
UPC VARCHAR(15),
QtyOrdered int,
DateOrdered date,
Closed bit
)
insert into #tempTbl (POID, UPC, QtyOrdered, DateOrdered, Closed)
exec dbo.sp_executesql #as4sql;
--exec (#as4sql)
select * from #tempTbl
Update:
Here's my set:
set #as4sql = 'INSERT INTO tblPODetail
(POID, UPC, QtyOrdered, DateOrdered, Closed)
(SELECT pod.PONumber, pod.UPC, pod.QtyOrdered, 1, 0
FROM [awx].[dbo].[POLines] pol
JOIN [awx].[dbo].[PODetails] pod
ON pol.PONumber = pod.PONumber
LEFT JOIN tblPODetail p
ON p.POID = pod.PONumber
WHERE p.POID IS NULL...
I have wrote this cursor for commission report. What happens is commission comes in one table, the records are another table. I match two based on certain critera (there is not exact match available). The problem is there are duplicates where records exist. When I match commission with the records table, it can result picking up these duplicates. Thus the rep gets paid more. On the other hand, there are duplicates in commission table also but those are valid beause they simple mean an account got paid for 2 months.
I wrote this query but it takes 5+ minutes to run. I have 50,000 records in records table and 100,000 in commission table. Is there any way I an improve this cursor?
/* just preparation of cursor, this is not time consuming */
CREATE TABLE #result
(
repid INT,
AccountNo VARCHAR(100),
supplier VARCHAR(15),
CompanyName VARCHAR(200),
StartDate DATETIME,
EndDate DATETIME,
Product VARCHAR(25),
commodity VARCHAR(25),
ContractEnd DATETIME,
EstUsage INT,
EnrollStatus VARCHAR(10),
EnrollDate DATETIME,
ActualEndDate DATETIME,
MeterStart DATETIME,
MeterEnd DATETIME,
ActualUsage INT
)
DECLARE #AccountNo VARCHAR(100)
DECLARE #supplier VARCHAR(10)
DECLARE #commodity VARCHAR(15)
DECLARE #meterstart DATETIME
DECLARE #meterEnd DATETIME
DECLARE #volume FLOAT
DECLARE #RepID INT
DECLARE #Month INT
DECLARE #Year INT
SET #repID = 80
SET #Month = 1
SET #year = 2012
/* the actual cursor */
DECLARE commission_cursor CURSOR FOR
SELECT AccountNo,
supplier,
commodity,
meterStart,
MeterEnd,
Volume
FROM commission
WHERE Datepart(m, PaymentDate) = #Month
AND Datepart(YYYY, PaymentDate) = #Year
OPEN commission_cursor
FETCH next FROM commission_cursor INTO #AccountNo, #supplier, #commodity, #MeterStart, #MeterEnd, #Volume;
WHILE ##fetch_status = 0
BEGIN
IF EXISTS (SELECT id
FROM Records
WHERE AccountNo = #AccountNo
AND supplier = #supplier
AND Commodity = #commodity
AND RepID = #repID)
INSERT INTO #result
SELECT TOP 1 RepID,
AccountNo,
Supplier,
CompanyName,
[Supplier Start Date],
[Supplier End Date],
Product,
Commodity,
[customer end date],
[Expected Usage],
EnrollStatus,
ActualStartDate,
ActualEndDate,
#meterstart,
#MeterEnd,
#volume
FROM Records
WHERE AccountNo = #AccountNo
AND supplier = #supplier
AND Commodity = #commodity
AND RepID = #repID
AND #MeterStart >= Dateadd(dd, -7, ActualStartDate)
AND #meterEnd <= Isnull(Dateadd(dd, 30, ActualEndDate), '2015-12-31')
FETCH next FROM commission_cursor INTO #AccountNo, #supplier, #commodity, #MeterStart, #MeterEnd, #Volume;
END
SELECT *
FROM #result
/* clean up */
CLOSE commission_cursor
DEALLOCATE commission_cursor
DROP TABLE #result
I have read answer to How to make a T-SQL Cursor faster?, for that what I get is rewrite this query in table form. But I do have another query which uses join and is lightening fast. The problem is, it can not differentiate between the dups in my records table.
Is there anything I can do to make is faster. This is primary question. If not, do you have any alternative way to do it.
I specifically need help with
Will using Views or store procedure help
I there a way I can use cache in Cursor to make it faster
Any other option in syntax
The very first option is to set the least resource intensive options for your cursor:
declare commission_cursor cursor
local static read_only forward_only
for
Next is to investigate whether you need a cursor at all. In this case I think you can do the same with a single pass and no loops:
;WITH x AS
(
SELECT
rn = ROW_NUMBER() OVER (PARTITION BY r.AccountNo, r.Supplier, r.Commodity, r.RepID
ORDER BY r.ActualEndDate DESC),
r.RepID,
r.AccountNo,
r.Supplier,
r.CompanyName,
StartDate = r.[Supplier Start Date],
EndDate = r.[Supplier End Date],
r.Product,
r.Commodity,
ContractEnd = r.[customer end date],
EstUsage = r.[Expected Usage],
r.EnrollStatus,
EnrollDate = r.ActualStartDate,
r.ActualEndDate,
c.MeterStart,
c.MeterEnd,
ActualUsage = c.Volume
FROM dbo.commission AS c
INNER JOIN dbo.Records AS r
ON c.AccountNo = r.AccountNo
AND c.Supplier = r.Supplier
AND c.Commodity = r.Commodity
AND c.RepID = r.RepID
WHERE
c.PaymentDate >= DATEADD(MONTH, #Month-1, CONVERT(CHAR(4), #Year) + '0101')
AND c.PaymentDate < DATEADD(MONTH, 1, CONVERT(CHAR(4), #Year) + '0101')
AND r.RepID = #RepID
)
SELECT RepID, AccountNo, Supplier, CompanyName, StartDate, EndDate,
Product, Commodity, ContractEnd, EstUsage, EnrollStatus, EnrollDate,
ActualEndDate, MeterStart, MeterEnd, ActualUsage
FROM x
WHERE rn = 1 --ORDER BY something;
If this is still slow, then the cursor probably wasn't the problem - the next step will be investigating what indexes might be implemented to make this query more efficient.
Temp tables are your friend
The way I solved my problem, merging data from two tables, removed duplicates in complex fashion and everything extremely fast was to use temporary table. This is what I did
Create a #temp table, fetch the merged data from both the tables. Make sure you include ID fields in both tables even if you do not required it. This will help remove duplicates.
Now you can do all sort of calculation on this table. Remove duplicates from table B, just remove duplicate table B IDs. Remove duplicates from table A, just remove duplicate table A Ids. There is more complexity to the problem but at least this is probably the best way to solve your problem and make it considerably faster if cursors are too expensive and takes considerable time to calculate. In my case it was taking +5 min. The #temp table query about about 5 sec, which had a lot more calculations in it.
While applying Aaron solution, the cursor did not get any faster. The second query was faster but it did not give me the correct answer, so finally I used temp tables. This is my own answer.
I have a SP that inserts records into a temp table, then selects the records and returns them. The SQL is this.
I troubleshot it by removing the INSERT INTO statement, and minimizing the SQL. The culprit is the SELECT * FROM #Worklist1. No idea why this does not work. I upgraded (just now) to latest version of Pervasive server ver 10 if that helps, but this issue was in 10.3 and its still there. Must be missing something.
CREATE PROCEDURE "Connect_Workflow"(
:StartDate DATETIME, :EndDate DATETIME)
RETURNS(Patient varchar(100) ,
AccessionNo varchar(25)
);
BEGIN
CREATE TABLE #WorkFlow1
(Patient varchar(100) null,
AccessionNo varchar(25) null
);
INSERT INTO #Workflow1(
SELECT
rtrim(p.LastName),--+ '^' + rtrim(p.FirstName) + isnull('^' + rtrim(p.Initial), ''),
v.VisitID -- equiv to EncounterID
FROM visit v
join patient p on v.patientnumber = p.patientnumber
WHERE v.VisitYY = '99'
);
SELECT * FROM #WorkFlow1;
DROP TABLE #Workflow1;
END
Update: After commenting out the SELECT * FROM #Worklist1; it still gives a invalid table error. If I remove the INSERT INTO and the SELECT * then finally the error is gone. Must be error in referencing the table.
Remove the DROP TABLE #Workflow1; from your query.
I believe it's dropping the table before the SP returns the data.
Okay i figured it out. Although the procedure should work fine, in fact Pervasive recommends something like this. use SELECT INTO
CREATE PROCEDURE "Connect_Workflow"(
:StartDate DATETIME, :EndDate DATETIME)
RETURNS(Patient varchar(100) ,
AccessionNo varchar(25)
);
BEGIN
SELECT
rtrim(p.LastName),--+ '^' + rtrim(p.FirstName) + isnull('^' + rtrim(p.Initial), ''),
v.VisitID -- equiv to EncounterID
FROM visit v
INTO #Workflow1
join patient p on v.patientnumber = p.patientnumber
WHERE v.VisitYY = '99'
);
SELECT * FROM #WorkFlow1;
END