I have created 2 temp tables, #Table1 being the source and #Table2 being the output table. I have to traverse through each row on #Table1 to make the necessary calculations before inserting each row into #Table2 because the calculation is also dependent on previous rows in #Table2.
I have all the rows in the order I want, and have all the calculations working perfectly. The only problem is that when I go through my 30,000 rows of test data, it takes over 30 minutes. Which means when I expand the query to the full scope of over 2,000,000 rows it will take far too long.
Example Tables:
My solution on traversing through each record in #Table1 to come up with #Table2 is provided below.
exec [database_name].dbo.drop_if_exists '#Table2'
create table #Table2 (ID int, Shipment varchar(8), Delivery int, Arrival datetime, ETA datetime, ETA_TimeStamp datetime, Compliant int)
declare #z int
declare #currentShipment varchar(8)
declare #currentStop int
declare #previousID int
declare #previousETA datetime
set #z = 1
while #z <= (Select max(ID) from #Table1)
Begin
Set #currentShipment = (Select Shipment from #Table1 where ID = #z)
Set #currentStop = (Select Delivery from #Table1 where ID = #z)
Set #previousID = (Select max(ID) from #Table2 where Shipment = #currentShipment AND Delivery = #currentStop AND ID < #Z AND Compliant = 1)
Set #previousETA = (Select ETA from #Table2 where ID = #previousID)
if (Select count(Shipment) from #Table2 where Shipment = #currentShipment AND Delivery = #currentStop AND ID < #Z AND Compliant = 1) = 0
BEGIN
insert into #Table2 Select ID
, Shipment
, Delivery
, Arrival
, ETA
, ETA_Timestamp
, Compliant_Prelim as [Compliant]
from #Table1
where ID = #Z
END
if (Select count(Shipment) from #Table2 where Shipment = #currentShipment AND Delivery = #currentStop AND ID < #Z AND Compliant = 1) > 0
BEGIN
insert into #Table2 Select ID
, Shipment
, Delivery
, Arrival
, ETA
, ETA_Timestamp
, case when (Select Compliant_Prelim from #Table1 where ID = #z) = 1
AND ETA_TimeStamp <= dateadd(hh, -1, #previousETA)
then 1
else 0 end as [Compliant]
from #Table1
where ID = #Z
END
Set #z = #z + 1
End
Select * from #Table2
How do I speed up the insert statement given this example?
The performance isn't in the "insert" itself. It is in the queries used to create the data (the select part of the insert). To fix this, consider the following.
Make sure that the id column in #table1 has an index:
create index on #table1(id);
And make sure that #table2(shipment) has an index:
create index on #table2(shipment);
Related
normally it works fine (it recalculates time to time production based on daily meter readings , the problem starts when other apps are inserting data into [dbo].[MeterReading] ( at least I belive this is the source of deadlocks), what would you reccomend to avoid exceptions ?
BEGIN
-- SET NOCOUNT ON added to prevent extra result sets from
-- interfering with SELECT statements.
SET NOCOUNT ON;
declare #ResultVar float;
DECLARE #Val1 TABLE
(
RowID INT IDENTITY ( 1 , 1 ),
InputId int,
TimeStampUtc1 smalldatetime,
Val1 float
);
DECLARE #Val2 TABLE
(
RowID INT IDENTITY ( 1 , 1 ),
InputId int,
TimeStampUtc2 smalldatetime,
Val2 float
);
WITH CTE
AS
(
SELECT [Val],m.InverterInputId, m.[TimeStampUtc]
,ROW_NUMBER() OVER(PARTITION BY m.InverterInputId ORDER BY m.[TimeStampUtc] DESC) AS RN
FROM [dbo].[MeterReading] AS m
inner join InverterInput AS ii on m.InverterInputId = ii.InverterInputId
inner join Inverter AS i on ii.InverterId = i.InverterId
where ii.InputName = 'TotalYield' and i.PlantId = #plantId and m.[TimeStampUtc] >= #from and m.[TimeStampUtc] <= #to
)
INSERT INTO #Val1 (Val1,InputId,TimeStampUtc1 ) Select [Val],[InverterInputId],[TimeStampUtc] FROM CTE WHERE RN = 1;
WITH CTE
AS
(
SELECT [Val],m.InverterInputId, m.[TimeStampUtc]
,ROW_NUMBER() OVER(PARTITION BY m.InverterInputId ORDER BY m.[TimeStampUtc] ASC) AS RN
FROM [dbo].[MeterReading] AS m
inner join InverterInput AS ii on m.InverterInputId = ii.InverterInputId
inner join Inverter AS i on ii.InverterId = i.InverterId
where ii.InputName = 'TotalYield' and i.PlantId = #plantId and m.[TimeStampUtc] >= #from and m.[TimeStampUtc] <= #to
)
INSERT INTO #Val2 (Val2,InputId,TimeStampUtc2) Select [Val],[InverterInputId],[TimeStampUtc] FROM CTE WHERE RN = 1;
--select Val1 , Val2, Val1-Val2 as Result, [#Val1].InputId,[#Val1].TimeStampUtc1,[#Val2].TimeStampUtc2 from #Val1
select Sum(Val1-Val2) as Result from #Val1
inner join #Val2 on [#Val1].InputId = [#Val2].InputId
END
I want to find PC model pairs with the same speed and memory. These pairs are listed only once.
Here is my data and desired results:
desired results: https://i.imgur.com/cJBdrvq.png
data: https://i.imgur.com/t8LiJ7G.png
I did get results but the query is too long, I know there is a shorter way. Hope everyone help me.
Here is my query
DECLARE #FOR INT = 1
DECLARE #SPEED INT
DECLARE #RAM INT
DECLARE #MODEL INT
DECLARE #LIST TABLE(SPEED INT, RAM INT)
DECLARE #LISTMODEL TABLE(MODEL INT)
DECLARE #RESULT TABLE(PC1 INT, PC2 INT)
DECLARE #RESULTREAL TABLE(COUPLE NVARCHAR(20), SPEED INT, RAM INT)
DECLARE #COUNT INT
WHILE(1=1)
BEGIN
IF(NOT EXISTS(SELECT TOP(1) SPEED FROM #LIST))
BEGIN
INSERT #LIST(SPEED,RAM)
SELECT speed,ram
FROM VW_count
END
BREAK
END
SET #COUNT = (SELECT COUNT(SPEED) FROM #LIST)
WHILE #FOR <= #COUNT
BEGIN
SET #SPEED = (SELECT KETQUA.SPEED FROM (SELECT ROW_NUMBER() OVER (ORDER BY SPEED) AS STT, SPEED FROM #LIST) AS KETQUA WHERE KETQUA.STT = 1)
SET #RAM = (SELECT KETQUA.RAM FROM (SELECT ROW_NUMBER() OVER (ORDER BY RAM) AS STT, RAM FROM #LIST) AS KETQUA WHERE KETQUA.STT = 1)
IF #SPEED IS NULL
BEGIN
BREAK
END
ELSE
BEGIN
IF(EXISTS(SELECT speed FROM PC WHERE speed = #SPEED AND ram = #RAM))
BEGIN
INSERT #LISTMODEL(MODEL)
SELECT model FROM PC WHERE speed = #SPEED AND ram = #RAM
INSERT #RESULT(PC1,PC2)
SELECT DISTINCT L1.MODEL, L2.MODEL FROM #LISTMODEL AS L1 , #LISTMODEL AS L2
INSERT #RESULTREAL(COUPLE,SPEED,RAM)
SELECT CONCAT(R1.PC1, ', ', R1.PC2), #SPEED, #RAM FROM #RESULT R1 WHERE R1.PC1 > R1.PC2 OR NOT EXISTS (SELECT * FROM #RESULT R2 WHERE R2.PC1 = R1.PC2 AND R1.PC2 = R2.PC1)
END
DELETE #RESULT
DELETE #LISTMODEL
END
SET #FOR = #FOR + 1
DELETE TOP(1) FROM #LIST
CONTINUE
END
SELECT * FROM #RESULTREAL
It's normally a bad idea to use procedural statements when you could also use set-based logic.
You could make a SELECT query using a self-join on table [PC] based on the value of the [speed] and [ram] fields, but where the [model] value of the second table is larger than that of the first table. Something like this:
DECLARE #RESULTREAL TABLE(COUPLE NVARCHAR(20), SPEED INT, RAM INT);
INSERT INTO #RESULTREAL
SELECT
CAST(T1.[model] AS NVARCHAR) + N', ' + CAST(T2.[model] AS NVARCHAR),
T1.[speed],
T1.[ram]
FROM
[PC] AS T1
INNER JOIN [PC] AS T2 ON
T2.[speed] = T1.[speed] AND
T2.[ram] = T1.[ram]
WHERE
T2.[model] > T1.[model];
I am completely new to stored procedure. This time, I need to create a stored procedure in MS SQL.
Let's say I have the following table.
Table name: ListOfProducts
--------------------------
SomeID, ProductID
34, 4
35, 8
35, 11
How do I pass in a SomeID. Use this SomeID to select a recordset from table, ListOfProducts. Then loop through this record set.
Let's say I pass in SomeID = 35.
So, the record set will return 2 records with SomeID 35. In the loop, I will get ProductID 8 and 11, which will be used to do another select from another table.
The stored procedure should return the results from the 2nd select.
How can I do this in MS SQL stored procedure?
Sorry, for this newbie question. Thanks for any help.
If you want looping through the records. You can do like:
--Container to Insert Id which are to be iterated
Declare #temp1 Table
(
tempId int
)
--Container to Insert records in the inner select for final output
Declare #FinalTable Table
(
Id int,
ProductId int
)
Insert into #temp1
Select Distinct SomeId From YourTable
-- Keep track of #temp1 record processing
Declare #Id int
While((Select Count(*) From #temp1)>0)
Begin
Set #Id=(Select Top 1 tempId From #temp1)
Insert Into #FinalTable
Select SomeId,ProductId From ListOfProducts Where Id=#Id
Delete #temp1 Where tempId=#Id
End
Select * From #FinalTable
There is probably no point in writing an explicit loop if you don't need to preform some action on the products that can't be done on the whole set. SQL Server can handle stuff like this much better on its own. I don't know what your tables look like, but you should try something that looks more like this.
CREATE PROC dbo.yourProcName
#SomeID int
AS
BEGIN
SELECT
P.ProductId,
P.ProductName
FROM
Product P
JOIN
ListOfProducts LOP
ON LOP.ProductId = P.ProductId
WHERE
LOP.SomeId = #SomeID
END
I had to do something similar in order to extract hours from a select resultset start/end times and then create a new table iterating each hour.
DECLARE #tCalendar TABLE
(
RequestedFor VARCHAR(50),
MeetingType VARCHAR(50),
RoomName VARCHAR(MAX),
StartTime DATETIME,
EndTime DATETIME
)
INSERT INTO #tCalendar(RequestedFor,MeetingType,RoomName,StartTime,EndTime)
SELECT req as requestedfor
,meet as meetingtype
,room as rooms
,start as starttime
,end as endtime
--,u.datetime2 as endtime
FROM mytable
DECLARE #tCalendarHours TABLE
(
RequestedFor VARCHAR(50),
MeetingType VARCHAR(50),
RoomName VARCHAR(50),
Hour INT
)
DECLARE #StartHour INT,#EndHour INT, #StartTime DATETIME, #EndTime DATETIME
WHILE ((SELECT COUNT(*) FROM #tCalendar) > 0)
BEGIN
SET #StartTime = (SELECT TOP 1 StartTime FROM #tCalendar)
SET #EndTime = (SELECT TOP 1 EndTime FROM #tCalendar)
SET #StartHour = (SELECT TOP 1 DATEPART(HOUR,DATEADD(HOUR,0,StartTime)) FROM #tCalendar)
SET #EndHour = (SELECT TOP 1 DATEPART(HOUR,DATEADD(HOUR,0,EndTime)) FROM #tCalendar)
WHILE #StartHour <= #EndHour
BEGIN
INSERT INTO #tCalendarHours
SELECT RequestedFor,MeetingType,RoomName,#StartHour FROM #tCalendar WHERE StartTime = #StartTime AND EndTime = #EndTime
SET #StartHour = #StartHour + 1
END
DELETE #tCalendar WHERE StartTime = #StartTime AND EndTime = #EndTime
END
Do something like this:
Declare #ID int
SET #ID = 35
SELECT
p.SomeID
,p.ProductID
FROM ListOfProducts p
WHERE p.SomeID = #ID
-----------------------
--Or if you have to join to get it
Declare #ID int
SET #ID = 35
SELECT
c.SomeID
,p.ProductID
,p.ProductName
FROM ListOfProducts p
INNER JOIN categories c on p.ProductID = c.SomeID
WHERE p.SomeID = #ID
You can use option with WHILE loop and BREAK/CONTINUE keywords
CREATE PROC dbo.yourProcName
#SomeID int
AS
BEGIN
IF OBJECT_ID('tempdb.dbo.#resultTable') IS NOT NULL DROP TABLE dbo.#resultTable
CREATE TABLE dbo.#resultTable
(Col1 int, Col2 int)
DECLARE #ProductID int = 0
WHILE(1=1)
BEGIN
SELECT #ProductID = MIN(ProductID)
FROM ListOfProducts
WHERE SomeID = #SomeID AND ProductID > #ProductID
IF #ProductID IS NULL
BREAK
ELSE
INSERT dbo.#resultTable
SELECT Col1, Col2
FROM dbo.yourSearchTable
WHERE ProductID = #ProductID
CONTINUE
END
SELECT *
FROM dbo.#resultTable
END
Demo on SQLFiddle
Arrrgh, I am not getting this.
I have a table of accounts from Dynamics GP that has 7 columns. I need to fill in the blank months for accounts that didn't have any activity for a given month.
I have created an in memory table #MONTHS that has lines like so:
Account, Description, Year, Month, Month Name, Netchange, PeriodBal
1110000, NULL, 2006, 1, NULL, 0, NULL
This should match up with the same information coming from Dynamics GP. A similar line from GP would look like this:
1110000, Petty Cash, 2006, 1, January, 15.00, 343.97
If we did not spend any petty cash in February, then there would be no line for that account in 2/2006, I want to make the #MONTHS table RIGHT JOIN with the DynamicsGP table so that empty months are filled in.
Here's the abbreviated SQL shortened for readability:
SELECT Z.GPACCOUNTNO,
Z.DESCRIPTION,
Z.FISCALYEAR,
Z.FISCALPERIOD,
Z.FISCALPERIODNAME,
Z.NETCHANGE,
Z.PERIODBALANCE
FROM Z
RIGHT JOIN #MONTHS M
ON Z.GPACCOUNTNO = M.GPACCOUNTNO
AND Z.FISCALPERIOD = M.FISCALPERIOD
AND Z.FISCALYEAR = M.FISCALYEAR
The SQL just runs forever. (i.e. 5 minutes before I lose my patience)
I have verified that my #MONTHS table looks like I intend. I have tried doing a "UNION ALL" with the two tables and it gives me duplicates.
If Table Z does not have a current line for a given account/year/month, I want my #MONTHS table to add that line with a Netchange balance of 0.
Thank you for your help. The full SQL is below.
/* Create in memory table to hold account numbers */
DECLARE #i int
DECLARE #c int
DECLARE #ACCT char(129)
DECLARE #numrows int
DECLARE #numyears int
DECLARE #y int
DECLARE #m int
DECLARE #ACCT_TABLE TABLE (
idx smallint Primary Key IDENTITY(1,1),
account char(129)
)
/* Populate account number table */
INSERT #ACCT_TABLE
select distinct ACTNUMST from SBM01.[dbo].[GL00105]
/* Year table reads available years in the DB */
DECLARE #YEAR_TABLE TABLE (
idx smallint Primary Key IDENTITY(1,1),
YEAR1 smallint
)
/* Populate year table */
INSERT #YEAR_TABLE
SELECT distinct YEAR1 FROM SBM01.dbo.SY40101 ORDER BY YEAR1
/* Create our table of months to UNION to the main accounts */
DECLARE #MONTHS table (
GPACCOUNTNO char(129),
DESCRIPTION char(51),
FISCALYEAR smallint ,
FISCALPERIOD smallint,
FISCALPERIODNAME char(21),
NETCHANGE numeric(19, 5),
PERIODBALANCE numeric(19, 5)
)
/* Here comes the heavy lifting.
We loop over the account numbers and add year and month values.
*/
SET #i = 1
SET #numrows = (SELECT COUNT(*) FROM #ACCT_TABLE)
IF #numrows > 0
WHILE(#i <= (SELECT MAX(idx) FROM #ACCT_TABLE))
BEGIN
/* Get the next account number */
SET #ACCT = (SELECT account FROM #ACCT_TABLE WHERE idx = #i)
SET #c = 1
SET #numyears = (SELECT COUNT(*) FROM #YEAR_TABLE)
WHILE(#c <= (SELECT MAX(idx) FROM #YEAR_TABLE))
BEGIN
SET #y = (SELECT YEAR1 FROM #YEAR_TABLE WHERE idx = #c)
SET #m = '0'
WHILE(#m < '13')
BEGIN
INSERT INTO #MONTHS (GPACCOUNTNO, DESCRIPTION, FISCALPERIOD, FISCALYEAR, FISCALPERIODNAME, NETCHANGE, PERIODBALANCE)
VALUES (#ACCT, NULL, #m, #y, NULL, '0', NULL)
SET #m = #m + 1
END
SET #c = #c + 1
END
SET #i = #i + 1
END
/* We should now have a populated Database */
SELECT Z.GPACCOUNTNO, Z.DESCRIPTION, Z.FISCALYEAR, Z.FISCALPERIOD, Z.FISCALPERIODNAME, Z.NETCHANGE, Z.PERIODBALANCE
FROM ( SELECT RTRIM(B.[ACTNUMST]) AS GPACCOUNTNO,
RTRIM(C.[ACTDESCR]) AS DESCRIPTION,
A.[YEAR1] AS FISCALYEAR,
A.[PERIODID] AS FISCALPERIOD,
E.[PERNAME] AS FISCALPERIODNAME,
ISNULL(A.[PERDBLNC], 0) AS NETCHANGE,
( SELECT ISNULL(SUM(D.[PERDBLNC]), 0)
FROM SBM01.[dbo].[GL10110] D
WHERE D.[ACTINDX] = A.[ACTINDX]
AND D.[YEAR1] = A.[YEAR1]
AND D.[PERIODID] <= A.[PERIODID]
) AS PERIODBALANCE
FROM SBM01.[dbo].[GL10110] A
INNER JOIN SBM01.[dbo].[GL00105] B ON B.[ACTINDX] = A.[ACTINDX]
INNER JOIN SBM01.[dbo].[GL00100] C ON C.[ACTINDX] = A.[ACTINDX]
INNER JOIN SBM01.[dbo].[SY40100] E ON E.[YEAR1] = A.[YEAR1]
AND E.[PERIODID] = A.[PERIODID]
AND E.[SERIES] = 0
UNION ALL
SELECT RTRIM(B.[ACTNUMST]) AS GPACCOUNTNO,
RTRIM(C.[ACTDESCR]) AS DESCRIPTION,
A.[YEAR1] AS FISCALYEAR,
A.[PERIODID] AS FISCALPERIOD,
E.[PERNAME] AS FISCALPERIODNAME,
ISNULL(A.[PERDBLNC], 0) AS NETCHANGE,
( SELECT ISNULL(SUM(D.[PERDBLNC]), 0)
FROM SBM01.[dbo].[GL10111] D
WHERE D.[ACTINDX] = A.[ACTINDX]
AND D.[YEAR1] = A.[YEAR1]
AND D.[PERIODID] <= A.[PERIODID]
) AS PERIODBALANCE
FROM SBM01.[dbo].[GL10111] A
INNER JOIN SBM01.[dbo].[GL00105] B ON B.[ACTINDX] = A.[ACTINDX]
INNER JOIN SBM01.[dbo].[GL00100] C ON C.[ACTINDX] = A.[ACTINDX]
INNER JOIN SBM01.[dbo].[SY40100] E ON E.[YEAR1] = A.[YEAR1]
AND E.[PERIODID] = A.[PERIODID]
AND E.[SERIES] = 0
) Z
RIGHT JOIN #MONTHS M
ON Z.GPACCOUNTNO = M.GPACCOUNTNO
AND Z.FISCALPERIOD = M.FISCALPERIOD
AND Z.FISCALYEAR = M.FISCALYEAR
ORDER BY Z.[GPACCOUNTNO],
M.[FISCALYEAR],
M.[FISCALPERIOD]
Why don't you use the #Months table as the starting point (since it already gives you all the months you need) and fill-in the values from Z if they are available?
SELECT
M.GPACCOUNTNO,
M.DESCRIPTION,
M.FISCALYEAR,
M.FISCALPERIOD,
M.FISCALPERIODNAME,
ISNULL(Z.NETCHANGE, 0) as NETCHANGE
ISNULL(Z.PERIODBALANCE, 0) as PERIODBALANCE
FROM #MONTHS M
LEFT JOIN Z
ON Z.GPACCOUNTNO = M.GPACCOUNTNO
AND Z.FISCALPERIOD = M.FISCALPERIOD
AND Z.FISCALYEAR = M.FISCALYEAR
You can use a SQL case statement to join when null
CREATE TABLE #TMP
(
id int,
[month] datetime
)
INSERT INTO #TMP(id,[month])values(1,GETDATE())
INSERT INTO #TMP(id,[month])values(2,null)
INSERT INTO #TMP(id,[month])values(3,GETDATE())
INSERT INTO #TMP(id,[month])values(4,GETDATE())
CREATE TABLE #TMP2
(
id int,
[month] datetime
)
INSERT INTO #TMP2(id,[month])values(1,GETDATE())
INSERT INTO #TMP2(id,[month])values(2,GETDATE())
INSERT INTO #TMP2(id,[month])values(3,GETDATE())
INSERT INTO #TMP2(id,[month])values(4,GETDATE())
select * from #TMP
select * from #TMP2
SELECT #TMP.[id], case when #TMP.[month] is null then #TMP2.[month] else #TMP.month end
from #tmp
inner join #tmp2 on #tmp.id= #tmp2.id
drop table #tmp,#tmp2
I have the following TSQL Statement, I am trying to figure out how I can keep getting the results (100 rows at a time), store them in a variable (as I will have to add the totals after each select) and continue to select in a while loop until no more records are found and then return the variable totals to the calling function.
SELECT [OrderUser].OrderUserId, ISNULL(SUM(total.FileSize), 0), ISNULL(SUM(total.CompressedFileSize), 0)
FROM
(
SELECT DISTINCT TOP(100) ProductSize.OrderUserId, ProductSize.FileInfoId,
CAST(ProductSize.FileSize AS BIGINT) AS FileSize,
CAST(ProductSize.CompressedFileSize AS BIGINT) AS CompressedFileSize
FROM ProductSize WITH (NOLOCK)
INNER JOIN [Version] ON ProductSize.VersionId = [Version].VersionId
) AS total RIGHT OUTER JOIN [OrderUser] WITH (NOLOCK) ON total.OrderUserId = [OrderUser].OrderUserId
WHERE NOT ([OrderUser].isCustomer = 1 AND [OrderUser].isEndOrderUser = 0 OR [OrderUser].isLocation = 1)
AND [OrderUser].OrderUserId = 1
GROUP BY [OrderUser].OrderUserId
Depending on the clustered index, if its by numbered id, then use the code below. If its by date, go in 10 - 60 minute increments. keep an eye on performance of other things, but the lovely part of this code is you can start and stop at anytime if you push the results to permanent temp table (real table, just temp)
Here's a sample:
declare #count int
Declare #batch int
declare #max int
create table #temp (id int identity(1,1) primary key, Batch int, value int)
select #max = max(OrderUserId), #count = 0, #batch = 1000 from table
while (#count < #max)
begin
insert into #temp (batch,value)
select #count, Sum(stuffs)
from table
where orderId >= #count
and orderid < #count + #batch
Set #count = #count + #batch
waitfor delay ('00:00:01')
Raiserror('On Batch %d',0,1,#Count) with nowait /* Will print progess */
end