What can be the reason for slow inserts into my SQL Server table compared to my local DB with 4GB of memory the script which is running less than 2 seconds while on my client's server with 1GB memory (attached image) it took 50 seconds.
Below is a sample query, records from my local and from my client's db are almost the same count.
IF OBJECT_ID('tempdb..#TempUnitServiceLogs') IS NOT NULL
drop table #TempUnitServiceLogs
Select
F.Unit_Id, MAX(F.UnitTransaction_Id) UnitTransaction_Id
into
#TempUnitServiceLogs
from
TempUnitServiceLogs F
group by F.Unit_Id
insert into TempUnitServiceLogs
Select
A.Unit_Id
,C.Brand_Name
,A.Unit_CompanyCode
,B.Project_Code
,A.Unit_Code
,A.Unit_Status
from Table1 A
inner join Table2 B on A.Project_Id = B.Project_Id
inner join Table3 C on B.Brand_Id = C.Brand_Id
left join #TempUnitServiceLogs D on A.Unit_Id = D.Unit_ID
left join Table4 ut on ut.Unit_Id = A.Unit_Id
left join Table5 pb on pb.ProspectiveBuyer_Id = ut.ProspectiveBuyer_Id
where 1=1
and d.UnitTransaction_Id IS NULL
and case when UnitTransaction_ModDate IS null
then convert(date,UnitTransaction_CrDate)
else convert(date,UnitTransaction_ModDate) end = CONVERT(date,GETDATE())
Related
I have a procedure which is basically inserting records to a table from selection of few table and views combinations
and SQL is like below
ALTER PROCEDURE [dbo].[aka_spring_rep_sum]
AS
INSERT INTO tbl_spring (col1,col2,....colx)
SELECT col1,col2...colx
FROM vw_tbl_spring bk
LEFT JOIN
(SELECT col1,col2,..
FROM vw_tbl_prices_spring) sp ON bk.col1 = sp.col1
AND bk.col2 = sp.col2
LEFT JOIN
(SELECT col1,col2...
FROM tbl_xx) stock ON bk.col1 = stock.col1
AND bk.col2 = stock.col2
LEFT JOIN
(SELECT DISTINCT col1,col2,....
FROM tbl_v) sf ON bk.col1 = sf.col1
AND bk.colx = sf.colx
LEFT JOIN
(SELECT DISTINCT col1, col2
FROM tbl_bb) vr ON sf.col1 = vr.col1
LEFT JOIN
(SELECT col1, is
FROM tbl_ss) sh ON bk.col1 = sh.col1
OPTION (RECOMPILE)
The stored procedure was taking less than 2-3 seconds only till today, but today all of a sudden this was taking very long time 30 minutes plus and never ending and forced to stop manually.
After breaking the different selections one by one I found that
select ..... FROM vw_tbl_spring bk
is ended up as a never ending call. Rest all select statements in the stored procedure are returning results less than 1 seconds.
ALTER VIEW [dbo].[vw_tbl_spring]
AS
SELECT col1, col2...., colx
FROM
(SELECT Icol1, col2, ....
FROM
(
SELECT DISTINCT col1,col2,... FROM tbl_pens s
INNER JOIN tbl_penh h ON s.col1 = h.col1 AND s.col2 = h.col2
WHERE s.col6 >= 21 AND h.col1 = 'X'
)
) b
LEFT JOIN
( SELECT col1,col2,...... FROM tbl_pens s
INNER JOIN tbl_penh h ON ON s.col1 = h.col1 AND s.col2 = h.col2
WHERE WHERE s.col6 >= 21 AND h.col1 = 'X'
) p ON b.col1 = p.col1 AND b.col1 = p.col1
LEFT JOIN vw_tbl_kk k ON p.col1 = k.col1 AND p.col1 = k.col2
Again filtering the different selections inside this view found out that the last left join is slowing things down
If we removed the last left join ie
LEFT JOIN vw_tbl_kk k ON p.col1 = k.col1 AND p.col1 = k.col2
Everything will be as normal ie will return results in less than 2-3 seconds
Unable to find what is the reason behind this sudden slowness
The same behaviour occurred a few months back and that time try to delete and recreate all associated views and stored procedure and then the issue was resolved. But this time this also didn't help
Any way to check what is causing this slowness in SQL Server?
Your query is a 6-way join. This means that if there are n rows that join in each joined table, there will be n6 resulting rows to insert. To highlight the exponential growth this causes:
rows in each table that join | number of resultant rows
1 | 1
2 | 64
3 | 729
4 | 4,096
10 | 1,000,000
There are probably suddenly more joining rows in the tables, not only making it slower, but also hitting the logging high water mark, which means the database must rollback the transaction, which is typically very slow.
A select query with join returns results in less than 1 sec(returns 1000 rows: 600 ms) but insert into temp table or physical table takes 15-16 seconds.
tested io performance : a select or insert into without any joins takes sub-second to write 1000 rows.
tried trace flag 1118
tried adding clustered index on the temp table and do insert into with tablock and maxdop hint.
None of these improved performance.
Thanks for all your comments. 6000 to 20000 rows that need to be inserted every 5 seconds from Kafka...
I get the data from kafka into sql server using table type variable
Pass it as a parameter to stored procedure
Load this data joining with other tables into a temporary table #table
I use the #table to merge the data into application table
Found a workaround that helps me achieve the target turnaround time but dont exactly know the reason for the behaviour. As I mentioned in the problem statement, the bottleneck was writing the resultset of the select statement that joins the table variable with various other tables to the temp table.
I put this into a stored prod and returned the execution of the stored proc to a temp table. Now the insert takes less than 1 sec
SELECT
i.Id AS IId,
df.Id AS dfid,
MAX(CASE
WHEN lp.Value IS NULL and f.pnp = 1 THEN 0
WHEN lp.Value = 0 and f.tzan = 1 and f.pnp = 0 THEN NULL
ELSE lp.Value
END) 'FV',
MAX(lp.TS),
MAX(lp.Eid),
MAX(0+lp.IsDelayedStream)
FROM
f1 f WITH (NOLOCK)
INNER JOIN ft1 ft WITH (NOLOCK) ON f.FeedTypeId = ft.Id
INNER JOIN FeedDataField fdf WITH (NOLOCK)
ON fdf.FeedId = f.Id
INNER JOIN df1 df WITH (NOLOCK)
ON fdf.dfId = df.Id
INNER JOIN ds1 ds WITH (NOLOCK)
ON df.dsid = ds.Id
INNER JOIN dp1 dp WITH (NOLOCK)
ON ds.dpId = dp.Id
INNER JOIN dc1 dc WITH (NOLOCK)
ON dc.dcId = ds.dcId
INNER JOIN i1 i WITH (NOLOCK)
ON f.iId = I.Id
INNER JOIN id1 id WITH (NOLOCK)
ON id.iId = i.Id
INNER JOIN IdentifierType it WITH (NOLOCK)
ON id.ItId = it.Id
INNER JOIN ivw_Tdf tdf WITH(NOEXPAND)
ON tdf.iId = i.Id
INNER JOIN z.dbo.[tlp] lp
ON lp.Ticker = id.Name AND lp.Field = df.SourceName AND
lp.Contributor = dc.Name AND lp.YellowKey = tdf.TextValue
WHERE
ft.Name in ('X', 'Y') AND f.SA = 1
AND dp.Name = 'B' AND (i.Inactive IS NULL OR i.Inactive = 0)
AND it.Name = 'T' AND id.ValidTo = #InfinityDate
AND tdf.SourceName = 'MSD'
AND tdf.ValidTo = #Infinity
GROUP BY i.Id, df.Id
OPTION(MAXDOP 4, OPTIMIZE FOR (#Infinity = '9999-12-31 23:59:59',
#InfinityDate = '9999-12-31))
My SQL query is taking a large amount of time to run. I wrote a similar query and pit them against each other and this one runs FASTER when a small dataset (10K lines) is used, but about 20-30x slower than the other one when a LARGE dataset (500K+ lines) is used. My first query however does not have ONE column that I need, and I cannot add it without going about it with this different approach.
SELECT a.[RFIDTAGID], a.[JOB_NUMBER], d.[PROJECT_NUMBER], a.[PART_NUMBER], a.[QUANTITY], b.[DESIGNATION] as LOCATION,
c.[DESIGNATION] as CONTAINER, a.[LAST_SEEN_TIME], b.[TYPE], b.[BLDG], d.[PBG], d.[PLANNED_MFG_DELIVERY_DATE], d.[EXTENSION_DATE], a.[ORG_ID]
FROM [LTS].[dbo].[LTS_PACKAGE] as a
LEFT OUTER JOIN (
SELECT [DESIGNATION], [CONTAINER_ID], [LOCATION_ID]
FROM [LTS].[dbo].[LTS_CONTAINER]
) c ON a.[CONTAINER_ID] = c.[CONTAINER_ID]
LEFT OUTER JOIN (
SELECT [DESIGNATION], [TYPE], [BLDG], [LOCATION_ID]
FROM [LTS].[dbo].[LTS_LOCATION]
) b ON a.[LAST_SEEN_LOC_ID] = b.[LOCATION_ID] OR b.[LOCATION_ID] = c.[LOCATION_ID]
INNER JOIN (
SELECT [PBG], [PLANNED_MFG_DELIVERY_DATE], [EXTENSION_DATE], [DISCRETE_JOB_NUMBER], [PROJECT_NUMBER]
FROM [LTS].[dbo].[LTS_DISCRETE_JOB_SUMMARY]
)d ON a.[JOB_NUMBER] = d.[DISCRETE_JOB_NUMBER]
WHERE
d.[PLANNED_MFG_DELIVERY_DATE] <= GETDATE()
AND b.[TYPE] NOT IN('MFG', 'Manufacturing')
AND (b.[DESIGNATION] IS NOT NULL OR c.[DESIGNATION] IS NOT NULL)
ORDER BY [JOB_NUMBER], d.[PLANNED_MFG_DELIVERY_DATE] desc, [RFIDTAGID];
You can see below the usage, 100% is roughly 20,000, whereas my other query is about 900:
Is there something I can do to speed up my query, or where did I bog it down?
Remove inner selects and join directly to the tables:
SELECT a.[RFIDTAGID], a.[JOB_NUMBER], d.[PROJECT_NUMBER], a.[PART_NUMBER], a.[QUANTITY], b.[DESIGNATION] as LOCATION,
c.[DESIGNATION] as CONTAINER, a.[LAST_SEEN_TIME], b.[TYPE], b.[BLDG], d.[PBG], d.[PLANNED_MFG_DELIVERY_DATE], d.[EXTENSION_DATE], a.[ORG_ID]
FROM [LTS].[dbo].[LTS_PACKAGE] a
LEFT OUTER JOIN [LTS].[dbo].[LTS_CONTAINER]
c ON a.[CONTAINER_ID] = c.[CONTAINER_ID]
LEFT OUTER JOIN [dbo].[LTS_LOCATION]
b ON a.[LAST_SEEN_LOC_ID] = b.[LOCATION_ID] OR b.[LOCATION_ID] = c.[LOCATION_ID]
INNER JOIN
[LTS].[dbo].[LTS_DISCRETE_JOB_SUMMARY]
d ON a.[JOB_NUMBER] = d.[DISCRETE_JOB_NUMBER]
WHERE
d.[PLANNED_MFG_DELIVERY_DATE] <= GETDATE()
AND b.[TYPE] NOT IN('MFG', 'Manufacturing')
AND (b.[DESIGNATION] IS NOT NULL OR c.[DESIGNATION] IS NOT NULL)
ORDER BY [JOB_NUMBER], d.[PLANNED_MFG_DELIVERY_DATE] desc, [RFIDTAGID];
I'm having a performance problem with a SQL query that is generated by a .NET application.
Basically what the query is doing is:
(query1) left join (query2) right join (queries3 to 30) WHERE (query1.ID IS NULL) OR (query3.ID IS NULL AND query4.ID IS NULL AND… queryN.ID IS NULL)
When the query only does WHERE A (query1.ID) the query is fast.
When the query only does WHERE B (query3 to 30) the query is fast
When A and B are a combined WHERE clause with an OR, the query is
very slow.
I'm looking for a way to optimize this query without variables or stored procedures.
The query:
SELECT DISTINCT [Table0].[FIELD]
FROM /*8*/ ([Table0] AS [Table0]
INNER JOIN
[XTABLE] AS [XTABLE0]
ON [Table0].ID = [XTABLE0].ID1
AND [XTABLE0].ID3 = 52)
RIGHT OUTER /*10*/ JOIN
[Table1] AS [Table1]
/*21*/ /*11*/ ON [XTABLE0].ID2 = [Table1].ID
AND [XTABLE0].ID3 = 52
LEFT OUTER JOIN
([XTABLE] AS [XTABLE1]
INNER JOIN
[Table2] AS [Table2]
ON [XTABLE1].ID1 = [Table2].ID
AND [XTABLE1].ID3 = 19
/*20a*/ INNER JOIN
[XTABLE] AS [XTABLE2]
ON [Table2].ID = [XTABLE2].ID1
AND [XTABLE2].ID3 = 8
INNER JOIN
[Table3] AS [Table3]
ON [XTABLE2].ID2 = [Table3].ID
AND [XTABLE2].ID3 = 8/*22*/ )
ON [Table1].ID = [XTABLE1].ID2
AND [XTABLE1].ID3 = 19
/*26 */ LEFT OUTER JOIN
([XTABLE] AS [XTABLE3]
... and tens of similar INNER JOIN blocks
WHERE (/*13*/ [XTABLE0].ID IS NULL)
OR (/*25*/ [XTABLE1].ID IS NULL
AND /*27b*/ [XTABLE3].ID IS NULL
AND /*27b*/ [XTABLE5].ID IS NULL
... and tens of similar lines
AND /*27b*/ [XTABLE131].ID IS NULL);
You are OUTER JOIN'ing the queries, so, when you start putting stuff in the WHERE clause from the result of the OUTER JOIN table expressions (derived table in this case) then it will more than likely be treat as an INNER JOIN - you can see that by checking the query plan.
I have this SP with more than 10tables involved. In the underlined lines, there is a table AllData which is being joined 3times because of the fieldname in the where clause.
Any suggestions on how to handle this complex query better will be greatly appreciated. Mostly, to avoid the multiple times I am joining AllData(with alias names ad1, adl2, adl3). This could affect the performance.
Here is the sp
ALTER PROCEDURE [dbo].[StoredProc1]
AS
select case when pd.Show_Photo = '1,1,1'
then i.id
else null
end as thumbimage,
t1.FPId,
'WebProfile' as profiletype,
mmbp.Name as Name,
t1.Age,
t1.Height,
adl.ListValue as AlldataValue1,
adl2.ListValue as AlldataValue2,
adl3.ListValue as AlldataValue3,
c.CName,
ed.ELevel,
ed.EDeg,
NEWID()
from Table2 mmbp, Table3 u
join Table1 t1 on t1.Pid = u.Pid
left join Table4 mmb on t1.Pid= mmb.Pid
join table5 i on t1.Pid = i.Pid
join table6 pd on t1.Pid = pd.Pid
join table7 ed on t1.Pid = ed.Pid
join table8 c on t1.xxx= c.xxx
join AllData adl on t1.xxx = adl.ListKey
join AllData adl2 on b.ms = adl2.ListKey
join AllData adl3 on b.Diet = adl3.ListKey
where adl.FieldName=xxx and
adl2.FieldName='ms' and
adl3.FieldName='Diet' and
------
I note that you appear to have a cartesian join between Table2 and Table3 - unless one of these tables is very small, this is likely to drastically affect performance. I suggest explicitly joining Table2 to one of the other tables in the query, to improve performance.
One thing you could try is moving the where conditions into the joins
join AllData ad1 on t1.xxx = ad1.ListKey AND ad1.FieldName = xxx
join AllData ad2 on b.ms = adl2.ListKey AND ad2.FieldName = 'ms'
join AllData ad3 on b.Diet = adl3.ListKey AND ad3.FieldName = 'Diet'
This would give better performance as the join size would be limited to only the records you want. To do this all in one join you could join AllData ad on (t1.xxx = ad.ListKey AND ad.FieldName = xxx) OR (b.ms = ad.ListKey AND ad.FieldName = 'ms').... The issue with this option is you no longer have distinct columns for ad1, ad2, etc.