Splitting SQL query with many joins into smaller ones helps? - sql

Question

We need to do some reporting every night on our SQL server 2008R2. Calculating the reports takes several hours. In order to shorten the time we precalculate a table. This table is created based on JOINining 12 quite big (tens of milions row) tables.
The calculation of this aggregation table took until few days ago cca 4 hours. Our DBA than split this big join into 3 smaller joins (each joining 4 tables). The temporar result is everytime saved into a temporary table, which is used in the next join.
The result of the DBA enhancment is, that the aggregation table is calculated in 15 minutes. I wondered how is that possible. DBA told me that it is because the number of data the server must process is smaller. In other words, that in the big original join the server has to work with more data than in summed smaller joins. However, I would presume that optimizer would take care of doing it effeciently with the original big join, splitting the joins on its own and sending only the number of columns needed to next joins.
The other thing he has done is that he created an index on one of the tmeporary tables. However, once again I would think that the optimizer will create the appropriate hash tables if needed and alltogether better optimize the computation.
I talked about this with our DBA, but he was himself uncertain about what cased the improvement in processing time. He just mentioned, that he would not blame the server as it can be overwhelming to compute such big data and that it is possible that the optimizer has hard time to predict the best execution plan ... . This I understand, but I would like to have more defining answer as to exactly why.
So, the questions are:
1. "What could possibly cause the big improvement?"
2. "Is it a standard procedure to split big joins into smaller?"
3. "Is the amount of data which srever has to process really smaller in case of multiple smaller joins?"
Here is the original query:
Insert Into FinalResult_Base
SELECT
TC.TestCampaignContainerId,
TC.CategoryId As TestCampaignCategoryId,
TC.Grade,
TC.TestCampaignId,
T.TestSetId
,TL.TestId
,TSK.CategoryId
,TT.[TestletId]
,TL.SectionNo
,TL.Difficulty
,TestletName = Char(65+TL.SectionNo) + CONVERT(varchar(4),6 - TL.Difficulty)
,TQ.[QuestionId]
,TS.StudentId
,TS.ClassId
,RA.SubjectId
,TQ.[QuestionPoints]
,GoodAnswer = Case When TQ.[QuestionPoints] Is null Then 0
When TQ.[QuestionPoints] > 0 Then 1
Else 0 End
,WrongAnswer = Case When TQ.[QuestionPoints] = 0 Then 1
When TQ.[QuestionPoints] Is null Then 1
Else 0 End
,NoAnswer = Case When TQ.[QuestionPoints] Is null Then 1 Else 0 End
,TS.Redizo
,TT.ViewCount
,TT.SpentTime
,TQ.[Position]
,RA.SpecialNeeds
,[Version] = 1
,TestAdaptationId = TA.Id
,TaskId = TSK.TaskId
,TaskPosition = TT.Position
,QuestionRate = Q.Rate
,TestQuestionId = TQ.Guid
,AnswerType = TT.TestletAnswerTypeId
FROM
[TestQuestion] TQ WITH (NOLOCK)
Join [TestTask] TT WITH (NOLOCK) On TT.Guid = TQ.TestTaskId
Join [Question] Q WITH (NOLOCK) On TQ.QuestionId = Q.QuestionId
Join [Testlet] TL WITH (NOLOCK) On TT.TestletId = TL.Guid
Join [Test] T WITH (NOLOCK) On TL.TestId = T.Guid
Join [TestSet] TS WITH (NOLOCK) On T.TestSetId = TS.Guid
Join [RoleAssignment] RA WITH (NOLOCK) On TS.StudentId = RA.PersonId And RA.RoleId = 1
Join [Task] TSK WITH (NOLOCK) On TSK.TaskId = TT.TaskId
Join [Category] C WITH (NOLOCK) On C.CategoryId = TSK.CategoryId
Join [TimeWindow] TW WITH (NOLOCK) On TW.Id = TS.TimeWindowId
Join [TestAdaptation] TA WITH (NOLOCK) On TA.Id = TW.TestAdaptationId
Join [TestCampaign] TC WITH (NOLOCK) On TC.TestCampaignId = TA.TestCampaignId
WHERE
T.TestTypeId = 1 -- eliminuji ankety
And t.ProcessedOn is not null -- ne vsechny, jen dokoncene
And TL.ShownOn is not null
And TS.Redizo not in (999999999, 111111119)
END;
The new splitted joins after DBA great work:
SELECT
TC.TestCampaignContainerId,
TC.CategoryId As TestCampaignCategoryId,
TC.Grade,
TC.TestCampaignId,
T.TestSetId
,TL.TestId
,TL.SectionNo
,TL.Difficulty
,TestletName = Char(65+TL.SectionNo) + CONVERT(varchar(4),6 - TL.Difficulty) -- prevod na A5, B4, B5 ...
,TS.StudentId
,TS.ClassId
,TS.Redizo
,[Version] = 1 -- ?
,TestAdaptationId = TA.Id
,TL.Guid AS TLGuid
,TS.TimeWindowId
INTO
[#FinalResult_Base_1]
FROM
[TestSet] [TS] WITH (NOLOCK)
JOIN [Test] [T] WITH (NOLOCK)
ON [T].[TestSetId] = [TS].[Guid] AND [TS].[Redizo] NOT IN (999999999, 111111119) AND [T].[TestTypeId] = 1 AND [T].[ProcessedOn] IS NOT NULL
JOIN [Testlet] [TL] WITH (NOLOCK)
ON [TL].[TestId] = [T].[Guid] AND [TL].[ShownOn] IS NOT NULL
JOIN [TimeWindow] [TW] WITH (NOLOCK)
ON [TW].[Id] = [TS].[TimeWindowId] AND [TW].[IsActive] = 1
JOIN [TestAdaptation] [TA] WITH (NOLOCK)
ON [TA].[Id] = [TW].[TestAdaptationId] AND [TA].[IsActive] = 1
JOIN [TestCampaign] [TC] WITH (NOLOCK)
ON [TC].[TestCampaignId] = [TA].[TestCampaignId] AND [TC].[IsActive] = 1
JOIN [TestCampaignContainer] [TCC] WITH (NOLOCK)
ON [TCC].[TestCampaignContainerId] = [TC].[TestCampaignContainerId] AND [TCC].[IsActive] = 1
;
SELECT
FR1.TestCampaignContainerId,
FR1.TestCampaignCategoryId,
FR1.Grade,
FR1.TestCampaignId,
FR1.TestSetId
,FR1.TestId
,TSK.CategoryId AS [TaskCategoryId]
,TT.[TestletId]
,FR1.SectionNo
,FR1.Difficulty
,TestletName = Char(65+FR1.SectionNo) + CONVERT(varchar(4),6 - FR1.Difficulty) -- prevod na A5, B4, B5 ...
,FR1.StudentId
,FR1.ClassId
,FR1.Redizo
,TT.ViewCount
,TT.SpentTime
,[Version] = 1 -- ?
,FR1.TestAdaptationId
,TaskId = TSK.TaskId
,TaskPosition = TT.Position
,AnswerType = TT.TestletAnswerTypeId
,TT.Guid AS TTGuid
INTO
[#FinalResult_Base_2]
FROM
#FinalResult_Base_1 FR1
JOIN [TestTask] [TT] WITH (NOLOCK)
ON [TT].[TestletId] = [FR1].[TLGuid]
JOIN [Task] [TSK] WITH (NOLOCK)
ON [TSK].[TaskId] = [TT].[TaskId] AND [TSK].[IsActive] = 1
JOIN [Category] [C] WITH (NOLOCK)
ON [C].[CategoryId] = [TSK].[CategoryId]AND [C].[IsActive] = 1
;
DROP TABLE [#FinalResult_Base_1]
CREATE NONCLUSTERED INDEX [#IX_FR_Student_Class]
ON [dbo].[#FinalResult_Base_2] ([StudentId],[ClassId])
INCLUDE ([TTGuid])
SELECT
FR2.TestCampaignContainerId,
FR2.TestCampaignCategoryId,
FR2.Grade,
FR2.TestCampaignId,
FR2.TestSetId
,FR2.TestId
,FR2.[TaskCategoryId]
,FR2.[TestletId]
,FR2.SectionNo
,FR2.Difficulty
,FR2.TestletName
,TQ.[QuestionId]
,FR2.StudentId
,FR2.ClassId
,RA.SubjectId
,TQ.[QuestionPoints] -- 1+ good, 0 wrong, null no answer
,GoodAnswer = Case When TQ.[QuestionPoints] Is null Then 0
When TQ.[QuestionPoints] > 0 Then 1 -- cookie
Else 0 End
,WrongAnswer = Case When TQ.[QuestionPoints] = 0 Then 1
When TQ.[QuestionPoints] Is null Then 1
Else 0 End
,NoAnswer = Case When TQ.[QuestionPoints] Is null Then 1 Else 0 End
,FR2.Redizo
,FR2.ViewCount
,FR2.SpentTime
,TQ.[Position] AS [QuestionPosition]
,RA.SpecialNeeds -- identifikace SVP
,[Version] = 1 -- ?
,FR2.TestAdaptationId
,FR2.TaskId
,FR2.TaskPosition
,QuestionRate = Q.Rate
,TestQuestionId = TQ.Guid
,FR2.AnswerType
INTO
[#FinalResult_Base]
FROM
[#FinalResult_Base_2] FR2
JOIN [TestQuestion] [TQ] WITH (NOLOCK)
ON [TQ].[TestTaskId] = [FR2].[TTGuid]
JOIN [Question] [Q] WITH (NOLOCK)
ON [Q].[QuestionId] = [TQ].[QuestionId] AND [Q].[IsActive] = 1
JOIN [RoleAssignment] [RA] WITH (NOLOCK)
ON [RA].[PersonId] = [FR2].[StudentId]
AND [RA].[ClassId] = [FR2].[ClassId] AND [RA].[IsActive] = 1 AND [RA].[RoleId] = 1
drop table #FinalResult_Base_2;
truncate table [dbo].[FinalResult_Base];
insert into [dbo].[FinalResult_Base] select * from #FinalResult_Base;
drop table #FinalResult_Base;

Answer 1

For the first, please, rebuilding indexes on your tables by this script (this may take a long time) -
SET NOCOUNT ON;
DECLARE
#SQL NVARCHAR(MAX)
, #IndexName SYSNAME
, #Output VARCHAR(200)
, #ServerVersion VARCHAR(100)
SELECT #ServerVersion = CAST(SERVERPROPERTY('Edition') AS VARCHAR(100))
DECLARE cur CURSOR LOCAL READ_ONLY FORWARD_ONLY FOR
SELECT
'ALTER INDEX [' + ix.name + N'] ON [' + SCHEMA_NAME(t.[schema_id]) + '].[' + t.name + '] ' +
CASE
WHEN ps.avg_fragmentation_in_percent > 50 THEN
CASE WHEN #ServerVersion LIKE 'Enterprise%' OR #ServerVersion LIKE 'Developer%' THEN
'REBUILD WITH (SORT_IN_TEMPDB = ON, ONLINE = ON ' + CASE WHEN ix.fill_factor > 0 THEN ', FILLFACTOR = ' + CAST(ix.fill_factor AS VARCHAR(3)) ELSE '' END + ') '
ELSE
'REBUILD WITH (SORT_IN_TEMPDB = ON' + CASE WHEN ix.fill_factor > 0 THEN ', FILLFACTOR = ' + CAST(ix.fill_factor AS VARCHAR(3)) ELSE '' END + ') '
END
ELSE 'REORGANIZE '
END +
CASE
WHEN ps.partition_number > 1 THEN N' PARTITION = ' + CAST(ps.partition_number AS NVARCHAR(MAX))
ELSE N''
END + ';', ix.name
FROM sys.indexes ix
JOIN sys.objects t ON t.[object_id] = ix.[object_id]
JOIN (
SELECT
[object_id]
, index_id
, avg_fragmentation_in_percent
, partition_number
FROM sys.dm_db_index_physical_stats(DB_ID(), NULL, NULL, NULL, N'LIMITED')
WHERE page_count > 100
AND avg_fragmentation_in_percent > 10
) ps ON t.[object_id] = ps.[object_id] AND ix.index_id = ps.index_id
WHERE t.[type] = 'U'
AND t.name IN (
'TestQuestion', 'TestTask', 'Question', 'Testlet',
'Test', 'TestSet', 'RoleAssignment', 'Task',
'category', 'TimeWindow', 'TestAdaptation', 'TestCampaign')
OPEN cur
FETCH NEXT FROM cur INTO #SQL, #IndexName
WHILE ##FETCH_STATUS = 0 BEGIN
SELECT #Output = CONVERT(NVARCHAR(15), GETDATE(), 114) + ': ' + #IndexName
RAISERROR(#Output, 0, 1) WITH NOWAIT
EXEC sys.sp_executesql #SQL
FETCH NEXT FROM cur INTO #SQL, #IndexName
END
CLOSE cur
DEALLOCATE cur
And after it try this query -
INSERT INTO dbo.FinalResult_Base
SELECT
TC.TestCampaignContainerId
, TestCampaignCategoryId = TC.CategoryID
, TC.Grade
, TC.TestCampaignId
, T.TestSetId
, TL.TestId
, TSK.CategoryID
, TT.[TestletId]
, TL.SectionNo
, TL.Difficulty
, TestletName = CHAR(65 + TL.SectionNo) + CONVERT(VARCHAR(4), 6 - TL.Difficulty)
, TQ.[QuestionId]
, TS.StudentId
, TS.ClassId
, RA.SubjectId
, TQ.[QuestionPoints]
, GoodAnswer =
CASE WHEN ISNULL(TQ.[QuestionPoints], 0) > 0
THEN 1
ELSE 0
END
, WrongAnswer =
CASE
WHEN ISNULL(TQ.[QuestionPoints], 0) = 0
THEN 1
ELSE 0
END
, NoAnswer =
CASE WHEN TQ.[QuestionPoints] IS NULL
THEN 1
ELSE 0
END
, TS.Redizo
, TT.ViewCount
, TT.SpentTime
, TQ.[Position]
, RA.SpecialNeeds
, [Version] = 1
, TestAdaptationId = TA.id
, TaskId = TSK.TaskId
, TaskPosition = TT.Position
, QuestionRate = Q.Rate
, TestQuestionId = TQ.guid
, AnswerType = TT.TestletAnswerTypeId
FROM dbo.TestQuestion TQ WITH (NOLOCK)
JOIN dbo.TestTask TT WITH (NOLOCK) ON TT.[guid] = TQ.TestTaskId
JOIN dbo.Question Q WITH (NOLOCK) ON TQ.QuestionId = Q.QuestionId
JOIN (
SELECT *
FROM dbo.Testlet TL WITH (NOLOCK)
WHERE TL.ShownOn IS NOT NULL
) TL ON TT.TestletId = TL.[guid]
JOIN (
SELECT *
FROM dbo.Test T WITH (NOLOCK)
WHERE T.TestTypeId = 1
AND T.ProcessedOn IS NOT NULL
) T ON TL.TestId = T.[guid]
JOIN (
SELECT *
FROM dbo.TestSet TS WITH (NOLOCK)
WHERE TS.Redizo NOT IN (999999999, 111111119)
) TS ON T.TestSetId = TS.[guid]
JOIN dbo.RoleAssignment RA WITH (NOLOCK) ON TS.StudentId = RA.PersonID AND RA.RoleId = 1
JOIN dbo.Task TSK WITH (NOLOCK) ON TSK.TaskId = TT.TaskId
JOIN dbo.category C WITH (NOLOCK) ON C.CategoryID = TSK.CategoryID
JOIN dbo.TimeWindow TW WITH (NOLOCK) ON TW.id = TS.TimeWindowId
JOIN dbo.TestAdaptation TA WITH (NOLOCK) ON TA.id = TW.TestAdaptationId
JOIN dbo.TestCampaign TC WITH (NOLOCK) ON TC.TestCampaignId = TA.TestCampaignId
And try this query -
SELECT TC.TestCampaignContainerId
,TC.CategoryID AS TestCampaignCategoryId
,TC.Grade
,TC.TestCampaignId
,T.TestSetId
,TL.TestId
,TL.SectionNo
,TL.Difficulty
,TestletName = CHAR(65 + TL.SectionNo) + CONVERT(VARCHAR(4), 6 - TL.Difficulty) -- prevod na A5, B4, B5 ...
,TS.StudentId
,TS.ClassId
,TS.Redizo
,[Version] = 1 -- ?
,TestAdaptationId = TA.id
,TL.guid AS TLGuid
,TS.TimeWindowId
INTO [#FinalResult_Base_1]
FROM (
SELECT *
FROM dbo.[TestSet] [TS] WITH (NOLOCK)
WHERE [TS].[Redizo] NOT IN (999999999, 111111119)
) TS
JOIN (
SELECT *
FROM dbo.[Test] [T] WITH (NOLOCK)
WHERE [T].[TestTypeId] = 1
AND [T].[ProcessedOn] IS NOT NULL
) T ON [T].[TestSetId] = [TS].[guid]
JOIN (
SELECT *
FROM dbo.[Testlet] [TL] WITH (NOLOCK)
WHERE [TL].[ShownOn] IS NOT NULL
) TL ON [TL].[TestId] = [T].[guid]
JOIN (
SELECT *
FROM dbo.[TimeWindow] [TW] WITH (NOLOCK)
WHERE [TW].[IsActive] = 1
) TW ON [TW].[id] = [TS].[TimeWindowId]
JOIN (
SELECT *
FROM dbo.[TestAdaptation] [TA] WITH (NOLOCK)
WHERE [TA].[IsActive] = 1
) TA ON [TA].[id] = [TW].[TestAdaptationId]
JOIN (
SELECT *
FROM dbo.[TestCampaign] [TC] WITH (NOLOCK)
WHERE [TC].[IsActive] = 1
) TC ON [TC].[TestCampaignId] = [TA].[TestCampaignId]
--possible unused join in this query
--JOIN dbo.[TestCampaignContainer] [TCC] WITH (NOLOCK) ON [TCC].[TestCampaignContainerId] = [TC].[TestCampaignContainerId] AND [TCC].[IsActive] = 1
SELECT FR1.TestCampaignContainerId
,FR1.TestCampaignCategoryId
,FR1.Grade
,FR1.TestCampaignId
,FR1.TestSetId
,FR1.TestId
,TSK.CategoryID AS [TaskCategoryId]
,TT.[TestletId]
,FR1.SectionNo
,FR1.Difficulty
,TestletName = CHAR(65 + FR1.SectionNo) + CONVERT(VARCHAR(4), 6 - FR1.Difficulty) -- prevod na A5, B4, B5 ...
,FR1.StudentId
,FR1.ClassId
,FR1.Redizo
,TT.ViewCount
,TT.SpentTime
,[Version] = 1 -- ?
,FR1.TestAdaptationId
,TaskId = TSK.TaskId
,TaskPosition = TT.Position
,AnswerType = TT.TestletAnswerTypeId
,TT.guid AS TTGuid
INTO [#FinalResult_Base_2]
FROM #FinalResult_Base_1 FR1
JOIN [TestTask] [TT] WITH (NOLOCK) ON [TT].[TestletId] = [FR1].[TLGuid]
JOIN [Task] [TSK] WITH (NOLOCK) ON [TSK].[TaskId] = [TT].[TaskId]
--possible unused join
--JOIN [category] [C] WITH (NOLOCK) ON [C].[CategoryID] = [TSK].[CategoryID]
WHERE [TSK].[IsActive] = 1
--AND [C].[IsActive] = 1
DROP TABLE [#FinalResult_Base_1]
CREATE NONCLUSTERED INDEX [#IX_FR_Student_Class]
ON [dbo].[#FinalResult_Base_2] ([StudentId], [ClassId])
INCLUDE ([TTGuid])
TRUNCATE TABLE [dbo].[FinalResult_Base];
INSERT INTO [dbo].[FinalResult_Base]
SELECT FR2.TestCampaignContainerId
,FR2.TestCampaignCategoryId
,FR2.Grade
,FR2.TestCampaignId
,FR2.TestSetId
,FR2.TestId
,FR2.[TaskCategoryId]
,FR2.[TestletId]
,FR2.SectionNo
,FR2.Difficulty
,FR2.TestletName
,TQ.[QuestionId]
,FR2.StudentId
,FR2.ClassId
,RA.SubjectId
,TQ.[QuestionPoints] -- 1+ good, 0 wrong, null no answer
, GoodAnswer =
CASE WHEN ISNULL(TQ.[QuestionPoints], 0) > 0
THEN 1
ELSE 0
END
, WrongAnswer =
CASE
WHEN ISNULL(TQ.[QuestionPoints], 0) = 0
THEN 1
ELSE 0
END
, NoAnswer =
CASE WHEN TQ.[QuestionPoints] IS NULL
THEN 1
ELSE 0
END
,FR2.Redizo
,FR2.ViewCount
,FR2.SpentTime
,TQ.[Position] AS [QuestionPosition]
,RA.SpecialNeeds -- identifikace SVP
,[Version] = 1 -- ?
,FR2.TestAdaptationId
,FR2.TaskId
,FR2.TaskPosition
,QuestionRate = Q.Rate
,TestQuestionId = TQ.guid
,FR2.AnswerType
FROM [#FinalResult_Base_2] FR2
JOIN [TestQuestion] [TQ] WITH (NOLOCK) ON [TQ].[TestTaskId] = [FR2].[TTGuid]
JOIN [Question] [Q] WITH (NOLOCK) ON [Q].[QuestionId] = [TQ].[QuestionId] AND [Q].[IsActive] = 1
JOIN [RoleAssignment] [RA] WITH (NOLOCK) ON [RA].[PersonID] = [FR2].[StudentId]
AND [RA].[ClassId] = [FR2].[ClassId]
AND [RA].[IsActive] = 1
AND [RA].[RoleId] = 1
DROP TABLE #FinalResult_Base_2;

Answer 2

IMHO It should not be the case, I faced similar problem and took the following steps to optimize it.
Try putting Indexes on the columns which are used in filtering (The
columns deciding the joins).
Try to put indexes on Views, It could be done but It needs some
special requirements.
Have some jobs for Index Rebuilding.
Use another mirror instance of DB for reporting leave the live DB
Don't use functions while joining if you want to transform data do it
after joining.
Use Query optimizer to view what part of join is taking most of the
time/resources.
Use an Archive table to purge data into it from live DB.
Hope it helps :)