Terminating a SSIS Foreach loop after x amount of execution time? - sql

Question

I have a collection, which gets executed in the For each loop of the SSIS package. However, i want to ensure that after lets say one hour of executing a particular item, i want the For each loop to terminate or exit. But it should not Stop the execution of that item or Time it out.
Is it possible to achieve ?

Answer 1

It has been quite some time since I've had a good SSIS problem that I've never solved, so thank you for your question.
The approach I came up with requires a Sequence Container that holds a ForEach Loop Container. My Foreach Loop has 6 items in the stack. My requirement is that my loop cannot start any new work after 5 seconds of execution. Inside my loop, I run a 3 second delay in the Execute SQL Task to simulate work. Therefore, a "full" run will take at least 18 seconds (6*3) but if we get the abort working, we'll see 2 total runs.
I have defined 5 variables. You'll notice that in the second column, 2 of those variables are scoped to specific containers. The Second icon in the Variables screen has an arrow -> which is how you change the Scope of a Variable.
The idea is that in the Sequence Container, we're going to use an Expression task to compute the end time. The expression is going to use the System scoped variable ContainerStartTime
#[User::SEQCEndTime] = DATEADD("second", #[User::TaskDuration_s], #[System::ContainerStartTime])
The reason we encapsulate this into a sequence container is there might have been work beforehand that takes 4 seconds to generate a data set for the Loop container and we do not want to be penalized for that work. The formula is driven by the Variable TaskDuration_s which is initialized to 5.
The two SCR Echo Back are just logging messages to the Information stream to "prove" things are working.
The FELC Enum Values has a variable IsLoopValid defined in it. This expression is
#[User::IsLoopValid] = #[System::ContainerStartTime] < #[User::SEQCEndTime]
Every loop of the ForEach container resets that ContainerStartTime, which is what we want. But if we had evaluated the IsLoopValid in the context of the Sequence container, it'd never change. So, important to have it computed here.
The f/x you see on the precedent constraint between "Set IsLoopValid" and "SQL Do thing" is a non-default constraint because we're going to make it "Expression and Constraint" instead of just "Constraint". The expression is
#[User::IsLoopValid]
Every Loop will compare current time to the max end time (SEQCEndTime) and as long as we have not slipped the boundary, it will go on to the next task. Once we've exceeded the boundary, the Loop container will continue iterating through the result set but since there's no work to be done, it'll finish right quick.
Results
A sample run with the precedent constraint set to just Constraint. We can see the IsLoopValid is flipped to False but since we do not have the Expression in there, it does all 6 loops
SSIS package "C:\Users\bfellows\source\repos\SO_Trash\SO_Trash\SO_73829526.dtsx" starting.
Information: 0x0 at SCR Echo back, SCR Echo Back 0: System::ContainerStartTime->9/23/2022 3:03:49 PM
Information: 0x0 at SCR Echo back, SCR Echo Back 0: User::SEQCEndTime->9/23/2022 3:03:54 PM
Information: 0x0 at SCR Echo back, SCR Echo Back: System::ContainerStartTime->9/23/2022 3:03:52 PM
Information: 0x0 at SCR Echo back, SCR Echo Back: User::IsLoopValid->True
Information: 0x0 at SCR Echo back, SCR Echo Back: User::SEQCEndTime->9/23/2022 3:03:54 PM
Information: 0x0 at SCR Echo back, SCR Echo Back: System::ContainerStartTime->9/23/2022 3:03:55 PM
Information: 0x0 at SCR Echo back, SCR Echo Back: User::IsLoopValid->True
Information: 0x0 at SCR Echo back, SCR Echo Back: User::SEQCEndTime->9/23/2022 3:03:54 PM
Information: 0x0 at SCR Echo back, SCR Echo Back: System::ContainerStartTime->9/23/2022 3:03:59 PM
Information: 0x0 at SCR Echo back, SCR Echo Back: User::IsLoopValid->False
Information: 0x0 at SCR Echo back, SCR Echo Back: User::SEQCEndTime->9/23/2022 3:03:54 PM
Information: 0x0 at SCR Echo back, SCR Echo Back: System::ContainerStartTime->9/23/2022 3:04:02 PM
Information: 0x0 at SCR Echo back, SCR Echo Back: User::IsLoopValid->False
Information: 0x0 at SCR Echo back, SCR Echo Back: User::SEQCEndTime->9/23/2022 3:03:54 PM
Information: 0x0 at SCR Echo back, SCR Echo Back: System::ContainerStartTime->9/23/2022 3:04:05 PM
Information: 0x0 at SCR Echo back, SCR Echo Back: User::IsLoopValid->False
Information: 0x0 at SCR Echo back, SCR Echo Back: User::SEQCEndTime->9/23/2022 3:03:54 PM
Information: 0x0 at SCR Echo back, SCR Echo Back: System::ContainerStartTime->9/23/2022 3:04:08 PM
Information: 0x0 at SCR Echo back, SCR Echo Back: User::IsLoopValid->False
Information: 0x0 at SCR Echo back, SCR Echo Back: User::SEQCEndTime->9/23/2022 3:03:54 PM
SSIS package "C:\Users\bfellows\source\repos\SO_Trash\SO_Trash\SO_73829526.dtsx" finished: Success.
Expression and Constraint
We see the package stop doing the expensive task after 2 loops but does complete successfully.
SSIS package "C:\Users\bfellows\source\repos\SO_Trash\SO_Trash\SO_73829526.dtsx" starting.
Information: 0x0 at SCR Echo back, SCR Echo Back 0: System::ContainerStartTime->9/23/2022 3:05:36 PM
Information: 0x0 at SCR Echo back, SCR Echo Back 0: User::SEQCEndTime->9/23/2022 3:05:41 PM
Information: 0x0 at SCR Echo back, SCR Echo Back: System::ContainerStartTime->9/23/2022 3:05:39 PM
Information: 0x0 at SCR Echo back, SCR Echo Back: User::IsLoopValid->True
Information: 0x0 at SCR Echo back, SCR Echo Back: User::SEQCEndTime->9/23/2022 3:05:41 PM
Information: 0x0 at SCR Echo back, SCR Echo Back: System::ContainerStartTime->9/23/2022 3:05:42 PM
Information: 0x0 at SCR Echo back, SCR Echo Back: User::IsLoopValid->True
Information: 0x0 at SCR Echo back, SCR Echo Back: User::SEQCEndTime->9/23/2022 3:05:41 PM
SSIS package "C:\Users\bfellows\source\repos\SO_Trash\SO_Trash\SO_73829526.dtsx" finished: Success.