Recurring tasks using SSIS - sql

Question

I created a new SSIS package (SQL 2017) that reads through an "Export Request" table and exports data based on the schedule to an external storage. For example, users can add a new row to this table like "select * from table 1 where col1=2" with the schedule as June 25, 2018 3:30 pm.
My package (which is running using SQL agent) loops all the time in search of new tasks in that table and when scheduled, exports the result of the queries to a folder at that time.
There is a new feature requested to have these exported scheduled for recurring. For example, customers might want to export every day starting June 25, 2018 3:30 pm. The schedule can get complicated for annually, monthly, daily, every 3 month, ...
What is the best way to implement recurring inside SSIS? I can use a 5 char field for Cron but don't know how to query for new tasks using their Cron schedule.
Any help?

Answer 1

To confirm what you already have: You have a TASK table with tasks in it and some sort of date flag for when it needs to run. Then your job runs every x number of minutes and checks if there are any tasks to be picked up/run. If so it runs them. Then i am guessing you have a flag in the table you set saying it is completed after you run it in your loop? If you have multiple tasks to run at a time it just runs them all one at a time?
Then what you want to do is add this to it: something I did something very similar. I created a lookup table for each task (in my process each task had a name and I used that name to reference the lookup table, I called it schedule lookup).
In that table I put when and how often the report would need to run (so 2 columns) time of day and frequency. One example is 7:00 and weekdays. So for this report it runs every weekday only (m-F) and at 7pm.
Then when my job runs it would run the task like you have above, but then have another step, that would flag that task as complete, but then insert a new task in the task table (the task details would be the same) but I would look at my schedule lookup table described above to figure out the next date/time when the job should run again and use that as my next run date/time in my task table.
Below is the SP I use in my process to get the next schedule day/time and to update the existing one to completed, and then to create the new one.
NOTE: My scheduling has some advanced options you may not need, I have LOTS of comments that should explain what/why I am doing everything. I am calling a few functions I created in places but I dont think you would need those and can figure out what to do instead of my functions, but if you have questions let me know:
This is what I used so it is using my table structures/etc, but you could convert it to yours easily enough.
--Purpose
----------------------------------------------------------------------------------
-- calculates the next time to run/schedule the job
----------------------------------------------------------------------------------
-- NOTE: TO USE you have to insert the first value manually in queue table
----- possibile scenerios
-- if we want to schedule every x hours, or x days
-- run every month only
-- run weekdays only
-- run on certain days of month only
-- TO ADD MORE COMPLEX or different types of schedules:
-- special - different times for different days of week
-- ex - so have dayofweek:2:00,dayofweek:3:00 (and we parse out the day of week and number splitting out the strings)
-- hourly - to do more then once a day??
-- WHEN #ScheduleLookupType = 'hourly' THEN DATEADD(DAY, 1, #CurrentScheduleDate) -- FIX FIX FIX
-- EXEC dbo.JobsDynamicRescheduleFindNextTimeToScheduleJob #ReportName = 'TestReport1'
----------------------------------------------------------------------------------
ALTER PROCEDURE [dbo].[JobsDynamicRescheduleFindNextTimeToScheduleJob]
#ReportName VARCHAR(50)
AS
SET NOCOUNT ON
BEGIN TRY
-- left here for testing outside of SP
-- this will be passed from SP
--DECLARE #ReportName AS VARCHAR(50)
--SET #ReportName = 'TESTREport'
-- this sets the first day of the week to Monday (we need it set to a value to do calcluations for weekdays, I set it to 1 for monday and 7 for sunday)
-- this is due to server settings could have somethign else so forcing it here
SET DATEFIRST 1
DECLARE #CurrentScheduleDate AS DATE -- find the current date for the job that just ran
DECLARE #CurrentScheduleDayNumberOfWeek AS SMALLINT -- this pulls the number of the day of week 1=monday, 2=tuesdday
DECLARE #ScheduleLookupType AS VARCHAR(20) -- this is the type of schedule to do calculations on
DECLARE #TimeOfDayToScheduleJob AS VARCHAR(20) -- look this up, its the time to schedule the job
DECLARE #SpecialScheduleValue AS VARCHAR(8000) -- this is special value to lookup (only needed if non standard one)
DECLARE #NewScheduleDateONLY AS DATETIME -- to hold just the date of the schedule before combinng with time
DECLARE #NewScheduleDateTime AS DATETIME -- to hold the new schedule date and time, actual value to insert into queue
-- pull the current schedule date/time from the queue table
SELECT #CurrentScheduleDate = NextRunDateTime
FROM dbo.GenericReportingQueue (NOLOCK)
WHERE IsGenerated IS NULL
AND ReportName = #ReportName
-- to override for testing
--SET #CurrentScheduleDate = '5/20/2016'
SET #CurrentScheduleDayNumberOfWeek = DATEPART(WEEKDAY, #CurrentScheduleDate)
-- pull these values from lookup table
SELECT #ScheduleLookupType = dbo.fn_GetValueLookupTableValue(#ReportName, 'RescheduleJobDynamic_ScheduleLookupType'),
#TimeOfDayToScheduleJob = dbo.fn_GetValueLookupTableValue(#ReportName, 'RescheduleJobDynamic_TimeOfDayToScheduleJob'),
#SpecialScheduleValue = dbo.fn_GetValueLookupTableValue(#ReportName, 'RescheduleJobDynamic_SpecialScheduleValue')
/*
-- reset for testing
SET #ScheduleLookupType = 'specialdays' -- weekly, weekdays, monthly, specialdays
SET #TimeOfDayToScheduleJob = '8:00'
SET #SpecialScheduleValue = '5,6'
*/
-- calculations to get the date to schedule the job next time based off logic
SELECT #NewScheduleDateONLY = CASE
WHEN #ScheduleLookupType = 'daily' THEN DATEADD(DAY, 1, #CurrentScheduleDate)
WHEN #ScheduleLookupType = 'weekly' THEN DATEADD(DAY, 7, #CurrentScheduleDate)
WHEN #ScheduleLookupType = 'monthly' THEN DATEADD(MONTH, 1, #CurrentScheduleDate)
WHEN #ScheduleLookupType = 'yearly' THEN DATEADD(YEAR, 1, #CurrentScheduleDate)
-- only run on weekdays and skip weekends
WHEN #ScheduleLookupType = 'weekdays' THEN
CASE
WHEN #CurrentScheduleDayNumberOfWeek IN (1, 2, 3, 4) THEN DATEADD(DAY, 1, #CurrentScheduleDate)
WHEN #CurrentScheduleDayNumberOfWeek = 5 THEN DATEADD(DAY, 3, #CurrentScheduleDate)
END -- end case for day of week
-- only run on weekends and skip weekdays
WHEN #ScheduleLookupType = 'weekends' THEN
CASE
WHEN #CurrentScheduleDayNumberOfWeek = 6 THEN DATEADD(DAY, 1, #CurrentScheduleDate)
WHEN #CurrentScheduleDayNumberOfWeek = 7 THEN DATEADD(DAY, 6, #CurrentScheduleDate)
END -- end case for weekends only
WHEN #ScheduleLookupType = 'specialdays' THEN
-- for this we need to determine the current day, and the next day we want to run on, then add that many days
-- if next day is not till the following week we just find the first day in the list
-- Take taht number and do dateadd to it
DATEADD(DAY,
-- this does the select to determine what number to add based off current day and next day list
(SELECT ISNULL(
-- if this one I want to take today value and subtract from next value found
-- then add that number to todays date to give me the next schedule date
(SELECT TOP 1 StringValue - #CurrentScheduleDayNumberOfWeek
FROM dbo.fn_ParseText2Table(#SpecialScheduleValue, ',')
WHERE StringValue > #CurrentScheduleDayNumberOfWeek
ORDER BY StringValue)
,
-- if none found above I need to go to the next weeks first value
-- I need to take 7 - todays number (to get the rest of the week) then add the next number for the next week to it
(SELECT TOP 1 (7 - #CurrentScheduleDayNumberOfWeek) + StringValue
FROM dbo.fn_ParseText2Table(#SpecialScheduleValue, ',')
ORDER BY StringValue)
)-- end is null
) -- end select
, #CurrentScheduleDate) -- end dateadd for speical days
END -- outer case
SET #NewScheduleDateTime = #NewScheduleDateONLY + ' ' + #TimeOfDayToScheduleJob
-- for testing
--SELECT #ScheduleLookupType AS ReportLookupType, #TimeOfDayToScheduleJob AS TimeOfDayToSchedule, #SpecialScheduleValue AS SpecialValuesForCalc, #NewScheduleDateTime AS NewDateTimeToRun,
--#CurrentScheduleDate AS CurrentDateSchedule, #CurrentScheduleDayNumberOfWeek AS CurrentNumberDayOfWeek, #NewScheduleDateONLY AS NewScheduleDateOnly
-- &&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&
-- now update and insert the new schedule date/time into the table
-- update existing record
UPDATE dbo.GenericReportingQueue
SET IsGenerated = 1,
DateReportRun = GETDATE(),
LastUpdateDate = GETDATE()
WHERE ISGenerated IS NULL
AND ReportName = #ReportName
-- insert new record with new date
INSERT INTO dbo.GenericReportingQueue (
ReportName, NextRunDateTime, CreatorID, CreateDate
)
SELECT #ReportName, #NewScheduleDateTime, 1, GETDATE()
END TRY
BEGIN CATCH
RETURN
END CATCH