Is there a way to cycle through consecutive data values for a variable in my code?
My date variable updates every Saturday, so whenever I update my table I have to manually backdate the code by changing the data variable every execution. This is a very lengthy process as I am backdating 30+ dates with a code that takes 1-2 minutes to run.
For example, if I loaded a secondary table with the date of every Saturday in 2020-2021 (for now), could I cycle the variable through the values?
Declare #Saturday Date
Set #Saturday = '2021-04-24'
Declare #Friday Date
Set #Friday = '2021-04-30'
;
I had a similar thing where I had to recalculate values per day and if some configuration changed we had to backdate those changes per day for x number of days
I used something like this to achieve the result, essentially starting at the furtherst point back and moving forwards but you can run it the other way if that is preferable. On the other hand the code I was running would take a few seconds per execution rather than 1-2 minutes.
Declare #Saturday Date
Set #Saturday = '2020-01-04';
while #Saturday < GETDATE()
begin
print #saturday; /*do something*/
set #Saturday = DATEADD(WEEK, 1, #Saturday);
end
Related
I created a new SSIS package (SQL 2017) that reads through an "Export Request" table and exports data based on the schedule to an external storage. For example, users can add a new row to this table like "select * from table 1 where col1=2" with the schedule as June 25, 2018 3:30 pm.
My package (which is running using SQL agent) loops all the time in search of new tasks in that table and when scheduled, exports the result of the queries to a folder at that time.
There is a new feature requested to have these exported scheduled for recurring. For example, customers might want to export every day starting June 25, 2018 3:30 pm. The schedule can get complicated for annually, monthly, daily, every 3 month, ...
What is the best way to implement recurring inside SSIS? I can use a 5 char field for Cron but don't know how to query for new tasks using their Cron schedule.
Any help?
To confirm what you already have: You have a TASK table with tasks in it and some sort of date flag for when it needs to run. Then your job runs every x number of minutes and checks if there are any tasks to be picked up/run. If so it runs them. Then i am guessing you have a flag in the table you set saying it is completed after you run it in your loop? If you have multiple tasks to run at a time it just runs them all one at a time?
Then what you want to do is add this to it: something I did something very similar. I created a lookup table for each task (in my process each task had a name and I used that name to reference the lookup table, I called it schedule lookup).
In that table I put when and how often the report would need to run (so 2 columns) time of day and frequency. One example is 7:00 and weekdays. So for this report it runs every weekday only (m-F) and at 7pm.
Then when my job runs it would run the task like you have above, but then have another step, that would flag that task as complete, but then insert a new task in the task table (the task details would be the same) but I would look at my schedule lookup table described above to figure out the next date/time when the job should run again and use that as my next run date/time in my task table.
Below is the SP I use in my process to get the next schedule day/time and to update the existing one to completed, and then to create the new one.
NOTE: My scheduling has some advanced options you may not need, I have LOTS of comments that should explain what/why I am doing everything. I am calling a few functions I created in places but I dont think you would need those and can figure out what to do instead of my functions, but if you have questions let me know:
This is what I used so it is using my table structures/etc, but you could convert it to yours easily enough.
--Purpose
----------------------------------------------------------------------------------
-- calculates the next time to run/schedule the job
----------------------------------------------------------------------------------
-- NOTE: TO USE you have to insert the first value manually in queue table
----- possibile scenerios
-- if we want to schedule every x hours, or x days
-- run every month only
-- run weekdays only
-- run on certain days of month only
-- TO ADD MORE COMPLEX or different types of schedules:
-- special - different times for different days of week
-- ex - so have dayofweek:2:00,dayofweek:3:00 (and we parse out the day of week and number splitting out the strings)
-- hourly - to do more then once a day??
-- WHEN #ScheduleLookupType = 'hourly' THEN DATEADD(DAY, 1, #CurrentScheduleDate) -- FIX FIX FIX
-- EXEC dbo.JobsDynamicRescheduleFindNextTimeToScheduleJob #ReportName = 'TestReport1'
----------------------------------------------------------------------------------
ALTER PROCEDURE [dbo].[JobsDynamicRescheduleFindNextTimeToScheduleJob]
#ReportName VARCHAR(50)
AS
SET NOCOUNT ON
BEGIN TRY
-- left here for testing outside of SP
-- this will be passed from SP
--DECLARE #ReportName AS VARCHAR(50)
--SET #ReportName = 'TESTREport'
-- this sets the first day of the week to Monday (we need it set to a value to do calcluations for weekdays, I set it to 1 for monday and 7 for sunday)
-- this is due to server settings could have somethign else so forcing it here
SET DATEFIRST 1
DECLARE #CurrentScheduleDate AS DATE -- find the current date for the job that just ran
DECLARE #CurrentScheduleDayNumberOfWeek AS SMALLINT -- this pulls the number of the day of week 1=monday, 2=tuesdday
DECLARE #ScheduleLookupType AS VARCHAR(20) -- this is the type of schedule to do calculations on
DECLARE #TimeOfDayToScheduleJob AS VARCHAR(20) -- look this up, its the time to schedule the job
DECLARE #SpecialScheduleValue AS VARCHAR(8000) -- this is special value to lookup (only needed if non standard one)
DECLARE #NewScheduleDateONLY AS DATETIME -- to hold just the date of the schedule before combinng with time
DECLARE #NewScheduleDateTime AS DATETIME -- to hold the new schedule date and time, actual value to insert into queue
-- pull the current schedule date/time from the queue table
SELECT #CurrentScheduleDate = NextRunDateTime
FROM dbo.GenericReportingQueue (NOLOCK)
WHERE IsGenerated IS NULL
AND ReportName = #ReportName
-- to override for testing
--SET #CurrentScheduleDate = '5/20/2016'
SET #CurrentScheduleDayNumberOfWeek = DATEPART(WEEKDAY, #CurrentScheduleDate)
-- pull these values from lookup table
SELECT #ScheduleLookupType = dbo.fn_GetValueLookupTableValue(#ReportName, 'RescheduleJobDynamic_ScheduleLookupType'),
#TimeOfDayToScheduleJob = dbo.fn_GetValueLookupTableValue(#ReportName, 'RescheduleJobDynamic_TimeOfDayToScheduleJob'),
#SpecialScheduleValue = dbo.fn_GetValueLookupTableValue(#ReportName, 'RescheduleJobDynamic_SpecialScheduleValue')
/*
-- reset for testing
SET #ScheduleLookupType = 'specialdays' -- weekly, weekdays, monthly, specialdays
SET #TimeOfDayToScheduleJob = '8:00'
SET #SpecialScheduleValue = '5,6'
*/
-- calculations to get the date to schedule the job next time based off logic
SELECT #NewScheduleDateONLY = CASE
WHEN #ScheduleLookupType = 'daily' THEN DATEADD(DAY, 1, #CurrentScheduleDate)
WHEN #ScheduleLookupType = 'weekly' THEN DATEADD(DAY, 7, #CurrentScheduleDate)
WHEN #ScheduleLookupType = 'monthly' THEN DATEADD(MONTH, 1, #CurrentScheduleDate)
WHEN #ScheduleLookupType = 'yearly' THEN DATEADD(YEAR, 1, #CurrentScheduleDate)
-- only run on weekdays and skip weekends
WHEN #ScheduleLookupType = 'weekdays' THEN
CASE
WHEN #CurrentScheduleDayNumberOfWeek IN (1, 2, 3, 4) THEN DATEADD(DAY, 1, #CurrentScheduleDate)
WHEN #CurrentScheduleDayNumberOfWeek = 5 THEN DATEADD(DAY, 3, #CurrentScheduleDate)
END -- end case for day of week
-- only run on weekends and skip weekdays
WHEN #ScheduleLookupType = 'weekends' THEN
CASE
WHEN #CurrentScheduleDayNumberOfWeek = 6 THEN DATEADD(DAY, 1, #CurrentScheduleDate)
WHEN #CurrentScheduleDayNumberOfWeek = 7 THEN DATEADD(DAY, 6, #CurrentScheduleDate)
END -- end case for weekends only
WHEN #ScheduleLookupType = 'specialdays' THEN
-- for this we need to determine the current day, and the next day we want to run on, then add that many days
-- if next day is not till the following week we just find the first day in the list
-- Take taht number and do dateadd to it
DATEADD(DAY,
-- this does the select to determine what number to add based off current day and next day list
(SELECT ISNULL(
-- if this one I want to take today value and subtract from next value found
-- then add that number to todays date to give me the next schedule date
(SELECT TOP 1 StringValue - #CurrentScheduleDayNumberOfWeek
FROM dbo.fn_ParseText2Table(#SpecialScheduleValue, ',')
WHERE StringValue > #CurrentScheduleDayNumberOfWeek
ORDER BY StringValue)
,
-- if none found above I need to go to the next weeks first value
-- I need to take 7 - todays number (to get the rest of the week) then add the next number for the next week to it
(SELECT TOP 1 (7 - #CurrentScheduleDayNumberOfWeek) + StringValue
FROM dbo.fn_ParseText2Table(#SpecialScheduleValue, ',')
ORDER BY StringValue)
)-- end is null
) -- end select
, #CurrentScheduleDate) -- end dateadd for speical days
END -- outer case
SET #NewScheduleDateTime = #NewScheduleDateONLY + ' ' + #TimeOfDayToScheduleJob
-- for testing
--SELECT #ScheduleLookupType AS ReportLookupType, #TimeOfDayToScheduleJob AS TimeOfDayToSchedule, #SpecialScheduleValue AS SpecialValuesForCalc, #NewScheduleDateTime AS NewDateTimeToRun,
--#CurrentScheduleDate AS CurrentDateSchedule, #CurrentScheduleDayNumberOfWeek AS CurrentNumberDayOfWeek, #NewScheduleDateONLY AS NewScheduleDateOnly
-- &&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&
-- now update and insert the new schedule date/time into the table
-- update existing record
UPDATE dbo.GenericReportingQueue
SET IsGenerated = 1,
DateReportRun = GETDATE(),
LastUpdateDate = GETDATE()
WHERE ISGenerated IS NULL
AND ReportName = #ReportName
-- insert new record with new date
INSERT INTO dbo.GenericReportingQueue (
ReportName, NextRunDateTime, CreatorID, CreateDate
)
SELECT #ReportName, #NewScheduleDateTime, 1, GETDATE()
END TRY
BEGIN CATCH
RETURN
END CATCH
I'm trying to create a flexible way to determine the last day of the previous week, knowing that this will be run in different environments that have different planning weeks (some customers use Sat-fri; Sun-Sat; Mon-Sun). I was planning on creating a variable to use as each customer's start day of the week (see below)
Set #BeginningWeek = 1 -- 1=Mon; 2=Tues; 3=Wed; 4=Thurs; 5=Fri; 6=Sat; 7=Sun
I've been trying to use a combination of dateadd and datepart, along with the variable above to try to get the previous last day of the week, but I'm having trouble with the logic. I do not want to use Set DateFirst, these scripts will be run on a customer environment, so I do not want to change anything in their DB.
Assuming the script is run on a wednesday (9/3); and the customer's planning week is sat-fri; I would want to set another variable #EndDate = the previous friday(8/29). I need the flexibility to use the same script (changing only the #BeginningWeek value) with another customer whose planning week is Mon-Sun and set #EndDate = the previous sunday (8/31).
This will get you the previous Saturday.
SELECT DATEADD(day,
-1 - (DATEPART(dw, GETDATE()) + ##DATEFIRST - 2) % 6,
GETDATE()
)
You should be able to replace the %6 bit with a parameter based on what the actual desired day is.
I'm looking to schedule a job on my SQL Server to run three times a day (8AM, 1PM, & 7PM) from January to June of every year and run the same job once a month for rest of the year.
I have tried to find an answer and so for I have not been successful. Any help will be greatly appreciated.
Thanks!
(This has largely been addressed by the comments. I am converting their insight into an official answer.)
The built-in scheduler is only so sophisticated. It doesn't support the same schedule varying depending on the time of the year. However, it does allow multiple schedules, and you can perform your own scheduling logic in the SQL statement.
If you don't mind adjusting your job schedules each year, you can avoid having any custom scheduling logic in the SQL statement by setting up 4 separate schedules. You would set up the first 3 schedules as occurring Daily, every 1 day, occurring once at the relevant time of day (8 AM, 1 PM, or 7 PM). You would set an End date for those schedules of June 30th, 2014. You would then create another schedule which occurs monthly, on whatever day of the month you would like, every 1 month, at whatever time you would like. You would set the start date of that schedule to July 1st, 2014 and you would specify the end date as December 31st, 2014.
As stated, there is a big disadvantage to this method; you must change the schedule each year, as the current year is hard-coded into the schedules. (If desired, you can add the next year's schedules at any time, in advance-- even making several years-- and you can leave the old schedules until well after their end dates.) However, there are two advantages to the method. First, you don't have to write any scheduling logic in SQL, which is more likely to have bugs and would hide the scheduling reality from the casual observer. Second, the job only fires when it is supposed to, so your job history shows only those executions which matter. (If you are performing the scheduling logic in SQL, then you have the fire the job more frequently, letting it be squelched by the SQL scheduling logic but also cluttering up the job history.) A third, minor advantage is avoiding the very minor performance hit of all of the extra job executions and SQL scheduling logic evaluation.
If you are interested in using some scheduling logic inside the job's SQL statement, instead of making a larger number of specifically defined schedules, you could create a single schedule which fired every hour of every day. Then you would wrap the existing SQL statement in logic like the following:
DECLARE #TestDate datetime, #ExecuteSQL bit;
SELECT #TestDate = getdate(), #ExecuteSQL = 0;
-- Uncomment each of these lines (one at a time) to test the logic with various dates.
--SET #TestDate = '2014-07-02 8:00 AM';
--SET #TestDate = '2014-07-01 8:00 AM';
--SET #TestDate = '2014-07-01 9:00 AM';
--SET #TestDate = '2014-06-01 9:00 AM';
--SET #TestDate = '2014-06-02 1:00 PM';
--SET #TestDate = '2014-06-02 6:03 PM';
--SET #TestDate = '2014-06-02 7:03 PM';
SET #ExecuteSQL =
CASE
WHEN
-- it's between January and June and it's in the 8 AM, 1 PM, or 7 PM hours
(
DATEPART(m,#TestDate) BETWEEN 1 AND 6
AND
DATEPART(hh,#TestDate) IN (8,13,19)
)
OR
-- it's not between January and June, but it's the first of the month during the 8 AM hour
(
DATEPART(m,#TestDate) NOT BETWEEN 1 AND 6
AND
DATEPART(d,#TestDate) = 1
AND
DATEPART(hh,#TestDate) IN (8)
)
THEN
1
ELSE
0
END;
IF #ExecuteSQL <> 0
BEGIN
SELECT 'Do Fun Stuff Here';
END
ELSE
BEGIN
SELECT 'You would probably omit this ELSE, but it is maintained for demonstration purposes.';
END
I need to get data from previous day (sql server db) for transferring to other DB (postgress), but since data is huge I want to transfer records for just 2 hr, I mean I will run this job 12 times a day and each time it will transfer rec for 2 hr, record transfered should not be duplicate.
So basically I need a query which I can schedule to run 12 times and which will transfer records for two hours each.
declare #StartHour datetime, #EndHour datetime
set #EndHour = dateadd(hh,datediff(hh,0,GetDate()),0)
set #StartHour = dateadd(hh,-2,#EndHour)
--The above make the query work with start and end on-the-hour
--so it can be run any time within one hour to get the data
--for the two hours ending on the previous hour
select * from whatever where TheDate between #StartHour and #EndHour
If you are timestamping inserts, then it should be simple to run a select to pull out only the preceding two hours worth of records.
SELECT * FROM tblFoo WHERE tblFoo.insertionDate>DATEADD(hour,-2,GETDATE())
(If you want to be exact, then don't use GETDATE but hold the last date that ran in a table or variable somewhere and add two hours to it each time then setting it after running the query)
Hopefully this will be an easy one to answer.
I am working on a table that requires MTD data. One of our SQL guys told me to use
MONTH (#monthtodate)= 11
Where #monthtodate is set to GetDate() in the parameter list in SQL Server Management Studio. So in "theory", he says, it should select the month (11) and then get today and return all the requested data in between those two dates. But I'm thinking this isn't correct.
In looking at my data I'm starting to think that It's just returning data for the whole month of November instead of just MTD. I guess, technically, anything that has 0 won't be calculated. However that just means it's poorly written code correct?
In your opinions, would this be the better way to return MTD data:
production_date <= #today and Production_Date >= DATEADD(mm, DATEDIFF(mm, 0, #today), 0)
Thanks in advance everyone!
Here's how I do it. This should work on pretty much any version of SQL Server.
One important thing to note: at the outset, one should always establish a single value that represents 'now', the current moment in time. If you do not have a consistent value for now in your query, you will eventually get bit when your query is executed such that it crosses a date boundary whilst in-flight. Nothing like billing somebody for something they already paid for last month. Worst, edge-case bugs like that are difficult to catch, either by developers or by QA, since neither is likely to be working, say, at 11:59 on December 31.
The code:
declare
#dtNow datetime ,
#Today datetime ,
#dtFrom datetime ,
#dtThru datetime
---------------------------------------------------------------------------------------
-- set our effective notion of 'now'-ness.
--
-- We need have a consistent notion of now, lest we get bit in the a$$
-- by an edge case where we cross a day/month/year boundary in mid-execution.
--
-- NOTE: Mostly, we're interested in the *DATE* rather than the actual moment-in-time.
-- So, we carry around two flavors here.
---------------------------------------------------------------------------------------
set #dtNow = current_timestamp
set #Today = convert(datetime,convert(varchar,#dtNow,112),112)
---------------------------------------------------------------------------------------
-- compute the current date.
--
-- 1. get the current date sans timestamp (effectively start-of-day)
-- 2. add 1 day, then back off 3 millseconds to set it to the last tick of the current day
--
-- NOTE: Depending on the requirements of your particular application (and the nature
-- of your data), you might want to use the actual current date/time value as
-- your upper bound.
--
-- FURTHER NOTE: How far to back off is dependent on your date/time type:
--
-- * For DateTime, the resolution is milliseconds and the last tick of the day
-- is 997 milliseconds, so you need to back off 3ms from the start of the
-- next day.
--
-- * SmallDateTime has a 1 second resolution. The last tick of the day, natch,
-- is 59 seconds, so you need to back off 1 second from the start of the next day.
--
-- * For DateTime2, the user declares the precision in decimal fractions of a second,
-- though its resolution is 100ns ticks. You'll need (especially if you're working
-- with DateTime2 columns/variables of differing precision) experiment to figure out
-- what traps Microsoft has set for you inside DateTime2 and what you need to do to
-- make things work properly.
--
---------------------------------------------------------------------------------------
set #dtThru = dateadd(ms,-3,dateadd(day,1,#Today))
--set #dtThru = #dtNow -- if you need the actual current date/time value
---------------------------------------------------------------------------------------
-- compute start of month
--
-- We do this by subtracting the day number of 'today' from the date/time value #today.
-- That gives us the last day of the prior month. Then we add one day to get the first
-- day of the current month.
---------------------------------------------------------------------------------------
set #dtFrom = dateadd(day,1-day(#Today),#Today)
---------------------------------------------------------------------------------------
-- finally, make your query for 'current month to date'
---------------------------------------------------------------------------------------
select *
from dbo.foobar t
where t.recorded_date between #dtFrom and #dtThru
If you are asking which of these 2 queries is better from a performance standpoint:
DECLARE #now datetime = GETDATE()
SELECT *
FROM yourTable
WHERE Production_Date >= DATEADD(mm, DATEDIFF(mm, 0, #now), 0)
AND Production_Date < #now
SELECT *
FROM yourTable
WHERE YEAR(Production_Date) = YEAR(#now)
AND MONTH(Production_Date) = MONTH(#now)
AND Production_Date < #now
Then the first one would be, since it will use the index on Production_Date if there is one. However, they should both return the same results.