pass multiple dates to foreach loop container - sql

I have a process where I want to pass multiple dates that have not been loaded to my foreach loop container to iterate through dates one at a time. if there are no start dates in the table that have been loaded I have the isnull in the where clause to tell it to start on the min date from that table. Currently if there is a date in the table that has been loaded it will grab the next calendar date from the table to process but wont loop through all the dates. My goal is to have it pass all dates that have not been loaded and iterate through each date one at a time. So what I would like to do is to take the min and max date from ods.CalendarDate and load everything between those dates and including those dates that has not been loaded to dbo.Customerinformation. Below is my query at the moment, please help me right this to where it will select all the dates that have not been processed from the table. Thank you
DECLARE #ProcessAllDataSet BIT = ?
DECLARE #StartDate Date =?
DECLARE #EndDate date = ?
IF #LoadFullDataSet = 1
BEGIN
SELECT #StartDate = MIN(StartDate),
#EndDate = Max(StartDate) FROM dbo.CustomerInformation
END
SELECT DateProcessed = MIN(DateCalendar)
FROM ods.CalendarDate c
WHERE c.CalendarDate > ISNULL(#EndDate,'2001-01-01')

I think you're pretty close. I'm assuming ods.CalendarDate is a table of calendar dates. If so, just modify your query to return all the dates between your #StartDate and #EndDate. Make sure you set your ResultSet property of the Execute SQL Task to Full Result Set and map the results to an Object-type SSIS variable.
DECLARE #ProcessAllDataSet BIT = ?
DECLARE #StartDate Date =?
DECLARE #EndDate date = ?
IF #ProcessAllDataSet = 1
BEGIN
SELECT
#StartDate = MIN(StartDate),
#EndDate = Max(StartDate)
FROM dbo.CustomerInformation
END
SELECT
DateProcessed = DateCalendar
FROM ods.CalendarDate c
WHERE c.CalendarDate BETWEEN ISNULL(#StartDate,'2001-01-01') AND ISNULL(#EndDate,'2100-01-01')
Then set your Foreach Loop to use the Foreach ADO Enumerator and use your Object variable as the ADO object source variable. Map the DateProcessed field to a Date variable and then you can use that Date variable as you iterate through process.

Related

Data factory- SQL query of historical data with periods of one minute

I'm trying to make a copy activity to copy data from an on-premise SQL database with data factory. I need the to modify dynamiccaly the query, so it takes a range of dates, for example ranges of every minute:
So for example: the first copy would be using dynamical content as
...
AND DateTime > '20230131 13:06'
AND DateTime < '20230131 13:07'
and then, the next run would be:
AND DateTime > '20230131 13:07'
AND DateTime < '20230131 13:08'
I tried to use variables like:
AND DateTime > #{variables('starttime')}
AND DateTime < #{variables('endtime')}
It works when I give the times manually to variables but the actual data is for a year, so I need to read the start and end times from a file or somehow automate it. I tried to use "ForEach" block, but the problem is that I can only set one variable as a loop, either start or end time.
what is the best solution for this?
How can I use "ForEach" block on start time and enter the end time in a way that is one minute after the start time?
You can read the start time file, loop through it and add a minute to each time in for loop.
The following is the output of my look up of csv file containing start_times.
Loop through this data. Inside for each I have 2 set variable activities, one for converting start time to a valid format so minute can be added using addMinutes() function.
#concat(substring(item().start_time,0,4),'-',substring(item().start_time,4,2),'-',substring(item().start_time,6,2),' ',substring(item().start_time,9,5))
Now, for this format, add minutes and replace the - value concatenated above.
#replace(addminutes(variables('tp'),1,'yyyy-MM-dd hh:mm'),'-','')
I'm not sure how this works in Data Factory but in t-sql, you can increase your variables by one minute with the DATEADD function. Fiddle
DATEADD(*interval*, *number of intervals*, *start datetime*)
I checked Books Online and it looks like data factory uses datetime_add. It appears to work the same way though.
DATETIME_ADD(*interval*, *number of intervals*, *start datetime*)
I hope this helps.
DECLARE #startdate datetime = '20230131 13:06'
, #enddate datetime
, #finishdate datetime;
SET #enddate = DATEADD(minute, 1, #startdate);
SET #finishdate = DATEADD(minute, 100, #startdate); --will loop 100 times.
--change that to whatever you want.
WHILE #startdate < #finishdate
BEGIN
SELECT data
INTO newTable
FROM oldTable
WHERE oldTableDate >= #startdate
AND oldTableDate <= #enddate;
SELECT #startdate = DATEADD(minute, 1, #startdate),
#enddate = DATEADD(minute, 1, #enddate)
END

SQL query based on two values in a column with a date range

So here is what I'm attempting to accomplish. I want to have returned all rows within a specified date range where the value in a column is one value 'C' or another 'X'. I'm not huge in SQL and much more comfortable in scriptwriting for excel or google sheets. I would assume there's a pretty simple solution but here is what I have written so far that's not working.
Declare #FromDate DateTime
Declare #ToDate DateTime
Set #FromDate='20201027'
Set #ToDAte ='20201102'
select * from oehdr where OrderStatus='C' or OrderStatus='X'
The errors I've received have been "Invalid SQL, No Memory" and a script error when I modify the current attempt.
There's not enough info in the question. We don't know the name of the date column, or whether the time is included, or whether the ToDate value is inclusive of the whole day. It's not even 100% clear exactly what DB you're using, as there's some conflicting info with the question.
But I'll do the best I can making some guessues:
Declare #FromDate DateTime = '20201027'
Declare #ToDate DateTime = '20201102'
SELECT *
FROM oehdr
WHERE OrderStatus IN ('C', 'X')
AND OrderDate >= #FromDate AND OrderDate < DATEADD(day, 1, #ToDate)
Notice I did not use the BETWEEN operator. BETWEEN is inclusive at both ends of the range, which is rarely what I want for dates. Instead, when I want the full day, I add one day and use an exclusive condition for the end of the range.

I am tring to add a year to a date if it is not before today.

I am tring to add a year to a date if it is not before today. So in the statement below I would like to display 4/20/2018
declare #StartDate datetime
set #StartDate='4/20/2015'
select case when dateadd(year,1,#StartDate)> GETDATE() then
dateadd(year,1,#StartDate) else dATEADD(year,1,(datepart(year,GETDATE()))) end
I've changed the else clause so it works out the difference between the start date and today's date +1 to alter make '4/20/2015' become '4/20/2018'
declare #StartDate date
set #StartDate='4/20/2015'
select
case
when dateadd(year,1,#StartDate)> GETDATE() then dateadd(year,1,#StartDate)
else DATEADD(year,(datepart(year,GETDATE())+1-datepart(year,#StartDate)),#StartDate)
end
Also if you are not using the time part of a datetime data type it's best practice to use date instead as it only requires 3 bytes of storage rather than 8 for datetime.
More on date here: https://learn.microsoft.com/en-us/sql/t-sql/data-types/date-transact-sql
And a cheatsheet of sql data types here: https://sqlserverrider.files.wordpress.com/2013/04/pic116.png

Counting the amount of Days in a date range

I'm trying to get a query that will multiply a static number by the amount of days in a date range the problem i'm having is that when a single day is selected it returns a result of 0 instead of 1: Ex:
Declare #Startdate DATE
Declare #enddate DATE
SET #Startdate='9/1/2013'
SET #enddate='9/1/2013'
SELECT 1154*(Select DATEDIFF(DAY, #startdate, #enddate))
This example returns 0 instead of 1. Should I be using something other than DateDiff?
Additional clarification - This will be used as part of a report where the date range will be dynamically entered by person calling the report.
Could just add 1:
Declare #Startdate DATE
Declare #enddate DATE
SET #Startdate='9/1/2013'
SET #enddate='9/1/2013'
SELECT (DATEDIFF(day,#Startdate,#enddate)+1)*1154
Update, as noted don't need the inner SELECT

Creating a list of all the dates between a range and then insert it

I have a problem were I need to insert a set of dates (one row for each date) that come from a date range that comes in to me as an ical format. I also need to insert the date if the date does not exist.
I'm happy manipulating the date formats so converting the incoming 20131201T000000 to smalldatetime is ok and I can build the SQL Not Exits as separate bits but how do I go about listing all the dates between 20131201 and 20140101 and then go about inserting them into the table if Not Exists along with some other data.
One idea I had would be to count the days using DateDiff which will give me a total number of days in the range and then I could DateAdd one day at a time and insert for each date until the DateDiff total is reached. However, it seems to be that its a very messy and long winded way to go about it and I would find it challenging to say the least.
Is there another method that someone could walk me through or point me in the right direction of?
In my opinion you are much better off letting SQL Server determine the set of dates that are missing than to create a set of dates, then pull back all of the existing dates in the table to compare them to your set, then insert the ones that don't match. By doing this you are taking what SQL Server is really, really good at, tossing it in the mud, and re-inventing a wheel that is now quite non-circular.
So I suggest you do this in SQL Server.
Imagine you have an existing table with some dates:
CREATE TABLE dbo.MyTable(TheDate DATE);
And it has some existing days in the specified range:
INSERT dbo.MyTable(TheDate) VALUES
('20131203'),('20131205'),('20131209'),('20131230');
You can easily derive a set without using loops as follows.
CREATE PROCEDURE dbo.GenerateDates
#start CHAR(17),
#end CHAR(17)
AS
BEGIN
SET NOCOUNT ON;
DECLARE #s DATE = LEFT(#start, 8), #e DATE = LEFT(#end, 8);
DECLARE #d SMALLINT = DATEDIFF(DAY, #s, #e);
;WITH x(d) AS
(
SELECT TOP (#d) DATEADD(DAY, number, #s)
FROM master..spt_values
WHERE type = N'P' ORDER BY number
)
-- INSERT dbo.MyTable(TheDate)
SELECT d FROM x WHERE NOT EXISTS
(SELECT 1 FROM dbo.MyTable WHERE TheDate = x.d);
END
GO
Sample usage:
EXEC dbo.GenerateDates
#start = '20131201T000000',
#end = '20140101T000000';
When you are happy this is giving the output you expect, alter the procedure and uncomment the INSERT.
(Also it would be best if you just pass date values from your app, instead of these 17-character strings, but I was working with what you have.)
Well lets see. You have a date called beginDate and date called endDate. Lets get all they days between those dates into list:
List<DateTime> betweenDates = new List<DateTime>();
var currentDate = beginDate;
while(currentDate <= endDate)
{
betweenDates.Add(currentDate);
currentDate = currentDate.AddDays(1);
}
Then lets fetch all the rows that are already in database:
// var datesInDb = SELECT DATE FROM TABLE_X WHERE DATE BETWEEN (beginDate, EndDate);
Now we have dates from database and all the dates that should be in database. Lets substract dates from database from our betweenDates
var datesWhichShouldBeInsertedIntoDB = betweenDates.Any(day => datesInDb.Contains(day) == false);
then insert datesWhichShouldBeInsertedIntoDB