I have a table of jobs that have ran for different source systems. These have a "RunDate" and then the FromDate and ToDate.
I want to find the gaps where we are missing any dates in the FromDate and ToDate fields to make sure that we have covered the data in those periods.
Many examples I've looked at work where a single date misses in a single column of ranges, however I have a From and To range that I need to test and ultimately work out where a date may be missed.
CREATE TABLE #temptable ( [SourceSystem] nchar(3), [RunDate] datetime, [ResubmitCount] int, [FromDate] date, [ToDate] date )
INSERT INTO #temptable
VALUES
( N'ILG', N'2021-07-28T15:35:23.207', 0, N'2021-06-01T00:00:00', N'2021-06-01T00:00:00' ),
( N'ILG', N'2021-07-28T15:35:23.707', 0, N'2021-06-05T00:00:00', N'2021-06-06T00:00:00' ),
( N'AAP', N'2021-07-28T15:35:23.833', 0, N'2021-06-01T00:00:00', N'2021-06-02T00:00:00' ),
( N'AAP', N'2021-07-28T15:35:23.833', 0, N'2021-06-04T00:00:00', N'2021-06-04T00:00:00' ),
( N'ZZP', N'2021-07-28T15:35:23.897', 0, N'2021-06-05T00:00:00', N'2021-06-05T00:00:00' )
DROP TABLE #temptable
So obviously using the example above I should be able to ascertain that the period between 2021-06-02 and 2021-06-04 for SourceSystem ILG and period 2021-06-03 to 2021-06-03 is missing for SourceSystem AAP.
Struggling to make it work for ranges, I can work with single dates but the system doesn't log them in this fasion.
UPDATE
I took the accepted answer and then tagged some code to it to be able to explode all the individual dates between the ranges specified.
Included the code in case anyone needs in the future.
WITH
a AS (
SELECT
SourceSystem, FromDate, ToDate,
LEAD(FromDate) OVER(
PARTITION BY SourceSystem
ORDER BY RunDate
) AS NextDate
FROM dbo.WDSubmission ws
),
gap_periods AS
(
SELECT
SourceSystem,
DATEADD(DAY, 1, ToDate) AS GapBeg,
DATEADD(DAY, -1, NextDate) AS GapFin
FROM a
WHERE
NextDate IS NOT NULL AND
DATEADD(DAY, -2, NextDate) >= ToDate
--AND a.SourceSystem = 'OGI'
) , E00(N) AS (SELECT 1 UNION ALL SELECT 1)
,E02(N) AS (SELECT 1 FROM E00 a, E00 b)
,E04(N) AS (SELECT 1 FROM E02 a, E02 b)
,E08(N) AS (SELECT 1 FROM E04 a, E04 b)
,E16(N) AS (SELECT 1 FROM E08 a, E08 b)
,E32(N) AS (SELECT 1 FROM E16 a, E16 b)
,cteTally(N) AS (SELECT ROW_NUMBER() OVER (ORDER BY (SELECT NULL)) FROM E32)
,DateRange AS
(
SELECT ExplodedDate = DATEADD(DAY,N - 1,'2021-01-01')
FROM cteTally
WHERE N <= DATEDIFF(DAY,'2021-01-01',GETDATE())
)
SELECT *
FROM gap_periods eh
JOIN DateRange d ON d.ExplodedDate >= eh.GapBeg
AND d.ExplodedDate <= eh.GapFin;
Try this:
WITH
a AS (
SELECT
SourceSystem, FromDate, ToDate,
LEAD(FromDate) OVER(
PARTITION BY SourceSystem
ORDER BY RunDate
) AS NextDate
FROM #temptable
)
SELECT
SourceSystem,
DATEADD(DAY, 1, ToDate) AS GapBeg,
DATEADD(DAY, -1, NextDate) AS GapFin
FROM a
WHERE
NextDate IS NOT NULL AND
DATEADD(DAY, -2, NextDate) >= ToDate;
Result:
+--------------+------------+------------+
| SourceSystem | GapBeg | GapFin |
+--------------+------------+------------+
| AAP | 2021-06-03 | 2021-06-03 |
| ILG | 2021-06-02 | 2021-06-04 |
+--------------+------------+------------+
db-fiddle
Its hard to display the missing dates and also it would be too much information to analyze. Bbut you can get the number of days skipped in between using the below query :
select *,datediff(day,lag(ToDate)over(partition by sourcesystem order by ToDate),fromdate) from #temptable
The data would look something like this (last column tells the number of days skipped) :
Related
I have data in a table in this format - where date range is multi-month:
SourceSink Class ShadowPrice Round Period StartDate EndDate
AEC Peak 447.038 3 WIN2020 2020-12-01 2021-02-28
I want to create a view/ insert into a new table - the above record broken by month as shown below:
SourceSink Class ShadowPrice Round Period StartDate EndDate
AEC Peak 447.038 3 WIN2020 2020-12-01 2021-12-31
AEC Peak 447.038 3 WIN2020 2021-01-01 2021-01-31
AEC Peak 447.038 3 WIN2020 2021-02-01 2021-02-28
Please advise.
One option is a recursive query. Assuming that periods always start on the the first day of a month and end on the last day of a month, as shown in your sample data, that would be:
with cte as (
select t.*, startDate newStartDate, eomonth(startDate) newEndDate
from mytable t
union all
select
sourceSink,
class,
shadowPrice,
period,
startDate,
endDate,
dateadd(month, 1, newStartDate),
eomonth(dateadd(month, 1, newStartDate))
from cte
where newStartDate < endDate
)
select * from cte
If periods start and end on variying month days, then we need a little more logic:
with cte as (
select
t.*,
startDate newStartDate,
case when eomonth(startDate) <= endDate then eomonth(startDate) else endDate end newEndDate
from mytable t
union all
select
sourceSink,
class,
shadowPrice,
period,
startDate,
endDate,
dateadd(month, 1, datefromparts(year(newStartDate), month(newStartDate), 1)),
case when eomonth(dateadd(month, 1, datefromparts(year(newStartDate), month(newStartDate), 1))) <= endDate
then eomonth(dateadd(month, 1, datefromparts(year(newStartDate), month(newStartDate), 1)))
else endDate
end
from cte
where datefromparts(year(newStartDate), month(newStartDate), 1) < endDate
)
select * from cte
Just another option using a CROSS APPLY and an ad-hoc tally table
Example
Select A.[SourceSink]
,A.[Class]
,A.[ShadowPrice]
,A.[Round]
,A.[Period]
,B.[StartDate]
,B.[EndDate]
From YourTable A
Cross Apply (
Select StartDate=min(D)
,EndDate =max(D)
From (
Select Top (DateDiff(DAY,[StartDate],[EndDate])+1)
D=DateAdd(DAY,-1+Row_Number() Over (Order By (Select Null)),[StartDate])
From master..spt_values n1,master..spt_values n2
) B1
Group By Year(D),Month(D)
) B
Returns
I have this table:
id | PlateNumber | ImgDate | ImgTime|
1 | abc123 | 2020-03-01 | 09:10:33|
2 | qwe128 | 2020-03-02| 09:13:39|
3 | abc123 | 2020-03-01| 09:11:21|
So, there are too many repeated plate numbers based on image capturing time.
I need a select statement that returns repeated records only when there is a time difference of more than 3 minutes. I hope that this makes sense.
Please help
Thank you guys!
You need to provide a bit more info, but you would want to compare the current row with the previous one (for each PlateNumber). You can do that using an analytical function like LAG.
WITH CTE as (
Select
id
,PlateNumber
,ImgDate
,ImgTime
,LAG(ImgDate) OVER (PARTITION BY PlateNumber ORDER BY ImgDate,ImgTime) as PreviousDate
,LAG(ImgTime) OVER (PARTITION BY PlateNumber ORDER BY ImgDate,ImgTime) as PreviousImgTime
FROM YourTable
)
Select id
,PlateNumber
,ImgDate
,ImgTime
FROM CTE
WHERE (PreviousDate is null and PreviousImgTime is null) -- to keep the first occurence
OR (
DATEDIFF(second,
DATEADD(day, 0, DATEDIFF(day, 0, PreviousDate)) + DATEADD(day, 0 - DATEDIFF(day, 0, PreviousImgTime), PreviousImgTime) ,
DATEADD(day, 0, DATEDIFF(day, 0, ImgDate)) + DATEADD(day, 0 - DATEDIFF(day, 0, ImgTime), ImgTime)
) > 180
)
If your columns are DATE and TIME datatype
OR (
DATEDIFF(second,
cast(PreviousDate as DATETIME) +cast(PreviousImgTime as DATETIME) ,
cast(ImgDate as DATETIME) + cast(ImgTime as DATETIME),
) > 180
)
In SQL Server, you can add datetime values, so you can do this basically just with lag():
select t.*
from (select t.*, v.dt,
lag(dt) over (partition by platenumber order by dt) as prev_dt
from t cross apply
(values (convert(datetime, t.imgdate) + convert(datetime, t.imgtime))
) as v(dt)
) t
where prev_dt is null or prev_dt < dateadd(minute, -3, dt);
I have two tables each containing the start and end dates of several periods. I want an efficient way to find periods (date ranges) where dates are within the ranges of the first table but not within ranges of the second table.
For example, if this is my first table (with dates that I want)
start_date end_date
2001-01-01 2010-01-01
2012-01-01 2015-01-01
And this is my second table (with dates that I do not want)
start_date end_date
2002-01-01 2006-01-01
2003-01-01 2004-01-01
2005-01-01 2009-01-01
2014-01-01 2018-01-01
Then output looks like
start_date end_date
2001-01-01 2001-12-31
2009-01-02 2010-01-01
2012-01-01 2013-12-31
We can safely assume that periods in the first table are non-overlapping, but can not assume periods in the second table are overlapping.
I already have a method for doing this but it is an order of magnitude slower than I can accept. So hoping someone can propose a faster approach.
My present method looks like:
merge table 2 into non-overlapping periods
find the inverse of table 2
join overlapping periods from table 1 and inverted-table-2
I am sure there is a faster way if some of these steps can be merged together.
In more detail
/* (1) merge overlapping preiods */
WITH
spell_starts AS (
SELECT [start_date], [end_date]
FROM table_2 s1
WHERE NOT EXISTS (
SELECT 1
FROM table_2 s2
WHERE s2.[start_date] < s1.[start_date]
AND s1.[start_date] <= s2.[end_date]
)
),
spell_ends AS (
SELECT [start_date], [end_date]
FROM table_2 t1
WHERE NOT EXISTS (
SELECT 1
FROM table_2 t2
WHERE t2.[start_date] <= t1.[end_date]
AND t1.[end_date] < t2.[end_date]
)
)
SELECT s.[start_date], MIN(e.[end_date]) as [end_date]
FROM spell_starts s
INNER JOIN spell_ends e
ON s.[start_date] <= e.[end_date]
GROUP BY s.[start_date]
/* (2) inverse table 2 */
SELECT [start_date], [end_date]
FROM (
/* all forward looking spells */
SELECT DATEADD(DAY, 1, [end_date]) AS [start_date]
,LEAD(DATEADD(DAY, -1, [start_date]), 1, '9999-01-01') OVER ( ORDER BY [start_date] ) AS [end_date]
FROM merge_table_2
UNION ALL
/* back looking spell (to 'origin of time') created separately */
SELECT '1900-01-01' AS [start_date]
,DATEADD(DAY, -1, MIN([start_date])) AS [end_date]
FROM merge_table_2
) k
WHERE [start_date] <= [end_date]
AND '1900-01-01' <= [start_date]
AND [end_date] <= '9999-01-01'
/* (3) overlap spells */
SELECT IIF(t1.start_date < t2.start_date, t2.start_date, t1.start_date) AS start_date
,IIF(t1.end_date < t2.end_date, t1.end_date, t2.end_date) AS end_date
FROM table_1 t1
INNER JOIN inverse_merge_table_2 t2
ON t1.start_date < t2.end_date
AND t2.start_date < t1.end_date
Hope this helps. I have comment the two ctes I am using for explanation purposes
Here you go:
drop table table1
select cast('2001-01-01' as date) as start_date, cast('2010-01-01' as date) as end_date into table1
union select '2012-01-01', '2015-01-01'
drop table table2
select cast('2002-01-01' as date) as start_date, cast('2006-01-01' as date) as end_date into table2
union select '2003-01-01', '2004-01-01'
union select '2005-01-01', '2009-01-01'
union select '2014-01-01', '2018-01-01'
/***** Solution *****/
-- This cte put all dates into one column
with cte as
(
select t
from
(
select start_date as t
from table1
union all
select end_date
from table1
union all
select dateadd(day,-1,start_date) -- for table 2 we bring the start date back one day to make sure we have nothing in the forbidden range
from table2
union all
select dateadd(day,1,end_date) -- for table 2 we bring the end date forward one day to make sure we have nothing in the forbidden range
from table2
)a
),
-- This one adds an end date using the lead function
cte2 as (select t as s, coalesce(LEAD(t,1) OVER ( ORDER BY t ),t) as e from cte a)
-- this query gets all intervals not in table2 but in table1
select s, e
from cte2 a
where not exists(select 1 from table2 b where s between dateadd(day,-1,start_date) and dateadd(day,1,end_date) and e between dateadd(day,-1,start_date) and dateadd(day,1,end_date) )
and exists(select 1 from table1 b where s between start_date and end_date and e between start_date and end_date)
and s <> e
If you want performance, then you want to use window functions.
The idea is to:
Combine the dates with flags of being in-and-out of the two tables.
Use cumulative sums to determine where dates start being in-and-out.
Then you have a gaps-and-islands problem where you want to combine the results.
Finally, filter on the particular periods you want.
This looks like:
with dates as (
select start_date as dte, 1 as in1, 0 as in2
from table1
union all
select dateadd(day, 1, end_date), -1, 0
from table1
union all
select start_date, 0, 1 as in2
from table2
union all
select dateadd(day, 1, end_date), 0, -1
from table2
),
d as (
select dte,
sum(sum(in1)) over (order by dte) as ins_1,
sum(sum(in2)) over (order by dte) as ins_2
from dates
group by dte
)
select min(dte), max(next_dte)
from (select d.*, dateadd(day, -1, lead(dte) over (order by dte)) as next_dte,
row_number() over (order by dte) as seqnum,
row_number() over (partition by case when ins_1 >= 1 and ins_2 = 0 then 'in' else 'out' end order by dte) as seqnum_2
from d
) d
group by (seqnum - seqnum_2)
having max(ins_1) > 0 and max(ins_2) = 0
order by min(dte);
Here is a db<>fiddle.
Thanks to #zip and #Gordon for their answers. Both were superior to my initial approach. However, the following solution was faster than both of their approaches in my environment & context:
WITH acceptable_starts AS (
SELECT [start_date] FROM table1 AS a
WHERE NOT EXISTS (
SELECT 1 FROM table2 AS b
WHERE DATEADD(DAY, 1, a.[end_date]) BETWEEN b.[start_date] AND b.
UNION ALL
SELECT DATEADD(DAY, 1, [end_date]) FROM table2 AS a
WHERE NOT EXISTS (
SELECT 1 FROM table2 AS b
WHERE DATEADD(DAY, 1, a.[end_date]) BETWEEN b.[start_date] AND b.[end_date]
)
),
acceptable_ends AS (
SELECT [end_date] FROM table1 AS a
WHERE NOT EXISTS (
SELECT 1 FROM table2 AS b
WHERE DATEADD(DAY, -1, a.[start_date]) BETWEEN b.[start_date] AND b.[end_date]
)
UNION ALL
SELECT DATEADD(DAY, -1, [start_date]) FROM table2 AS a
WHERE NOT EXISTS (
SELECT 1 FROM table2 AS b
WHERE DATEADD(DAY, -1, a.[start_date]) BETWEEN b.[start_date] AND b.[end_date]
)
)
SELECT s.[start_date], MIN(e.[end_date]) AS [end_date]
FROM acceptable_starts
INNER JOIN acceptable_ends
ON s.[start_date] < e.[end_date]
I am explain problem in short.
select distinct DATE from #Table where DATE >='2016-01-01'
Output :
Date
2016-11-23
2016-11-22
2016-11-21
2016-11-19
2016-11-18
Now i need to find out missing date a compare to our calender dates from year '2016'
i.e. Here date '2016-11-20' is missing.
I want list of missing dates.
Thanks for reading this. Have nice day.
You need to generate dates and you have to find missing ones. Below with recursive cte i have done it
;WITH CTE AS
(
SELECT CONVERT(DATE,'2016-01-01') AS DATE1
UNION ALL
SELECT DATEADD(DD,1,DATE1) FROM CTE WHERE DATE1<'2016-12-31'
)
SELECT DATE1 MISSING_ONE FROM CTE
EXCEPT
SELECT * FROM #TABLE1
option(maxrecursion 0)
Using CTE and get all dates in CTE table then compare with your table.
CREATE TABLE #yourTable(_Values DATE)
INSERT INTO #yourTable(_Values)
SELECT '2016-11-23' UNION ALL
SELECT '2016-11-22' UNION ALL
SELECT '2016-11-21' UNION ALL
SELECT '2016-11-19' UNION ALL
SELECT '2016-11-18'
DECLARE #DATE DATE = '2016-11-01'
;WITH CTEYear (_Date) AS
(
SELECT #DATE
UNION ALL
SELECT DATEADD(DAY,1,_Date)
FROM CTEYear
WHERE _Date < EOMONTH(#DATE,0)
)
SELECT * FROM CTEYear
WHERE NOT EXISTS(SELECT 1 FROM #yourTable WHERE _Date = _Values)
OPTION(maxrecursion 0)
You need to generate the dates and then find the missing ones. A recursive CTE is one way to generate a handful of dates. Another way is to use master..spt_values as a list of numbers:
with n as (
select row_number() over (order by (select null)) - 1 as n
from master..spt_values
),
d as (
select dateadd(day, n.n, cast('2016-01-01' as date)) as dte
from n
where n <= 365
)
select d.date
from d left join
#table t
on d.dte = t.date
where t.date is null;
If you are happy enough with ranges of missing dates, you don't need a list of dates at all:
select date, (datediff(day, date, next_date) - 1) as num_missing
from (select t.*, lead(t.date) over (order by t.date) as next_date
from #table t
where t.date >= '2016-01-01'
) t
where next_date <> dateadd(day, 1, date);
What I need is to calculate the missing time periods within the calendar year given a table such as this in SQL:
DatesTable
|ID|DateStart |DateEnd |
1 NULL NULL
2 2015-1-1 2015-12-31
3 2015-3-1 2015-12-31
4 2015-1-1 2015-9-30
5 2015-1-1 2015-3-31
5 2015-6-1 2015-12-31
6 2015-3-1 2015-6-30
6 2015-7-1 2015-10-31
Expected return would be:
1 2015-1-1 2015-12-31
3 2015-1-1 2015-2-28
4 2015-10-1 2015-12-31
5 2015-4-1 2015-5-31
6 2015-1-1 2015-2-28
6 2015-11-1 2015-12-31
It's essentially work blocks. What I need to show is the part of the calendar year which was NOT worked. So for ID = 3, he worked from 3/1 through the rest of the year. But he did not work from 1/1 till 2/28. That's what I'm looking for.
You can do it using LEAD, LAG window functions available from SQL Server 2012+:
;WITH CTE AS (
SELECT ID,
LAG(DateEnd) OVER (PARTITION BY ID ORDER BY DateEnd) AS PrevEnd,
DateStart,
DateEnd,
LEAD(DateStart) OVER (PARTITION BY ID ORDER BY DateEnd) AS NextStart
FROM DatesTable
)
SELECT ID, DateStart, DateEnd
FROM (
-- Get interval right before current [DateStart, DateEnd] interval
SELECT ID,
CASE
WHEN DateStart IS NULL THEN '20150101'
WHEN DateStart > start THEN start
ELSE NULL
END AS DateStart,
CASE
WHEN DateStart IS NULL THEN '20151231'
WHEN DateStart > start THEN DATEADD(d, -1, DateStart)
ELSE NULL
END AS DateEnd
FROM CTE
CROSS APPLY (SELECT COALESCE(DATEADD(d, 1, PrevEnd), '20150101')) x(start)
-- If there is no next interval then get interval right after current
-- [DateStart, DateEnd] interval (up-to end of year)
UNION ALL
SELECT ID, DATEADD(d, 1, DateEnd) AS DateStart, '20151231' AS DateEnd
FROM CTE
WHERE DateStart IS NOT NULl -- Do not re-examine [Null, Null] interval
AND NextStart IS NULL -- There is no next [DateStart, DateEnd] interval
AND DateEnd < '20151231' -- Current [DateStart, DateEnd] interval
-- does not terminate on 31/12/2015
) AS t
WHERE t.DateStart IS NOT NULL
ORDER BY ID, DateStart
The idea behind the above query is simple: for every [DateStart, DateEnd] interval get 'not worked' interval right before it. If there is no interval following the current interval, then also get successive 'not worked' interval (if any).
Also note that I assume that if DateStart is NULL then DateStart is also NULL for the same ID.
Demo here
If your data is not too big, this approach will work. It expands all the days and ids and then re-groups them:
with d as (
select cast('2015-01-01' as date)
union all
select dateadd(day, 1, d)
from d
where d < cast('2015-12-31' as date)
),
td as (
select *
from d cross join
(select distinct id from t) t
where not exists (select 1
from t t2
where d.d between t2.startdate and t2.enddate
)
)
select id, min(d) as startdate, max(d) as enddate
from (select td.*,
dateadd(day, - row_number() over (partition by id order by d), d) as grp
from td
) td
group by id, grp
order by id, grp;
An alternative method relies on cumulative sums and similar functionality that is much easier to expression in SQL Server 2012+.
Somewhat simpler approach I think.
Basically create a list of dates for all work block ranges (A). Then create a list of dates for the whole year for each ID (B). Then remove the A from B. Compile the remaining list of dates into date ranges for each ID.
DECLARE #startdate DATETIME, #enddate DATETIME
SET #startdate = '2015-01-01'
SET #enddate = '2015-12-31'
--Build date ranges from remaining date list
;WITH dateRange(ID, dates, Grouping)
AS
(
SELECT dt1.id, dt1.Dates, dt1.Dates + row_number() over (order by dt1.id asc, dt1.Dates desc) AS Grouping
FROM
(
--Remove (A) from (B)
SELECT distinct dt.ID, tmp.Dates FROM DatesTable dt
CROSS APPLY
(
--GET (B) here
SELECT DATEADD(DAY, number, #startdate) [Dates]
FROM master..spt_values
WHERE type = 'P' AND DATEADD(DAY, number, #startdate) <= #enddate
) tmp
left join
(
--GET (A) here
SELECT DISTINCT T.Id,
D.Dates
FROM DatesTable AS T
INNER JOIN master..spt_values as N on N.number between 0 and datediff(day, T.DateStart, T.DateEnd)
CROSS APPLY (select dateadd(day, N.number, T.DateStart)) as D(Dates)
WHERE N.type ='P'
) dr
ON dr.Id = dt.Id and dr.Dates = tmp.Dates
WHERE dr.id is null
) dt1
)
SELECT ID, CAST(MIN(Dates) AS DATE) DateStart, CAST(MAX(Dates) AS DATE) DateEnd
FROM dateRange
GROUP BY ID, Grouping
ORDER BY ID
Heres the code:
http://sqlfiddle.com/#!3/f3615/1
I hope this helps!