Related
I have two tables each containing the start and end dates of several periods. I want an efficient way to find periods (date ranges) where dates are within the ranges of the first table but not within ranges of the second table.
For example, if this is my first table (with dates that I want)
start_date end_date
2001-01-01 2010-01-01
2012-01-01 2015-01-01
And this is my second table (with dates that I do not want)
start_date end_date
2002-01-01 2006-01-01
2003-01-01 2004-01-01
2005-01-01 2009-01-01
2014-01-01 2018-01-01
Then output looks like
start_date end_date
2001-01-01 2001-12-31
2009-01-02 2010-01-01
2012-01-01 2013-12-31
We can safely assume that periods in the first table are non-overlapping, but can not assume periods in the second table are overlapping.
I already have a method for doing this but it is an order of magnitude slower than I can accept. So hoping someone can propose a faster approach.
My present method looks like:
merge table 2 into non-overlapping periods
find the inverse of table 2
join overlapping periods from table 1 and inverted-table-2
I am sure there is a faster way if some of these steps can be merged together.
In more detail
/* (1) merge overlapping preiods */
WITH
spell_starts AS (
SELECT [start_date], [end_date]
FROM table_2 s1
WHERE NOT EXISTS (
SELECT 1
FROM table_2 s2
WHERE s2.[start_date] < s1.[start_date]
AND s1.[start_date] <= s2.[end_date]
)
),
spell_ends AS (
SELECT [start_date], [end_date]
FROM table_2 t1
WHERE NOT EXISTS (
SELECT 1
FROM table_2 t2
WHERE t2.[start_date] <= t1.[end_date]
AND t1.[end_date] < t2.[end_date]
)
)
SELECT s.[start_date], MIN(e.[end_date]) as [end_date]
FROM spell_starts s
INNER JOIN spell_ends e
ON s.[start_date] <= e.[end_date]
GROUP BY s.[start_date]
/* (2) inverse table 2 */
SELECT [start_date], [end_date]
FROM (
/* all forward looking spells */
SELECT DATEADD(DAY, 1, [end_date]) AS [start_date]
,LEAD(DATEADD(DAY, -1, [start_date]), 1, '9999-01-01') OVER ( ORDER BY [start_date] ) AS [end_date]
FROM merge_table_2
UNION ALL
/* back looking spell (to 'origin of time') created separately */
SELECT '1900-01-01' AS [start_date]
,DATEADD(DAY, -1, MIN([start_date])) AS [end_date]
FROM merge_table_2
) k
WHERE [start_date] <= [end_date]
AND '1900-01-01' <= [start_date]
AND [end_date] <= '9999-01-01'
/* (3) overlap spells */
SELECT IIF(t1.start_date < t2.start_date, t2.start_date, t1.start_date) AS start_date
,IIF(t1.end_date < t2.end_date, t1.end_date, t2.end_date) AS end_date
FROM table_1 t1
INNER JOIN inverse_merge_table_2 t2
ON t1.start_date < t2.end_date
AND t2.start_date < t1.end_date
Hope this helps. I have comment the two ctes I am using for explanation purposes
Here you go:
drop table table1
select cast('2001-01-01' as date) as start_date, cast('2010-01-01' as date) as end_date into table1
union select '2012-01-01', '2015-01-01'
drop table table2
select cast('2002-01-01' as date) as start_date, cast('2006-01-01' as date) as end_date into table2
union select '2003-01-01', '2004-01-01'
union select '2005-01-01', '2009-01-01'
union select '2014-01-01', '2018-01-01'
/***** Solution *****/
-- This cte put all dates into one column
with cte as
(
select t
from
(
select start_date as t
from table1
union all
select end_date
from table1
union all
select dateadd(day,-1,start_date) -- for table 2 we bring the start date back one day to make sure we have nothing in the forbidden range
from table2
union all
select dateadd(day,1,end_date) -- for table 2 we bring the end date forward one day to make sure we have nothing in the forbidden range
from table2
)a
),
-- This one adds an end date using the lead function
cte2 as (select t as s, coalesce(LEAD(t,1) OVER ( ORDER BY t ),t) as e from cte a)
-- this query gets all intervals not in table2 but in table1
select s, e
from cte2 a
where not exists(select 1 from table2 b where s between dateadd(day,-1,start_date) and dateadd(day,1,end_date) and e between dateadd(day,-1,start_date) and dateadd(day,1,end_date) )
and exists(select 1 from table1 b where s between start_date and end_date and e between start_date and end_date)
and s <> e
If you want performance, then you want to use window functions.
The idea is to:
Combine the dates with flags of being in-and-out of the two tables.
Use cumulative sums to determine where dates start being in-and-out.
Then you have a gaps-and-islands problem where you want to combine the results.
Finally, filter on the particular periods you want.
This looks like:
with dates as (
select start_date as dte, 1 as in1, 0 as in2
from table1
union all
select dateadd(day, 1, end_date), -1, 0
from table1
union all
select start_date, 0, 1 as in2
from table2
union all
select dateadd(day, 1, end_date), 0, -1
from table2
),
d as (
select dte,
sum(sum(in1)) over (order by dte) as ins_1,
sum(sum(in2)) over (order by dte) as ins_2
from dates
group by dte
)
select min(dte), max(next_dte)
from (select d.*, dateadd(day, -1, lead(dte) over (order by dte)) as next_dte,
row_number() over (order by dte) as seqnum,
row_number() over (partition by case when ins_1 >= 1 and ins_2 = 0 then 'in' else 'out' end order by dte) as seqnum_2
from d
) d
group by (seqnum - seqnum_2)
having max(ins_1) > 0 and max(ins_2) = 0
order by min(dte);
Here is a db<>fiddle.
Thanks to #zip and #Gordon for their answers. Both were superior to my initial approach. However, the following solution was faster than both of their approaches in my environment & context:
WITH acceptable_starts AS (
SELECT [start_date] FROM table1 AS a
WHERE NOT EXISTS (
SELECT 1 FROM table2 AS b
WHERE DATEADD(DAY, 1, a.[end_date]) BETWEEN b.[start_date] AND b.
UNION ALL
SELECT DATEADD(DAY, 1, [end_date]) FROM table2 AS a
WHERE NOT EXISTS (
SELECT 1 FROM table2 AS b
WHERE DATEADD(DAY, 1, a.[end_date]) BETWEEN b.[start_date] AND b.[end_date]
)
),
acceptable_ends AS (
SELECT [end_date] FROM table1 AS a
WHERE NOT EXISTS (
SELECT 1 FROM table2 AS b
WHERE DATEADD(DAY, -1, a.[start_date]) BETWEEN b.[start_date] AND b.[end_date]
)
UNION ALL
SELECT DATEADD(DAY, -1, [start_date]) FROM table2 AS a
WHERE NOT EXISTS (
SELECT 1 FROM table2 AS b
WHERE DATEADD(DAY, -1, a.[start_date]) BETWEEN b.[start_date] AND b.[end_date]
)
)
SELECT s.[start_date], MIN(e.[end_date]) AS [end_date]
FROM acceptable_starts
INNER JOIN acceptable_ends
ON s.[start_date] < e.[end_date]
In need of some help :)
So I have a table of records with the following columns:
Key (PK, FK, int) DT (smalldatetime) Value (real)
The DT is a datetime for every half hour of the day with an associated value
E.g.
Key DT VALUE
1000 2010-01-01 08:00:00 80
1000 2010-01-01 08:30:00 75
1000 2010-01-01 09:00:00 100
I have a Query that finds the max value every 24 hour period and its associated time however, on one day the max value occurs twice and hence duplicates the date which is causing processing issues. I have tried using rownumber() which works but I can't use a calculated column in my where clause?
Currently I have:
SELECT cast(T1.DT as date) as 'Date',Cast(T1.DT as time(0)) as 'HH', ROW_NUMBER() over (PARTITION BY cast(DT as date) ORDER BY DT) AS 'RowNumber'
FROM TABLE_1 AS T1
INNER JOIN (
SELECT CAST([DT] as date) as 'DATE'
, MAX([VALUE]) as 'MAX_HH'
FROM TABLE_1
WHERE DT > '6-nov-2016' and [KEY] = '1000'
GROUP BY CAST([DT] as date)
) AS MAX_DT
ON MAX_DT.[DATE] = CAST(T1.[DT] as date)
AND T1.VALUE = MAX_DT.MAX_HH
WHERE DT > '6-nov-2016' and [KEY] = '1000'
ORDER BY DT
This results in
Key DT VALUE HH
1000 2010-01-01 80 07:00:00
1000 2010-02-01 100 17:30:00
1000 2010-02-01 100 18:00:00
I need to remove the duplicate date (I Have no preference which HH it takes)
I think I've explained that terribly, let me know if it makes no sense and i'll try and re write
Any ideas?
Can you try this the new code is in ** **:
SELECT cast(T1.DT as date) as 'Date', ** MIN(Cast(T1.DT as time(0))) as 'HH' **
FROM TABLE_1 AS T1
INNER JOIN (
SELECT CAST([DT] as date) as 'DATE'
, MAX([VALUE]) as 'MAX_HH'
FROM TABLE_1
WHERE DT > '6-nov-2016' and [KEY] = '1000'
GROUP BY CAST([DT] as date)
) AS MAX_DT
ON MAX_DT.[DATE] = CAST(T1.[DT] as date)
AND T1.VALUE = MAX_DT.MAX_HH
WHERE DT > '6-nov-2016' and [KEY] = '1000'
here put the group by
GROUP BY cast(T1.DT as date)
ORDER BY DT
i would do something like this
i didnt try it but i think it s correct.
SELECT cast(T1.DT as date) as 'Date',Cast(T1.DT as time(0)) as 'HH', VALUE
FROM TABLE_1 T1
WHERE [DT] IN (
--select the max date from Table_1 for each day
SELECT MAX([DT]) max_date FROM TABLE_1
WHERE (CAST([DT] as date) ,value) IN
(
SELECT CAST([DT] as date) as 'CAST_DATE'
,MAX([VALUE]) as 'MAX_HH'
FROM TABLE_1
WHERE DT > '6-nov-2016' and [KEY] = '1000'
GROUP BY CAST([DT] as date
)group by [DT]
)
WHERE DT > '6-nov-2016' and [KEY] = '1000'
Change the JOIN to an APPLY.
The APPLY operation will allow you to limit the connected relation to just one result for each source relation.
SELECT v.[Key], cast(v.DT As Date) as "Date", v.[Value], cast(v.DT as Time(0)) as "HH"
FROM
( -- First a projection to get just the exact dates you want
SELECT DISTINCT [Key], CAST(DT as DATE) as DT
FROM Table_1
WHERE [Key] = '1000' AMD DT > '20161106'
) dates
CROSS APPLY (
-- Then use APPLY rather than JOIN to find just the exact one record you need for each date
SELECT TOP 1 *
FROM Table_1
WHERE [Key] = dates.[Key] AND cast(DT as DATE) = dates.DT ORDER BY [Value] DESC
) v
A final note: Both this query and your sample query in the question will include values from Nov 6, 2016. The query says > 2016-11-05 with an exlusive inequality, but the original was still comparing using full DateTime values, meaning there is a implied 0 as a time component. So 12:01 AM on Nov 6 is still greater than 12:00:00.001 AM on Nov 6. If you want to exclude all Nov 6 dates from the query, you either need to change this to use a time value at the end of the date, or cast to date before making that > comparison.
With SQL you can use SELECT DISTINCT,
The SELECT DISTINCT statement is used to return only distinct (different) values.
Inside a table, a column often contains many duplicate values; and sometimes you only want to list the different (distinct) values.
The SELECT DISTINCT statement is used to return only distinct (different) values.
What I need is to calculate the missing time periods within the calendar year given a table such as this in SQL:
DatesTable
|ID|DateStart |DateEnd |
1 NULL NULL
2 2015-1-1 2015-12-31
3 2015-3-1 2015-12-31
4 2015-1-1 2015-9-30
5 2015-1-1 2015-3-31
5 2015-6-1 2015-12-31
6 2015-3-1 2015-6-30
6 2015-7-1 2015-10-31
Expected return would be:
1 2015-1-1 2015-12-31
3 2015-1-1 2015-2-28
4 2015-10-1 2015-12-31
5 2015-4-1 2015-5-31
6 2015-1-1 2015-2-28
6 2015-11-1 2015-12-31
It's essentially work blocks. What I need to show is the part of the calendar year which was NOT worked. So for ID = 3, he worked from 3/1 through the rest of the year. But he did not work from 1/1 till 2/28. That's what I'm looking for.
You can do it using LEAD, LAG window functions available from SQL Server 2012+:
;WITH CTE AS (
SELECT ID,
LAG(DateEnd) OVER (PARTITION BY ID ORDER BY DateEnd) AS PrevEnd,
DateStart,
DateEnd,
LEAD(DateStart) OVER (PARTITION BY ID ORDER BY DateEnd) AS NextStart
FROM DatesTable
)
SELECT ID, DateStart, DateEnd
FROM (
-- Get interval right before current [DateStart, DateEnd] interval
SELECT ID,
CASE
WHEN DateStart IS NULL THEN '20150101'
WHEN DateStart > start THEN start
ELSE NULL
END AS DateStart,
CASE
WHEN DateStart IS NULL THEN '20151231'
WHEN DateStart > start THEN DATEADD(d, -1, DateStart)
ELSE NULL
END AS DateEnd
FROM CTE
CROSS APPLY (SELECT COALESCE(DATEADD(d, 1, PrevEnd), '20150101')) x(start)
-- If there is no next interval then get interval right after current
-- [DateStart, DateEnd] interval (up-to end of year)
UNION ALL
SELECT ID, DATEADD(d, 1, DateEnd) AS DateStart, '20151231' AS DateEnd
FROM CTE
WHERE DateStart IS NOT NULl -- Do not re-examine [Null, Null] interval
AND NextStart IS NULL -- There is no next [DateStart, DateEnd] interval
AND DateEnd < '20151231' -- Current [DateStart, DateEnd] interval
-- does not terminate on 31/12/2015
) AS t
WHERE t.DateStart IS NOT NULL
ORDER BY ID, DateStart
The idea behind the above query is simple: for every [DateStart, DateEnd] interval get 'not worked' interval right before it. If there is no interval following the current interval, then also get successive 'not worked' interval (if any).
Also note that I assume that if DateStart is NULL then DateStart is also NULL for the same ID.
Demo here
If your data is not too big, this approach will work. It expands all the days and ids and then re-groups them:
with d as (
select cast('2015-01-01' as date)
union all
select dateadd(day, 1, d)
from d
where d < cast('2015-12-31' as date)
),
td as (
select *
from d cross join
(select distinct id from t) t
where not exists (select 1
from t t2
where d.d between t2.startdate and t2.enddate
)
)
select id, min(d) as startdate, max(d) as enddate
from (select td.*,
dateadd(day, - row_number() over (partition by id order by d), d) as grp
from td
) td
group by id, grp
order by id, grp;
An alternative method relies on cumulative sums and similar functionality that is much easier to expression in SQL Server 2012+.
Somewhat simpler approach I think.
Basically create a list of dates for all work block ranges (A). Then create a list of dates for the whole year for each ID (B). Then remove the A from B. Compile the remaining list of dates into date ranges for each ID.
DECLARE #startdate DATETIME, #enddate DATETIME
SET #startdate = '2015-01-01'
SET #enddate = '2015-12-31'
--Build date ranges from remaining date list
;WITH dateRange(ID, dates, Grouping)
AS
(
SELECT dt1.id, dt1.Dates, dt1.Dates + row_number() over (order by dt1.id asc, dt1.Dates desc) AS Grouping
FROM
(
--Remove (A) from (B)
SELECT distinct dt.ID, tmp.Dates FROM DatesTable dt
CROSS APPLY
(
--GET (B) here
SELECT DATEADD(DAY, number, #startdate) [Dates]
FROM master..spt_values
WHERE type = 'P' AND DATEADD(DAY, number, #startdate) <= #enddate
) tmp
left join
(
--GET (A) here
SELECT DISTINCT T.Id,
D.Dates
FROM DatesTable AS T
INNER JOIN master..spt_values as N on N.number between 0 and datediff(day, T.DateStart, T.DateEnd)
CROSS APPLY (select dateadd(day, N.number, T.DateStart)) as D(Dates)
WHERE N.type ='P'
) dr
ON dr.Id = dt.Id and dr.Dates = tmp.Dates
WHERE dr.id is null
) dt1
)
SELECT ID, CAST(MIN(Dates) AS DATE) DateStart, CAST(MAX(Dates) AS DATE) DateEnd
FROM dateRange
GROUP BY ID, Grouping
ORDER BY ID
Heres the code:
http://sqlfiddle.com/#!3/f3615/1
I hope this helps!
I have a result set in generated as CTE using Union that contains duplicate records. as in image below:
And the query is:
WITH CTE (StartTime ,EndTime )
AS
(
SELECT StartTime ,EndTime, Null as Exclude, SupplierId FROM cms.TimeSlotMaster
WHERE Monday = 1 AND SupplierID IS NULL
UNION
SELECT StartTime ,EndTime FROM cms.TimeSlotOverRider
WHERE SupplierID IS NULL
AND StartDate <= cast(GETDATE() as DATE) AND EndDate >= cast(GETDATE() as DATE)
)
Now I am trying to remove the duplicate results from this result set at all. So finally the results set should be only 2 rows. So it should look like below:
Any help would be appreciated. Thanks.
For more information the first result set is generated using below CTE
You can use NOT EXISTS:
SELECT t.*
FROM dbo.TableName t
WHERE NOT EXISTS
(
SELECT 1 FROM dbo.TableName t2
WHERE t. ID <> t2.ID
AND t.StartTime = t2.StartTime
AND t.EndTime = t2.EndTime
)
or - if you don't have a primary key in this table:
WITH CTE AS
(
SELECT t.*, cnt = COUNT(*) OVER (PARTITION BY StartTime, EndTime)
FROM dbo.TableName t
)
SELECT StartTime, EndTime
FROM CTE
WHERE cnt = 1
I'm trying to export the latest records from SQL Server 2005 database once weekly. This is my table:
agent_name date ID
ALEX 2015-05-25 13
ALEX 2015-05-22 13
ALICE 2015-05-24 10
ALICE 2015-05-26 10
How to create output table should like this:
agent_name date ID
ALEX 2015-05-25 13
ALICE 2015-05-26 10
My SQL script:
SELECT a.agent_name,
a.date,
a.ID
FROM Payment a
INNER JOIN agentmaster b ON a.ID = b.ID2
WHERE b.agent ='Y'
AND a.date >= DATEADD(day, -7, GETDATE())
If you want latest rows for each ID and agent_name use this:
SELECT a.agent_name,
MAX(a.[date]) [date],
a.ID
FROM Payment a
GROUP BY
a.agent_name,
a.ID
If you want to have latest rows in each week use this:
SELECT a.agent_name,
a.date,
a.ID
FROM (SELECT *, ROW_NUMBER() OVER (PARTITION BY DATEPART(WEEK, [date]) ORDER BY [date] DESC) seq
FROM Payment ) a
WHERE
seq = 1
If you want latest rows for each ID and agent_name at each week use this:
;WITH p AS (
/* add your query here */
)
SELECT a.agent_name,
a.date,
a.ID
FROM (SELECT *, ROW_NUMBER() OVER (PARTITION BY agent_name, ID, DATEPART(WEEK, [date]) ORDER BY [date] DESC) seq
FROM p ) a
WHERE
seq = 1
Try This
SELECT ID, Agent_Name, MAX(Date) Max_Date
FROM AgentMaster
GROUP BY Agent_Name, ID
If you want to add the date filter then add the where condition
date >= DATEADD(day,-7, GETDATE())
This will return records from the first day of the current week to the last day of the current week
SELECT a.agent_name
,a.date
,a.ID
FROM Payment a INNER JOIN agentmaster b
ON a.ID = b.ID2
WHERE b.agent ='Y' AND
(a.date BETWEEN
DATEADD(wk, DATEDIFF(wk, 0, GetDate()),0) AND
DATEADD(wk, DATEDIFF(wk, 0, GETDATE()),6))
In this solution, I make a table in a CTE and defines the first day of week within a period. And then make a JOIN between the first day of weeks and our actual query
Sample data
DECLARE #tbl TABLE(agent_name VARCHAR(10) ,[date] date, Id INT)
INSERT #tbl
SELECT 'ALEX' , '2015-05-25' , 13 UNION
SELECT 'ALEX' , '2015-05-22' , 13 UNION
SELECT 'ALICE' , '2015-05-24' , 10 UNION
SELECT 'ALICE' , '2015-05-26' , 10
SELECT * FROM #tbl
Query
-- This period can be changed
DECLARE #StartDate date = '01/01/2015'
DECLARE #EndDate date = '01/01/2016'
;WITH Numbers (Number)
AS
(
SELECT number
FROM master..spt_values
WHERE [type] = 'P'
)
,firstDayOfWeek (fWDate)
AS
(
SELECT DATEADD(DAY, n.Number, #StartDate)
FROM Numbers AS n
WHERE n.Number <= DATEDIFF(DAY, #StartDate, #EndDate)
AND DATEPART(WEEKDAY, DATEADD(DAY, n.Number, #StartDate)) = 1 --Defines first day of week
)
SELECT t.agent_name, Max(t.date) AS [date], t.Id
FROM #tbl AS t
INNER JOIN firstDayOfWeek AS fw ON t.date >= fw.fWDate
AND t.[date] < DATEADD(DAY, 7, fw.fWDate)
GROUP BY t.Id, t.agent_name