I have a requirement where I will have to split overlapping records on a given table with 2 date fields.
Consider this to be my input table TableT.
ID
EFFECTIVE_DATE
END_DATE
JKL
2016-01-01
2016-12-31
JKL
2016-04-01
2016-12-31
JKL
2016-01-01
2016-03-04
JKL
2016-04-01
2016-12-31
JKL
2016-01-01
2016-12-31
I would want my output to look like below. I need to achieve this in both SQL Server and Oracle\DB2 so I am looking for a generic solution.
ID
EFFECTIVE_DATE
END_DATE
JKL
2016-01-01
2016-03-04
JKL
2016-03-05
2016-03-31
JKL
2016-04-01
2016-12-31
This is what I have tried
With EndDates as (
select END_DATE as END_DATE,TRIM(ID) as ID FROM TableT
union all
select ADD_DAYS(EFFECTIVE_DATE, -1) as END_DATE,TRIM(ID) as ID FROM TableT
), Periods as (
select ID as ID,MIN(EFFECTIVE_DATE) as EFFECTIVE_DATE,
(select MIN(END_DATE) from EndDates e
where e.ID = t.ID and
e.END_DATE >= MIN(EFFECTIVE_DATE)) as END_DATE
from
TableT t
group by ID),
EXTN_PERIOD as (select p.ID as ID, ADD_DAYS(p.END_DATE, 1) as EFFECTIVE_DATE,e.END_DATE as END_DATE
from
Periods p
inner join
EndDates e
on
p.ID = e.ID and
p.END_DATE < e.END_DATE
where
not exists (select * from EndDates e2 where
e2.ID = p.ID and
e2.END_DATE > p.END_DATE and
e2.END_DATE < e.END_DATE)
)
select * from EXTN_PERIOD
union
select * from PERIODS
It works partially fine but does not give me the desired output.
This is what the output I get when I run the above query:
ID
EFFECTIVE_DATE
END_DATE
JKL
2016-01-01
2016-03-04
JKL
2016-03-05
2016-03-31
Thanks in advance!
WITH
/*
MY_TAB (ID, EFFECTIVE_DATE, END_DATE) AS
(
VALUES
('JKL', DATE('2016-01-01'), DATE('2016-12-31'))
, ('JKL', DATE('2016-04-01'), DATE('2016-12-31'))
, ('JKL', DATE('2016-01-01'), DATE('2016-03-04'))
, ('JKL', DATE('2016-04-01'), DATE('2016-12-31'))
, ('JKL', DATE('2016-01-01'), DATE('2016-12-31'))
)
,
*/
A AS
(
SELECT DISTINCT T.ID, DECODE(V.I, 1, T.EFFECTIVE_DATE, 2, T.END_DATE + 1) DT
FROM MY_TAB T, (VALUES 1, 2) V(I)
)
, INTL AS
(
SELECT
ID
, LAG(DT) OVER (PARTITION BY ID ORDER BY DT) AS EFF_DT
, DT AS END_DT
FROM A
)
SELECT ID, EFF_DT, END_DT - 1 AS END_DT
FROM INTL
WHERE EFF_DT IS NOT NULL
ORDER BY 1, 2;
Almost universal. The only customization is the way the "virtual" table with the correlation name V of 2 rows (with INTEGERS 1 and 2) is generated.
The idea is to convert your data first to [inclusive, exclusive) form to simplify further calculations. Then we merge all effective and end dates and construct intervals using the OLAP LAG function. Finally we revert to your [inclusive, inclusive] form.
db<>fiddle link to test.
In Oracle you could do something like this:
with
tablet (id, effective_date, end_date) as (
select 'JKL', date '2016-01-01', date '2016-12-31' from dual union all
select 'JKL', date '2016-04-01', date '2016-12-31' from dual union all
select 'JKL', date '2016-01-01', date '2016-03-04' from dual union all
select 'JKL', date '2016-04-01', date '2016-12-31' from dual union all
select 'JKL', date '2016-01-01', date '2016-12-31' from dual
)
, prep (id, dt) as (
select distinct id, case col when 'EFF' then val else val + 1 end
from tablet
unpivot (val for col in (effective_date as 'EFF', end_date as 'END'))
)
, almost_done (id, effective_date, end_date) as (
select id, dt, lead(dt) over (partition by id order by dt) - 1
from prep
)
select id, effective_date, end_date
from almost_done
where end_date is not null
;
ID EFFECTIVE_DATE END_DATE
--- -------------- ----------
JKL 2016-01-01 2016-03-04
JKL 2016-03-05 2016-03-31
JKL 2016-04-01 2016-12-31
Notice the first CTE (tablet, used to generate testing data - you don't need it in your real-life case). Then, the first step is to unpivot the data; I don't know how SQL Server supports unpivoting, worst case you can do it manually with a cross join. (NOT with UNION ALL - that is inefficient.) Then you remove duplicates, and the rest is easy with the LEAD analytic function, which SQL Server should support too.
Related
I have below data
Record_date ID
28-feb-2022 xyz
31-Jan-2022 ABC
30-nov-2022 jkl
31-oct-2022 dcs
I want to get last 3 months data from given date column. We don't have to consider the missing month.
Output should be:
Record_date ID
28-feb-2022 xyz
31-Jan-2022 ABC
30-nov-2022 jkl
In the last 3 months Dec is missing but we have to ignore it as the data is not available. Tried many things but not working.
Any suggestions?
Assuming you are using Oracle then you can use Oralce ADD_MONTHS function and filter the data.
--- untested
-- Assumption Record_date is a date column
SELECT * FROM table1
where Record_date > ADD_MONTHS(SYSDATE, -3)
To get the data for the three months that are latest in the table, you can use:
SELECT record_date,
id
FROM (
SELECT t.*,
DENSE_RANK() OVER (ORDER BY TRUNC(Record_date, 'MM') DESC) AS rnk
FROM table_name t
)
WHERE rnk <= 3;
Which, for the sample data:
CREATE TABLE table_name (Record_date, ID) AS
SELECT DATE '2022-02-28', 'xyz' FROM DUAL UNION ALL
SELECT DATE '2022-01-31', 'ABC' FROM DUAL UNION ALL
SELECT DATE '2022-11-30', 'jkl' FROM DUAL UNION ALL
SELECT DATE '2022-10-31', 'dcs' FROM DUAL;
Outputs:
RECORD_DATE
ID
2022-11-30 00:00:00
jkl
2022-10-31 00:00:00
dcs
2022-02-28 00:00:00
xyz
db<>fiddle here
I have a problem with fetching few exceptions from DB.
Example, table b:
sn
v_num
start_date
end_date
1
001
01-01-2019
31-12-2099
1
002
01-01-2021
31-01-2022
1
003
01-02-2022
31-12-2099
2
001
01-01-2022
31-12-2099
2
002
01-07-2022
31-07-2022
2
003
01-08-2022
31-12-2099
Expected output:
sn
v_num
start_date
end_date
1
003
01-02-2022
31-12-2099
2
001
01-01-2022
31-12-2099
Currently I'm here:
SELECT * FROM table a, table b
WHERE a.sn = b.sn
AND b.v_num = (SELECT max (v_num) FROM b WHERE a.sn = b.sn)
but obviously that is not good because of a few cases like this with sn = 2.
Conclusion, I need to get unique sn record where v_num is max (95% of them in DB) except in case if start_date of max v_num record is > today.
Filter using start_date <= TRUNC(SYSDATE) then use the ROW_NUMBER analytic function:
SELECT *
FROM (
SELECT a.*,
ROW_NUMBER() OVER (PARTITION BY sn ORDER BY v_num DESC) AS rn
FROM "TABLE" a
WHERE start_date <= TRUNC(SYSDATE)
)
WHERE rn = 1;
If the start_date has a time component then you can use start_date < TRUNC(SYSDATE) + INTERVAL '1' DAY to get all the values for today from 00:00:00 to 23:59:59.
If you can have ties for the maximum and want to return all the ties then you can use the RANK analytic function instead of ROW_NUMBER.
Which, for the sample data:
CREATE TABLE "TABLE" (sn, v_num, start_date, end_date) AS
SELECT 1, '001', DATE '2022-01-01', DATE '2099-12-31' FROM DUAL UNION ALL
SELECT 1, '002', DATE '2022-01-01', DATE '2022-01-31' FROM DUAL UNION ALL
SELECT 1, '003', DATE '2022-02-01', DATE '2099-12-31' FROM DUAL UNION ALL
SELECT 2, '001', DATE '2022-01-01', DATE '2099-12-31' FROM DUAL UNION ALL
SELECT 2, '002', DATE '2022-07-01', DATE '2022-07-31' FROM DUAL UNION ALL
SELECT 2, '003', DATE '2022-08-01', DATE '2099-12-31' FROM DUAL;
Outputs:
SN
V_NUM
START_DATE
END_DATE
RN
1
003
2022-02-01 00:00:00
2099-12-31 00:00:00
1
2
001
2022-01-01 00:00:00
2099-12-31 00:00:00
1
db<>fiddle here
I have the below table. I need to count how many ids were active in a given month. So thinking I'll need to create a row for each id that was active during that month so that id can be counted each month. A row should be generated for a term_dt during that month.
active_dt term_dt id
1/1/2018 101
1/1/2018 5/15/2018 102
3/1/2018 6/1/2018 103
1/1/2018 4/25/18 104
Apparently this is a "count number of overlapping intervals" problem. The algorithm goes like this:
Create a sorted list of all start and end points
Calculate a running sum over this list, add one when you encounter a start and subtract one when you encounter an end
If two points are same then perform subtractions first
You will end up with list of all points where the sum changed
Here is a rough outline of the query. It is for SQL Server but could be ported to any RDBMS that supports window functions:
WITH cte1(date, val) AS (
SELECT active_dt, 1 FROM #t AS t
UNION ALL
SELECT COALESCE(term_dt, '2099-01-01'), -1 FROM #t AS t
-- if end date is null then assume the row is valid indefinitely
), cte2 AS (
SELECT date, SUM(val) OVER(ORDER BY date, val) AS rs
FROM cte1
)
SELECT YEAR(date) AS YY, MONTH(date) AS MM, MAX(rs) AS MaxActiveThisYearMonth
FROM cte2
GROUP BY YEAR(date), MONTH(date)
DB Fiddle
I was toying with a simpler query, that seemed to do the trick, for Oracle:
with candidates (month_start) as (
select to_date ('2018-' || column_value || '-01','YYYY-MM-DD')
from
table
(sys.odcivarchar2list('01','02','03','04','05',
'06','07','08','09','10','11','12'))
), sample_data (active_dt, term_dt, id) as (
select to_date('01/01/2018', 'MM/DD/YYYY'), null, 101 from dual
union select to_date('01/01/2018', 'MM/DD/YYYY'),
to_date('05/15/2018', 'MM/DD/YYYY'), 102 from dual
union select to_date('03/01/2018', 'MM/DD/YYYY'),
to_date('06/01/2018', 'MM/DD/YYYY'), 103 from dual
union select to_date('01/01/2018', 'MM/DD/YYYY'),
to_date('04/25/2018', 'MM/DD/YYYY'), 104 from dual
)
select c.month_start, count(1)
from candidates c
join sample_data d
on c.month_start between d.active_dt and nvl(d.term_dt,current_date)
group by c.month_start
order by c.month_start
An alternative solution would be to use a hierarchical query, e.g.:
WITH your_table AS (SELECT to_date('01/01/2018', 'dd/mm/yyyy') active_dt, NULL term_dt, 101 ID FROM dual UNION ALL
SELECT to_date('01/01/2018', 'dd/mm/yyyy') active_dt, to_date('15/05/2018', 'dd/mm/yyyy') term_dt, 102 ID FROM dual UNION ALL
SELECT to_date('01/03/2018', 'dd/mm/yyyy') active_dt, to_date('01/06/2018', 'dd/mm/yyyy') term_dt, 103 ID FROM dual UNION ALL
SELECT to_date('01/01/2018', 'dd/mm/yyyy') active_dt, to_date('25/04/2018', 'dd/mm/yyyy') term_dt, 104 ID FROM dual)
SELECT active_month,
COUNT(*) num_active_ids
FROM (SELECT add_months(TRUNC(active_dt, 'mm'), -1 + LEVEL) active_month,
ID
FROM your_table
CONNECT BY PRIOR ID = ID
AND PRIOR sys_guid() IS NOT NULL
AND LEVEL <= FLOOR(months_between(coalesce(term_dt, SYSDATE), active_dt)) + 1)
GROUP BY active_month
ORDER BY active_month;
ACTIVE_MONTH NUM_ACTIVE_IDS
------------ --------------
01/01/2018 3
01/02/2018 3
01/03/2018 4
01/04/2018 4
01/05/2018 3
01/06/2018 2
01/07/2018 1
01/08/2018 1
01/09/2018 1
01/10/2018 1
Whether this is more or less performant than the other answers is up to you to test.
I have a SQL table that contains employeeid, StartDateTime and EndDatetime as follows:
CREATE TABLE Sample
(
SNO INT,
EmployeeID NVARCHAR(10),
StartDateTime DATE,
EndDateTime DATE
)
INSERT INTO Sample
VALUES
( 1, 'xyz', '2018-01-01', '2018-01-02' ),
( 2, 'xyz', '2018-01-03', '2018-01-05' ),
( 3, 'xyz', '2018-01-06', '2018-02-01' ),
( 4, 'xyz', '2018-02-15', '2018-03-15' ),
( 5, 'xyz', '2018-03-16', '2018-03-19' ),
( 6, 'abc', '2018-01-16', '2018-02-25' ),
( 7, 'abc', '2018-03-08', '2018-03-19' ),
( 8, 'abc', '2018-02-26', '2018-03-01' )
I want the result to be displayed as
EmployeeID | StartDateTime | EndDateTime
------------+-----------------+---------------
xyz | 2018-01-01 | 2018-02-01
xyz | 2018-02-15 | 2018-03-19
abc | 2018-01-16 | 2018-03-01
abc | 2018-03-08 | 2018-03-19
Basically, I want to recursively look at records of each employee and datemine the continuity of Start and EndDates and make a set of continuous date records.
I wrote my query as follows:
SELECT *
FROM dbo.TestTable T1
LEFT JOIN dbo.TestTable t2 ON t2.EmpId = T1.EmpId
WHERE t1.EndDate = DATEADD(DAY, -1, T2.startdate)
to see if I could decipher something from the output looking for a pattern. Later realized that with the above approach, I need to join the same table multiple times to get the output I desire.
Also, there is a case that there can be multiple employee records, so I need direction on efficient way of getting this desired output.
Any help is greatly appreciated.
This will do it for you. Use a recursive CTE to get all the adjacent rows, then get the highest end date for each start date, then the first start date for each end date.
;with cte as (
select EmployeeID, StartDateTime, EndDateTime
from sample s
union all
select CTE.EmployeeID, CTE.StartDateTime, s.EndDateTime
from sample s
join cte on cte.EmployeeID=s.EmployeeID and s.StartDateTime=dateadd(d,1,CTE.EndDateTime)
)
select EmployeeID, Min(StartDateTime) as StartDateTime, EndDateTime from (
select EmployeeID, StartDateTime, Max(EndDateTime) as EndDateTime from cte
group by EmployeeID, StartDateTime
) q group by EmployeeID, EndDateTime
You can use this.
WITH T AS (
SELECT S1.SNO,
S1.EmployeeID,
S1.StartDateTime,
ISNULL(S2.EndDateTime, S1.EndDateTime) EndDateTime,
ROW_NUMBER() OVER(PARTITION BY S1.EmployeeId ORDER BY S1.StartDateTime)
- ROW_NUMBER() OVER(PARTITION BY S1.EmployeeId, CASE WHEN S2.StartDateTime IS NULL THEN 0 ELSE 1 END ORDER BY S1.StartDateTime ) RN,
ROW_NUMBER() OVER(PARTITION BY S1.EmployeeId, ISNULL(S2.EndDateTime, S1.EndDateTime) ORDER BY S1.EmployeeId, S1.StartDateTime) RN_END
FROM Sample S1
LEFT JOIN Sample S2 ON DATEADD(DAY,1,S1.EndDateTime) = S2.StartDateTime
)
SELECT EmployeeID, MIN(StartDateTime) StartDateTime,MAX(EndDateTime) EndDateTime FROM T
WHERE RN_END = 1
GROUP BY EmployeeID, RN
ORDER BY EmployeeID DESC, StartDateTime
Result:
EmployeeID StartDateTime EndDateTime
---------- ------------- -----------
xyz 2018-01-01 2018-02-01
xyz 2018-02-15 2018-03-19
abc 2018-01-16 2018-03-01
abc 2018-03-08 2018-03-19
I have following table with ID and DATE
ID DATE
123 7/1/2015
123 6/1/2015
123 5/1/2015
123 4/1/2015
123 9/1/2014
123 8/1/2014
123 7/1/2014
123 6/1/2014
456 11/1/2014
456 10/1/2014
456 9/1/2014
456 8/1/2014
456 5/1/2014
456 4/1/2014
456 3/1/2014
789 9/1/2014
789 8/1/2014
789 7/1/2014
789 6/1/2014
789 5/1/2014
789 4/1/2014
789 3/1/2014
In this table, I have three customer ids, 123, 456, 789 and date column which shows which month they worked.
I want to find out which of the customers have gap in their work.
Our customers work record is kept per month...so, dates are monthly..
and each customer have different start and end dates.
Expected results:
ID First_Absent_date
123 10/01/2014
456 06/01/2014
To get a simple list of the IDs with gaps, with no further details, you need to look at each ID separately, and as #mikey suggested you can count the number of months and look at the first and last date to see if how many months that spans.
If your table has a column called month (since date isn't allowed unless it's a quoted identifier) you could start with:
select id, count(month), min(month), max(month),
months_between(max(month), min(month)) + 1 as diff
from your_table
group by id
order by id;
ID COUNT(MONTH) MIN(MONTH) MAX(MONTH) DIFF
---------- ------------ ---------- ---------- ----------
123 8 01-JUN-14 01-JUL-15 14
456 7 01-MAR-14 01-NOV-14 9
789 7 01-MAR-14 01-SEP-14 7
Then compare the count with the month span, in a having clause:
select id
from your_table
group by id
having count(month) != months_between(max(month), min(month)) + 1
order by id;
ID
----------
123
456
If you can actually have multiple records in a month for an ID, and/or the date recorded might not be the start of the month, you can do a bit more work to normalise the dates:
select id,
count(distinct trunc(month, 'MM')),
min(trunc(month, 'MM')),
max(trunc(month, 'MM')),
months_between(max(trunc(month, 'MM')), min(trunc(month, 'MM'))) + 1 as diff
from your_table
group by id
order by id;
select id
from your_table
group by id
having count(distinct trunc(month, 'MM')) !=
months_between(max(trunc(month, 'MM')), min(trunc(month, 'MM'))) + 1
order by id;
Oracle Setup:
CREATE TABLE your_table ( ID, "DATE" ) AS
SELECT 123, DATE '2015-07-01' FROM DUAL UNION ALL
SELECT 123, DATE '2015-06-01' FROM DUAL UNION ALL
SELECT 123, DATE '2015-05-01' FROM DUAL UNION ALL
SELECT 123, DATE '2015-04-01' FROM DUAL UNION ALL
SELECT 123, DATE '2014-09-01' FROM DUAL UNION ALL
SELECT 123, DATE '2014-08-01' FROM DUAL UNION ALL
SELECT 123, DATE '2014-07-01' FROM DUAL UNION ALL
SELECT 123, DATE '2014-06-01' FROM DUAL UNION ALL
SELECT 456, DATE '2014-11-01' FROM DUAL UNION ALL
SELECT 456, DATE '2014-10-01' FROM DUAL UNION ALL
SELECT 456, DATE '2014-09-01' FROM DUAL UNION ALL
SELECT 456, DATE '2014-08-01' FROM DUAL UNION ALL
SELECT 456, DATE '2014-05-01' FROM DUAL UNION ALL
SELECT 456, DATE '2014-04-01' FROM DUAL UNION ALL
SELECT 456, DATE '2014-03-01' FROM DUAL UNION ALL
SELECT 789, DATE '2014-09-01' FROM DUAL UNION ALL
SELECT 789, DATE '2014-08-01' FROM DUAL UNION ALL
SELECT 789, DATE '2014-07-01' FROM DUAL UNION ALL
SELECT 789, DATE '2014-06-01' FROM DUAL UNION ALL
SELECT 789, DATE '2014-05-01' FROM DUAL UNION ALL
SELECT 789, DATE '2014-04-01' FROM DUAL UNION ALL
SELECT 789, DATE '2014-03-01' FROM DUAL;
Query:
SELECT ID,
MIN( missing_date )
FROM (
SELECT ID,
CASE WHEN LEAD( "DATE" ) OVER ( PARTITION BY ID ORDER BY "DATE" )
= ADD_MONTHS( "DATE", 1 ) THEN NULL
WHEN LEAD( "DATE" ) OVER ( PARTITION BY ID ORDER BY "DATE" )
IS NULL THEN NULL
ELSE ADD_MONTHS( "DATE", 1 )
END AS missing_date
FROM your_table
)
GROUP BY ID
HAVING COUNT( missing_date ) > 0;
Output:
ID MIN(MISSING_DATE)
---------- -------------------
123 2014-10-01 00:00:00
456 2014-06-01 00:00:00
You could use a Lag() function to see if records have been skipped for a particular date or not.Lag() basically helps in comparing the data in current row with previous row. So if we order by DATE, we could easily compare and find any gaps.
select * from
(
select ID,DATE_, case when DATE_DIFF>1 then 1 else 0 end comparison from
(
select ID, DATE_ ,DATE_-LAG(DATE_, 1) OVER (PARTITION BY ID ORDER BY DATE_) date_diff from trial
)
)
where comparison=1 order by ID,DATE_;
This groups all the entries by id, and then arranges the records by date. If a customer is always present, there would not be a gap in his date. So anyone who has a date difference greater than 1 had a gap. You could tweak this as per your requirement.
EDIT : Just observed that you are storing data in mm/dd/yyyy format, when I closely observed above answers.You are storing only first date of every month. So, the above query can be tweaked as :
select * from
(
select ID,DATE_,PREV_DATE,last_day(PREV_DATE)+1 ABSENT_DATE, case when DATE_DIFF>31 then 1 else 0 end comparison from
(
select ID, DATE_ ,LAG(DATE_,1) OVER (PARTITION BY ID ORDER BY DATE_) PREV_DATE,DATE_-LAG(DATE_, 1) OVER (PARTITION BY ID ORDER BY DATE_) date_diff from trial
)
)
where comparison=1 order by ID,DATE_;