Query to remove continuous eligibility records - sql

Question

I need to get rid of the records whose eligibility already exists in another record of the memid. In the below example I need the output with only rows where I have mentioned Y. Table has memid,effdate,termdate. I have just prefixed Y to mention the record I need as output. How can we do this.Thanks.
MEMID EFFDATE TERMDATE
Y A1 2012-01-01 2078-12-31
A1 2012-02-01 2078-12-31
Y B1 2007-05-01 2008-12-31
Y B1 2009-10-01 2010-04-30
Y A2 1999-01-01 2078-12-31
A2 2006-01-01 2011-04-28
B2 1999-01-01 1999-10-01
Y B2 1999-01-01 2000-09-30
Y B2 2006-01-01 2006-01-01
Y B2 2009-08-01 2078-12-31
Y A3 2000-03-01 2009-01-31
A3 2002-04-01 2009-01-31
A3 2003-01-01 2006-06-30
A3 2006-01-01 2009-01-31
Y A3 2009-10-01 2010-07-31
Y A3 2011-06-01 2012-09-30
A3 2011-09-01 2012-09-30
Y A3 2013-06-01 2078-12-31
A3 2013-07-01 2078-12-31
B3 1999-01-01 2008-11-30
Y B3 1999-01-01 2078-12-31
B3 2006-01-01 2008-11-30

Answer 1

select all ranges where there is no covering bigger range with NOT EXISTS. Then remove duplicates with DISTINCT.
select distinct memid, effdate, termdate
from mytable
where not exists
(
select *
from mytable bigger
where bigger.memid = mytable.memid
and
(
(bigger.effdate <= mytable.effdate and bigger.termdate > mytable.termdate)
or
(bigger.effdate < mytable.effdate and bigger.termdate >= mytable.termdate)
)
);

Answer 2

This is a variation of the Gaps and Islands problem. Since you have tagged both MySQL and SQL-Server, and have not responded to my question asking you to clarify which, I will give a solution for both.
Your first step is expand all your ranges into continuous data by joining with a numbers table. This will turn a single row into one row for every date in the range:
SQL Server
SELECT t.memid, Date = DATEADD(DAY, n.Number, t.EffDate)
FROM YourTable t
INNER JOIN Numbers n
ON n.Number BETWEEN 0 AND DATEDIFF(DAY, t.EffDate, t.TermDate);
MySQL
SELECT t.memid, DATE_ADD(t.EffDate, INTERVAL n.Number DAY) AS Date
FROM YourTable t
INNER JOIN Numbers n
ON n.Number BETWEEN 0 AND DATEDIFF(t.EffDate, t.TermDate);
This will turn this row:
memid EFFDATE TERMDATE
A3 2009-10-01 2009-10-05
Into
memid Date
A3 2009-10-01
A3 2009-10-02
A3 2009-10-03
A3 2009-10-04
A3 2009-10-05
If you don't have a numbers table, then you should probably create one. (In each of the SQL Fiddle's below I have created a numbers table, so you can find ways of doing so there.
Now you have your continuous ranges you can apply the appropriate gaps and islands solution.
If you are using SQL Server then you can use ranking functions to resolve this:
WITH ContinuousRange AS
( SELECT t.memid,
d.Date,
GroupingSet = DATEADD(DAY, -DENSE_RANK() OVER(PARTITION BY memid
ORDER BY d.Date), d.Date)
FROM T
INNER JOIN Numbers n
ON n.Number BETWEEN 0 AND DATEDIFF(DAY, t.EffDate, t.TermDate)
OUTER APPLY (SELECT Date = DATEADD(DAY, n.Number, t.EffDate)) d
)
SELECT cr.MemID,
EffDate = MIN(cr.Date),
TermDate = MAX(cr.Date)
FROM ContinuousRange cr
GROUP BY cr.MemID, cr.GroupingSet
ORDER BY cr.MemID, cr.GroupingSet;
Simplified Example on SQL Fiddle
This works on the basis that end number in a sequence minus it's order in the sequence will give a constant for a continuous range, e.g.:
Sequence | OrderInSequence | (Sequence - OrderInSequence)
---------+-----------------+------------------------------
1 | 1 | 0
2 | 2 | 0
3 | 3 | 0
5 | 4 | 1
6 | 5 | 1
As you can see, where there is a gap in the sequence (between 3 and 5) the value in the 3rd column changes, this is how the column GroupingSet is calculated:
GroupingSet = DATEADD(DAY, -DENSE_RANK() OVER(PARTITION BY memid ORDER BY d.Date), d.Date)
Then when you can use thisc column to get the minimum and maximum value of each consectutive sequence (or island).
Since MySQL does not have ranking functions you will need to use user defined variables to emulate them:
SELECT MemID,
MIN(Date) AS EffDate,
MAX(Date) AS TermDate
FROM ( SELECT t.memid,
#i:= CASE WHEN t.MemID = #m
AND DATE_ADD(t.EffDate, INTERVAL n.Number DAY)
<= DATE_ADD(#d, INTERVAL 1 DAY)
THEN #i
ELSE #i + 1
END AS GroupingSet,
#m:= t.memid,
#d:= DATE_ADD(t.EffDate, INTERVAL n.Number DAY) AS Date
FROM t
INNER JOIN Numbers n
ON n.Number BETWEEN 0 AND DATEDIFF(t.TermDate, t.EffDate)
CROSS JOIN (SELECT #M:= '', #i:= 0, #d:= NULL) i
ORDER BY t.memid, DATE_ADD(t.EffDate, INTERVAL n.Number DAY)
) t
GROUP BY MemID, GroupingSet;
Simplified Example on SQL Fiddle
Getting the Grouping Set here is a more iterative process. The data is ordered by MemID and the date, then at each row the value of the date and the memid is stored in the variables #d and #m respectively. If the memid in the new row is the same as #m (i.e still in the same group of memid's) and the date in the new row is 1 day ahead of, or the same as #d then the grouping set does not increment, if it is a new memid, or the date is further than 1 day ahead of the previous date, then it is a new 'island' and the grouping set increments.
EDIT
To help with memory issues then you can deal with different scenarios differently. The first step is to remove any records contained entirely within another, e.g.
MemID EffDate TermDate
A1 2012-01-01 2078-12-31
A1 2012-02-01 2078-12-31
With these two, the second row is not required since its date range is contained entirely within the first. So this can be removed (This is done in the CTE called Filtered in the below query).
The second way to help is to remove the range expansion where it is not required, so with the above example we are just left with:
MemID EffDate TermDate
A1 2012-01-01 2078-12-31
And this is the only row for A1, it is therefore not necessary to expand this out into all the days between the EffDate and TermDate, and then take a minimum and maximum, we can simply use the EffDate and TermDate as they are. This is the query below the UNION ALL in the query below.
WITH Filtered AS
( SELECT MemID, EffDate, TermDate
FROM T
WHERE NOT EXISTS
( SELECT 1
FROM T T2
WHERE T.MemID = T2.MemID
AND T2.EffDate < T.EffDate
AND T2.TermDate >= T.TermDate
)
), ContinuousRange AS
( SELECT t.memid,
d.Date,
GroupingSet = DATEADD(DAY, -DENSE_RANK() OVER(PARTITION BY memid
ORDER BY d.Date), d.Date)
FROM Filtered T
INNER JOIN Numbers n
ON n.Number BETWEEN 0 AND DATEDIFF(DAY, t.EffDate, t.TermDate)
OUTER APPLY (SELECT Date = DATEADD(DAY, n.Number, t.EffDate)) d
WHERE EXISTS
( SELECT 1
FROM Filtered T2
WHERE T.MemID = T2.MemID
AND ( (T2.EffDate > T.EffDate AND T2.EffDate < T.TermDate)
OR (T2.TermDate > T.EffDate AND T2.TermDate < T.TermDate)
)
)
)
SELECT cr.MemID,
EffDate = MIN(cr.Date),
TermDate = MAX(cr.Date), 1
FROM ContinuousRange cr
GROUP BY cr.MemID, cr.GroupingSet
UNION ALL
SELECT T.MemID, T.EffDate, T.TermDate, 0
FROM Filtered T
WHERE NOT EXISTS
( SELECT 1
FROM Filtered T2
WHERE T.MemID = T2.MemID
AND ( (T2.EffDate > T.EffDate AND T2.EffDate < T.TermDate)
OR (T2.TermDate > T.EffDate AND T2.TermDate < T.TermDate)
)
)
ORDER BY MemID, EffDate;
Example on SQL Fiddle