I'm struggling with log aggregation with window functions. I have table with columns
Item (string)
User (string)
TimeStart (timestamp)
With a CTE, I'm trying to merge all entries within 5 min into one, to create a time interval. In C1 I'm looking for the start of next interval, C2 is grouping records into intervals. To this point it works fine.
When I selected C2 I got my data labeled with group ID. But when I'm trying to catch max time from my group as TimeEnd in C3 I got messed data that is useless. As I understand OVER PARTITION BY it should be a simple comparison of TimeStart values within partition and return of MAX, but instead of I get messed group IDs, and TimeEnds
Code below, SQL Server Express:
WITH C1 AS
(
SELECT
Item,
User,
TimeStart,
CASE
WHEN DATEDIFF(MINUTE, LAG(TimeStart) OVER (PARTITION BY Item, User ORDER BY TimeStart), TimeStart) < 5
THEN 0
ELSE 1
END AS isstart
FROM
Log
),
C2 AS
(
SELECT
*,
SUM(isstart) OVER (ORDER BY TimeStart ROWS UNBOUNDED PRECEDING) AS grp
FROM
C1
),
C3 AS
(
SELECT
*,
MAX(TimeStart) OVER (PARTITION BY grp) AS TimeEnd
FROM
C2
)
SELECT *
FROM C3
Could you explain to me what happened here and how to solve it?
PS: I could use GROUP BY clause but I'll lose non aggregated columns.
Thanks to #Alex, I got this solution:
WITH C1 AS
(
SELECT
Item,
User,
TimeStart,
CASE
WHEN DATEDIFF(MINUTE, LAG(TimeStart) OVER (PARTITION BY Item, User ORDER BY TimeStart), TimeStart) < 5
THEN 0
ELSE 1
END AS isstart
FROM
Log
),
C2 AS
(
SELECT
*,
SUM(isstart) OVER ( ORDER BY ItemPath,UserName,TimeStart
ROWS UNBOUNDED PRECEDING) as grp
FROM
C1
),
C3 AS(
SELECT *, MAX(TimeStart) OVER (PARTITION BY grp ) AS [TimeEnd] FROM C2
)
SELECT * FROM C3
where isstart=1
ORDER BY grp
The problem was in second CTE with the wrong ORDER of rows.
Also I send preview of output data
Output data preview
Related
I have a table with Name and Date columns. I want to get the earliest date when the current name appeared. For example:
Name
Date
X
30-Jan-2021
X
29-Jan-2021
X
28-Jan-2021
Y
27-Jan-2021
Y
26-Jan-2021
Y
25-Jan-2021
Y
24-Jan-2021
X
23-Jan-2021
X
22-Jan-2021
Now when I try to get the earliest date when current name (X) started to appear, I want 28-Jan, but the sql query would give 22-Jan-2021 because that's when X appeared originally for the first time.
Update: This was the query I was using:
Select min(Date) from myTable where Name='X'
I am using older SQL Server 2008 (in the process of upgrading), so do not have access to LEAD/LAG functions.
The solutions suggested below do work as intended. Thanks.
This is a type of gaps-and-islands problem.
There are many solutions. Here is one that is optimized for your case
Use LEAD/LAG to identify the first row in each grouping
Filter to only those rows
Number them rows and take the first one
WITH StartPoints AS (
SELECT *,
IsStart = CASE WHEN Name <> LEAD(Name, 1, '') OVER (ORDER BY Date DESC) THEN 1 END
FROM YourTable
),
Numbered AS (
SELECT *,
rn = ROW_NUMBER() OVER (PARTITION BY Name ORDER BY Date DESC)
FROM StartPoints
WHERE IsStart = 1 AND Name = 'X'
)
SELECT
Name, Date
FROM Numbered
WHERE rn = 1;
db<>fiddle
For SQL Server 2008 or earlier (which I strongly suggest you upgrade from), you can use a self-join with row-numbering to simulate LEAD/LAG
WITH RowNumbered AS (
SELECT *,
AllRn = ROW_NUMBER() OVER (ORDER BY Date ASC)
FROM YourTable
),
StartPoints AS (
SELECT r1.*,
IsStart = CASE WHEN r1.Name <> ISNULL(r2.Name, '') THEN 1 END
FROM RowNumbered r1
LEFT JOIN RowNumbered r2 ON r2.AllRn = r1.AllRn - 1
),
Numbered AS (
SELECT *,
rn = ROW_NUMBER() OVER (PARTITION BY Name ORDER BY Date DESC)
FROM StartPoints
WHERE IsStart = 1
)
SELECT
Name, Date
FROM Numbered
WHERE rn = 1;
This is a gaps and island problem. Based on the sample data, this will work:
WITH Groups AS(
SELECT YT.[Name],
YT.[Date],
ROW_NUMBER() OVER (ORDER BY YT.Date DESC) -
ROW_NUMBER() OVER (PARTITION BY YT.[Name] ORDER BY Date DESC) AS Grp
FROM dbo.YourTable YT),
FirstGroup AS(
SELECT TOP (1) WITH TIES
G.[Name],
G.[Date]
FROM Groups G
WHERE [Name] = 'X'
ORDER BY Grp ASC)
SELECT MIN(FG.[Date]) AS Mi
db<>fiddle
If i did understand, you want to know when the X disappeared and reappeared again. in that case you can search for gaps in dates by group.
this and example how to detect that
SELECT name
,DATE
FROM (
SELECT *
,DATEDIFF(day, lead(DATE) OVER (
PARTITION BY name ORDER BY DATE DESC
), DATE) DIF
FROM YourTable
) a
WHERE DIF > 1
I have a peculiar problem at hand. I need to rank in the following manner:
Each ID gets a new rank.
rank #1 is assigned to the ID with the lowest date. However, the subsequent dates for that particular ID can be higher but they will get the incremental rank w.r.t other IDs.
(E.g. ADF32 series will be considered to be ranked first as it had the lowest date, although it ends with dates 09-Nov, and RT659 starts with 13-Aug it will be ranked subsequently)
For a particular ID, if the days are consecutive then ranks are same, else they add by 1.
For a particular ID, ranks are given in date ASC.
How to formulate a query?
You need two steps:
select
id_col
,dt_col
,dense_rank()
over (order by min_dt, id_col, dt_col - rnk) as part_col
from
(
select
id_col
,dt_col
,min(dt_col)
over (partition by id_col) as min_dt
,rank()
over (partition by id_col
order by dt_col) as rnk
from tab
) as dt
dt_col - rnk caluclates the same result for consecutives dates -> same rank
Try datediff on lead/lag and then perform partitioned ranking
select t.ID_COL,t.dt_col,
rank() over(partition by t.ID_COL, t.date_diff order by t.dt_col desc) as rankk
from ( SELECT ID_COL,dt_col,
DATEDIFF(day, Lag(dt_col, 1) OVER(ORDER BY dt_col),dt_col) as date_diff FROM table1 ) t
One way to think about this problem is "when to add 1 to the rank". Well, that occurs when the previous value on a row with the same id_col differs by more than one day. Or when the row is the earliest day for an id.
This turns the problem into a cumulative sum:
select t.*,
sum(case when prev_dt_col = dt_col - 1 then 0 else 1
end) over
(order by min_dt_col, id_col, dt_col) as ranking
from (select t.*,
lag(dt_col) over (partition by id_col order by dt_col) as prev_dt_col,
min(dt_col) over (partition by id_col) as min_dt_col
from t
) t;
I need an sql code for the below. I want it to RANK however if DSLR >= 60 then I want the rank to start again like below.
Thanks
Assuming that you have a column that defines the ordering of the rows, say id, you can address this as a gaps-and-islands problem. Islands are group of adjacent record that start with a dslr above 60. We can identify them with a window sum, then rank within each island:
select dslr, rank() over(partition by grp order by id) as rn
from (
select t.*,
sum(case when dslr >= 60 then 1 else 0 end) over(order by id) as grp
from mytable t
) t
How would I construct a query to receive the MAX TurnTime per ID of the first 2 rounds? Rounds being defined as minimum Beginning_Date to mininmum End_Date of an ID. Without reusing either of the dates for the second round Turn Time calculation.
You can use row_number() . . . twice:
select d.*
from (select d.*,
row_number() over (partition by id order by turn_time desc) as seqnum_turntime
from (select d.*,
row_number() over (partition by id order by beginning_end desc) as seqnum_round
from data d
) d
where seqnum_round <= 2
) d
where seqnum_turntime = 1;
The innermost subquery gets the first two rounds. The outer subquery gets the maximum.
You could express this without window functions as well:
select top (1) with ties d.*
from data d
where d.beginning_date <= (select d2.beginning_date
from data d2
where d2.id = d.id
offset 1 fetch first 1 row only
)
order by row_number() over (partition by id order by turntime desc);
SELECT
ID
,turn_time
,beginning_date
,end_date
FROM
(
SELECT
ID
,MAX(turn_time) OVER (PARTITION BY Id ORDER BY BeginningDate ROWS BETWEEN 1 PRECEDING AND CURRENT ROW) AS turn_time --Maximum turn time of the current row and preceding row
,MIN(BeginningDate) OVER (PARTITION BY Id ORDER BY BeginningDate ROWS BETWEEN 1 PRECEDING AND CURRENT ROW) AS beginning_date --Minimum begin date over current row and preceding row (could also use LAG)
,end_date
,ROW_NUMBER() OVER (PARTITION BY Id ORDER BY BeginningDate) AS Turn_Number
FROM
<whatever your table is>
) turn_summary
WHERE
Turn_Number = 2
This is a simplified view of a table. I apologize, but I could not save a picture of the table so I hope this is ok.
c1___c2
1____a
1____b
2____a
2____b
2____c
2____d
3____e
3____a
4____z
5____d
The result is that due to the relationships of column C2,
Group 1 would include, 1,2,3,5 (because they have overlapping c2 values basically stating a=b=c=d=e)
Group 2 would include 4
I have millions of rows with this kind of data and currently there is a cursor job that runs x number of times to build these groups. I am able to visualize how this should work, but I have not been able to build a query that can pull out this relationship.
Any suggestions?
Thank you
Tested on SQL Server 2012:
WITH t AS (
SELECT
t.c1,
t.c2,
tm.c1_min
FROM
Test t
JOIN
(
SELECT
c2,
MIN(c1) AS c1_min
FROM
Test
GROUP BY
c2
) AS tm
ON
t.c2 = tm.c2
),
rt AS (
SELECT
c1_min,
c1,
1 AS cnt
FROM
t
UNION ALL
SELECT
rt.c1_min,
t.c1,
rt.cnt + 1 AS cnt
FROM
rt
JOIN
t
ON
rt.c1 = t.c1_min
AND
rt.c1 < t.c1
)
SELECT
SUM(t.rst) OVER (ORDER BY t.ord ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) AS group_number,
t.c1
FROM
(
SELECT
t.c1,
t.rst,
t.ord
FROM
(
SELECT
rt.c1,
CASE
WHEN rt.c1_min = MIN(rt.c1_min) OVER (ORDER BY rt.c1_min, rt.c1 ROWS BETWEEN 1 PRECEDING AND 1 PRECEDING) THEN 0
ELSE 1
END AS rst,
ROW_NUMBER() OVER (ORDER BY rt.c1_min, rt.c1) AS ord,
ROW_NUMBER() OVER (PARTITION BY rt.c1 ORDER BY rt.c1_min, rt.cnt) AS qfy
FROM
rt
) AS t
WHERE
t.qfy = 1
) AS t;