TSQL - Referencing a changed value from previous row - sql

I am trying to do a row calculation whereby the larger value will carry forward to the subsequent rows until a larger value is being compared. It is done by comparing the current value to the previous row using the lag() function.
Code
DECLARE #TAB TABLE (id varchar(1),d1 INT , d2 INT)
INSERT INTO #TAB (id,d1,d2)
VALUES ('A',0,5)
,('A',1,2)
,('A',2,4)
,('A',3,6)
,('B',0,4)
,('B',2,3)
,('B',3,2)
,('B',4,5)
SELECT id
,d1
,d2 = CASE WHEN id <> (LAG(id,1,0) OVER (ORDER BY id,d1)) THEN d2
WHEN d2 < (LAG(d2,1,0) OVER (ORDER BY id,d1)) THEN (LAG(d2,1,0) OVER (ORDER BY id,d1))
ELSE d2 END
Output (Added row od2 for clarity)
+----+----+----+ +----+
| id | d1 | d2 | | od2|
+----+----+----+ +----+
| A | 0 | 5 | | 5 |
| A | 1 | 5 | | 2 |
| A | 2 | 4 | | 4 |
| A | 3 | 6 | | 6 |
| B | 0 | 4 | | 4 |
| B | 2 | 4 | | 3 |
| B | 3 | 3 | | 2 |
| B | 4 | 5 | | 5 |
+----+----+----+ +----+
As you can see from the output it lag function is referencing the original value of the previous row rather than the new value. Is there anyway to achieve this?
Desired Output
+----+----+----+ +----+
| id | d1 | d2 | | od2|
+----+----+----+ +----+
| A | 0 | 5 | | 5 |
| A | 1 | 5 | | 2 |
| A | 2 | 5 | | 4 |
| A | 3 | 6 | | 6 |
| B | 0 | 4 | | 4 |
| B | 2 | 4 | | 3 |
| B | 3 | 4 | | 2 |
| B | 4 | 5 | | 5 |
+----+----+----+ +----+

Try this:
SELECT id
,d1
,d2
,MAX(d2) OVER (PARTITION BY ID ORDER BY d1)
FROM #TAB
The idea is to use the MAX to get the max value from the beginning to the current row for each partition.

Thanks for providing the DDL scripts and the DML.
One way of doing it would be using recursive cte as follows.
1. First rank all the records according to id, d1 and d2. -> cte block
2. Use recursive cte and get the first elements using rnk=1
3. the field "compared_val" will check against the values from the previous rnk to see if the value is > than the existing and if so it would swap
DECLARE #TAB TABLE (id varchar(1),d1 INT , d2 INT)
INSERT INTO #TAB (id,d1,d2)
VALUES ('A',0,5)
,('A',1,2)
,('A',2,4)
,('A',3,6)
,('B',0,4)
,('B',2,3)
,('B',3,2)
,('B',4,5)
;with cte
as (select row_number() over(partition by id order by d1,d2) as rnk
,id,d1,d2
from #TAB
)
,data(rnk,id,d1,d2,compared_val)
as (select rnk,id,d1,d2,d2 as compared_val
from cte
where rnk=1
union all
select a.rnk,a.id,a.d1,a.d2,case when b.compared_val > a.d2 then
b.compared_val
else a.d2
end
from cte a
join data b
on a.id=b.id
and a.rnk=b.rnk+1
)
select * from data order by id,d1,d2

Related

Get maximum of sequence

+----+-------+
| id | value |
+----+-------+
| 1 | A |
| 2 | B |
| 3 | C |
| 4 | D |
| 5 | D |
| 6 | D |
| 7 | N |
| 8 | P |
| 9 | P |
+----+-------+
Desired output
+----+-------+---------------------+
| id | value | calc ↓ |
+----+-------+---------------------+
| 1 | A | 1 |
| 2 | B | 2 |
| 3 | C | 3 |
| 4 | D | 6 |
| 5 | D | 6 |
| 6 | D | 6 |
| 7 | N | 7 |
| 8 | P | 9 |
| 9 | P | 9 |
| 10 | D | 11 |
| 11 | D | 11 |
| 12 | Z | 12 |
+----+-------+---------------------+
Can you help me for a solution for this ? Id is identity, id must be present in output, must have the same 9 rows in output.
New note: I added rows 10,11,12. Notice that id 10 and 11 which has letter 'D' is in a different group from id 4,5,6
thanks
If the grouping also depends on the surrounding ids then this turns into something like the gaps and islands problem https://www.red-gate.com/simple-talk/sql/t-sql-programming/the-sql-of-gaps-and-islands-in-sequences/#:~:text=The%20SQL%20of%20Gaps%20and%20Islands%20in%20Sequences,...%204%20Performance%20Comparison%20of%20Gaps%20Solutions.%20
You could use the Tabibitosan method https://rwijk.blogspot.com/2014/01/tabibitosan.html
Here you also need to group by your value column but that doesn't complicate it too much:
select id, value, max(id) over (partition by value, island) calc
from (
select id, value, id - row_number() over(partition by value order by id) island
from my_table
) as sq
order by id;
The id - row_number() over(partition by value order by id) expression gives you a number which changes each time the ID value changes by more than 1 for each value of value. This gets included in the max(id) over (partition by value, island) expression. The island number is only valid for that particular value. In your case, both values N and D have a computed island number of 6 but they need to be considered differently.
Db-fiddle https://www.db-fiddle.com/f/jahP7T6xBt3cpbLRhZZdQG/1
For this sample date you need MAX() window function:
SELECT id, value,
MAX(id) OVER (PARTITION BY value) calc
FROM tablename
SELECT id, value, (SELECT max(id) FROM TABLE inner where inner.value = outer.value)
FROM table as outer

Using LAG function with higher offset

Suppose we have the following input table
cat | value | position
------------------------
1 | A | 1
1 | B | 2
1 | C | 3
1 | D | 4
2 | C | 1
2 | B | 2
2 | A | 3
2 | D | 4
As you can see, the values A,B,C,D change position in each category, I want to track this change by adding a column change in front of each value, the output should look like this:
cat | value | position | change
---------------------------------
1 | A | 1 | NULL
1 | B | 2 | NULL
1 | C | 3 | NULL
1 | D | 4 | NULL
2 | C | 1 | 2
2 | B | 2 | 0
2 | A | 3 | -2
2 | D | 4 | 0
For example C was in position 3 in category 1 and moved to position 1 in category 2 and therefore has a change of 2. I tried inmplementing this using the LAG() function with an offset of 4 but I failed, how can I write this query.
Use lag() - with the proper partition by clause:
select
t.*,
lag(position) over(partition by value order by cat) - position change
from mytable t
You can use lag and then order by to maintain original order. Here is the demo.
select
*,
lag(position) over (partition by value order by cat) - position as change
from yourTable
order by
cat, position
output:
| cat | value | position | change |
| --- | ----- | -------- | ------ |
| 1 | A | 1 | null |
| 1 | B | 2 | null |
| 1 | C | 3 | null |
| 1 | D | 4 | null |
| 2 | C | 1 | 2 |
| 2 | B | 2 | 0 |
| 2 | A | 3 | -2 |
| 2 | D | 4 | 0 |
I think you just want lag() with the right partition by:
select t.*,
(lag(position) over (partition by value order by cat) - position) as change
from t;
Here is a db<>fiddle.

Spark dataframe add Missing Values

I have a dataframe of the following format. I want to add empty rows for missing time stamps for each customer.
+-------------+----------+------+----+----+
| Customer_ID | TimeSlot | A1 | A2 | An |
+-------------+----------+------+----+----+
| c1 | 1 | 10.0 | 2 | 3 |
| c1 | 2 | 11 | 2 | 4 |
| c1 | 4 | 12 | 3 | 5 |
| c2 | 2 | 13 | 2 | 7 |
| c2 | 3 | 11 | 2 | 2 |
+-------------+----------+------+----+----+
The resulting table should be of the format
+-------------+----------+------+------+------+
| Customer_ID | TimeSlot | A1 | A2 | An |
+-------------+----------+------+------+------+
| c1 | 1 | 10.0 | 2 | 3 |
| c1 | 2 | 11 | 2 | 4 |
| c1 | 3 | null | null | null |
| c1 | 4 | 12 | 3 | 5 |
| c2 | 1 | null | null | null |
| c2 | 2 | 13 | 2 | 7 |
| c2 | 3 | 11 | 2 | 2 |
| c2 | 4 | null | null | null |
+-------------+----------+------+------+------+
I have 1 Million customers and 360(in the above example only 4 is depicted) Time slots.
I figured out a way to create a Dataframe with 2 columns (Customer_id,Timeslot) with (1 M x 360 rows) and do a Left outer join with the original dataframe.
Is there a better way to do this?
You can express this as a SQL query:
select df.customerid, t.timeslot,
t.A1, t.A2, t.An
from (select distinct customerid from df) c cross join
(select distinct timeslot from df) t left join
df
on df.customerid = c.customerid and df.timeslot = t.timeslot;
Notes:
You should probably put this into another dataframe.
You might have tables with the available customers and/or timeslots. Use those instead of the subqueries.
I think can used the answer of gordon linoff but you can add the following thinsg as you stated that there are millions of customer and you are performing join in them.
use tally table for TimeSlot?? because it might give a better performance.
for more usabllity please refer the following link
http://www.sqlservercentral.com/articles/T-SQL/62867/
and I think you should use partition or row number function to divide you column customerid and select the customers based on some partition value. For example just select the row number values and then cross join with the tally table. it can imporove your performance .

Return a limit of 2 records for each distinct column value

Assume I have a table that looks like this:
| Col A | Col B | Col C |
|-------|-------|-------|
| 1 | A | 54 |
| 1 | A | 56 |
| 1 | B | 55 |
| 1 | B | 51 |
| 1 | C | 36 |
| 1 | C | 23 |
| 1 | D | 62 |
| 1 | D | 11 |
| 2 | B | 88 |
| 2 | B | 17 |
| 2 | C | 56 |
| 2 | C | 86 |
| 2 | D | 47 |
| 2 | D | 29 |
What I want to do is grab the table to look like this:
| Col A | Col B | Col C |
|-------|-------|-------|
| 1 | A | 54 |
| 1 | A | 56 |
| 2 | B | 88 |
| 2 | B | 17 |
I'm pretty sure there is a way to do this, I just don't know how. First, I thought a DISTINCT ON selector would work, but that only returns one record for each value. In this case, I need two records for each value.
One way to do this would be to use a window function to add a row number to each partition of data ordered by however you want and then select the anything with a row number less than 2.
With CTE AS (
SELECT colA, ColB, ColC, Row_Number() over (Partition by ColA ORDER By ColB , ColC) RN
FROM Table)
Select * from cte where RN <=2
Since I didn't know what values of c you wanted, I choose to order by colC (ascending) so the lowest values of C would be returned for a given A+B combination.
with
grp as (select col_a from table group by col_a) -- It should be only index scan, not scanning the whole table
select * from grp join lateral (
select * from table
where grp.col_a = table.col_a
order by <desired order here>
limit 2) on true -- It also avoiding the full scan if properly indexes provided

group by top two results based on order

I have been trying to get this to work with some row_number, group by, top, sort of things, but I am missing some fundamental concept. I have a table like so:
+-------+-------+-------+
| name | ord | f_id |
+-------+-------+-------+
| a | 1 | 2 |
| b | 5 | 2 |
| c | 6 | 2 |
| d | 2 | 1 |
| e | 4 | 1 |
| a | 2 | 3 |
| c | 50 | 4 |
+-------+-------+-------+
And my desired output would be:
+-------+---------+--------+-------+
| f_id | ord_n | ord | name |
+-------+---------+--------+-------+
| 2 | 1 | 1 | a |
| 2 | 2 | 5 | b |
| 1 | 1 | 2 | d |
| 1 | 2 | 4 | e |
| 3 | 1 | 2 | a |
| 4 | 1 | 50 | c |
+-------+---------+--------+-------+
Where data is ordered by the ord value, and only up to two results per f_id. Should I be working on a Stored Procedure for this or can I just do it with SQL? I have experimented with some select TOP subqueries, but nothing has even come close..
Here are some statements to create the test table:
create table help(name varchar(255),ord tinyint,f_id tinyint);
insert into help values
('a',1,2),
('b',5,2),
('c',6,2),
('d',2,1),
('e',4,1),
('a',2,3),
('c',50,4);
You may use Rank or DENSE_RANK functions.
select A.name, A.ord_n, A.ord , A.f_id from
(
select
RANK() OVER (partition by f_id ORDER BY ord asc) AS "Rank",
ROW_NUMBER() OVER (partition by f_id ORDER BY ord asc) AS "ord_n",
help.*
from help
) A where A.rank <= 2
Sqlfiddle demo