I have table like this:
value nextValue
1 2
2 3
3 20
20 21
21 22
22 23
23 NULL
Value is ordered ASC, nextValue is next row Value.
requirement is group by with condition nextValue-value>10, and count how many values in different groups.
For example, there should be two groups (1,2,3) and (20,21,22,23), first group count is 3, the second group count is 4.
I'm trying to mark each group with unique number, so I could group by these marked nums
value nextValue mark
1 2 1
2 3 1
3 20 1
20 21 2
21 22 2
22 23 2
23 NULL 2
But I don't know how to write mark column, I need an autocrement variable when nextValue-value>10.
Can I make it happen in Hive? Or there's better solution for the requirement?
If I understand correctly, you can use a cumulative sum. The idea is to set a flag when next_value - value > 10. This identifies the groups. So, this query adds a group number:
select t.*,
sum(case when nextvalue > value + 10 then 1 else 0 end) over (order by value desc) as mark
from t
order by value;
You might not find this solution satisfying, because the numbering is in descending order. So, a bit more arithmetic fixes that:
select t.*,
(sum(case when nextvalue > value + 10 then 1 else 0 end) over () + 1 -
sum(case when nextvalue > value + 10 then 1 else 0 end) over (order by value desc)
) as mark
from t
order by value;
Here is a db<>fiddle.
Calculate previous value, then calculate new_group_flag if value-prev_value >10, then calculate cumulative sum of new_group_flag to get group number (mark). Finally you can calculate group count using analytics function or group-by (in my example analytics count is used to show you the full dataset with all intermediate calculations). See comments in the code.
Demo:
with your_data as (--use your table instead of this
select stack(10, --the number of tuples generated
1 ,
2 ,
3 ,
20 ,
21 ,
22 ,
23 ,
40 ,
41 ,
42
) as value
)
select --4. Calculate group count, etc, etc
value, prev_value, new_group_flag, group_number,
count(*) over(partition by group_number) as group_count
from
(
select --3. Calculate cumulative sum of new group flag to get group number
value, prev_value, new_group_flag,
sum(new_group_flag) over(order by value rows between unbounded preceding and current row)+1 as group_number
from
(
select --2. calculate new_group_flag
value, prev_value, case when value-prev_value >10 then 1 else 0 end as new_group_flag
from
(
select --1 Calculate previous value
value, lag(value) over(order by value) prev_value
from your_data
)s
)s
)s
Result:
value prev_value new_group_flag group_number group_count
1 \N 0 1 3
2 1 0 1 3
3 2 0 1 3
20 3 1 2 4
21 20 0 2 4
22 21 0 2 4
23 22 0 2 4
40 23 1 3 3
41 40 0 3 3
42 41 0 3 3
This works for me
It needs "rows between unbounded preceding and current row" in my case.
select t.*,
sum(case when nextvalue > value + 10 then 1 else 0 end) over (order by value desc rows between unbounded preceding and current row) as mark
from t
order by value;
Related
i have a table like this:
ID
num_A
num_B
1
1
168
2
1
4
2
5
24
2
6
24
3
1
36
So, num_A and num_B represent a fraction. That means for ID=1, i have 1/168, ID=2 ---> (1/4)+(5/24)+(6/24) = 17/24, ID=3 --> 1/36....
I need to add 2 columns, one with the sum(num_A) and one with the denominator num_B, for those with the same ID. So the example should be:
ID
num_A
num_B
sumA
denom_B
1
1
168
1
168
2
1
4
17
24
2
5
24
17
24
2
6
24
17
24
3
1
36
1
36
My problem is that i dont know how to calculate the denominator for each different fraction in postgres.
In general PostgreSQL provides the LCM function that returns the least common multiple (the smallest strictly positive number that is an integral multiple of both inputs), but it takes only two arguments and cannot be used to process rowset column values.
Thus, to get the LCM of rows with the same ID value, you can use a recursive CTE to process the rows one by one, using the LCM function with the LCM calculated in the previous step (in the first step equal to the value of num_B ) and the current value of num_B as arguments. This will produce the LCM value of all previous num_B and the current value for each row.
Finally, you can get the maximum (the last if to be exact, it would be the maximum anyway) calculated LCM value for rows grouped by ID and that will be the LCM for all num_B values with the same ID.
The rest is simple - divide, multiply and sum.
Query:
WITH t_rn AS (
SELECT *, ROW_NUMBER() OVER (PARTITION BY id ORDER BY num_b) AS rn FROM t
),
least_common_multiple AS (
WITH RECURSIVE least_multiples AS (
SELECT
id,
num_b,
num_b AS lm,
rn
FROM t_rn
WHERE rn = 1
UNION ALL
SELECT
t_rn.id,
t_rn.num_b,
LCM(t_rn.num_b, lm.lm),
t_rn.rn
FROM t_rn
JOIN least_multiples lm ON lm.id = t_rn.id AND t_rn.rn = lm.rn + 1
)
SELECT
id,
MAX(lm) AS lcm
FROM least_multiples
GROUP BY id
)
SELECT
t.*,
SUM(t.num_a * lm.lcm / t.num_b) OVER (PARTITION BY t.id) AS suma,
lm.lcm AS denom_b
FROM t
JOIN least_common_multiple lm ON t.id = lm.id
Output
id
num_a
num_b
suma
denom_b
1
1
168
1
168
2
1
4
17
24
2
5
24
17
24
2
6
24
17
24
3
1
36
1
36
DEMO
I think you are trying to simulate fraction addition,
Try the following query:
with find_mutiplication As
(
Select Id, num_a, num_b,
ROUND(EXP(SUM(LN(ABS(num_b))) over (partition by id))) as mutiplication,
ROUND(EXP(SUM(LN(ABS(num_b))) over (partition by id))) / num_b * num_a as unified
From mytable
)
,
calc as
(
Select *,
mutiplication/ GCD(mutiplication::int, SUM(unified::int)over (partition by id)) denom_B,
num_a * (mutiplication/ GCD(mutiplication::int, SUM(unified::int)over (partition by id)) / num_b) as dv
From find_mutiplication
)
Select id, num_a, num_b,
SUM(dv) Over (Partition By id) As sumA,
denom_b
From calc
Order By id
See demo from db<>fiddle.
To understand how the query works consider the following image:
where the ROUND(EXP(SUM(LN(num_b)) over (partition by id))) will find the multiplication of the dividends for each id. (According to this post)
I have the following example dataset:
ID
Value
Row index (for reference purposes only, does not need to exist in final output)
a
4
1
a
7
2
a
12
3
a
12
4
a
13
5
b
1
6
b
2
7
b
3
8
b
4
9
b
5
10
I would like to write a SQL script that returns the next row which has a Value of N or more than the previously returned row starting from the first row per ID and ordered ascending by [Value]. An example of the final table for N = 3 should look like the following:
ID
Value
Row index
a
4
1
a
7
2
a
12
3
b
1
6
b
4
9
Can this script be written in a vectorised manner? Or must a loop be utilised? Any advice would be greatly appreciated. Thanks!
SQL tables represent unordered sets. There is no definition of "previous" value, unless you have a column that specifies the ordering. With such a column, you can use lag():
select t.*
from (select t.*,
lag(value) over (partition by id order by <ordering column>) as prev_value
from t
) t
where prev_value is null or prev_value <= value - 3;
EDIT:
I think I misunderstood what you want to do. You seem to want to start with the first row for each id. Then get the next row that is 3 or higher in value. Then hold onto that value and get the next that is 3 or higher than that. And so on.
You can do this in SQL using a recursive CTE:
with ts as (
select distinct t.id, t.value, dense_rank() over (partition by id order by value) as seqnum
from t
),
cte as (
select id, value, value as grp_value, 1 as within_seqnum, seqnum
from ts
where seqnum = 1
union all
select ts.id, ts.value,
(case when ts.value >= cte.grp_value + 3 then ts.value else cte.grp_value end),
(case when ts.value >= cte.grp_value + 3 then 1 else cte.within_seqnum + 1 end),
ts.seqnum
from cte join
ts
on ts.id = cte.id and ts.seqnum = cte.seqnum + 1
)
select *
from cte
where within_seqnum = 1
order by id, value;
Here is a db<>fiddle.
I want to take the max value of each partitioned block and find the correlating id(in the same row). I then want to use the singular show_id as the 'winner' and bool_flag all rows in the same partition with a matching show_id.
I am having trouble implementing this, especially the window function-- I have hit multiple issues saying that the subquery is not supported, or "must appear in the GROUP BY clause or be used in an aggregate function sql"
subQ1 as (
select subQ0.*,
case
**when show_id =
(select id from (select show_id, max(rn_max_0)
over (partition by tv_id, show_id)))**
then 1
else 0
end as winner_flag
from subQ0
)
What I have:
tv_id show_id partition_count
1 42 1
1 42 2
1 42 3
1 7 1
2 12 1
2 12 2
2 12 3
2 27 1
What I want:
tv_id show_id partition_count flag
1 42 1 1
1 42 2 1
1 42 3 1
1 7 1 0
2 12 1 1
2 12 2 1
2 12 3 1
2 27 1 0
Because tv_id 1 has the most connections to show_id 42, those rows get flagged.
Ideally, something similar to SQL select only rows with max value on a column, but the partitions and grouping have led to issues. This dataset also has billions of rows so a union would be a nightmare.
Thanks in advance!
For each tv_id, you seem to want the show_id that appears the most. If so:
select s.*,
(case when cnt = max(cnt) over (partition by tv_id)
then 1 else 0
end) as flag
from (select s.*, count(*) over (partition by tv_id, show_id) as cnt
from subQ0 s
) s;
Can I somehow assign a new group to a row when a value in a column changes in T-SQL?
I would be grateful if you can provide solution that will work on unlimited repeating numbers without CTE and functions. I made a solution that work in sutuation with 100 consecutive identical numbers(with
coalesce(lag()over(), lag() over(), lag() over() ) - it is too bulky
but can not make a solution for a case with unlimited number of consecutive identical numbers.
Data
id somevalue
1 0
2 1
3 1
4 0
5 0
6 1
7 1
8 1
9 0
10 0
11 1
12 0
13 1
14 1
15 0
16 0
Expected
id somevalue group
1 0 1
2 1 2
3 1 2
4 0 3
5 0 3
6 1 4
7 1 4
8 1 4
9 0 5
10 0 5
11 1 6
12 0 7
13 1 8
14 1 8
15 0 9
16 0 9
If you just want a group identifier, you can use:
select t.*,
min(id) over (partition by some_value, seqnum - seqnum_1) as grp
from (select t.*,
row_number() over (order by id) as seqnum,
row_number() over (partition by somevalue order by id) as sequm_1
from t
) t;
If you want them enumerated . . . well, you can enumerate the id above using dense_rank(). Or you can use lag() and a cumulative sum:
select t.*,
sum(case when some_value = prev_sv then 0 else 1 end) over (order by id) as grp
from (select t.*,
lag(somevalue) over (order by id) as prev_sv
from t
) t;
Here's a different approach:
First I created a view to provide the group increment on each row:
create view increments as
select
n2.id,n2.somevalue,
case when n1.somevalue=n2.somevalue then 0 else 1 end as increment
from
(select 0 as id,1 as somevalue union all select * from mytable) n1
join mytable n2
on n2.id = n1.id+1
Then I used this view to produce the group values as cumulative sums of the increments:
select id, somevalue,
(select sum(increment) from increments i1 where i1.id <= i2.id)
from increments i2
How to get optimized query for this
date_one | date_two
------------------------
01.02.1999 | 31.05.2003
01.01.2004 | 01.01.2010
02.01.2010 | 10.10.2011
11.10.2011 | (null)
I need to get this
date_one | date_two | group
------------------------------------
01.02.1999 | 31.05.2003 | 1
01.01.2004 | 01.01.2010 | 2
02.01.2010 | 10.10.2011 | 2
11.10.2011 | (null) | 2
The group number is assigned as follows. Order the rows by date_one ascending. First row gets group = 1. Then for each row if date_one is the date immediately following date_two of the previous row, the group number stays the same as in the previous row, otherwise it increases by one.
You can do this using left join and a cumulative sum:
select t.*, sum(case when tprev.date_one is null then 1 else 0 end) over (order by t.date_one) as grp
from t left join
t tprev
on t.date_one = tprev.date_two + 1;
The idea is to find where the gaps begin (using the left join) and then do a cumulative sum of such beginnings to define the group.
If you want to be more inscrutable, you could write this as:
select t.*,
count(*) over (order by t.date_one) - count(tprev.date_one) over (order by t.date_one) as grp
from t left join
t tprev
on t.date_one = tprev.date_two + 1;
One way is using window function:
select
date_one,
date_two,
sum(x) over (order by date_one) grp
from (
select
t.*,
case when
lag(date_two) over (order by date_one) + 1 =
date_one then 0 else 1 end x
from t
);
It finds the date_two from the last row using analytic function lag and check if it in continuation with date_one from this row (in increasing order of date_one).
How it works:
lag(date_two) over (order by date_one)
(In the below explanation, when I say first, next, previous or last row, it's based on increasing order of date_one with null values at the end)
The above produces produces NULL for the first row as there is no row before it to get date_two from and previous row's date_two for the subsequent rows.
case when
lag(date_two)
over (order by date_one) + 1 = date_one then 0
else 1 end
Since, the lag produces NULL for the very first row (since NULL = anything expression always finally evaluates to false), output of case will be 1.
For further rows, similar check will be done to produce a new column x in the query output which has value 1 when the previous row's date_two is not in continuation with this row's date_one.
Then finally, we can do an incremental sum on x to find the required group values. See the value of x below for understanding:
SQL> with t (date_one,date_two) as (
2 select to_date('01.02.1999','dd.mm.yyyy'),to_date('31.05.2003','dd.mm.yyyy') from dual union
all
3 select to_date('01.01.2004','dd.mm.yyyy'),to_date('01.01.2010','dd.mm.yyyy') from dual union
all
4 select to_date('02.01.2010','dd.mm.yyyy'),to_date('10.10.2011','dd.mm.yyyy') from dual union
all
5 select to_date('11.10.2011','dd.mm.yyyy'),null from dual
6 )
7 select
8 date_one,
9 date_two,
10 x,
11 sum(x) over (order by date_one) grp
12 from (
13 select
14 t.*,
15 case when
16 lag(date_two) over (order by date_one) + 1 =
17 date_one then 0 else 1 end x
18 from t
19 );
DATE_ONE DATE_TWO X GRP
--------- --------- ---------- ----------
01-FEB-99 31-MAY-03 1 1
01-JAN-04 01-JAN-10 1 2
02-JAN-10 10-OCT-11 0 2
11-OCT-11 0 2
SQL>