SQL window excluding current group? - sql

I'm trying to provide rolled up summaries of the following data including only the group in question as well as excluding the group. I think this can be done with a window function, but I'm having problems with getting the syntax down (in my case Hive SQL).
I want the following data to be aggregated
+------------+---------+--------+
| date | product | rating |
+------------+---------+--------+
| 2018-01-01 | A | 1 |
| 2018-01-02 | A | 3 |
| 2018-01-20 | A | 4 |
| 2018-01-27 | A | 5 |
| 2018-01-29 | A | 4 |
| 2018-02-01 | A | 5 |
| 2017-01-09 | B | NULL |
| 2017-01-12 | B | 3 |
| 2017-01-15 | B | 4 |
| 2017-01-28 | B | 4 |
| 2017-07-21 | B | 2 |
| 2017-09-21 | B | 5 |
| 2017-09-13 | C | 3 |
| 2017-09-14 | C | 4 |
| 2017-09-15 | C | 5 |
| 2017-09-16 | C | 5 |
| 2018-04-01 | C | 2 |
| 2018-01-13 | D | 1 |
| 2018-01-14 | D | 2 |
| 2018-01-24 | D | 3 |
| 2018-01-31 | D | 4 |
+------------+---------+--------+
Aggregated results:
+------+-------+---------+----+------------+------------------+----------+
| year | month | product | ct | avg_rating | avg_rating_other | other_ct |
+------+-------+---------+----+------------+------------------+----------+
| 2018 | 1 | A | 5 | 3.4 | 2.5 | 4 |
| 2018 | 2 | A | 1 | 5 | NULL | 0 |
| 2017 | 1 | B | 4 | 3.6666667 | NULL | 0 |
| 2017 | 7 | B | 1 | 2 | NULL | 0 |
| 2017 | 9 | B | 1 | 5 | 4.25 | 4 |
| 2017 | 9 | C | 4 | 4.25 | 5 | 1 |
| 2018 | 4 | C | 1 | 2 | NULL | 0 |
| 2018 | 1 | D | 4 | 2.5 | 3.4 | 5 |
+------+-------+---------+----+------------+------------------+----------+
I've also considered producing two aggregates, one with the product in question and one without, but having trouble with creating the appropriate joining key.

You can do:
select year(date), month(date), product,
count(*) as ct, avg(rating) as avg_rating,
sum(count(*)) over (partition by year(date), month(date)) - count(*) as ct_other,
((sum(sum(rating)) over (partition by year(date), month(date)) - sum(rating)) /
(sum(count(*)) over (partition by year(date), month(date)) - count(*))
) as avg_other
from t
group by year(date), month(date), product;
The rating for the "other" is a bit tricky. You need to add everything up and subtract out the current row -- and calculate the average by doing the sum divided by the count.

Related

windowing function avg in Hive with - over (order by colName)

i'm trying to understand how windowing function avg works, and somehow it seems to not be working as i expect.
here is the dataset :
select * from winsales;
+-------------------+------------------+--------------------+-------------------+---------------+-----------------------+--+
| winsales.salesid | winsales.dateid | winsales.sellerid | winsales.buyerid | winsales.qty | winsales.qty_shipped |
+-------------------+------------------+--------------------+-------------------+---------------+-----------------------+--+
| 30001 | NULL | 3 | b | 10 | 10 |
| 10001 | NULL | 1 | c | 10 | 10 |
| 10005 | NULL | 1 | a | 30 | NULL |
| 40001 | NULL | 4 | a | 40 | NULL |
| 20001 | NULL | 2 | b | 20 | 20 |
| 40005 | NULL | 4 | a | 10 | 10 |
| 20002 | NULL | 2 | c | 20 | 20 |
| 30003 | NULL | 3 | b | 15 | NULL |
| 30004 | NULL | 3 | b | 20 | NULL |
| 30007 | NULL | 3 | c | 30 | NULL |
| 30001 | NULL | 3 | b | 10 | 10 |
+-------------------+------------------+--------------------+-------------------+---------------+-----------------------+--+
When i fire the following query ->
select salesid, sellerid, qty, avg(qty) over (order by sellerid) as avg_qty from winsales order by sellerid,salesid;
I get the following ->
+----------+-----------+------+---------------------+--+
| salesid | sellerid | qty | avg_qty |
+----------+-----------+------+---------------------+--+
| 10001 | 1 | 10 | 20.0 |
| 10005 | 1 | 30 | 20.0 |
| 20001 | 2 | 20 | 20.0 |
| 20002 | 2 | 20 | 20.0 |
| 30001 | 3 | 10 | 18.333333333333332 |
| 30001 | 3 | 10 | 18.333333333333332 |
| 30003 | 3 | 15 | 18.333333333333332 |
| 30004 | 3 | 20 | 18.333333333333332 |
| 30007 | 3 | 30 | 18.333333333333332 |
| 40001 | 4 | 40 | 19.545454545454547 |
| 40005 | 4 | 10 | 19.545454545454547 |
+----------+-----------+------+---------------------+--+
Question is - how is the avg(qty) being calculated.
Since i'm not using partition by, i would expect the avg(qty) to be the same for all rows.
Any ideas ?
if you want to have same avg(qty) to get for all rows then remove order by sellerid in over clause, then you are going to have 19.545454545454547 value for all the rows.
Query to get same avg(qty) for all rows:
hive> select salesid, sellerid, qty, avg(qty) over () as avg_qty from winsales order by sellerid,salesid;
If we include order by sellerid in over clause then you are getting cumulative avg is caluculated for each sellerid.
i.e. for
sellerid 1 you are having 2 records total 2 records with qty as 10,30 so avg would be
(10+30)/2.
sellerid 2 you are having 2 records total 4 records with qty as 20,20 so avg would be
(10+30+20+20)/4 = 20.0
sellerid 3 you are having 5 records total 9 records with qty as so 10,10,15,20,30 avg would be
(10+30+20+20+10+10+15+20+30)/9 = 18.333
sellerid 4 avg is 19.545454545454547
when we include over clause then this is an expected behavior from hive.

Why do you need to include a field in GROUP BY when using OVER (PARTITION BY x)?

I have a table for which I want to do a simple sum of a field, grouped by two columns. I then want the total for all values for each year_num.
See example: http://rextester.com/QSLRS68794
This query is throwing: "42803: column "foo.num_cust" must appear in the GROUP BY clause or be used in an aggregate function", and I cannot figure out why. Why would an aggregate function using the OVER (PARTITION BY x) require the summed field to be in GROUP BY??
select
year_num
,age_bucket
,sum(num_cust)
--,sum(num_cust) over (partition by year_num) --THROWS ERROR!!
from
foo
group by
year_num
,age_bucket
order by 1,2
TABLE:
| loc_id | year_num | gen | cust_category | cust_age | num_cust | age_bucket |
|--------|-----------|------|----------------|-----------|-----------|-------------|
| 1 | 2016 | M | cash | 41 | 2 | 04_<45 |
| 1 | 2016 | F | Prepaid | 41 | 1 | 03_<35 |
| 1 | 2016 | F | cc | 61 | 1 | 05_45+ |
| 1 | 2016 | F | cc | 19 | 2 | 02_<25 |
| 1 | 2016 | M | cc | 64 | 1 | 05_45+ |
| 1 | 2016 | F | cash | 46 | 1 | 05_45+ |
| 1 | 2016 | F | cash | 27 | 3 | 03_<35 |
| 1 | 2016 | M | cash | 42 | 1 | 04_<45 |
| 1 | 2017 | F | cc | 35 | 1 | 04_<45 |
| 1 | 2017 | F | cc | 37 | 1 | 04_<45 |
| 1 | 2017 | F | cash | 46 | 1 | 05_45+ |
| 1 | 2016 | F | cash | 19 | 4 | 02_<25 |
| 1 | 2017 | M | cash | 43 | 1 | 04_<45 |
| 1 | 2017 | M | cash | 29 | 1 | 03_<35 |
| 1 | 2016 | F | cc | 13 | 1 | 01_<18 |
| 1 | 2017 | F | cash | 16 | 2 | 01_<18 |
| 1 | 2016 | F | cc | 17 | 2 | 01_<18 |
| 1 | 2016 | M | cc | 17 | 2 | 01_<18 |
| 1 | 2017 | F | cash | 18 | 9 | 02_<25 |
DESIRED OUTPUT:
| year_num | age_bucket | sum | sum over (year_num) |
|----------|------------|-----|---------------------|
| 2016 | 01_<18 | 5 | 21 |
| 2016 | 02_<25 | 6 | 21 |
| 2016 | 03_<35 | 4 | 21 |
| 2016 | 04_<45 | 3 | 21 |
| 2016 | 05_45+ | 3 | 21 |
| 2017 | 01_<18 | 2 | 16 |
| 2017 | 02_<25 | 9 | 16 |
| 2017 | 03_<35 | 1 | 16 |
| 2017 | 04_<45 | 3 | 16 |
| 2017 | 05_45+ | 1 | 16 |
You need to nest the sum()s:
select year_num, age_bucket, sum(num_cust),
sum(sum(num_cust)) over (partition by year_num) --WORKS!!
from foo
group by year_num, age_bucket
order by 1, 2;
Why? Well, the window function is not doing aggregation. The argument needs to be an expression that can be evaluated after the group by (because this is an aggregation query). Because num_cust is not a group by key, it needs an aggregation function.
Perhaps this is clearer if you used a subquery:
select year_num, age_bucket, sum_num_cust,
sum(sum_num_cust) over (partition by year_num)
from (select year_num, age_bucket, sum(num_cust) as sum_num_cust
from foo
group by year_num, age_bucket
) ya
order by 1, 2;
These two queries do exactly the same thing. But with the subquery it should be more obvious why you need the extra aggregation.

Join Distinct or First

I have a table structure for SalesItems, and Sales.
SalesItems is setup something like this
| SaleItemID | SaleID | ProductID | ProductType |
| 1 | 1 | 1 | 1 |
| 2 | 1 | 2 | 2 |
| 3 | 1 | 15 | 1 |
| 4 | 2 | 5 | 2 |
| 5 | 3 | 1 | 1 |
| 6 | 3 | 8 | 5 |
And Sales is setup something like this
| Sale | Cash |
| 1 | 1.00 |
| 2 | 10.00 |
| 3 | 28.50 |
I am trying to export a basic 'Daily History' that uses joins to spit out the information like this.
| Date | StoreID | Type1Sales | Type2Sales | ... | Cash Taken |
| 5/2 | 50 | 50 | 40 | ... | 39.50 |
| 5/3 | 50 | 10 | 32.50 | ... | 48.50 |
The issue I'm having is if I do an inner join From Sales to Sales Items, I'll end up with this.
| SaleItemID | SaleID | ProductID | ProductType | Sale | Cash |
| 1 | 1 | 1 | 1 | 1 | 1.00 |
| 2 | 1 | 2 | 2 | 1 | 1.00 |
| 3 | 1 | 15 | 1 | 1 | 1.00 |
| 4 | 2 | 5 | 2 | 2 | 10.00 |
| 5 | 3 | 1 | 1 | 3 | 28.50 |
| 6 | 3 | 8 | 5 | 3 | 28.50 |
So if I do a SUM(Cash), then I'll end up returning $70.00, instead of the correct $39.50. I'm not the best with joins, so I've been researching outer joins and such, but none of those seem to work as it's still matching up. Is there a way to only match on the FIRST instance, and return NULL for the rest? For example, something like this
| SaleItemID | SaleID | ProductID | ProductType | Sale | Cash |
| 1 | 1 | 1 | 1 | 1 | 1.00 |
| 2 | 1 | 2 | 2 | 1 | NULL |
| 3 | 1 | 15 | 1 | 1 | NULL |
| 4 | 2 | 5 | 2 | 2 | 10.00 |
| 5 | 3 | 1 | 1 | 3 | 28.50 |
| 6 | 3 | 8 | 5 | 3 | NULL |
Or do you have any other suggestions for returning back the correct amount of Cash for each particular day?
Use DISTINCT(SaleID) in your SELECT to return a single row for each Sale ID.

how to write a query to get multilevel data

I have four tables as below:
tblAccount
Id i sprimary key
+----+-----------------+
| Id | AccName |
+----+-----------------+
| 1 | AccountA |
| 2 | AccountB |
+----+-----------------+
tblLocation
Id is primary key.
+----+---------------+
| Id | LocName |
+----+---------------+
| 1 | LocationA |
| 2 | LocationB |
| 3 | LocationC |
+----+---------------+
tblAccountwiseLocation
Id i sprimary key.LocId and AccId are foreign key.
+----+---------------+---------------+
| Id | LocId | AccId |
+----+---------------+---------------+
| 1 | 1 | 1 |
| 2 | 2 | 1 |
| 3 | 3 | 1 |
| 4 | 1 | 2 |
| 5 | 2 | 2 |
| 6 | 3 | 2 |
+----+---------------+---------------+
tblRSCMaster
Id i sprimary key.LocId and AccId are foreign key.
+----+---------------+---------------+----------------+------------------+
| Id | LocId | AccId | RSCNo | DateOfAddition |
+----+---------------+---------------+----------------+------------------+
| 1 | 1 | 1 | Acc1_Loc1_1_14 | 15/01/2014 |
| 2 | 2 | 1 | Acc1_Loc2_1_14 | 15/01/2014 |
| 3 | 3 | 1 | Acc1_Loc2_1_14 | 15/01/2014 |
| 4 | 1 | 2 | Acc2_Loc1_1_14 | 15/01/2014 |
| 5 | 2 | 2 | Acc2_Loc2_1_14 | 15/01/2014 |
| 6 | 3 | 2 | Acc2_Loc3_1_14 | 15/01/2014 |
| 7 | 1 | 1 | Acc1_Loc1_2_14 | 15/02/2014 |
| 8 | 2 | 1 | Acc1_Loc2_2_14 | 15/02/2014 |
| 9 | 3 | 1 | Acc1_Loc3_2_14 | 15/02/2014 |
| 10 | 1 | 2 | Acc2_Loc1_2_14 | 15/02/2014 |
| 11 | 2 | 2 | Acc2_Loc2_2_14 | 15/02/2014 |
| 12 | 3 | 2 | Acc2_Loc3_2_14 | 15/02/2014 |
| 13 | 1 | 1 | Acc1_Loc1_3_14 | 15/03/2014 |
| 14 | 2 | 1 | Acc1_Loc2_3_14 | 15/03/2014 |
| 15 | 3 | 1 | Acc1_Loc3_3_14 | 15/03/2014 |
| 16 | 1 | 2 | Acc2_Loc1_3_14 | 15/03/2014 |
| 17 | 2 | 2 | Acc2_Loc2_3_14 | 15/03/2014 |
| 18 | 3 | 2 | Acc2_Loc3_3_14 | 15/03/2014 |
| 19 | 1 | 1 | Acc1_Loc1_4_14 | 15/04/2014 |
| 20 | 2 | 1 | Acc1_Loc2_4_14 | 15/04/2014 |
| 21 | 3 | 1 | Acc1_Loc3_4_14 | 15/04/2014 |
| 22 | 1 | 2 | Acc2_Loc1_4_14 | 15/04/2014 |
| 23 | 2 | 2 | Acc2_Loc2_4_14 | 15/04/2014 |
| 24 | 3 | 2 | Acc2_Loc3_4_14 | 15/04/2014 |
| 25 | 1 | 1 | Acc1_Loc1_5_14 | 15/05/2014 |
| 26 | 2 | 1 | Acc1_Loc2_5_14 | 15/05/2014 |
| 27 | 3 | 1 | Acc1_Loc3_5_14 | 15/05/2014 |
| 28 | 1 | 2 | Acc2_Loc1_5_14 | 15/05/2014 |
| 29 | 2 | 2 | Acc2_Loc2_5_14 | 15/05/2014 |
| 30 | 3 | 2 | Acc2_Loc3_5_14 | 15/05/2014 |
+----+---------------+---------------+----------------+------------------+
Acc1_Loc1_1_14 resembles RSC for LocationA of AccountA for Jan 2014.
I need to get a output as below from tblRSCMaster.
+---------------+---------------+----------------+------------------+
| LocId | AccId | RSCNo | DateOfAddition |
+---------------+---------------+----------------+------------------+
| 1 | 1 | Acc1_Loc1_3_14 | 15/03/2014 |
| 1 | 1 | Acc1_Loc1_4_14 | 15/04/2014 |
| 1 | 1 | Acc1_Loc1_5_14 | 15/05/2014 |
| 2 | 1 | Acc1_Loc2_3_14 | 15/03/2014 |
| 2 | 1 | Acc1_Loc2_4_14 | 15/04/2014 |
| 2 | 1 | Acc1_Loc2_5_14 | 15/05/2014 |
| 3 | 1 | Acc1_Loc3_3_14 | 15/03/2014 |
| 3 | 1 | Acc1_Loc3_4_14 | 15/04/2014 |
| 3 | 1 | Acc1_Loc3_5_14 | 15/05/2014 |
+---------------+---------------+----------------+------------------+
Each account has multiple locations and each location has multiple RSCs.
I need to get last three RSCs for each location for AccountA.
I have tried the below query:
SELECT tblAccountwiseLocation.LocId,tblAccountwiseLocation.AccId,tblRSCMaster.RSCNo,tblRSCMaster.DateOfAddition FROM tblAccountwiseLocation
INNER JOIN tblRSCMaster ON tblAccountwiseLocation.LocId= tblRSCMaster.LocId
where tblRSCMaster.AccId=1
But not getting the proper output.
Please help me out.
Thank you all in advance.
You can wrap the existing query inside a common table expression, and use ROW_NUMBER() to get only the last 3 (by tblRSCMaster.DateOfAddition) entries per tblAccountwiseLocation.LocId.
WITH cte AS (
SELECT tblAccountwiseLocation.LocId,
tblAccountwiseLocation.AccId,
tblRSCMaster.RSCNo,
tblRSCMaster.DateOfAddition,
ROW_NUMBER() OVER (PARTITION BY tblAccountwiseLocation.LocId
ORDER BY tblRSCMaster.DateOfAddition DESC) rn
FROM tblAccountwiseLocation
INNER JOIN tblRSCMaster
ON tblAccountwiseLocation.LocId = tblRSCMaster.LocId
AND tblAccountwiseLocation.AccId = tblRSCMaster.AccId
WHERE tblRSCMaster.AccId=1
)
SELECT LocId, AccId, RSCNo, DateOfAddition
FROM cte
WHERE rn <= 3
ORDER BY LocId, AccId, DateOfAddition
An SQLfiddle to test with.
Is this what you need?
select m.*
from (select m.*, row_number() over (partition by accID
order by DateOfAddition desc) as seqnum
from tblRSCMaster
where m.locid = 1
) m
where seqnum <= 3
order by AccId, DateOfAddition;
I think you need to filter on the locid rather than on the AccId to get what you want.

Ask about query in sql server

i have table like this:
| ID | id_number | a | b |
| 1 | 1 | 0 | 215 |
| 2 | 2 | 28 | 8952 |
| 3 | 3 | 10 | 2000 |
| 4 | 1 | 0 | 215 |
| 5 | 1 | 0 |10000 |
| 6 | 3 | 10 | 5000 |
| 7 | 2 | 3 |90933 |
I want to sum a*b where id_number is same, what the query to get all value for every id_number? for example the result is like this :
| ID | id_number | result |
| 1 | 1 | 0 |
| 2 | 2 | 523455 |
| 3 | 3 | 70000 |
This is a simple aggregation query:
select id_number, sum(a*b)
from t
group by id_number
I'm not sure what the first column is for.