I am working on Hive and am facing an issue with rolling counts. The sample data I am working on is as shown below:
and the output I am expecting is as shown below:
I tried using the following query but it is not returning the rolling count:
select event_dt,status, count(distinct account) from
(select *, row_number() over (partition by account order by event_dt
desc)
as rnum from table.A
where event_dt between '2018-05-02' and '2018-05-04') x where rnum =1
group by event_dt, status;
Please help me with this if some one has solved a similar issue.
You seem to just want conditional aggregation:
select event_dt,
sum(case when status = 'Registered' then 1 else 0 end) as registered,
sum(case when status = 'active_acct' then 1 else 0 end) as active_acct,
sum(case when status = 'suspended' then 1 else 0 end) as suspended,
sum(case when status = 'reactive' then 1 else 0 end) as reactive
from table.A
group by event_dt
order by event_dt;
EDIT:
This is a tricky problem. The solution I've come up with does a cross-product of dates and users and then calculates the most recent status as of each date.
So:
select a.event_dt,
sum(case when aa.status = 'Registered' then 1 else 0 end) as registered,
sum(case when aa.status = 'active_acct' then 1 else 0 end) as active_acct,
sum(case when aa.status = 'suspended' then 1 else 0 end) as suspended,
sum(case when aa.status = 'reactive' then 1 else 0 end) as reactive
from (select d.event_dt, ac.account, a.status,
max(case when a.status is not null then a.timestamp end) over (partition by ac.account order by d.event_dt) as last_status_timestamp
from (select distinct event_dt from table.A) d cross join
(select distinct account from table.A) ac left join
(select a.*,
row_number() over (partition by account, event_dt order by timestamp desc) as seqnum
from table.A a
) a
on a.event_dt = d.event_dt and
a.account = ac.account and
a.seqnum = 1 -- get the last one on the date
) a left join
table.A aa
on aa.timestamp = a.last_status_timestamp and
aa.account = a.account
group by d.event_dt
order by d.event_dt;
What this is doing is creating a derived table with rows for all accounts and dates. This has the status on certain days, but not all days.
The cumulative max for last_status_timestamp calculates the most recent timestamp that has a valid status. This is then joined back to the table to get the status on that date. Voila! This is the status used for the conditional aggregation.
The cumulative max and join is a work-around because Hive does not (yet?) support the ignore nulls option in lag().
Related
I have a code as below where I want to count number of first purchases for a given period of time. I have a column in my sales table where if the buyer is not a first time buyer, then is_first_purchase = 0
For example:
buyer_id = 456391 is already an existing buyer who made purchases on 2 different dates.
Hence is_first_purchase column will show as 0 as per below.
If i do a count() on is_first_purchase for this buyer_id = 456391 then it should return 0 instead of 2.
My query is as follows:
with first_purchases as
(select *,
case when is_first_purchase = 1 then 'Yes' else 'No' end as first_purchase
from sales)
select
count(case when first_purchase = 'Yes' then 1 else 0 end) as no_of_first_purchases
from first_purchases
where buyer_id = 456391
and date_id between '2021-02-01' and '2021-03-01'
order by 1 desc;
It returned the below which is not an intended output
Appreciate if someone can help explain how to exclude is_first_purchase = 0 from the count, thanks.
Because COUNT function count when the value isn't NULL (include 0), if you don't want to count, need to let CASE WHEN return NULL
There are two ways you can count as your expectation, one is SUM other is COUNT but remove the part of else 0
SUM(case when first_purchase = 'Yes' then 1 else 0 end) as no_of_first_purchases
COUNT(case when first_purchase = 'Yes' then 1 end) as no_of_first_purchases
From your question, I would combine CTE and main query as below
select
COUNT(case when is_first_purchase = 1 then 1 end) as no_of_first_purchases
from sales
where buyer_id = 456391
and date_id between '2021-02-01' and '2021-03-01'
order by 1 desc;
I think that you are using COUNT() when you want SUM().
with first_purchases as
(select *,
case when is_first_purchase = 1 then 'Yes' else 'No' end as first_purchase
from sales)
select
SUM(case when first_purchase = 'Yes' then 1 else 0 end) as no_of_first_purchases
from first_purchases
where buyer_id = 456391
and date_id between '2021-02-01' and '2021-03-01'
order by 1 desc;
You could simplify your query as:
SELECT COUNT(*) AS
FROM sales no_of_first_purchases
WHERE is_first_purchase = 1
AND buyer_id = 456391
AND date_id BETWEEN '2021-02-01' AND '2021-03-01'
ORDER BY 1 DESC;
It is better to avoid the use of functions like IF and CASE when it can be done with WHERE.
The simplest approach for Trino (f.k.a. Presto SQL) is to use an aggregate with a filter:
count(name) FILTER (WHERE first_purchase = 'Yes') AS no_of_first_purchases
This is a practice question from stratascratch and I'm literally stuck at the final HAVING statement.
Problem statement:
Find the total number of downloads for paying and non-paying users by date. Include only records where non-paying customers have more downloads than paying customers. The output should be sorted by earliest date first and contain 3 columns date, non-paying downloads, paying downloads.
There are three tables:
ms_user_dimension (user_id, acc_id)
ms_acc_dimension (acc_id, paying_customer)
ms_download_facts (date, user_id, downloads)
This is my code so far
SELECT date,
SUM(CASE WHEN paying_customer = 'no' THEN cnt END) AS no,
SUM(CASE WHEN paying_customer = 'yes' THEN cnt END) AS yes
FROM (
SELECT date, paying_customer, SUM(downloads) AS cnt
FROM ms_download_facts d
LEFT JOIN ms_user_dimension u ON d.user_id = u.user_id
LEFT JOIN ms_acc_dimension a ON u.acc_id = a.acc_id
GROUP BY 1, 2
ORDER BY 1, 2
) prePivot
GROUP BY date
HAVING no > yes;
If I remove the HAVING no > yes at the end, the code will run and I can see I have three columns: date, yes, and no. However, if I add the HAVING statement, I get the error "column "no" does not exist...LINE 13: HAVING no > yes"
Can't figure out for the sake of my life what's going on here. Please let me know if anyone figures out something. TIA!
You don't need a subquery for this:
SELECT d.date,
SUM(CASE WHEN a.paying_customer = 'no' THEN d.downloads END) AS no,
SUM(CASE WHEN a.paying_customer = 'yes' THEN d.downloads END) AS yes
FROM ms_download_facts d LEFT JOIN
ms_user_dimension u
ON d.user_id = u.user_id LEFT JOIN
ms_acc_dimension a
ON u.acc_id = a.acc_id
GROUP BY d.date
HAVING SUM(CASE WHEN a.paying_customer = 'no' THEN d.downloads END) > SUM(CASE WHEN a.paying_customer = 'yes' THEN d.downloads END);
You can simplify the HAVING clause to:
HAVING SUM(CASE WHEN a.paying_customer = 'no' THEN 1 ELSE -1 END) > 0
This version assumes that paying_customer only takes on the values 'yes' and 'no'.
You may be able to simplify the query further, depending on the database you are using.
It doesn't like aliases in the having statement. Replace no with:
SUM(CASE WHEN paying_customer = 'no' THEN cnt END)
and do the similar thing for yes.
SELECT date,
SUM(CASE WHEN paying_customer = 'no' THEN cnt END) AS no,
SUM(CASE WHEN paying_customer = 'yes' THEN cnt END) AS yes
FROM (
SELECT date, paying_customer, SUM(downloads) AS cnt
FROM ms_download_facts d
LEFT JOIN ms_user_dimension u ON d.user_id = u.user_id
LEFT JOIN ms_acc_dimension a ON u.acc_id = a.acc_id
GROUP BY 1, 2
ORDER BY 1, 2
) prePivot
GROUP BY date
HAVING SUM(CASE WHEN paying_customer = 'no' THEN cnt END) > SUM(CASE WHEN paying_customer = 'yes' THEN cnt END);
I have Table a(Dimension table) and Table B(Fact table) stores transaction shopper history.
Table a : shopped id(surrogate key) created for unique combination(any of column 2,colum3,column4 repeated it will have same shopper id)
Table b is transaction data.
I am trying to identify New customers and repeated customers for each week, expected output is below.
I am thinking following SQL Statement
Select COUNT(*) OVER (PARTITION BY shopperid,weekdate) as total_new_shopperid for Repeated customer,
for Identifying new customer(ie unique) in same join condition, I am stuck on window function..
thanks,
Sam
You can use the DENSE_RANK analytical function along with aggregate function as follows:
SELECT WEEK_DATE,
COUNT(DISTINCT CASE WHEN DR = 1 THEN SHOPPER_ID END) AS TOTAL_NEW_CUSTOMER,
SUM(CASE WHEN DR = 1 THEN AMOUNT END) AS TOTAL_NEW_CUSTOMER_AMT,
COUNT(DISTINCT CASE WHEN DR > 1 THEN SHOPPER_ID END) AS TOTAL_REPEATED_CUSTOMER,
SUM(CASE WHEN DR > 1 THEN AMOUNT END) AS TOTAL_REPEATED_CUSTOMER_AMT
FROM
(
select T.*,
DENSE_RANK() OVER (PARTITION BY SHOPPER_ID ORDER BY WEEK_DATE) AS DR
FROM YOUR_TABLE T);
GROUP BY WEEK_DATE;
Cheers!!
Tejash's answer is fine (and I'm upvoting it).
However, Oracle is quite efficient with aggregation, so two levels of aggregation might have better performance (depending on the data):
select week_date,
sum(case when min_week_date = week_date then 1 else 0 end) as new_shoppers,
sum(case when min_week_date = week_date then amount else 0 end) as new_shopper_amount,
sum(case when min_week_date > week_date then 1 else 0 end) as returning_shoppers,
sum(case when min_week_date > week_date then amount else 0 end) as returning_amount
from (select shopper_id, week_date,
sum(amount) as amount,
min(week_date) over (partition by shopper_id) as min_week_date
from t
group by shopper_id, week_date
) sw
group by week_date
order by week_date;
Note: If this has better performance, it is probably due to the elimination of count(distinct).
I would like a cross table from the following table.
The cross table should look like this
A pivot table does not seem to solve the problem, because only one column can be used at a time. But in our case we are dealing with 4 different columns. (payment, month, year and free of charge)
I solved the problem by splitting these 4 columns into four different pivot tables, using temporary tables and finally reassembling the obtained data. But this is very complicated, long and confusing, in short not very nice...
The years and months should be shown in ascending form, exactly as you can see in the cross table above.
I have been looking for a solution for quite a while but I can't find the same problem anywhere.
If someone would give me a short, elegant solution I would be very grateful.
Under http://www.sqlfiddle.com/#!18/7216f/2 you can see the problem definition.
Thank you!
You can rank records by date in a subquery with row_number(), and then pivot with conditional aggregation:
select
ClientId,
max(case when rn = 1 then Payment end) Payment1,
max(case when rn = 2 then Payment end) Payment2,
max(case when rn = 3 then Payment end) Payment3,
max(case when rn = 1 then [Month] end) Month1,
max(case when rn = 2 then [Month] end) Month2,
max(case when rn = 3 then [Month] end) Month3,
max(case when rn = 1 then [Year] end) Year1,
max(case when rn = 2 then [Year] end) Year2,
max(case when rn = 3 then [Year] end) Year3,
max(case when rn = 1 then FreeOfCharge end) FreeOfCharge1,
max(case when rn = 2 then FreeOfCharge end) FreeOfCharge2,
max(case when rn = 3 then FreeOfCharge end) FreeOfCharge3
from (
select
t.*,
row_number() over(partition by ClientId order by [Year], [Month]) rn
from mytable t
) t
group by ClientId
You can join the table with itself a few times, as in:
with p as (
select
*, row_number() over(partition by clientid order by year, month) as n
from Payment
)
select
p1.clientid,
p1.payment, p2.payment, p3.payment,
p1.month, p2.month, p3.month,
p1.year, p2.year, p3.year,
p1.freeofcharge, p2.freeofcharge, p3.freeofcharge
from p p1
left join p p2 on p2.clientid = p1.clientid and p2.n = 2
left join p p3 on p3.clientid = p1.clientid and p3.n = 3
where p1.n = 1
See Fiddle.
I have table that looks like this:
I'm trying to build a query, that will show specific partnerId counters groupped by keywordName and month.
To solve first part(without grouping by month), I've built this query:
SELECT keywordName, COUNT(keywordName) as total, IFNULL(b.ebay_count, 0) as ebay, IFNULL(c.amazon_count, 0) as amazon,
FROM LogFilesv2_Dataset.FR_Clickstats_v2 a
LEFT JOIN
(SELECT keywordName as kw , SUM(CASE WHEN partnerId='eBay' THEN 1 ELSE 0 END) as ebay_count
FROM LogFilesv2_Dataset.FR_Clickstats_v2
WHERE partnerId = 'eBay' GROUP BY kw) b
ON keywordName = b.kw
LEFT JOIN
(SELECT keywordName as kw , SUM(CASE WHEN partnerId='AmazonApi' THEN 1 ELSE 0 END) as amazon_count
FROM LogFilesv2_Dataset.FR_Clickstats_v2
WHERE partnerId = 'AmazonApi' GROUP BY kw) c
ON keywordName = c.kw
WHERE keywordName = 'flipper' -- just to filter out single kw.
GROUP BY keywordName, ebay, amazon
It works quite well and returns following output:
Now I'm trying to make additional group by month, but all my attempts returned incorrect results.
Final output supposed to be similar to this:
You can do this with conditional aggregation:
select
date_trunc(dt, month) dt,
keywordName,
count(*) total,
sum(case when partnerId = 'eBay' then 1 else 0 end) ebay,
sum(case when partnerId = 'AmazonApi' then 1 else 0 end) amazon
from LogFilesv2_Dataset.FR_Clickstats_v2
group by date_trun(dt, month), keywordName