Related
I have data from two different sources. On one hand I have user data from our app. This has a primary key of ID and UTC date. There are only rows for UTC dates when are users uses the app. On the other hand I have advertisement campaign attribition data for the users (which can be multiple advertisment campaigns per user). This table has a primary key of ID and campaign and a metric containing a advertisment attribution timestamp. I want to combine the two data sources such that I can compute if a campaign is generating more revenue than it costs among other campaign statistics.
App data example:
SELECT
*
FROM UNNEST(ARRAY<STRUCT<ID INT64, UTC_Date DATE, Revenue FLOAT64>>
[(1, DATE('2021-01-01'), 0),
(1, DATE('2021-01-05'), 5),
(1, DATE('2021-01-10'), 0),
(2, DATE('2021-01-03'), 10),
(2, DATE('2021-01-08'), 0),
(2, DATE('2021-01-09'), 0)])
advertisement campaign attribition data example:
SELECT
*
FROM UNNEST(ARRAY<STRUCT<ID INT64, Attribution_Timestamp Timestamp, campaign_name STRING>>
[(1, TIMESTAMP('2021-01-01 09:54:31'), "A"),
(1, TIMESTAMP('2021-01-09 22:32:51'), "B"),
(2, TIMESTAMP('2021-01-03 19:12:11'), "A")])
The end result I would like to get is:
SELECT
*
FROM UNNEST(ARRAY<STRUCT<ID INT64, UTC_Date DATE, Revenue FLOAT64, campaign_name STRING>>
[(1, DATE('2021-01-01'), 0, "A"),
(1, DATE('2021-01-05'), 5, "A"),
(1, DATE('2021-01-10'), 0, "B"),
(2, DATE('2021-01-03'), 10, "A"),
(2, DATE('2021-01-08'), 0, "A"),
(2, DATE('2021-01-09'), 0, "A")])
This can be achieved by somehow joining the campaign attribution data to the app data and then forward filling.
The problem I have is that the advertisment attribution timestamp can have a mismatch with the UTC dates in the app data table. This means I cannot use a left join as it will not assign campaign_name B to ID 1. Does anyone know an elegant way to solve this problem?
Found a solution! Here is what I did (and a little bit more sample data):
WITH app_data AS
(
SELECT
*
FROM UNNEST(ARRAY<STRUCT<adid INT64, utc_date DATE, Revenue FLOAT64>>
[(1, DATE('2021-01-01'), 0),
(1, DATE('2021-01-05'), 5),
(1, DATE('2021-01-10'), 0),
(1, DATE('2021-01-12'), 0),
(1, DATE('2021-01-15'), 0),
(1, DATE('2021-01-16'), 15),
(1, DATE('2021-01-18'), 0),
(2, DATE('2021-01-03'), 10),
(2, DATE('2021-01-08'), 0),
(2, DATE('2021-01-09'), 0),
(2, DATE('2021-01-15'), 4),
(2, DATE('2021-02-01'), 0),
(2, DATE('2021-02-08'), 8),
(2, DATE('2021-02-15'), 0),
(2, DATE('2021-03-04'), 0),
(2, DATE('2021-03-06'), 12),
(3, DATE('2021-02-15'), 10),
(3, DATE('2021-02-23'), 5),
(3, DATE('2021-03-25'), 0),
(3, DATE('2021-03-30'), 0)])
),
advertisment_attribution_data AS
(
SELECT
*
FROM UNNEST(ARRAY<STRUCT<adid INT64, utc_date DATE, campaign_name STRING>>
[(1, DATE(TIMESTAMP('2021-01-01 09:54:31')), "A"),
(1, DATE(TIMESTAMP('2021-01-09 22:32:51')), "B"),
(1, DATE(TIMESTAMP('2021-01-17 14:30:05')), "C"),
(2, DATE(TIMESTAMP('2021-01-03 19:12:11')), "A"),
(1, DATE(TIMESTAMP('2021-01-15 18:17:57')), "B"),
(3, DATE(TIMESTAMP('2021-03-14 22:32:51')), "C")])
)
SELECT
t1.*,
IFNULL(LAST_VALUE(t2.campaign_name IGNORE NULLS) OVER (PARTITION BY t1.adid ORDER BY t1.utc_date ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW), "Organic") as campaign_name
FROM
app_data t1
LEFT JOIN
advertisment_attribution_data t2
ON t1.adid = t2.adid
AND t1.utc_date = (SELECT MIN(t3.utc_date) FROM app_data t3 WHERE t2.adid=t3.adid AND t2.utc_date <= t3.utc_date)
EDIT
It doesn't work when I select a real table in app_data. It says: Unsupported subquery with table in join predicate.
EDIT 2
Found a way to solve the problem where you cannot use subqueries in joins (apparently it is possible for tables which are not selected from an existing table...) This is the way it works in any case:
WITH app_data AS
(
SELECT
*
FROM UNNEST(ARRAY<STRUCT<adid INT64, utc_date DATE, Revenue FLOAT64>>
[(1, DATE('2021-01-01'), 0),
(1, DATE('2021-01-05'), 5),
(1, DATE('2021-01-10'), 0),
(1, DATE('2021-01-12'), 0),
(1, DATE('2021-01-15'), 0),
(1, DATE('2021-01-16'), 15),
(1, DATE('2021-01-18'), 0),
(2, DATE('2021-01-03'), 10),
(2, DATE('2021-01-08'), 0),
(2, DATE('2021-01-09'), 0),
(2, DATE('2021-01-15'), 4),
(2, DATE('2021-02-01'), 0),
(2, DATE('2021-02-08'), 8),
(2, DATE('2021-02-15'), 0),
(2, DATE('2021-03-04'), 0),
(2, DATE('2021-03-06'), 12),
(3, DATE('2021-02-15'), 10),
(3, DATE('2021-02-23'), 5),
(3, DATE('2021-03-25'), 0),
(3, DATE('2021-03-30'), 0)])
),
advertisment_attribution_data AS
(
SELECT
*,
(
SELECT
MIN(t2.utc_date)
FROM app_data t2
WHERE t1.adid=t2.adid
AND t1.utc_date <= t2.utc_date
) as attribution_join_date -- is the closest next date for this adid in app_data to the attribution date. This ensures the join lateron works.
FROM UNNEST(ARRAY<STRUCT<adid INT64, utc_date DATE, campaign_name STRING>>
[(1, DATE(TIMESTAMP('2021-01-01 09:54:31')), "A"),
(1, DATE(TIMESTAMP('2021-01-09 22:32:51')), "B"),
(1, DATE(TIMESTAMP('2021-01-17 14:30:05')), "C"),
(2, DATE(TIMESTAMP('2021-01-03 19:12:11')), "A"),
(1, DATE(TIMESTAMP('2021-01-15 18:17:57')), "B"),
(3, DATE(TIMESTAMP('2021-03-14 22:32:51')), "C")]) t1
)
SELECT
t1.*,
IFNULL(LAST_VALUE(t2.campaign_name IGNORE NULLS) OVER (PARTITION BY t1.adid ORDER BY t1.utc_date ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW), 'Organic') as campaign_name
FROM
app_data t1
LEFT JOIN
advertisment_attribution_data t2
ON t1.adid = t2.adid
AND t1.utc_date = t2.attribution_join_date
So I have an account number and a reading number that I want to take the cumulative sum of but reset at the beginning of a new reading cycle (I want to reset the running sum).
I am using a window function but cannot figure out how to set it when the new reading cycle exists.
Data has the following format:
The Reading cycle Volume value is what I am attempting to achieve.
Currently I have tried SUM(Value) OVER(PARTITION BY ACCOUNT ORDER BY OBS)
I do not know how to reset it when reading # = 1.
I have tried:
Case
when [Reading #] = 1 THEN value
ELSE SUM(Value) OVER(PARTITION BY ACCOUNT ORDER BY OBS)
END AS [Running Total]
If I understand the question correctly and the values, stored in the Obs and [Reading #] columns are without gaps, the next approach is an option:
Table:
SELECT *
INTO Data
FROM (VALUES
(1, 1, 1, 5),
(1, 2, 2, 6),
(1, 3, 3, 5),
(1, 4, 4, 6),
(1, 5, 5, 5),
(1, 6, 6, 5),
(1, 7, 1, 5),
(1, 8, 2, 6),
(1, 9, 3, 5),
(1, 10, 4, 6),
(1, 11, 5, 5),
(1, 12, 6, 5),
(2, 1, 1, 7),
(2, 2, 2, 8),
(2, 3, 3, 9),
(2, 4, 4, 10),
(2, 5, 5, 11),
(2, 6, 6, 12),
(2, 7, 1, 7),
(2, 8, 2, 8),
(2, 9, 3, 9),
(2, 10, 4, 10),
(2, 11, 5, 11),
(2, 12, 6, 12)
) v (Account, Obs, [Reading #], [Value])
Statement:
SELECT
Account, Obs, [Reading #], [Value],
SUM([Value]) OVER (PARTITION BY Account, [Group] ORDER BY Account, Obs) AS [Ready Cicle Value]
FROM (
SELECT
*,
(Obs - [Reading #]) AS [Group]
FROM Data
) t
One additional option (as a more general approach) is to create groups when [Reading #] is equal to 1:
SELECT
Account, Obs, [Reading #], [Value],
SUM([Value]) OVER (PARTITION BY Account, [Group] ORDER BY Obs) AS [Ready Cicle Value]
FROM (
SELECT *, SUM([Change]) OVER (PARTITION BY Account ORDER BY Obs) AS [Group]
FROM (
SELECT *, CASE WHEN [Reading #] = 1 THEN 1 ELSE 0 END AS [Change]
FROM Data
) a
) b
Help us to help you. Always include a minimal set of data and the code we need, which we can copy and paste to immediately be on the same page as you without wasting time & effort that is better spent helping others. Note how you can simply copy and paste our solutions and play with them? They are complete and stand alone. That is what we are looking for from you.
You are close. The piece you are missing is that you need some way to group your readings and then you can include that in your partitioning as well.
There are any number of ways to create the new derived value for "reading_group" the following is just one way.
DECLARE #t_customer_readings TABLE
( account_number INT,
observation INT,
reading_number INT,
reading_value INT
)
INSERT INTO #t_customer_readings
VALUES (1, 1 , 1, 3),
(1, 2 , 2, 6),
(1, 3 , 3, 9),
(1, 4 , 4, 5),
(1, 5 , 5, 5),
(1, 6 , 6, 8),
(1, 7 , 1, 1),
(1, 8 , 2, 4),
(1, 9 , 3, 7),
(1, 10, 4, 0),
(1, 11, 5, 3),
(1, 12, 6, 6),
(2, 1 , 1, 9),
(2, 2 , 2, 2),
(2, 3 , 3, 5),
(2, 4 , 4, 8),
(2, 5 , 5, 1),
(2, 6 , 6, 4),
(2, 7 , 1, 7),
(2, 8 , 2, 0),
(2, 9 , 3, 3),
(2, 10, 1, 6), -- note I have split this group into 2 to show that the reading numbers do not need to be sequential.
(2, 11, 5, 9),
(2, 12, 6, 2)
SELECT r.*,
-- reading_group = CASE WHEN r.reading_number = 1 THEN observation ELSE rg.reading_group END,
ready_cycle_volume = SUM(reading_value) OVER(PARTITION BY account_number,
CASE WHEN r.reading_number = 1 THEN observation
ELSE rg.reading_group
END
ORDER BY observation)
FROM #t_customer_readings r
CROSS APPLY
(SELECT reading_group = MAX(observation) -- I picked observation but you could use whatever value you like. we are just creating something we can group on.
FROM #t_customer_readings
WHERE account_number = r.account_number
AND observation < r.observation
AND reading_number = 1) rg
I have a simple table with two fields.
1) book_id - int
2) tag_id - int
One book can have multiple tags, such as
book_id:1 - tag_id: 2, 3, 5, 9
Here is the question: how can i get a similar books from specific book? Also, they should be ordered desc by something like "likeness" count.
Example: i wanna get all book_ids with similar tags from book_id = 1 ordered by similar tags count.
Specific book: book_id: 1 - tag_id: 2, 3, 5 , 9
Result:
book_id: 54 - tag_id: 2, 3, 5, 14
book_id: 104 - tag id: 2, 3, 10
You can order the books by the number of tags they have in common with your given book:
select bt2.book_id, count(*) as tags_in_common
from book_tags bt join
book_tags bt2
on bt.tag_id = bt2.tag_id
where bt.book_id = ?
group by bt2.book_id
order by tags_in_common desc;
Let's say your table and data look like this:
create table #books_tags(
book_id int,
tag_id int
)
insert into #books_tags values(1, 2), (1, 3), (1, 5), (1, 9) -- book_id:1
insert into #books_tags values(54, 2), (54, 3), (54, 5), (54, 14) -- book_id:54
insert into #books_tags values(104, 2), (104, 3), (104, 10) -- book_id:104
insert into #books_tags values(2, 3), (2, 5), (2, 11), (2, 14) -- book_id:2
insert into #books_tags values(3, 3), (3, 9), (3, 10), (3, 11) -- book_id:3
Then your query is this:
select a.book_id,
b.book_id similar_book_id,
count(*) matching_tags,
string_agg(a.tag_id, ',') tag_ids
from #books_tags a
left join #books_tags b on b.tag_id = a.tag_id and b.book_id <> a.book_id
group by a.book_id, b.book_id
order by matching_tags desc, a.book_id
(in SQL Server 2017 or later)
I'm not sure the title describes exactly what I want to do very well, but here's a working example which I need to speed up...
/*
Aim:
2018-01-01, 1
2018-09-01, 2
2019-01-01, 1
2019-04-01, 3
2020-04-01, 1
2020-09-01, 2
2021-01-01, 1
*/
;WITH temp (ID, GroupID, Date, Value) AS (
SELECT * FROM (VALUES
(1, 1, CAST('2018-01-01' AS DATE), 1),
(2, 1, CAST('2018-04-01' AS DATE), 1),
(3, 1, CAST('2018-09-01' AS DATE), 2),
(4, 1, CAST('2019-01-01' AS DATE), 1),
(5, 1, CAST('2019-04-01' AS DATE), 3),
(6, 1, CAST('2019-09-01' AS DATE), 3),
(7, 1, CAST('2020-01-01' AS DATE), 3),
(8, 1, CAST('2020-04-01' AS DATE), 1),
(9, 1, CAST('2020-09-01' AS DATE), 2),
(10, 1, CAST('2021-01-01' AS DATE), 1),
(11, 2, CAST('2018-01-01' AS DATE), 1),
(12, 2, CAST('2018-04-01' AS DATE), 1),
(13, 2, CAST('2018-09-01' AS DATE), 2),
(14, 2, CAST('2019-01-01' AS DATE), 1),
(15, 2, CAST('2019-04-01' AS DATE), 3),
(16, 2, CAST('2019-09-01' AS DATE), 3),
(17, 2, CAST('2020-01-01' AS DATE), 3),
(18, 2, CAST('2020-04-01' AS DATE), 1),
(19, 2, CAST('2020-09-01' AS DATE), 2),
(20, 2, CAST('2021-01-01' AS DATE), 1)
) AS X(ID, GroupID, Date, Value)
)
select t1.* from temp t1
left join temp t2 on t1.GroupID = t2.GroupID
and t1.Value = t2.Value
and t2.Date < t1.Date
and not exists (
select *
from temp t3
where t1.groupID = t3.GroupID
and t3.Value != t1.Value
and t3.Date between t2.Date and t1.Date
)
where t2.ID is null
order by t1.GroupID asc, t1.Date asc
This is returning the results that I want, but when I use real data the query is really slow. It seems to be the NOT EXISTS check which is taking so long
For each GroupID I want to select the earliest Date for each Value until the next Value happens and I'm sure there must be a better/faster way of doing this, but can't think of one at the minute.
Any suggestions appreciated
This seems to be an example of a groups-and-islands problem. In this case, the best solution is to use lag():
select t.*
from (select t.*,
lag(value) over (partition by groupid order by date) as prev_value
from temp t
) t
where prev_value is null or prev_value <> value;
Here is a db<>fiddle.
I have a table that has 3 columns
Plate_Id, Prod_id and Location
You can say that the plate_id is the "header" column.
The prod_id groups the 'locations' together for a particular 'plate_id'
Given a particular set of values, I only want to pick locations that are 'COMMON' amongst prod_ids for a particular plate_id.
NOTE: My table can have multiple plate_ids
I am close, but its not perfect.
I tried to isolate the smallest group for a given plate_id and then tried to inner join it with the original list, but it fails for the scenario where I have 3 prod_ids and a location is common for even one group(i need only locations that are strictly in every prod_id)
Following is the result I desire, based on the how far I have gotten so far,
-- DESIRED RESULT:
-- plate_id location
-- 100 1
-- 100 2
-- 200 3
-- 200 4
-- 300 1
-- 300 2
-- 300 5
create table #AllTab
(
plate_id int,
prod_id int,
location int
)
insert into #AllTab
values
(100,10, 1),
(100,10, 2),
(100,10, 3),
(100,10, 4),
(100,20, 1),
(100,20, 2),
(100,20, 3),
(100,20, 4),
(100,20, 5),
(100,20, 6),
(100,20, 9),
(100,30, 1),
(100,30, 2),
(100,30, 9),
(100,40, 1),
(100,40, 2),
(100,40, 12),
(100,40, 14),
(100,40, 1),
(100,40, 2),
(100,40, 25),
(100,40, 30),
-----------------
(200,10, 1),
(200,10, 2),
(200,10, 3),
(200,10, 4),
(200,20, 1),
(200,20, 2),
(200,20, 3),
(200,20, 4),
(200,20, 5),
(200,20, 6),
(200,20, 7),
(200,30, 3),
(200,30, 4),
(200,30, 9),
-----------------
(300,10, 1),
(300,10, 2),
(300,10, 3),
(300,10, 5),
(300,20, 1),
(300,20, 2),
(300,20, 3),
(300,20, 4),
(300,20, 5),
(300,20, 6),
(300,20, 7),
(300,20, 9),
(300,30, 1),
(300,30, 2),
(300,30, 5)
-- The #SubTab table isolates the smallest group from the above table
-- for a particular plate_id
create table #SubTab
(
plate_id int,
prod_id int,
location int
)
insert into #SubTab
values
(100,30, 1),
(100,30, 2),
(100,30, 9),
------------
(200,30, 3),
(200,30, 4),
(200,30, 9),
------------
(300,30, 1),
(300,30, 2),
(300,30, 5)
select distinct pr.plate_id, pr.prod_id, pr.location from #SubTab pr
inner join #AllTab pl on pr.plate_id = pl.plate_id
and pr.location = pl.location
where pr.Prod_Id <> pl.prod_id
group by pr.plate_id, pr.prod_id, pr.location
This query returns the locations that are in all the products for a given plate:
select plate_id, location
from #alltab a
group by plate_id, location
having count(distinct prod_id) = (select count(distinct prod_id) from #alltab a2 where a2.plate_id = a.plate_id);
This assumes no duplicates in the table -- a reasonable assumption given your data.
Here is a rextester.
try this:
;with cte1
AS
(
Select Plate_Id,Count(DISTINCT prod_id) as ProdCount
From #AllTab
Group by Plate_Id
)
,cte2
AS
(
Select Plate_Id,Location,Count(Location) As LocCount
from #AllTab
Group by Plate_Id,Location
)
SELECT t1.plate_id ,t2.location
FROM cte1 t1 JOIN cte2 t2
ON t1.Plate_Id =t2.Plate_Id
Where LocCount>=ProdCount
I have a solution, it might be a tad lengthy, but works,
SELECT
SubGroupCounts.plate_id,
LocationSubGroupCounts.location
FROM
(-- Number of sub-grouping relative to main grouping
SELECT
plate_id,
count(distinct prod_id) as num
FROM
AllTab
GROUP BY
plate_id) SubGroupCounts
INNER JOIN
(-- Count the number of sub-groups each location appears in
SELECT
plate_id,
Location,
COUNT(distinct prod_id) AS num
FROM
AllTab
GROUP BY
Location, plate_id) LocationSubGroupCounts ON LocationSubGroupCounts.plate_id = SubGroupCounts.plate_id
AND LocationSubGroupCounts.num = SubGroupCounts.num