Joining two tables on two criteria "cannot partition on repeated field" - sql

Question

I'm using BigQuery for this.
I have a subquery that pulls data from a table that has an account_id, product, date, and product_spend fields. This subquery calculates the total lifetime spend for each product for each 'account_id' by adding up each of the line items.
SELECT account_id,
product,
SUM(product_spend)/1000000 lifetime_product_spend
FROM usage
GROUP BY 1, 2
The result looks like this:
table: lifetime
account_id product lifetime_product_spend
===========================================================
A product1 50
A product2 20
B product2 100
B product3 150
C product3 500
I'm trying to preserve the values and join them with a larger query:
SELECT account_id,
product,
month,
SUM(spend)
FROM data_source
WHERE month >= DATE_ADD(today ,-5,"MONTH")
GROUP BY 1, 2, 3
This query has a table that looks like this:
table: monthly
account_id product month spend
=================================================================
A product1 1 10
A product1 2 20
A product1 3 30
A product2 1 5
A product2 2 15
B product2 2 100
B product3 2 100
B product3 3 50
C product3 1 100
C product3 2 400
I'm not using an aggregate to calculate lifetime_product_spend on the second table. Due to the sheer amount of data, I'm only able to include the last 6 months data. That's why I'm calculating the lifetime spend in a different table and joining them.
My current query is failing:
SELECT d.account_id,
d.product,
d.month,
sum(d.spend),
u.lifetime_product_spend
FROM data_source d
LEFT JOIN (SELECT account_id,
product,
SUM(product_spend)/1000000 lifetime_product_spend
FROM usage
GROUP BY account_id, product) u
ON d.account_id = u.account_id
WHERE d.month >= DATE_ADD(today ,-5,"MONTH")
GROUP BY d.account_id, d.product, d.month, u.lifetime_product_spend
because it doesn't seem to have assigned the lifetime figures to each product as in the Lifetime table. That's because I'm only joining on account_id. See below for the bad output. I've truncated this table because it basically added the # of outputs I have for lifetime_product_spend (5) and put one for each month, product, and company...because it's ignoring the 'product' assignment for these values:
table: monthly
account_id product month spend lifetime_product_spend
=====================================================================================
A product1 1 10 50
A product1 1 10 20
A product1 1 10 100
A product1 1 10 150
A product1 1 10 500
A product1 2 20 50
A product1 2 20 20
A product1 2 20 100
A product1 2 20 150
A product1 2 20 500
Is there a way for me to join on both of them? I've tried doing a JOIN ON x = x AND y = y:
SELECT d.account_id,
d.product,
d.month,
sum(d.spend),
u.lifetime_product_spend
FROM data_source d
LEFT JOIN (SELECT account_id,
product,
SUM(product_spend)/1000000 lifetime_product_spend
FROM usage
GROUP BY account_id, product) u
ON (d.account_id = u.account_id AND d.product = u.product)
WHERE d.month >= DATE_ADD(today ,-5,"MONTH")
GROUP BY d.account_id, d.product, d.month, u.lifetime_product_spend
but it gives me this error : "Execution Failed
Error: Cannot partition on repeated field d.product".
I want my final table to look like this:
table: monthly
account_id product month spend lifetime_product_spend
=====================================================================================
A product1 1 10 50
A product1 2 20 50
A product1 3 30 50
A product2 1 5 20
A product2 2 15 20
B product2 2 100 100
B product3 2 100 150
B product3 3 50 150
C product3 1 100 500
C product3 2 400 500
I think I need "FLATTEN" somewhere, but I can't seem to get it in the right place. Thanks for reading.

Answer 1

SELECT d.account_id,
d.product,
d.month,
sum(d.spend),
u.lifetime_product_spend
FROM FLATTEN(data_source, product) d
LEFT JOIN (SELECT account_id,
product,
SUM(product_spend)/1000000 lifetime_product_spend
FROM usage
GROUP BY account_id, product) u
ON (d.account_id = u.account_id AND d.product = u.product)
WHERE d.month >= DATE_ADD(today ,-5,"MONTH")
GROUP BY d.account_id, d.product, d.month, u.lifetime_product_spend
The above works with the original data source flattened around the repeated field d.product. Thanks for the comments and help.

Answer 2

Write "Select .... from usage" as a sub-query and apply an INNER JOIN or LEFT JOIN on data_source table.
SELECT d.account_id,
d.product,
d.month,
sum(d.spend),
u.lifetime_product_spend
from data_source d
left join (SELECT account_id,
product,
SUM(product_spend)/1000000 lifetime_product_spend
FROM usage
GROUP BY account_id, product) u
on(d.account_id=u.account_id and d.product=u.product)
WHERE d.month >= DATE_ADD(today ,-5,"MONTH")
GROUP BY d.account_id, d.product, d.month, u.lifetime_product_spend