Create a table with multiple nesting levels from flattened structure - google-bigquery

Question

A bigquery challenge :
input
I have a table with incoming product batches that go into the factory and multiple sensors along the way measure different defects of different parts of the individual products. We are reading out the data from the devices in a flat structure.
The data is written to a incoming table.
Batch_id|Sensor_id|Product_part_id|defect_id|Count_defects|Event_Date
1.......|.1.......|1..............|2........|.5...........|.2018-7-1
1.......|.2.......|1..............|2........|.6...........|.2018-7-1
1.......|.2.......|2..............|3........|.7...........|.2018-7-1
1.......|.3.......|2..............|3........|.8...........|.2018-7-1
1.......|.3.......|2..............|4........|.9...........|.2018-7-1
1.......|.3.......|3..............|5........|.10...........|.2018-7-1
We can do de-duplication on theses tables as the same sensor might spit out the same data multiple times (by mistake or on purpose when the count-defects is updated) based on the last [updated_time]
the problem: transform into multiple nested repeated structs
Now I'm trying to materialize the raw input into fact tables partitioned by Event_Date but for max performance and cheapest storage, I want to achieve the following structure :
Batch_id|Sensor_id|Product_part_id|defect_id|Count_defects|Event_Date
1.......|.1.......|1..............|2........|.5...........|.2018-7-1
........|.2.......|1..............|2........|.6...........|.2018-7-1
........|.........|2..............|3........|.7...........|.2018-7-1
........|.3.......|2..............|3........|.8...........|.2018-7-1
........|.........|...............|4........|.9...........|.2018-7-1
........|.........|3..............|5........|.10..........|.2018-7-1
I cannot do multiple nested ARRAY() calls, it is not allowed and also badly performing as this would take the same base table as input multiple time.
Looking for suggestions on how to tackle this.
Thanks!

Answer 1

I'm using sequencial application of array_agg() + GROUP BY for that, starting with the innermost array. After the first iteration I put the query into a WITH and start over with creating the next array again using array_agg() + GROUP BY.
Performance-wise this approach has the same constraints all GROUP BY queries have - you should avoid skewed group sizes if you can - otherwise it will just take longer because BigQuery has to re-plan resources in the background when it realizes a group takes to much memory. But you can optimize using the query execution plan.
For your example table my result query looks like this:
WITH t AS (
SELECT 1 as batch_id, 1 as sensor_id, 1 as product_part_id, 2 as defect_id, 5 as count_defects, '2018-7-1' as event_date
UNION ALL SELECT 1 as batch_id, 2 as sensor_id, 1 as product_part_id, 2 as defect_id, 6 as count_defects, '2018-7-1' as event_date
UNION ALL SELECT 1 as batch_id, 2 as sensor_id, 2 as product_part_id, 3 as defect_id, 7 as count_defects, '2018-7-1' as event_date
UNION ALL SELECT 1 as batch_id, 3 as sensor_id, 2 as product_part_id, 3 as defect_id, 8 as count_defects, '2018-7-1' as event_date
UNION ALL SELECT 1 as batch_id, 3 as sensor_id, 2 as product_part_id, 4 as defect_id, 9 as count_defects, '2018-7-1' as event_date
UNION ALL SELECT 1 as batch_id, 3 as sensor_id, 3 as product_part_id, 5 as defect_id, 10 as count_defects, '2018-7-1' as event_date
),
defect_nesting as (
SELECT
batch_id,
sensor_id,
product_part_id,
array_agg(STRUCT(defect_id, count_defects, event_date) ORDER BY defect_id) defectInfo
FROM t
GROUP BY 1, 2, 3
),
product_nesting as (
SELECT
batch_id,
sensor_id,
array_agg(STRUCT(product_part_id, defectInfo) ORDER BY product_part_id) productInfo
FROM defect_nesting
GROUP BY 1,2
)
SELECT
batch_id,
array_agg(STRUCT(sensor_id, productInfo) ORDER BY sensor_id) sensorInfo
FROM product_nesting
GROUP BY 1
The resulting JSON:
[
{
"batch_id": "1",
"sensorInfo": [
{
"sensor_id": "1",
"productInfo": [
{
"product_part_id": "1",
"defectInfo": [
{
"defect_id": "2",
"count_defects": "5",
"event_date": "2018-7-1"
}
]
}
]
},
{
"sensor_id": "2",
"productInfo": [
{
"product_part_id": "1",
"defectInfo": [
{
"defect_id": "2",
"count_defects": "6",
"event_date": "2018-7-1"
}
]
},
{
"product_part_id": "2",
"defectInfo": [
{
"defect_id": "3",
"count_defects": "7",
"event_date": "2018-7-1"
}
]
}
]
},
{
"sensor_id": "3",
"productInfo": [
{
"product_part_id": "2",
"defectInfo": [
{
"defect_id": "3",
"count_defects": "8",
"event_date": "2018-7-1"
},
{
"defect_id": "4",
"count_defects": "9",
"event_date": "2018-7-1"
}
]
},
{
"product_part_id": "3",
"defectInfo": [
{
"defect_id": "5",
"count_defects": "10",
"event_date": "2018-7-1"
}
]
}
]
}
]
}
]
Hope that helps!