Complex Clustering of Rows in Large Table - SQL in Google BigQuery - sql

Question

I try to find common clusters in a large table. I have my data in Google BigQuery.
My data consists of many transactions tx from different user groups. User groups can have multiple ids and I try to cluster all ids to the respective user group by analyzing their transactions.
I identified four rules that identify ids from the same user group. In the brackets I named this cluster rule so that you can map the rule to the SQL below:
all ids that belong to the same tx where is_type_0 = TRUE belong to the same user group (="cluster_is_type_0")
all ids that belong to the same tx where is_type_1 = FALSE belong to the same user group (="cluster_is_type_1")
all ids that belong to the same tx where is_type_1 = TRUE that did not exist in the row numbers before (row_nr), belong to the same user group (="cluster_id_occurence")
all ids with the same id belong to the same user group (="cluster_id")
Here is some example data:
row_nr
tx
id
is_type_1
is_type_0
expected_cluster
1
z0
a1
true
true
1
2
z1
b1
true
true
2
3
z1
b2
true
true
2
4
z2
c1
true
true
3
5
z2
c2
true
true
3
6
z3
d1
true
true
4
7
z
a1
false
false
1
8
z
b1
true
false
2
9
z
a2
true
false
1
10
y
b1
false
false
2
11
y
b2
false
false
2
12
y
a2
true
false
1
13
x
c1
false
false
3
14
x
c2
false
false
3
15
x
b1
true
false
2
16
x
c3
true
false
3
17
w
a2
false
false
1
18
w
c1
true
false
3
19
w
a3
true
false
1
20
v
b1
false
false
2
21
v
b2
false
false
2
22
v
a2
true
false
1
This is what I already tried:
WITH data AS (
SELECT *
FROM UNNEST([
STRUCT
(1 as row_nr, 'z0' as tx, 'a1' as id, TRUE as is_type_1, TRUE as is_type_0, 1 as expected_cluster),
(2, 'z1', 'b1', TRUE, TRUE, 2),
(3, 'z1', 'b2', TRUE, TRUE, 2),
(4, 'z2', 'c1', TRUE, TRUE, 3),
(5, 'z2', 'c2', TRUE, TRUE, 3),
(6, 'z3', 'd1', TRUE, TRUE, 4),
(7, 'z', 'a1', FALSE, FALSE, 1),
(8, 'z', 'b1', TRUE, FALSE, 2),
(9, 'z', 'a2', TRUE, FALSE, 1),
(10, 'y', 'b1', FALSE, FALSE, 2),
(11, 'y', 'b2', FALSE, FALSE, 2),
(12, 'y', 'a2', TRUE, FALSE, 1),
(13, 'x', 'c1', FALSE, FALSE, 3),
(14, 'x', 'c2', FALSE, FALSE, 3),
(15, 'x', 'b1', TRUE, FALSE, 2),
(16, 'x', 'c3', TRUE, FALSE, 3),
(17, 'w', 'a2', FALSE, FALSE, 1),
(18, 'w', 'c1', TRUE, FALSE, 3),
(19, 'w', 'a3', TRUE, FALSE, 1),
(20, 'v', 'b1', FALSE, FALSE, 2),
(21, 'v', 'b2', FALSE, FALSE, 2),
(22, 'v', 'a2', TRUE, FALSE, 1)
])
)
, first_cluster as (
SELECT *
, ROW_NUMBER() OVER (PARTITION BY id ORDER BY row_nr) as id_occurence
, CASE WHEN NOT is_type_1 THEN DENSE_RANK() OVER (ORDER BY tx) END AS cluster_is_type_1
, CASE WHEN is_type_0 THEN DENSE_RANK() OVER (ORDER BY tx) END AS cluster_is_type_0
, DENSE_RANK() OVER (ORDER BY id) AS cluster_id
FROM data
ORDER BY row_nr
)
, second_cluster AS (
SELECT *
, CASE WHEN id_occurence = 1 THEN MIN(cluster_is_type_1) OVER (PARTITION BY tx) END AS cluster_id_occurence
FROM first_cluster
ORDER BY row_nr
)
, third_cluster AS (
SELECT *
, COALESCE(cluster_is_type_1, cluster_id_occurence, cluster_is_type_0, cluster_id) AS combined_cluster
FROM second_cluster
ORDER BY row_nr
)
SELECT *
-- , ARRAY_AGG(combined_cluster) OVER (PARTITION BY id) AS combined_cluster_agg
, MIN(combined_cluster) OVER (PARTITION BY id) AS result_cluster
FROM third_cluster
ORDER BY id
But the result is not as expected. id a1, a2 and a3 are not considered to be the same cluster, and also COALESCE(cluster_is_type_1, cluster_id_occurence, cluster_is_type_0, cluster_id) AS combined_cluster can lead to some unwanted behavior as the defined clusters always start at 1 with the dense_rank and when you combine them like so it might be that ids end up in the same cluster that do not belong together.
I appreciate every help!