BigQuery how to use MERGE to load array columns - google-bigquery

Question

I have a test table that I am trying to load from GCS storage:
CREATE OR REPLACE TABLE ta_producer_conformed.test
(
id NUMERIC,
array_string ARRAY<STRING>,
array_struct_string_string ARRAY<STRUCT<key STRING, value STRING>>,
array_struct_string_numeric ARRAY<STRUCT<key STRING, value NUMERIC>>,
array_struct_string_int64 ARRAY<STRUCT<key STRING, value INT64>>
)
I have defined an external storage table as:
{
"autodetect": true,
"csvOptions": {
"encoding": "UTF-8",
"quote": "\"",
"fieldDelimiter": "\t"
},
"sourceFormat": "CSV",
"sourceUris": [
"gs://my_bucket/test/input/*.tsv"
]
}
In it I am using JSON to hold the ARRAY types:
"id" "array_string" "struct_string_string" "struct_string_numberic" "struct_string_int64"
1 ["one", "two", "three"] [{"key":"one", "value":"1"},{"key":"two", "value":"2"},{"key":"three", "value":"3"}] [{"key":"one", "value":1.1},{"key":"two", "value":2.2}] [{"key":"one", "value":11},{"key":"two", "value":22}]
2 ["four", "five", "six"] [{"key":"four", "value":"4"},{"key":"five", "value":"5"},{"key":"six", "value":"6"}] [{"key":"three", "value":3.3},{"key":"four", "value":4.4}] [{"key":"three", "value":33},{"key":"four", "value":44}]
I then want to using a MERGE to upsert the data in the target table. When I run this:
CREATE TEMPORARY FUNCTION ARRAY_OF(json STRING)
RETURNS ARRAY<STRING>
LANGUAGE js AS """
let parsed = JSON.parse(json);
return parsed;
""";
CREATE TEMPORARY FUNCTION ARRAY_STRUCT_STRING_STRING_OF(json STRING)
RETURNS ARRAY<STRUCT<key STRING, value STRING>>
LANGUAGE js AS """
let parsed = JSON.parse(json);
return parsed;
""";
CREATE TEMPORARY FUNCTION ARRAY_STRUCT_STRING_NUMERIC_OF(json STRING)
RETURNS ARRAY<STRUCT<key STRING, value NUMERIC>>
LANGUAGE js AS """
let parsed = JSON.parse(json);
return parsed;
""";
CREATE TEMPORARY FUNCTION ARRAY_STRUCT_STRING_INT64_OF(json STRING)
RETURNS ARRAY<STRUCT<key STRING, value INT64>>
LANGUAGE js AS """
let parsed = JSON.parse(json);
return parsed;
""";
MERGE ta_producer_conformed.test T
USING ta_producer_raw.test_raw S
ON
S.id = T.id
WHEN NOT MATCHED THEN
INSERT (id, array_string, array_struct_string_string, array_struct_string_numeric, array_struct_string_int64)
VALUES (
id,
ARRAY_OF(array_string),
ARRAY_STRUCT_STRING_STRING_OF(struct_string_string),
ARRAY_STRUCT_STRING_NUMERIC_OF(struct_string_numberic),
ARRAY_STRUCT_STRING_INT64_OF(struct_string_int64)
)
WHEN MATCHED THEN UPDATE SET
T.id = S.id,
T.array_string = ARRAY_OF(S.array_string),
T.struct_string_string = ARRAY_STRUCT_STRING_STRING_OF(S.struct_string_string),
T.ARRAY_STRUCT_STRING_NUMERIC_OF(S.struct_string_numberic),
T.ARRAY_STRUCT_STRING_INT64_OF(S.struct_string_int64)
I get this error:
Error in query string: Error processing job 'xxxx-10843454-datamesh-
dev:bqjob_r4c426875_00000173fcfd2294_1': Syntax error: Expected "." or "=" or
"[" but got "(" at [1:1312]
If I delete the whole last section for WHEN MATCHED such that it only INSERTS the temporary functions work fine. So the problem appears to be that in final THEN UPDATE SET section I cannot use the temporary functions.
How can I get data types such as ARRAY<STRING> and ARRAY<STRUCT<STRING,STRING>> to load from an external bucket ideally using a single MERGE statement?
Update: I tried to use a Common Table Expression to pre-process the data using:
WITH cteConvertJason AS (
SELECT
id,
ARRAY_OF(array_string) AS array_string,
ARRAY_STRUCT_STRING_STRING_OF(struct_string_string) AS struct_string_string,
ARRAY_STRUCT_STRING_NUMERIC_OF(struct_string_numberic) AS struct_string_numberic,
ARRAY_STRUCT_STRING_INT64_OF(struct_string_int64) AS struct_string_int64
FROM
ta_producer_raw.test_raw
)
MERGE ta_producer_conformed.test T
USING cteConvertJason S
...
That gave an error so it looks like you combine WITH and MERGE.
Update: We were trying out TSV for legacy reasons. It is a far better idea to use NEWLINE_DELIMITED_JSON as the format such that you do not need to explicitly parse the nested or repeated columns.

Answer 1

It turns out that the MERGE target USING source can use a query as the source. That query can run the temporary functions to pre-process the source data. Then the rest of the MERGE statement can be vanilla and works:
CREATE TEMPORARY FUNCTION ARRAY_OF(json STRING)
RETURNS ARRAY<STRING>
LANGUAGE js AS """
let parsed = JSON.parse(json);
return parsed;
""";
CREATE TEMPORARY FUNCTION ARRAY_STRUCT_STRING_STRING_OF(json STRING)
RETURNS ARRAY<STRUCT<key STRING, value STRING>>
LANGUAGE js AS """
let parsed = JSON.parse(json);
return parsed;
""";
CREATE TEMPORARY FUNCTION ARRAY_STRUCT_STRING_NUMERIC_OF(json STRING)
RETURNS ARRAY<STRUCT<key STRING, value NUMERIC>>
LANGUAGE js AS """
let parsed = JSON.parse(json);
return parsed;
""";
CREATE TEMPORARY FUNCTION ARRAY_STRUCT_STRING_INT64_OF(json STRING)
RETURNS ARRAY<STRUCT<key STRING, value INT64>>
LANGUAGE js AS """
let parsed = JSON.parse(json);
return parsed;
""";
MERGE ta_producer_conformed.test T
USING
(
SELECT
id,
ARRAY_OF(array_string) AS array_string,
ARRAY_STRUCT_STRING_STRING_OF(struct_string_string) AS struct_string_string,
ARRAY_STRUCT_STRING_NUMERIC_OF(struct_string_numberic) AS struct_string_numberic,
ARRAY_STRUCT_STRING_INT64_OF(struct_string_int64) AS struct_string_int64
FROM
ta_producer_raw.test_raw
)
S
ON
S.id = T.id
WHEN NOT MATCHED THEN
INSERT (id, array_string, array_struct_string_string, array_struct_string_numeric, array_struct_string_int64)
VALUES (
id,
array_string,
struct_string_string,
struct_string_numberic,
struct_string_int64
)
WHEN MATCHED THEN UPDATE SET
T.array_string = S.array_string,
T.array_struct_string_string = S.struct_string_string,
T.array_struct_string_numeric = S.struct_string_numberic,
T.array_struct_string_int64 = S.struct_string_int64