erroneous function call with case when inside UDF - sql

Question

I'm noticing some really weird behavior when using a CASE WHEN statement inside a UDF in one of my queries in google bigquery. The result is really bizarre, so either I'm missing something really obvious, or there's some weird behavior in the query execution.
(side note: if there's a more efficient way to implement the query logic below I'm all ears, my query takes forever)
I'm processing some log lines, where each line contains a data: string and topics: array<string> field that are used for decoding. Each type of log line will have different length of topics, and requires different decoding logic. I use a CASE WHEN inside a UDF to switch to different methods of decoding. I originally got a strange error of indexing too far into an array. This would either mean non-conformant data, or the wrong decoder getting called at some point. I verified that all the data conformed to the spec, so it must be the latter.
I've narrowed it down to an erroneous / extraneous decoder being executed inside my CASE WHEN for the wrong type.
The weirdest thing is that when I go and insert fixed value instead of my decoding functions, the return value of the CASE WHEN doesn't indicate that it's an erroneous match. Somehow the first function is getting called when I use functions, but when debugging I get the value from the proper value from the second WHEN.
I pulled the logic out of the udf, and implemented it with an if(..) instead of CASE WHEN and everything decodes fine. I'm wondering what's going on here, if it's a bug in bigquery, or something weird happens when using UDFs.
Here's a stripped down version of the query
-- helper function to normalize different payloads into a flattened struct
create temporary function wrap_struct(payload array<struct<name string, value string>>) as (
(select as struct
decode_field_type1(field1) as field1,
decode_field_type1(field2) as field2,
decode_field_type2(field3) as field3,
-- a bunch more fields
from (select * from
(select p.name, p.value
from unnest(payload) as p) pivot(string_agg(value) for name in (
'field1', 'field2', 'field3', --a bunch more fields
)
)
))
);
-- this topic uses the data and topics in the decoding, and has a topics array of length 4
-- this gets called from the switch with a payload from topics2, which has a shorter topics array of length 1, causing a failure
create temporary function decode_topic1(data string, topics array<string>) as
(
wrap_struct([
struct("field1" as name, substring(topics[offset(1)], 3) as value),
struct("field2" as name, substring(topics[offset(2)], 3) as value),
struct("field3" as name, substring(topics[offset(3)], 3) as value),
struct("field4" as name, substring(data, 3, 64) as value)
])
);
--this uses only the data_payload, and has a topics array of length 1
create temporary function decode_topic2(data string, topics array<string>) as
(
wrap_struct([
struct("field1" as name, substring(data, 3, 64) as value),
struct("field5" as name, substring(data, 67, 64) as value),
struct("field6" as name, substring(data, 131, 64) as value)
])
);
create temporary function decode_event_data(data string, topics array<string>) as
(
-- first element of topics denotes the type of event
case
-- somehow the function decode_topic1 gets called when topics[0] == topic2
-- HOWEVER, when i replaced the functions with a fixed value to debug
-- i get the expected results, indicating a proper match.
-- this is not unique these topics
-- it happens with other combinations also.
when topics[offset(0)] = 'topic1' then decode_topic1(data, topics)
when topics[offset(0)] = 'topic2' then decode_topic2(data, topics)
-- a bunch more topics
else wrap_struct([])
end
);
select
id, data, topics,
decode_event_data(data, topics) as decoded_payload
from (select * from mytable
where
topics[offset(0)] = 'topic1'
or topics[offset(0)] = 'topic2'
when i change the base query to:
select
id, data, topics, decode_topic2(data, topics)
from (select * from mytable
where
topics[offset(0)] = 'topic2'
it decodes fine.
What's up with the CASE WHEN?
edit: Here's a query on a public dataset that can generate the problem:
concat('0x', substring(raw, 25, 40))
);
create temporary function decode_amount(raw string) as (
concat('0x', raw)
);
create temporary function wrap_struct(payload array<struct<name string, value string>>) as (
(select as struct
decode_address(sender) as reserve,
decode_address(`to`) as `to`,
decode_amount(amount1) as amount1,
decode_amount(amount2) as amount2,
from (select * from
(select p.name, p.value
from unnest(payload) as p) pivot(string_agg(value) for name in (
'sender', 'to', 'amount1', 'amount2'
)
)
))
);
create temporary function decode_mint(data_payload string, topics array<string>) as
(
wrap_struct([
struct("sender" as name, substring(topics[offset(1)], 3) as value),
struct("amount1" as name, substring(data_payload, 3, 64) as value),
struct("amount2" as name, substring(data_payload, 67, 64) as value)
])
);
create temporary function decode_burn(data_payload string, topics array<string>) as
(
wrap_struct([
struct("sender" as name, substring(topics[offset(1)], 3) as value),
struct("amount1" as name, substring(data_payload, 3, 64) as value),
struct("amount2" as name, substring(data_payload, 67, 64) as value),
struct("to" as name, substring(topics[offset(2)], 67, 64) as value)
])
);
select
*,
case
when topics[offset(0)] = '0x4c209b5fc8ad50758f13e2e1088ba56a560dff690a1c6fef26394f4c03821c4f' then decode_mint(data, topics)
when topics[offset(0)] = '0xd78ad95fa46c994b6551d0da85fc275fe613ce37657fb8d5e3d130840159d822' then decode_burn(data, topics)
end as decoded_payload
from `public-data-finance.crypto_ethereum_kovan.logs`
where
array_length(topics) > 0
and (
(array_length(topics) = 2 and topics[offset(0)] = '0x4c209b5fc8ad50758f13e2e1088ba56a560dff690a1c6fef26394f4c03821c4f')
or (array_length(topics) = 3 and topics[offset(0)] = '0xd78ad95fa46c994b6551d0da85fc275fe613ce37657fb8d5e3d130840159d822')
)

Answer 1

The offset(nn) is killing your function, change it for safe_offset(nn) and it will solve the problem. Also, this to field is generally empty in decode_burn, or null when decode_mint, so, at least with this data its just causing problem.
The following works and solve this issue:
create temporary function decode_burn(data_payload string, topics array<string>) as
(
wrap_struct([
struct("sender" as name, substring(topics[offset(1)], 3) as value),
struct("amount1" as name, substring(data_payload, 3, 64) as value),
struct("amount2" as name, substring(data_payload, 67, 64) as value),
struct("to" as name, substring(topics[SAFE_OFFSET(2)], 67, 64) as value)
])
);
Edit1:
I have analyzed in detail the data sent to the functions and steps and you a right about the filters and how it is working. Seams you have reached a specific corner case (or bug ?):
As far as a could understand by the processing steps, BQ is optimizing your functions, as they are very similar, except by the fact that there is an additional field in one of them.
So, BQ engine is using the same optimized function for both data, which is causing the exception when the input is the data of rows with topics[OFFSET(0)] = '0x4c209b5fc8ad50758f13e2e1088ba56a560dff690a1c6fef26394f4c03821c4f'
As this is happening:
using the safe_offset still a good call;
Or, Create only one function and use the conditional on it, in this case the query will be processed correctly:
CREATE TEMPORARY FUNCTION decode(data_payload string, topics ARRAY<string>) AS (
wrap_struct([
STRUCT("sender" AS name,
SUBSTRING(topics[OFFSET(1)], 3) AS value),
struct("amount1" as name, substring(data_payload, 3, 64) as value),
struct("amount2" as name, substring(data_payload, 67, 64) as value),
if(topics[OFFSET(0)] = '0xd78ad95fa46c994b6551d0da85fc275fe613ce37657fb8d5e3d130840159d822',
STRUCT("to" AS name, topics[OFFSET(2)] AS value),null )])
);
select *, decode(DATA,topics) ...
In parallel, you can open a case on issue tracker