Processing AI Platform Batch Results BigQuery - sql

Question

The output from AI Platform for tabular dataset looks something like this:
or
{
"classes": ["a","b","c"],
"scores": [0.9,0.1,0.0]
}
There are two arrays within in a record field. predicted_label.classes is the label, and predicted_label.scores is the score produced by AI Platform.
I would like to select the class based on the highest score. i.e in the above example I would like to have an output like row=0, class="a", score=0.9
UNNEST does not immediately solve my issue from my understanding, as it requires the input to be an array. I believe if the ouput was a repeated RECORD it would be easier.
What SQL query will enable me to extract the right label from the AI Platform batch results?

Answer 1

Try this:
with testdata as (
select struct(["a", "b", "c"] as classes, [0.9, 0.1, 0.0] as scores) as predicted_label
)
select (
select struct(offset, class, score)
from unnest(predicted_label.classes) as class with offset
join unnest(predicted_label.scores) as score with offset
using (offset)
order by score desc
limit 1
) as highest
from testdata

Answer 2

You should design your prediction list so that each label and score is represented as a key-value pair.
That BigQuery table looks like this array.
prediction RECORD REPEATED
prediction.label STRING REQUIRED
prediction.score FLOAT REQUIRED
Why?
This a correct representation of your real world situation.
You need no further verification that both list keep the elements in the correct pairing order (on write and on read).
With two loose lists you create a pitfall that will hounds you.
SQL example
with this_model as (
select [
STRUCT ('a' as label, 0.9 as score)
, STRUCT ('b' as label, 0.1 as score)
, STRUCT ('c' as label, 0.0 as score)
] as prediction
)
select pair.label, pair.score
from this_model, UNNEST(prediction) pair
order by pair.score desc
limit 1;