SQL, BigQuery - completing missing values with other part of rows - sql

Question

I'm using Firebase data exported to BigQuery (data contains events data coming from mobile application). I've made an update to the application and new parameter is being reported. Unfortunately, not all users have the latest version of app. This is why I have rows with that parameter as well as rows without it.
In event_params I have something like:
| No | contentId | contentName |
|----|-----------|---------------------|
| 1 | abc | (parameter missing) |
| 2 | abc | Name of ABC |
| 3 | cde | Name of CDE |
| 4 | efg | Name of EFG |
| 5 | abc | (parameter missing) |
| 6 | cde | Name of CDE |
Now, when I query that table and I specify (using UNNEST) that I need contentName parameter, I don't get rows where that parameter is missing.
I have query:
SELECT
ep.value.string_value as ContentID,
ep2.value.string_value as ContentName,
COUNT(1) as `Count`
FROM
`mydataset.mytable.events_*`,
UNNEST(event_params) as ep,
UNNEST(event_params) as ep2
WHERE
event_name="my_event_name" AND
ep.key="contentID" AND
ep2.key="contentName"
GROUP BY 1,2
and I get:
| No | contentId | contentName | Count |
|----|-----------|-------------|-------|
| 1 | abc | Name of ABC | 1 |
| 2 | cde | Name of CDE | 2 |
| 3 | efg | Name of EFG | 1 |
However, I would like to get:
| No | contentId | contentName | Count |
|----|-----------|-------------|-------|
| 1 | abc | Name of ABC | 3 |
| 2 | cde | Name of CDE | 2 |
| 3 | efg | Name of EFG | 1 |
I want to complete somehow rows with missing contentName parameters using values from other rows with the same contentId (we can assume that each contentId has the same, constant contentName)
How can I achieve it? I thougt about SELF JOIN, but it's rather not recommended by BigQuery.

Answer 1

The solution provided by Gordon can be slightly modified in order to achieve what you intend:
SELECT contentId.value.string_value as ContentID,
MAX(contentName.value.string_value) as ContentName,
COUNT(1) as `Count`
FROM `mydataset.mytable.events_*` e LEFT JOIN
UNNEST(e.event_params) as contentId
ON contentId.key = 'contentID' LEFT JOIN
UNNEST(e.event_params) contentName
ON contentName.key = 'contentName'
WHERE e.event_name = 'my_event_name'
GROUP BY 1;
Note that I am grouping only by the ContentID and I am aggregating the ContentNames using MAX, which ignores null values.
I have recreated your example table and it works as expected.

Answer 2

You can update the table so that you fill the nulls and then make your query
[1]
UPDATE `your_project.your_dataset.your_table` t_incomplete
SET t_incomplete.contentName = t_complete.contentName
FROM `your_project.your_dataset.your_table` t_complete
WHERE t_incomplete.contentId = t_complete.contentId
AND t_complete.contentName IS NOT NULL
I am not sure how will this work with nested tables but you can always
UPDATE UNNESTING
UPDATE WITH QUERY [1]
UPDATE NESTING
You can picture the idea behind with this sample CREATE TABLE
CREATE TABLE `your_project.your_dataset.sample_table`
(
id INT64,
nullable STRING
);
INSERT INTO `your_project.your_dataset.sample_table`
VALUES (1, 'foo');
INSERT INTO `your_project.your_dataset.sample_table`
VALUES (1, null);
INSERT INTO `your_project.your_dataset.sample_table`
VALUES (2, 'lel');
INSERT INTO `your_project.your_dataset.sample_table`
VALUES (1, null);
INSERT INTO `your_project.your_dataset.sample_table`
VALUES (2, null);
and QUERY[2]
UPDATE `your_project.your_dataset.sample_table` t_incomplete
SET t_incomplete.nullable = t_complete.nullable
FROM `wave27-sellbytel-aalbesa.trial_dataset.with_and_update` t_complete
WHERE t_incomplete.id = t_complete.id
AND t_complete.nullable IS NOT NULL
This way you actually give the corresponding value to the cell and you can run your query without worries. I hope this works!

Answer 3

Do you just need an OR condition?
WHERE event_name = 'my_event_name' AND
ep.key = 'contentID' AND
(ep2.key = 'contentName' OR ep2.key IS NULL)
EDIT:
I think you need LEFT JOINs:
SELECT contentId.value.string_value as ContentID,
contentName.value.string_value as ContentName,
COUNT(1) as `Count`
FROM `mydataset.mytable.events_*` e LEFT JOIN
UNNEST(e.event_params) as contentId
ON contentId.key = 'contentID' LEFT JOIN
UNNEST(e.event_params) contentName
ON contentName.key = 'contentName'
WHERE e.event_name = 'my_event_name'
GROUP BY 1, 2;
Note: This should preserve the counts you want but might result in extra rows in the result set.