Creating a new view in BigQuery, with case statements - take 2 - google-bigquery

Question

I had asked this question before, but my original code snippet was wrong and I didn't see it until an answer came in.
The original question:
I am trying to create a new view in BigQuery using some of the Google hosted data. The data is for traffic collisions in New York.
For each unique day in the dataset, I want to find the borough and sum up some fields (people injured, killed, etc.)
Now, the dataset does have a borough field, but this is incomplete, and what I have seen is that there are also latitude and longitude fields. However these are also not complete. So I see 3 scenarios.
The borough is set, use that
No borough, but there is lat and long, so use those in a sub query.
There is no lat, long or borough, so just enter an "unknown" here to find the borough from lat and long, there is another public dataset, and I used this with a lat and long to check
SELECT UPPER(tz_loc.borough) FROM `bigquery-public-data.new_york_taxi_trips.taxi_zone_geom` tz_loc
WHERE (ST_DWithin(tz_loc.zone_geom, ST_GeogPoint(-73.94398, 40.680088),0))
Originally, I tried this:
CREATE VIEW `your-project-id.your_dataset_id.collisions_data_bourgh` AS
SELECT CAST(timestamp as DATE) as collision_date,
COUNT(CAST(timestamp as DATE)) as NUM_COLLISIONS,
CASE
WHEN ds.borough IS NOT NULL THEN CAST(borough as STRING) -- when the borough is set
WHEN ((ds.latitude IS NOT NULL or ds.longitude IS NOT NULL) AND ds.borough IS NULL) THEN (SELECT CAST(UPPER(tz_loc.borough)as STRING) FROM `bigquery-public-data.new_york_taxi_trips.taxi_zone_geom` tz_loc WHERE (ST_DWithin(tz_loc.zone_geom, ST_GeogPoint(CAST(ds.longitude AS FLOAT64), CAST(ds.latitude AS FLOAT64)),0))) -- when the borough is null and either lat or long is not null
WHEN (ds.latitude IS NULL OR ds.longitude IS NULL OR ds.borough IS NULL) THEN "Unknown"
END AS NEIGHBORHOOD,
SUM(CAST(number_of_cyclist_killed as INT64)) as CYCLISTS_KILLED,
SUM(CAST(number_of_cyclist_injured as INT64)) as CYCLISTS_INJURED,
SUM(CAST(number_of_motorist_killed as INT64)) as MOTORISTS_KILLED,
SUM(CAST(number_of_motorist_injured as INT64)) as MOTORISTS_INJURED,
SUM(CAST(number_of_pedestrians_killed as INT64)) as PEDS_KILLED,
SUM(CAST(number_of_pedestrians_injured as INT64)) as PEDS_INJURED,
SUM(CAST(number_of_persons_killed as INT64)) as PERSONS_KILLED,
SUM(CAST(number_of_persons_injured as INT64)) as PERSONS_INJURED,
FROM `bigquery-public-data.new_york_mv_collisions.nypd_mv_collisions` ds
GROUP BY collision_date, NEIGHBORHOOD;
but was corrected by another user to the code below:
CREATE VIEW `uhi-assignment-1.assignment.collisions_data_bourgh22` AS
SELECT
CAST(timestamp AS DATE) AS collision_date,
COUNT(CAST(timestamp AS DATE)) AS NUM_COLLISIONS,
CASE
WHEN ds.borough IS NOT NULL THEN CONCAT('AA ', CAST(borough AS STRING)) -- when the borough is set
WHEN ds.borough IS NULL AND ds.location IS NOT NULL
THEN (
SELECT UPPER(tz_loc.borough) as STRING)
FROM bigquery-public-data.new_york_taxi_trips.taxi_zone_geom tz_loc
WHERE
ST_DWithin(tz_loc.zone_geom,
ST_GeogPoint(CAST(ds.longitude AS FLOAT64),
CAST(ds.latitude AS FLOAT64)),0)
AND tz_loc.borough = ds.borough
)
WHEN (ds.latitude IS NULL AND ds.longitude IS NULL AND ds.borough IS NULL) THEN "CC Unknown"
END
AS NEIGHBORHOOD,
SUM(CAST(number_of_cyclist_killed AS INT64)) AS CYCLISTS_KILLED,
SUM(CAST(number_of_cyclist_injured AS INT64)) AS CYCLISTS_INJURED,
SUM(CAST(number_of_motorist_killed AS INT64)) AS MOTORISTS_KILLED,
SUM(CAST(number_of_motorist_injured AS INT64)) AS MOTORISTS_INJURED,
SUM(CAST(number_of_pedestrians_killed AS INT64)) AS PEDS_KILLED,
SUM(CAST(number_of_pedestrians_injured AS INT64)) AS PEDS_INJURED,
SUM(CAST(number_of_persons_killed AS INT64)) AS PERSONS_KILLED,
SUM(CAST(number_of_persons_injured AS INT64)) AS PERSONS_INJURED,
FROM
bigquery-public-data.new_york_mv_collisions.nypd_mv_collisions ds
GROUP BY
collision_date,
NEIGHBORHOOD
But this doesn't do as I needed. After some playing, I have found that I had no records being inserted using the seperate query.
From looking at the query, I think the line:
AND tz_loc.borough = ds.borough
is the issue as the ds.borough will be null (which is reason it goes down to check the taxi geo query.) But this line is needed to prevent a LEFT JOIN issue.
Anyone have any ideas?

Answer 1

The right way to do things in SQL is to think about sets, not per-row operations. The CASE in the suggested expression is rather anti-pattern.
Thinking about SET operations, we need to join rows with empty borough to zones, to get borough info this way, then use this data in original table:
join the collisions table rows where we don't have borough to the zones table - this will drop some rows, but that's OK. This only computes the join when needed, where borough is not filled out.
left join the collision table with the result of step 1 - this adds another column with neighborhood computed from zones table, without dropping anything.
use COALESCE(borough, tz_borough, 'UNKNOWN') to choose the first available data: first try original borough field, if not available - borough from taxi zones, otherwise UNKNOWN.
finally, group and count.
I dropped extra aggregations, but overall idea should be clear:
WITH taxi_zones_join AS (
SELECT ds.unique_key, tz_borough
FROM bigquery-public-data.new_york_mv_collisions.nypd_mv_collisions ds
JOIN (
SELECT borough AS tz_borough, zone_geom AS tz_geom
FROM bigquery-public-data.new_york_taxi_trips.taxi_zone_geom tz_loc
)
ON ST_INTERSECTS(tz_geom, ST_GeogPoint(CAST(longitude AS FLOAT64), CAST(latitude AS FLOAT64)))
WHERE ds.borough IS NULL
),
collisions_plus AS (
SELECT
ds.* EXCEPT(borough),
UPPER(COALESCE(borough, tz_borough, 'UNKNOWN')) AS NEIGHBORHOOD
FROM bigquery-public-data.new_york_mv_collisions.nypd_mv_collisions ds
LEFT JOIN taxi_zones_join bj
USING(unique_key)
)
SELECT
CAST(timestamp AS DATE) AS collision_date,
COUNT(*) AS NUM_COLLISIONS,
NEIGHBORHOOD
FROM collisions_plus
GROUP BY collision_date, NEIGHBORHOOD