How to access BigQuery table metadata in DBT using jinja? - sql

Question

I'd like to access the last modified time column from the metadata of a BigQuery table that acts as a source. I want to create a generic test that checks if the last modified date of the source table is equal to today.
In BigQuery you can access this data in this way:
SELECT
last_modified_time
FROM `project.dataset.__TABLES__`
WHERE table_id = 'table_id'
My goal is to make the project.dataset dynamic depending on the model this test is applied to. Similarly, I'd like for table_id to be dynamic.
Given that DBT mentions on their documentation that the dataset of BigQuery is similar in definition to 'schema', I tried this but it didn't work.
{% test last_modified_time(schema, model) %}
SELECT
last_modified_time
FROM `{{ database }}.{{ schema }}.__TABLES__`
WHERE table_id = {{ model }}
{% endtest %}
What this does is it renders the project name for both database and schema.
Also, model will (of course) render the project.dataset.table_id path while I only need the table_id.
I'm fairly new to DBT but I couldn't find anything that resembles what I'm looking for.

Answer 1

I tinkered with your solution for a little bit and this works flawlessly.
Thank you so much!
{% test last_modified_time(model) %}
WITH t AS (
SELECT DATE(TIMESTAMP_MILLIS(last_modified_time)) AS lmt
FROM `{{ model.database }}.{{ model.schema }}.__TABLES__`
WHERE table_id = '{{ model.identifier }}'
)
SELECT
lmt
FROM t
WHERE lmt < CURRENT_DATE()
{% endtest %}

Answer 2

There are a few changes you need to make:
Your generic test is accepting an argument named schema, which dbt won't provide when you execute the test. The test should only accept model, and then you'll want to config your yaml file so the test is on the model (not on a column):
models:
- name: my_model
tests:
- last_modified_time
The model argument is a Relation, and you can use that to grab the database/project, schema/dataset, and identifier (the materialized name) of the model.
Tests fail if they return any records. So your test will always fail. You need to compare the last_modified_time to the current date, and only return records that are older than the current date.
Putting that all together:
{% test last_modified_time(model) %}
with t as (
SELECT last_modified_time
FROM `{{ model.database }}.{{ model.schema }}.__TABLES__`
WHERE table_id = '{{ model.identifier }}'
)
select *
from t
where t.last_modified_time < current_date()
{% endtest %}