Why the DBT macro executes before the pre_hook statement in an incremental model? - dbt

Question

Does anyone know why the pre_hook statement gets executed after the macro append_placekeys is called?
I am creating a Temporary table in the pre_hook configuration which is going to be called by the macro. However, DBT logs showed the macro executing before the creation of the temp table.
{{
config(
materialized='incremental',
unique_key='DX_ID',
pre_hook=[
"""
CREATE OR REPLACE TEMP TABLE DXH_STORE_PLCK_IN
AS
SELECT DISTINCT
s.dx_id AS msa_id,
s.nm AS store_name,
s.city AS city,
s.state AS state,
s.address AS street_addr,
s.zip5 AS zipcode,
s.country AS COUNTRY_CODE,
s.latitude AS LATITUDE,
s.longitude AS LONGITUDE,
COALESCE(p.last_updt_dt, TO_DATE('2023-01-01')) AS last_updt_dt
FROM {{ ref('dx_store_attr') }} s
{%- if is_incremental() %}
FULL JOIN {{ this }} p USING (dx_id)
WHERE
(
p.placekey IS NULL
OR
DATEDIFF(day, p.last_updt_dt::DATE, CURRENT_TIMESTAMP()::DATE) > 90
)
ORDER BY last_updt_dt ASC, s.dx_id ASC
LIMIT 3000
{%- else %}
LIMIT 50000
{%- endif -%}
;
"""
],
tags=["dx", "refresh:daily"]
)
}}
--load dx_store_plck
{{
append_placekeys(
tbl_in='DXH_STORE_PLCK_IN',
tbl_out='DXH_STORE_PLCK_TEMP'
)
}}
SELECT
t.msa_id AS dx_id,
t.placekey AS placekey,
t.error AS error,
CURRENT_TIMESTAMP() AS last_updt_dt
FROM
{{ database }}.{{ schema }}.DXH_STORE_PLCK_TEMP AS t
DBT Logs:
[0m20:44:55.939031 [debug] [Thread-1 ]: SQL status: SUCCESS 1 in 0.19 seconds
[0m20:44:55.940927 [debug] [Thread-1 ]: Using snowflake connection "model.XXX.dx_store_plck"
[0m20:44:55.942096 [debug] [Thread-1 ]: On model.XXX.dx_store_plck: /* {"app": "dbt", "dbt_version": "1.3.0", "profile_name": "XXX", "target_name": "dev", "node_id": "model.XXX.dx_store_plck"} */
CALL DB.SCHEMA_DEV.APPEND_PLACEKEYS(
'DXH_STORE_PLCK_IN',
(
SELECT object_construct(MAPPING.*)
FROM (
SELECT
'msa_id' AS PRIMARY_KEY,
'store_name' AS LOCATION_NAME,
'city' AS CITY,
'state' AS REGION,
'street_addr' AS STREET_ADDRESS,
'zipcode' AS POSTAL_CODE,
'LATITUDE' AS LATITUDE,
'LONGITUDE' AS LONGITUDE,
'false' AS strict_address_match,
'false' AS strict_name_match
) AS MAPPING
),
'DXH_STORE_PLCK_TEMP', 'temp', 'get_placekeys_v', 1000
);
[0m20:44:56.882161 [debug] [Thread-1 ]: SQL status: SUCCESS 1 in 0.94 seconds
[0m20:44:56.888032 [debug] [Thread-1 ]: Writing injected SQL for node "model.XXX.dx_store_plck"
[0m20:44:56.891930 [debug] [Thread-1 ]: finished collecting timing info
[0m20:44:56.893285 [debug] [Thread-1 ]: Began executing node model.XXX.dx_store_plck
[0m20:44:56.897175 [debug] [Thread-1 ]: Using snowflake connection "model.XXX.dx_store_plck"
[0m20:44:56.898207 [debug] [Thread-1 ]: On model.sdna_us_project.dx_store_plck: /* {"app": "dbt", "dbt_version": "1.3.0", "profile_name": "XXX", "target_name": "dev", "node_id": "model.XXX.dx_store_plck"} */
select current_warehouse() as warehouse
[0m20:44:57.061101 [debug] [Thread-1 ]: SQL status: SUCCESS 1 in 0.16 seconds
[0m20:44:57.063617 [debug] [Thread-1 ]: Using snowflake connection "model.XXX.dx_store_plck"
[0m20:44:57.064833 [debug] [Thread-1 ]: On model.XXX.dx_store_plck: /* {"app": "dbt", "dbt_version": "1.3.0", "profile_name": "XXX", "target_name": "dev", "node_id": "model.XXX.dx_store_plck"} */
use warehouse XXX_M;
[0m20:44:57.326229 [debug] [Thread-1 ]: SQL status: SUCCESS 1 in 0.26 seconds
[0m20:44:57.356896 [debug] [Thread-1 ]: Using snowflake connection "model.XXX.dx_store_plck"
[0m20:44:57.358488 [debug] [Thread-1 ]: On model.XXX.dx_store_plck: /* {"app": "dbt", "dbt_version": "1.3.0", "profile_name": "sdna_us", "target_name": "dev", "node_id": "model.XXX.dx_store_plck"} */
USE ROLE FUNCTIONAL_ROLE_XXX;
[0m20:44:57.508780 [debug] [Thread-1 ]: SQL status: SUCCESS 1 in 0.15 seconds
[0m20:44:57.510018 [debug] [Thread-1 ]: Using snowflake connection "model.XXX.dx_store_plck"
[0m20:44:57.510897 [debug] [Thread-1 ]: On model.XXX.dx_store_plck: /* {"app": "dbt", "dbt_version": "1.3.0", "profile_name": "sdna_us", "target_name": "dev", "node_id": "model.XXX.dx_store_plck"} */
alter session set query_tag ='DB.SCHEMA_DEV.DX_STORE_PLCK';
[0m20:44:57.648295 [debug] [Thread-1 ]: SQL status: SUCCESS 1 in 0.14 seconds
[0m20:44:57.652786 [debug] [Thread-1 ]: Using snowflake connection "model.XXX.dx_store_plck"
[0m20:44:57.653900 [debug] [Thread-1 ]: On model.sdna_us_project.dx_store_plck: /* {"app": "dbt", "dbt_version": "1.3.0", "profile_name": "XXX", "target_name": "dev", "node_id": "model.XXX.dx_store_plck"} */
CREATE OR REPLACE TEMP TABLE DXH_STORE_PLCK_IN
AS
SELECT DISTINCT
s.dx_id AS msa_id,
s.nm AS store_name,
s.city AS city,
s.state AS state,
s.address AS street_addr,
s.zip5 AS zipcode,
s.country AS COUNTRY_CODE,
s.latitude AS LATITUDE,
s.longitude AS LONGITUDE,
COALESCE(p.last_updt_dt, TO_DATE('2023-01-01')) AS last_updt_dt
FROM DB.SCHEMA_DEV.store_attr s
FULL JOIN DB.SCHEMA_DEV.dx_store_plck p USING (dx_id)
WHERE
(
p.placekey IS NULL
OR
DATEDIFF(day, p.last_updt_dt::DATE, CURRENT_TIMESTAMP()::DATE) > 90
)
ORDER BY last_updt_dt ASC, s.dx_id ASC
LIMIT 3000;
[0m20:44:59.140263 [debug] [Thread-1 ]: SQL status: SUCCESS 1 in 1.49 seconds

Answer 1

dbt takes two passes over your model file; first to parse it and build the dag, and then to actually execute it.
In order to properly template (parse) your model, it has to execute any macros contained in it. (Among other reasons, ref is also a macro, and is critical for this step). This means that your append_placekeys macro gets executed twice; first, when the model is parsed, and then, when it is executed.
You can prevent this by using the special jinja variable called execute. execute is False during the first parsing pass, but True during the actual model execution. I would probably edit the macro itself to add an {% if execute %} block around the database call, but you could also just gate the whole macro call in the model file:
{{
config(
...
)
}}
--load dx_store_plck
{% if execute %}
{{
append_placekeys(
tbl_in='DXH_STORE_PLCK_IN',
tbl_out='DXH_STORE_PLCK_TEMP'
)
}}
{% endif %}
SELECT
t.msa_id AS dx_id,
t.placekey AS placekey,
t.error AS error,
CURRENT_TIMESTAMP() AS last_updt_dt
FROM
{{ database }}.{{ schema }}.DXH_STORE_PLCK_TEMP AS t
All that aside, this code contains a bunch of dbt antipatterns. You should really never have to write any DDL when using dbt. Both your pre-hook and the macro are relatively fragile operations, and you should re-write your model logic so you don't need them. Finally, models should pretty much always select from {{ ref('a_model') }} or from {{ source('a_source', 'a_source_tbl') }}, and not directly from a database relation, as you're doing here.

Answer 2

This was the resulting final model:
{{
config(
materialized='incremental',
unique_key='DX_ID',
pre_hook=[
"""
CREATE OR REPLACE TABLE {{ database }}.{{ schema }}.dxh_store_plck_in
AS
SELECT DISTINCT
s.dx_id AS msa_id,
s.btg_store_nm AS store_name,
s.btg_city AS city,
s.btg_state AS state,
s.btg_address AS street_addr,
s.btg_zip5 AS zipcode,
'US' AS COUNTRY_CODE,
s.latitude AS LATITUDE,
s.longitude AS LONGITUDE,
COALESCE(p.last_updt_dt, TO_DATE('2023-01-01')) AS last_updt_dt
FROM {{ ref('dx_store_attr') }} s
FULL JOIN {{ this }} p USING (dx_id)
{%- if is_incremental() %}
WHERE
(
p.placekey IS NULL
OR
DATEDIFF(day, p.last_updt_dt::DATE, CURRENT_TIMESTAMP()::DATE) > 90
)
ORDER BY last_updt_dt ASC, s.dx_id ASC
LIMIT 5000
{%- endif -%}
;
""",
"""
{{
append_placekeys(
tbl_in=database|as_text ~'.' ~ schema|as_text ~'.dxh_store_plck_in',
tbl_out=database|as_text ~'.' ~ schema|as_text ~'.dxh_store_plck_temp'
)
}}
"""
],
tags=["dx", "refresh:daily"]
)
}}
--load dx_store_plck
SELECT
t.msa_id AS dx_id,
t.placekey AS placekey,
t.error AS error,
CURRENT_TIMESTAMP() AS last_updt_dt
FROM {{ database }}.{{ schema }}.dxh_store_plck_temp AS t