Using one YAML definition for the same column as it moves through models - dbt

Question

I have some model YAML:
version: 2
models:
- name: my_model
columns:
- name: foo
description: My bestest column
...If I make other models which inherit from this one, is there any way to refer back to this column definition when documentation is generated, or do I need to copy-paste the column definition for each model in which the column appears?
In other words, is there a way of defining a column only once to make edits and updates easier.
Cheers,
Graham

Answer 1

I think there are two ways to do it.
1. using macro
create a file that contains all those reusable descriptions. you could even use params to customize the description.
e.g. doc_library.sql
{% macro column_bestest_doc(col_name) %}
My bestest column {{ col_name }}
{% endmacro %}
then use it in dbt_project.yml
version: 2
models:
- name: my_model
columns:
- name: foo_1
description: {{column_bestest_doc(foo_1)}}
- name: my_model_another
columns:
- name: foo_2
description: {{column_bestest_doc(foo_2)}}
2. using YAML anchor
you could do YAML anchors in dbt_project.yml as in any other yml files.
version: 2
models:
- name: my_model
columns:
- name: &foo
description: My bestest column
- name: my_model_another
columns:
- name: *foo
ref:
https://support.atlassian.com/bitbucket-cloud/docs/yaml-anchors/
https://medium.com/#kinghuang/docker-compose-anchors-aliases-extensions-a1e4105d70bd

Answer 2

We have been thinking about this problem extensively as well... our current solution is to modify the generate_model_yaml macro:
{% macro generate_model_yaml(model_name) %}
{% set model_yaml=[] %}
{% set existing_descriptions = fetch_existing_descriptions(model_name) %}
-- # TO DO: pass model to fetch()
-- if column not blank on current model, use description in returned dict
-- otherwise, use global
-- also extract tests on column anywhere in global scope
{% do model_yaml.append('version: 2') %}
{% do model_yaml.append('') %}
{% do model_yaml.append('models:') %}
{% do model_yaml.append(' - name: ' ~ model_name | lower) %}
{% do model_yaml.append(' description: ""') %}
{% do model_yaml.append(' columns:') %}
{% set relation=ref(model_name) %}
{%- set columns = adapter.get_columns_in_relation(relation) -%}
{% for column in columns %}
{%- set column = column.name | lower -%}
{%- set col_description = existing_descriptions.get(column, '') %}
{% do model_yaml.append(' - name: ' ~ column ) %}
{% do model_yaml.append(' description: "' ~ col_description ~ '"') %}
{% do model_yaml.append('') %}
{% endfor %}
{% if execute %}
{% set joined = model_yaml | join ('\n') %}
{{ log(joined, info=True) }}
{% do return(joined) %}
{% endif %}
{% endmacro %}
And then get the first description found that matches the column, with fetch_existing_descriptions():
{% macro fetch_existing_descriptions(current_model) %}
{% set description_dict = {} %}
{% set current_model_dict = {} %}
{% for node in graph.nodes.values() | selectattr("resource_type", "equalto", "model") %}
{% for col_dict in node.columns.values() %}
{% if node.name == current_model %}
-- Add current model description to seperate dict to overwrite with later
{% set col_description = {col_dict.name: col_dict.description} %}
{% do current_model_dict.update(col_description) %}
{% elif description_dict.get(col_dict.name, '') == '' %}
{% set col_description = {col_dict.name: col_dict.description} %}
{% do description_dict.update(col_description) %}
{% endif %}
{% endfor %}
{% endfor %}
-- Overwrite description_dict with current descriptions
{% do description_dict.update(current_model_dict) %}
{% if var('DEBUG', False) %}
{{ log(tojson(description_dict), info=True) }}
{% else %}
{{ return(description_dict) }}
{% endif %}
{% endmacro %}
Finally, we use a bash script with the path to the model to write/overwrite a yaml file using the modified generate_model_yaml above:
#!/bin/bash
# Generates documentation for dbt models.
# Usage:
# Run from within the gold folder.
# Run with no args for all models. Provide optional relative model path to generate docs for just one model:
# Eg. `$ bash ./scripts/generate_docs.sh [./models/path/to_your_model.sql]`
if [[ -n $1 ]]; then # Build array of one model if optional arg provided
yml="${1%.sql}.yml" # Create model yml filename
touch "$yml" && rm "$yml" || exit 1 # Ensure filepath works by testing creation
array=("$yml")
else # Create array of yml
array=()
while IFS= read -r -d $'\0'; do
if [[ ${REPLY} != *"src"* ]]; then # Only proceed for model yml files (don't contain "src")
if [[ -n $1 ]]; then
# Include only the model yml of the optional arg
if [[ $(basename $yml) == $(basename $REPLY) ]]; then
array+=("$REPLY")
fi
else
array+=("$REPLY")
fi
fi
done < <(find "./models/" -name "*.yml" -print0)
fi
# Create copy model yml with prescribed yml containing docs standard.
for i in "${array[#]}"
do
model=$(basename $i | sed -E 's|[.]yml||g')
generated_yml=$(dbt run-operation generate_model_yaml --args "model_name: $model" | sed '1d')
echo "$generated_yml" > "${i}_copy" # Create non-yml copy file to allow script to complete
done
# Once all copies are created, replace originals
for i in "${array[#]}"
do
cat "${i}_copy" > $i
rm "${i}_copy"
done