I'm trying to have a structure to run a dbt project, where I have multiple entities (bank, names, cars). I'm going to have exactly the same code for them all.
Based on that, I'm trying to have several folder with the same code, where I can define inside the dbt_project.yamlfile. The idea is something like this:
vars:
db_name: 'db_official'
staging:
bank: 'variable_bank'
car: 'variable_car'
name: 'variable_name'
The variable "db_name" works. So, my the two problems I'm having are:
How to have this structure inside the the yaml file?
How to reference this structure inside each file?
(extra) Any other ideas how to handle this?
Thanks!
vars are basically globals. They can be scoped to your whole project, or to a package within your project, but not more specifically than that (they share a flat namespace). See the docs.
I would pull out the common code into a macro, then call that macro from each model file, passing in the unique values as string literals in the model file:
-- models/staging/bank.sql
{{ my_model_template('variable_bank') }}
-- models/staging/car.sql
{{ my_model_template('variable_car') }}
Related
I have a generic test and need it to be always saved under a particular name for the given table it is running on, e.g. on table report_revenue the generated generic test name will always be diff_check_report_revenue. Right now the default dbt naming behavior for generic tests is kinda messy (it sets the test name based on the test config, which is a great idea for most cases, but unfortunately not for my particular one).
According to the docs it looks like the [alias]https://docs.getdbt.com/reference/resource-configs/alias is exactly what I need. However, I also need to set the name dynamically based on the table that is tested. I know it can be set in the yml config by setting the field alias, but I hope there might be a more elegant solution.
When I try the following:
{{
config({
"severity": 'warn',
"tags": ["diff_check"],
"alias": 'diff_check_' + {{ model | replace("XXXXXXX") | trim }}
})
}}
It just doesn't work and dbt completely ignores the alias property. model is the relation on which the test is running. It's probably just my own wrong syntax, but I'm stuck and humbly asking for advice. Thanks a lot in advance!
The docs are super confusing on test config, since they group together generic tests and singular tests, and the behavior is different.
You can use a config() block inside the definition for a generic test to configure it, and some keys (e.g., severity) work fine, but alias is not one of them.
I think alias is meant for singular tests only. To give generic tests a unique identifier (only possible since v1.1), you are supposed to use the name property (not config). Docs. Does this make sense? No. Does it make it easy for you to do what you want to do? Also no.
I'll point out that the default naming convention for a generic test includes the name of the test followed by the name of the model, but assuming that isn't good enough, your only option will be to add a name property to every test, where you define the test in the properties (fka schema.yml) file. And it looks like the name property doesn't jinja-template its value (so you can't use jinja to populate the test name). So you'll have to do a lot of this:
models:
- name: my_model
tests:
- diff_check:
name: diff_check_my_model
You could fork dbt-core. The relevant code is here.
I have model properties file defined in .yml and query defined in .sql. There are many models and there are some properties which are shared across models. So for instance I have used meta tag in .yml model property file and need to define a value let say a: . Now this is used across multiple model meta's. How can I define and use t hem in the dbt model properties file?
Variables not only works in .sql but also works in .yml in the model properties file. Here one sample of it
https://github.com/dbt-labs/dbt-core/pull/2015
works
We started using the seeds functionality in DBT, we put a single CSV file in the data folder, and configured the seed to use a custom schema named util --- and it works (i.e. - it creates a table in the correct schema).
yaml looks like this:
seeds:
my_project_name:
+schema: util
However, when we refer to it using ref in our models:
{{ref('my_seed')}}
it looks for it in our default target schema for the environment (public ), instead of the custom one we defined --- how come?
I should mention that we also used the macro trick mentioned here:
https://docs.getdbt.com/docs/building-a-dbt-project/building-models/using-custom-schemas
Update:
Adding the macro code we used (as the file get_custom_schema.sql):
{% macro generate_schema_name(custom_schema_name, node) -%}
{{ generate_schema_name_for_env(custom_schema_name, node) }}
{%- endmacro %}
Not sure the + is needed in front of schema, based on the code example here.
Another option would be to define the schema for the specific seed table:
seeds:
my_project_name:
my_seed:
schema: util
After some time I spent staring at the jbake code, I figured out that if I declare my own property in jbake.properties :
...
foo=bar
...
I can reuse that in files that go through a template engine by referencing it as ${config.foo}. I'd like to have this substitution working also on the content lvl, i.e. for files written in asciidoc, living inside the content directory.
Is there any non-trivial way to achieve it? How can I make the templating engine to proccess the result of asciidoc parses engine, or make it running it before the asciidoctor?
I found the answer myself.
To use the property substitution in asciidoc files, add following to the jbake.properties:
...
asciidoctor.attributes.export=true
foo=world
...
and reference the variable in aFile.adoc this way:
Hello {foo}!
I can't figure out how to get variables into Sphinx documents via a template. I am certainly making an elementary mistake, but while there is lots of documentation for using Jinja/Flask templates for web service and some documentation for Sphinx using it, I am having trouble doing the following. Maybe it's not possible and I'm barking up the wrong tree, but then this is fairly different from how variables work in general in web (as opposed to doc) templates?
I am working within a much larger project. Suppose in my project's conf.py I make a variable, say
LANGS = ['en', 'de', 'cn']
I know that this works because if I do the docbuild (we have a custom docbuild but I don't think it does anything really crazy other than a customized logger and eating a bunch of 'chatter') with the following line in conf.py
print len(LANGS)
it shows up during the docbuild.
But now of course I want to access this variable in my template. As far as I can tell, we override index.rst with templates/index.html, which inherits from the basic layout.html for Sphinx. And if I do
<p>We have {{ LANGS|len }} languages</p>
I get
We have 0 languages
Now, this is weird, because sometimes I can cause an error in the build by referring to variables not defined (though not consistently), so that somehow it 'knows' that the variable is defined but thinks it has length zero. Or does a "null" variable have length zero automatically?
How do I get this variable defined - or is it not possible?
What I want to do is then do something for each language in the list (make an outside link, in particular), but I figure there is no point in trying {% for %}/{% endfor %} or whatever if I can't get this working. Maybe Sphinx implements only a subset of Jinja?
Anyway, please help!
There are at least two ways to pass variables to a template:
Via html_context:
A dictionary of values to pass into the template engine’s context for all pages. Single values can also be put in this dictionary using the -A command-line option of sphinx-build.
Example:
# conf.py:
html_context = {'num_langs': len(LANGS)}
<!-- template: -->
<p>We have {{ num_langs }} languages</p>
Via the html_theme_options. This requires adding an option to theme.conf (you can create a theme by inheriting from a standard one):
[options]
num_langs = 1
Then you can set num_langs in conf.py via html_theme_options:
html_theme_options = {'num_langs': len(LANGS)}
and use it in a template:
<p>We have {{ theme_num_langs }} languages</p>