We started using the seeds functionality in DBT, we put a single CSV file in the data folder, and configured the seed to use a custom schema named util --- and it works (i.e. - it creates a table in the correct schema).
yaml looks like this:
seeds:
my_project_name:
+schema: util
However, when we refer to it using ref in our models:
{{ref('my_seed')}}
it looks for it in our default target schema for the environment (public ), instead of the custom one we defined --- how come?
I should mention that we also used the macro trick mentioned here:
https://docs.getdbt.com/docs/building-a-dbt-project/building-models/using-custom-schemas
Update:
Adding the macro code we used (as the file get_custom_schema.sql):
{% macro generate_schema_name(custom_schema_name, node) -%}
{{ generate_schema_name_for_env(custom_schema_name, node) }}
{%- endmacro %}
Not sure the + is needed in front of schema, based on the code example here.
Another option would be to define the schema for the specific seed table:
seeds:
my_project_name:
my_seed:
schema: util
Related
I'm looking to self-reference the table I'm working on within a model file in the config block to alias the table name. Right now, I'm naming dynamically naming the alias using a Python file for loop but would prefer if the model file recognized and designated the table name in itself.
{{ config(
alias=model.table ### this.name? not sure what syntax to use here ###
) }}
select *
from {{ source('aceso', 'aceso_accountlookup') }}
{% if is_incremental() %}
where _FIVETRAN_SYNCED > (select max(_FIVETRAN_SYNCED) from {{ this }} )
{% endif %}
Currently I have no idea the format of syntax required to get dbt to understand what I want it to do
dbt currently has a strong one-database-object-per-model association, so what you seem to be trying to describe (based on reading your answers to #tconbeer's questions in the comments) isn't really possible without something hacky like you're already doing.
There is a GitHub Discussion around making it possible for dbt to generate multiple objects from a single model here that you may wish to contribute to.
Let's say I have a project in dbt. When I run it, it generates a bunch of tables. Now I want to change the underlying SQL and see what happens to these tables, how they differ from before the change. So I want to be able to compare all the tables generated by the old version to all the tables generated by the new version. Ideally I would like the method to work for any number of versions, not just two. Basically the question is how to put each version in its own namespace.
Method 1: run the new version of the project in a new schema, so I can compare old.foo to new.foo. But getting another schema from the database admins is a painful process.
Method 2: Have both versions in the same schema, but add a prefix, like new_ to the table name for the new version. So, old version has table foo, new version has new_foo, and I compare foo to new_foo.
Is there any convenient way to do Method 2 in dbt? Is there a third method I should be considering? Or am I doing something fundamentally wrong to even find myself in this situation? It seems like it shouldn't be such a rare problem but I can't find any information about what I can do in this situation.
One possible way to do this is to override the default alias macro. The macro gets called even if there is no alias defined in the configuration, so you can use that as an opportunity to rename the target table.
The version below will prefix any model that does not have an alias set in the configuration with name of the target profile when the run is not against the prod profile.
{% macro generate_alias_name(custom_alias_name=none, node=none) -%}
{%- if target.name != 'prod' and custom_alias_name is none -%}
{{ target.name ~ "_" ~ node.name }}
{%- elif target.name == 'prod' -%}
{{ node.name }}
{%- else -%}
{{ custom_alias_name | trim }}
{%- endif -%}
{%- endmacro %}
If your model is foo.sql and you run this against a profile named "prod", the table will be foo. If you run it against "dev", it will be dev_foo. If your model has an alias, then the alias name will take precedence regardless of the target profile. You can decide if you want to include the special behavior if the model has an alias name. Just modify the else block.
I'm trying to have a structure to run a dbt project, where I have multiple entities (bank, names, cars). I'm going to have exactly the same code for them all.
Based on that, I'm trying to have several folder with the same code, where I can define inside the dbt_project.yamlfile. The idea is something like this:
vars:
db_name: 'db_official'
staging:
bank: 'variable_bank'
car: 'variable_car'
name: 'variable_name'
The variable "db_name" works. So, my the two problems I'm having are:
How to have this structure inside the the yaml file?
How to reference this structure inside each file?
(extra) Any other ideas how to handle this?
Thanks!
vars are basically globals. They can be scoped to your whole project, or to a package within your project, but not more specifically than that (they share a flat namespace). See the docs.
I would pull out the common code into a macro, then call that macro from each model file, passing in the unique values as string literals in the model file:
-- models/staging/bank.sql
{{ my_model_template('variable_bank') }}
-- models/staging/car.sql
{{ my_model_template('variable_car') }}
I am developing a Shopify Theme. The structure I have now is (excerpt):
Snippets
custom-message-snippet
Sections
custom-message-section
welcome-page-a-section
welcome-page-b-section
Templates
welcome-page-a-template
welcome-page-b-template
...
Custom message snippet uses settings that are in the custom-message-section, that is:
a) message
b) header text
I'd like for users to be able to add custom-message-snippet to welcome-page-a and welcome-page-b in a way that settings for both are different.
I can not render section 'custom-message-section" inside welcome pages because it is not possible (and a workaround is nasty).
There are fifty welcome pages. Every welcome page is totally different.
My question is:
How to allow users to use custom-message-snippet in all welcome pages without copying and pasting custom-message-page setting schema to each and every welcome page?
combine custom-message-section to custom-message-snippet
combine welcome-page-\w+-template to welcome-page-template, you only need to have one template and schema set, the user will control the sections in schema selection.
in template, use schema to select the page-content; this, defined by a bunch of if-else /switch statement.
{% section "welcome-page-custom-message-section" %}
{% section "welcome-page-content" %}
in welcome-page-custom-message-section
{% render "custom-message-snippet" with "string from welcome-page-section schema " %}
create checkbox select schema for custom message to appear or not.
Short: You can't.
Long: Using storefront 2.0, you end you having blocks, which can be added manually to any template.
It is possible if you have a little knowledge of Node.js and you're using the Shopify CLI.
Check the solution here: Shopify Section / Block Schema In A Separate File. It is possible
I can't figure out how to get variables into Sphinx documents via a template. I am certainly making an elementary mistake, but while there is lots of documentation for using Jinja/Flask templates for web service and some documentation for Sphinx using it, I am having trouble doing the following. Maybe it's not possible and I'm barking up the wrong tree, but then this is fairly different from how variables work in general in web (as opposed to doc) templates?
I am working within a much larger project. Suppose in my project's conf.py I make a variable, say
LANGS = ['en', 'de', 'cn']
I know that this works because if I do the docbuild (we have a custom docbuild but I don't think it does anything really crazy other than a customized logger and eating a bunch of 'chatter') with the following line in conf.py
print len(LANGS)
it shows up during the docbuild.
But now of course I want to access this variable in my template. As far as I can tell, we override index.rst with templates/index.html, which inherits from the basic layout.html for Sphinx. And if I do
<p>We have {{ LANGS|len }} languages</p>
I get
We have 0 languages
Now, this is weird, because sometimes I can cause an error in the build by referring to variables not defined (though not consistently), so that somehow it 'knows' that the variable is defined but thinks it has length zero. Or does a "null" variable have length zero automatically?
How do I get this variable defined - or is it not possible?
What I want to do is then do something for each language in the list (make an outside link, in particular), but I figure there is no point in trying {% for %}/{% endfor %} or whatever if I can't get this working. Maybe Sphinx implements only a subset of Jinja?
Anyway, please help!
There are at least two ways to pass variables to a template:
Via html_context:
A dictionary of values to pass into the template engine’s context for all pages. Single values can also be put in this dictionary using the -A command-line option of sphinx-build.
Example:
# conf.py:
html_context = {'num_langs': len(LANGS)}
<!-- template: -->
<p>We have {{ num_langs }} languages</p>
Via the html_theme_options. This requires adding an option to theme.conf (you can create a theme by inheriting from a standard one):
[options]
num_langs = 1
Then you can set num_langs in conf.py via html_theme_options:
html_theme_options = {'num_langs': len(LANGS)}
and use it in a template:
<p>We have {{ theme_num_langs }} languages</p>