How do I use a yaml selector? - dbt

I am experimenting with yaml selectors, so far without success. My selectors.yml:
selectors:
- name: daily
description: selects models for daily run
definition:
exclude:
- union:
- "tag:pp_backfill"
- "tag:inc_transactions_raw_data"
- "tag:hourly"
And when I try to use it I get an error:
$ dbt ls --selector daily
* Deprecation Warning: dbt v0.17.0 introduces a new config format for the
dbt_project.yml file. Support for the existing version 1 format will be removed
in a future release of dbt. The following packages are currently configured with
config version 1:
- honey_dbt
- dbt_utils
- audit_helper
- codegen
For upgrading instructions, consult the documentation:
https://docs.getdbt.com/docs/guides/migration-guide/upgrading-to-0-17-0
* Deprecation Warning: The "adapter_macro" macro has been deprecated. Instead,
use the `adapter.dispatch` method to find a macro and call the result.
adapter_macro was called for: dbt_utils.intersect
Encountered an error:
Runtime Error
Could not find selector named daily, expected one of []
I tried this in both 0.18.1 and 0.19.0, with and without config-version: 2. Any thoughts?

I think the blocker here might be that you are not currently selecting anything initially to then exclude specific models from using the tag method. Here is solution in my project and then an adoption that might work in your case.
Context
I'm running dbt version 0.19.0 on dbt Cloud. Both of these compiled and successfully ran dbt run --selector daily.
Jaffle Shop Example
stg_customers is tagged with dont_run_me and stg_orders is tagged with also_dont_run_me
selector.yml is the following at the root of the dbt project
selectors:
- name: daily
description: selects models for daily run
definition:
union:
- method: path
value: models
- exclude:
- method: tag
value: dont_run_me
- method: tag
value: also_dont_run_me
The logic here is that I'm first select all the models and then exclude the union of the models that have tags dont_run_me and also_dont_run_me.
dbt run --selector daily ended up running everything in my project except stg_customers and stg_orders
Specific Case
If you are trying to select all models except for ones that are tagged as pp_backfill, inc_transactions_raw_data, and hourly, I think the following will do the trick:
selectors:
- name: daily
description: selects models for daily run
definition:
union:
- method: path
value: models
- exclude:
- union:
- method: tag
value: pp_backfill
- method: tag
value: inc_transactions_raw_data
- method: tag
value: hourly

Related

DBT - [WARNING]: Did not find matching node for patch

I keep getting the error below when I use dbt run - I can't find anything on why this error occurs or how to fix it within the dbt documentation.
[WARNING]: Did not find matching node for patch with name 'vGenericView' in the 'models' section of file 'models\generic_schema\schema.sql'
did you by chance recently upgrade to dbt 1.0.0? If so, this means that you have a model, vGenericView defined in a schema.yml but you don't have a vGenericView.sql model file to which it corresponds.
If all views and tables defined in schema are 1 to 1 with model files then try to run dbt clean and test or run afterward.
Not sure what happened to my project, but ran into frustration looking for missing and/or misspelled files when it was just leftovers from different compiled files not cleaned out. Previously moved views around to different schemas and renamed others.
So the mistake is here in the naming:
The model name in the models.yml file should for example be: employees
And the sql file should be named: employees.sql
So your models.yml will look like:
version: 2
models:
- name: employees
description: "View of employees"
And there must be a model with file name: employees.sql
One case when this will happen is if you have the same data source defined in two different schema.yml file (or whatever you call it)

Configuring raw and analytics databases with dbt

I have been reading dbt's How we configure Snowflake guide which explains the rationale behind having a raw database and an analytics database. Raw data is loaded into your warehouse into raw (e.g. by using Fivetran) and analytics is used by dbt to save transformed data/views for data analysts/scientists.
However, I can't seem to find any guides on how to actually set this up. The profiles.yml file needs to point to where the raw data is, so that dbt can begin transforming. However, this file also seems to dictate the database and schema into which transformed data/views are saved.
Where in dbt's many .yml files do I specify globally where to save transformed data?
Set up your profiles.yml, which does NOT live in the actual project but rather in the ~/.dbt folder on your machine, such that it refers to your target database/schema. For development, this would look like what you see below. For production on dbt Cloud. Now, you just set up your sources like usual (see third block below). There is no universal sources option, just a target database/schema.
Profiles.yml Docs and Snowflake Profile Docs
-- profiles.yml
my_profile:
target: dev
outputs:
dev:
type: snowflake
account: <snowflake_server>
user: my_user
password: my_password
role: my_role
database: analytics
warehouse: dev_wh
schema: dbt_<myname>
threads: 1
client_session_keep_alive: False
-- dbt_project.yml
name: 'my_dbt_models'
version: '1.0.0'
config-version: 2
profile: 'my_profile'
...
...
...
Sources Docs
-- src.yml
version: 2
sources:
- name: jaffle_shop
database: raw
tables:
- name: orders
In the model:
raw.jaffle_shop.orders becomes {{ source( 'jaffle_shop' , 'orders' ) }}
Note, dbt processes this source such that it assumes the name is the schema by default, however, I've discovered that you can really name it whatever you want and add in a schema if you want to give it a special name.
For example…
sources:
- name: my_special_name
database: raw
schema: jaffle_shop
tables:
- name: orders
In the model:
raw.jaffle_shop.orders becomes {{ source( 'my_special_name' , 'orders' ) }}
I hope all that made sense.

what are drone.io 0.8.5 plugin/gcr secretes' acceptable values?

I'm having trouble pushing to gcr with the following
gcr:
image: plugins/gcr
registry: us.gcr.io
repo: dev-221608/api
tags:
- ${DRONE_BRANCH}
- ${DRONE_COMMIT_SHA}
- ${DRONE_BUILD_NUMBER}
dockerfile: src/main/docker/Dockerfile
secrets: [GOOGLE_CREDENTIALS]
when:
branch: [prod]
...Where GOOGLE_CREDENTIALS will work, but if named say GOOGLE_CREDENTIALS_DEV it will not be properly picked up. GCR_JSON_KEY works fine. I recall reading legacy documentation that spelled out the acceptable variable names, of which GOOGLE_CREDENTIALS and GCR_JSON_KEY were listed among other variants but as of version 1 they've done some updates omitting that info.
So, question is, is the plugin capable of accepting whatever variable name or is it expecting specific variable names and if so what are they?
The Drone GCR plugin accepts the credentials in a secret named PLUGIN_JSON_KEY, GCR_JSON_KEY, GOOGLE_CREDENTIALS, or TOKEN (see code here)
If you stored the credentials in drone as GOOGLE_CREDENTIALS_DEV then you can rename it in the .drone.yml file like this:
...
secrets:
- source: GOOGLE_CREDENTIALS_DEV
target: GOOGLE_CREDENTIALS
...

How to override environment variables in jenkins_job_builder at job level?

I am trying to find a way to inherit/override environment variables in jenkins jobs defined via jenkins-job-builder (jjb).
Here is one template that does not work:
#!/usr/bin/env jenkins-jobs test
- defaults: &sample_defaults
name: sample_defaults
- job-template:
name: 'sample-{product_version}'
project-type: pipeline
dsl: ''
parameters:
- string:
name: FOO
default: some-foo-value-defined-at-template-level
- string:
name: BAR
default: me-bar
- project:
defaults: sample_defaults
name: sample-{product_version}
parameters:
- string:
name: FOO
value: value-defined-at-project-level
jobs:
- 'sample-{product_version}':
product_version:
- '1.0':
parameters:
- string:
name: FOO
value: value-defined-at-job-level-1
- '2.0':
# this job should have:
# FOO=value-defined-at-project-level
# BAR=me-bar
Please note that it is key to be able to override these parameters at job or project level instead of template.
Requirements
* be able to add as many environment variables like this without having to add one JJB variable for each of them
* user should not be forced to define these at template or job levels
* those var need to endup being exposed as environment variables at runtime for pipelines and freestyle jobs.
* syntax is flexible but a dictionary approach would be highly appreciated, like:
vars:
FOO: xxx
BAR: yyy
The first thing to understand is how JJB priorities where it will pull variables in from.
job-group section definition
project section definition
job-template variable definition
defaults definition
(This is not an exhaustive list but it's covers the features I use)
From this list we can immediately see that if we want to make job-templates have override-able then using JJB defaults configuration is useless as it has the lowest precedence when JJB is deciding where to pull from.
On the other side of the spectrum, job-groups has the highest precedence. Which unfortunately means if you define a variable in a job-group with the intention of of overriding it at the project level then you are out of luck. For this reason I avoid setting variables in job-groups unless I want to enforce a setting for a set of jobs.
Declaring variable defaults
With that out of the way there are 2 ways JJB allows us to define defaults for a parameter in a job-template:
Method 1) Using {var|default}
In this method we can define the default along with the definition of the variable. For example:
- job-template:
name: '{project-name}-verify'
parameters:
- string:
name: BRANCH
default: {branch|master}
However where this method falls apart if you need to use the same JJB variable in more than one place as you will have multiple places to define the default value for the template. For example:
- job-template:
name: '{project-name}-verify'
parameters:
- string:
name: BRANCH
default: {branch|master}
scm:
- git:
refspec: 'refs/heads/{branch|master}'
As you can see we now have 2 places were we are declaring {branch|master} not ideal.
Method 2) Defining the default variable value in the job-template itself
With this method we declare the default value of the variable in the job-template itself just once. I like to section off my job-templates like this:
- job-template:
name: '{project-name}-verify'
#####################
# Variable Defaults #
#####################
branch: master
#####################
# Job Configuration #
#####################
parameters:
- string:
name: BRANCH
default: {branch}
scm:
- git:
refspec: 'refs/heads/{branch}'
In this case there is still 2 branch definitions for the job-template. However we also provide the default value for the {branch} variable at the top of the file. Just once. This will be the value that the job takes on if it is not passed in by a project using the template.
Overriding job-templates variables
When a project now wants to use a job-template I like to use one of 2 methods depending on the situation.
- project:
name: foo
jobs:
- '{project-name}-merge'
- '{project-name}-verify'
branch: master
This is the standard way that most folks use and it will set branch: master for every job-template in the list. However sometimes you may want to provide an alternative value for only 1 job in the list. In this case the more specific declaration takes precendence.
- project:
name: foo
jobs:
- '{project-name}-merge':
branch: production
- '{project-name}-verify'
branch: master
In this case the verify job will get he value "master" but the merge job will instead get the branch value "production".

How to use a dependency of a module within a Play app

I am writing a Play Framework module in order to share some common logic among multiple Play apps. One of the things I would like my module to do is provide some frequently-used functionality by way of 3rd-party modules, for example the excellent Markdown module.
First of all, is it possible to do this? I want all the apps that include my module to be able to use the .markdown().raw() String extension without needing to explicitly declare the Markdown module as a dependency. The Play Framework Cookbook chapter 5 seems to imply that it is possible, unless I am reading it wrong.
Secondly, if it is possible, how does it work? I have created the following vanilla example case, but I'm still getting errors.
I created a new, empty application "myapp", and a new, empty module "mymod", both in the same parent directory. I then modified mymod/conf/dependencies.yml to:
self: mymod -> mymod 0.1
require:
- play
- play -> markdown [1.5,)
I ran play deps on mymod and it successfully downloaded and installed the Markdown module. Running play build-module also worked fine with no errors.
Then, I modified myapp/conf/dependencies.yml to:
# Application dependencies
require:
- play
- mymod -> mymod 0.1
repositories:
- Local Modules:
type: local
artifact: ${application.path}/../[module]
contains:
- mymod
I ran play deps on myapp and it successfully found mymod, and generated the myapp/modules/mymod file, containing the absolute path to mymod.
I ran myapp using play run and was able to see the welcome page on http://localhost:9000/. So far so good.
Next, I modified myapp/app/views/Application/index.html to:
#{extends 'main.html' /}
#{set title:'Home' /}
${"This is _MarkDown_, by [John Gruber](http://daringfireball.net/projects/markdown/).".markdown().raw()}
I restarted myapp, and now I get the following error.
09:03:23,425 ERROR ~
#6a6eppo46
Internal Server Error (500) for request GET /
Template execution error (In /app/views/Application/index.html around line 4)
Execution error occured in template /app/views/Application/index.html. Exception raised was MissingMethodException : No signature of method: java.lang.String.markdown() is applicable for argument types: () values: [].
play.exceptions.TemplateExecutionException: No signature of method: java.lang.String.markdown() is applicable for argument types: () values: []
at play.templates.BaseTemplate.throwException(BaseTemplate.java:86)
at play.templates.GroovyTemplate.internalRender(GroovyTemplate.java:257)
at play.templates.Template.render(Template.java:26)
at play.templates.GroovyTemplate.render(GroovyTemplate.java:187)
at play.mvc.results.RenderTemplate.<init>(RenderTemplate.java:24)
at play.mvc.Controller.renderTemplate(Controller.java:660)
at play.mvc.Controller.renderTemplate(Controller.java:640)
at play.mvc.Controller.render(Controller.java:695)
at controllers.Application.index(Application.java:13)
at play.mvc.ActionInvoker.invokeWithContinuation(ActionInvoker.java:548)
at play.mvc.ActionInvoker.invoke(ActionInvoker.java:502)
at play.mvc.ActionInvoker.invokeControllerMethod(ActionInvoker.java:478)
at play.mvc.ActionInvoker.invokeControllerMethod(ActionInvoker.java:473)
at play.mvc.ActionInvoker.invoke(ActionInvoker.java:161)
at Invocation.HTTP Request(Play!)
Caused by: groovy.lang.MissingMethodException: No signature of method: java.lang.String.markdown() is applicable for argument types: () values: []
at /app/views/Application/index.html.(line:4)
at play.templates.GroovyTemplate.internalRender(GroovyTemplate.java:232)
... 13 more
And just to confirm I'm not crazy, I tried adding the play -> markdown [1.5,) line to myapp/conf/dependencies.yml and restarted the app, and confirmed that it works.
I feel like I'm missing something obvious. Many thanks in advance to anyone who can help! :)
Yes I had the same problem, it seems that transitive dependencies through custom home made modules does not work