reference a global var in yml file in dbt - dbt

Referencing a global variable in a .yml file in dbt
For example, if I have a column or a table name that is defined as a global in my dbt_project.yml file:
# dbt_project.yml
name: 'my_project'
version: '1.0.0'
config-version: 2
vars:
my_var: 'my_special_var'
how can I reference my_var in a .yml file?

There are a couple of ways, if you use dbt core, you can export the environment variables e.g.
export DBT_USER_NAME=andy
then you can reference it in the yaml file:
models:
test_project:
AA:
databaseusername: {{ var("DBT_USER_NAME") }}
you can also define variables in the yaml file
vars:
test_project:
env: "dev"
etl_schema: "etl"
...
models:
test_project:
AA:
database: test_{{ var("env") }}_raw
schema: {{ var("etl_schema") }}
then overwrite the variable value in the command line
dbt run --vars '{"env": "prod"}'

here's an example from a github issue comment:
# models/sources.yml
version: 2
sources:
- name: my_source
tables: "{{ var('my_list_of_tables') }}"

Related

Can github actions edit parts of README.md?

I want to update ONLY the first header of a readme so it always has the repo's name.
I know how to get the repo name in github actions by doing something like:
name: CI
on:
push:
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v3
- name: Run a one-line script
run: echo "REPO_NAME=${{ github.event.repository.name }}" >> $GITHUB_ENV
However, I want to access my readme.md and add the 'github.event.repository.name' to the header.
So I would make a Readme with something like:
Introduction for: {{ github.event.repository.name }}
hoping I can get something like this with gitactions:
Introduction for: RepoName
I tried using some marketplace github actions but the one I tried seems to not do variables and it seems to update the whole readme.md not just the header: https://github.com/marketplace/actions/dynamic-readme
Here is the failed example for this marketplace plugin:
name: Dynamic Template
on:
push:
branches:
- main
workflow_dispatch:
jobs:
update_templates:
name: "Update Templates"
runs-on: ubuntu-latest
steps:
- name: "📥 Fetching Repository Contents"
uses: actions/checkout#main
# Runs a single command using the runners shell
- name: Run a one-line script
run: echo "REPO_NAME=${{ github.event.repository.name }}" >> $GITHUB_ENV
# Runs a set of commands using the runners shell
- name: Update GitHub Profile README
uses: theboi/github-update-readme#v1.0
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
REPO_NAME: ${{ github.event.repository.name }}
with:
header: $REPO_NAME
Is there any way to make the readme.md file have the repo name dynamically with a variable in github actions?
EDIT: I think I am very close but I don't know how to commit code in github actions. I figured, I can do this manually by using the sed command in bash in github actions. I think it worked but I think I have to commit the code to make it save. Here is the code I have so far:
name: Dynamic Template
on:
push:
branches:
- main
workflow_dispatch:
jobs:
update_templates:
name: "Update Templates"
runs-on: ubuntu-latest
steps:
- name: "📥 Fetching Repository Contents"
uses: actions/checkout#main
# Runs a single command using the runners shell
- name: Run a one-line script
run: echo "REPO_NAME=${{ github.event.repository.name }}" >> $GITHUB_ENV
# Runs a set of commands using the runners shell
- name: Run a multi-line script
run: |
ls
sed -i 's/<reponame>/$REPO_NAME/' README.md
cat README.md
echo $REPO_NAME
I figured it out. You can it manually by using the sed command in bash within a runner in github actions. Set your README.md with a variable that you want to replace like <reponame> then use github actions to find that string and replace it with something you want (for me the repo name).
name: Dynamic Template
on:
create:
jobs:
update_templates:
name: "Update Templates"
runs-on: ubuntu-latest
steps:
- name: "📥 Fetching Repository Contents"
uses: actions/checkout#main
# Runs a set of commands using the runners shell
- name: Update README.md
run: |
sed -i 's/<reponame>/'${{ github.event.repository.name }}'/' README.md
git config user.email "41898282+github-actions[bot]#users.noreply.github.com"
git config user.name "github-actions[bot]"
git commit -am "Automated report"
git push
The email I used is a dependabot mentioned from here: https://github.com/orgs/community/discussions/26560#discussioncomment-3531273
This method needs four files
README.md
python script #replace tag
github_action.yml #run it automatically
variable.json #define variable
#PythonKiddieScripterX provides a good idea.
Based on the idea that context inside <> those <information will be hidden>. We can update our readme files by replacing those tags <> with the texts we want.
Example of my README.md file
This answer is version `1.0`. It will be updated automatically by loading `json` file and using GitHub Action.
In my readme.json file
{
VERSION: 1.0
}
Then I will only need a script to do the replacement. In this example, I use python saved as
replace_tag.py.
import re
import os
import json
# read all .md files
readmefiles = []
for root, dirs, files in os.walk("."):
for file in files:
if file.endswith(".md"):
readmefiles.append(os.path.join(root, file))
# load variable json
with open('readme.json') as f:
var_dic = json.load(f)
# match pattern (<variable-*.?-tag>)(`*.?`)
# example: (<variable-VERSION-tag>)(`1.1`)
for filename in readmefiles:
with open(filename,"r") as f:
content = f.read()
# update readme variables
for key, value in var_dic.items():
pattern = r"(<variable-{}-tag>)(`.*?`)".format(key)
replacement = r"\1`{}`".format(value)
content = re.sub(pattern, replacement, content)
with open(filename,"w") as f:
f.write(content)
Then the final thing is to use GitHub Action to run this python script every time there is a change.
yml file could be
name: README Dynamic Update
on:
push:
paths:
- *.md
- readme.json
- **/*.md
workflow_dispatch:
jobs:
update_templates:
name: "Update Templates"
runs-on: ubuntu-latest
steps:
- name: "📥 Update GitHub Readme files"
uses: actions/checkout#main
# Runs a set of commands using the runners shell
- name: Update README.md
run: |
python replace_tag.py
- name: pull-request
uses: repo-sync/pull-request#v2
with:
destination_branch: "main"
github_token: ${{ secrets.GITHUB_TOKEN }}
- name: commit
run: |
git config --global user.email youremail
git config --global user.name yourusername
git add .
git commit -m "README update Automation" -a
- name: Push changes
uses: ad-m/github-push-action#master
with:
github_token: ${{ secrets.GITHUB_TOKEN }}
Also for those text inside the code blocks, we can use <pre></pre> to solve the hidding issue.
Note that
need to add tag in this format <variable-YOURTAG-tag>"variable"

dbt env macro produces wrong output

i've encountered strange problem with env var.
Here's my profiles.yml file
project:
outputs:
prod:
type: postgres
threads: 4
host: my_host
port: 5432
user: "{{ env_var('DBT_USER') }}"
pass: "{{ env_var('DBT_PASS') }}"
dbname: my_db
schema: my_schema
target: prod
Here's my env variables:
echo $DBT_USER;
my_user
echo $DBT_PASS;
my_password
dbt-debug output:
connection to server at "my_host", port 5432 failed: FATAL: password authentication failed for user "my_user"
But if I change
pass: "{{ env_var('DBT_PASS') }}"
to
my_password
in profiles.yml, dbt-debug shows no error.
dbt version is 1.0.1.
What am i doing wrong?
#dcpt The way you used the env_var function looks correct per the env_var docs but there's two things I can think of.
Are you restarting your shell after setting your env variables? Maybe there's something about the CLI session that needs to be renewed after a profiles.yml or env_var change.
I think that you need to layout your profiles.yml with target as a higher header? Looking at the example here on profiles.yml it looks like:
profiles.yml
<profile-name>:
target: <target-name>
outputs:
<target-name>:
type: <bigquery | postgres | redshift | snowflake | other>
schema: <schema_identifier>
threads: <natural_number>
But you have your target: at the bottom? Is that supposed to be an additional target definition or should it be higher above your prod: tag.

Use pool name as variable from variable group for deployment task

This question came from fact that you can only use global variables for deployment:
https://learn.microsoft.com/en-us/azure/devops/pipelines/process/deployment-jobs?view=azure-devops#schema
Use only global level variables for defining a pool name. Stage/job level variables are not supported to define pool name.
But my config is a bit more complex. I do have main pipeline.yml with several stages. Stage's jobs defined in the template, so I have following structure:
# pipeline.yml (some cuts)
....
stages:
- stage: Build
displayName: Build all
jobs:
- template: ../templates/build.yml
parameters:
stage: 'DEV'
- stage: Dev
displayName: Deploy Dev
variables:
- group: GROUP-DEV
pool:
name: '$(env._global.agentPool)' # env._global.agentPool defined in in the variable group
jobs:
- template: ../templates/deployment.yml
parameters:
stage: 'DEV'
# other stages here ....
# templates/deployment.yml
parameters:
- name: stage
displayName: 'Stage'
type: string
jobs:
- deployment: Deploy${{ parameters.stage }}
displayName: Deploy ${{ parameters.stage }}
environment: ${{ parameters.stage }}
strategy:
runOnce:
deploy:
steps:
- download: current
artifact: art_${{ parameters.stage }}
displayName: 'Download art_${{ parameters.stage }}'
# more steps here.....
Now the problem: with such config I got error:
##[error]Pipeline does not have permissions to use the referenced pool(s) . For authorization details, refer to https://aka.ms/yamlauthz.
which means we faced with limitation mentioned in the first quote (proof: https://developercommunity.visualstudio.com/t/pool-name-in-deployment-job-cannot-be-in-a-variabl/940174)
As soon as I change either pool name to hardcoded value or remove -deployment and use regular steps all works fine.
Question: with that, is there any chance to re-work templates to still use agent pool name from group variable? I need the -deployment task to reference environment.
What was tried already:
use pool name as param of the templates/deployment.yml and use pool inside steps. No luck, same story

How to load an image in gitlab-ci.yml when the version tag is inside a file?

I want to create a file .terraform-version with the content:
0.13.4
Then in gitlab-ci.yml I want to load the version for select image, like:
variables:
TERRAFORM_DOCKER_TAG_VERSION: $(head -n 1 .terraform-version)
# ...
PlanMRDev:
image:
name: hashicorp/terraform:$TERRAFORM_DOCKER_TAG_VERSION
But the content of the $TERRAFORM_DOCKER_TAG_VERSION will be the string $(head -n 1 .terraform-version).
I also tried with:
before_script:
- export TERRAFORM_DOCKER_TAG_VERSION=$(cat .terraform-version)
But in this case the $TERRAFORM_DOCKER_TAG_VERSION will be empty.
Can anyone help me?
Thanks :)
You could use Gitlab Dotenv Artifacts. This is a feature of Gitlab that allows you to define a special kind of artifact - a file containing environment variables - that will be loaded automatically by all subsequent jobs that would normally load the artifacts (previous stages or defined with a need clause).
This is how the jobs would look:
prepare:
script:
- echo "export TERRAFORM_DOCKER_TAG_VERSION=$(cat .terraform-version)" >> version.env
artifacts:
reports:
dotenv: version.env
PlanMRDev:
image:
name: hashicorp/terraform:$TERRAFORM_DOCKER_TAG_VERSION

Github Action - use Variable in the script [duplicate]

This question already has answers here:
How do I set an env var with a bash expression in GitHub Actions?
(2 answers)
Closed 3 years ago.
With Github Actions how to do a backup of a file?
I need to create a new folder named with the date/time of today.
name: Backup Database
on:
push:
branches:
- master
jobs:
build:
runs-on: ubuntu-latest
steps:
- name: Backup
uses: appleboy/ssh-action#master
with:
host: ${{ secrets.HOST }}
username: ${{ secrets.USERNAME }}
key: ${{ secrets.KEY }}
port: 22
script: |
DATE=${{ NEW DATE }} # Need to create a variable with date/time
mkdir backup_${{ DATE }} # Create a new folder with the date/time
This user found a hack to create vars that are generated with bash commands output.
In his words:
I found a way to hack this limitation. Write your VAR on disk (the CI
system disk), then cat $my_var to use your VAR in every step you need
Reference: "How to define env variable?"