dbtvault has already been added to the packages.yml file, the yaml file looks like this:
I am following a tutorial that can be found here: dbtset_up
This is a tutorial that I will on snowflake so that I can be able to provide the metadata and it will generate the sql for us and the required links and hubs.
name: dbtvault_snowflake_demo
profile: dbtvault_snowflake_demo
version: '5.3.0'
require-dbt-version: ['>=1.0.0', '<2.0.0']
config-version: 2
analysis-paths:
- analysis
clean-targets:
- target
seed-paths:
- seeds
macro-paths:
- macros
model-paths:
- models
test-paths:
- tests
target-path: target
vars:
load_date: '1992-01-08'
tpch_size: 10 #1, 10, 100, 1000, 10000
models:
dbtvault_snowflake_demo:
raw_stage:
tags:
- 'raw'
materialized: view
stage:
tags:
- 'stage'
enabled: true
materialized: view
raw_vault:
tags:
- 'raw_vault'
materialized: incremental
hubs:
tags:
- 'hub'
links:
tags:
- 'link'
sats:
tags:
- 'satellite'
t_links:
tags:
- 't_link'
I am getting an error when I ran this command:
dbt depsdbt
The error is as follows:
usage: dbt [-h] [--version] [-r RECORD_TIMING_INFO] [-d] [--log-format {text,json,default}]
[-
-no-write-json]
[--use-colors | --no-use-colors] [--printer-width PRINTER_WIDTH] [--warn-error] [--no-
version-check]
[--partial-parse | --no-partial-parse] [--use-experimental-parser] [--no-static-parser]
[--profiles-dir PROFILES_DIR]
[--no-anonymous-usage-stats] [-x] [--event-buffer-size EVENT_BUFFER_SIZE] [-q] [--no-
print]
[--cache-selected-only | --no-cache-selected-only]
{docs,source,init,clean,debug,deps,list,ls,build,snapshot,run,compile,parse,test,seed,run-
operation} ...
dbt: error: argument
{docs,source,init,clean,debug,deps,list,ls,build,snapshot,run,compile,parse,test,seed,run-
operation}: invalid choice: 'depsdbt' (choose from 'docs', 'source', 'init', 'clean', '
'debug', 'deps', 'list', 'ls', 'build', 'snapshot', 'run', 'compile', 'parse', 'test',
'seed', 'run-operation')
The command is:
dbt deps
I believe you tried to execute dbt depsdbt, which is not a command
Related
I have the following code in a GitHub Action config:
name: Build & Tests
on:
pull_request:
env:
CARGO_TERM_COLOR: always
ZEROCOPY_MSRV: 1.61.0
ZEROCOPY_CURRENT_STABLE: 1.64.0
ZEROCOPY_CURRENT_NIGHTLY: nightly-2022-09-26
jobs:
build_test:
runs-on: ubuntu-latest
strategy:
matrix:
# See `INTERNAL.md` for an explanation of these pinned toolchain
# versions.
channel: [ ${{ env.ZEROCOPY_MSRV }}, ${{ env.ZEROCOPY_CURRENT_STABLE }}, ${{ env.ZEROCOPY_CURRENT_NIGHTLY }} ]
target: [ "i686-unknown-linux-gnu", "x86_64-unknown-linux-gnu", "arm-unknown-linux-gnueabi", "aarch64-unknown-linux-gnu", "powerpc-unknown-linux-gnu", "powerpc64-unknown-linux-gnu", "wasm32-wasi" ]
features: [ "" , "alloc,simd", "alloc,simd,simd-nightly" ]
exclude:
# Exclude any combination which uses a non-nightly toolchain but
# enables nightly features.
- channel: ${{ env.ZEROCOPY_MSRV }}
features: "alloc,simd,simd-nightly"
- channel: ${{ env.ZEROCOPY_CURRENT_STABLE }}
features: "alloc,simd,simd-nightly"
I'm getting the following parsing error on this file:
Invalid workflow file: .github/workflows/ci.yml#L19
You have an error in your yaml syntax on line 19
It appears to be referring to this line (it's actually one off, but maybe it's zero-indexing its line numbers?):
channel: [ ${{ env.ZEROCOPY_MSRV }}, ${{ env.ZEROCOPY_CURRENT_STABLE }}, ${{ env.ZEROCOPY_CURRENT_NIGHTLY }} ]
Is there any way to use variables in the matrix definition like this? Or do I just need to hard-code everything?
According to the documentation (reference 1 and reference 2)
Environment variables (at the workflow level) are available to the steps of all jobs in the workflow.
In your example, the environment variables are used at the job level (inside the job strategy / matrix definition), not inside the job steps.
At that level, environment variables aren't interpolated by the GitHub interpreter.
First alternative
Hardcode the values inside the channel field inside your matrix strategy:
Example:
channel: [ "1.61.0", "1.64.0", "nightly-2022-09-26" ]
However, you'll have to do this for each job (bad for maintenance as duplicated code).
Second alternative
Use inputs (with reusable workflow workflow_call trigger, or with workflow_dispatch trigger.
Example:
on:
workflow_dispatch: # or workflow_call
inputs:
test1:
description: "Test1"
required: false
default: "test1"
test2:
description: "Test2"
required: false
default: "test2"
test3:
description: "Test3"
required: false
default: "test3"
jobs:
build_test:
runs-on: ubuntu-latest
strategy:
matrix:
channel: [ "${{ inputs.test1 }}", "${{ inputs.test2 }}", "${{ inputs.test3 }}" ]
In that case, inputs will be interpolated by the GitHub interpreter.
However, you'll need to trigger the workflow from another workflow, or through the GitHub API to send the inputs (in some way, it gives you more flexibility with the values, but increase the complexity).
I am writing an azure pipeline yml requesting to pass variables between jobs but the variables are not passing through. however, it wasn't successful and it returns an empty variable.
here is my pipeline:
jobs:
- job: UpdateVersion
variables:
terraformRepo: ${{ parameters.terraformRepo }}
pool:
vmImage: ubuntu-latest
steps:
- checkout: self
persistCredentials: true
- checkout: ${{ parameters.terraformRepo }}
- task: AzureCLI#2
displayName: PerformVerUpdate
inputs:
azureSubscription: ${{ parameters.azureSubscriptionName }}
scriptType: bash
scriptLocation: inlineScript
inlineScript: |
echo Step 3 result
echo "Reponame $Reponame"
echo "notify $notify"
echo "pullRequestId $pullRequestId"
echo "##vso[task.setvariable variable=pullRequestId;isOutput=true;]$pullRequestId"
echo "##vso[task.setvariable variable=Reponame;isOutput=true;]$Reponame"
echo "##vso[task.setvariable variable=notify;isOutput=true;]true"
Name: PerformVerUpdate
- job: SlackSuccessNotification
dependsOn: UpdateVersion
condition: and(succeeded(), eq(dependencies.UpdateVersion.outputs['PerformVerUpdate.notify'], 'true'))
pool:
vmImage: 'ubuntu-latest'
variables:
- group: platform-alerts-webhooks
- name: notify_J1
value: $[ dependencies.UpdateVersion.outputs['PerformVerUpdate.notify'] ]
- name: pullRequestId_J1
value: $[ dependencies.UpdateVersion.outputs['PerformVerUpdate.pullRequestId'] ]
- name: Reponame_J1
value: $[ dependencies.UpdateVersion.outputs['PerformVerUpdate.Reponame'] ]
steps:
- task: AzurePowerShell#5
displayName: Slack Notification
inputs:
pwsh: true
azureSubscription: ${{ parameters.azureSubscriptionName }}
ScriptType: 'InlineScript'
TargetAzurePs: LatestVersion
inline: |
write-host "Reponame $(Reponame_J1)"
write-host "pullRequest $(pullRequestId_J1)"
I've tried so many different syntax for it but the variables are still not able to pass through between both jobs - e.g. The condition is passing Null result to second job "(Expanded: and(True, eq(Null, 'true'))". Could anyone help with this?
Firstly 'Name' should be 'name' in lowercase
Name: PerformVerUpdate
The rest of syntax seems fine(I have tested it on Bash task because I do not have Azure subscription).
If renaming 'Name' does not help I suppose the problem may be that your Bash task is running within AzureCLI#2 task.
Maybe as workaround you could add new Bash task right after AzureCLI#2 and try to set there output variable for next job?
Pipeline validation fails with Unexpected value 'steps' in acr-login.yaml. Even I tribble-checked the docs and stackoverflows, I can't find the issue in my pipeline:
pipeline.yaml
trigger: none
pool:
name: MyPool
variables:
- template: vars/global.yaml
- template: vars/stage.yaml
stages:
- stage: Import
jobs:
- template: steps/acr-login.yaml
parameters:
registry_name: ${{variables.registry_name}}
acr-login.yaml
parameters:
- name: registry_name
type: string
steps:
- bash: |
az login --identity --output none
az acr login --name ${{ parameters.registry_name }} --output none
Your acr-login.yaml must contain (one or multible) jobs since you are using it as job template:
parameters:
- name: registry_name
type: string
jobs:
- job: myStep
steps:
- bash: |
az login --identity --output none
az acr login --name ${{ parameters.registry_name }} --output none
Following is how our.drone.yml looks like (and template also listed below) this an example configuration very much how we want in our production. The reason we are using a template is that our staging and production have similar configurations with values different in them(hence circuit template). And we wanted to remove duplication using the template circuit.yaml.
But currently, we are unable to do so df I don’t define the test.yaml(template) and have test step imported without template (and have the circuit template define to avoid the duplicate declaration of staging and production build) the drone build fails with
"template converter: template name given not found
If I define the test step as a template. I see the test step working but on creating the tag I see the following error
{"commit":"28ac7ad3a01728bd1e9ec2992fee36fae4b7c117","event":"tag","level":"info","msg":"trigger: skipping build, no matching pipelines","pipeline":"test","ref":"refs/tags/v1.4.0","repo":"meetme2meat/drone-example","time":"2022-01-07T19:16:15+05:30"}
---
kind: template
load: test.yaml
data:
commands:
- echo "machine github.com login $${GITHUB_LOGIN} password $${GITHUB_PASSWORD}" > /root/.netrc
- chmod 600 /root/.netrc
- go clean -testcache
- echo "Running test"
- go test -race ./...
---
kind: template
load: circuit.yaml
data:
deploy: deploy
create_tags:
commands:
- echo "Deploying version $DRONE_SEMVER"
- echo -n "$DRONE_SEMVER,latest" > .tags
backend_image:
version: ${DRONE_SEMVER}
tags:
- '${DRONE_SEMVER}'
- latest
And the template is below
test.yaml
kind: pipeline
type: docker
name: test
steps:
- name: test
image: golang:latest
environment:
GITHUB_LOGIN:
from_secret: github_username
GITHUB_PASSWORD:
from_secret: github_token
commands:
{{range .input.commands }}
- {{ . }}
{{end}}
volumes:
- name: deps
path: /go
- name: build
image: golang:alpine
commands:
- go build -v -o out .
volumes:
- name: deps
path: /go
volumes:
- name: deps
temp: {}
trigger:
branch:
- main
event:
- push
- pull_request
circuit.yaml
kind: pipeline
type: docker
name: {{ .input.deploy }}
steps:
- name: create-tags
image: alpine
commands:
{{range .input.create_tags.commands }}
- {{ . }}
{{end}}
- name: build
image: plugins/docker
environment:
GITHUB_LOGIN:
from_secret: github_username
GITHUB_PASSWORD:
from_secret: github_token
VERSION: {{ .input.backend_image.version }}
SERVICE: circuits
settings:
auto_tag: false
repo: ghcr.io/meetme2meat/drone-ci-example
registry: ghcr.io
frameworkVersion: '2'
plugins:
- serverless-step-functions
- serverless-python-requirements
- serverless-parameters
- serverless-pseudo-parameters
provider:
name: aws
region: us-east-2
stage: ${opt:stage, 'dev'}
runtime: python3.7
versionFunctions: false
iam:
role: arn:aws:iam::#{AWS::AccountId}:role/AWSLambdaVPCAccessExecutionRole
apiGateway:
shouldStartNameWithService: true
lambdaHashingVersion: 20201221
package:
exclude:
- node_modules/**
- venv/**
# Lambda functions
functions:
generateAlert:
handler: handler.generateAlert
generateData:
handler: handler.generateDataHandler
timeout: 600
approveDenied:
handler: handler.approveDenied
timeout: 600
stepFunctions:
stateMachines:
"claims-etl-and-insight-generation-${self:provider.stage}":
loggingConfig:
level: ALL
includeExecutionData: true
destinations:
- Fn::GetAtt: ["ETLStepFunctionLogGroup", Arn]
name: "claims-etl-and-insight-generation-${self:provider.stage}"
definition:
Comment: "${self:provider.stage} ETL Workflow"
StartAt: RawQualityJob
States:
# Raw Data Quality Check Job Start
RawQualityJob:
Type: Task
Resource: arn:aws:states:::glue:startJobRun.sync
Parameters:
JobName: "data_quality_v2_${self:provider.stage}"
Arguments:
"--workflow-name": "${self:provider.stage}-Workflow"
"--dataset_id.$": "$.datasetId"
"--client_id.$": "$.clientId"
Next: DataQualityChoice
Retry:
- ErrorEquals: [States.ALL]
MaxAttempts: 2
IntervalSeconds: 10
BackoffRate: 5
Catch:
- ErrorEquals: [States.ALL]
Next: GenerateErrorAlertDataQuality
# End Raw Data Quality Check Job
DataQualityChoice:
Type: Task
Resource:
Fn::GetAtt: [approveDenied, Arn]
Next: Is Approved ?
Is Approved ?:
Type: Choice
Choices:
- Variable: "$.quality_status"
StringEquals: "Denied"
Next: FailState
Default: HeaderLineJob
FailState:
Type: Fail
Cause: "Denied status"
# Header Line Job Start
HeaderLineJob:
Type: Parallel
Branches:
- StartAt: HeaderLineIngestion
States:
HeaderLineIngestion:
Type: Task
Resource: arn:aws:states:::glue:startJobRun.sync
Parameters:
JobName: headers_lines_etl_rs_v2
Arguments:
"--workflow-name.$": "$.Arguments.--workflow-name"
"--dataset_id.$": "$.Arguments.--dataset_id"
"--client_id.$": "$.Arguments.--client_id"
End: True
Retry:
- ErrorEquals: [States.ALL]
MaxAttempts: 2
IntervalSeconds: 10
BackoffRate: 5
Catch:
- ErrorEquals: [States.ALL]
Next: GenerateErrorAlertHeaderLine
End: True
# Header Line Job End
GenerateErrorAlertDataQuality:
Type: Task
Resource:
Fn::GetAtt: [generateAlert, Arn]
End: true
resources:
Resources:
# Cloudwatch Log
"ETLStepFunctionLogGroup":
Type: AWS::Logs::LogGroup
Properties:
LogGroupName: "ETLStepFunctionLogGroup_${self:provider.stage}"
This is what my serverless.yml file looks like.
When I run the command:
sls deploy --stage staging
It show
Type Error ----------------------------------------------
TypeError: Cannot read property 'stage' of undefined
at Variables.getValueFromOptions (/snapshot/serverless/lib/classes/Variables.js:648:37)
at Variables.getValueFromSource (/snapshot/serverless/lib/classes/Variables.js:579:17)
at /snapshot/serverless/lib/classes/Variables.js:539:12
Your Environment Information ---------------------------
Operating System: linux
Node Version: 14.4.0
Framework Version: 2.30.3 (standalone)
Plugin Version: 4.5.1
SDK Version: 4.2.0
Components Version: 3.7.4
How I can fix this? I tried with different version of serverless.
There is error in yamlParser file, which is provided by serverless-step-functions.
Above is my serverless config file.
It looks like a $ sign is missing from your provider -> stage?
provider:
name: aws
region: us-east-2
stage: ${opt:stage, 'dev'} # $ sign is missing?
runtime: python3.7
versionFunctions: false
iam:
role: arn:aws:iam::#{AWS::AccountId}:role/AWSLambdaVPCAccessExecutionRole
apiGateway:
shouldStartNameWithService: true
lambdaHashingVersion: 20201221