I am working through the fundamentals course. In the models section of the yml, I have specified the +materialized: table instruction, yet the dim_customers is still created as a view in snowflake.
But if I put in the config block atop the sql block, it works fine.
Might you have any tips?
yml
name: 'jaffle_shop'
version: '1.0.0'
config-version: 2
profile: 'default'
model-paths: ["models"]
analysis-paths: ["analyses"]
test-paths: ["tests"]
seed-paths: ["seeds"]
macro-paths: ["macros"]
snapshot-paths: ["snapshots"]
target-path: "target" # directory which will store compiled SQL files
clean-targets: # directories to be removed by `dbt clean`
- "target"
- "dbt_packages"
models:
jaffle:
marts:
core:
+materialized: table
staging:
+materialized: view
project structure
You need to add the name of your project into the models: config in your dbt_project.yml:
models:
jaffle_shop:
jaffle:
marts:
core:
+materialized: table
staging:
+materialized: view
file-tree & yml
snowflake
Hi, I am facing a similar issue, even though the project name is added.
I also tried adding/removing the +enabled option.
Overriding with config macro works.
Related
The problem I’m having is after the profiles.yml All checks passed! (the connection successful)
when i run dbt get this message
Configuration paths exist in your dbt_project.yml file which do not apply to any resources. There are 1 unused configuration paths:- models.dbt_project.example
when I run
dbt --version
Core: - installed: 1.4.1
- latest: 1.4.1 - Up to date!
Plugins: - sqlserver: 1.3.0 - Not compatible!
- synapse: 1.3.2 - Not compatible!
At least one plugin is out of date or incompatible with dbt-core.
You can find instructions for upgrading here:
https://docs.getdbt.com/docs/installation
The error message is telling you what is wrong. There is config in your dbt_project.yml file that does not apply to any files in your models directory.
When you run dbt init, it creates some example models (.sql files) in the models/example directory, like my_first_dbt_model.sql. It also adds some example config to dbt_project.yml that looks like this:
# towards the bottom...
models:
your_project_name:
# Config indicated by + and applies to all files under models/example/
example:
+materialized: view
You need to delete the example key, since you deleted that directory. You could also delete the keys above it, or keep them, since you'll probably add config at some point.
The OTHER error is probably because you installed dbt-core separately from dbt-synapse. You should delete your virtual environment and start over by just running pip install dbt-synapse, which will automatically install a compatible version of dbt-core. You should NEVER pin versions of dbt-core, only the version of your adapter (since the adapter will specify its compatible versions of dbt-core).
My goal is to update all includes in my gitlab-ci.yml files. By default without any custom configuration renovate create MR with an include update based on gitlab-release.
But today, I have some includes only based on tag and there isn't any release associated to this tag. I looking for a solution to update also these includes.
To explain, if I have a release for myprojet named 1.2.3 and tag 1.2 and 1
include:
# Bash template
- project: "myproject"
ref: "1.2.2"
file: "templates/gitlab-ci.yml"
renovate detects there is a new release on 1.2.3 - It's OK
If I have :
include:
# Bash template
- project: "myproject"
ref: "1.1"
file: "templates/gitlab-ci.yml"
renovate don't detect tag named 1.2 for myproject
have you tried regex managers?
here's an example, change it for your needs:
"regexManagers": [
{
"fileMatch": ["(^|/)\\.?gitlab-ci\\.yml$"],
"matchStringsStrategy": "combination",
"matchStrings": [
"\\s\\sCHART_SOURCES_URL: \"(?<depName>.*?)\"\n",
"\\s\\sCHART_SOURCES_VERSION: \"(?<currentValue>.*?)\"\n"
],
"datasourceTemplate": "git-tags"
}
],
i believe you could also set enabled=false for the actual gitlab-ci manager using matchManagers in package rules
I'm hitting a strange error in dbt
I am using the following package to enable external tables in bigquery.
packages:
- package: dbt-labs/dbt_external_tables
version: 0.8.0
$ dbt run-operation stage_external_sources
An error occurred in the dbt Server. Please contact support if this issue persists.
RPC server failed to compile project, call the "status" method for compile status: Compilation Error
dbt found two resources with the name
"dbt_user_test_ext". Since these resources have the same name, dbt will be unable to find the correct resource when source("dbt_user", "test_ext") is used. To fix this change the name of one of these resources:
source.dbt_user.test_ext (models/sources.yml)
source.dbt_user.test_ext (models/sources.yml) Code: 10011
models/sources.yml looks something like this:
sources:
- name: "{{ target.schema }}"
tables:
- name: test_ext
external:
location: "gs://test-bucket/test_ext.csv"
options:
format: csv
skip_leading_rows: 1
Seems to only happen the after the first time I run the command.
Question: what am I doing wrong, and/or how to fix?
I tried to built github .yml file but I’m getting error like
|GitHub Actions/ Main Workflow
Invalid workflow file
The workflow is not valid. .github/workflows/build.yml (Line: 22, Col: 22): Unexpected symbol: '<hash_value>'. Located at position 9 within expression: secrets.<hash_value>|
CODE
on:
Trigger analysis when pushing in master or pull requests, and when creating
a pull request.
push:
branches:
- master
pull_request:
types: [opened, synchronize, reopened]
name: Main Workflow
jobs:
sonarcloud:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v2
with:
# Disabling shallow clone is recommended for improving relevancy of reporting
fetch-depth: 0
- name: SonarCloud Scan
uses: SonarSource/sonarcloud-github-action#v1.3
env:
GITHUB_TOKEN: {{ secrets.<hash_value>}} SONAR_TOKEN: {{ secrets.<hash_value>}}
AND
#Configure here general information about the environment, such as SonarQube server connection details for example
#No information about specific project should appear here
#----- Default SonarQube server
sonar.host.url=https://sonarcloud.io/
#----- Default source code encoding
#sonar.sourceEncoding=UTF-8
sonar.organization=blah blah
sonar.projectKey=blah blah
— optional properties —
defaults to project key
sonar.projectName=Toolsdemo
defaults to ‘not provided’
sonar.projectVersion=1.0
Path is relative to the sonar-project.properties file. Defaults to .
sonar.sources=https://github.com/abcd/xyz
Encoding of the source code. Default is default system encoding
sonar.sourceEncoding=UTF-8
Please help . I need to test my code very fast .
I am doing the DBT hello world tutorial found here, and I have created my first project on a windows machine. My profiles.yml file looks like this:
my-bigquery-db:
target: dev
outputs:
dev:
type: bigquery
method: service-account
project: abiding-operand-286102
dataset: xxxDBTtestO1
threads: 1
keyfile: C:\Users\xxx\.dbt\abiding-operand-286102-ed9e3c9c13cd.json
timeout_seconds: 300
when I execute dbt run I get:
Running with dbt=0.17.2 Encountered an error while reading profiles: ERROR Runtime Error
dbt encountered an error while trying to read your profiles.yml
file.
Profile my-bigquery-db in profiles.yml is empty
Defined profiles:
my-bigquery-db
target
outputs
dev
type
method
project
dataset
threads
keyfile
timeout_seconds
Any idea?
At first glance from both your code and the source walkthrough, this is just a yml config problem. YML is a markup language that is white-space sensitive. And by just looking at the example that you may have pulled from - it doesn't look appropriately white spaced to me.
I'm not sure if you can simply copy from the below but it might be worth a shot.
my-bigquery-db:
target: dev
outputs:
dev:
type: bigquery
method: service-account
project: abiding-operand-286102
dataset: xxxDBTtestO1
threads: 1
keyfile: C:\Users\xxx\.dbt\abiding-operand-286102-ed9e3c9c13cd.json
timeout_seconds: 300
Basically - your dbt profile.yml needs to be setup with the sections at certain levels (not unlike python indentation or any other white spacing scheme).