I want to set the owner for dbt models from yml file. I have tried everything.
---
version: 2
models:
- name: my_model
description: "Foo bar"
owner: 'XXXX' # not work
config:
owner: 'XXXX' # not work
meta:
owner: 'XXXX' # not work
dbt-core 1.2.2
Finally I can do it
A. Add schema file
make this file
src/models/some_version/some_squad/some_domain/schema.yml
B. Content
version: 2
models:
- name: some_model_name
meta:
owner: 'some_owner'
domain:
- some_domain_
- some_domain_2
description: |
some description
C. Final result
Related
dbt version: 1.3.1
python version: 3.9.6
adapter = dbt-synapse.yml
# profiles.yml
default: dbt_project
dbt_project:
target: dev
outputs:
dev:
type: synapse
driver: 'ODBC Driver 17 for SQL Server'
server: XXXXXXX
database: ###
port: 1433
schema: #######
user: ######
password: #####
azure_blob:
target: dev
outputs:
dev:
type: azure_blob
account_name: ##
account_key: ##
container: ##
prefix: delta_lake
--- after implied this change here is the error message a get--01/30/2023--- #2:32 pm central time----
i get this error when try to read the file from azure blob storage
-- the is the profiles.yml--------
default: dbt_project
dbt_project:
target: dev
outputs:
dev:
type: synapse #synapse #type: Azuresynapse
driver: 'ODBC Driver 17 for SQL Server' # (The ODBC Driver installed on your system)
server: XXXXXXX
database: XXXXXXX
port: 1433
schema: XXXXXXX
#authentication: sqlpassword
user: XXXXXXX
password: XXXXXXX
azure_blob:
type: azure_blob
account_name: XXXXXXX
account_key: XXXXXXX
container: data-platform-archive #research-container/Bronze/Freedom/ABS_VESSEL/
prefix: abc/FGr1/fox/
--------------- dbt_project.yml-------------------------
name or the intended use of these models
name: 'dbt_project'
version: '1.0.0'
config-version: 2
# This setting configures which "profile" dbt uses for this project.
profile: 'dbt_project'
model-paths: ["models"]
analysis-paths: ["analyses"]
test-paths: ["tests"]
seed-paths: ["seeds"]
macro-paths: ["macros"]
snapshot-paths: ["snapshots"]
target-path: "target"
clean-targets:
- "target"
- "dbt_packages"
models:
dbt_project:
staging:
+materialized: table
utilities:
+materialized: view
azure_Blob:
staging:
+materialized: view
--------------------------------
Model name=dbt_stg_DL_abs_acm_users.sql"
and here is the code
{{ config(
materialized='view',
connection='azure_blob'
) }}
select *
from {{ source('data-platform-archive/abc/FGr1/fox/', 'abc.parquet') }}
Compilation Error in model dbt_stg_DL_abs_acm_users
Model 'model.dbt_project.dbt_stg_DL_abs_acm_users' 'abc.parquet' which was not found
Yes, what you've shown here is multiple profiles in a single profiles.yml file. However, there is no default key in the profiles.yml file, since the profile must be specified by the project.
In your dbt_project.yml file, there is a key that names the profile that the project should use. This project config use the dbt_project profile that you have defined:
# dbt_project.yml
name: 'jaffle_shop'
profile: 'dbt_project'
And this one will use the azure_blob profile that you have defined:
# dbt_project.yml
name: 'jaffle_shop'
profile: 'azure_blob'
This is a common pattern if you are developing on multiple projects on a single machine. For a SINGLE project, it is more common to define multiple targets:
# profiles.yml
dbt_project:
target: dev
outputs:
dev:
type: synapse
...
blob:
type: azure_blob
...
You can select which target to use at runtime by passing the -t or --target parameter to the CLI: dbt run -t blob would run the project against the azure_blob connection. The target: dev line in the file above specifies that the dev target (so the Synapse connection) should be the default, if the target is not specified at runtime.
It's somewhat unusual to have multiple targets use different types of databases, and some care must be taken to write compatible syntax. But it is possible -- many packages do this for integration tests, for example.
can we have two profile in one profiles.yml file in root
For example, to have a profile for Azure Blob and another profile for Azure Synapse SQL
for info:
dbt version: 1.3.1
python version: 3.9.6
adapter = dbt-synapse
# profiles.yml
default: dbt_project
dbt_project:
target: dev
outputs:
dev:
type: synapse
driver: 'ODBC Driver 17 for SQL Server' # (The ODBC Driver installed on your system)
server: XXXXXXX
database: ###
port: 1433
schema: #######
user: ######
password: #####
azure_blob:
target: dev
outputs:
dev:
type: azure_blob
account_name: ##
account_key: ##
container: ##
prefix: delta_lake
when i r deb-debug
get this error
02:25:37 Encountered an error:
Runtime Error : dbt encountered an error while trying to read your profiles.yml file.
the error starting on line azure_blob:
The issue is that the outputs: line under azure_blob is indented when it shouldn't be. If you unindent this line you should be good. Make it look like:
azure_blob:
target: dev
outputs:
dev:
type: azure_blob
account_name: ##
account_key: ##
container: ##
prefix: delta_lake
(P.S, there is a typo in the start of your file defaul should be default.)
Hi I just created a dbt project using
dbt init puddle
I have a MySQL database running locally and have defined my profiles.yml as follows
puddle:
target: dev
outputs:
dev:
type: mysql5
server: localhost
port: 3306 # optional
database: puddle
dbname: puddle
schema: puddle
username: test
password: test
driver: MySQL ODBC 8.0 ANSI Driver
prod:
type: mysql5
server: [server/host]
port: [port] # optional
database: [schema] # optional, should be same as schema
schema: [schema]
username: [username]
password: [password]
driver: MySQL ODBC 8.0 ANSI Driver
But when I run dbt run I get the following error
02:48:52 1 of 2 START table model puddle.my_first_dbt_model.............................. [RUN]
02:48:52 1 of 2 ERROR creating table model puddle.my_first_dbt_model..................... [ERROR in 0.11s]
02:48:52 2 of 2 SKIP relation puddle.my_second_dbt_model................................. [SKIP]
02:48:52
02:48:52 Finished running 1 table model, 1 view model in 0.31s.
02:48:52
02:48:52 Completed with 1 error and 0 warnings:
02:48:52
02:48:52 Database Error in model my_first_dbt_model (models/example/my_first_dbt_model.sql)
02:48:52 1046 (3D000): No database selected
02:48:52 compiled SQL at target/run/puddle/models/example/my_first_dbt_model.sql
How am I suppose to select the database??
I am new to CircleCI, and was attempting to setup a project. The initial setup and commit built the configuration file fine, however I needed to use environment variables, so I attempted to setup a context: https://circleci.com/docs/2.0/contexts/. I made a context with my environment variable and named it 'test-context'
Here is my config file:
version: 2.1
orbs:
python: circleci/python#1.2
workflows:
sample:
jobs:
- build-and-test:
context:
- test-context
jobs:
build-and-test:
- image: cimg/python:3.6
parallelism: 4
steps:
- checkout
- python/install-packages:
pkg-manager: pip
- run:
...
I get the following Linter error:
ERROR IN CONFIG FILE:
[#/workflows/sample] only 1 subschema matches out of 2
1. [#/workflows/sample/jobs/0] 0 subschemas matched instead of one
| 1. [#/workflows/sample/jobs/0] expected type: String, found: Mapping
| | SCHEMA:
| | type: string
| | INPUT:
| | build-and-test: null
| | context:
| | - test-context
| 2. [#/workflows/sample/jobs/0/context] expected type: Mapping, found: Sequence
| | SCHEMA:
| | type: object
| | INPUT:
| | - test-context
and - test-context is underlined with the error: Incorrect type. Expected "jobRef"
How can I use contexts to properly integrate environment variables with my project?
#figbar, the way you are writing it on the workflow has syntax issues.
you should correct it as following.
orbs:
python: circleci/python#1.2
workflows:
sample:
jobs:
- build-and-test:
context:
- test-context
jobs:
build-and-test:
docker:
- image: cimg/python:3.6
parallelism: 4
steps:
- checkout
- python/install-packages:
- run:
...
You can get further information regarding the context on this documentation. CircleCI Context
I am quite new to the Elastic stack and trying to experiment with visualization of apache log files in Kibana. I am using filebeat to ingest the apache logs. However when I run .\filebeat.exe setup -e, I get the following error:
2019-02-05T20:53:10.515+0530 INFO elasticsearch/client.go:165 Elasticsearch url: http://localhost:9200
2019-02-05T20:53:10.520+0530 INFO elasticsearch/client.go:721 Connected to Elasticsearch version 6.6.0
2019-02-05T20:53:10.520+0530 INFO kibana/client.go:118 Kibana url: http://localhost:5601
2019-02-05T20:53:10.567+0530 WARN fileset/modules.go:388 X-Pack Machine Learning is not enabled
2019-02-05T20:53:10.572+0530 ERROR instance/beat.go:911 Exiting: 1 error: error loading config file: invalid con
fig: yaml: line 4: did not find expected hexdecimal number
My filebeat.yml file looks like this:
filebeat.inputs:
- type: log
enabled: true
paths: C:\Users\bigdataadmin\Downloads\ApacheLogs\*
#============================= Filebeat modules ===============================
filebeat.config.modules:
path: C:\Program Files\Filebeat\modules.d\*.yml
reload.enabled: true
reload.period: 60s
#==================== Elasticsearch template setting ==========================
setup.template.settings:
index.number_of_shards: 3
setup.kibana:
host: "localhost:5601"
output.elasticsearch:
hosts: ["localhost:9200"]
# Configure processors to enhance or manipulate events generated by the beat.
processors:
- add_host_metadata: ~
- add_cloud_metadata: ~
I also checked the yml on http://www.yamllint.com/ but didn't find any problems. I can't seem to figure out what's wrong with line 4 of this file.
I am using filebeat 6.6
The path key(on line 4) is an array. So you need to represent an array there.
Example :
filebeat.inputs:
- type: log
enabled: true
paths:
- C:\Users\bigdataadmin\Downloads\ApacheLogs\*
Please be very cautious about the data type you are representing in such config files, I had made the same mistake while I was working on Filebeat and I had to spend a lot of time for a small mistake...