I am doing the DBT hello world tutorial found here, and I have created my first project on a windows machine. My profiles.yml file looks like this:
my-bigquery-db:
target: dev
outputs:
dev:
type: bigquery
method: service-account
project: abiding-operand-286102
dataset: xxxDBTtestO1
threads: 1
keyfile: C:\Users\xxx\.dbt\abiding-operand-286102-ed9e3c9c13cd.json
timeout_seconds: 300
when I execute dbt run I get:
Running with dbt=0.17.2 Encountered an error while reading profiles: ERROR Runtime Error
dbt encountered an error while trying to read your profiles.yml
file.
Profile my-bigquery-db in profiles.yml is empty
Defined profiles:
my-bigquery-db
target
outputs
dev
type
method
project
dataset
threads
keyfile
timeout_seconds
Any idea?
At first glance from both your code and the source walkthrough, this is just a yml config problem. YML is a markup language that is white-space sensitive. And by just looking at the example that you may have pulled from - it doesn't look appropriately white spaced to me.
I'm not sure if you can simply copy from the below but it might be worth a shot.
my-bigquery-db:
target: dev
outputs:
dev:
type: bigquery
method: service-account
project: abiding-operand-286102
dataset: xxxDBTtestO1
threads: 1
keyfile: C:\Users\xxx\.dbt\abiding-operand-286102-ed9e3c9c13cd.json
timeout_seconds: 300
Basically - your dbt profile.yml needs to be setup with the sections at certain levels (not unlike python indentation or any other white spacing scheme).
Related
I'm trying to define my serverless framework deployment bucket.
My serverless.yml looks like this:
provider:
name: aws
runtime: nodejs14.x
region: us-east-1
stage: dev
deploymentBucket:
name: ${self:environment.DEPLOYMENT_BUCKET}
environment:
${file(../evn.${opt:stage, 'dev'}.json)}
and the evn.dev.json file looks like this:
{
"DEPLOYMENT_BUCKET": "myBucketName"
}
(both of these files have non-relevant parts removed)
I'm getting a "cannot resolve variable at "provicer.deploymentBucket.name" error when trying to deploy.
How do I reference the DEPLOYMENT_BUCKET variable in the serverless.yml file?
EDIT: Other errors:
${environment}:DEPLOYMENT_BUCKET -> Could not locate deployment bucket. Error: The specified bucket is not valid
name: ${environment:DEPLOYMENT_BUCKET}1 -> Unrecognized configuration variable sources: "environment"
name: ${self:provider.environment:DEPLOYMENT_BUCKET}
and
name: ${self:environment:DEPLOYMENT_BUCKET}
-> Cannot resolve serverless.yml: Variables resolution errored with - Cannot resolve variable at "provider.deploymentBucket.name": Value not found at "self" source
I was able to solve the problem with this:
${file(../evn.${opt:stage, 'dev'}.json):DEPLOYMENT_BUCKET}
But 'reading' that file twice -- both here and in the 'environment' area seems to somewhat defeat the purpose of the environments area.
I'm hitting a strange error in dbt
I am using the following package to enable external tables in bigquery.
packages:
- package: dbt-labs/dbt_external_tables
version: 0.8.0
$ dbt run-operation stage_external_sources
An error occurred in the dbt Server. Please contact support if this issue persists.
RPC server failed to compile project, call the "status" method for compile status: Compilation Error
dbt found two resources with the name
"dbt_user_test_ext". Since these resources have the same name, dbt will be unable to find the correct resource when source("dbt_user", "test_ext") is used. To fix this change the name of one of these resources:
source.dbt_user.test_ext (models/sources.yml)
source.dbt_user.test_ext (models/sources.yml) Code: 10011
models/sources.yml looks something like this:
sources:
- name: "{{ target.schema }}"
tables:
- name: test_ext
external:
location: "gs://test-bucket/test_ext.csv"
options:
format: csv
skip_leading_rows: 1
Seems to only happen the after the first time I run the command.
Question: what am I doing wrong, and/or how to fix?
I tried commenting out some lines from a .gitlab-ci.yml file and the pipeline is failing saying the YAML is not properly formatted.
Below is the picture of what I see in the GitLab CI-Pipeline UI.
Below is the .gitlab-ci.yml file that is error.
stages:
- build
- test
# - release
build-demo-commands:
extends: .npm-job
stage: build
script:
- npm run build-demo-prod
include:
- project: 'myproj/gitlab-ci'
ref: '1.2.0'
file: 'templates/npm.gitlab-ci.yml'
# - project: 'myproj/gitlab-ci'
# ref: '1.2.0'
# file: 'templates/Kaniko-npm.gitlab-ci.yml'
I suspect that there is a pre-processor for the CI-pipeline which supports a comment syntax.
What is the correct way to comment out lines in a .gitlab-ci.yml file?
Searching for answer
I found this Add comments to .gitlab-ci.yml, which was merged two years ago. Now I just need to find the page that says how to do it!
https://docs.gitlab.com/ee/development/code_comments.html - not helpful
https://docs.gitlab.com/ee/development/cicd/templates.html#template-types -
https://docs.gitlab.com/ee/development/cicd/templates.html#explain-the-template-with-comments - Looks like the answer so why does it complain?
https://blog.wplauncher.com/add-comments-to-yaml-file/#:~:text=In%20order%20to%20add%20comments,line%20in%20a%20YAML%20file. This says the # is the way to do comments.
EDIT/UPDATE:
I believe the problem is with the included scripts and not with adding the comments. When I removed the comments I still got an error. The error was:
Found errors in your .gitlab-ci.yml:
npm-deploy-snapshot job: chosen stage does not exist; available stages are .pre
build
test
.post
You can also test your .gitlab-ci.yml in CI Lint
So I now believe that using the hash (#) character is the correct way to comment a .gitlab-ci.yml file.
Comments in YAML use a # character which comments out the remainder of the line
Wikipedia says:
Comments begin with the number sign (#), can start anywhere on a line and continue until the end of the line. Comments must be separated from other tokens by whitespace characters.[16] If # characters appear inside of a string, then they are number sign (#) literals.
I tried to built github .yml file but I’m getting error like
|GitHub Actions/ Main Workflow
Invalid workflow file
The workflow is not valid. .github/workflows/build.yml (Line: 22, Col: 22): Unexpected symbol: '<hash_value>'. Located at position 9 within expression: secrets.<hash_value>|
CODE
on:
Trigger analysis when pushing in master or pull requests, and when creating
a pull request.
push:
branches:
- master
pull_request:
types: [opened, synchronize, reopened]
name: Main Workflow
jobs:
sonarcloud:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v2
with:
# Disabling shallow clone is recommended for improving relevancy of reporting
fetch-depth: 0
- name: SonarCloud Scan
uses: SonarSource/sonarcloud-github-action#v1.3
env:
GITHUB_TOKEN: {{ secrets.<hash_value>}} SONAR_TOKEN: {{ secrets.<hash_value>}}
AND
#Configure here general information about the environment, such as SonarQube server connection details for example
#No information about specific project should appear here
#----- Default SonarQube server
sonar.host.url=https://sonarcloud.io/
#----- Default source code encoding
#sonar.sourceEncoding=UTF-8
sonar.organization=blah blah
sonar.projectKey=blah blah
— optional properties —
defaults to project key
sonar.projectName=Toolsdemo
defaults to ‘not provided’
sonar.projectVersion=1.0
Path is relative to the sonar-project.properties file. Defaults to .
sonar.sources=https://github.com/abcd/xyz
Encoding of the source code. Default is default system encoding
sonar.sourceEncoding=UTF-8
Please help . I need to test my code very fast .
Using filebeat 6.8 open source version, I'm trying to use the field rename feature. I'm not seeing any errors in startup or processing, but the field isn't getting renamed. The logs are JSON formatted. Am I missing something in my config, or is this combination not supported yet?
filebeat.yml
processors:
- rename:
fields:
- from: "a"
to: "b"
filebeat.inputs:
- type: log
enabled: true
json.keys_under_root: true
fields_under_root: true
sample log
{
"a": "blah"
}
Edit:
According to the official documentation, processors can be placed at top level or under an input.
So, perhaps what your configuration is missing is the file paths to prospect.
I'm running filebeat 7.6.2 and I use this feature without any issues, but I'm pretty sure it works on version 6.8 just as well since these fields are documented for this version. See here.
Have you tried:
filebeat.inputs:
- type: log
paths:
- /path/to/your/logs
enabled: true
json.keys_under_root: true
fields_under_root: true
processors:
- rename:
fields:
- from: "a"
to: "b"
Processors should be declared after inputs AFAIK. As for not seeing any errors, I'm not sure how far beyond simple yaml linting filebeat goes when it comes to validating your configuration file.