I'm hitting a strange error in dbt
I am using the following package to enable external tables in bigquery.
packages:
- package: dbt-labs/dbt_external_tables
version: 0.8.0
$ dbt run-operation stage_external_sources
An error occurred in the dbt Server. Please contact support if this issue persists.
RPC server failed to compile project, call the "status" method for compile status: Compilation Error
dbt found two resources with the name
"dbt_user_test_ext". Since these resources have the same name, dbt will be unable to find the correct resource when source("dbt_user", "test_ext") is used. To fix this change the name of one of these resources:
source.dbt_user.test_ext (models/sources.yml)
source.dbt_user.test_ext (models/sources.yml) Code: 10011
models/sources.yml looks something like this:
sources:
- name: "{{ target.schema }}"
tables:
- name: test_ext
external:
location: "gs://test-bucket/test_ext.csv"
options:
format: csv
skip_leading_rows: 1
Seems to only happen the after the first time I run the command.
Question: what am I doing wrong, and/or how to fix?
Related
I'm trying to define my serverless framework deployment bucket.
My serverless.yml looks like this:
provider:
name: aws
runtime: nodejs14.x
region: us-east-1
stage: dev
deploymentBucket:
name: ${self:environment.DEPLOYMENT_BUCKET}
environment:
${file(../evn.${opt:stage, 'dev'}.json)}
and the evn.dev.json file looks like this:
{
"DEPLOYMENT_BUCKET": "myBucketName"
}
(both of these files have non-relevant parts removed)
I'm getting a "cannot resolve variable at "provicer.deploymentBucket.name" error when trying to deploy.
How do I reference the DEPLOYMENT_BUCKET variable in the serverless.yml file?
EDIT: Other errors:
${environment}:DEPLOYMENT_BUCKET -> Could not locate deployment bucket. Error: The specified bucket is not valid
name: ${environment:DEPLOYMENT_BUCKET}1 -> Unrecognized configuration variable sources: "environment"
name: ${self:provider.environment:DEPLOYMENT_BUCKET}
and
name: ${self:environment:DEPLOYMENT_BUCKET}
-> Cannot resolve serverless.yml: Variables resolution errored with - Cannot resolve variable at "provider.deploymentBucket.name": Value not found at "self" source
I was able to solve the problem with this:
${file(../evn.${opt:stage, 'dev'}.json):DEPLOYMENT_BUCKET}
But 'reading' that file twice -- both here and in the 'environment' area seems to somewhat defeat the purpose of the environments area.
I am doing the DBT hello world tutorial found here, and I have created my first project on a windows machine. My profiles.yml file looks like this:
my-bigquery-db:
target: dev
outputs:
dev:
type: bigquery
method: service-account
project: abiding-operand-286102
dataset: xxxDBTtestO1
threads: 1
keyfile: C:\Users\xxx\.dbt\abiding-operand-286102-ed9e3c9c13cd.json
timeout_seconds: 300
when I execute dbt run I get:
Running with dbt=0.17.2 Encountered an error while reading profiles: ERROR Runtime Error
dbt encountered an error while trying to read your profiles.yml
file.
Profile my-bigquery-db in profiles.yml is empty
Defined profiles:
my-bigquery-db
target
outputs
dev
type
method
project
dataset
threads
keyfile
timeout_seconds
Any idea?
At first glance from both your code and the source walkthrough, this is just a yml config problem. YML is a markup language that is white-space sensitive. And by just looking at the example that you may have pulled from - it doesn't look appropriately white spaced to me.
I'm not sure if you can simply copy from the below but it might be worth a shot.
my-bigquery-db:
target: dev
outputs:
dev:
type: bigquery
method: service-account
project: abiding-operand-286102
dataset: xxxDBTtestO1
threads: 1
keyfile: C:\Users\xxx\.dbt\abiding-operand-286102-ed9e3c9c13cd.json
timeout_seconds: 300
Basically - your dbt profile.yml needs to be setup with the sections at certain levels (not unlike python indentation or any other white spacing scheme).
I'm trying to deploy the DVSA Serverless App via Serverless (SLS) and I'm hitting this error:
Serverless: Packaging service...
Serverless Error ---------------------------------------
FeedbackBucket - Bucket name should not contain uppercase characters. Please check provider.s3.FeedbackBucket and/or s3 events of function "FeedbackUploads".
Get Support --------------------------------------------
Docs: docs.serverless.com
Bugs: github.com/serverless/serverless/issues
Issues: forum.serverless.com
Your Environment Information ---------------------------
Operating System: linux
Node Version: 10.21.0
Framework Version: 1.73.1
Plugin Version: 3.6.13
SDK Version: 2.3.1
Components Version: 2.31.2
The repo is here https://github.com/OWASP/DVSA and it looks to be the file https://github.com/OWASP/DVSA/blob/master/backend/src/functions/processing/sls.yml which is causing the issue. With the S3 line - should this be a reference rather than a string? (New to SLS so apologies if this is an obvious question). Thanks!
It looks like the error is line 26 where the bucket name has caps in it: https://github.com/OWASP/DVSA/blob/b26c8a744293cd192383e4a61e0699563505c5a8/backend/src/functions/processing/sls.yml#L26
all
As it said in the title, I just delete all the cloudformation stack and try to do the fresh deploy again. but when I run the sls deploy --verbose, it shows the error
The CloudFormation template is invalid: Template format error:
Unresolved resource dependencies [ServerlessDeploymentBucket] in the
Resources block of the template
For debugging logs, run again after setting the "SLS_DEBUG=*"
environment variable.
I double check my serverless.yml file, but found I don’t define a bucket with Name ServerlessDeploymentBucket.
Then I go to the AWS console, after clicking my root stack (I am using the serverless-split-stack plugin),
there is only a simple template
AWSTemplateFormatVersion: 2010-09-09
Description: The AWS CloudFormation template for this Serverless application
Resources:
ServerlessDeploymentBucket:
Type: 'AWS::S3::Bucket'
Outputs:
ServerlessDeploymentBucketName:
Value: !Ref ServerlessDeploymentBucket
while actually in my origin serverless.yaml, there are more than 1200 lines. Quite wondering what is the error related to the serverless. Appreciated for anyone could help. Thanks.
Edit: there are a bunch of plugin I am using which maybe useful for troubleshooting the error:
serverless-content-encoding
serverless-pseudo-parameters
serverless-webpack
serverless-offline
serverless-plugin-split-stacks
serverless-plugin-custom-roles
serverless-domain-manager
serverless-s3-deploy serverless-plugin-tracing
Regards.
is there any chance you are using template exported from an old stack?
By the way What happens if you just
delete the stack
and just sls deploy?
It will create the template for you (and the stack) and deploy it.
I have the following semver setup:
- name: version
type: semver
source:
driver: gcs
bucket: my-ci
json_key: ((my.serviceaccount))
key: version/version.txt
initial_version: 0.0.0
In my publishjob, I have the following:
name: publish
serial_groups: [version]
plan:
- get: version
passed: [build]
trigger: true
So, basically the publish job is triggered after build job is passed (version updated)
Now, in the publish job I am creating a docker image and pushing it to gcr.
- put: my-gcr
params:
additional_tags: my/ci/tags
build: mycode
get_params: {skip_download: true}
Here, the image is correctly tagged based on the values in the tags file. However, I want to set these values dynamically based on the current version which can be retreived following this:
https://concoursetutorial.com/miscellaneous/versions-and-buildnumbers/#display-version
How can I use this version number to tag my docker image?
I solved it using the following code:
- put: artifacts
params:
additional_tags: version/number
build: mycode
get_params: {skip_download: true}