I'm a new user of dbt, trying to write a relationship test:
- name: PROTOCOL_ID
tests:
- relationships:
to: ref('Animal_Protocols')
field: id
I am getting this error:
Compilation Error
Invalid test config given in models/Animal_Protocols/schema.yml:
test definition dictionary must have exactly one key, got [('relationships', None), ('to', "ref('Animal_Protocols')"), ('field', 'id')] instead (3 keys)
#: UnparsedNodeUpdate(original_file_path='model...ne)
"unique" and "not-null" tests in the same file are working fine, but I have a similar error with "accepted_values".
I am using dbt cli version 0.21.0 with Snowflake on MacOS Big Sur 11.6.
You are very close! I'm 96% sure that this is an indentation issue -- the #1 pain point of working with YAML. The solution is that both to and field need to be indented below the relationships key as opposed to at the same level.
See the Tests dbt docs page for an example
- name: PROTOCOL_ID
tests:
- relationships:
to: ref('Animal_Protocols')
field: id
Related
I want to prevent the scenario where my model runs, even though any number of source tables are (erroneously) empty. The phrase coming to mind is a "pre-hook," although I'm not sure that's the right terminology.
Ideally I'd run dbt run --select MY_MODEL and as a part of that, these tests for non-emptiness in the source tables would run. However, I'm not sure dbt works like that. Currently I'm thinking I'll have to apply these tests to the sources and run those tests (according to this document), prior to executing dbt run.
Is there a more direct way of having dbt run fail if any of these sources are empty?
Personally the way I'd go about this would be to define your my_source.yml
to have not_null tests on every column using something like this docs example
version: 2
sources:
- name: jaffle_shop
database: raw
schema: public
loader: emr # informational only (free text)
loaded_at_field: _loaded_at # configure for all sources
tables:
- name: orders
identifier: Orders_
loaded_at_field: updated_at # override source defaults
columns:
- name: id
tests:
- not_null
- name: price_in_usd
tests:
- not_null
And then in your run / build, use the following order of operations:
dbt test --select source:*
dbt build
In this circumstance, I'd highly recommend making your own variation on the generate_source macro from dbt-codegen which automatically defines your sources with columns & not_null tests included.
I am trying to run the example "optaplanner-mixedvrp-experiment" developed by Geoffrey De Smet and when I run it it throws me the following error:
Caused by: java.lang.IllegalStateException: The entity (MY) has a
variable (previousStandstill) with value (MUNO) which has a
sourceVariableName variable (nextVisit) with a value (WERBOMONT) which
is not null. Verify the consistency of your input problem for that
sourceVariableName variable.
I have not made any change, I have only cloned and executed it, I import and solve it and it throws me this error.
Do you know what could be happening?
I am applying it in the development of a variant of VRP with multiple deliveries and collections, but it throws me the same error. I have activated the FULL_ASSERT mode and nextVisit, previousStandstill, visitIndex are always null
It's been a long time since I looked at that code, so it's using an old version of optaplanner. Our goal is still to clean it up and offer an out of the box example for VRPPD (and probably remove some boilerplate along the way, using the upcoming #CollectionPlanningVariabe etc). That being said, we have multiple users&customers who used that optaplanner-mixedvrp-experiment to successfully build VRPPD implementations.
Which dataset did you try?
FWIW, that IllegalStateException says that when A.previous = B, the B.next is not A. So either the dataset importer didn't import it correctly - before calling solve() - especially if it fails before the first CH step in FULL_ASSERT. Or one of the custom moves corrupted the model.
I am working on a repository in GitHub and learning to use their Workflows and Actions to execute CI tests. I have created a simple workflow that runs against a shell script to test a simple mathematical expression y-x=expected_val. This workflow isn't that different from other automatic tests I have set up on code in the past, but I cannot figure out how to perform negative test cases.
on:
push:
branches:
- 'Math-Test-Pass*'
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v1
- name: T1. Successful math test
uses: ./.github/actions/mathTest
with:
OPERAND1: 3
OPERAND2: 5
ANSWER: 2
- name: T2. Mismatch answer math test
if: ${{ always() }}
uses: ./.github/actions/mathTest
with:
OPERAND1: -3
OPERAND2: 2
ANSWER: 1
- name: T3. Missing operand math test
if: ${{ always() }}
uses: ./.github/actions/mathTest
with:
OPERAND1: -3
ANSWER: 5
- name: T4. Another test should pass
if: ${{ always() }}
uses: ./.github/actions/mathTest
with:
OPERAND1: 6
OPERAND2: 9
ANSWER: 3
- name: T5. Another test should pass
uses: ./.github/actions/mathTest
with:
OPERAND1: 1
OPERAND2: 9
ANSWER: 8
Now, I expected tests T.2 and T.3 to fail, but I run into two problems. First, I want all the steps to execute and the errors thrown by T.2 and T.3 make the job status a failure. Github's default response is to not run any additional steps unless I force it with something like if: ${{ always() }} This means that T.3 and T.4 only run because of that logic and T.5 doesn't run at all. See below.
The second problem is that while the mathTest action failed on T.2 and T.3 that was the intended behavior. It did exactly what it was supposed to do by failing. I wanted to show that by improperly configuring the parameters the script would fail. These negative pass tests shouldn't show up as failures, but as successes. The whole math test should pass to show that the script in question was prompting the right errors as well as the right answers.
There is a third case that doesn't show here. I definitely don't want to use continue on error. If the script failed to throw an error I want the test case to fail. There should be a failure and then the rest of the tests should continue. My ideal solution would show a pass on T.2 and T.3 and run T.4 and T.5. The same solution would also fail on T.2 or T.3 if they didn't generate an exception and still run T.4 and T.5. I just don't know how to fix that.
I have considered a couple of options but I don't know what is usually done. I expect that while I could jury rig something (e.g. put the failure into the script as another parameter, nest the testing in a second script that passes the parameters and catches the error, etc.), there is some standard way of doing this that I haven't considered. I'm looking for anyone who can tell me how it should be done.
I obtained an answer from the GitHub community that I want to share here.
https://github.community/t/negative-testing-with-workflows/116559
The answer is that the workflow should kick off one of several tools instead of multiple actions and that the tools can handle positive/negative testing on their own. The example given by the respondent is https://github.com/lee-dohm/close-matching-issues/blob/c65bd332c8d7b63cc77e463d0103eed2ad6497d2/.github/workflows/test.yaml#L16 which uses npm for testing.
When doing "DBT run" I get the following error
{{ config(materialized='table') }}
SELECT customer_id FROM `hello-data-pipeline.adwords.google_ads_campaign_stats`
I am making sure that my FROM location contains 3 parts
A project (hello-data-pipeline)
A database (adwords)
A table (google_ads_campaign_stats)
But I get the following error
15:41:51 | 2 of 3 START table model staging_benjamin.yo......................... [RUN]
15:41:51 | 2 of 3 ERROR creating table model staging_benjamin.yo................ [ERROR in
0.32s]
Runtime Error in model yo (models/yo.sql)
404 Not found: Dataset hello-data-pipeline:staging_benjamin was not found in location EU
NB. Bigquery does not show any error when doing this query in Bigquery Editor.
NB 2 DBT does not show any error when "running sql" command directly in the script editor
What I am doing wrong ?
You may need to specify a location where your query will run. Queries that run in a specific location may only reference data in that location. You may choose auto-select to run the query in the location where the data resides.
Read more about Dataset locations
OK I found. I needed to specify the location in the profile.yml file.
=> https://docs.getdbt.com/reference/warehouse-profiles/bigquery-profile/#dataset-locations
In DBT cloud you will find it when setting up your project
I had a similar error to your 'hello-data-pipeline:staging_benjamin was not found in location EU'
However, my issue was not that the dataset was not in the incorrect location. If was that DBT was not targeting the schema I wanted.
e.g. for your example it would be that hello-data-pipeline:staging_benjamin would actually not be the target schema you initially wanted.
Adding this bit of code on top of my query solved the issue.
{{ config(schema='marketing') }}
select ...
cf DBT's schemas: https://docs.getdbt.com/docs/building-a-dbt-project/building-models/using-custom-schemas
here is another doc that helped me understand why this was happening:
"dbt Cloud IDE: The values are defined by your connection and credentials. To check any of these values, head to your account (via your profile image in the top right hand corner), and select the project under "Credentials".
https://docs.getdbt.com/reference/dbt-jinja-functions/target
I followed these steps to get liquibase-hibernate working. I hope I correctly understood the instructions in the wiki.
Our hibernate entities are declared in the file applicationContext.xml. We do not have a hibernate.cfg.xml. My liquibase properties are:
url=jdbc:postgresql://localhost:1234/MY_DATABASE
username=user
password=pass
referenceUrl=hibernate:spring:somePackage?dialect=org.hibernate.dialect.PostgreSQLDialect
The thing is no matter what I enter as somePackage, liquibase shows everything (tables, columns, constraints) as "unexpected". Liquibase "finds" somePackage even if it does not exist.
liquibase diff
INFO 09.08.17 10:41: liquibase-hibernate: Reading hibernate configuration hibernate:spring:somePackage?dialect=org.hibernate.dialect.PostgreSQLDialect
INFO 09.08.17 10:41: liquibase-hibernate: Found package somePackage
And the comparison result is like
Reference Database: null # hibernate:spring:somePackage?dialect=org.hibernate.dialect.PostgreSQLDialect (Default Schema: HIBERNATE)
Comparison Database: postgres # jdbc:postgresql://localhost:1234/MY_DATABASE (Default Schema: public)
Compared Schemas: HIBERNATE -> public
Product Name:
Reference: 'Hibernate'
Target: 'PostgreSQL'
Product Version:
Reference: '4.3.11.Final'
Target: '9.5.4'
Missing Catalog(s): NONE
Unexpected Catalog(s): NONE
Changed Catalog(s):
HIBERNATE
name changed from 'HIBERNATE' to 'MY_DATABASE'
Missing Column(s): NONE
[...]
Unexpected Table(s):
activityentity
addressentity
advertisemententity
advertisementusageentity
[...]
I really don't know what's going on or whether I'm doing something wrong. Any help would be appreciated.