How to test a relationship in seeds with a source() in dbt - testing

There is any way to test a seed with a relatioship coming from a external source?
I'm trying to test the seeds before creating them, to do this I'm creating in this moment a relationship test but with an external source.
I would like to do something like this in the seeds/schema.yml:
version: 2
seeds:
- name: products_market
description: ""
columns:
- name: id_source
description: "ID source"
- name: id_target
description: "id target"
tests:
- relationship:
to: source('dt', 'products')
field: id_product
In this case the field id_target would be checked if exists the field id_product from dt.products
But I get the error
'test_relationship' is undefined. This can happen when calling a macro that does not exist. Check for typos and/or install package dependencies with "dbt deps".
The sources.yml
version: 2
sources:
- name: dt
tables:
- name: products
description: General Products
I saw this info but not sure if I'm doing it well or maybe there is another way to do this
https://docs.getdbt.com/reference/seed-properties
https://docs.getdbt.com/reference/resource-properties/tests
EDIT:
I was missing an 's' in the test relationships statement in the schema.yml. Anyway it doesn't look i'm doing it properly, it says there is no test or do nothing when I do dbt seed... or dbt build
version: 2
seeds:
- name: products_market
description: ""
columns:
- name: id_source
description: "ID source"
- name: id_target
description: "id target"
tests:
- relationship**s**:
to: source('dt', 'products')
field: id_product
Thanks in advance

Related

Running dbt tests as a pre-hook, prior to running a model

I want to prevent the scenario where my model runs, even though any number of source tables are (erroneously) empty. The phrase coming to mind is a "pre-hook," although I'm not sure that's the right terminology.
Ideally I'd run dbt run --select MY_MODEL and as a part of that, these tests for non-emptiness in the source tables would run. However, I'm not sure dbt works like that. Currently I'm thinking I'll have to apply these tests to the sources and run those tests (according to this document), prior to executing dbt run.
Is there a more direct way of having dbt run fail if any of these sources are empty?
Personally the way I'd go about this would be to define your my_source.yml
to have not_null tests on every column using something like this docs example
version: 2
sources:
- name: jaffle_shop
database: raw
schema: public
loader: emr # informational only (free text)
loaded_at_field: _loaded_at # configure for all sources
tables:
- name: orders
identifier: Orders_
loaded_at_field: updated_at # override source defaults
columns:
- name: id
tests:
- not_null
- name: price_in_usd
tests:
- not_null
And then in your run / build, use the following order of operations:
dbt test --select source:*
dbt build
In this circumstance, I'd highly recommend making your own variation on the generate_source macro from dbt-codegen which automatically defines your sources with columns & not_null tests included.

Home Assistance Script for Automation

Trying to config my first script
My goal is to automate an alert if my heater is in Error… there are many type error … The only state that is good is E-00: OK’
i would like to trigger the script only if the value is <> to state: "E-00: OK’
Is there a way to do that?
Script Yaml
alias: >-
Heater E10
sequence:
condition: state
entity_id: sensor.heater_error_string
state: "E-00: OK’
mode: single
icon: mdi:radiator
Yes, you can create server-side automation script which is triggered only when your text sensor entity value changes from OK to any other.
For example, you may try:
automation:
trigger:
- platform: state
entity_id: sensor.heater_error_string
from:
- "E-00: OK"
action:
- service: notify.mobile_phone_app
data:
message: heater is not ok
title: Heater Notification
mode: single

dbt relationship test compilation error: test definition dictionary must have exactly one key

I'm a new user of dbt, trying to write a relationship test:
- name: PROTOCOL_ID
tests:
- relationships:
to: ref('Animal_Protocols')
field: id
I am getting this error:
Compilation Error
Invalid test config given in models/Animal_Protocols/schema.yml:
test definition dictionary must have exactly one key, got [('relationships', None), ('to', "ref('Animal_Protocols')"), ('field', 'id')] instead (3 keys)
#: UnparsedNodeUpdate(original_file_path='model...ne)
"unique" and "not-null" tests in the same file are working fine, but I have a similar error with "accepted_values".
I am using dbt cli version 0.21.0 with Snowflake on MacOS Big Sur 11.6.
You are very close! I'm 96% sure that this is an indentation issue -- the #1 pain point of working with YAML. The solution is that both to and field need to be indented below the relationships key as opposed to at the same level.
See the Tests dbt docs page for an example
- name: PROTOCOL_ID
tests:
- relationships:
to: ref('Animal_Protocols')
field: id

liquibase-hibernate shows all tables as "unexpected"

I followed these steps to get liquibase-hibernate working. I hope I correctly understood the instructions in the wiki.
Our hibernate entities are declared in the file applicationContext.xml. We do not have a hibernate.cfg.xml. My liquibase properties are:
url=jdbc:postgresql://localhost:1234/MY_DATABASE
username=user
password=pass
referenceUrl=hibernate:spring:somePackage?dialect=org.hibernate.dialect.PostgreSQLDialect
The thing is no matter what I enter as somePackage, liquibase shows everything (tables, columns, constraints) as "unexpected". Liquibase "finds" somePackage even if it does not exist.
liquibase diff
INFO 09.08.17 10:41: liquibase-hibernate: Reading hibernate configuration hibernate:spring:somePackage?dialect=org.hibernate.dialect.PostgreSQLDialect
INFO 09.08.17 10:41: liquibase-hibernate: Found package somePackage
And the comparison result is like
Reference Database: null # hibernate:spring:somePackage?dialect=org.hibernate.dialect.PostgreSQLDialect (Default Schema: HIBERNATE)
Comparison Database: postgres # jdbc:postgresql://localhost:1234/MY_DATABASE (Default Schema: public)
Compared Schemas: HIBERNATE -> public
Product Name:
Reference: 'Hibernate'
Target: 'PostgreSQL'
Product Version:
Reference: '4.3.11.Final'
Target: '9.5.4'
Missing Catalog(s): NONE
Unexpected Catalog(s): NONE
Changed Catalog(s):
HIBERNATE
name changed from 'HIBERNATE' to 'MY_DATABASE'
Missing Column(s): NONE
[...]
Unexpected Table(s):
activityentity
addressentity
advertisemententity
advertisementusageentity
[...]
I really don't know what's going on or whether I'm doing something wrong. Any help would be appreciated.

EventLog & ConvertFrom-String

i am trying to objectify the security event log by using the ConvertFrom-String PowerShell cmdlet, but am not able to work it out.
First i am getting the event/s from my DC.
$events = Get-WinEvent -ComputerName $comp FilterHashtable #{logname='security';id=4727}
Next i define my template.
$tmpl = #'
{Event:A security-enabled global group was created.}
Subject:
Security ID: S-1-5*
{SubjectName:Account Name: andrew}
Account Domain: DOMAIN
Logon ID: 0x16D280EB
New Group:
Security ID: S-1-5*
{GroupName:Group Name: test1}
Group Domain: DOMAIN
Attributes:
SAM Account Name: test1
SID History: -
Additional Information:
Privileges: -
'#
Finally i try to turn it into objects.
($events).message | ConvertFrom-String -TemplateContent $tmpl
But my output is only
Event: A security-enabled global group was created.
Instead, I want to get something like;
Event: A security-enabled global group was created
SubjectName: andrew
GroupName: test1
And i would like this to be compatible to loop through many similar events to pull out the right bits??
I posted the same question in the Microsoft forums and got an immediate answer, and so for those interested, here it is.
https://social.technet.microsoft.com/Forums/windowsserver/en-US/42f8e6a3-4304-4215-b521-d611e3216e1c/eventlog-convertfromstring?forum=winserverpowershell