Hello Could anyone help me how to simulate this scenario. Example I want to validate these 3 fields on my table "symbol_type", "symbol_subtype", "taker_symbol" and return unique combination/result.
I tried to use this command, however Its not working properly on my test. Not sure if this is the correct syntax to simulate my scenario. Your response is highly appreciated.
Expected Result: These 3 fields should return my unique combination using DBT commands.
I'd recommend to either:
use the generate_surrogate_key (docs) macro in the model, or
use the dbt_utils.unique_combination_of_columns (docs) generic test.
For the first case, you would need to define the following in the model:
select
{{- dbt_utils.generate_surrogate_key(['symbol_type', 'symbol_subtype', 'taker_symbol']) }} as hashed_key_,
(...)
from your_model
This would create a hashed value of the three columns. You could then use a unique test in your YAML file.
For the second case, you would only need to add the generic test in your YAML file as follows:
# your model's YAML file
- name: your_model_name
description: ""
tests:
- dbt_utils.unique_combination_of_columns:
combination_of_columns:
- symbol_type
- symbol_subtype
- taker_symbol
Both these approaches will let you check whether the combination of the three columns is unique over the whole model's output.
Related
I am trying to setup a singular test in dbt (it’s a test for one specific table - TableA), so I wrote an SQL query which I placed in tests folder. It returns failing rows.
However, when I run dbt test —-select tableA, in case the test passes (no failing records), I get the following error:
14:20:57 Running dbt Constraints
14:20:58 Database error while running on-run-end
14:20:59 Encountered an error:
Compilation Error in operation dbt_constraints-on-run-end-0 (./dbt_project.yml)
'dbt.tableA.graph.compiled.CompiledSingularTestNode object' has no attribute 'test_metadata’
In case the test fails, it returns the failing rows, which is correct behaviour.
I am using dbt_constraints package (v0.3.0), which seems to be causing this problem, specifically this script which runs in the on-run-end hook https://github.com/Snowflake-Labs/dbt_constraints/blob/main/macros/create_constraints.sql
I am guessing I need to add some test metadata to the singular test, but not sure how to do it.
Here is what the test looks like
tests/table_a_test.sql
SELECT *
FROM {{ ref('TableA') }}
WHERE param_1 NOT IN
(SELECT TableB_id
FROM {{ ref('TableB') }}
UNION
SELECT TableC_id
FROM {{ ref('TableC') }}
UNION
SELECT TableD_id
FROM {{ ref('TableD') }}
UNION
SELECT TableE_id
FROM {{ ref ('TableE') }} )
and param_2 is null
Thank you!
This seems to be a bug in that package; I would open an issue in the dbt-constraints repo. There is no documented way to add metadata to a Singular test, but that code assumes that all tests will have test_metadata.name.
I doubt this would work, but what happens if you add a schema.yml file to the tests directory, alongside your singular test? The contents would look like:
version: 2
tests:
- name: table_a_test
sounds like your call should be dbt test —-select table_a_test instead of dbt test —-select tableA. I think, you need to call the test name not the table name, which is already hard coded in the (singular) test. does that work?
Have you tried to run the test with a + sign in front of it? Since you are using ref in the test, you might need to build everything before test.
We're working on converting our Classic Azure Pipelines to YAML Pipelines. One thing that is not clear is how to ensure that two different variable groups with variables with the same name but different meaning don't step on each other.
For example, if I have variable groups vg1 and vg2, each with variable named secretDataDestination, how do I ensure that the correct secretDataDestination is used in a YAML Pipeline?
A more concerning example is, if we initially have two variable groups without overlapping variable names, how do we ensure that adding a newly-overlapping variable name to a group doesn't replace use of the variable as originally intended?
A workaround is leveraging output variables in Azure DevOps with some small inline PowerShell task code.
First, create 2 jobs. Each job with their own variable group, in this case Staging and Prod. Both groups contain the variables apimServiceName and apimPrefix. Add the variables as a job output by echoing them as isOutput=true like this:
- job: StagingVars
dependsOn:
variables:
- group: "Staging"
steps:
- powershell: >-
echo "##vso[task.setvariable variable=apimServiceName;isOutput=true]$(apimServiceName)"
echo "##vso[task.setvariable variable=apimPrefix;isOutput=true]$(apimPrefix)"
name: setvarStep
- job: ProdVars
dependsOn:
variables:
- group: "Prod"
steps:
- powershell: >-
echo "##vso[task.setvariable variable=apimServiceName;isOutput=true]$(apimServiceName)"
echo "##vso[task.setvariable variable=apimPrefix;isOutput=true]$(apimPrefix)"
name: setvarStep
Then, use the variables in a new job, where you specify a new variable name and navigate to the job output to get a value, this works because the variable groups are each placed into their own job, so they will not overwrite any variable:
- job:
dependsOn:
- StagingVars
- ProdVars
variables:
ServiceNameSource: "$[ dependencies.StagingVars.outputs['setvarStep.apimServiceName'] ]"
UrlprefixSource: "$[ dependencies.StagingVars.outputs['setvarStep.apimPrefix'] ]"
ServiceNameDestination: "$[ dependencies.ProdVars.outputs['setvarStep.apimServiceName'] ]"
UrlprefixDestination: "$[ dependencies.ProdVars.outputs['setvarStep.apimPrefix'] ]"
if I have variable groups vg1 and vg2, each with variable named secretDataDestination, how do I ensure that the correct secretDataDestination is used in a YAML Pipeline?
Whether we use classic mode or YAML, it is not recommended to define a variable with the same name in different variable groups. Because when you refer to different variable groups containing the same variable name in the same pipeline, you cannot avoid step on each other.
When you use the same variable name in different variable group in the same pipeline, just like Matt said,
"You can reference multiple variable groups in the same pipeline. If
multiple variable groups include the same variable, the variable group
included last in your YAML file will set the variable's value."
variables:
- group: variable-group1
- group: variable-group2
That means that the variable value in the variable group written later will overwrite the variable value in the variable group written first
I guess you already know this, so you post your second question. Let us now turn to the second question.
if we initially have two variable groups without overlapping variable
names, how do we ensure that adding a newly-overlapping variable name
to a group doesn't replace use of the variable as originally intended?
Indeed, Azure devops currently does not have such a function or mechanism to intelligently detect whether different variable groups have the same variable name, and give a prompt.
I think this is a reasonable request, I add your request for this feature on our UserVoice site which is our main forum for product suggestions:
The ability to detect the same variable in a variable group
As workaround, the simplest and most direct way is that open the variable group of your pipeline link in the Library tab, and directly ctrl + F to search for the existence of the same variable.
Another way is to use REST API Variablegroups - Get Variable Groups By Id to get all the variables, then the loop compares with the variable we are going to enter whether the same variable exists.
Is there any easy way to handle huge query param like below. Also I would like to know how can I do run time parameterisation for some values?
http://154.213.196.243:7941/v1/banking/Jumio/callback?callBackType=NetVerifyId&jumioIdScanReference=123abcde-1244-8571-3454-abcd12345567&merchantIdScanReference=66a9ff2e-d8ec-e811-a956-000d3ab3f117&verificationStatus=APPROVED_VERIFIED&idScanStatus=SUCCESS&id+ScanSource=API&idCheckDataPositions=OK&idCheckDocumentValidation=OK&idCheckHologram=OK&idCheckMRZcode=OK&idCheckMicroprint=OK&idCheckSecurityFeatures=OK&idCheckSignature=OK&transactionDate=2018-11-20T20%3A53%3A25.797Z&callbackDate=2018-11-20T20%3A53%3A25.797Z&idType=DRIVING_LICENSE&idCountry=GBR&idScanImage+=https%3A%2F%2Fnetverify.com%2Frecognition%2Fv1%2Fidscan%2F123abcde-1244-8571-3454-abcd12345567%2Ffront&idFirstName=ILARIA&idLastName=FURS&idDob=1976-12-23&idExpiry=2025-12-31&personalNumber=123456789&clientIp=xxx.xxx.xxx.xxx&idAddress=%7B%22country%22%3A%22USA%22%2C%20%22stateCode%22%3A%22US-OH%22%7D&idNumber=P12345&idStatus=TESTER961260SS9DL54&identityVerification=%7B%22similarity%22%3A%22MATCH%22%2C%22validity%22%3Atrue%7D HTTP/1.1
Yes. Read the docs: https://github.com/intuit/karate#param
For example:
* param callBackType = 'NetVerifyId'
and so on. And look at params where you can set all keys up as one single JSON and also do parameterization if needed, there are multiple possibilities: https://github.com/intuit/karate#params
See this example as well: dynamic-params.feature
I have a response with hundreds of attributes while matching the attributes the scripts getting failed and further steps are not getting executed. because of this we have to validate the same case multiple times to validate the attribute values. is they a option like #ContinueNextStepsOnException to execute all the steps and it is hard to script using karate.match(actual, expected) for more than 100 attributes I have give actual and expected values if in case of any failure to continue.
No, there is no such option. If your scripts are getting failed - it is because Karate is doing its job correctly !
If you feel you want to skip certain fields, you can easily do so by using match ... contains syntax.
I think you are using multiple lines instead of matching the entire JSON in one-line which you can easily do in Karate. For example:
* def response = { a: 1, b: 2 }
# not recommended
* match response.a == 1
* match response.b == 2
# recommended
* match response == { a: 1, b: 2 }
Is it so hard to create the above match, even in development mode ? Just cut and paste valid JSON, and you are done ! I have hardly ever heard users complain about this.
Please forgive me if my question does not make sense.
What im trying to do is to inject in values for query parameters
GET1 File
Scenario:
Given path 'search'
And param filter[id] = id (default value or variable from another feature file)
POST1 File
Scenario:
def newid = new id made by a post call
def checkid = read call(GET1) {id : newid}
like if one of my feature files creates a new id then i want to do a get call with the above scenario. therefore i need a parameter there which takes in the new id.
On the other hand if i do not have an id newly created or the test creating it is not part of the suite. i want to still be able to run the above mentioned scenario but this time it has a default value to it.
Instead of param use params. It is designed so that any keys with null values are ignored.
After the null is set on the first line below, you can make a call to another feature, and overwrite the value of criteria. If it still is null, no params will be set.
* def criteria = null
Given path 'search'
And params { filter: '#(criteria)' }
There are multiple other ways to do this, also refer to this set of examples for data-driven search params: dynamic-params.feature
The doc on conditional logic may also give you some ideas.