I have a rule with a schema function. The rule works fine in that it's identifying the problem, but it's also highlighting the entire object that fails the schema. Instead, I'd like it to highlight just the specific field inside the object that's causing the problem.
I've tried adding the field property, but that just causes the rule to not run (without error).
if-without-then-or-else:
description: if requires then or else.
severity: error
given:
- $
- $..[?(#.if)]
then:
- function: schema
functionOptions:
schema:
if:
required:
- if
then:
anyOf:
- required:
- then
- required:
- else
{
"if": {}
}
I ended up inverting the schema so that it tests the other way around.
if-requires-then-or-else:
description: 'The `if` keyword will be ignored unless an adjacent `then` and/or `else` keyword is present.'
severity: error
given:
- $
- $..[?(#.if)]
then:
- function: schema
functionOptions:
schema:
if:
not:
anyOf:
- required:
- then
- required:
- else
then:
properties:
if: false
It's basically saying, "If either then or else are not present, then if is also disallowed.
This way, it targets the if as the thing that shouldn't be there.
Related
The default names that dbt chooses for a test can be very long and when it's too long, dbt chooses to hash the last part.
Example:
dbt_expectations_source_expect_table_row_count_to_equal_other_table_exponea_purchase_ref_test_exponea_sdv_orders_v___eventoccuredtime_yesterday_timestamp_AND_eventoccuredtime_today_timestamp___timestamp_yesterday_timestamp_AND_timestamp_today_timestamp_
How can I rename a dbt test (give a custom name) so that it is much more clear in the logs what a test was doing? (when it failed)
You can use the name: keyword to give a test a custom name:
tests:
- dbt_expectations.expect_table_row_count_to_be_between:
name: exponea_purchase_row_count_yesterday_at_least_2500
min_value: 2500
row_condition: "timestamp >= {{ yesterday_timestamp() }}"
However for the not_null generic test this doesn't seem to work, so then you have to use name for the custom name and with test_name you can specify that you want not_null:
columns:
- name: timestamp
description: the view is partitioned on this column
tests:
- name: exponea_purchase_timestamp_not_null
test_name: not_null
See also the dbt documentation here:
https://docs.getdbt.com/reference/resource-properties/tests#define-a-custom-name-for-one-test
Or the explanation on the forum here:
https://discourse.getdbt.com/t/customizing-dbt-test-output-name/5550
i would like to add new field extracted from the path what will be used. I have two path, see below.
paths:
- /home/*/app/logs/*.log
# - /home/v209/app/logs/*.log
# - /home/v146/app/logs/*.log
fields:
campaign: v209
fields_under_root: true
i would like to create new field campaign only with folder name like v209 or v146 any idea, how to do this in filebeads?
Thank you in advance!
Here are three suggested solutions tested with Filebeat 7.1.3
1) Static configuration of campaign field per input
filebeat.inputs:
- type: filestream
id: v209
paths:
- "/home/v209/app/logs/*.log"
fields:
campaign: v209
fields_under_root: true
- type: filestream
id: v146
paths:
- "/home/v146/app/logs/*.log"
fields:
campaign: v146
fields_under_root: true
output.console:
pretty: true
Explanation: This solution is simple. Each file input will have a field set (campaign) based on a static config.
Pros/Cons: This option has the problem of having to add a new campaign field every time you add a new path. For dynamic environments, this can pose a serious operational problem but it's dead simple to implement.
2) Dynamically extract campaign name from file path
processors:
- dissect:
tokenizer: "/%{key1}/%{campaign}/%{key3}/%{key4}/%{key5}"
field: "log.file.path"
target_prefix: ""
- drop_fields:
when:
has_fields: ['key1','key3','key4','key5']
fields: ['key1','key3','key4','key5']
Explanation: These processors work on top of your filestream or log input messages. The dissect processor will tokenize your path string and extract each element of your full path. The drop_fields processor will remove all fields of no interest and only keep the second path element (campaign id).
Pros/Cons: Assuming your path structures are stable, with this solution you don't have to do anything when new files appear under /home/*/app/logs/*.log
3) Script your way around
If you wish to setup a more custom parsing logic, I'd suggest trying out the script processor and hack your way until your requirements are met:
https://www.elastic.co/guide/en/beats/filebeat/7.17/processor-script.html
I'm trying to create a rule to enforce the usage of related_name when defining a ForeignKey field in Django, as well as its format.
Enforcing the usage of the field is done by the following rule:
- id: python.custom.foreign-key-must-set-related-name
message: ForeignKey relationships must explicitly set the `related_name` property as `<model>_<field>_set`
languages: [python]
severity: ERROR
patterns:
- pattern-inside: |
class $M(...):
...
- pattern-not: $F = django.db.models.ForeignKey(..., related_name=..., ...)
- pattern: $F = django.db.models.ForeignKey(...)
My goal now is to modify this rule to also be able to validate the format of the related_name, so that it is: <model>_<field>_set (or, following the variables in the rule: $M_$F_set)
I've been trying several combinations and haven't been lucky so far.
Is this possible to do?
Edit:
As an example, look at the following model:
class MyCoolModel(models.Model):
correct_field = models.ForeignKey(to='something', related_name='mycoolmodel_correct_field_set')
incorrect_field = models.ForeignKey(to='something', related_name='something_set')
In karate version 0.9.6 I used the following match statement in a .feature file and it worked for validating the value to be an empty array or a key that was not present.
def example = {}
match example.errors == '##[0]'
In 1.0 the documentation example suggests that this should check for the key being present and null or an empty array and testing this fails with a validation error that the value is not present.
From https://karatelabs.github.io/karate/#schema-validation
# should be null or an array of strings
* match foo == '##[] #string'
This appears to be an undocumented breaking change from pre-1.0 to 1.0.
My question is: how do I construct a validator to cover this case correctly when the key is allowed to be absent but if it is present it must be an empty array?
I've found an undesirable solution for now but am leaving this open in case someone has a better answer.
I'm validating the entire parent object with a minimal schema:
Replace
match $.errors == '##[0]'
With
* match $ == { data: '#object', extensions: '##object', errors: '##[0]' }
While more brittle and verbose it is technically working.
This indeed looks like an in-intended breaking change. Here is another workaround:
* def example = {}
* def expected = example.errors ? '#[0]' : '#notpresent'
* match example.errors == expected
I see you have opened an issue here: https://github.com/karatelabs/karate/issues/1825
EDIT: this might be an improvement over the workaround you came up with in your answer:
* match example contains { errors: '##[0]' }
I recently refactored my declartive pipeline into a scripted form. Even if everything seems to work fine, I have a problem coming from the initialization of a multiple valued paramenter.
In my declarative pipeline, I was using the following definition of multiple valued parameter (which was working as it should):
parameters {
choice(choices: ['fix', 'major', 'minor', 'none'], description: "Increase version's number: MAJOR.MINOR.FIX", name: "VERSIONING")
}
I refactored it into this form for the scripted pipeline:
properties([
parameters([
choice(choices: ['fix\nmajor\nminor\nnone'], description: "Increase version's number: MAJOR.MINOR.FIX", name: "VERSIONING"),
]),
])
The problem is that when I realised that something wasn't working as it should, and printed the variable value with a sh """echo "Versioning parameter check:" ${params.VERSIONING}""" step, I got this from the Jenkins' console:
Versioning parameter check: false
Which is both a value not in the list, and of a different type (boolean instead of string).
Is there a way to implement multiple value parameter initialization in Jenkins scripted pipelines?
Why this directive doesn't work out of the box in the scripted pipeline, whereas it does in the declarative type?
Is this a bug or am I doing something wrong?
Your definition is absolutely fine. You just need to pass the choices as list items and not as \n seperated values.
properties([
parameters([
choice(choices: ['fix', 'major', 'minor', 'none'], description: "Increase version's number: MAJOR.MINOR.FIX", name: "VERSIONING"),
])
])
Try the other option for defining choice parameter:
properties([
parameters([
[$class: 'ChoiceParameterDefinition',
choices: 'fix\nmajor\nminor\nnone\n',
name: 'VERSIONING',
description: "Increase version's number: MAJOR.MINOR.FIX"
],
]),
])
We use the following for a choice parameter, so it looks like your own definition but without the brackets:
properties([
parameters([
choice(choices: 'fix\nmajor\nminor\nnone', description: "Increase version's number: MAJOR.MINOR.FIX", name: "VERSIONING"),
]),
])
If you want the default value to be empty, just add an empty first choice like this:
choice(choices: '\nfix\nmajor\nminor\nnone'