How do I use multiple selector tags to exclude them from dbt run - dbt

I need to exclude a few selector tags in the dbt run, instead to excluding models which are more in number compared to tags.
I am running
dbt run --fail-fast --profiles-dir . --exclude tag:selector1,tag:selector2,tag:selector3
getting error:
The selection criterion '[tag:selector1,tag:selector2,tag:selector3]' does not match any nodes
my selector entry looks like:
selectors:
- name: selector1
definition:
union:
- intersection:
- method: tag
value: selector1

selectors and tags are two different things. There is no such thing as a "selector tag." It's hard for me to understand exactly what you're trying to do, but to explain the building blocks:
tags are configs that can be applied to any dbt resource (e.g., models, seeds, tests). I can add a tag to a model like this:
-- my_model.sql
{{ config(tags=['tag1', 'tag2']) }}
select 1
I can then run this model (and all other models with tag1) with:
dbt run -s tag:tag1
I can run all models tagged with either tag1 or tag2 by using union syntax (a space):
dbt run -s tag:tag1 tag:tag2
Or I can run only the models tagged with both tag1 and tag2 by using intersection syntax (a comma):
dbt run -s tag:tag1,tag:tag2
If this gets too complicated, instead of typing in this selection syntax every time at the command line, I can define a custom selector in a .yml file, and reference that selector in my CLI command.
Here's a yml selector for either tag1 or tag2 (union):
selectors:
- name: union_tag1_tag2
description: *either* `tag1` *or* `tag2`
definition:
union:
- method: tag
value: tag1
- method: tag
value: tag2
I would then use this selector at the command line like this:
dbt run --selector union_tag1_tag2
And again, for my second example above, for just models with both tag1 and tag2 (intersection):
selectors:
- name: intersection_tag1_tag2
description: *both* `tag1` and `tag2`
definition:
intersection:
- method: tag
value: tag1
- method: tag
value: tag2
Then I use the new name at the command line:
dbt run --selector intersection_tag1_tag2
I can use --exclude with tag:
dbt run --exclude tag:tag1
But I can't use --exclude with a selector. Instead, I define a selector that does the excluding:
selectors:
- name: exclude_intersection_tag1_tag2
description: run all models except those tagged with *both* `tag1` and `tag2`
definition:
exclude:
intersection:
- method: tag
value: tag1
- method: tag
value: tag2
Then I run it with:
dbt run --selector exclude_intersection_tag1_tag2

Related

Gitlab CI only variables that come from extends

I have the following setup (simplified version), which doesn't run the expected merge::my when I use tag that includes the string "TEST". I can't figure out why is it happening - I know that only doesn't support variable expansion, but here the variable is just a string, that is being set up in a different extend - is that a problem? Would using yaml anchors be better? Are there different suggestions?
The reason that I check for only:variable in merge_builds is because I have many languages, in this case I used en, but I have many others, and I don't want to do the only:variables for each (the real matching is more complex - I stripped it to bare minimum for the example)
.merge_builds:
script:
- echo 'test'
only:
variables:
- $CI_COMMIT_TAG =~ $VARIABLEMATCH
.en_variables:
variables:
VARIABLEMATCH: /^$|(?i)EN/
merge::en:
extends:
- .en_variables
- .merge_builds
Based on GitLab issue 35438, I'd say that it is not currently possible to use a variable (as opposed to a literal) as the regular expression pattern.
Within issue 35438, #furkanayhan explains in a comment titled "Introduction" from 2021-09-06 (sorry, I wasn't able to get a permalink to it) that GitLab will make a simple string comparison between a value and a pattern given as a variable:
variables:
teststring: 'abcde'
pattern: '/^ab.*/'
test1:
script: exit 0
rules:
- if: '$teststring =~ $pattern'
test2:
script: exit 0
rules:
- if: '$teststring =~ /^ab.*/'
The test1 job is not created because the backend makes string comparison between "abcde" and "/'^ab.*/".
The test2 job is created because the backend makes regexp comparison between "abcde" and /'^ab.*/.
I believe that you are encountering the same behavior that caused "test1 job" not to be created.
However, issue 35438 shows that GitLab is planning on offering a fix in version 15.0, scheduled for 2022-05-22.
One other thing you might want to check on is the regular expression itself. GitLab's regexp doc (here) states that GitLab uses the re2 regular expression syntax for these kinds of comparison. To achieve case insensitivity, I believe one appends the "i" flag as in:
/pattern/i

DevOps Pipeline Variables cannot be compared

I have a question to DevOps Pipelines.
I have created a Pipeline and during this creation I add a variable with this values “ Name: RESIDENCE ; value: ISS“. So this Value is defined outside from any script.
Pipeline Variable
Inside the .yml file I use this code:
variables:
- name: shuttle
value: Columbia
- name: pipe_var
value: $(RESIDENCE)
- name: location
${{ if eq(variables.pipe_var, 'ISS') }}:
value: pretty_cool
${{ if eq(variables.pipe_var, 'MIR') }}:
value: not_possible
steps:
- script: |
echo space shuttle is: $(shuttle)
echo residence is: $(pipe_var)
echo place to be is: $(location)
But The Pipeline output is only showing:
space shuttel is: Columbia
resicende is: ISS
place to be is:
So as it can be seen in the line “resicende is: ISS “ the value from the outside variable “RESIDENCE” is shown correct. To show this value I use the detour over the variable “pipe_var”. But when I try to compare this variable value with the “if equal” lines then I get no results.
What I’m doing wrong? Is there a special way to compare string values in a pipeline?
It would be nice if someone could give me a hint.

Azure Devops YAML: Looping Jobs With Different Variables Passed

I am struggling to figure out how to execute an API test using a pipeline where the command used can be modified using a loop. For example:
TEMPLATE.yaml
parameters:
JobName: ''
TestDirectory: '.\tests\smoke\'
PositiveTest: ''
NegativeTest: ''
- name: environments
type: object
values:
- dev01
- dev02
- test01
- test02
jobs:
- job: ${{ parameters.JobName }}
pool:
name: pool
demands:
- Cmd
variables:
PosTest: ${{ parameters.PositiveTest }}
NegTest: ${{ parameters.NegativeTest }}
Directory: ${{ parameters.TestDirectory }}
- script: |
call .\venv\Scripts\activate.bat
cd $(Directory)
python $(PosTest)
displayName: 'Executing Positive Test Scenarios'
condition: and(succeeded(), ne('${{ variables.PosTest }}', ''))
- script: |
call .\venv\Scripts\activate.bat
cd $(Directory)
python $(NegTest)
displayName: 'Executing Negative Test Scenarios'
condition: and(succeeded(), ne('${{ variables.NegTest }}', ''))
TEST_FILE.yaml
...
jobs:
# Get this file: templates\TEMPLATE.yml from the `build` repository (imported above)
- template: templates\api-test-build.yml#build
- ${{ each env in parameters.environments }}:
parameters:
TestDirectory: '.\tests\smoke\job_class'
PositiveTest: 'python_test.py http://apient${{ env }}/arbitrary/api/path/name'
NegativeTest: ''
This of course doesn't work (the each directive returns an error like "the first property must be template". If I move it up a line it then says "the first property must be a job" and this cycle of errors just continues...).
The idea is just that I have a loop that iterates through environment strings (top of the TEMPLATE.yaml example). The yaml file that references the template passes the command python_test.py http://apient<whatever env string the current iteration is on>/arbitrary/api/path/name for each iterated string (bottom of TEST_FILE.yaml) and the template just executes each of those api tests. At the end of a run there should be 4 environments that have been tested on.
This is just an idea I have and I am still learned all the in's and out's of Azure Devops YAML. If anyone knows how I can get this to work, any improvements I can make to the idea itself or any other workarounds/solutions that would be highly appreciated. Thank you!
You can try using Multi-job configuration (matrix) in your pipeline.
When you want to run the same job with multi-configuration in a pipeline, the matrix strategy is a good choose.
For example, in your pipeline, you want to run the jobs that have the same steps and input parameters but different values of the input parameters. You can just set up one job with the matrix strategy in the pipeline.

Delimit BigQuery REGEXP_EXTRACT strings in Google Cloud Build YAML script

I have a complex query that creates a View within the BigQuery console.
I have simplified it to the following to illustrate the issue
SELECT
REGEXP_EXTRACT(FIELD1, r"[\d]*") as F1,
REGEXP_REPLACE(FIELD2, r"\'", "") AS F2,
FROM `project.mydataset.mytable`
Now I am trying to automate the creation of the view with cloud build.
I cannot workout how to delimit the strings inside the regex to work with both yaml and SQL.
- name: 'gcr.io/cloud-builders/gcloud'
entrypoint: 'bq'
args: [
'mk',
'--use_legacy_sql=false',
'--project_id=${_PROJECT_ID}',
'--expiration=0',
'--view=
REGEXP_EXTRACT(FIELD1, r"[\d]*") as F1 ,
REGEXP_REPLACE(FIELD2, r"\'", "") AS F2,
REGEXP_EXTRACT(FIELD3, r"\[(\d{3,12}).*\]") AS F3
FROM `project.mydataset.mytable`"
'${_TARGET_DATASET}.${_TARGET_VIEW}'
]
I get the following error
Failed to trigger build: failed unmarshalling build config
cloudbuild/build-views.yaml: json: cannot unmarshal number into Go
value of type string
I have tried using Cloud Build substitution parameters, and as many combinations of SQL and YAML escape sequences as I can think of to find a working solution.
Generally, you want to use block scalars in such cases, as they do not process any special characters inside them and are terminated via indentation.
I have no idea how the command is supposed to look, but here's something that's at least valid YAML:
- name: 'gcr.io/cloud-builders/gcloud'
entrypoint: 'bq'
args:
- 'mk'
- '--use_legacy_sql=false'
- '--project_id=${_PROJECT_ID}'
- '--expiration=0'
- >- # folded block scalar; newlines are folded into spaces
--view=
REGEXP_EXTRACT(FIELD1, r"[\d]*") as F1,
REGEXP_REPLACE(FIELD2, r"\'", "") AS F2,
REGEXP_EXTRACT(FIELD3, r"\[(\d{3,12}).*\]") AS F3
FROM `project.mydataset.mytable`"
'${_TARGET_DATASET}.${_TARGET_VIEW}'
- dummy value to show that the scalar ends here
A folded block scalar is started with >, the following minus tells YAML to not append the final newline to its value.

Test only one it or describe with Rspec

On TestUnit you can launch one test in file with -n option
for example
require 'test_helper'
class UserTest < ActiveSupport::TestCase
test "the truth" do
assert true
end
test "the truth 2" do
assert true
end
end
You can execute only the test the truth
ruby -Itest test/unit/user_test.rb -n test_the_truth
The ouput
1 tests, 1 assertions, 0 failures, 0 errors, 0 skip
How can that with rspec ?
The command seem not work
rspec spec/models/user_spec.rb -e "User the truth"
You didn't include the source of your spec, so it's hard to say where the problem is, but in general you can use the -e option to run a single example. Given this spec:
# spec/models/user_spec.rb
require 'spec_helper'
describe User do
it "is true" do
true.should be_true
end
describe "validation" do
it "is also true" do
true.should be_true
end
end
end
This command line:
rspec spec/models/user_spec.rb -e "User is true"
Will produce this output:
Run filtered including {:full_description=>/(?-mix:User\ is\ true)/}
.
Finished in 0.2088 seconds
1 example, 0 failures
And if you wanted to invoke the other example, the one nested inside the validation group, you'd use this:
rspec spec/models/user_spec.rb -e "User validation is also true"
Or to run all the examples in the validation group:
rspec spec/models/user_spec.rb -e "User validation"
You can also select in which line is the test case you want to execute.
rspec spec/models/user_spec.rb:8
By passing any line inside the scope of the test case, only this test case will be executed. You can also use this to execute a whole context inside your test.
At least in Rspec 2.11.1 you can use all of the following options:
** Filtering/tags **
In addition to the following options for selecting specific files, groups,
or examples, you can select a single example by appending the line number to
the filename:
rspec path/to/a_spec.rb:37
-P, --pattern PATTERN Load files matching pattern (default: "spec/**/*_spec.rb").
-e, --example STRING Run examples whose full nested names include STRING (may be
used more than once)
-l, --line_number LINE Specify line number of an example or group (may be
used more than once).
-t, --tag TAG[:VALUE] Run examples with the specified tag, or exclude examples
by adding ~ before the tag.
- e.g. ~slow
- TAG is always converted to a symbol