how to use an "or" statement to make a dependent job in Gitlab CI? - gitlab-ci

I have a pipeline I'm putting together that has three different "deploy" steps, each with its own unique deployment but can be triggered by the same job. Ideally, I would like to find a way to "or" the items inside of the needs section to make the job automatically run after one of those previous jobs completes.
I know I can create a separate job for each job to "run" but I'd like to avoid repeating myself if possible.
Int (Dry Run):
extends: .stageBatchDryRunJob
stage: Deploy Non-Prod
except:
variables:
- $CI_COMMIT_TAG =~ /^v[0-9]+\.[0-9]+\.[0-9]+$/
variables:
<<: *hubNonVars
kube_cluster_id: HubInt
kube_env: int
environment: int
Int (Rollback):
extends: .stageBatchRollbackJob
stage: Deploy Non-Prod
except:
variables:
- $CI_COMMIT_TAG =~ /^v[0-9]+\.[0-9]+\.[0-9]+$/
variables:
<<: *hubNonVars
kube_cluster_id: HubInt
kube_env: int
environment: int
I would like to have a single "run" job that requires one of these two above jobs to be completed.
Int (run):
extends: .run-batch
variables:
TOWER_JOB_TEMPLATE: $TOWER_JOB_TEMPLATE_INT
kube_cluster_id: HubInt
kube_env: int
needs: [Int (Dry Run), Int (Rollback)] # can this be "or-ed?" IE needs Int (Dry Run) or Int (Rollback)

If both jobs actually exist in the pipeline, no. There is no way to do what you describe with a single job and have the downstream job start immediately.
However, if only one of the two jobs will exist in the pipeline (for example one is excluded by a rules configuration), you can leverage the needs optional feature.
needs:
- job: "Int (Dry Run)"
optional: true
- job: "Int (Rollback)"
optional: true
Of course this introduces the possibility that the job could run when neither of these jobs exist. So, you may want to run a check in your job to see if expected artifacts or variables exist.
You can make the two-job approach a bit more DRY, too, you can use !reference tags or other YAML features like anchors/extends (which you're already using in your provided code).

Related

Running dbt tests as a pre-hook, prior to running a model

I want to prevent the scenario where my model runs, even though any number of source tables are (erroneously) empty. The phrase coming to mind is a "pre-hook," although I'm not sure that's the right terminology.
Ideally I'd run dbt run --select MY_MODEL and as a part of that, these tests for non-emptiness in the source tables would run. However, I'm not sure dbt works like that. Currently I'm thinking I'll have to apply these tests to the sources and run those tests (according to this document), prior to executing dbt run.
Is there a more direct way of having dbt run fail if any of these sources are empty?
Personally the way I'd go about this would be to define your my_source.yml
to have not_null tests on every column using something like this docs example
version: 2
sources:
- name: jaffle_shop
database: raw
schema: public
loader: emr # informational only (free text)
loaded_at_field: _loaded_at # configure for all sources
tables:
- name: orders
identifier: Orders_
loaded_at_field: updated_at # override source defaults
columns:
- name: id
tests:
- not_null
- name: price_in_usd
tests:
- not_null
And then in your run / build, use the following order of operations:
dbt test --select source:*
dbt build
In this circumstance, I'd highly recommend making your own variation on the generate_source macro from dbt-codegen which automatically defines your sources with columns & not_null tests included.

Groking the parallel-matrix

This seems to be a mis-use of the new Parallel Matrix feature in GitLab 13.3 (https://docs.gitlab.com/ee/ci/yaml/#parallel-matrix-jobs)
I have a collection of parallel jobs for a set of services: build (docker image), test, release, delete.... and the code-base is created so that each parallel service is in a separate sub-directory.
This way I can have a common template:
variables:
IMAGE_NAME: $CI_REGISTRY_IMAGE/$LOCATION
.build-template:
script:
- docker build --tag $IMAGE_NAME:$CI_PIPELINE_ID-$CI_COMMIT_REF_SLUG --tag $IMAGE_NAME:latest $LOCATION
stage: build
when: manual
then multiple build jobs:
build-alpha:
extends: .build-template
variables:
LOCATION: alpha
build-beta:
extends: .build-template
variables:
LOCATION: beta
.... and repeat as needed.
I can then do the same thing for test, release, and delete jobs: a common template that takes just the one variable to distinguish the service.
Matrix to the rescue?
It would seem to me that
variables:
IMAGE_NAME: $CI_REGISTRY_IMAGE/$NOTEBOOK
build-services:
parallel:
matrix:
- LOCATION: alpha
- LOCATION: beta
script:
- docker build --tag $IMAGE_NAME:$CI_PIPELINE_ID-$CI_COMMIT_REF_SLUG --tag $IMAGE_NAME:latest $LOCATION
stage: build
when: manual
would be an ideal candidate for this matrix form.... but apparently matrix requires two variables.
Has anyone got a good solution for this multiple-parallel jobs problem?
would be an ideal candidate for this matrix form.... but apparently matrix requires two variables.
Not anymore.
See GitLab 13.5 (October 2020)
Allow one dimensional parallel matrices
Previously, the parallel: matrix keyword, which runs a matrix of jobs in parallel, only accepted two-dimensional matrix arrays. This was limiting if you wanted to specify your own array of values for certain jobs.
In this release, you now have more flexibility to run your jobs the way that works best for your development workflow.
You can run a parallel matrix of jobs in a one-dimensional array, making your pipeline configuration much simpler. Thanks Turo Soisenniemi for your amazing contribution!
Here’s a basic example of this in practice that will run 3 test jobs for different versions of Node.js, but you can apply this approach to your specific use cases and easily add or remove jobs in your pipeline as well:
See Documentation and Issue.
And with GitLab 13.10 (March 2021)
Use 'parallel: matrix' with trigger jobs
You can use the parallel: matrix keywords to run jobs multiple times in parallel, using different variable values for each instance of the job.
Unfortunately, you could not use it with trigger jobs.
In this release, we’ve expanded the parallel matrix feature to support trigger jobs as well, so you can now run multiple downstream pipelines (child or multi-project pipelines) in parallel using different variables value for each downstream pipeline.
This lets you configure CI/CD pipelines that are faster and more flexible.
See Documentation and Issue.
This is a known issue by Gitlab.
There is a workaround using a "dummy" value for the second value (I'm working with Gitlab 13.3.1).
For your example, it gives this :
parallel:
matrix:
- LOCATION: ['alpha', 'beta']
DUMMY: 'dummy'
Did you try:
parallel:
matrix:
- LOCATION: [alpha, beta]

How to create a negative test case in GitHub

I am working on a repository in GitHub and learning to use their Workflows and Actions to execute CI tests. I have created a simple workflow that runs against a shell script to test a simple mathematical expression y-x=expected_val. This workflow isn't that different from other automatic tests I have set up on code in the past, but I cannot figure out how to perform negative test cases.
on:
push:
branches:
- 'Math-Test-Pass*'
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v1
- name: T1. Successful math test
uses: ./.github/actions/mathTest
with:
OPERAND1: 3
OPERAND2: 5
ANSWER: 2
- name: T2. Mismatch answer math test
if: ${{ always() }}
uses: ./.github/actions/mathTest
with:
OPERAND1: -3
OPERAND2: 2
ANSWER: 1
- name: T3. Missing operand math test
if: ${{ always() }}
uses: ./.github/actions/mathTest
with:
OPERAND1: -3
ANSWER: 5
- name: T4. Another test should pass
if: ${{ always() }}
uses: ./.github/actions/mathTest
with:
OPERAND1: 6
OPERAND2: 9
ANSWER: 3
- name: T5. Another test should pass
uses: ./.github/actions/mathTest
with:
OPERAND1: 1
OPERAND2: 9
ANSWER: 8
Now, I expected tests T.2 and T.3 to fail, but I run into two problems. First, I want all the steps to execute and the errors thrown by T.2 and T.3 make the job status a failure. Github's default response is to not run any additional steps unless I force it with something like if: ${{ always() }} This means that T.3 and T.4 only run because of that logic and T.5 doesn't run at all. See below.
The second problem is that while the mathTest action failed on T.2 and T.3 that was the intended behavior. It did exactly what it was supposed to do by failing. I wanted to show that by improperly configuring the parameters the script would fail. These negative pass tests shouldn't show up as failures, but as successes. The whole math test should pass to show that the script in question was prompting the right errors as well as the right answers.
There is a third case that doesn't show here. I definitely don't want to use continue on error. If the script failed to throw an error I want the test case to fail. There should be a failure and then the rest of the tests should continue. My ideal solution would show a pass on T.2 and T.3 and run T.4 and T.5. The same solution would also fail on T.2 or T.3 if they didn't generate an exception and still run T.4 and T.5. I just don't know how to fix that.
I have considered a couple of options but I don't know what is usually done. I expect that while I could jury rig something (e.g. put the failure into the script as another parameter, nest the testing in a second script that passes the parameters and catches the error, etc.), there is some standard way of doing this that I haven't considered. I'm looking for anyone who can tell me how it should be done.
I obtained an answer from the GitHub community that I want to share here.
https://github.community/t/negative-testing-with-workflows/116559
The answer is that the workflow should kick off one of several tools instead of multiple actions and that the tools can handle positive/negative testing on their own. The example given by the respondent is https://github.com/lee-dohm/close-matching-issues/blob/c65bd332c8d7b63cc77e463d0103eed2ad6497d2/.github/workflows/test.yaml#L16 which uses npm for testing.

Salt Stack - Best way to get schedule output

I'm using saltstack on my servers. I did a simple schedule:
job1:
schedule.present:
- function: state.apply
- seconds: 1800
- splay: 5
Now I want to get the output of the schedule back on my master
(or on my minion but I just like to know the best way)
I don't really know how to use salt mine or returners or what is best preferred for my needs.
thank you :)
Job data return and job metadata
By default, data about jobs runs from the Salt scheduler is returned to the master.
It can therefore be useful to include specific data to differentiate a job from other jobs. By using the metadata parameter a value can be associated with a scheduled job, although these metadata values aren't used in the execution of the job they can be used to search for specific job later.
job1:
schedule.present:
- function: state.apply
- seconds: 1800
- splay: 5
- metadata:
foo: bar
An example job search on the salt-master might look similar to this:
salt-run jobs.list_jobs search_metadata='{"foo": "bar"}'
Take a look at the salt.runners.jobs documentation for more examples.
Scheduler with returner
The schedule.present function includes a parameter to set a returner to use to return the results of the scheduled job. You could for example use the pushover returner:
job1:
schedule.present:
- function: state.apply
- seconds: 1800
- splay: 5
- returner: pushover
Take a look at the returners documentation for a full list of of builtin returners and usage information.

How to give a slave/node as a dynamic parameter in hudson?

I have a list of jobs(say 20) in hudson, which are run in sequence(Job1,2,3,....20) and which are parameterized(parameters given for job1 are passed to other jobs) .
All the jobs run on a node, say 'A'.Now if i wan't to run the same 20 jobs next time on server 'B', I have to go to each job's configuration matrix and change the node from 'A' to 'B'. Since I have 20 jobs, I've to do this tedious job of changing the node 20 times. Is there a way to give the node as a parameter when starting job1, so that i don't have to do put in a lot of effort everytime?
We have one plugin Link : https://wiki.jenkins-ci.org/display/JENKINS/NodeLabel+Parameter+Plugin which allow to use NODE as Parameter
And in First job you can use the option in post-build action "Trigger Parameterized build on other projects" and then try to pass the node parameter to next job.