Argo Workflow to continue processing during fan-out - dynamic

General question here, wondering if anyone has any ideas or experience trying to achieve something I am right now. I'm not entirely sure if its even possible in the argo workflow system...
I'm wondering if it is possible to continue a workflow regardless if a dynamic fanout has finished. By dynamic fanout I mean that B1/B2/B3 can go to B30 potentially.
I want to see if C1 can start when B1 has finished. The B stage is creating a small file which then in C stage I need to run an api request that it has finished and upload said file. But in this scenario B2/B3 still are processing.
And finally, D1 would have to wait for all of C1/2/3-C# to finish to complete
Diagram what I'm trying to achieve
# *
# |
# A1 (generates a dynamic list that can change depending on the inputs)
# |
# / | \
# B1 B2 B3 +++ B#
# | | |
# C1 +++ C#
# * * *
# \ | /
# \ | /
# D1
I was viewing https://github.com/argoproj/argo-workflows/blob/master/docs/enhanced-depends-logic.md but I cant wrap my head around if this is what I need to achieve this. Especially if the fan-out steps are dynamic.
Seems to me that it would bind C stage to the entirety of B stage and require for for B to finish

Something like this should work:
apiVersion: argoproj.io/v1alpha1
kind: Workflow
spec:
templates:
- name: main
steps:
- - name: A
template: A
- - name: B_C
template: B_C
arguments:
parameters:
- name: item
value: "{{item}}"
withParam: "{{steps.A.outputs.parameters.items}}"
- - name: D
template: D
- name: A
# container or script spec here
outputs:
parameters:
- name: items
valueFrom:
path: /tmp/items.json
- name: B_C
inputs:
parameters:
- name: item
steps:
- - name: B
template: B
arguments:
parameters:
- name: item
value: "{{inputs.parameters.item}}"
- - name: C
template: C
arguments:
artifacts:
- name: file
from: "{{steps.B.outputs.artifacts.file}}"
- name: B
inputs:
parameters:
- name: item
# container or script spec here
outputs:
artifacts:
- name: file
path: /tmp/file
- name: C
inputs:
artifacts:
- name: file
# container or script spec here
- name: D
# container or script spec here
Step B_C in the main template runs instances of the B_C template in parallel.
Template B_C runs B and C in series. Once template B_C starts, it runs as quickly as possible, completely unaware of any concurrent executions of the B_C template. So C1 blocks only on B1, never on B2 or B3 or any other B#.
Once all instances of B_C are finished, the main template finally invokes the D template.

Related

Gitlab-CI: variable in variable in downstream trigger job

(How) Can I use variable value as a name for another variable?
I have a job with matrix and dotenv artifacts as follows:
build-names:
stage: build
...
script:
...
<lines omitted>
...
- echo "${NAME}_DEB_PACKAGE_VERSION=${DEB_PACKAGE_VERSION}" >> build.env
artifacts:
reports:
dotenv: build.env
parallel:
matrix:
- NAME:
- name1
- name2
type:
- 0
- 1
environment: $NAME/$TYPE
Then I have a downstream trigger job again using matrix and I want to pass the appropriate package version based on the ${NAME}
build-images:
stage: .post
needs:
- job: build-names
artifacts: true
variables:
PACKAGE_VERSION_VARIABLE_NAME: ${NAME}_DEB_PACKAGE_VERSION
PACKAGE_VERSION: ${${BACKEND_VERSION_VARIABLE_NAME}}
// OR
PACKAGE_VERSION_VARIABLE_NAME: ${NAME}_DEB_PACKAGE_VERSION
PACKAGE_VERSION: ${!BACKEND_VERSION_VARIABLE_NAME}
trigger:
project: group/project-${NAME}
parallel:
matrix:
- NAME:
- name1
- name2
Neither of the two approaches above (double ${${}} or !) works in the variables section.
I could 'generate' the variable within script section, but AFAIK you cannot have both trigger and script within the same job.
Is there a workaround for similar use cases?
(Using self-hosted gitlab 15.4)

Ansible script with | differences of empty map fails [duplicate]

I'm customizing linux users creation inside my role. I need to let users of my role customize home_directory, group_name, name, password.
I was wondering if there's a more flexible way to cope with default values.
I know that the code below is possible:
- name: Create default
user:
name: "default_name"
when: my_variable is not defined
- name: Create custom
user:
name: "{{my_variable}}"
when: my_variable is defined
But as I mentioned, there's a lot of optional variables and this creates a lot of possibilities.
Is there something like the code above?
user:
name: "default_name", "{{my_variable}}"
The code should set name="default_name" when my_variable isn't defined.
I could set all variables on defaults/main.yml and create the user like that:
- name: Create user
user:
name: "{{my_variable}}"
But those variables are inside a really big hash and there are some hashes inside that hash that can't be a default.
You can use Jinja's default:
- name: Create user
user:
name: "{{ my_variable | default('default_value') }}"
Not totally related, but you can also check for both undefined AND empty (for e.g my_variable:) variable. (NOTE: only works with ansible version > 1.9, see: link)
- name: Create user
user:
name: "{{ ((my_variable == None) | ternary('default_value', my_variable)) \
if my_variable is defined else 'default_value' }}"
If anybody is looking for an option which handles nested variables, there are several such options in this github issue.
In short, you need to use "default" filter for every level of nested vars. For a variable "a.nested.var" it would look like:
- hosts: 'localhost'
tasks:
- debug:
msg: "{{ ((a | default({})).nested | default({}) ).var | default('bar') }}"
or you could set default values of empty dicts for each level of vars, maybe using "combine" filter. Or use "json_query" filter. But the option I chose seems simpler to me if you have only one level of nesting.
In case you using lookup to set default read from environment you have also set the second parameter of default to true:
- set_facts:
ansible_ssh_user: "{{ lookup('env', 'SSH_USER') | default('foo', true) }}"
You can also concatenate multiple default definitions:
- set_facts:
ansible_ssh_user: "{{ some_var.split('-')[1] | default(lookup('env','USER'), true) | default('foo') }}"
If you are assigning default value for boolean fact then ensure that no quotes is used inside default().
- name: create bool default
set_fact:
name: "{{ my_bool | default(true) }}"
For other variables used the same method given in verified answer.
- name: Create user
user:
name: "{{ my_variable | default('default_value') }}"
If you have a single play that you want to loop over the items, define that list in group_vars/all or somewhere else that makes sense:
all_items:
- first
- second
- third
- fourth
Then your task can look like this:
- name: List items or default list
debug:
var: item
with_items: "{{ varlist | default(all_items) }}"
Pass in varlist as a JSON array:
ansible-playbook <playbook_name> --extra-vars='{"varlist": [first,third]}'
Prior to that, you might also want a task that checks that each item in varlist is also in all_items:
- name: Ensure passed variables are in all_items
fail:
msg: "{{ item }} not in all_items list"
when: item not in all_items
with_items: "{{ varlist | default(all_items) }}"
The question is quite old, but what about:
- hosts: 'localhost'
tasks:
- debug:
msg: "{{ ( a | default({})).get('nested', {}).get('var','bar') }}"
It looks less cumbersome to me...
#Roman Kruglov mentioned json_query. It's perfect for nested queries.
An example of json_query sample playbook for existing and non-existing value:
- hosts: localhost
gather_facts: False
vars:
level1:
level2:
level3:
level4: "LEVEL4"
tasks:
- name: Print on existing level4
debug:
var: level1 | json_query('level2.level3.level4') # prints 'LEVEL4'
when: level1 | json_query('level2.level3.level4')
- name: Skip on inexistent level5
debug:
var: level1 | json_query('level2.level3.level4.level5') # skipped
when: level1 | json_query('level2.level3.level4.level5')

Azure Devops YAML: Looping Jobs With Different Variables Passed

I am struggling to figure out how to execute an API test using a pipeline where the command used can be modified using a loop. For example:
TEMPLATE.yaml
parameters:
JobName: ''
TestDirectory: '.\tests\smoke\'
PositiveTest: ''
NegativeTest: ''
- name: environments
type: object
values:
- dev01
- dev02
- test01
- test02
jobs:
- job: ${{ parameters.JobName }}
pool:
name: pool
demands:
- Cmd
variables:
PosTest: ${{ parameters.PositiveTest }}
NegTest: ${{ parameters.NegativeTest }}
Directory: ${{ parameters.TestDirectory }}
- script: |
call .\venv\Scripts\activate.bat
cd $(Directory)
python $(PosTest)
displayName: 'Executing Positive Test Scenarios'
condition: and(succeeded(), ne('${{ variables.PosTest }}', ''))
- script: |
call .\venv\Scripts\activate.bat
cd $(Directory)
python $(NegTest)
displayName: 'Executing Negative Test Scenarios'
condition: and(succeeded(), ne('${{ variables.NegTest }}', ''))
TEST_FILE.yaml
...
jobs:
# Get this file: templates\TEMPLATE.yml from the `build` repository (imported above)
- template: templates\api-test-build.yml#build
- ${{ each env in parameters.environments }}:
parameters:
TestDirectory: '.\tests\smoke\job_class'
PositiveTest: 'python_test.py http://apient${{ env }}/arbitrary/api/path/name'
NegativeTest: ''
This of course doesn't work (the each directive returns an error like "the first property must be template". If I move it up a line it then says "the first property must be a job" and this cycle of errors just continues...).
The idea is just that I have a loop that iterates through environment strings (top of the TEMPLATE.yaml example). The yaml file that references the template passes the command python_test.py http://apient<whatever env string the current iteration is on>/arbitrary/api/path/name for each iterated string (bottom of TEST_FILE.yaml) and the template just executes each of those api tests. At the end of a run there should be 4 environments that have been tested on.
This is just an idea I have and I am still learned all the in's and out's of Azure Devops YAML. If anyone knows how I can get this to work, any improvements I can make to the idea itself or any other workarounds/solutions that would be highly appreciated. Thank you!
You can try using Multi-job configuration (matrix) in your pipeline.
When you want to run the same job with multi-configuration in a pipeline, the matrix strategy is a good choose.
For example, in your pipeline, you want to run the jobs that have the same steps and input parameters but different values of the input parameters. You can just set up one job with the matrix strategy in the pipeline.

Multiple extends or multiple stages?

I want to have a CI to deploy two commands ("bash X" and "bash Y") on different production servers (server 1, server 2, server 3, etc.).
I looked for multiple stages but it don't seems to answer my question.
I don't really care if it runs in parallel or B after A. (the manual section is for debugging)
I don't know how to do it : I tried with multiple extends but it only takes the last one (bashB) in my pipeline.
stages:
- get_password
- bashA
- bashB
get_password:
stage: get_password
# Steps
.bashA:
stage: bashA
script:
- lorem ipsum
when: manual
only:
changes:
- script/bashA.sh
.bashB:
stage: bashB
script:
- ipsum loreem
when: manual
only:
changes:
- script/bashB.sh
# SRV1
deploy-srv1:
extends:
- .bashA
- .bashB
variables:
SRV_1: urlsrv1
# SRV2
deploy-srv2:
extends:
- .bashA
- .bashB
variables:
SRV_1: urlsrv2
I just want to be able to deploy bashA and bash B on X servers (I just took 2 servers for example).
When using multiple extend in GitLab, some of the values will not be merged, but overwritten. If you check the documentation here:
https://docs.gitlab.com/ee/ci/yaml/#extends
They write:
The algorithm used for merge is “closest scope wins”, so keys from the last member will always shadow anything defined on other levels
You are not alone in wanting a feature to be able to merge scripts instead of overwriting them. Here's an open issue on GitLab to do what you described:
https://gitlab.com/gitlab-org/gitlab/issues/16376
In the meantime, and only looking at the example you provided, you can get something like what you want by manually merging bashA and bashB into one job:
stages:
- get_password
- bash
get_password:
stage: get_password
# Steps
.bash_both:
stage: bash
script:
- lorem ipsum
- ipsum loreem
when: manual
only:
changes:
- script/bashA.sh
- script/bashB.sh
# SRV1
deploy-srv1:
extends:
- .bash_both
variables:
SRV_1: urlsrv1
# SRV2
deploy-srv2:
extends:
- .bash_both
variables:
SRV_1: urlsrv2

Ansible variable list span

When adding a variable list in Ansible how would one achieve a span of similar values? For instance "000-100" - in an Ansible hosts file this can be done by listing like so, "hostname-[a:v].com". Would this process be the similar in a variable list?
My use case is to provision many VM's within oVirt in a single go without having to make a line by line list.
---
- name: Create VM based on template
hosts: ovirt-engine
become: yes
become_method: sudo
vars:
- temp: '{{temp_fedora25}}'
- iname:
- db-aa
- db-ab
- db-ac
tasks:
- name: Giving Birth to lil Baby VM's
ovirt:
user: '{{ovirt_usr}}'
password: '{{ovirt_pass}}'
url: '{{engine_url}}'
instance_name: "{{item}}"
instance_nic: ovirtmgmt
resource_type: template
image: '{{temp}}'
zone: superblade-a
disk_alloc: preallocated
with_items: "{{iname}}"
You can use sequence lookup:
- name: numeric
debug:
msg: "{{ item }}"
with_sequence: start=1 count=10 format=server-%0d
- name: characters from small 'a'
debug:
msg: "{{ item }}"
with_sequence: start=0x61 count=10 format=server-%c
- name: save for future use
set_fact:
my_seq: "{{ lookup('sequence','start={} count={} format={}{}'.format(beg,cnt,pref,fmt),wantlist=True) }}"
vars:
beg: 1
cnt: 10
pref: host-
fmt: '%0d'
You can skip set_fact and define my_seq in vars section, but if you use my_seq much, list generation will be done internally every time. With set_fact list is generated once.
With respect to the correct answer from Konstantin, I'm adding the full solution as per my case....
My goal is to be able to reuse the sequenced values as registered variables in order to pass the instance name to host name. This works so far but Im sure it can be streamlined by nesting variables perhaps?
---
- name: Create VM based on template
hosts: ovirt-engine
become: yes
become_method: sudo
vars:
- temp: '{{temp_fedora25}}'
- host_pre: db
- host_seq: a%c
- host_cnt: 3
- host_srt: 0x61
tasks:
- name: Giving Birth to lil Baby VM's
ovirt:
user: '{{ovirt_usr}}'
password: '{{ovirt_pass}}'
url: '{{engine_url}}'
instance_name: "{{item}}"
instance_nic: ovirtmgmt
resource_type: template
image: '{{temp}}'
zone: superblade-a
disk_alloc: preallocated
with_sequence: start="{{host_srt}}" count="{{host_cnt}}" format="{{host_pre}}-{{host_seq}}"