I am having a problem where an MR can still be merged during freeze period if it has a passing pipeline from before the freeze.
I have set up my .gitlab-ci.yml like below:
code-freeze-check:
stage: test
script:
- echo "No merge should happen during a freeze period"
- exit 1
rules:
- if: $CI_DEPLOY_FREEZE != null
It behaves as expected, failing the pipeline during a freeze period.
Is there a method besides manually running the pipeline to make sure the MR has a failing pipeline during freeze periods?
Related
We have a Gitlab CI pipeline (.gitlab-ci.yml file)
In that file there are 3 stages
stages:
build
validate
deploy:stage
Firstly, there are 2 jobs
validate:rules:
stage: validate
validate:charts:
stage: validate
Secondly, there is a job that does not have a stage announced inside,
.deploy: &deploy
Later, there is another job:
job-abc:
<<: *deploy
stage: deploy:stage
Q1: can we call same stage in 2 jobs as it has been done in case of validate:rules and validate:charts.
Q2: what is the generic purpose of using double colon in job names such as validate:rules.
Q3: It will be helpful if someone can explain ".deploy: &deploy" and "<<: *deploy" above.
I have a step in a pipeline that creates a MR environment and then when the MR is merged tears down the MR environment using the on_stop keyword.
Example step
deploy-mr-env:
rules:
- if: $CI_PIPELINE_SOURCE == 'merge_request_event'
exists:
- helm/config/values-sandbox.yml
extends:
- .deploy-mr-env-base
variables:
HELM_SANDBOX_CONFIGS: >-
-f helm/config/values-sandbox.yml
-f helm/config/secrets-sandbox.yml
environment:
name: ${MR_ENVIRONMENT}
url: https://${CI_PROJECT_NAME}-mr${CI_MERGE_REQUEST_IID}.sandbox.com
on_stop: stop-mr-env
The bug we found with our pipeline looks like this:
A MR is created and a MR environment is stood up.
Changes are made based on what is discovered with the MR env deployment.
A change is pushed to MR branch (kicking off the pipeline) prior to the MR being merged, some anomalous event occurs on the cloud platform used to host the MR env, and the deploy-mr-env step fails (i.e. - oc login command fails for some weird reason).
MR is merged.
Because deploy-mr-env failed when the last changes were pushed to MR branch on_stop doesn't run.
Normally on_stop runs when a MR is merged, but this is a weird little edge case in our pipeline. How would we make sure on_stop runs during a merge event even if the deploy-mr-env step failed the previous time changes were pushed to the MR branch?
Is a multiline script block in a gitlab-ci pipeline immediately aborted if a call within these instructions returns an exit code that is not equal to zero?
[gitlab-ci.yml]
stage: test
script:
- export ENVIRONMENTS=$(cat $STAGE-environments.txt)
- |
for ENVIRONMENT in ${ENVIRONMENTS}; do
echo "create tests for stage '$STAGE' and environment '$ENVIRONMENT'"
create_test $STAGE $ENVIRONMENT
fill_test $STAGE $ENVIRONMENT
done
The two calls "create_test" and "fill_test" are two bash script functions that return an exit code not equal to zero in the event of an error. If this happens, I want the gitlab-ci stage "test" to stop immediately and mark the job as failed.
Do I have to adapt the multiline script for this?
I need to call a bash script from my gitlab cicd pipeline. When I call it, the input parameter needs to change depending on whether or not this is a merge into master. Basically what I want is this:
if master:
test:
stage: test
script:
- INPUT="foo"
- $(myscript.sh $INPUT)
if NOT master:
test:
stage: test
script:
- INPUT=""
- $(myscript.sh $INPUT)
I'm trying to figure out a way to set INPUT depending on which branch the pipeline is running on. I know there are rules as well as only/except, but they don't seem to allow you to set variable, only test them. I know the brute force way is to just write this twice, one with "only master" and another with "except master", but I would really like to avoid that.
Thanks
I implement this kind of thing using yml anchors to extend tasks.
I find it easier to read and customization can include other things, not only variables.
For example:
.test_my_software:
stage: test
script:
- echo ${MESSAGE}
- bash test.sh
test stable:
<<: *test_my_software
variables:
MESSAGE: testing stable code!
only:
- /stable.*/
test:
<<: *test_my_software
variables:
MESSAGE: testing our code!
except:
- /stable.*/
you get the idea...
Why not have two jobs to run the script and use rules to control when they are ran against master:
test:
stage: test
script:
- if [$CI_COMMIT_REF_NAME == 'master']; then export INPUT="foo"; else INPUT=""; fi
- $(myscript.sh $INPUT)
I have a program that will dynamically release resources during job execution, using the command:
scontrol update JobId=$SLURM_JOB_ID NodeList=${remaininghosts}
However, this results in some very weird behavior sometimes. Where the job is re-queued. Below is the output of sacct
sacct -j 1448590
JobID NNodes State Start End NodeList
1448590 4 RESIZING 20:47:28 01:04:22 [0812,0827],[0663-0664]
1448590.0 4 COMPLETED 20:47:30 20:47:30 [0812,0827],[0663-0664]
1448590.1 4 RESIZING 20:47:30 01:04:22 [0812,0827],[0663-0664]
1448590 3 RESIZING 01:04:22 01:06:42 [0812,0827],0663
1448590 2 RESIZING 01:06:42 1:12:42 0827,tnxt-0663
1448590 4 COMPLETED 05:33:15 Unknown 0805-0807,0809]
The first lines show everything works fine, nodes are getting released but in the last line, it shows a completely different set of nodes with an unknown end time. The slurm logs show the job got requeued:
requeue JobID=1448590 State=0x8000 NodeCnt=1 due to node failure.
I suspect this might happen because the head node is killed, but the slurm documentation doesn't say anything about that.
Does anybody had an idea or suggestion?
Thanks
In this post there was a discussion about resizing jobs.
In your particular case, for shrinking I would use:
Assuming that j1 has been submitted with:
$ salloc -N4 bash
Update j1 to the new size:
$ scontrol update jobid=$SLURM_JOBID NumNodes=2
$ scontrol update jobid=$SLURM_JOBID NumNodes=ALL
And update the environmental variables of j1 (the script is created by the previous commands):
$ ./slurm_job_$SLURM_JOBID_resize.sh
Now, j1 has 2 nodes.
In your example, your "remaininghost" list, as you say, may exclude the head node that is needed by Slurm to shrink the job. If you provide a quantity instead of a list, the resize should work.