I have the following script that pulls from a remote template. The remote template has the following stages: build, test, code_analysis, compliance, deploy.
The deploy step is manually triggered and executed AWS CLI to deploy a SAM project.
I want to add an additional step such that when the deploy step fails, it will execute a script to rollback the cloudformation stack to its last operational state.
I created a "cleanup-cloudformation-stack-failure" job and tried adding "extends: .deploy", but that didn't work.
I then added an additional stage called "cloudformation_stack_rollback" in the serverless-template.yml file and tried to use a mix of rules and when to get it to trigger on failure, but I'm getting errors flagged by GitLab's linter.
Does anyone know what I'm doing wrong?
include:
- remote: 'https://my-gitlab-server.com/ci-templates/-/raw/master/serverless-template.yml'
deploy-qas:
extends: .deploy
variables:
....
PARAMETER_OVERRIDES: "..."
environment: qas
only:
- qas
tags:
- serverless
cleanup-cloudformation-stack-failure:
variables:
STACK_NAME: $CI_PROJECT_NAME-$CI_ENVIRONMENT_NAME
stage: cloudformation_stack_rollback
rules:
- if: '$CI_JOB_MANUAL == true'
when: on_failure
script:
- aws cloudformation continue-update-rollback --stack-name ${STACK_NAME} --resources-to-skip ${STACK_NAME}
You forgot double quotes around true, however you can use Directed Asyclic Graphs to execute jobs conditionally
include:
- remote: 'https://my-gitlab-server.com/ci-templates/-/raw/master/serverless-template.yml'
deploy-qas:
extends: .deploy
variables:
....
PARAMETER_OVERRIDES: "..."
environment: qas
only:
- qas
tags:
- serverless
cleanup-cloudformation-stack-failure:
needs:
- deploy-qas
when: on_failure
variables:
STACK_NAME: $CI_PROJECT_NAME-$CI_ENVIRONMENT_NAME
stage: cloudformation_stack_rollback
script:
- aws cloudformation continue-update-rollback --stack-name ${STACK_NAME} --reso
Related
Seeing Found errors in your .gitlab-ci.yml:
'app-deploy' job needs 'app-verify' job
but 'app-verify' is not in any previous stage
You can also test your .gitlab-ci.yml in CI Lint
Where as both stages are defined
Cache set as below
cache:
key: ${CI_PIPELINE_ID}
paths:
- $CI_PROJECT_DIR/
- $CI_PROJECT_DIR/$CONTEXT/
Stages defined as below, snippets
app-build:
stage: build
# Extending the maven-build function from maven.yaml
extends: .maven-build
app-deploy:
stage: deploy
extends: .docker-publish
cache:
key: ${CI_PIPELINE_ID}
paths:
- $CI_PROJECT_DIR/
- $CI_PROJECT_DIR/$CONTEXT/
variables:
DOCKERFILE: Dockerfile
CONTEXT: app
OUTPUT: app.war
needs:
- app-build
- app-verify
dependencies:
- app-build
- app-verify
How to resolve the above error.
Error should go away and no error in pipeline run.
I set a schedule for my gitlab.yml file to run the pipeline. In my job I have set rules to run/not run the job. However, in my schedule the job is running no matter if any of my rules met.
here is the simplified yml file:
stages:
- build
variables:
DOCKER_DRIVER: overlay2
DOCKER_TLS_CERTDIR : ""
DOCKER_NETWORK: "gitlab-network"
.docker_dind_service: &docker_dind_service
services:
- name: docker:20.10-dind
command: ["--insecure-registry", "my_server.net:7000"]
docker:custom:
stage: build
<<: *docker_dind_service
tags:
- docker_runner
image: docker
rules:
- if: '$FORCE_BUILD_DOCKER_IMAGE == "1"'
when: always
- changes:
- Dockerfile
- when: never
script:
- docker build -t my_image .
for the case above, the job is added to the schedule even though there is no change in my Dockerfile. I think I am lost, because when I do changes in my yml file and push it, this job is not added, which is right because there is no change in the Dockerfile. However, it is running for every scheduled pipeline.
Apparently according to the Gitlab documentation:
https://docs.gitlab.com/ee/ci/yaml/#using-onlychanges-without-pipelines-for-merge-requests
You should use rules: changes only with branch pipelines or merge request pipelines. You can use rules: changes with other pipeline types, but rules: changes always evaluates to true when there is no Git push event. Tag pipelines, scheduled pipelines, manual pipelines, and so on do not have a Git push event associated with them. A rules: changes job is always added to those pipelines if there is no if that limits the job to branch or merge request pipelines.
after adding dynamic child pipelines to our CI pipeline on stop action(eg. deleting branch), stopped working.
In stop job we are deleting created k8s resources, so its important to be executed.
What i noticed is that defining environment in child pipeline is probable cause(without environment section, on stop action is working).
Any ideas?
gitlab-ci.yaml looks like this
stages:
....
- deploy
- tests_prepare
- maintenance_tests
....
deploy_branch_to_k8s:
stage: deploy
only:
- branches
except:
- master
dependencies:
- build_api
environment:
name: branches/$CI_COMMIT_REF_NAME
on_stop: stop_deployed_branch_in_k8s
script:
- deploy to k8s
stop_deployed_branch_in_k8s:
stage: deploy
only:
- branches
except:
- master
when: manual
dependencies: []
variables:
GIT_STRATEGY: none
environment:
name: branches/$CI_COMMIT_REF_NAME
action: stop
script:
- delete k8s resources
generate_config_tests:
stage: tests_prepare
only:
- branches
except:
- master
dependencies:
- build_api
....
script:
- python3 ./utils/generate-jobs-config.py > generated-config.yml
artifacts:
paths:
- generated-config.yml
create_maintenance_tests_pipeline:
stage: maintenance_tests
only:
- branches
except:
- master
trigger:
strategy: depend
include:
- artifact: generated-config.yml
job: generate_config_tests
variables:
PARENT_PIPELINE_ID: $CI_PIPELINE_ID
generated-config.yml looks something like this
stages:
- tests
run_maintenance_test_job___TEST_NAME__:
stage: tests
retry: 2
environment:
name: branches/$CI_COMMIT_REF_NAME
needs:
- pipeline: $PARENT_PIPELINE_ID
job: generate_config_maintenance_tests
script:
- deploy a run tests in k8s
If I'm not wrong, you should skip the needs part altogether in the child pipeline, since it is only used for jobs in the same pipeline. Its upstream will be the parent pipeline anyway.
I have a pipeline with review apps. So when the pipeline runs in the context of a Merge Request / Pull Request then I run:
build and upload a docker image to ECR tagged with $CI_PROJECT_NAME.$CI_MERGE_REQUEST_ID
deploy a helm chart that configures the app to be visible at $CI_PROJECT_NAME-$CI_MERGE_REQUEST_ID.reviewapps.example.com
I want to delete that docker image tag and kubernetes deployment after the merge request is merged/closed, so I added the following stop review app job:
deploy review app:
stage: deploy
image: alpine/helm:3.5.0
dependencies: []
script:
- helm -n "$KUBE_NAMESPACE" upgrade
--install --wait "$CI_PROJECT_NAME-$CI_MERGE_REQUEST_ID" chart
-f helm-reviewapp-values.yaml
--set-string "ingress.annotations.external-dns\.alpha\.kubernetes\.io/hostname=$CI_PROJECT_NAME-$CI_MERGE_REQUEST_ID.reviewapps.example.com."
--set-string "ingress.host=$CI_PROJECT_NAME-$CI_MERGE_REQUEST_ID.reviewapps.example.com"
--set-string "image=$AWS_REPOSITORY:$CI_PROJECT_NAME.$CI_MERGE_REQUEST_ID"
--set "deploymentAnnotations.app\.gitlab\.com/app=${CI_PROJECT_PATH_SLUG}"
--set "deploymentAnnotations.app\.gitlab\.com/env=${CI_ENVIRONMENT_SLUG}"
--set "podAnnotations.app\.gitlab\.com/app=${CI_PROJECT_PATH_SLUG}"
--set "podAnnotations.app\.gitlab\.com/env=${CI_ENVIRONMENT_SLUG}"
environment:
name: review/$CI_PROJECT_NAME-$CI_MERGE_REQUEST_ID
url: https://$CI_PROJECT_NAME-$CI_MERGE_REQUEST_ID.reviewapps.example.com
on_stop: stop review app
auto_stop_in: 1 day
needs:
- build docker image review app
rules:
- if: $CI_MERGE_REQUEST_ID
stop review app:
stage: cleanup approval
script: echo approved
dependencies: []
environment:
name: review/$CI_PROJECT_NAME-$CI_MERGE_REQUEST_ID
action: stop
needs:
- deploy review app
rules:
- if: $CI_MERGE_REQUEST_ID
when: manual
allow_failure: true
uninstall helm chart:
stage: cleanup
image: alpine/helm:3.5.0
dependencies: []
environment:
name: review/$CI_PROJECT_NAME-$CI_MERGE_REQUEST_ID
action: stop
script:
- helm -n "$KUBE_NAMESPACE" uninstall "$CI_PROJECT_NAME-$CI_MERGE_REQUEST_ID"
needs:
- stop review app
rules:
- if: $CI_MERGE_REQUEST_ID
allow_failure: true
delete ecr image:
stage: cleanup
image: amazon/aws-cli:2.1.19
dependencies: []
script:
- aws ecr batch-delete-image --repository-name XXXX --image-ids "imageTag=$CI_PROJECT_NAME.$CI_MERGE_REQUEST_ID"
needs:
- stop review app
rules:
- if: $CI_MERGE_REQUEST_ID
allow_failure: true
As you can see the stop review job is
referred in the "deploy review app" in the environment:on_stop, making use of the environment auto-stop feature
marked as when:manual
made optional with allow_failure: true
Then the pipeline looks like this
the stop review app still "blocks" the pipeline, it shows as running until the stop job runs:
This is bothering me because when people click on the Merge when pipeline succeeds nothing will really happen until the environment is manually stopped (by clicking the play button on the stop review app job).
I also tried removing the allow_failure from the stop job but the only difference is that the pipeline will be stuck in state blocked instead of running.
Is there a way to use the Merge when pipeline succeeds together with Review apps (that need a stop job)?
This caused by the needs: stop review app in the downstream jobs.
As a workaround you can create a single job that performs all the cleanup instead of having uninstall helm chart and delete ecr image depending on stop review app via needs:.
You will need to use a docker image for the job that has all the tools required (helm and aws-cli in your case).
The following pipeline .gitlab-ci.yml will turn to passed after deploy review app passed. The single optional stop job stop review app does not force the pipeline to remain in running or blocked and the pipeline succeeds without having to run that particular job:
stages:
- test
- package
- deploy
- cleanup approval
- cleanup
build docker image review app:
stage: package
script:
- echo hello
rules:
- if: $CI_MERGE_REQUEST_ID
deploy review app:
stage: deploy
image: alpine/helm:3.5.0
dependencies: []
script:
- echo hello
environment:
name: review/$CI_PROJECT_NAME-$CI_MERGE_REQUEST_ID
url: https://$CI_PROJECT_NAME-$CI_MERGE_REQUEST_ID.reviewapps.tdhb2bdev.com
on_stop: stop review app
auto_stop_in: 1 day
needs:
- build docker image review app
rules:
- if: $CI_MERGE_REQUEST_ID
stop review app:
stage: cleanup approval
script:
- echo helm uninstall xxxx
- echo aws ecr batch-delete-image xxxx
dependencies: []
environment:
name: review/$CI_PROJECT_NAME-$CI_MERGE_REQUEST_ID
action: stop
needs:
- deploy review app
rules:
- if: $CI_MERGE_REQUEST_ID
when: manual
allow_failure: true
I'm trying to create a pipeline with a production and a development deployment. In both environments the application should be built with docker. But only when something changed in the according directory.
For example:
When something changed in the frontend directory the frontend should be build and deployed
When something changed in the backend directory the backend should be build and deployed
At first I didn't had the needs: keyword. The pipeline always executed the deploy_backend and deploy_frontend even when the build jobs were not executed.
Now I've added the needs: keyword, but Gitlab says yaml invalid when there was only a change in one directory. When there is a change in both directories the pipeline works fine. When there for exaple a change in the README.md outside the 2 directories the says yaml invalid as well.
Does anyone knows how I can create a pipeline that only runs when there is a change in a specified directory and only runs the according deploy job when the build job has ran?
gitlab-ci.yml:
stages:
- build
- deploy
build_frontend:
stage: build
only:
refs:
- master
- development
changes:
- frontend/*
script:
- cd frontend
- docker build -t frontend .
build_backend:
stage: build
only:
refs:
- master
- development
changes:
- backend/*
script:
- cd backend
- docker build -t backend .
deploy_frontend_dev:
stage: deploy
only:
refs:
- development
script:
- "echo deploy frontend"
needs: ["build_frontend"]
deploy_backend_dev:
stage: deploy
only:
refs:
- development
- pipeline
script:
- "echo deploy backend"
needs: ["build_backend"]
The problem here is that your deploy jobs require the previous build jobs to actually exist.
However, by using the only.changes-rule, they only exist if actually something changed within those directories.
So when only something in the frontend-folder changed, the build_backend-Job is not generated at all. But the deploy_backend_dev job still is and then misses it's dependency.
A quick fix would be to add the only.changes configuration also to the deployment-jobs like this:
deploy_frontend_dev:
stage: deploy
only:
refs:
- development
changes:
- frontend/*
script:
- "echo deploy frontend"
needs: ["build_frontend"]
deploy_backend_dev:
stage: deploy
only:
refs:
- development
- pipeline
changes:
- backend/*
script:
- "echo deploy backend"
needs: ["build_backend"]
This way, both jobs will only be created if the dependent build job is created as well and the yaml will not be invalid.