I don't get it. I have two repos. One for infrastructure, and the other for project code. Inside project code, I have .gitlab-ci.yml file that will include one job for creating env variables, another include file that will include other stages inside the job. Every other stage is triggered, but the first include is not triggered no matter what I do. What am I doing wrong?
Project gitlab-ci
# Stages
stages:
- pre-build test
- build
- post-build test
- deploy
- environment
- e2e
# Main Variables
variables:
GIT_SUBMODULE_STRATEGY: normal
IMAGE_VERSION: "latest"
CARGO_HOME: $CI_PROJECT_DIR/.cargo
FF_USE_FASTZIP: "true"
ARTIFACT_COMPRESSION_LEVEL: "fast"
CACHE_COMPRESSION_LEVEL: "fastest"
STAGING_BRANCH: "master"
VARIABLES_FILE: variables.txt
# Include main CI files from infrastructure repository
include:
- project: 'project/repo-one'
ref: master
file: '/gitlab-ci/env/ci-app-env.yml'
- project: 'project/repo-one'
ref: master
file: '/gitlab-ci/app/ci-merge-request.yml'
env ci file
env mr:
stage: .pre
before_script:
- TEST_VAR="TEST"
- IMAGE_PATH="/var/www"
script:
- echo "export TEST_VAR=$TEST_VAR" > $VARIABLES_FILE
- echo "export IMAGE_PATH=$IMAGE_PATH" > $VARIABLES_FILE
- cat $VARIABLES_FILE
artifacts:
paths:
- $VARIABLES_FILE
Related
I have a CI pipeline in Gitlab (relevant part only):
default:
image: docker:latest
variables:
DOCKER_APP_TAG: ${REGISTRY_URL}/${APP_NAME}:${CI_COMMIT_SHORT_SHA}
stages:
- build
.config:
only:
- branches
- merge_requests
- tags
except:
- triggers
tags:
- prod
build-app:
extends: .config
stage: build
script:
- docker build --target production -t ${DOCKER_APP_TAG} -f ${CI_PROJECT_DIR}/etc/node/Dockerfile .
When I build from a branch (i.e. push to main branch) everything works well. The docker build command is ran with the proper value available in S{DOCKER_APP_TAG}.
However after I create a TAG in GitLab (and a release), the build on this GitLab TAG fails at the docker build ... line complaining that the docker tag is not valid:
invalid argument "/:e5dc27fd" for "-t, --tag" flag: invalid reference format
The variables ${REGISTRY_URL} and ${APP_NAME} are not expanded. I have checked GitLab docs and the only limitations I see is if I was running in a service. But it is not the case.
What am I missing to expand properly the variables even with tag builds?
I'm trying to setup a CI/CD with GitLab and what I want to achieve is to replace all the variables in .env.production file with the ones stored in the Gitlab environment variable.
I search a lot and I could not find any specific example for Vuejs. Anyone did this already, or knows how to do it?
hereafter is an exemple of one of my .gitlab-ci.yaml :
build:
stage: build
image: node:12.18.3-buster
script:
# Set environment variables
- export VUE_APP_API_BASE_PATH="$API_BASE_PATH"
- npm install
- npm run build
- tar -zcf dist.tar.gz dist
artifacts:
expire_in: 15 min
paths:
- dist.tar.gz
only:
- master
I'm trying to create a pipeline with a production and a development deployment. In both environments the application should be built with docker. But only when something changed in the according directory.
For example:
When something changed in the frontend directory the frontend should be build and deployed
When something changed in the backend directory the backend should be build and deployed
At first I didn't had the needs: keyword. The pipeline always executed the deploy_backend and deploy_frontend even when the build jobs were not executed.
Now I've added the needs: keyword, but Gitlab says yaml invalid when there was only a change in one directory. When there is a change in both directories the pipeline works fine. When there for exaple a change in the README.md outside the 2 directories the says yaml invalid as well.
Does anyone knows how I can create a pipeline that only runs when there is a change in a specified directory and only runs the according deploy job when the build job has ran?
gitlab-ci.yml:
stages:
- build
- deploy
build_frontend:
stage: build
only:
refs:
- master
- development
changes:
- frontend/*
script:
- cd frontend
- docker build -t frontend .
build_backend:
stage: build
only:
refs:
- master
- development
changes:
- backend/*
script:
- cd backend
- docker build -t backend .
deploy_frontend_dev:
stage: deploy
only:
refs:
- development
script:
- "echo deploy frontend"
needs: ["build_frontend"]
deploy_backend_dev:
stage: deploy
only:
refs:
- development
- pipeline
script:
- "echo deploy backend"
needs: ["build_backend"]
The problem here is that your deploy jobs require the previous build jobs to actually exist.
However, by using the only.changes-rule, they only exist if actually something changed within those directories.
So when only something in the frontend-folder changed, the build_backend-Job is not generated at all. But the deploy_backend_dev job still is and then misses it's dependency.
A quick fix would be to add the only.changes configuration also to the deployment-jobs like this:
deploy_frontend_dev:
stage: deploy
only:
refs:
- development
changes:
- frontend/*
script:
- "echo deploy frontend"
needs: ["build_frontend"]
deploy_backend_dev:
stage: deploy
only:
refs:
- development
- pipeline
changes:
- backend/*
script:
- "echo deploy backend"
needs: ["build_backend"]
This way, both jobs will only be created if the dependent build job is created as well and the yaml will not be invalid.
Depending on branch the build comes from I need to use slightly different command line arguments. Particularly I would like to upload snapshot nexus artifacts when building from a branch, and release artifact when building off master.
Is there a way to conditionally alter variables?
I tried to use except/only keywords like this
stages:
- stage
variables:
TYPE: Release
.upload_common:
stage: stage
tags: ["Win"]
script:
- echo Uploading %TYPE%
.upload_snapshot:
variables:
TYPE: "Snapshot"
except:
- master
upload:
extends:
- .upload_common
- .upload_snapshot
Unfortunately it skips whole upload step when building off master.
The reason I am using 'extends' pattern here is that I have win and mac platforms which use slightly different variables substitution syntax ($ vs %). I also have a few different build configuration - Debug/Release, 32bit/64bit.
The code below actually works, but I had to duplicate steps for release and snapshot, one is enabled at a time.
stages:
- stage
.upload_common:
stage: stage
tags: ["Win"]
script:
- echo Uploading %TYPE%
.upload_snapshot:
variables:
TYPE: "Snapshot"
except:
- master
.upload_release:
variables:
TYPE: "Release"
only:
- master
upload_release:
extends:
- .upload_common
- .upload_release
upload_snapshot:
extends:
- .upload_common
- .upload_snapshot
The code gets much larger when snapshot/release configuration is multiplied by Debug/Release, Mac/Win, and 32/64bits. I would like to keep number of configurations at minimum.
Having ability to conditionally altering just a few variables would help me reducing this code a lot.
Another addition in GitLab 13.7 are the rules:variables. This allows some logic in setting vars:
job:
variables:
DEPLOY_VARIABLE: "default-deploy"
rules:
- if: $CI_COMMIT_REF_NAME == $CI_DEFAULT_BRANCH
variables: # Override DEPLOY_VARIABLE defined
DEPLOY_VARIABLE: "deploy-production" # at the job level.
- if: $CI_COMMIT_REF_NAME =~ /feature/
variables:
IS_A_FEATURE: "true" # Define a new variable.
script:
- echo "Run script with $DEPLOY_VARIABLE as an argument"
- echo "Run another script if $IS_A_FEATURE exists"
Unfortunatelly YAML-anchors or GitLab-CI's extends don't seem to allow to combine things in script array of commands as of today.
I use built-in variable CI_COMMIT_REF_NAME in combination with global or job-only before_script to solve similar tasks without repeating myself.
Here is an example of my workaround on how to set environment variable to different values dynamically for PROD and DEV during delivery or deployment:
.provide ssh private deploy key: &provide_ssh_private_deploy_key
before_script:
- mkdir -p ~/.ssh
- chmod 700 ~/.ssh
- echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config
- |
if [ "$CI_COMMIT_REF_NAME" == "master" ]; then
echo "$SSH_PRIVATE_DEPLOY_KEY_PROD" > ~/.ssh/id_rsa
MY_DYNAMIC_VAR="we are in master (PROD)"
else
echo "$SSH_PRIVATE_DEPLOY_KEY_DEV" > ~/.ssh/id_rsa
MY_DYNAMIC_VAR="we are NOT in master (DEV)"
fi
- chmod 600 ~/.ssh/id_rsa
deliver-via-ssh:
stage: deliver
<<: *provide_ssh_private_deploy_key
script:
- echo Stage is deliver
- echo $MY_DYNAMIC_VAR
- ssh ...
Also consider this workaround for concatenation of "script" commands: https://stackoverflow.com/a/57209078/470108
hopefully it helps.
A nice way to prepare variables for other jobs is the dotenv report artifact. Unfortunately, these variables cannot be used in many places, but if you only need to access them from other jobs scripts, this is the way:
# prepare environment variables for other jobs
env:
stage: .pre
script:
# Set application version from GIT tag or fake it
- echo "APPVERSION=${CI_COMMIT_TAG:-0.1-dev-$CI_COMMIT_REF_SLUG}+$CI_COMMIT_SHORT_SHA" | tee -a .env
artifacts:
reports:
dotenv: .env
In the script of this job you can conditionally prepare and write your environment values into a file, then make a dotenv artifact out of it. Subsequent - or better yet dependent - jobs will pick up the variables for their scripts from there.
Dears
My continuous integration + tests project is running with the gitlabci file on a dedicated server. Run with multiple environments is my goal to complete this project.
For example: I have two environments (desenv and homol) and my desire is build the project and after run in each one these tests changing only some variables (link, db user) in the test-automation-inscricao-vestib.dll.config.
I createad a yml file with 3 jobs:
build:test -> nuget restore and build project
test:desenv -> change file.dll.config to the respective configuration file of the environment and run testes (NUnit)
test:homol -> change file.dll.config to the respective configuration file of the environment and run testes (NUnit)
Is there a way correctly to do this? Because my gitlabci-runner doesnt work fine with this configuration. E.g.:
Here is my yml code:
stages:
- build
- test
build:test:
only:
- schedules
- web
stage: build
tags:
- windows
script:
#Restore Nuget
- '"C:\\Gitlab-Runner\\nuget.exe" restore "test-automation-inscricao-vestib.sln"'
#Build project
- '"C:\\Program Files (x86)\\MSBuild\\14.0\\Bin\\msbuild.exe" /t:Clean,Build /p:Configuration=Debug "test-automation-inscricao-vestib.sln"'
artifacts:
paths:
- test-automation-inscricao-vestib\bin\Debug
test:desenv:
only:
- schedules
- web
stage: test
tags:
- teste
script:
#Change the environment to DESENV
- powershell Remove-Item test-automation-inscricao-vestib\bin\Debug\test-automation-inscricao-vestib.dll.config
- powershell Rename-Item test-automation-inscricao-vestib\test-automation-inscricao-vestib_DESENV.dll.config test-automation-inscricao-vestib\bin\Debug\test-automation-inscricao-vestib.dll.config
#Run tests
- cd test-automation-inscricao-vestib/bin/Debug
- '"C:\\Program Files (x86)\NUnit.org\nunit-console\nunit3-console.exe" "test-automation-inscricao-vestib.dll" --where "cat==Producao"'
dependencies:
- build:test
test:homol:
only:
- schedules
- web
stage: test
tags:
- teste
script:
#Change the environment to HOMOL
- powershell Remove-Item test-automation-inscricao-vestib\bin\Debug\test-automation-inscricao-vestib.dll.config
- powershell Rename-Item test-automation-inscricao-vestib\test-automation-inscricao-vestib_HOMOL.dll.config test-automation-inscricao-vestib\bin\Debug\test-automation-inscricao-vestib.dll.config
#Run tests
- cd test-automation-inscricao-vestib/bin/Debug
- '"C:\\Program Files (x86)\NUnit.org\nunit-console\nunit3-console.exe" "test-automation-inscricao-vestib.dll" --where "cat==Producao"'
dependencies:
- build:test
I found a solution for this mistake. I add some tags in each NUnit running line and it works! Look here:
Before:
- '"C:\\Program Files (x86)\NUnit.org\nunit-console\nunit3-console.exe" "test-automation-inscricao-vestib.dll" --where "cat==Producao"'
After:
- '"C:\\Program Files (x86)\NUnit.org\nunit-console\nunit3-console.exe" "test-automation-inscricao-vestib.dll" --inprocess --labels=On --where "cat==Producao"'
Fixed only including these tags: --inprocess --labels=On