Push existing tarball image with kaniko - gitlab-ci

I want to build a Docker image (tarball) in my GitLab CI pipeline using kaniko, then scan it with trivy and push it to an AWS ECR using kaniko.
Step 1: kaniko build (tarball)
Step 2: trivy scan
Step 3: kaniko push (to AWS ECR!)
Unfortunately I can't find a way to push an existing tarball image with kaniko without rebuilding it.
I also tried crane for the push, but can't get a login due to the non-existent credHelper.
I don't actually want to do big installations, nor do I want to create a custom image for this.
Is this possible? What would be potential solutions?

Coincidentally, I did exactly this a while ago. Here is how I did it:
docker:build:
stage: build
image:
name: Kaniko image
entrypoint: [""]
script:
- mkdir tar_images
- mkdir -p /kaniko/.docker
- echo "{\"auths\":{\"$CI_REGISTRY\":{\"username\":\"$CI_REGISTRY_USER\",\"password\":\"$CI_REGISTRY_PASSWORD\"}}}" > /kaniko/.docker/config.json
- /kaniko/executor --context ${CI_PROJECT_DIR} --no-push --destination $CI_REGISTRY/<image_name>:<image_tag> --tarPath tar_images/$file_name.tar
artifacts:
paths:
- tar_images
when: on_success
# scan all built images
# currently bug with grype as docker registry is not public!
docker:scan:
stage: scan
image:
name: trivy
entrypoint: [""]
script:
- mkdir scan_result
- cd tar_images
- |
for tar_image in *.tar;
do
[ -e "$tar_image" ] || continue;
file_name=${tar_image%.*};
echo $file_name;
if [ "$vulnerability_scanner" = "trivy" ]; then
trivy image --timeout 15m --offline-scan --input $tar_image -f json -o ../scan_result/$file_name.json --severity CRITICAL;
fi
done
artifacts:
paths:
- scan_result
expire_in: 1 month
# push all images without detected security issues
docker:push:
stage: push
image:
name: gcr.io/go-containerregistry/crane:debug
entrypoint: [""]
rules:
- if: $UPDATE
script:
- cd tar_images
- |
for tar_image in *.tar;
do
file_name=${tar_image%.*};
vulnerabilities=`awk -F '[:,]' '/"Vulnerabilities"/ {gsub("[[:blank:]]+", "", $2); print $2}' ../scan_result/$file_name.json`; # find vulnerabilities in json file
if ! [ -z "$vulnerabilities" ]; then # if vulnerabilities found in image
echo "There are security issues with the image $img.Dockerfile. Image is not pushed to registry!";
else # push image
crane auth login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY;
crane push $tar_image $CI_REGISTRY_IMAGE/<image_name>:<image_tag>;
fi
done
What happens here is that in the first job the images are built using kaniko. They are stored as tar files and made accessible to the next job via artifacts. In the next job they are scanned using trivy and the scan results are stored as artifacts. Then the scan reports are analyzed and if no vulnerabilities had been detected the image is pushed using crane.
The code above probably does not work out of the box as I copied it out of a bigger yaml file.

Related

Trigger GitLab pipeline based on an external event (Example - A binary image has been uploaded to Nexus)

I have a problem where the application code is developed and built in vendor premises and only code binary is sent in the form of docker image and uploaded to Nexus repository. Now, I need to create a deployment pipeline in GitLab which will be triggered as soon as a new image is uploaded to nexus.
Please suggest any solution.
I have tried to create a parent child pipeline where I am trying to check the HTTP resonse status from Nexus through curl request, trigger the child pipeline only when the status is 200. The parent pipeline can be configured as a scheduled pipeline which will check the image availability on a regular interval.
Parent yml:
stages:
- build
- trigger_child
build:
stage: build
script:
- echo "img_status=`curl -s -o /dev/null -w "%{http_code}" https://www.google.com`" >> build.env
artifacts:
reports:
dotenv: build.env
trigger_child:
stage: trigger_child
trigger:
include:
- local: App/.gitlab-ci.yml
strategy: depend
needs:
- job: build
artifacts: true
rules:
- if: $img_status == "200"
when: always
Child yml:
stages:
- deploy
deploy:
stage: deploy
script:
- echo "This job deploys something new"

Get error x509 in release job in my local gitlab pipeline

I am running a local gitlab server with self-signed certificate, My pipline builds my application and create a release but I have x509 I tried the workaround mentionned on gitlab documenation but it doesn't work. Everything works fine when tested in gitlab.com
To summerize first I build my application to generate a war file as an artifact, then the artifact is uploaded using gitlab API to generate URL and file path after that release job add tags and generate the release page
my gitlab-ci.yaml
---
variables:
PACKAGE_VERSION: "V7"
GENERIC_WAR: "mypackage-${PACKAGE_VERSION}.war"
PACKAGE_REGISTRY_URL: "${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/packages/generic/${CI_PROJECT_NAME}/${PACKAGE_VERSION}"
workflow:
rules:
- if: $CI_COMMIT_BRANCH == "main"
when: always
variables:
SERVER: "${PROD_SERVER}"
- if: $CI_COMMIT_BRANCH == "test"
when: always
variables:
SERVER: "${TEST_SERVER}"
- if: $CI_COMMIT_BRANCH == "feature/release"
when: always
variables:
SERVER: "${TEST_SERVER}"
stages:
- build
- upload
- prepare
- release
- deploy
build-application:
stage: build
image: maven:3.8.4-jdk-8
script:
- mvn clean package -U -DskipTests=true
- echo $CI_COMMIT_TAG
artifacts:
expire_in: 2h
when: always
paths:
- target/*.war
upload:
stage: upload
image: curlimages/curl:latest
needs:
- job: build-application
artifacts: true
# rules:
# - if: $CI_COMMIT_TAG
script:
- |
curl -k --header "JOB-TOKEN: ${CI_JOB_TOKEN}" --upload-file target/*.war "${PACKAGE_REGISTRY_URL}/${GENERIC_WAR}"
prepare_job:
stage: prepare
rules:
- if: $CI_COMMIT_TAG
when: never
- if: $CI_COMMIT_BRANCH == "feature/release"
script:
- echo "EXTRA_DESCRIPTION=some message" >> variables.env # Generate the EXTRA_DESCRIPTION and TAG environment variables
- echo "TAG=v$(cat VERSION)" >> variables.env
artifacts:
reports:
dotenv: variables.env
release_job:
stage: release
image: registry.gitlab.com/gitlab-org/release-cli:latest
needs:
- job: prepare_job
artifacts: true
rules:
- if: $CI_COMMIT_TAG
when: never
- if: $CI_COMMIT_BRANCH == "feature/release"
before_script:
- apk --no-cache add openssl ca-certificates
- mkdir -p /usr/local/share/ca-certificates/extra
- openssl s_client -connect ${CI_SERVER_HOST}:${CI_SERVER_PORT} -servername ${CI_SERVER_HOST} -showcerts </dev/null 2>/dev/null | sed -e '/-----BEGIN/,/-----END/!d' | tee "/usr/local/share/ca-certificates/${CI_SERVER_HOST}.crt" >/dev/null
- update-ca-certificates
script:
- echo 'running release_job for $TAG'
release:
name: "Release $TAG"
description: "Created using the release-cli $EXTRA_DESCRIPTION"
tag_name: "$TAG"
ref: "$CI_COMMIT_SHA"
assets:
links:
- name: "{$GENERIC_WAR}"
url: "${PACKAGE_REGISTRY_URL}"
filepath: "/${GENERIC_WAR}"
Release job execution
Running with gitlab-runner 14.5.2 (e91107dd)
on Shared-Docker mdaS6_cA
Preparing the "docker" executor
00:03
Using Docker executor with image registry.gitlab.com/gitlab-org/release-cli:latest ...
Pulling docker image registry.gitlab.com/gitlab-org/release-cli:latest ...
Using docker image sha256:c2d3a3c3b9ad5ef63478b6a6b757632dd7994d50e603ec69999de6b541e1dca8 for registry.gitlab.com/gitlab-org/release-cli:latest with digest registry.gitlab.com/gitlab-org/release-cli#sha256:68e201226e1e76cb7edd327c89eb2d5d1a1d2b0fd4a6ea5126e24184d9aa4ffc ...
Preparing environment
00:01
Running on runner-mdas6ca-project-32-concurrent-0 via Docker-Server1...
Getting source from Git repository
00:01
Fetching changes with git depth set to 50...
Reinitialized existing Git repository in /builds/Saiida/backend-endarh/.git/
Checking out 7735e9ea as feature/release...
Removing target/
Removing variables.env
Skipping Git submodules setup
Executing "step_script" stage of the job script
00:02
Using docker image sha256:c2d3a3c3b9ad5ef63478b6a6b757632dd7994d50e603ec69999de6b541e1dca8 for registry.gitlab.com/gitlab-org/release-cli:latest with digest registry.gitlab.com/gitlab-org/release-cli#sha256:68e201226e1e76cb7edd327c89eb2d5d1a1d2b0fd4a6ea5126e24184d9aa4ffc ...
$ apk --no-cache add openssl ca-certificates
fetch https://dl-cdn.alpinelinux.org/alpine/v3.13/main/x86_64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.13/community/x86_64/APKINDEX.tar.gz
(1/2) Installing ca-certificates (20191127-r5)
(2/2) Installing openssl (1.1.1l-r0)
Executing busybox-1.32.1-r6.trigger
Executing ca-certificates-20191127-r5.trigger
OK: 7 MiB in 16 packages
$ mkdir -p /usr/local/share/ca-certificates/extra
$ openssl s_client -connect ${CI_SERVER_HOST}:${CI_SERVER_PORT} -servername ${CI_SERVER_HOST} -showcerts </dev/null 2>/dev/null | sed -e '/-----BEGIN/,/-----END/!d' | tee "/usr/local/share/ca-certificates/${CI_SERVER_HOST}.crt" >/dev/null
$ update-ca-certificates
Warning! Cannot copy to bundle: /usr/local/share/ca-certificates/extra
WARNING: ca-certificates.crt does not contain exactly one certificate or CRL: skipping
WARNING: ca-cert-extra.pem does not contain exactly one certificate or CRL: skipping
$ echo 'running release_job for $TAG'
running release_job for $TAG
Executing "step_release" stage of the job script
00:01
$ release-cli create --name "Release $TAG" --description "Created using the release-cli $EXTRA_DESCRIPTION" --tag-name "$TAG" --ref "$CI_COMMIT_SHA" --assets-link "{\"url\":\"${PACKAGE_REGISTRY_URL}\",\"name\":\"{$GENERIC_WAR}\",\"filepath\":\"/${GENERIC_WAR}\"}"
time="2021-12-23T08:47:48Z" level=info msg="Creating Release..." cli=release-cli command=create name="Release v" project-id=32 ref=7735e9ea9422e20b09cae2072c692843b118423a server-url="https://gitlab.endatamweel.tn" tag-name=v version=0.10.0
time="2021-12-23T08:47:48Z" level=fatal msg="run app" cli=release-cli error="failed to create release: failed to do request: Post \"https://gitlab.endatamweel.tn/api/v4/projects/32/releases\": x509: certificate relies on legacy Common Name field, use SANs or temporarily enable Common Name matching with GODEBUG=x509ignoreCN=0" version=0.10.0
Cleaning up project directory and file based variables
00:00
ERROR: Job failed: exit code 1
I managed to get it work by replacing the yaml format of the release job with the release-cli command and arguments and set --insecure-https option not optimised for production of course
release:
stage: release
image: registry.gitlab.com/gitlab-org/release-cli:latest
needs:
- job: prepare_job
artifacts: true
rules:
- if: $CI_COMMIT_TAG
when: never # Do not run this job when a tag is created manually
- if: $CI_COMMIT_BRANCH == "feature/release" # Run this job when commits are pushed or merged to the default branch
script:
- |
release-cli --insecure-https=true create --name "Release $TAG" --tag-name $TAG --ref $CI_COMMIT_SHA \
--assets-link "{\"name\":\"${GENERIC_WAR}\",\"url\":\"${PACKAGE_REGISTRY_URL}/${GENERIC_WAR}\", \"link_type\":\"package\"}"

report generation zap-full-scan

hello I'm trying to do a CI / CD integration of zap on gitlab. To do this, I wrote the following code but after running the report is not generated. What to do please.
the scan is well done but the report is not available
test_site:
stage: test
image: owasp/zap2docker-stable
when: always
script:
- mkdir -p /zap/wrk/
- zap-full-scan.py -t https://example.com -r report.html
- cp /zap/wrk/report.html .
artifacts:
when: always
paths: [report.html]
allow_failure: false
I managed to generate the report by adding to the command || true or the -I option but the objective is to generate the report without adding
I have struggled to get this done for more than 2 days. I started with this approach of using image: owasp/zap2docker-stable however then had to go to the vanilla docker commands to execute it.
The reason for that is, if you use -r parameter, zap will attempt to generate the file report.html at location /zap/wrk/. In order to make this work, we have to mount a directory to this location /zap/wrk.
But when you do so, it is important that the zap container is able to perform the write operations on the mounted directory.
So, below is the working solution.
test_site:
stage: test
image: docker:latest
script:
# The folder zap-reports created locally will be mounted to owasp/zap2docker docker container,
# On execution it will generate the reports in this folder. Current user is passed so reports can be generated"
- mkdir zap-reports
- cd zap-reports
- docker pull owasp/zap2docker-stable:latest || echo
- docker run --name zap-container --rm -v $(pwd):/zap/wrk -u $(id -u ${USER}):$(id -g ${USER}) owasp/zap2docker-stable zap-baseline.py -t "https://example.com" -r report.html
artifacts:
when: always
paths:
- zap-reports
allow_failure: true
So the trick in the above code is
Mount local directory zap-reports to /zap/wrk as in $(pwd):/zap/wrk
Pass the current user and group on the host machine to the docker container so the process is using the same user / group. This allows writing of reports on the directory mounted from local host. This is done by -u $(id -u ${USER}):$(id -g ${USER})
Below is the working code with image: owasp/zap2docker-stable
test_site:
variables:
GIT_STRATEGY: none
stage: test
image:
name: owasp/zap2docker-stable:latest
before_script:
- mkdir -p /zap/wrk
script:
- zap-baseline.py -t "https://example.com" -g gen.conf -I -r testreport.html
- cp /zap/wrk/testreport.html testreport.html
artifacts:
when: always
paths:
- zap.out
- testreport.html

Make kubectl work in gitlab ci

I am searching for a way to use kubectl in gitlab.
So far I have the following script:
deploy_to_dev:
stage: deploy
image: docker:dind
environment:
name: dev
script:
- mkdir -p $HOME/.kube
- echo $KUBE_CONFIG | base64 -d > $HOME/.kube/config
- kubectl config view
only:
- develop
But it says that gitlab does not know kubectl. So can you point me in the right direction.
You are using docker:dindimage which does not have the kubectl binary, you should bring your own image with the binary or download it in the process
deploy_to_dev:
stage: deploy
image: alpine:3.7
environment:
name: dev
script:
- apk update && apk add --no-cache curl
- curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl
- chmod +x ./kubectl && mv ./kubectl /usr/local/bin/kubectl
- mkdir -p $HOME/.kube
- echo -n $KUBE_CONFIG | base64 -d > $HOME/.kube/config
- kubectl config view
only:
- develop
Use image google/cloud-sdk which has a preinstalled installation of gcloud and kubectl.
build:
stage: build
image: google/cloud-sdk
services:
- docker:dind
script:
# Make gcloud available
- source /root/.bashrc

Use GitLab CI to run tests locally?

If a GitLab project is configured on GitLab CI, is there a way to run the build locally?
I don't want to turn my laptop into a build "runner", I just want to take advantage of Docker and .gitlab-ci.yml to run tests locally (i.e. it's all pre-configured). Another advantage of that is that I'm sure that I'm using the same environment locally and on CI.
Here is an example of how to run Travis builds locally using Docker, I'm looking for something similar with GitLab.
Since a few months ago this is possible using gitlab-runner:
gitlab-runner exec docker my-job-name
Note that you need both docker and gitlab-runner installed on your computer to get this working.
You also need the image key defined in your .gitlab-ci.yml file. Otherwise won't work.
Here's the line I currently use for testing locally using gitlab-runner:
gitlab-runner exec docker test --docker-volumes "/home/elboletaire/.ssh/id_rsa:/root/.ssh/id_rsa:ro"
Note: You can avoid adding a --docker-volumes with your key setting it by default in /etc/gitlab-runner/config.toml. See the official documentation for more details. Also, use gitlab-runner exec docker --help to see all docker-based runner options (like variables, volumes, networks, etc.).
Due to the confusion in the comments, I paste here the gitlab-runner --help result, so you can see that gitlab-runner can make builds locally:
gitlab-runner --help
NAME:
gitlab-runner - a GitLab Runner
USAGE:
gitlab-runner [global options] command [command options] [arguments...]
VERSION:
1.1.0~beta.135.g24365ee (24365ee)
AUTHOR(S):
Kamil TrzciƄski <ayufan#ayufan.eu>
COMMANDS:
exec execute a build locally
[...]
GLOBAL OPTIONS:
--debug debug mode [$DEBUG]
[...]
As you can see, the exec command is to execute a build locally.
Even though there was an issue to deprecate the current gitlab-runner exec behavior, it ended up being reconsidered and a new version with greater features will replace the current exec functionality.
Note that this process is to use your own machine to run the tests using docker containers. This is not to define custom runners. To do so, just go to your repo's CI/CD settings and read the documentation there. If you wanna ensure your runner is executed instead of one from gitlab.com, add a custom and unique tag to your runner, ensure it only runs tagged jobs and tag all the jobs you want your runner to be responsible of.
I use this docker-based approach:
Edit: 2022-10
docker run --entrypoint bash --rm -w $PWD -v $PWD:$PWD -v /var/run/docker.sock:/var/run/docker.sock gitlab/gitlab-runner:latest -c 'git config --global --add safe.directory "*";gitlab-runner exec docker test'
For all git versions > 2.35.2. You must add safe.directory within the container to avoid fatal: detected dubious ownership in repository at.... This also true for patched git versions < 2.35.2. The old command will not work anymore.
Details
0. Create a git repo to test this answer
mkdir my-git-project
cd my-git-project
git init
git commit --allow-empty -m"Initialize repo to showcase gitlab-runner locally."
1. Go to your git directory
cd my-git-project
2. Create a .gitlab-ci.yml
Example .gitlab-ci.yml
image: alpine
test:
script:
- echo "Hello Gitlab-Runner"
3. Create a docker container with your project dir mounted
docker run -d \
--name gitlab-runner \
--restart always \
-v $PWD:$PWD \
-v /var/run/docker.sock:/var/run/docker.sock \
gitlab/gitlab-runner:latest
(-d) run container in background and print container ID
(--restart always) or not?
(-v $PWD:$PWD) Mount current directory into the current directory of the container - Note: On Windows you could bind your dir to a fixed location, e.g. -v ${PWD}:/opt/myapp. Also $PWD will only work at powershell not at cmd
(-v /var/run/docker.sock:/var/run/docker.sock) This gives the container access to the docker socket of the host so it can start "sibling containers" (e.g. Alpine).
(gitlab/gitlab-runner:latest) Just the latest available image from dockerhub.
4. Execute with
Avoid fatal: detected dubious ownership in repository at... More info
docker exec -it -w $PWD gitlab-runner git config --global --add safe.directory "*"
Actual execution
docker exec -it -w $PWD gitlab-runner gitlab-runner exec docker test
# ^ ^ ^ ^ ^ ^
# | | | | | |
# (a) (b) (c) (d) (e) (f)
(a) Working dir within the container. Note: On Windows you could use a fixed location, e.g. /opt/myapp.
(b) Name of the docker container
(c) Execute the command "gitlab-runner" within the docker container
(d)(e)(f) run gitlab-runner with "docker executer" and run a job named "test"
5. Prints
...
Executing "step_script" stage of the job script
$ echo "Hello Gitlab-Runner"
Hello Gitlab-Runner
Job succeeded
...
Note: The runner will only work on the commited state of your code base. Uncommited changes will be ignored. Exception: The .gitlab-ci.yml itself does not have be commited to be taken into account.
Note: There are some limitations running locally. Have a look at limitations of gitlab runner locally.
I'm currently working on making a gitlab runner that works locally.
Still in the early phases, but eventually it will become very relevant.
It doesn't seem like gitlab want/have time to make this, so here you go.
https://github.com/firecow/gitlab-runner-local
If you are running Gitlab using the docker image there: https://hub.docker.com/r/gitlab/gitlab-ce, it's possible to run pipelines by exposing the local docker.sock with a volume option: -v /var/run/docker.sock:/var/run/docker.sock. Adding this option to the Gitlab container will allow your workers to access to the docker instance on the host.
The GitLab runner appears to not work on Windows yet and there is an open issue to resolve this.
So, in the meantime I am moving my script code out to a bash script, which I can easily map to a docker container running locally and execute.
In this case I want to build a docker container in my job, so I create a script 'build':
#!/bin/bash
docker build --pull -t myimage:myversion .
in my .gitlab-ci.yaml I execute the script:
image: docker:latest
services:
- docker:dind
before_script:
- apk add bash
build:
stage: build
script:
- chmod 755 build
- build
To run the script locally using powershell I can start the required image and map the volume with the source files:
$containerId = docker run --privileged -d -v ${PWD}:/src docker:dind
install bash if not present:
docker exec $containerId apk add bash
Set permissions on the bash script:
docker exec -it $containerId chmod 755 /src/build
Execute the script:
docker exec -it --workdir /src $containerId bash -c 'build'
Then stop the container:
docker stop $containerId
And finally clean up the container:
docker container rm $containerId
Another approach is to have a local build tool that is installed on your pc and your server at the same time.
So basically, your .gitlab-ci.yml will basically call your preferred build tool.
Here an example .gitlab-ci.yml that i use with nuke.build:
stages:
- build
- test
- pack
variables:
TERM: "xterm" # Use Unix ASCII color codes on Nuke
before_script:
- CHCP 65001 # Set correct code page to avoid charset issues
.job_template: &job_definition
except:
- tags
build:
<<: *job_definition
stage: build
script:
- "./build.ps1"
test:
<<: *job_definition
stage: test
script:
- "./build.ps1 test"
variables:
GIT_CHECKOUT: "false"
pack:
<<: *job_definition
stage: pack
script:
- "./build.ps1 pack"
variables:
GIT_CHECKOUT: "false"
only:
- master
artifacts:
paths:
- output/
And in nuke.build i've defined 3 targets named like the 3 stages (build, test, pack)
In this way you have a reproducible setup (all other things are configured with your build tool) and you can test directly the different targets of your build tool.
(i can call .\build.ps1 , .\build.ps1 test and .\build.ps1 pack when i want)
I am on Windows using VSCode with WSL
I didn't want to register my work PC as a runner so instead I'm running my yaml stages locally to test them out before I upload them
$ sudo apt-get install gitlab-runner
$ gitlab-runner exec shell build
yaml
image: node:10.19.0 # https://hub.docker.com/_/node/
# image: node:latest
cache:
# untracked: true
key: project-name
# key: ${CI_COMMIT_REF_SLUG} # per branch
# key:
# files:
# - package-lock.json # only update cache when this file changes (not working) #jkr
paths:
- .npm/
- node_modules
- build
stages:
- prepare # prepares builds, makes build needed for testing
- test # uses test:build specifically #jkr
- build
- deploy
# before_install:
before_script:
- npm ci --cache .npm --prefer-offline
prepare:
stage: prepare
needs: []
script:
- npm install
test:
stage: test
needs: [prepare]
except:
- schedules
tags:
- linux
script:
- npm run build:dev
- npm run test:cicd-deps
- npm run test:cicd # runs puppeteer tests #jkr
artifacts:
reports:
junit: junit.xml
paths:
- coverage/
build-staging:
stage: build
needs: [prepare]
only:
- schedules
before_script:
- apt-get update && apt-get install -y zip
script:
- npm run build:stage
- zip -r build.zip build
# cache:
# paths:
# - build
# <<: *global_cache
# policy: push
artifacts:
paths:
- build.zip
deploy-dev:
stage: deploy
needs: [build-staging]
tags: [linux]
only:
- schedules
# # - branches#gitlab-org/gitlab
before_script:
- apt-get update && apt-get install -y lftp
script:
# temporarily using 'verify-certificate no'
# for more on verify-certificate #jkr: https://www.versatilewebsolutions.com/blog/2014/04/lftp-ftps-and-certificate-verification.html
# variables do not work with 'single quotes' unless they are "'surrounded by doubles'"
- lftp -e "set ssl:verify-certificate no; open mediajackagency.com; user $LFTP_USERNAME $LFTP_PASSWORD; mirror --reverse --verbose build/ /var/www/domains/dev/clients/client/project/build/; bye"
# environment:
# name: staging
# url: http://dev.mediajackagency.com/clients/client/build
# # url: https://stg2.client.co
when: manual
allow_failure: true
build-production:
stage: build
needs: [prepare]
only:
- schedules
before_script:
- apt-get update && apt-get install -y zip
script:
- npm run build
- zip -r build.zip build
# cache:
# paths:
# - build
# <<: *global_cache
# policy: push
artifacts:
paths:
- build.zip
deploy-client:
stage: deploy
needs: [build-production]
tags: [linux]
only:
- schedules
# - master
before_script:
- apt-get update && apt-get install -y lftp
script:
- sh deploy-prod
environment:
name: production
url: http://www.client.co
when: manual
allow_failure: true
The idea is to keep check commands outside of .gitlab-ci.yml. I use Makefile to run something like make check and my .gitlab-ci.yml runs the same make commands that I use locally to check various things before committing.
This way you'll have one place with all/most of your commands (Makefile) and .gitlab-ci.yml will have only CI-related stuff.
I have written a tool to run all GitLab-CI job locally without have to commit or push, simply with the command ci-toolbox my_job_name.
The URL of the project : https://gitlab.com/mbedsys/citbx4gitlab
Years ago I build this simple solution with Makefile and docker-compose to run the gitlab runner in docker, you can use it to execute jobs locally as well and should work on all systems where docker works:
https://gitlab.com/1oglop1/gitlab-runner-docker
There are few things to change in the docker-compose.override.yaml
version: "3"
services:
runner:
working_dir: <your project dir>
environment:
- REGISTRATION_TOKEN=<token if you want to register>
volumes:
- "<your project dir>:<your project dir>"
Then inside your project you can execute it the same way as mentioned in other answers:
docker exec -it -w $PWD runner gitlab-runner exec <commands>..
I recommend using gitlab-ci-local
https://github.com/firecow/gitlab-ci-local
It's able to run specific jobs as well.
It's a very cool project and I have used it to run simple pipelines on my laptop.