I'm new to Gitlab CI.
I have configured .gitlab-ci.yml file, and using CI Lint it has passed the validation process.
Based on this documentation, I can see a specific runner should be configured on a virtual machine, a VPS, a bare-metal machine, a docker container or
even a cluster of containers.
But I can see gitlab has its own shared runners and enabled by default.
The question is how to use this shared runner?
When I visit the Pipeline page I can only see the blue Get Started with Pipeline button and when clicked I was redirected to this page.
Here's my .gitlab-ci.yml content :
before_script:
- eval $(ssh-agent -s)
stage_deploy:
only:
- testing
script:
- ssh-add <(echo "$STAGING_PRIVATE_KEY")
- ssh root#1.2.3.4 "sh update_app.sh"
It will only run the job for your testing branch, have you added the .gitlab-ci.yml file to that branch too?
Related
Since I need to use a self-hosted runner, the option to use existing marketplace SSH Actions is not viable because they tend to docker and build an image on GitHub Actions which fails because of a lack of permissions for an unknown user. Existing GitHub SSH Actions such as appleboy fail due to reliance on Docker for me.
Please note: Appleboy & other similar actions work great when using GitHub Actions Runners.
Since it's a self-hosted runner i.e. I have access to it all the time. So using an SSH profile and then using native-run command works pretty well.
To create an SSH profile & use it in GitHub Actions:
Create (if doesn't exist already) a "config" file in "~/.ssh/".
Add a profile in "~/.ssh/config". For example-
Host dev
HostName qa.bluerelay.com
User ec2-user
Port 22
IdentityFile /home/ec2-user/mykey/something.pem
Now to test it, on self-hosted runner run:
ssh dev ls
This should run the ls command inside the dev server.
4. Now since the SSH profile is set up, you can easily run remote SSH commands using GitHub Actions. For example-
name: Test Workflow
on:
push:
branches: ["main"]
jobs:
build:
runs-on: self-hosted
steps:
- uses: actions/checkout#v2
- name: Download Artifacts into Dev
run: |
ssh dev 'cd GADeploy && pwd && sudo wget https://somelink.amazon.com/web.zip'
- name: Deploy Web Artifact to Dev
run: |
ssh dev 'cd GADeploy && sudo ./deploy-web.sh web.zip'
This works like a charm!!
Independently of the runner itself, you would need first to check if SSH does work from your server
curl -v telnet://github.com:22
# Assuming your ~/.ssh/id_rsa.pub is copied to your GitHub profile page
git ls-remote git#github.com:you/YourRepository
Then you can install your runner, without Docker.
Finally, your runner script can execute jobs.<job_id>.steps[*].run commands, using git with SSH URLs.
Like in docker we can enter a container by and have an interactive shell
docker-compose exec containername /bin/bash
Similary in the script in gitlab CI/CD can we enter into it. Like it provides an interactive shell
Eg:
build:
stage: build
script:
- pwd; ls -al
HERE I WANT TO HAVE AN INTERACTIVE SHELL SO THAT I CAN CHECK FEW THINGS
I think we need to do an small detour here and explain how jobs are working in GitLab CI.
Each job is an encapsulated docker container. The container only executes things you like to be executed within the script directive. By default the jobs on shared runners are using a ruby container image.
If you want to check, what you have available within your image, or you want try things out locally. You can do so running a container with this image locally and mounting your project folder into it.
docker run --rm -v "$(pwd):/build/project" -w "/build/project" -it <the job image> /bin/bash # or /bin/sh or whatever shell is available in the image.
# -v mounts the current directory int /build/project in your container
# -w changes the working directory to the mounting point
# /bin/bash starts the shell, it might be that there are others within the image
If you want to use a different docker image, lets say because you are running some other build tool, you can specify this with the image directive like:
build:
image: maven:latest
script:
- echo "some output"
You do have the functionality available within your job, which is provided by the image. As the job will run within a container of that image.
You can even use some tools like https://github.com/firecow/gitlab-ci-local to verify this locally. But in the end those are just docker images, and you can easily recreate the flow on your own.
I have setup my CI so that I can manually create a release-tag when all tests succeeds for a new commit on master branch. For this I have created a manual step in the CI config like so:
.release-template:
stage:
releasing
dependencies:
- assemble
script:
- ./gradlew reckonTagPush -Preckon.scope=$scope -Preckon.stage=$stage -Dorg.ajoberstar.grgit.auth.username=$GIT_USER -Dorg.ajoberstar.grgit.auth.password=$GIT_PASSSWORD
only:
- master
when: manual #ONLY MANUAL RELEASES, ONLY FROM MASTER
release-major:
extends: .release-template
variables:
scope: major
stage: final
release-minor:
extends: .release-template
variables:
scope: minor
stage: final
release-patch:
extends: .release-template
variables:
scope: patch
stage: final
This setup fails with an authentication error.
Execution failed for task ':reckonTagPush'.
> org.eclipse.jgit.api.errors.TransportException: https://gitlab-ci-token#gitlab.com/<group>/<project>.git: not authorized
I am running this on gitlab.com on a shared runner.
The username and password are configured in gitlab ci variables for the project. When running this locally inside the same docker image that is used in the gitlab runner, it works fine. So there must be something special about the way the gitlab runner is executing the gradle tasks, or communicating with the gitlab git repo.
Solved the issue with access to pushing to the git repo by adding the following script :
script:
- url_host=`git remote get-url origin | sed -e "s/https:\/\/gitlab-ci-token:.*#//g"`
- git remote set-url origin "https://gitlab-ci-token:$GIT_TOKEN#$url_host"
- ./gradlew reckonTagPush -Preckon.scope=$scope -Preckon.stage=$stage -Dorg.ajoberstar.grgit.auth.username="$GIT_USER" -Dorg.ajoberstar.grgit.auth.password="$GIT_TOKEN"
The notably changes here are setting the git remote url, as well as surrounding the gitlab ci variables with " when passing them to the reckon plugin
I want to run CI pipline with 2 jobs:
job will boot up a docker image with docker-runner and run test inside docker
will run under ssh runner and pull code on a remote server.
Is it possible?
Yes, it's possible. You need to:
Register two GitLab Runners with needed executor (docker and shell), each witch different tag (or, at least one of them with a build tag).
Declare a specific tag for given job in your .gitlab-ci.yml, .
Shell runner registration:
[root#jsc00mca ~]# gitlab-runner register
Running in system-mode.
Please enter the gitlab-ci coordinator URL (e.g. https://gitlab.com/):
https://example.com/
Please enter the gitlab-ci token for this runner:
1a2b3c
Please enter the gitlab-ci description for this runner:
[jsc00mca.example.com]: my-shell-runner
Please enter the gitlab-ci tags for this runner (comma separated):
shell
Whether to run untagged builds [true/false]:
[false]:
Whether to lock the Runner to current project [true/false]:
[true]:
Registering runner... succeeded runner=ajgHxcNz
Please enter the executor: virtualbox, docker+machine, kubernetes, docker, shell, ssh, docker-ssh+machine, docker-ssh, parallels:
shell
Runner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded!
Docker runner registration:
[root#jsc00mca ~]# gitlab-runner register
Running in system-mode.
Please enter the gitlab-ci coordinator URL (e.g. https://gitlab.com/):
https://example.com/
Please enter the gitlab-ci token for this runner:
1a2b3c
Please enter the gitlab-ci description for this runner:
[jsc00mca.example.com]: my-docker-runner
Please enter the gitlab-ci tags for this runner (comma separated):
docker
Whether to run untagged builds [true/false]:
[false]:
Whether to lock the Runner to current project [true/false]:
[true]:
Registering runner... succeeded runner=ajgHxcNz
Please enter the executor: virtualbox, docker+machine, kubernetes, docker, shell, ssh, docker-ssh+machine, docker-ssh, parallels:
docker
Please enter the default Docker image (e.g. ruby:2.1):
alpine:latest
Runner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded!
.gitlab-ci.yml
buildWithShell:
stage: build
tags:
- shell
script:
- echo 'Building with the shell executor...'
buildWithDocker:
image: alpine:latest
stage: build
tags:
- docker
script:
- echo 'Building with the docker executor...'
Yes you can trigger different/mixed runners from a single gitlab-ci pipeline.
First you should register a shell runner on the target host and give it a tag (truncated):
$ gitlab-runner register
...
Please enter the gitlab-ci tags for this runner (comma separated):
my_shell_runner
...
Please enter the executor: virtualbox, docker+machine, docker-ssh+machine, docker, docker-ssh, parallels, shell, ssh:
shell
Within your gitlab-ci.yaml something like this should work.
The 'test' job runs your test command in a docker container based on the image NAME_OF_IMAGE.
If that succeeds, the 'deploy' job chooses your shell runner based on the tag 'my_shell_runner' and will execute all commands within the script tag on the runner's host (truncated):
test:
stage: test
services:
- docker:dind
tags:
- docker-executor
script:
- docker run --rm NAME_OF_IMAGE sh -c "TEST_COMMAND_TO_RUN"
deploy:
stage: deploy
tags:
- my_shell_runner
script:
- COMMAND_TO_RUN
- COMMAND_TO_RUN
- COMMAND_TO_RUN
Gitlab provides a .gitlab-ci.yml template for building and publishing images to its own registry (click "new file" in one of your project, select .gitlab-ci.yml and docker). The file looks like this and it works out of the box :)
# This file is a template, and might need editing before it works on your project.
# Official docker image.
image: docker:latest
services:
- docker:dind
before_script:
- docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY
build-master:
stage: build
script:
- docker build --pull -t "$CI_REGISTRY_IMAGE" .
- docker push "$CI_REGISTRY_IMAGE"
only:
- master
build:
stage: build
script:
- docker build --pull -t "$CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG" .
- docker push "$CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG"
except:
- master
But by default, this will publish to gitlab's registry. How can we publish to docker hub instead?
No need to change that .gitlab-ci.yml at all, we only need to add/replace the environment variables in project's pipeline settings.
1. Find the desired registry url
Using hub.docker.com won't work, you'll get the following error:
Error response from daemon: login attempt to https://hub.docker.com/v2/ failed with status: 404 Not Found
Default docker hub registry url can be found like this:
docker info | grep Registry
Registry: https://index.docker.io/v1/
index.docker.io is what I was looking for.
2. Set the environment variables in gitlab settings
I wanted to publish gableroux/unity3d images using gitlab-ci, here's what I used in Gitlab's project > Settings > CI/CD > Variables
CI_REGISTRY_USER=gableroux
CI_REGISTRY_PASSWORD=********
CI_REGISTRY=docker.io
CI_REGISTRY_IMAGE=index.docker.io/gableroux/unity3d
CI_REGISTRY_IMAGE is important to set.
It defaults to registry.gitlab.com/<username>/<project>
regsitry url needs to be updated so use index.docker.io/<username>/<project>
Since docker hub is the default registry when using docker, you can also use <username>/<project> instead. I personally prefer when it's verbose so I kept the full registry url.
This answer should also cover other registries, just update environment variables accordingly. 🙌
To expand on the GabLeRoux's answer,
I had issues on the pushing stage of the GitLab CI build:
denied: requested access to the resource is denied
By changing my CI_REGISTRY to docker.io (remove the index.) I was able to successfully push.