I've got Gitlab installed on Kubernetes using Helm and try to get Auto DevOps working but I'm getting the following issue from the Auto DevOps pipeline:
Executing "step_script" stage of the job script 00:01
$ if [[ -z "$CI_COMMIT_TAG" ]]; then # collapsed multi-line command
$ /build/build.sh
Logging in to GitLab Container Registry with CI credentials...
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
Login Succeeded
Building Dockerfile-based application...
Attempting to pull a previously built image for use with --cache-from...
Cannot connect to the Docker daemon at tcp://localhost:2375. Is the docker daemon running?
Cannot connect to the Docker daemon at tcp://localhost:2375. Is the docker daemon running?
No previously cached image found. The docker build will proceed without using a cached image
Cannot connect to the Docker daemon at tcp://localhost:2375. Is the docker daemon running?
Cleaning up file based variables 00:00
ERROR: Job failed: command terminated with exit code 1
I've read abt the issues with Docker-in-Docker, but I couldn't get it to work.
I also tried setting the DOCKER_HOST: tcp://docker:2375/ variable, but I still get the same error.
Cannot connect to the Docker daemon at tcp://docker:2375/. Is the docker daemon running?
My Gitlab Runner Helm values looks like this:
gitlabUrl: https://gitlab.mydomain.com
privileged: true
rbac:
create: true
runnerRegistrationToken: mytoken
runners:
config: |
[[runners]]
executor = "docker"
privileged = true
environment = ["DOCKER_TLS_CERTDIR="]
[runners.docker]
tls_verify = false
privileged = true
I tried using the Auto-DevOps .gitlab-ci.yml template.
My cluster is integrated into Gitlab using the cluster certificate method and running simple CI/CD like
test_project:
stage: test
script:
- npm run test
Works fine
I tried docker:18.09.7, docker:19.03.1 and docker:latest (and the corresponding -dind services, with or without setting DOCKER_HOST: tcp://docker:2375/)
I don't know what I'm missing.
// Update
I got this issue resolved with help from the Gitlab Support.
My issue was that I had to use [runners.kubernetes] config and set privileged to true there. I was only setting it in [runners.docker] as I thought that Kubernetes Gitlab Runners would that config.
Related
Since I need to use a self-hosted runner, the option to use existing marketplace SSH Actions is not viable because they tend to docker and build an image on GitHub Actions which fails because of a lack of permissions for an unknown user. Existing GitHub SSH Actions such as appleboy fail due to reliance on Docker for me.
Please note: Appleboy & other similar actions work great when using GitHub Actions Runners.
Since it's a self-hosted runner i.e. I have access to it all the time. So using an SSH profile and then using native-run command works pretty well.
To create an SSH profile & use it in GitHub Actions:
Create (if doesn't exist already) a "config" file in "~/.ssh/".
Add a profile in "~/.ssh/config". For example-
Host dev
HostName qa.bluerelay.com
User ec2-user
Port 22
IdentityFile /home/ec2-user/mykey/something.pem
Now to test it, on self-hosted runner run:
ssh dev ls
This should run the ls command inside the dev server.
4. Now since the SSH profile is set up, you can easily run remote SSH commands using GitHub Actions. For example-
name: Test Workflow
on:
push:
branches: ["main"]
jobs:
build:
runs-on: self-hosted
steps:
- uses: actions/checkout#v2
- name: Download Artifacts into Dev
run: |
ssh dev 'cd GADeploy && pwd && sudo wget https://somelink.amazon.com/web.zip'
- name: Deploy Web Artifact to Dev
run: |
ssh dev 'cd GADeploy && sudo ./deploy-web.sh web.zip'
This works like a charm!!
Independently of the runner itself, you would need first to check if SSH does work from your server
curl -v telnet://github.com:22
# Assuming your ~/.ssh/id_rsa.pub is copied to your GitHub profile page
git ls-remote git#github.com:you/YourRepository
Then you can install your runner, without Docker.
Finally, your runner script can execute jobs.<job_id>.steps[*].run commands, using git with SSH URLs.
I am a newbie in gitlabci. I want to understand why do we need docker dind image in order to build a docker image in GitLab CI jobs. Why can't we use the docker executor and run docker commands under scripts?
When we register docker executor gitlab runner, we choose one image..
Again inside gitlabci, we choose an image under image: or services: fields. So does that mean this GitLab CI job container runs inside the docker executor container?
why do we need docker dind image in order to build a docker image in GitLab CI jobs. Why can't we use the docker executor and run docker commands under scripts?
This partly depends on how you have configured your GitLab runner.
Why docker doesn't work inside containers
When you invoke docker commands, they are really talking to a docker daemon which is needed to perform builds and carry out other docker commands. Typically, jobs running under the docker executor do not have access to any docker daemon by default. It's the same kind of problem you would face if you tried to run docker inside of a docker container you started locally.
Even if I can run docker successfully on my host:
$ docker run --rm docker /bin/sh -c 'hello from container $HOSTNAME'
hello from container 2b51479b11b1
I cannot run docker inside the container
$ docker run --rm docker /bin/sh -c 'docker info'
errors pretty printing info
Client:
Context: default
Debug Mode: false
Server:
ERROR: error during connect: Get "http://docker:2375/v1.24/info": dial tcp: lookup docker on 192.168.65.5:53: no such host
The same error would happen trying to run any other significant docker command like build, run, etc.
An exception to this would be if you configured your GitLab runner to run containers in privileged mode and mount /var/run/docker.sock to all your jobs (this would not be advisable) in which case all your jobs could talk directly to the docker daemon on the host. Another exception might be if you use the shell executor instead and you have docker installed on the host where the runner is running.
How the dind service fixes this
The docker:dind service is a daemon that is created just for your job. This is incredibly important because it can prevent concurrent jobs from stepping on one another or being able to escalate access where they might not otherwise have it.
When the build starts, the GitLab runner will create two containers: your job container and the docker:dind container; they are linked together. When your job invokes docker commands, your job connects to the docker:dind container, which then carries out the requested commands.
Any containers created by your job (say, by invoking docker run or docker build as part of your job) are managed by the daemon running on the docker:dind container, not the host daemon. If you run docker ps inside the job, you'll notice that none of the containers run on the host daemon are listed, despite the fact that if you ran docker ps on the host, you would see the job container, the dind container, and any other running containers.
To clarify your other questions:
When we register docker executor gitlab runner, we choose one image
The image specified in your runner configuration is simply the default docker image to be used if a job doesn't declare any image: key. It does not affect how the runner runs in any way.
inside gitlabci, we choose an image under image: or services: fields
When the docker executor runs your job, it uses docker run to do so. The image: key determines which image is used to run your job. Similarly, services: define the image used for service containers -- service containers are siblings to the job container and are connected with links.
So does that mean this GitLab CI job container runs inside the docker executor container?
No. I'd also like to clear up: the runner/executor doesn't run in a container, necessarily. Runners might be installed as a Windows service, or simply even a process running directly on a system. You can use runners that happen to be inside containers, but it doesn't materially affect how jobs are run.
In any case, the containers where your job run are generally always going to be run directly by the host docker daemon.
I have a CakePHP Application with codeception-plugin for testing.'
Locally I run it in a ddev docker environment and everything works fine.
Trying to run automated tests with gitlab-ci gives me following error:
Running with gitlab-runner 11.1.0 (081978aa)
on shared runner 601c0f11
Using Docker executor with image kevinliteon/cakephp:php7 ...
Starting service mysql:latest ...
Pulling docker image mysql:latest ...
Using docker image sha256:6a834f03bd02bb88cdbe0e289b9cd6056f1d42fa94792c524b4fddc474dab628 for mysql:latest ...
Waiting for services to be up and running...
*** WARNING: Service runner-601c0f11-project-94-concurrent-0-mysql-0 probably didn't start properly.
Health check error:
service "runner-601c0f11-project-94-concurrent-0-mysql-0-wait-for-service" timeout
Health check container logs:
Service container logs:
2018-10-04T12:12:18.904025613Z Initializing database
2018-10-04T12:12:18.925096235Z 2018-10-04T12:12:18.919745Z 0 [Warning] [MY-011070] [Server] 'Disabling symbolic links using --skip-symbolic-links (or equivalent) is the default. Consider not using this option as it' is deprecated and will be removed in a future release.
2018-10-04T12:12:18.925195518Z 2018-10-04T12:12:18.919970Z 0 [System] [MY-013169] [Server] /usr/sbin/mysqld (mysqld 8.0.12) initializing of server in progress as process 30
2018-10-04T12:12:50.330736417Z 2018-10-04T12:12:50.330487Z 5 [Warning] [MY-010453] [Server] root#localhost is created with an empty password ! Please consider switching off the --initialize-insecure option.
*********
Pulling docker image kevinliteon/cakephp:php7 ...
Using docker image sha256:bd4a83b02647ad93a356b343d2ce5ae3a9a1177aea2cd76c61b009abc7df8990 for kevinliteon/cakephp:php7 ...
Running on runner-601c0f11-project-94-concurrent-0 via d7f4a5e71b47...
Fetching changes...
Removing vendor/
HEAD is now at 92cb022 test
Checking out 92cb0223 as deployment...
Skipping Git submodules setup
Checking cache for default...
Successfully extracted cache
$ vendor/bin/codecept run Unit
Codeception PHP Testing Framework v2.3.9
Powered by PHPUnit 6.5.13 by Sebastian Bergmann and contributors.
In Db.php line 308:
Db: SQLSTATE[HY000] [2002] Connection refused while creating PDO connection
My gitlab-ci.yml (partly):
services:
- mysql:latest
variables:
MYSQL_ROOT_PASSWORD: mysql123456789
MYSQL_DATABASE: test_db
MYSQL_USER: db
MYSQL_PASSWORD: db
build:
...
codecept:Unit:
stage: test
script:
- vendor/bin/codecept run Unit
In my codeception.yml I configured the Db module:
modules:
config:
Db:
dsn: 'mysql:host=mysql;dbname=test_db'
user: 'db'
password: 'db'
cleanup: true # reload dump between tests
populate: true # load dump before all tests
reconnect: true
I also tryed using the root user - without success.
Problem is, that I can not connect to the DB for whatever reasons... Maybe the warnings while initializing the service container have something to do with that, but I could not figure out how to fix them or if this is the problem.
I really tried a lot of things without any success! Basically my code depends on the documentations of gitlab-ci and codeception so it should work.
Anybody implemented this scenario successfully or know what I'm doing wrong?
Thanks for any help!
I want to answer how I solved it:
First thing was, I had to add the env-varibale "db_dsn" like this:
export db_dsn="mysql://user:paswd#host/db"
Then I still got the health-check error. Only way I found to successfully setup was to use another docker image for the db-service. I choose "mariadb:latest" - and then it worked for me.
On my local dev machine I use docker.from_env(version=auto) (docker-py) to get a docker client without any issues. If I run the same statement inside a gitlab-ci job which run inside a docker container (or with gitlab-runner exec docker ... during local debugging) to get a docker client I get a DockerException: Error while fetching server API version: ('Connection aborted.', error(2, 'No such file or directory')). How can I get a docker client with docker-py inside a gitlab-ci job which runs inside a docker container?
I think you'll need to build and deploy your own runner with the following binding '/var/sock/docker.sock:/var/sock/docker.sock.
Gitlab provides a .gitlab-ci.yml template for building and publishing images to its own registry (click "new file" in one of your project, select .gitlab-ci.yml and docker). The file looks like this and it works out of the box :)
# This file is a template, and might need editing before it works on your project.
# Official docker image.
image: docker:latest
services:
- docker:dind
before_script:
- docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY
build-master:
stage: build
script:
- docker build --pull -t "$CI_REGISTRY_IMAGE" .
- docker push "$CI_REGISTRY_IMAGE"
only:
- master
build:
stage: build
script:
- docker build --pull -t "$CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG" .
- docker push "$CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG"
except:
- master
But by default, this will publish to gitlab's registry. How can we publish to docker hub instead?
No need to change that .gitlab-ci.yml at all, we only need to add/replace the environment variables in project's pipeline settings.
1. Find the desired registry url
Using hub.docker.com won't work, you'll get the following error:
Error response from daemon: login attempt to https://hub.docker.com/v2/ failed with status: 404 Not Found
Default docker hub registry url can be found like this:
docker info | grep Registry
Registry: https://index.docker.io/v1/
index.docker.io is what I was looking for.
2. Set the environment variables in gitlab settings
I wanted to publish gableroux/unity3d images using gitlab-ci, here's what I used in Gitlab's project > Settings > CI/CD > Variables
CI_REGISTRY_USER=gableroux
CI_REGISTRY_PASSWORD=********
CI_REGISTRY=docker.io
CI_REGISTRY_IMAGE=index.docker.io/gableroux/unity3d
CI_REGISTRY_IMAGE is important to set.
It defaults to registry.gitlab.com/<username>/<project>
regsitry url needs to be updated so use index.docker.io/<username>/<project>
Since docker hub is the default registry when using docker, you can also use <username>/<project> instead. I personally prefer when it's verbose so I kept the full registry url.
This answer should also cover other registries, just update environment variables accordingly. 🙌
To expand on the GabLeRoux's answer,
I had issues on the pushing stage of the GitLab CI build:
denied: requested access to the resource is denied
By changing my CI_REGISTRY to docker.io (remove the index.) I was able to successfully push.