How to publish docker images to docker hub from gitlab-ci - gitlab-ci

Gitlab provides a .gitlab-ci.yml template for building and publishing images to its own registry (click "new file" in one of your project, select .gitlab-ci.yml and docker). The file looks like this and it works out of the box :)
# This file is a template, and might need editing before it works on your project.
# Official docker image.
image: docker:latest
services:
- docker:dind
before_script:
- docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY
build-master:
stage: build
script:
- docker build --pull -t "$CI_REGISTRY_IMAGE" .
- docker push "$CI_REGISTRY_IMAGE"
only:
- master
build:
stage: build
script:
- docker build --pull -t "$CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG" .
- docker push "$CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG"
except:
- master
But by default, this will publish to gitlab's registry. How can we publish to docker hub instead?

No need to change that .gitlab-ci.yml at all, we only need to add/replace the environment variables in project's pipeline settings.
1. Find the desired registry url
Using hub.docker.com won't work, you'll get the following error:
Error response from daemon: login attempt to https://hub.docker.com/v2/ failed with status: 404 Not Found
Default docker hub registry url can be found like this:
docker info | grep Registry
Registry: https://index.docker.io/v1/
index.docker.io is what I was looking for.
2. Set the environment variables in gitlab settings
I wanted to publish gableroux/unity3d images using gitlab-ci, here's what I used in Gitlab's project > Settings > CI/CD > Variables
CI_REGISTRY_USER=gableroux
CI_REGISTRY_PASSWORD=********
CI_REGISTRY=docker.io
CI_REGISTRY_IMAGE=index.docker.io/gableroux/unity3d
CI_REGISTRY_IMAGE is important to set.
It defaults to registry.gitlab.com/<username>/<project>
regsitry url needs to be updated so use index.docker.io/<username>/<project>
Since docker hub is the default registry when using docker, you can also use <username>/<project> instead. I personally prefer when it's verbose so I kept the full registry url.
This answer should also cover other registries, just update environment variables accordingly. 🙌

To expand on the GabLeRoux's answer,
I had issues on the pushing stage of the GitLab CI build:
denied: requested access to the resource is denied
By changing my CI_REGISTRY to docker.io (remove the index.) I was able to successfully push.

Related

GitHub Actions: How to run remote SSH commands using a self hosted runner

Since I need to use a self-hosted runner, the option to use existing marketplace SSH Actions is not viable because they tend to docker and build an image on GitHub Actions which fails because of a lack of permissions for an unknown user. Existing GitHub SSH Actions such as appleboy fail due to reliance on Docker for me.
Please note: Appleboy & other similar actions work great when using GitHub Actions Runners.
Since it's a self-hosted runner i.e. I have access to it all the time. So using an SSH profile and then using native-run command works pretty well.
To create an SSH profile & use it in GitHub Actions:
Create (if doesn't exist already) a "config" file in "~/.ssh/".
Add a profile in "~/.ssh/config". For example-
Host dev
HostName qa.bluerelay.com
User ec2-user
Port 22
IdentityFile /home/ec2-user/mykey/something.pem
Now to test it, on self-hosted runner run:
ssh dev ls
This should run the ls command inside the dev server.
4. Now since the SSH profile is set up, you can easily run remote SSH commands using GitHub Actions. For example-
name: Test Workflow
on:
push:
branches: ["main"]
jobs:
build:
runs-on: self-hosted
steps:
- uses: actions/checkout#v2
- name: Download Artifacts into Dev
run: |
ssh dev 'cd GADeploy && pwd && sudo wget https://somelink.amazon.com/web.zip'
- name: Deploy Web Artifact to Dev
run: |
ssh dev 'cd GADeploy && sudo ./deploy-web.sh web.zip'
This works like a charm!!
Independently of the runner itself, you would need first to check if SSH does work from your server
curl -v telnet://github.com:22
# Assuming your ~/.ssh/id_rsa.pub is copied to your GitHub profile page
git ls-remote git#github.com:you/YourRepository
Then you can install your runner, without Docker.
Finally, your runner script can execute jobs.<job_id>.steps[*].run commands, using git with SSH URLs.

Docker Tag Error 25 on gitlab-ci.yml trying to start GitLab Pipeline

I'm going through the "Scalable FastAPI Application on AWS" course. My gitlab-ci.yml file is below.
stages:
- docker
variables:
DOCKER_DRIVER: overlay2
DOCKER_TLS_CERTDIR: "/certs"
cache:
key: ${CI_JOB_NAME}
paths:
- ${CI_PROJECT_DIR}/services/talk_booking/.venv/
build-python-ci-image:
image: docker:19.03.0
services:
- docker:19.03.0-dind
stage: docker
before_script:
- cd ci_cd/python/
script:
- docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY
- docker build -t registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim .
- docker push registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim
My Pipeline fails with this error:
See https://docs.docker.com/engine/reference/commandline/login/#credentials-store
Login Succeeded
$ docker build -t registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim .
invalid argument "registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim" for "-t, --tag" flag: invalid reference format
See 'docker build --help'.
Cleaning up project directory and file based variables
ERROR: Job failed: exit code 125
It may or may not be relevant but the Container Registry for the GitLab project says there's a Docker connection error.
Thanks
I created a new GitLab account with a new username and things are working now. The underscore does appear to have been the issue.

BitBucket deployment using SSH keys to remote server

I am trying to write a YAML pipeline script to deploy files that have been altered from my bitbucket repository to my remote server using ssh keys. The document that I have in place at the moment was copied from bitbucket itself and has errors:
pipelines:
default:
- step:
name: Deploy to test
deployment: test
script:
- pipe: atlassian/sftp-deploy:0.3.1
- variables:
USER: $USER
SERVER: $SERVER
REMOTE_PATH: $REMOTE_PATH
LOCAL_PATH: $LOCAL_PATH
I am getting the following error
Configuration error
There is an error in your bitbucket-pipelines.yml at [pipelines > default > 0 > step > script > 1]. To be precise: Missing or empty command string. Each item in this list should either be a single command string or a map defining a pipe invocation.
My ssh public and private keys are setup in bitbucket along with the fingerprint and host. The variables have also been setup.
How do I go about setting up my YAML deploy script to connect to my remote server via ssh and transfer the files?
Try to update the variables section become:
- variables:
- USER: $USER
- SERVER: $SERVER
- REMOTE_PATH: $REMOTE_PATH
- LOCAL_PATH: $LOCAL_PATH
Here is am example about how to set variables: https://confluence.atlassian.com/bitbucket/configure-bitbucket-pipelines-yml-792298910.html#Configurebitbucket-pipelines.yml-ci_variablesvariables
Your directive - step has to be intended.
I have bitbucket-pipelines.yml like that (using rsync instead of ssh):
# This is a sample build configuration for PHP.
# Check our guides at https://confluence.atlassian.com/x/e8YWN for more examples.
# Only use spaces to indent your .yml configuration.
# -----
# You can specify a custom docker image from Docker Hub as your build environment.
image: php:7.2.1-fpm
pipelines:
default:
- step:
script:
- apt-get update
- apt-get install zip -y
- apt-get install unzip -y
- apt-get install libgmp3-dev -y
- curl -sS https://getcomposer.org/installer | php -- --install-dir=/usr/local/bin --filename=composer
- composer install
- cp .env.example .env
#- vendor/bin/phpunit
- pipe: atlassian/rsync-deploy:0.2.0
variables:
USER: $DEPLOY_USER
SERVER: $DEPLOY_SERVER
REMOTE_PATH: $DEPLOY_PATH
LOCAL_PATH: '.'
I suggest to use their online editor in repository for editing bitbucket-pipelines.yml, it checks all formal yml structure and you can't commit invalid file.
Even if you check file on some other yaml editor, it may look fine, but not necessary according to bitbucket specification. Their online editor does fine job.
Also, I suggest to visit their community on atlasian community as it's very active, sometimes their staff members are providing answers.
However, I struggle with plenty dependencies needed to run tests properly. (actual bitbucket-pipelines.yml is becoming bigger and bigger).
Maybe there is some nicely prepared Docker image for this job.

Gitlab-CI cannot clone

I have a very basic integration configured for Gitlab-CI but it fails almost at the beginning when it has to clone the code.
My integration is this:
image: node:latest
stages:
- build
- test
cache:
paths:
- node_modules/
- dist/
build-prod:
stage: build
script:
- npm install
- npm run build-prod
artifacts:
paths:
- node_modules/
- dist/
test_with_karma:
stage: test
script: ng test
And the error that I get is this:
Running with gitlab-runner 11.7.0 (8bb608ff)
on fakehost 2eaf11ea
Using Docker executor with image node:latest ...
Pulling docker image node:latest ...
Using docker image sha256:8c67bfd7b95bdc535edc4a4144f5392b0f73efd6385fbcb47747d028d7059359 for node:latest ...
Running on runner-2eaf11ea-project-56-concurrent-0 via fakehost...
Cloning repository...
Cloning into '/builds/redacted/frontend'...
remote: You are not allowed to download code from this project.
fatal: unable to access 'https://gitlab-ci-token:xxxxxxxxxxxxxxxxxxxx#working-domain.com/redacted/frontend.git/': The requested URL returned error: 403
/bin/bash: line 65: cd: /builds/redacted/frontend: No such file or directory
ERROR: Job failed: exit code 1
What is the problem here?
Check if this is covered by gitlab-org/gitlab-ce issue 39469
YAY - it works for me. This problem seems to have multiple solutions.
The one that worked for me is #44855
To summarize. Being an Administrator on Gitlab does not mean you have the "access" to do whatever you want to do in Gitlab.
"Unable to access" permissions applies to the person who is logged into Gitlab and running the job.
To fix the problem - the person / account running the job must be a member (master) of the project.
This will apply to private projects.
It is not necessary to make a private project Public even though that appears to fix the problem. GITLAB suggests you must have https for the project to work you can use http.
SOLUTION - add your account to the project even if you are the Administrator
And:
Conrad has described it correctly.
You need to have rights to the project to run pipeline, however, as administrator, you can start any pipeline.
I've got the case when the user being Admin in Gitlab could push his commit from command line, although theoretically having no rights to project - and the pipeline has failed.
This inconsistency need to be fixed, either Admin user should not be able to push/start pipeline, having no rights for it, or he should authomatically be granted all rights to all projects. I'd prefer the first one, because it separates gitlab administration from project rights. Sometimes I prefer not having full rights, just like working as non-root under Linux.

Drone CI get and use build metadata

I'm using Drone CI (0.7) in a self-hosted system. It's hooked up to GitLab and my private registry and working great!
However, I'd like to do more with the .drone.yml file. Especially when publishing images to the registry.
The docs show how to do substitution, but doesn't explain how those variables are set. Below is an example from the docs:
pipeline:
docker:
image: plugins/docker
+ tags: ${DRONE_TAG}
This is exactly what I'd like to do. Create a git tag on the repo (on the release branch) and have that tag become the tag for my Docker image.
I've tried git tag -a v1.0 -m "Initial release" then git push origin v1.0. The Drone CI build kicks off as I've set the Tag Hooks for the repository, but when the build completes, and publishes the image to the registry, the tag is set to latest. This also happens when pushing a regular code change commit.
Is this something that needs to be done with the CLI, or am I missing something in my .drone.yml file (posted below)? It looks like an environment variable I would need to set, but that seems strange to have to set that (I'm assuming) just before committing code to start a build...
Any and all help is appreciated!
pipeline:
build:
image: node:latest
commands:
- npm install
- npm test
docker:
image: plugins/docker
repo: private.registry.com/tester
registry: private.registry.com
secrets: [ docker_username, docker_password ]
tags: ${DRONE_TAG}
when:
branch: release
Actually your docker step is only being triggered when you push a new commit to the branch release, you should use the tag event.
Yo should have some similar configuration
tags:
- latest
- ${DRONE_TAG##v}
when:
event: tag
note ${DRONE_TAG##v} will strip the prefix v, since you are naming your tag v1.0 drone will parse this into
tags:
- latest
- 1.0
when:
event: tag