I'm using Drone CI (0.7) in a self-hosted system. It's hooked up to GitLab and my private registry and working great!
However, I'd like to do more with the .drone.yml file. Especially when publishing images to the registry.
The docs show how to do substitution, but doesn't explain how those variables are set. Below is an example from the docs:
pipeline:
docker:
image: plugins/docker
+ tags: ${DRONE_TAG}
This is exactly what I'd like to do. Create a git tag on the repo (on the release branch) and have that tag become the tag for my Docker image.
I've tried git tag -a v1.0 -m "Initial release" then git push origin v1.0. The Drone CI build kicks off as I've set the Tag Hooks for the repository, but when the build completes, and publishes the image to the registry, the tag is set to latest. This also happens when pushing a regular code change commit.
Is this something that needs to be done with the CLI, or am I missing something in my .drone.yml file (posted below)? It looks like an environment variable I would need to set, but that seems strange to have to set that (I'm assuming) just before committing code to start a build...
Any and all help is appreciated!
pipeline:
build:
image: node:latest
commands:
- npm install
- npm test
docker:
image: plugins/docker
repo: private.registry.com/tester
registry: private.registry.com
secrets: [ docker_username, docker_password ]
tags: ${DRONE_TAG}
when:
branch: release
Actually your docker step is only being triggered when you push a new commit to the branch release, you should use the tag event.
Yo should have some similar configuration
tags:
- latest
- ${DRONE_TAG##v}
when:
event: tag
note ${DRONE_TAG##v} will strip the prefix v, since you are naming your tag v1.0 drone will parse this into
tags:
- latest
- 1.0
when:
event: tag
Related
I have 7 stages in my pipeline. I need ruby for 3 of the stages.
things I have tried two different options,
Install ruby on each of the required stage,
Install ruby as part of the before_script section
Using before_script takes up too much of time trying to install ruby on the 4 other stages that does not require it.
Is there a way to do install dependencies as part of one stage and carry it forward for rest of the stages.
example yml
image: ubuntu:21.10
before_script:
- apt update
- apt install ruby-full
- apt install python3.8
stages:
- s1
- s2
- s3
- s4
s1:
stage: s1
script: ruby s1.rb
s2:
stage: s2
script: ruby s2.rb
s3:
stage: s3
script: python3 s3.py
s4:
stage: s4
script: python3 s4.py
There's a few elements here to understand. Generally, every job starts with the same fresh environment. The only differences to this would be files passed through artifacts: or files restored from cache: configurations. Actions performed in one job generally otherwise have no effect on any other jobs.
Using before_script takes up too much of time trying to install ruby on the 4 other stages that does not require it.
It's also important to know that before_script can be set for each job independently. If one job doesn't need it, just override the before_script: key in that job.
Anyhow. There are a few ways you might optimize your build speed with respect to dependencies:
Docker image containing your dependencies
Typically, you would just use a ruby image as your image: for jobs requiring ruby. Usually an official image from dockerhub will work, like ruby:3.1-alpine.
some_ruby_job:
image: "ruby:3.1-alpine"
script: # ruby is already available by default
- echo "hello ruby"
- ruby -v
some_other_job:
image: alpine:latest
script:
- echo "this job does not need ruby"
Making a custom docker image
If your dependencies are very complex, you may even choose to create your own docker images and push them to the project's container registry so you can use the custom image with all your dependencies as your image:.
You could even build an image in one stage and use it as the image: in subsequent stages. This example uses docker caching with --cache-from to further speed up that process.
build:
image: docker:19.03.12
stage: .pre
services:
- docker:19.03.12-dind
script:
- docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
- docker pull $CI_REGISTRY/group/project/image:$CI_BRANCH_NAME || true
- docker build --cache-from $CI_REGISTRY/group/project/image:$CI_BRANCH_NAME -t $CI_REGISTRY/group/project/image:$CI_BRANCH_NAME .
- docker push $CI_REGISTRY/group/project/image:$CI_BRANCH_NAME
some_ruby_job:
stage: test
# This is the image that was built in the previous stage!
image: $CI_REGISTRY/group/project/image:$CI_BRANCH_NAME
script:
- echo "all my dependencies are here!"
- ruby -v
Caching
To further speed things along, you may also choose to cache your ruby dependencies (say, if you install gems as part of your job)
Something like:
some_ruby_job:
stage: one
cache:
key:
files:
- Gemfile.lock
paths:
- vendor/ruby
# ...
That way the vendor/ruby directory is cached which will avoid the need to download the gems again in every stage.
Cache policy
You can also speed up caching behavior in subsequent stages by setting the cache policy to pull (to avoid time spent uploading the cache after the job). In other words, only one job is responsible for generating the cache, the other jobs reuse the same cache.
ruby_jobs_in_future_stages:
cache:
key:
files:
- Gemfile.lock
paths:
- vendor/ruby
policy: pull # only download the cache, don't upload it
I have a CI pipeline in Gitlab (relevant part only):
default:
image: docker:latest
variables:
DOCKER_APP_TAG: ${REGISTRY_URL}/${APP_NAME}:${CI_COMMIT_SHORT_SHA}
stages:
- build
.config:
only:
- branches
- merge_requests
- tags
except:
- triggers
tags:
- prod
build-app:
extends: .config
stage: build
script:
- docker build --target production -t ${DOCKER_APP_TAG} -f ${CI_PROJECT_DIR}/etc/node/Dockerfile .
When I build from a branch (i.e. push to main branch) everything works well. The docker build command is ran with the proper value available in S{DOCKER_APP_TAG}.
However after I create a TAG in GitLab (and a release), the build on this GitLab TAG fails at the docker build ... line complaining that the docker tag is not valid:
invalid argument "/:e5dc27fd" for "-t, --tag" flag: invalid reference format
The variables ${REGISTRY_URL} and ${APP_NAME} are not expanded. I have checked GitLab docs and the only limitations I see is if I was running in a service. But it is not the case.
What am I missing to expand properly the variables even with tag builds?
I am trying to execute my packer build into a Gitlab pipeline, i didn't find examples on the internet but i have seen there is a docker image, so i was hoping this yaml would do the job:
image: hashicorp/packer
stages:
- build
build:
stage: build
script:
- echo "Hello world"
- packer build ./definition.json
only:
- master
But i don't understand the behavior, the CI pull the image, clone the repo, then it ends up like this:
Skipping Git submodules setup
Usage: packer [--version] [--help] <command> [<args>]
Available commands are:
build build image(s) from template
console creates a console for testing variable interpolation
fix fixes templates from old versions of packer
inspect see components of a template
validate check that a template is valid
version Prints the Packer version
Usage: packer [--version] [--help] <command> [<args>]
Available commands are:
build build image(s) from template
console creates a console for testing variable interpolation
fix fixes templates from old versions of packer
inspect see components of a template
validate check that a template is valid
version Prints the Packer version
ERROR: Job failed: exit code 127
It doesn't even print my echo Hello World, and it prints 2 times how i should iterract with the CLI, why this behavior?
I found how to fix it, i had to change:
image: hashicorp/packer
into:
image:
name: hashicorp/packer
entrypoint:
- '/usr/bin/env'
- 'PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin'
I'm using drone 0.8.1 with gogs. When I commit to master a build is triggered. When I tag the repo in gogs, no build is triggered and no docker image built like I would expect. Here is my .drone.yml file:
pipeline:
docker:
image: plugins/docker
repo: larktech/mirror-repos
default_tags: true
when:
event: [ push, tag ]
Using gogs's release functionality, I tagged the repo v1.0.1, however no build ensued. Committing to master triggered the latest build as expected. What am I missing?
Currently drone does not support the release event webhook, which is what is called when I tag a release in the gogs release tab.
With help from https://discourse.drone.io/t/tags-not-building-for-me/1236 I learned that pushing tags manually will trigger my build:
git tag -a v1.0.6 -m 'another test'
git push origin v1.0.6
Gitlab provides a .gitlab-ci.yml template for building and publishing images to its own registry (click "new file" in one of your project, select .gitlab-ci.yml and docker). The file looks like this and it works out of the box :)
# This file is a template, and might need editing before it works on your project.
# Official docker image.
image: docker:latest
services:
- docker:dind
before_script:
- docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY
build-master:
stage: build
script:
- docker build --pull -t "$CI_REGISTRY_IMAGE" .
- docker push "$CI_REGISTRY_IMAGE"
only:
- master
build:
stage: build
script:
- docker build --pull -t "$CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG" .
- docker push "$CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG"
except:
- master
But by default, this will publish to gitlab's registry. How can we publish to docker hub instead?
No need to change that .gitlab-ci.yml at all, we only need to add/replace the environment variables in project's pipeline settings.
1. Find the desired registry url
Using hub.docker.com won't work, you'll get the following error:
Error response from daemon: login attempt to https://hub.docker.com/v2/ failed with status: 404 Not Found
Default docker hub registry url can be found like this:
docker info | grep Registry
Registry: https://index.docker.io/v1/
index.docker.io is what I was looking for.
2. Set the environment variables in gitlab settings
I wanted to publish gableroux/unity3d images using gitlab-ci, here's what I used in Gitlab's project > Settings > CI/CD > Variables
CI_REGISTRY_USER=gableroux
CI_REGISTRY_PASSWORD=********
CI_REGISTRY=docker.io
CI_REGISTRY_IMAGE=index.docker.io/gableroux/unity3d
CI_REGISTRY_IMAGE is important to set.
It defaults to registry.gitlab.com/<username>/<project>
regsitry url needs to be updated so use index.docker.io/<username>/<project>
Since docker hub is the default registry when using docker, you can also use <username>/<project> instead. I personally prefer when it's verbose so I kept the full registry url.
This answer should also cover other registries, just update environment variables accordingly. 🙌
To expand on the GabLeRoux's answer,
I had issues on the pushing stage of the GitLab CI build:
denied: requested access to the resource is denied
By changing my CI_REGISTRY to docker.io (remove the index.) I was able to successfully push.