I use drone as CI and want to know how I can disable simultaneous build. What's happening is that when I submit two commits to git repo, drone will trigger two build on each of the submit. How can I let the second build wait until the first one finish?
Regarding the open source version of Drone: set the DOCKER_MAX_PROCS environment variable of your drone agent to 1, i.e. docker run -e DOCKER_MAX_PROCS=1 [...] drone/drone:0.5 agent. The agent will run one build concurrently, other builds will queue up.
See the Installation Reference section in the readme for more info.
Related
I'm using a webhook to trigger my Gitlab pipeline. Sometimes, this trigger is triggered a bunch of times, but my pipelines only has to run the last one (static site generation). Right now, it will run as many pipelines as I have triggered. My pipelines takes 20 minutes so sometimes it's running the rest of the day, which is completely unnecessary.
https://docs.gitlab.com/ee/ci/yaml/#interruptible and https://docs.gitlab.com/ee/user/project/pipelines/settings.html#auto-cancel-pending-pipelines only work on pushed commits, not on triggers
A similar problem is discussed in gitlab-org/gitlab-foss issue 41560
Example of a use-case:
I want to always push the same Docker "image:tag", for example: "myapp:dev-CI". The idea is that "myapp:dev-CI" should always be the latest Docker image of the application that matches the HEAD of the develop branch.
However if 2 commits are pushed, then 2 pipelines are triggered and executed in paralell. Then the latest triggered pipeline often finishes before the oldest one.
As a consequence the pushed Docker image is not the latest one.
Proposition:
As a workaround for *nix you can get running pipelines from API and wait until they finished or cancel them with the same API.
In the example below script checks for running pipelines with lower id's for the same branch and sleeps.
jq package is required for this code to work.
Or:
Create a new runner instance
Configure it to run jobs marked as deploy with concurrency 1
Add the deploy tag to your CD job.
It's now impossible for two deploy jobs to run concurrently.
To guard against a situation where an older pipeline may run after a new one, add a check in your deploy job to exit if the current pipeline ID is less than the current deployment.
Slight modification:
For me, one slight change: I kept the global concurrency setting the same (8 runners on my machine so concurrency: 8).
But, I tagged one of the runners with deploy and added limit: 1 to its config.
I then updated my .gitlab-ci.yml to use the deploy tag in my deploy job.
Works perfectly: my code_tests job can run simultaneously on 7 runners but deploy is "single threaded" and any other deploy jobs go into pending state until that runner is freed up.
I have a drone.yml running 3 pipelines on my github repo. One for pullrequests that get run as soon as some one submits a pull request, a pipeline for releases that makes docker containers and outputs a docker-compose, and now I'm making a pipeline that runs integration tests after a merge into master.
One of the steps is that it updates a test server, which is making the task challenging. Is there a way to force this specific drone pipeline to only run if there isn't any other of this pipeline running?
You can used depends_on to force the order of the pipelines execution.
Pipeline: Graph Execution
We upgraded from Gitlab 7.11.4 to 9 in one fell swoop (by accident). Now we are trying to get CI set up the way it use to run for us before. I understand that CI is an integrated thing now.
One of my coworkers got a multi-runner thing going. The running command looks like so:
/usr/bin/gitlab-ci-multi-runner run --working-directory /home/gitlab-runner --config /etc/gitlab-runner/config.toml --service gitlab-runner --syslog --user gitlab-runner
But previously we had 1 runner for each project and we had a user associated for each project. So, if we have 2 projects called "portal" and "engine", we would have users created thusly:
gitlab-runner-fps-portal
gitlab-runner-fps-engine
And being users, they would have home folders like:
/home/gitlab-runner-fps-portal
/home/gitlab-runner-fps-engine
In the older version of CI, you'd have a config.yml with the url of CI and the runners token. Now you have config.toml.
I want to "divorce" the engine runner from this multi setup which runs under user "gitlab-runner" and have its own runner that runs under "gitlab-runner-fps-engine".
Easy to do? Right now since all of this docker business is new to us, we're continuing on to use "shell" as our executor in gitlab, if that information is useful.
There are at least two ways you can do it:
Register a specific runner in each of the projects and disable the shared runners.
Use tags to specify the job must be run on a specific runner. This way you can have some CI jobs run on your defined environment while others (like lint for example) can be run on tagged shared runners.
I want to deploy a continuous integration platform based on Jenkins. As I have various kinds of projects (PHP / Symfony, node, angular, …) and as I want these tests to run both locally and on Jenkins, I was thinking about using Dockers containers.
The process I’m aiming for is :
A merge request is opened on Github / Gitlab
A webhook notifies Jenkins of the merge request
Jenkins pulls the repo, builds the containers and runs a shell script to execute the tests
Once the tests are finished, Jenkins retrieves the results from one of the containers (through a shared volume) and process the results.
I do not want Jenkins to be in a container.
With this kind of process, I’m hoping to be able to run very easily the tests on each developer machine with something like a docker-composer up and then in one of the container ./tests all.
I’m not very familiar with Jenkins. I’ve read a lot of documentation, but most of them suggested to define Jenkins slaves for each kind of projects beforehand. I would like everything to be as dynamic as possible and require as less configuration on Jenkins as possible.
I would appreciate a description of your test process if you have ever implemented something similar. If you think what I’m aiming for is impossible, I would also appreciate if you could explain to me why.
A setup I suggest is Docker in Docker.
The base is a derived Docker image, which extends the jenkins:2.x image by adding a Docker commandline client.
The Jenkins is started as a container with its home folder (a folder e.g. /var/jenkins_home mounted from the Docker host) and the Docker socket file to be able to start Docker containers from Jenkins build jobs.
docker run -d --name jenkins -v /var/jenkins_home:/var/jenkins_home -v /var/run/docker.sock:/var/run/docker.sock ... <yourDerivedJenkinsImage>
To check, if this setup is working just execute following command after starting the Jenkins container:
docker exec jenkins docker version
If the "docker version" output does NOT show:
Is the docker daemon running on this host?
Everythin is fine.
In your build jobs, you could configure the process you mentioned above. Let Jenkins simply check out the repository. The repository should contain your build and test scripts.
Use a freestyle build job with a shell execution. A shell execution could look like this:
docker run --rm --volumes-from jenkins <yourImageToBuildAndTestTheProject> bash $WORKSPACE/<pathToYourProjectWithinTheGitRepository>/build.sh
This command simply starts a new container (to build and/or test your project) with the volumes from jenkins. Which means that the cloned repository will be available under $WORKSPACE. So if you run "bash $WORKSPACE/<pathToYourProjectWithinTheGitRepository>/build.sh" your project will be built within a container of "yourImageToBuildAndTestTheProject". After running this, you could start other containers for integration tests or combine this with "docker-compose" by installing it on the derived Jenkins image.
Advantages are the minimal configuration affort you have within Jenkins - only the SCM configuration for cloning the GIT repository is required. Since each Jenkins job uses the Docker client directly you could use for each project one or Docker image to build and/or test, WITHOUT further Jenkins configuration.
If you need additional configuration e.g. SSH keys or Maven settings, just put them on the Docker host and start the Jenkins container with the additional volumes, which contain those configuration files.
Using this Docker option within the shell execution of your build jobs:
--volumes-from jenkins
Automatically adds workspace and configuration files to each of your build jobs.
Is there a way to kill a build in drone before it finishes or times out?
The default timeout in drone is 6 hours (https://github.com/drone/drone/blob/master/cmd/drone/drone.go#L32)
And if you have a mistake in your makefile that just get's stuck then you need to wait for 6 hours.
This is specially annoying if you have limited number of simultaneous builds.
My question is about the self hosted, open source version, not the hosted version if it makes any difference.
You can stop drone build using CLI:
drone build stop <repo/name> <build>
If build cannot be stopped/canceled, you can kill it:
drone build kill <repo/name> <build>
See more commands in drone CLI docs.
I just pushed a new commit and it automatically stopped the stuck build and started a new one. No need to wait for 6 hours. ;)
This is possible from the UI in Drone 0.4.
To stop the build using the drone cli, use the following command:
drone build stop <root/name> <DRONE_BUILD_NUMBER>
Make sure the following are exported:
export DRONE_SERVER=https://drone.server.com
export DRONE_TOKEN=<secret_drone_token>
It is also possible to stop the build using API:
DELETE /api/repos/{owner}/{repo}/builds/{build}