I'm using GitLab's shared CI runners for training. Since this morning I can't build
my project because the pipeline's status is "pending".
Is it because the number of shared runners is maxed out ? Too busy running other people's
code ?
Is there a setting I need to check ? I know I can pay for a dedicated runner but I'm not
in commercial setting at this point therefore I'm sticking with shared ones.
Thank you for assistance.
In such situations I would highly recommend to make the best use of tags. You can use tags to select a specific runner from the list of all runners that are available for the project.
In this example, the job is run by a runner that
has both ruby and postgres tags defined.
job:
tags:
- ruby
- postgres
If you have your own runners setup then you can use tags to run different jobs on different platforms. For
example, if you have an OS X runner with tag osx and a Windows runner with tag
windows, you can run a job on each platform:
windows job:
stage:
- build
tags:
- windows
script:
- echo Hello, %USERNAME%!
osx job:
stage:
- build
tags:
- osx
script:
- echo "Hello, $USER!"
If this is still a problem then consider using your own private runner.
Related
I'm looking for a way to run parallels jobs in different runners. I have several powerful runners set up for GitLab CI. In general, it's ok to run jobs on the same runner because they're executed in Docker container.
However, now I have a Pipeline that jobs are executed in parallel and each job consumes lots of CPU and Mem.(it's by design, not an issue). If it's unlucky that GitLab CI schedules those jobs to the same runner, job fails.
And, I want this limitation applies to this project ONLY, as my runners have 30+ CPU and 120GB+ Memory.
Thanks in advance.
It is possible, if you have set up say two runners (either specific, shared or group runners) with tags.
Say, runner1 has tags runner1-ci, my-runner1
Similarly, runner2 has tags runner2-ci, my-runner2
Now, in your .gitlab-ci.yml file, you can use the tags like below, so a job will pick up that particular runner and execute the job.
image: maven:latest
stages:
- build
- test
- deploy
install_dependencies:
stage: build
tags:
- runner1-ci
script:
- pwd
- echo "Build"
test:
stage: test
tags:
- runner2-ci
script:
- echo "Testing"
deploy:
stage: deploy
tags:
- runner1-ci
script:
- echo "Deploy to nexus"
Note: This is just an example .gitlab-ci.yml to demonstrate the use of tags in pipeline.
How can I run different CI deployment scripts on merge to master depending on the labels attached to the merge request?
I have a repository from which I build different versions of my software. I keep it in one repository as the systems share 90% of the code but there are differences that defitively need code modifications. On merge requests all versions are buildt and a suite of tests is run. Usually I want to deploy on accepting the merge request.
As not always the changes are relevant for all systems I would like to attach labels to the merge request that decide which deployments scripts are run on accepting the merge request. I already tried to automatically decide on the changed code parts but this is not possible as often I expand a shared library but this is only relevant for one of the systems.
I am aware of variables but I don't know how to apply them on merge accept in YML like this
deploy:
stage: deploy
script:
...
only:
- master
Update on strategy:
As CI_MERGE_REQUEST_LABELS is not available with only:master I will try to do a beta deployment depending on merge request labels in only:merge-request. In only:master I will deploy the betas that have changed. This most likely will fit my needs. I will add it as a solution once it works.
I finally solved it this way:
My YML script has three stages:
stages:
- buildtest
- createbeta
- deploy
buildtest:
stage: buildtest
script:
- ... run unit tests
- ... build all systems
- ... run scripted tests on all systems
only:
refs:
- merge_requests
createbeta:
stage: createbeta
script:
- ... run setup and update package creation with parameter $CI_MERGE_REQUEST_LABELS
- ... run update package tests with parameter $CI_MERGE_REQUEST_LABELS
- ... run beta deployment scripts with parameter $CI_MERGE_REQUEST_LABELS (see text)
only:
refs:
- merge_requests
deploy:
stage: deploy
script:
- ... run production deployment scripts (see text)
only:
refs:
- master
The first stages are run on merge request creation.
As changes to shared libraries might affect all systems all builds and tests are run in stage "buildtest".
The scripts in stage "createbeta" check for existance of the merge request label for the corresponding system and are skipped if the system is not involved by the labels.
The script for beta deployment creates a signal file "deploy_me" in the beta folder (important) if it runs
When the request is merged the deployment script runs in stage "deploy". It checks for the existance of the "deploy_me" file and only deploys and informs via mail if the file exists.
This way I can easily decide which system I want to deploy by applying a labes to the merge request. I can thorowly test the new feature with the beta version and make sure that changes do not break the other systems as unittests and system tests are run for all systems.
As the GitLab runner runs in a Windows environment (yes, this makes sense as I work with Delphi) here is the way I find the system label in a Windows cmd file for those who are interested. I use %* as the labels are separated by spaces and treated as individual command line parameters.
echo %* | findstr /i /c:"MyCoolSystem" > nul
if %ERRORLEVEL% EQU 0 goto runit
rem If the label is not supplied with the merge request, do nothing
goto ok
:runit
... content
:ok
Perhaps this helps someone with a similar environment and similar workflow.
1. The Summary of the Problem
I would like a Travis CI setup that would let me run flutter driver tests inside an Android and an iOS environments. For this to happen, I expect I somehow have to install Flutter, Android and iOS in different environments.
2. What I Have So Far
Most of the posts I've been able to find on this topic are terribly outdated or feature incredibly complicated setups. Some of those that keep on appearing in my searches are:
Test Flutter apps on Travis, by Yegor Jbanov. This one covers unit and widget testing (flutter test), but not integration tests.
It's from early 2017 and Travis CI has maybe simplified its API, because I've managed to make it work with only this:
language: dart
dart:
- stable
dart_task:
- dartfmt
install:
- git clone https://github.com/flutter/flutter.git -b stable
script:
- ./flutter/bin/flutter doctor
- ./flutter/bin/flutter test
One resource that I've found very useful is the .travis.yml in the Flutter samples repo. The setup there seems very complicated to me though.
The closest I could get to what I wanted is similar to Maurice McCabe's Flutter unit, widget and integration testing with IOS and Android emulators on Travis-CI.
Again, this seems overcomplicated and outdated.
3. The Sketch of What I Have in Mind
The script and install steps in the example I mentioned previously could be replaced by jobs with stages. In this way, each stage would represent one sort of step. Unit and Widget stages in one, integration tests on Android and iOS in two others, which is similar to what Maurice McCabe and Flutter samples show. For example:
jobs:
include:
- stage: Flutter Test
language: dart
os: linux
install: git clone $FLUTTER_GITHUB -b stable
before_script:
- ./flutter/bin/flutter doctor
script:
- ./flutter/bin/flutter test
- stage: Integration Test on Android
os: linux
dist: trusty
language: android
android: # the things here are what probably needs to be fixed
components:
- build-tools-28.0.3
- android-28
install: git clone $FLUTTER_GITHUB -b stable
before_script:
- ./flutter/bin/flutter doctor
script:
- ./flutter/bin/flutter drive --target=test_driver/app.dart
If I could create a stage for the dartfmt task that would also be nice in terms of organization.
1. The Overview
Kudos to #MirceaMatei and Maurice McCabe for helping out in this.
I haven't yet been able to make Android Integration Tests work (the last stage of the code below), but at least iOS is working.
Android is much harder to get right because of different versions and licenses, something Apple does much better.
The code you will find below is part of a custom mono repo setup I'm working on.
The folder structure consists of an app and a packages folders.
I haven't yet segmented code coverage by those two folders either, so this setup is currently overwriting code coverage from one folder to another.
Using a top level install section makes Travis repeat that setup for every stage, so that avoids having to do that repeatedly for every stage.
I'm sharing my current — albeit incomplete — setup below, but continuous improvement will happen through this Github Gist, which is a much better way of interacting with the community to improve code than StackOverflow — it's not its purpose.
All in all, I think Travis CI is really dropping the ball when it comes to Flutter — and maybe Android in general. An easier alternative that has been hyped by many practitioners I've come into contact recently is Codemagic, which also offers code signing and automated deployment to the iOS and Android app stores.
2. Useful Resources
I didn't really like Travis documentation for the integration tests. People creating pure Android apps filed similar issues.
Anyway, here are some useful resources I've found during my searches:
Issue Comment on "Error: Target id is not valid. Use 'android list targets' to get the target ids."
“Invalid --abi armeabi-v7a for the selected target” with Google APIs
Is there a way to start android emulator in Travis CI build?
Travis-CI Android 28 licenses have not been accepted
Flutter Samples Travis Setup
#MirceaMatei's Travis Setup Recommendation
Travis Building Android Projects Documentation
Maurice McCabe's Unit and Integration Tests with Travis Article on Medium
3. The Code
Please, do not post code improvement suggestions here, but in the the Github Gist. I'll be updating the yaml code below myself every time a useful iteration arrives.
language: dart
env:
global:
- DARTSDK=./flutter/bin/cache/dart-sdk/bin
- DARTFMT=$DARTSDK/dartfmt
- FLUTTER=./flutter/bin/flutter
- FLUTTER_UP=../flutter/bin/flutter
- FLUTTER_GITHUB=https://github.com/flutter/flutter.git
- CODECOV=https://codecov.io/bash
- PACKAGES=packages
- APP=app
- ANDROID_API=28
install:
- git clone $FLUTTER_GITHUB -b stable --depth 1
- $FLUTTER doctor
- $FLUTTER pub get
jobs:
include:
- stage: Formatting
script:
- $DARTFMT -n $PACKAGES --set-exit-if-changed
- stage: Packages Flutter Test
script:
- $FLUTTER test --coverage $PACKAGES
after_success:
- bash <(curl -s $CODECOV)
- stage: App Flutter Test
script:
- cd $APP
- $FLUTTER_UP test --coverage
after_success:
- bash <(curl -s $CODECOV)
- stage: iOS Integration Tests
os: osx
osx_image: xcode11
before_script:
- open /Applications/Xcode.app/Contents/Developer/Applications/Simulator.app
- export HOMEBREW_NO_AUTO_UPDATE=1
- brew install libimobiledevice
- brew install ideviceinstaller
- brew install ios-deploy
- brew install cocoapods || echo 'ignore exit(1)'
- brew link --overwrite cocoapods
script:
- cd $APP
- $FLUTTER_UP driver --target=test_driver/app.dart77
- stage: Android Integration Tests
language: android
dist: trusty
android:
components:
- tools
- platform-tools
- build-tools-25.0.3
- android-24
- android-22
- extra-google-google_play_services
- extra-google-m2repository
- extra-android-m2repository
- sys-img-armeabi-v7a-android-22
licenses:
- 'android-sdk-preview-license-52d11cd2'
- 'android-sdk-license-.+'
- 'google-gdk-license-.+'
before_install:
- yes | sdkmanager "platforms;android-28"
- echo no | android create avd --force -n test -t android-22 --abi armeabi-v7a -c 32M
- emulator -avd test -no-audio -no-window &
- android-wait-for-emulator
- adb devices
- adb shell input keyevent 82 &
script:
- $FLUTTER --version
- $FLUTTER doctor
- cd $APP
- $FLUTTER_UP devices
- $FLUTTER_UP driver --target=test_driver/app.dart
You can take a look at this project
Sqlite Scaffolding Generator # Dart Framework ORM M8
I think it has all the components you are looking for. It's a Dart package that has, in the solution, an example integration project for Flutter.
The CI pipeline is built for Travis CI (based mostly on Maurice McCabe's article). It's possible that the travis.yml file has all the sections you need.
I got following environment and setup:
3 load balanced production servers and
one development server
a working auto deployment to my development server with
one gitlab-runner which connects by ssh to my dev-server and pulls the dev-branch and
a gitlab-ci.yml which is limited to my dev-branch
How can I achieve:
further auto-deploy to my development server if I push into my dev-branch
auto deploy to my 3 production servers if I push into my master-branch
The question is based on my current setup where following questions came up
can I let a gitlab-runner run just locally (also not in a docker container because I havn't installed one) so that its just up to the gitlab-ci.yml to differ on branches and deploy to the specific servers or
can I install multiple gitlab-runners which are taking aktion on just specific branches?
or is there another solution which lets me achieve my plans?
"Can I let a gitlab-runner run just locally (also not in a docker container because I havn't installed one) so that its just up to the
gitlab-ci.yml to differ on branches and deploy to the specific
servers"
Yes, register local runners as specific runners with Shell executors on the machines you want to deploy to so they can run local commands just like you use SSH now and only your specific projects can use them. Then take a look at the next sub-answer regarding tags.
"Can I install multiple gitlab-runners which are taking aktion on just specific branches?"
Use either tags to pin certain jobs to runners (e.g. your deploy job) or use only or except to pin jobs to branches or
tags. (e.g. your deploy_prod job only run on master branch)
Example .gitlab-ci.yml file (abstract):
deploy-dev:
tags:
- dev-runner
only:
- dev-branch
script:
- cd mydir
- git pull
deploy-prod:
tags:
- prod-runner
only:
- master
script:
- cd mydir
- git pull
We upgraded from Gitlab 7.11.4 to 9 in one fell swoop (by accident). Now we are trying to get CI set up the way it use to run for us before. I understand that CI is an integrated thing now.
One of my coworkers got a multi-runner thing going. The running command looks like so:
/usr/bin/gitlab-ci-multi-runner run --working-directory /home/gitlab-runner --config /etc/gitlab-runner/config.toml --service gitlab-runner --syslog --user gitlab-runner
But previously we had 1 runner for each project and we had a user associated for each project. So, if we have 2 projects called "portal" and "engine", we would have users created thusly:
gitlab-runner-fps-portal
gitlab-runner-fps-engine
And being users, they would have home folders like:
/home/gitlab-runner-fps-portal
/home/gitlab-runner-fps-engine
In the older version of CI, you'd have a config.yml with the url of CI and the runners token. Now you have config.toml.
I want to "divorce" the engine runner from this multi setup which runs under user "gitlab-runner" and have its own runner that runs under "gitlab-runner-fps-engine".
Easy to do? Right now since all of this docker business is new to us, we're continuing on to use "shell" as our executor in gitlab, if that information is useful.
There are at least two ways you can do it:
Register a specific runner in each of the projects and disable the shared runners.
Use tags to specify the job must be run on a specific runner. This way you can have some CI jobs run on your defined environment while others (like lint for example) can be run on tagged shared runners.