I have a cucumber test suite for my rails app, comprising around 500 Scenarios with about 5000 Steps between them.
I have set up my github repository to use Travis-CI, using the following .travis.yml.
language: ruby
rvm:
- "1.9.2"
script:
- RAILS_ENV=test bundle exec rake db:migrate --trace
- bundle exec cucumber -f progress -r features features/cards/base_game
- bundle exec cucumber -f progress -r features features/cards/basic_cards
- bundle exec cucumber -f progress -r features features/cards/intrigue
- bundle exec cucumber -f progress -r features features/cards/seaside
- bundle exec cucumber -f progress -r features features/cards/prosperity
- bundle exec cucumber -f progress -r features features/cards/interactions
before_script:
- cp config/database.travis.yml config/database.yml
- psql -c 'create database dominion_test' -U postgres
I have split the cucumber execution up as Travis was throwing Out Of Memory if I just ran bundle exec cucumber to run all the cases.
However, my most recent push spawned a Travis task which took over 50 minutes to run all my tests, and was therefore killed. Am I just being unreasonable with that many scenarios, or is there anything I could do to speed up execution?
Edit: In case it matters, I should clarify that my Scenarios don't test the GUI. They're testing the rules of a card-game server, so they invoke model methods directly.
I found a solution, after much googling, at this page of Travis' docs.
Basically, it's allowed (recommended, even!) to parallelise the runs. With the following .travis.yml, I end up with six concurrent jobs, none of which takes over 15 minutes, and which therefore all run to completion:
language: ruby
rvm:
- "1.9.2"
env:
- CARD_SET=base_game
- CARD_SET=basic_cards
- CARD_SET=intrigue
- CARD_SET=seaside
- CARD_SET=prosperity
- CARD_SET=interactions
script:
- RAILS_ENV=test bundle exec rake db:migrate --trace
- bundle exec cucumber -f progress -r features features/cards/$CARD_SET
before_script:
- cp config/database.travis.yml config/database.yml
- psql -c 'create database dominion_test' -U postgres
Related
I've been working on setting up an automated RPM build and I'd like to perform a simple test on the SPEC file before proceeding with any build steps. The problem I am having is that the job always seems to jump to the deploy stage. Here is the relevant snippet from my .gitlab-ci.yml:
stages:
- test
- build
- deploy
job1:
stage: test
script:
# Test the SPEC file
- su - newbuild -c "rpmbuild --nobuild -vv ~/rpmbuild/SPECS/package.SPEC"
stage: build
script:
# Install our required packages
- yum -y install openssl-devel freetype-devel fontconfig-devel libicu-devel sqlite-devel libpng-devel libjpeg-devel ruby
# Initialize the submodules to build
- git submodule update --init
# build the RPM
- su - newbuild -c "rpmbuild -ba --target=`uname -m` -vv ~/rpmbuild/SPECS/package.SPEC"
stage: deploy
script:
# move the RPM/SRPM
- mkdir -pv $BUILD_DIR/$RELEASEVER/{SRPMS,x86_64}
- 'for f in $WORK_DIR/rpmbuild/RPMS/x86_64/*; do cp -v "$f" $BUILD_DIR/$RELEASEVER/x86_64; done'
- 'for f in $WORK_DIR/rpmbuild/SRPMS/*; do cp -v "$f" $BUILD_DIR/$RELEASEVER/SRPMS; done'
# create the repo
- createrepo -dvp $BUILD_DIR/$RELEASEVER
# update latest
- 'if [ $CI_BUILD_REF_NAME == "master" ]; then rm $PROJECT_DIR/latest; ln -sv $(basename $BUILD_DIR) $PROJECT_DIR/latest; fi'
- 'if [ $CI_BUILD_REF_NAME == "devel" ]; then rm $PROJECT_DIR/latest-dev; ln -sv $(basename $BUILD_DIR) $PROJECT_DIR/latest-dev; fi'
tags:
- repos
I've not found any questions or online documentation to properly explain this to me so any help is appreciated!
You have all stages in one job which does not work. You need to split it up into individual jobs for the three different stages.
Quote from the documentation:
First all jobs of build are executed in parallel.
If all jobs of build succeeds, the test jobs are executed in parallel.
If all jobs of test succeeds, the deploy jobs are executed in parallel.
If all jobs of deploy succeeds, the commit is marked as success.
If any of the previous jobs fails, the commit is marked as failed and no jobs of further stage are executed.
Something like this should work:
stages:
- test
- build
- deploy
do_things_on_stage_test:
script:
- do things
stage: test
do_things_on_stage_build:
script:
- do things
stage: build
do_things_on_stage_deploy:
script:
- do things
stage: deploy
I think you assume that the stages are build on top of each other, which is not the case. If one of your stages needs something like pre-installed packages, you have to add a before_script directive. Think of the stages as in: test-if-build-succeeds, test-if-depoy-succeeds, etc.
If a GitLab project is configured on GitLab CI, is there a way to run the build locally?
I don't want to turn my laptop into a build "runner", I just want to take advantage of Docker and .gitlab-ci.yml to run tests locally (i.e. it's all pre-configured). Another advantage of that is that I'm sure that I'm using the same environment locally and on CI.
Here is an example of how to run Travis builds locally using Docker, I'm looking for something similar with GitLab.
Since a few months ago this is possible using gitlab-runner:
gitlab-runner exec docker my-job-name
Note that you need both docker and gitlab-runner installed on your computer to get this working.
You also need the image key defined in your .gitlab-ci.yml file. Otherwise won't work.
Here's the line I currently use for testing locally using gitlab-runner:
gitlab-runner exec docker test --docker-volumes "/home/elboletaire/.ssh/id_rsa:/root/.ssh/id_rsa:ro"
Note: You can avoid adding a --docker-volumes with your key setting it by default in /etc/gitlab-runner/config.toml. See the official documentation for more details. Also, use gitlab-runner exec docker --help to see all docker-based runner options (like variables, volumes, networks, etc.).
Due to the confusion in the comments, I paste here the gitlab-runner --help result, so you can see that gitlab-runner can make builds locally:
gitlab-runner --help
NAME:
gitlab-runner - a GitLab Runner
USAGE:
gitlab-runner [global options] command [command options] [arguments...]
VERSION:
1.1.0~beta.135.g24365ee (24365ee)
AUTHOR(S):
Kamil Trzciński <ayufan#ayufan.eu>
COMMANDS:
exec execute a build locally
[...]
GLOBAL OPTIONS:
--debug debug mode [$DEBUG]
[...]
As you can see, the exec command is to execute a build locally.
Even though there was an issue to deprecate the current gitlab-runner exec behavior, it ended up being reconsidered and a new version with greater features will replace the current exec functionality.
Note that this process is to use your own machine to run the tests using docker containers. This is not to define custom runners. To do so, just go to your repo's CI/CD settings and read the documentation there. If you wanna ensure your runner is executed instead of one from gitlab.com, add a custom and unique tag to your runner, ensure it only runs tagged jobs and tag all the jobs you want your runner to be responsible of.
I use this docker-based approach:
Edit: 2022-10
docker run --entrypoint bash --rm -w $PWD -v $PWD:$PWD -v /var/run/docker.sock:/var/run/docker.sock gitlab/gitlab-runner:latest -c 'git config --global --add safe.directory "*";gitlab-runner exec docker test'
For all git versions > 2.35.2. You must add safe.directory within the container to avoid fatal: detected dubious ownership in repository at.... This also true for patched git versions < 2.35.2. The old command will not work anymore.
Details
0. Create a git repo to test this answer
mkdir my-git-project
cd my-git-project
git init
git commit --allow-empty -m"Initialize repo to showcase gitlab-runner locally."
1. Go to your git directory
cd my-git-project
2. Create a .gitlab-ci.yml
Example .gitlab-ci.yml
image: alpine
test:
script:
- echo "Hello Gitlab-Runner"
3. Create a docker container with your project dir mounted
docker run -d \
--name gitlab-runner \
--restart always \
-v $PWD:$PWD \
-v /var/run/docker.sock:/var/run/docker.sock \
gitlab/gitlab-runner:latest
(-d) run container in background and print container ID
(--restart always) or not?
(-v $PWD:$PWD) Mount current directory into the current directory of the container - Note: On Windows you could bind your dir to a fixed location, e.g. -v ${PWD}:/opt/myapp. Also $PWD will only work at powershell not at cmd
(-v /var/run/docker.sock:/var/run/docker.sock) This gives the container access to the docker socket of the host so it can start "sibling containers" (e.g. Alpine).
(gitlab/gitlab-runner:latest) Just the latest available image from dockerhub.
4. Execute with
Avoid fatal: detected dubious ownership in repository at... More info
docker exec -it -w $PWD gitlab-runner git config --global --add safe.directory "*"
Actual execution
docker exec -it -w $PWD gitlab-runner gitlab-runner exec docker test
# ^ ^ ^ ^ ^ ^
# | | | | | |
# (a) (b) (c) (d) (e) (f)
(a) Working dir within the container. Note: On Windows you could use a fixed location, e.g. /opt/myapp.
(b) Name of the docker container
(c) Execute the command "gitlab-runner" within the docker container
(d)(e)(f) run gitlab-runner with "docker executer" and run a job named "test"
5. Prints
...
Executing "step_script" stage of the job script
$ echo "Hello Gitlab-Runner"
Hello Gitlab-Runner
Job succeeded
...
Note: The runner will only work on the commited state of your code base. Uncommited changes will be ignored. Exception: The .gitlab-ci.yml itself does not have be commited to be taken into account.
Note: There are some limitations running locally. Have a look at limitations of gitlab runner locally.
I'm currently working on making a gitlab runner that works locally.
Still in the early phases, but eventually it will become very relevant.
It doesn't seem like gitlab want/have time to make this, so here you go.
https://github.com/firecow/gitlab-runner-local
If you are running Gitlab using the docker image there: https://hub.docker.com/r/gitlab/gitlab-ce, it's possible to run pipelines by exposing the local docker.sock with a volume option: -v /var/run/docker.sock:/var/run/docker.sock. Adding this option to the Gitlab container will allow your workers to access to the docker instance on the host.
The GitLab runner appears to not work on Windows yet and there is an open issue to resolve this.
So, in the meantime I am moving my script code out to a bash script, which I can easily map to a docker container running locally and execute.
In this case I want to build a docker container in my job, so I create a script 'build':
#!/bin/bash
docker build --pull -t myimage:myversion .
in my .gitlab-ci.yaml I execute the script:
image: docker:latest
services:
- docker:dind
before_script:
- apk add bash
build:
stage: build
script:
- chmod 755 build
- build
To run the script locally using powershell I can start the required image and map the volume with the source files:
$containerId = docker run --privileged -d -v ${PWD}:/src docker:dind
install bash if not present:
docker exec $containerId apk add bash
Set permissions on the bash script:
docker exec -it $containerId chmod 755 /src/build
Execute the script:
docker exec -it --workdir /src $containerId bash -c 'build'
Then stop the container:
docker stop $containerId
And finally clean up the container:
docker container rm $containerId
Another approach is to have a local build tool that is installed on your pc and your server at the same time.
So basically, your .gitlab-ci.yml will basically call your preferred build tool.
Here an example .gitlab-ci.yml that i use with nuke.build:
stages:
- build
- test
- pack
variables:
TERM: "xterm" # Use Unix ASCII color codes on Nuke
before_script:
- CHCP 65001 # Set correct code page to avoid charset issues
.job_template: &job_definition
except:
- tags
build:
<<: *job_definition
stage: build
script:
- "./build.ps1"
test:
<<: *job_definition
stage: test
script:
- "./build.ps1 test"
variables:
GIT_CHECKOUT: "false"
pack:
<<: *job_definition
stage: pack
script:
- "./build.ps1 pack"
variables:
GIT_CHECKOUT: "false"
only:
- master
artifacts:
paths:
- output/
And in nuke.build i've defined 3 targets named like the 3 stages (build, test, pack)
In this way you have a reproducible setup (all other things are configured with your build tool) and you can test directly the different targets of your build tool.
(i can call .\build.ps1 , .\build.ps1 test and .\build.ps1 pack when i want)
I am on Windows using VSCode with WSL
I didn't want to register my work PC as a runner so instead I'm running my yaml stages locally to test them out before I upload them
$ sudo apt-get install gitlab-runner
$ gitlab-runner exec shell build
yaml
image: node:10.19.0 # https://hub.docker.com/_/node/
# image: node:latest
cache:
# untracked: true
key: project-name
# key: ${CI_COMMIT_REF_SLUG} # per branch
# key:
# files:
# - package-lock.json # only update cache when this file changes (not working) #jkr
paths:
- .npm/
- node_modules
- build
stages:
- prepare # prepares builds, makes build needed for testing
- test # uses test:build specifically #jkr
- build
- deploy
# before_install:
before_script:
- npm ci --cache .npm --prefer-offline
prepare:
stage: prepare
needs: []
script:
- npm install
test:
stage: test
needs: [prepare]
except:
- schedules
tags:
- linux
script:
- npm run build:dev
- npm run test:cicd-deps
- npm run test:cicd # runs puppeteer tests #jkr
artifacts:
reports:
junit: junit.xml
paths:
- coverage/
build-staging:
stage: build
needs: [prepare]
only:
- schedules
before_script:
- apt-get update && apt-get install -y zip
script:
- npm run build:stage
- zip -r build.zip build
# cache:
# paths:
# - build
# <<: *global_cache
# policy: push
artifacts:
paths:
- build.zip
deploy-dev:
stage: deploy
needs: [build-staging]
tags: [linux]
only:
- schedules
# # - branches#gitlab-org/gitlab
before_script:
- apt-get update && apt-get install -y lftp
script:
# temporarily using 'verify-certificate no'
# for more on verify-certificate #jkr: https://www.versatilewebsolutions.com/blog/2014/04/lftp-ftps-and-certificate-verification.html
# variables do not work with 'single quotes' unless they are "'surrounded by doubles'"
- lftp -e "set ssl:verify-certificate no; open mediajackagency.com; user $LFTP_USERNAME $LFTP_PASSWORD; mirror --reverse --verbose build/ /var/www/domains/dev/clients/client/project/build/; bye"
# environment:
# name: staging
# url: http://dev.mediajackagency.com/clients/client/build
# # url: https://stg2.client.co
when: manual
allow_failure: true
build-production:
stage: build
needs: [prepare]
only:
- schedules
before_script:
- apt-get update && apt-get install -y zip
script:
- npm run build
- zip -r build.zip build
# cache:
# paths:
# - build
# <<: *global_cache
# policy: push
artifacts:
paths:
- build.zip
deploy-client:
stage: deploy
needs: [build-production]
tags: [linux]
only:
- schedules
# - master
before_script:
- apt-get update && apt-get install -y lftp
script:
- sh deploy-prod
environment:
name: production
url: http://www.client.co
when: manual
allow_failure: true
The idea is to keep check commands outside of .gitlab-ci.yml. I use Makefile to run something like make check and my .gitlab-ci.yml runs the same make commands that I use locally to check various things before committing.
This way you'll have one place with all/most of your commands (Makefile) and .gitlab-ci.yml will have only CI-related stuff.
I have written a tool to run all GitLab-CI job locally without have to commit or push, simply with the command ci-toolbox my_job_name.
The URL of the project : https://gitlab.com/mbedsys/citbx4gitlab
Years ago I build this simple solution with Makefile and docker-compose to run the gitlab runner in docker, you can use it to execute jobs locally as well and should work on all systems where docker works:
https://gitlab.com/1oglop1/gitlab-runner-docker
There are few things to change in the docker-compose.override.yaml
version: "3"
services:
runner:
working_dir: <your project dir>
environment:
- REGISTRATION_TOKEN=<token if you want to register>
volumes:
- "<your project dir>:<your project dir>"
Then inside your project you can execute it the same way as mentioned in other answers:
docker exec -it -w $PWD runner gitlab-runner exec <commands>..
I recommend using gitlab-ci-local
https://github.com/firecow/gitlab-ci-local
It's able to run specific jobs as well.
It's a very cool project and I have used it to run simple pipelines on my laptop.
This is my command
bundle exec rake resque:work QUEUE="*" --trace
I want to run this command on my server as a background process.
please help me.
A method I often use is:
nohup bundle exec rake resque:work QUEUE="*" --trace > rake.out 2>&1 &
This will keep the task running even if you exit your shell. Then if I want to just observe trace output live, I do:
tail -f rake.out
And you can examine rake.out at any time.
If you need to kill it before completion, you can find it with ps and kill the pid.
Just in case somebody finds this 4 years later, bundle has an elegant way of doing this now. For example if you want to run sidekiq in the background you can do:
bundle exec sidekiq -e production -d -L ./log/sidekiq.log
The -d daemonizes to run in the background, but you will also need to use the -L to provide a logfile, else bundler will refuse to run your command in the background (deamonize).
Tested with bundler version 1.15.4
Update Oct 2019.
While the command still works in general, the specific command above will no longer work for sidekiq 6.0+, you'll need to use Upstart or Systemd if you use Linux: https://github.com/mperham/sidekiq/wiki/Deployment#running-your-own-process
I run a rake task every night via cron (as root), when it runs it gives the error:
rake aborted!
no such file to load -- bundler/setup
which I get in an email
When I run it manually (as root), it runs just fine.
I am running rvm if that helps.
I am not really sure what to add to help, but here are a few things.
# ruby -v
ruby 1.9.2p180 (2011-02-18 revision 30909) [i686-linux]
# rails -v
Rails 3.0.9
# gem -v
1.8.5
try to set up your cron task like this
* * * * * /bin/bash -l -c 'rake blah:blah'
I'm trying to set up a cron job with the following command:
crontab -l
Begin Whenever generated tasks for: myapp
* * * * * /bin/bash -l -c 'cd /Users/boris/projects/myapp && script/rails runner "Resque.enqueue(MyModel)"'
I get the following error; in which I see its loading Ruby 1.8. The problem is I'm using RVM with ruby 1.9.2. How do I specify the correct RVM path in CRON?
Subject: Cron <boris#jz> /bin/bash -l -c cd /Users/boris/projects/myapp && script/rails runner "Resque.enqueue(Place)"
X-Cron-Env: <SHELL=/bin/sh>
X-Cron-Env: <PATH=/usr/bin:/bin>
X-Cron-Env: <LOGNAME=boris>
X-Cron-Env: <USER=boris>
X-Cron-Env: <HOME=/Users/boris>
Message-Id: <20110523022400.A5B242C608D#jz.local>
Date: Sun, 22 May 2011 19:24:00 -0700 (PDT)
/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/rubygems/custom_require.rb:31:in `gem_original_require': no such file to load -- bundler/setup (LoadError)
from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/rubygems/custom_require.rb:31:in `require'
from /Users/boris/projects/myapp/config/boot.rb:6
from script/rails:5:in `require'
from script/rails:5
How do I specify the correct RVM path in CRON?
Thanks in advance
Ruby path with which ruby:
/Users/boris/.rvm/rubies/ruby-1.9.2-p180/bin/ruby
Please do not use the -l switch in cron jobs. The --login switch instructs bash to run as a login shell. Therefore, it will load your environment, and things might appear to work. However, cron jobs are by nature non-interactive, non-login shells. Invoking them as if they were is just bad practice. Also, when bash starts a login shell, it first loads the system environment (/etc/profile), and if in that file something needs to print to the screen (like motd), your cron job will report nasty errors like this:
stty: TIOCGETD: Inappropriate ioctl for device
You don't need to write a cron runner neither (following that logic, you might as well write a cron runner runner). Please keep things simple. All you need to do is configure your cron job to launch a bash shell, and make that bash shell load your environment.
The shebang line in your script should not refer directly to a ruby executable, but to rvm's ruby:
#!/usr/bin/env ruby
This instructs the script to load the environment and run ruby as we would on the command line with rvm loaded.
On many UNIX derived systems, crontabs can have a configuration section before the actual lines that define the jobs to be run. If this is the case, you would then specify:
SHELL=/path/to/bash
This will ensure that the cron job will be spawned from bash. Still, your environment is missing, so to instruct bash to load your environment, you will want to add to the configuration section the following:
BASH_ENV=/path/to/environment (typically .bash_profile or .bashrc)
HOME is automatically derived from the /etc/passwd line of the crontab owner, but you can override it.
HOME=/path/to/home
After this, a cron job might look like this:
15 14 1 * * $HOME/rvm_script.rb
What if your crontab doesn't support the configuration section. Well, you will have to give all the environment directives in one line, with the job itself. For example,
15 14 1 * * export BASH_ENV=/path/to/environment && /full/path/to/bash -c '/full/path/to/rvm_script.rb'
Full blog post on the subject
Your problem is that you're executing two commands but not as you expect. The two commands are:
/bin/bash -l -c cd /Users/boris/projects/myapp
script/rails runner "Resque.enqueue(MyModel)"
With the second only executing if the first succeeded. I think you just need some quotes:
* * * * * /bin/bash -l -c 'cd /Users/boris/projects/myapp && script/rails runner "Resque.enqueue(MyModel)"'
Those single quotes will feed your cd ... && script/rails ... pair to /bin/bash as a single command and that should change the current working directory to what you want when script/rails is executed.
Easiest solution is to use this command instead:
Begin Whenever generated tasks for: myapp
* * * * * /bin/bash -l -c 'cd /Users/boris/projects/myapp && ./script/rails runner "Resque.enqueue(MyModel)"'