Can't get tests to pass on Gitlab CI - activejdbc

I've been trying to get our tests to pass on our Gitlab CI, but can't. I'm using the stock pipelines config that comes with Gitlab. All I've had to do is provide the gitlab yaml file to config the CI.
This is what we're using
image: maven:3.5.0-jdk-8-alpine
services:
- postgres:latest
variables:
POSTGRES_DB: my_test_db
POSTGRES_USER: my_test_user
POSTGRES_PASSWORD: ""
MAVEN_OPTS: "-Dmaven.repo.local=.m2/repository -Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jMavenTransferListener=WARN -Dorg.slf4j.simpleLogger.showDateTime=true -Djava.awt.headless=true"
MAVEN_CLI_OPTS: "--batch-mode --errors --fail-at-end --show-version -DinstallAtEnd=true -DdeployAtEnd=true"
ACTIVE_ENV: test
connect:
image: postgres
script:
# official way to provide password to psql: http://www.postgresql.org/docs/9.3/static/libpq-envars.html
- export PGPASSWORD=$POSTGRES_PASSWORD
- psql -h "postgres" -U "$POSTGRES_USER" -d "$POSTGRES_DB" -c "SELECT 'OK' AS status;"
stages:
- test
test:
stage: test
script:
- "mvn -Denvironments=test -B db-migrator:migrate; mvn -Denvironments=test -DACTIVE_ENV=test -B test"
Everything works perfectly up to the point where the tests run. Then they all error out with similar messages:
383 [main] WARN org.javalite.activeweb.DBSpecHelper - no DB connections are configured, none opened
456 [main] WARN org.javalite.activeweb.DBSpecHelper - no DB connections are configured, none opened
Tests run: 2, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 0.528 sec <<< FAILURE! - in app.models.RoleTest
validatePresenceOfUsers(app.models.RoleTest) Time elapsed: 0.071 sec <<< ERROR!
org.javalite.activejdbc.DBException: Failed to retrieve metadata from DB, connection: 'default' is not available
I have one database.properties file that is checked in and is for tests only (our dev and prod envs use jndi). It looks like so:
test.driver=org.postgresql.Driver
test.username=my_test_user
test.password=
test.url=jdbc:postgresql://postgres/edv_test
Again, migrations run using all this exact same config. I just can't figure out why the tests won't run. I understand why it's saying there's no default db, but I don't get why it's not seeing the test settings and configuring that connection as expected.

So you know, the Maven flag environments like this: mvn test -Denvironments=test only works for the DB-Migrator, and not for the tests. Any JavaLite application in a standard running mode or as a test, it will be looking at ACTIVE_ENV. If this is not set, it will assume development. In test mode, it will be looking at database.properties block development.test.xxx=yyy as in http://javalite.io/database_configuration#property-file-configuration.
Think of it as "development" environment, "test" mode.
Additionally, DbConfig is not involved in tests, as database connections in test have a special treatment (rollback transactions), see: http://javalite.io/testing_with_db_connection

Related

Executing reckonTagCreate from gitlab ci with authentication failure

I have setup my CI so that I can manually create a release-tag when all tests succeeds for a new commit on master branch. For this I have created a manual step in the CI config like so:
.release-template:
stage:
releasing
dependencies:
- assemble
script:
- ./gradlew reckonTagPush -Preckon.scope=$scope -Preckon.stage=$stage -Dorg.ajoberstar.grgit.auth.username=$GIT_USER -Dorg.ajoberstar.grgit.auth.password=$GIT_PASSSWORD
only:
- master
when: manual #ONLY MANUAL RELEASES, ONLY FROM MASTER
release-major:
extends: .release-template
variables:
scope: major
stage: final
release-minor:
extends: .release-template
variables:
scope: minor
stage: final
release-patch:
extends: .release-template
variables:
scope: patch
stage: final
This setup fails with an authentication error.
Execution failed for task ':reckonTagPush'.
> org.eclipse.jgit.api.errors.TransportException: https://gitlab-ci-token#gitlab.com/<group>/<project>.git: not authorized
I am running this on gitlab.com on a shared runner.
The username and password are configured in gitlab ci variables for the project. When running this locally inside the same docker image that is used in the gitlab runner, it works fine. So there must be something special about the way the gitlab runner is executing the gradle tasks, or communicating with the gitlab git repo.
Solved the issue with access to pushing to the git repo by adding the following script :
script:
- url_host=`git remote get-url origin | sed -e "s/https:\/\/gitlab-ci-token:.*#//g"`
- git remote set-url origin "https://gitlab-ci-token:$GIT_TOKEN#$url_host"
- ./gradlew reckonTagPush -Preckon.scope=$scope -Preckon.stage=$stage -Dorg.ajoberstar.grgit.auth.username="$GIT_USER" -Dorg.ajoberstar.grgit.auth.password="$GIT_TOKEN"
The notably changes here are setting the git remote url, as well as surrounding the gitlab ci variables with " when passing them to the reckon plugin

Using multiple runners in one gitlab-ci

I want to run CI pipline with 2 jobs:
job will boot up a docker image with docker-runner and run test inside docker
will run under ssh runner and pull code on a remote server.
Is it possible?
Yes, it's possible. You need to:
Register two GitLab Runners with needed executor (docker and shell), each witch different tag (or, at least one of them with a build tag).
Declare a specific tag for given job in your .gitlab-ci.yml, .
Shell runner registration:
[root#jsc00mca ~]# gitlab-runner register
Running in system-mode.
Please enter the gitlab-ci coordinator URL (e.g. https://gitlab.com/):
https://example.com/
Please enter the gitlab-ci token for this runner:
1a2b3c
Please enter the gitlab-ci description for this runner:
[jsc00mca.example.com]: my-shell-runner
Please enter the gitlab-ci tags for this runner (comma separated):
shell
Whether to run untagged builds [true/false]:
[false]:
Whether to lock the Runner to current project [true/false]:
[true]:
Registering runner... succeeded runner=ajgHxcNz
Please enter the executor: virtualbox, docker+machine, kubernetes, docker, shell, ssh, docker-ssh+machine, docker-ssh, parallels:
shell
Runner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded!
Docker runner registration:
[root#jsc00mca ~]# gitlab-runner register
Running in system-mode.
Please enter the gitlab-ci coordinator URL (e.g. https://gitlab.com/):
https://example.com/
Please enter the gitlab-ci token for this runner:
1a2b3c
Please enter the gitlab-ci description for this runner:
[jsc00mca.example.com]: my-docker-runner
Please enter the gitlab-ci tags for this runner (comma separated):
docker
Whether to run untagged builds [true/false]:
[false]:
Whether to lock the Runner to current project [true/false]:
[true]:
Registering runner... succeeded runner=ajgHxcNz
Please enter the executor: virtualbox, docker+machine, kubernetes, docker, shell, ssh, docker-ssh+machine, docker-ssh, parallels:
docker
Please enter the default Docker image (e.g. ruby:2.1):
alpine:latest
Runner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded!
.gitlab-ci.yml
buildWithShell:
stage: build
tags:
- shell
script:
- echo 'Building with the shell executor...'
buildWithDocker:
image: alpine:latest
stage: build
tags:
- docker
script:
- echo 'Building with the docker executor...'
Yes you can trigger different/mixed runners from a single gitlab-ci pipeline.
First you should register a shell runner on the target host and give it a tag (truncated):
$ gitlab-runner register
...
Please enter the gitlab-ci tags for this runner (comma separated):
my_shell_runner
...
Please enter the executor: virtualbox, docker+machine, docker-ssh+machine, docker, docker-ssh, parallels, shell, ssh:
shell
Within your gitlab-ci.yaml something like this should work.
The 'test' job runs your test command in a docker container based on the image NAME_OF_IMAGE.
If that succeeds, the 'deploy' job chooses your shell runner based on the tag 'my_shell_runner' and will execute all commands within the script tag on the runner's host (truncated):
test:
stage: test
services:
- docker:dind
tags:
- docker-executor
script:
- docker run --rm NAME_OF_IMAGE sh -c "TEST_COMMAND_TO_RUN"
deploy:
stage: deploy
tags:
- my_shell_runner
script:
- COMMAND_TO_RUN
- COMMAND_TO_RUN
- COMMAND_TO_RUN

Using services: mysql for codecption test in gitlab-ci fails with "Connection refused"

I have a CakePHP Application with codeception-plugin for testing.'
Locally I run it in a ddev docker environment and everything works fine.
Trying to run automated tests with gitlab-ci gives me following error:
Running with gitlab-runner 11.1.0 (081978aa)
on shared runner 601c0f11
Using Docker executor with image kevinliteon/cakephp:php7 ...
Starting service mysql:latest ...
Pulling docker image mysql:latest ...
Using docker image sha256:6a834f03bd02bb88cdbe0e289b9cd6056f1d42fa94792c524b4fddc474dab628 for mysql:latest ...
Waiting for services to be up and running...
*** WARNING: Service runner-601c0f11-project-94-concurrent-0-mysql-0 probably didn't start properly.
Health check error:
service "runner-601c0f11-project-94-concurrent-0-mysql-0-wait-for-service" timeout
Health check container logs:
Service container logs:
2018-10-04T12:12:18.904025613Z Initializing database
2018-10-04T12:12:18.925096235Z 2018-10-04T12:12:18.919745Z 0 [Warning] [MY-011070] [Server] 'Disabling symbolic links using --skip-symbolic-links (or equivalent) is the default. Consider not using this option as it' is deprecated and will be removed in a future release.
2018-10-04T12:12:18.925195518Z 2018-10-04T12:12:18.919970Z 0 [System] [MY-013169] [Server] /usr/sbin/mysqld (mysqld 8.0.12) initializing of server in progress as process 30
2018-10-04T12:12:50.330736417Z 2018-10-04T12:12:50.330487Z 5 [Warning] [MY-010453] [Server] root#localhost is created with an empty password ! Please consider switching off the --initialize-insecure option.
*********
Pulling docker image kevinliteon/cakephp:php7 ...
Using docker image sha256:bd4a83b02647ad93a356b343d2ce5ae3a9a1177aea2cd76c61b009abc7df8990 for kevinliteon/cakephp:php7 ...
Running on runner-601c0f11-project-94-concurrent-0 via d7f4a5e71b47...
Fetching changes...
Removing vendor/
HEAD is now at 92cb022 test
Checking out 92cb0223 as deployment...
Skipping Git submodules setup
Checking cache for default...
Successfully extracted cache
$ vendor/bin/codecept run Unit
Codeception PHP Testing Framework v2.3.9
Powered by PHPUnit 6.5.13 by Sebastian Bergmann and contributors.
In Db.php line 308:
Db: SQLSTATE[HY000] [2002] Connection refused while creating PDO connection
My gitlab-ci.yml (partly):
services:
- mysql:latest
variables:
MYSQL_ROOT_PASSWORD: mysql123456789
MYSQL_DATABASE: test_db
MYSQL_USER: db
MYSQL_PASSWORD: db
build:
...
codecept:Unit:
stage: test
script:
- vendor/bin/codecept run Unit
In my codeception.yml I configured the Db module:
modules:
config:
Db:
dsn: 'mysql:host=mysql;dbname=test_db'
user: 'db'
password: 'db'
cleanup: true # reload dump between tests
populate: true # load dump before all tests
reconnect: true
I also tryed using the root user - without success.
Problem is, that I can not connect to the DB for whatever reasons... Maybe the warnings while initializing the service container have something to do with that, but I could not figure out how to fix them or if this is the problem.
I really tried a lot of things without any success! Basically my code depends on the documentations of gitlab-ci and codeception so it should work.
Anybody implemented this scenario successfully or know what I'm doing wrong?
Thanks for any help!
I want to answer how I solved it:
First thing was, I had to add the env-varibale "db_dsn" like this:
export db_dsn="mysql://user:paswd#host/db"
Then I still got the health-check error. Only way I found to successfully setup was to use another docker image for the db-service. I choose "mariadb:latest" - and then it worked for me.

Gitlab-runner has 'access denied' when pushing built image

I have a problem with pushing a built image with gitlab-runner to a gitlab repository.
My gitlab-ci.yml:
image: docker:latest
services:
- docker:dind
stages:
- build
- release
variables:
TEST_IMAGE: registry.gitlab.com/myhost/haproxy:$CI_COMMIT_REF_NAME
RELEASE_IMAGE: registry.gitlab.com/myhost/haproxy:latest
before_script:
- docker login -u gitlab-ci-token -p $CI_JOB_TOKEN gitlab.com
build:
stage: build
script:
- docker build --pull -t $TEST_IMAGE .
- docker push $TEST_IMAGE
release:
stage: release
script:
- docker pull $TEST_IMAGE
- docker tag $TEST_IMAGE $RELEASE_IMAGE
- docker push $RELEASE_IMAGE
only:
- master
The docker login works - I got "Login success" - but when it comes to the push operation I get:
$ docker push $TEST_IMAGE
The push refers to repository [registry.gitlab.com/myhost/haproxy]
d77ab2f42dd4: Preparing
c70258f465dd: Preparing
96b45c1aa07c: Preparing
28587e66f3e8: Preparing
21b59fc0e3a3: Preparing
9c46f426bcb7: Preparing
9c46f426bcb7: Waiting
denied: access forbidden
ERROR: Job failed: exit code 1
The runner is on my own server, and I'm pushing to gitlab.com
I have also checked on my local machine, executing in terminal commands like in the script - login, build and push - and everything works, but if I run locally with the runner, register it and get the job, I also get an access forbidden error.
So I think the problem is in runner, bo what.
I compared the behaviour on a few versions of gitlab-runner from 10.6 to newest 11.0
Any ideas?
So the problem was wrong registry address - it should be registry.gitlab.com
It misled me that it shows "Loged in" in terminal even without "registry" prefix, so the best solution is using build in variables during login in gitlab-ci.yml:
docker login -u gitlab-ci-token -p $CI_JOB_TOKEN $CI_REGISTRY

How to bind Jenkins build output with tests result?

I'm setting automated protractor tests to run in a docker container with the help of jenkins. But not been able to make a the jenkins build result to reflect the testing outcome (if some test fail, build should fail also).
Important to say that all tests should run, even if the first one fails.
The tests are initiated with docker-compose up --abort-on-container-exit and my docker-compose file looks like:
version: '2'
services:
selenium:
image: selenium/standalone-chrome
ports:
- 4444:4444
volumes:
- /dev/shm:/dev/shm
protractor:
volumes:
- ./reporting:/assets/reporting
image: protractor-test
command: "dockerize -wait http://selenium:4444 -timeout 60m protractor /assets/conf.js"
Looks like your docker-compose command is returning exit code 0 no matter what.
How about using a Jasmine xunit reporter to generate a test report, copy the generated xml test report to outside the container (using docker cp), and then publish it using Jenkins' post-build action?
The job will be marked as failed if the xml is not present, which means there's an error during the test runtime or it will be marked as unstable, if it has failed any of the test asserts.