Gradle and Mysql in gitlab CI - gitlab-ci

I have a test job which requires gradle to run the test as well as mysql. This example
shows how to use mysql. The problem is the image tag in the link overrides my global image tag of gradle because of which gradle is not found. Is there a way to use multiple images in one job or any other work-around.
This is a shortened version of .gitlab-ci.yml. This is the full one:
image: gradle:jdk11
# Disable the Gradle daemon for Continuous Integration servers as correctness
# is usually a priority over speed in CI environments. Using a fresh
# runtime for each build is more reliable since the runtime is completely
# isolated from any previous builds.
variables:
# Configure mysql service (https://hub.docker.com/_/mysql/)
MYSQL_ROOT_PASSWORD: mysql
GRADLE_OPTS: "-Dorg.gradle.daemon=false"
## build configs....
test_MariaDBImpl:
needs:
- build_MariaDBImpl
stage: test
services:
- mysql
image: mysql
script:
- echo "create user if not exists 'test'#'localhost'; grant all privileges on *.* to 'test'#'localhost'; flush privileges;" | mysql --user=root --password="$MYSQL_ROOT_PASSWORD" --host=mysql
- gradle cleanTest :MariaDBImpl:test
Edit 1: One work around that I thought of was to add a before_script and download either gradle or mysql in it but I guess there is a better way?
Edit 2: Or maybe create a docker image with all three and use that one instead?

That's the expected behaviour with GitLab-CI. You set image: gradle:jdk11 as default image to use for the CI job unless overridden in the job configuration like you did with image: mysql.
As you said you have several options here if you need both gradle and mysql-client.
Build your own Docker image that contains both
Remove image: mysql, use the default image: gradle:jdk11 and install mysql-client at the beginning of your script block or with before_script. gradle:jdk11 is based on Ubuntu so you can run: apt-get update && apt-get install -y mysql-client
Keep image: mysql and install gradle with apt-get update && apt-get install -y gradle (I don't recommend that, installing gradle via APT install a lot of packages and will slow your CI job down unnecessarily)
Keep image: mysql, add gradle wrapper to your repository and directly call it from gitlab-ci. Edit: This is not very useful as it would also require to download the JDK to build the project. It works best if the base image already contains it.
My preference would be to keep on using gradle:jdk11 as base image and just install the mysql-client package for that specific job since your other jobs only need gradle.

Related

running the singularity image and finding the results

I am trying to make the image for the Singularity using the following file(which is Singularity.recepie):
Bootstrap: docker
From: ubuntu:18.04
%post
apt-get -y update
apt-get install -y python3-pip
%files
QC.py /
%runscript
python3 /QC.py
I build the image using the following command:
sudo singularity build ubuntu.simg Singularity.recepie
and run the image using this command:
./ubuntu.simg
when I run the image it works but I do not see the result files in the directory that I am working at. do you know how I can the results and copy them to my current directory?
By default, your current working directory is mounted into the image and used as the working directory. If no files are being created and they should be, then you need to look at where your script is trying to write to. Also check the singularity error/warnings output for additional info.
If you're using an older version of Singularity (implied by .simg instead of .sif extension), you may also be running on a kernel that does not support overlayfs, which can cause issues attempting to mount to a non-existing folder. If this is the case, singularity will give a warning but continue on and possibly fail in a hard to detect manner. More info on mounting with overlay here.

Installation gitlab-runner on windows with chocolatey

I'm trying to deploy gitlab-runner on windows using chocolatey.
My intention is to install gitlab-runner in E:\gitlab-runner directory, automatically register the runner and start it as a service
I executed the following line :
choco install -y gitlab-runner /InstallDir E:\gitlab-runner /Service --source https://mynexusproxy/repository/chocolatey-org/
and I got the following output :
Chocolatey v0.10.11
Installing the following packages:
gitlab-runner;/InstallDir;E:\gitlab-runner;/Service
By installing you accept licenses for the packages.
Progress: Downloading gitlab-runner 11.8.0... 100%
gitlab-runner v11.8.0
gitlab-runner package files install completed. Performing other installation steps.
Using previous gitlab-runner install path: e:\gitlab-runner
Installing x64 bit version
Added C:\ProgramData\chocolatey\bin\gitlab-runner.exe shim pointed to 'e:\gitlab-runner\gitlab-runner.exe'.
The install of gitlab-runner was successful.
Software install location not explicitly set, could be in package or
default install location if installer.
Second path fragment must not be a drive or UNC name.
Parameter name: path2
The gitlab-runner.exe is correctly downloaded in E:\gitlab-runner but the register is not done and no service is created. A register_example.ps1 is also download in E:\gitlab-runner.
What's wrong with my installation procedure ? Do I need to modify the register_example.ps1 with custom values ?
The command provided by #Gary Ewan Park is good but it's not sufficient to fully manage the gitlab-runner installation on Windows.
Installation procedure
Install the gitlab-runner with chocolatey
choco install -y gitlab-runner --params="'/InstallDir=E:\gitlab-runner /Service'" --source https://mynexusproxy/repository/chocolatey-org/
it will download an exe in the E:\gitlab-runner directory, create a service gitlab-runner and create a register_example.ps1 in the same directory.
Modify the register_example.ps1 with your desired value and rename it in register.ps1
Execute register.ps1 to register the runner in your Gitlab server. It will create also a config.toml in the directory where you execute the register.ps1 script.
Per default, the gitlab-runner service executable path is
E:\gitlab-runner\gitlab-runner.exe run --working-directory C:\Windows\system32 --config C:\Windows\system32\config.toml --service gitlab-runner --syslog
which was not good for me. To change the executable path in command line, see this thread.
After that, the service is up and running and the runner is correctly registered in the Giltab Server.
Upgrade procedure
stop the gitlab-runner service (otherwise chocolatey crash because it cannot create a file when that file already exists)
execute the upgrade with chocolatey
choco upgrade -y gitlab-runner --source https://mynexusproxy/repository/chocolatey-org/
start the gitlab-runner service
It is unfortunate that this particular package does not provide an example in the description to show how to correctly pass the package parameters. However, you can find more information, and examples of how to do this here:
https://chocolatey.org/docs/how-to-parse-package-parameters-argument
I believe you are going to want a command similar to the following:
choco install -y gitlab-runner --params="'/InstallDir=E:\gitlab-runner /Service'" --source https://mynexusproxy/repository/chocolatey-org/
Otherwise, the parameters that you are trying to pass in will simply be ignored.

how to ask ambari agent not keep quiet when install packages in ubuntu

I'm installing new service in ambari managed cluster. my connection to hortonwork's repo seems too bad, it took lots of time to install ambari-server and ambari-agent it self (onece i almost thought it was dead).
the problem is that when i'm installing new service (e.g zeppelin, it also requires install other dependencies), the progress bar get stuck at 26% percent. it confused me if the installation get stucked or it's just too slow.
the command
ps -ef|grep apt
shows the the install is still there. only can't track the progress via /var/log/apt/term.log like normal apt-get installation (it shows the download progress).
the command shows the apt-get comes with an "-q" options which prevent me from tracking the log file.
so my question is that is there any place i can change the default behavior?
The -q option is added to "apt-get" command based on log level of ambari-agent. I.e. if log level for ambari-agent is set to DEBUG then apt-get without -q option is executed.
You can edit /etc/ambari-agent/conf/logging.conf, in [logger_root] section comment default line level=WARNING and add level=DEBUG. Then retstart ambari-agent. Now if you install any service them apt-get without -q will be executed.
Note:
If /etc/ambari-agent/conf/logging.conf does not exist then copy it from /etc/ambari-agent/conf/logging.conf.sample
Ambari agent specific changed need to be done on all cluster node

Create Docker image from existing Ubuntu + App

I installed Moodle (eLearning PHP based app, but it could be any app) locally on Ubuntu and would like to package it as Docker image/container. There were whole bunch of installations and configurations done. I'd like to package all that so that I can deploy to some Docker enabled hosting service, such as Digital Ocean or AWS.
How do I create Docker image?
Do I need to handle networking, ports and Apache configuration for production deployment?
There ara a lot of Moodle images in dockerhub. just use one of them
The process to create docker images is well documented on Docker's documentation site. See: Build your own images
The idea is simple: You inherit/extend an existing image and make additions to it. This is done in a provisioning file called Dockerfile
Dockerfile Example:
FROM debian:8.4
MAINTAINER John Doe (j.doe#example.com)
# update aptitude
RUN apt-get clean && apt-get update
# utilities
RUN apt-get -y install vim git php5.6 apache2
In the example above I extend a Debian image, update aptitude and install a series of packages.
A full list of commands available in Dockerfiles is available at https://docs.docker.com/engine/reference/builder/
Once your Dockerfile is ready you can build the image using the following command:
docker build -t debian/enhanced:8.4 /path/to/Dockerfile

Debian: How can I pull a single package with dependencies from another repository?

I am on debian etch and I want to pull subversion1.5.1 from testing though it is a production machine. I need to keep the risk minimal.
Any hints?
Just add the testing repository to your sources.list and pin the priority of the testing packages to a very low value:
Add the following to /etc/apt/preferences:
Package: *
Pin: release a=testing
Pin-Priority: 200
200 means that new packages in testing still override local packages that are not in stable (local is always 100), but not ones that are in the stable repo as well.
Read apt_preferences(5) for more information on pinning.
Then, you can install packages from testing by doing
$ apt-get install -t testing $some_package
but they won't be installed by normal upgrade operations or won't be the default when installed with apt.