how to deploy same job on all my runners? - gitlab-ci

Question

I have several VMs running gilab-runner,and I'm using gitlab-ci to deploy microservices into those VMs. Now I want to monitoring those VMs with prometheus and grafana, but i need to setup node-exporter/cadvisor etc. service into those VMs.
My Ideas is using gitlab-ci to define a common job for those VMs.
I have already write the docker-compose.yml and .gitlab-ci.yml.
version: '3.8'
services:
node-exporter:
image: prom/node-exporter:latest
container_name: node-exporter
restart: unless-stopped
volumes:
- /proc:/host/proc:ro
- /sys:/host/sys:ro
- /:/rootfs:ro
command:
- '--path.procfs=/host/proc'
- '--path.rootfs=/rootfs'
- '--path.sysfs=/host/sys'
- '--collector.filesystem.mount-points-exclude=^/(sys|proc|dev|host|etc)($$|/)'
ports:
- "9100:9100"
cadvisor:
image: google/cadvisor
container_name: cadvisor
restart: unless-stopped
volumes:
- /:/rootfs:ro
- /var/run:/var/run:rw
- /sys:/sys:ro
- /var/lib/docker/:/var/lib/docker:ro
- /dev/disk/:/dev/disk:ro
ports:
- "8080:8080"
deploy-workers:
tags:
- worker
stage: deploy-workers
script:
- docker-compose -f docker-compose.worker.yaml pull
- docker-compose -f docker-compose.worker.yaml down
- docker-compose -f docker-compose.worker.yaml up -d
then I register the runner in all my VMs with 'worker' tag.
However, only one worker job is triggered during ci.
I have about 20 VMs to go.
Do anyone have suggestions?

Answer 1

This is probably not a good way to be deploying your services onto virtual machines. You don't want to just launch your GitLab CI job and then hope that it results in what you want. Managing each VM separately is both going to be tedious and error-prone.
You probably want to do is have a method that has a declarative way to define/describe your infrastructure and the state of how that infrastructure should be configured and the applications running on it.
For example, you could:
Use a proper orchestrator, such as docker swarm or Kubernetes AND/OR
Use a provisioning tool, such as Ansible connected to each VM, or if your VMs run in the cloud, Terraform or similar.
In both these examples, you can leverage these tools from a single GitLab CI job and deploy changes to all of your VMs/clusters at once.
Using docker swarm
For example, instead of running your docker-compose on 20 hosts, you can join all 20 VMs to the same docker swarm.
Then in your compose file, you create a deploy key specifying how many replicas you want across the swarm, including numbers per node. Or use mode: global to simply specify you want one container of the service per host in your cluster.
services:
node-exporter:
deploy:
mode: global # deploy exactly one container per node in the swarm
# ...
cadvisor:
deploy:
mode: global # deploy exactly one container per node in the swarm
Then running docker stack deploy from any manager node will do the right thing to all your swarm worker nodes. Docker swarm will also automatically restart your containers if they fail.
See deploy reference.
Using swarm (or any orchestrator) has a lot of other benefits, too, like health checking, rollbacks, etc. that will make your deployment process a lot safer and more maintainable.
If you must use a job per host
Set a unique tag for each runner on each VM. Then use a parallel matrix with a job set to each tag.
job:
parallel:
matrix:
RUNNER: [vm1, vm2, vm3, vm4, vm5] # etc.
tags:
- $RUNNER
See run a matrix of parallel jobs
You want to make sure the tag is unique and covers all your hosts, or you may run the same job on the same host multiple times.
This will let you do what you were seeking to do. However, it's not an advisable practice. As a simple example: there's no guarantee that your docker-compose up will succeed and you may just take down your entire cluster all at once.