. gitlab-ci. yml pipeline run only on one branch - gitlab-ci

i have . gitlab-ci. yml file. when i push to stage branch it make stage commands (only stage) but when i merge to main it's still make "only stage" command
what i am missing ??
variables:
DOCKER_REGISTRY: 036470204880.dkr.ecr.us-east-1.amazonaws.com
AWS_DEFAULT_REGION: us-east-1
APP_NAME: apiv6
APP_NAME_STAGE: apiv6-test
DOCKER_HOST: tcp://docker:2375
publish:
image:
name: amazon/aws-cli
entrypoint: [""]
services:
- docker:dind
before_script:
- amazon-linux-extras install docker
- aws --version
- docker --version
script:
- docker build -t $DOCKER_REGISTRY/$APP_NAME:latest .
- aws ecr get-login-password | docker login --username AWS --password-stdin $DOCKER_REGISTRY
- docker push $DOCKER_REGISTRY/$APP_NAME:latest
- aws ecs update-service --cluster apiv6 --service apiv6 --force-new-deployment
only:
- main
publish:
image:
name: amazon/aws-cli
entrypoint: [""]
services:
- docker:dind
before_script:
- amazon-linux-extras install docker
- aws --version
- docker --version
script:
- docker build -t $DOCKER_REGISTRY/$APP_NAME_STAGE:latest .
- aws ecr get-login-password | docker login --username AWS --password-stdin $DOCKER_REGISTRY
- docker push $DOCKER_REGISTRY/$APP_NAME_STAGE:latest
- aws ecs update-service --cluster apiv6-test --service apiv6-test-service --force-new-deployment
only:
- stage

Itamar, I believe this is a YAML limitation. See this GitLab issue as reference.
The problem is that you have two jobs with the same name. But when the YAML file is parsed, you're actually overriding the first job.
Also, from the official GitLab documentation:
Use unique names for your jobs. If multiple jobs have the same name, only one is added to the pipeline, and it’s difficult to predict which one is chosen
Please, try renaming one of your jobs and test it again.

Related

ECR login fails in gitlab runner

I'm trying to deploy ECS with task definition and I'm using ECR to store my docker image in was. When I try to login ECR in GitLab CI/CD with shared runner. I'm getting errors.
image: docker:19.03.10
services:
- docker:dind
variables:
REPOSITORY_URL: <REPOSITORY_URL>
TASK_DEFINITION_NAME: <Task_Definition>
CLUSTER_NAME: <CLUSTER_NAME>
SERVICE_NAME: <SERVICE_NAME>
before_script:
- apk add --no-cache curl jq python py-pip
- pip install awscli
- aws configure set aws_access_key_id $AWS_ACCESS_KEY_ID
- aws configure set aws_secret_access_key $AWS_SECRET_ACCESS_KEY
- aws configure set region $AWS_DEFAULT_REGION
- $(aws ecr get-login --no-include-email --region "${AWS_DEFAULT_REGION}")
- IMAGE_TAG="$(echo $CI_COMMIT_SHA | head -c 8)"
stages:
- build
- deploy
build:
stage: build
script:
- echo "Building image..."
- docker build -t $REPOSITORY_URL:latest .
- echo "Tagging image..."
- docker tag $REPOSITORY_URL:latest $REPOSITORY_URL:$IMAGE_TAG
- echo "Pushing image..."
- docker push $REPOSITORY_URL:latest
- docker push $REPOSITORY_URL:$IMAGE_TAG
Error details:
There are two approaches that you can take to access a private registry. Both require setting the CI/CD variable DOCKER_AUTH_CONFIG with appropriate authentication information.
Per-job: To configure one job to access a private registry, add DOCKER_AUTH_CONFIG as a CI/CD variable.
Per-runner: To configure a runner so all its jobs can access a private registry, add DOCKER_AUTH_CONFIG as an environment variable in the runner’s configuration.
https://docs.gitlab.com/ee/ci/docker/using_docker_images.html#access-an-image-from-a-private-container-registry
I see the following issues in your config:
docker login is missing
without DOCKER_HOST docker:dind will not work
Please try to follow this tutorial - link, youtube video about the mentioned tutorial is here.

Bitbucket Pippelines EKS Container image Change

It is required to deploy the ECR Image to EKS via Bitbucket pipelines.
So I have created the step below. But I am not sure about the correct command for the KUBECTL_COMMAND to change (set) the deployment image with the new one in a namespace in the EKS cluster:
- step:
name: 'Deployment to Production'
deployment: Production
trigger: 'manual'
script:
- pipe: atlassian/aws-eks-kubectl-run:2.2.0
variables:
AWS_ACCESS_KEY_ID: $AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY: $AWS_SECRET_ACCESS_KEY
AWS_DEFAULT_REGION: $AWS_DEFAULT_REGION
CLUSTER_NAME: 'xxx-zaferu-dev'
KUBECTL_COMMAND: 'set image deployment.apps/xxx-dev xxx-application=123456789.dkr.ecr.eu-west-1.amazonaws.com/ci-cd-test:latest'
- echo " Deployment has been finished successfully..."
So I am looking for the correct way for this step!
If this is not the best way for the CI/CD deployment, I am planning to use basic command to change the conatiner image :
image: python:3.8
pipelines:
default:
- step:
name: Update EKS deployment
script:
- aws eks update-kubeconfig --name <cluster-name>
- kubectl set image deployment/<deployment-name> <container-name>=<new-image>:<tag> -n <namespace>
- aws eks describe-cluster --name <cluster-name>
I tried to use:
KUBECTL_COMMAND: 'set image deployment.apps/xxx-dev xxx-application=123456789.dkr.ecr.eu-west-1.amazonaws.com/ci-cd-test:latest'
but it gives an error :
INFO: Successfully updated the kube config.
Error from server (NotFound): deployments.apps "xxx-app" not found
sorry I got my bug, a missing namespace :)
- kubectl set image deployment/<deployment-name> <container-name>=<new-image>:<tag> -n <namespace>
I forgot to add -n and then I realized.

Docker Tag Error 25 on gitlab-ci.yml trying to start GitLab Pipeline

I'm going through the "Scalable FastAPI Application on AWS" course. My gitlab-ci.yml file is below.
stages:
- docker
variables:
DOCKER_DRIVER: overlay2
DOCKER_TLS_CERTDIR: "/certs"
cache:
key: ${CI_JOB_NAME}
paths:
- ${CI_PROJECT_DIR}/services/talk_booking/.venv/
build-python-ci-image:
image: docker:19.03.0
services:
- docker:19.03.0-dind
stage: docker
before_script:
- cd ci_cd/python/
script:
- docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY
- docker build -t registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim .
- docker push registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim
My Pipeline fails with this error:
See https://docs.docker.com/engine/reference/commandline/login/#credentials-store
Login Succeeded
$ docker build -t registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim .
invalid argument "registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim" for "-t, --tag" flag: invalid reference format
See 'docker build --help'.
Cleaning up project directory and file based variables
ERROR: Job failed: exit code 125
It may or may not be relevant but the Container Registry for the GitLab project says there's a Docker connection error.
Thanks
I created a new GitLab account with a new username and things are working now. The underscore does appear to have been the issue.

How to set .gitlab-ci.yml to only run the tasks on one node and only update or push the repo to other nodes (docker-swarm)?

This is my .gitlab-ci.yml file in my repo:
image: docker
#services:
# - docker:dind
stages:
- build
- deploy
build-prod:
stage: build
only:
- master
tags:
- docker
script:
- docker network create -d overlay reprox
environment: master
deploy-prod:
stage: deploy
only:
- master
tags:
- docker
script:
- docker stack deploy -c ./site1/docker-compose.yml site1
- docker stack deploy -c ./site2/docker-compose.yml site2
- docker stack deploy -c ./site3/docker-compose.yml site3
- docker stack deploy -c ./reverse-proxy/docker-compose.yml proxy
environment: master
So my setup is 1 manager and 2 worker nodes and I only need to run build and deploy jobs on manager node, other nodes just need to have the repo, no need for running the bash commands on worker nodes.
I added a manager runner with "docker" tag and worker nodes with "runner" tag.
Remove your docker tag. You can configure your workers to work only on specific tags
job1:
tags:
- dockernode_1
job2:
tags:
- dockernode_2
Your previously used docker tag was probably just a workaround (or from a tutorial) to make the runners work on all jobs. If you don't want a runner to care about tagging, you can make him pick up all available jobs.

Gitlab-runner has 'access denied' when pushing built image

I have a problem with pushing a built image with gitlab-runner to a gitlab repository.
My gitlab-ci.yml:
image: docker:latest
services:
- docker:dind
stages:
- build
- release
variables:
TEST_IMAGE: registry.gitlab.com/myhost/haproxy:$CI_COMMIT_REF_NAME
RELEASE_IMAGE: registry.gitlab.com/myhost/haproxy:latest
before_script:
- docker login -u gitlab-ci-token -p $CI_JOB_TOKEN gitlab.com
build:
stage: build
script:
- docker build --pull -t $TEST_IMAGE .
- docker push $TEST_IMAGE
release:
stage: release
script:
- docker pull $TEST_IMAGE
- docker tag $TEST_IMAGE $RELEASE_IMAGE
- docker push $RELEASE_IMAGE
only:
- master
The docker login works - I got "Login success" - but when it comes to the push operation I get:
$ docker push $TEST_IMAGE
The push refers to repository [registry.gitlab.com/myhost/haproxy]
d77ab2f42dd4: Preparing
c70258f465dd: Preparing
96b45c1aa07c: Preparing
28587e66f3e8: Preparing
21b59fc0e3a3: Preparing
9c46f426bcb7: Preparing
9c46f426bcb7: Waiting
denied: access forbidden
ERROR: Job failed: exit code 1
The runner is on my own server, and I'm pushing to gitlab.com
I have also checked on my local machine, executing in terminal commands like in the script - login, build and push - and everything works, but if I run locally with the runner, register it and get the job, I also get an access forbidden error.
So I think the problem is in runner, bo what.
I compared the behaviour on a few versions of gitlab-runner from 10.6 to newest 11.0
Any ideas?
So the problem was wrong registry address - it should be registry.gitlab.com
It misled me that it shows "Loged in" in terminal even without "registry" prefix, so the best solution is using build in variables during login in gitlab-ci.yml:
docker login -u gitlab-ci-token -p $CI_JOB_TOKEN $CI_REGISTRY