I'm trying to deploy ECS with task definition and I'm using ECR to store my docker image in was. When I try to login ECR in GitLab CI/CD with shared runner. I'm getting errors.
image: docker:19.03.10
services:
- docker:dind
variables:
REPOSITORY_URL: <REPOSITORY_URL>
TASK_DEFINITION_NAME: <Task_Definition>
CLUSTER_NAME: <CLUSTER_NAME>
SERVICE_NAME: <SERVICE_NAME>
before_script:
- apk add --no-cache curl jq python py-pip
- pip install awscli
- aws configure set aws_access_key_id $AWS_ACCESS_KEY_ID
- aws configure set aws_secret_access_key $AWS_SECRET_ACCESS_KEY
- aws configure set region $AWS_DEFAULT_REGION
- $(aws ecr get-login --no-include-email --region "${AWS_DEFAULT_REGION}")
- IMAGE_TAG="$(echo $CI_COMMIT_SHA | head -c 8)"
stages:
- build
- deploy
build:
stage: build
script:
- echo "Building image..."
- docker build -t $REPOSITORY_URL:latest .
- echo "Tagging image..."
- docker tag $REPOSITORY_URL:latest $REPOSITORY_URL:$IMAGE_TAG
- echo "Pushing image..."
- docker push $REPOSITORY_URL:latest
- docker push $REPOSITORY_URL:$IMAGE_TAG
Error details:
There are two approaches that you can take to access a private registry. Both require setting the CI/CD variable DOCKER_AUTH_CONFIG with appropriate authentication information.
Per-job: To configure one job to access a private registry, add DOCKER_AUTH_CONFIG as a CI/CD variable.
Per-runner: To configure a runner so all its jobs can access a private registry, add DOCKER_AUTH_CONFIG as an environment variable in the runner’s configuration.
https://docs.gitlab.com/ee/ci/docker/using_docker_images.html#access-an-image-from-a-private-container-registry
I see the following issues in your config:
docker login is missing
without DOCKER_HOST docker:dind will not work
Please try to follow this tutorial - link, youtube video about the mentioned tutorial is here.
Related
It is required to deploy the ECR Image to EKS via Bitbucket pipelines.
So I have created the step below. But I am not sure about the correct command for the KUBECTL_COMMAND to change (set) the deployment image with the new one in a namespace in the EKS cluster:
- step:
name: 'Deployment to Production'
deployment: Production
trigger: 'manual'
script:
- pipe: atlassian/aws-eks-kubectl-run:2.2.0
variables:
AWS_ACCESS_KEY_ID: $AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY: $AWS_SECRET_ACCESS_KEY
AWS_DEFAULT_REGION: $AWS_DEFAULT_REGION
CLUSTER_NAME: 'xxx-zaferu-dev'
KUBECTL_COMMAND: 'set image deployment.apps/xxx-dev xxx-application=123456789.dkr.ecr.eu-west-1.amazonaws.com/ci-cd-test:latest'
- echo " Deployment has been finished successfully..."
So I am looking for the correct way for this step!
If this is not the best way for the CI/CD deployment, I am planning to use basic command to change the conatiner image :
image: python:3.8
pipelines:
default:
- step:
name: Update EKS deployment
script:
- aws eks update-kubeconfig --name <cluster-name>
- kubectl set image deployment/<deployment-name> <container-name>=<new-image>:<tag> -n <namespace>
- aws eks describe-cluster --name <cluster-name>
I tried to use:
KUBECTL_COMMAND: 'set image deployment.apps/xxx-dev xxx-application=123456789.dkr.ecr.eu-west-1.amazonaws.com/ci-cd-test:latest'
but it gives an error :
INFO: Successfully updated the kube config.
Error from server (NotFound): deployments.apps "xxx-app" not found
sorry I got my bug, a missing namespace :)
- kubectl set image deployment/<deployment-name> <container-name>=<new-image>:<tag> -n <namespace>
I forgot to add -n and then I realized.
i have . gitlab-ci. yml file. when i push to stage branch it make stage commands (only stage) but when i merge to main it's still make "only stage" command
what i am missing ??
variables:
DOCKER_REGISTRY: 036470204880.dkr.ecr.us-east-1.amazonaws.com
AWS_DEFAULT_REGION: us-east-1
APP_NAME: apiv6
APP_NAME_STAGE: apiv6-test
DOCKER_HOST: tcp://docker:2375
publish:
image:
name: amazon/aws-cli
entrypoint: [""]
services:
- docker:dind
before_script:
- amazon-linux-extras install docker
- aws --version
- docker --version
script:
- docker build -t $DOCKER_REGISTRY/$APP_NAME:latest .
- aws ecr get-login-password | docker login --username AWS --password-stdin $DOCKER_REGISTRY
- docker push $DOCKER_REGISTRY/$APP_NAME:latest
- aws ecs update-service --cluster apiv6 --service apiv6 --force-new-deployment
only:
- main
publish:
image:
name: amazon/aws-cli
entrypoint: [""]
services:
- docker:dind
before_script:
- amazon-linux-extras install docker
- aws --version
- docker --version
script:
- docker build -t $DOCKER_REGISTRY/$APP_NAME_STAGE:latest .
- aws ecr get-login-password | docker login --username AWS --password-stdin $DOCKER_REGISTRY
- docker push $DOCKER_REGISTRY/$APP_NAME_STAGE:latest
- aws ecs update-service --cluster apiv6-test --service apiv6-test-service --force-new-deployment
only:
- stage
Itamar, I believe this is a YAML limitation. See this GitLab issue as reference.
The problem is that you have two jobs with the same name. But when the YAML file is parsed, you're actually overriding the first job.
Also, from the official GitLab documentation:
Use unique names for your jobs. If multiple jobs have the same name, only one is added to the pipeline, and it’s difficult to predict which one is chosen
Please, try renaming one of your jobs and test it again.
I'm going through the "Scalable FastAPI Application on AWS" course. My gitlab-ci.yml file is below.
stages:
- docker
variables:
DOCKER_DRIVER: overlay2
DOCKER_TLS_CERTDIR: "/certs"
cache:
key: ${CI_JOB_NAME}
paths:
- ${CI_PROJECT_DIR}/services/talk_booking/.venv/
build-python-ci-image:
image: docker:19.03.0
services:
- docker:19.03.0-dind
stage: docker
before_script:
- cd ci_cd/python/
script:
- docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY
- docker build -t registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim .
- docker push registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim
My Pipeline fails with this error:
See https://docs.docker.com/engine/reference/commandline/login/#credentials-store
Login Succeeded
$ docker build -t registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim .
invalid argument "registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim" for "-t, --tag" flag: invalid reference format
See 'docker build --help'.
Cleaning up project directory and file based variables
ERROR: Job failed: exit code 125
It may or may not be relevant but the Container Registry for the GitLab project says there's a Docker connection error.
Thanks
I created a new GitLab account with a new username and things are working now. The underscore does appear to have been the issue.
I have a problem with pushing a built image with gitlab-runner to a gitlab repository.
My gitlab-ci.yml:
image: docker:latest
services:
- docker:dind
stages:
- build
- release
variables:
TEST_IMAGE: registry.gitlab.com/myhost/haproxy:$CI_COMMIT_REF_NAME
RELEASE_IMAGE: registry.gitlab.com/myhost/haproxy:latest
before_script:
- docker login -u gitlab-ci-token -p $CI_JOB_TOKEN gitlab.com
build:
stage: build
script:
- docker build --pull -t $TEST_IMAGE .
- docker push $TEST_IMAGE
release:
stage: release
script:
- docker pull $TEST_IMAGE
- docker tag $TEST_IMAGE $RELEASE_IMAGE
- docker push $RELEASE_IMAGE
only:
- master
The docker login works - I got "Login success" - but when it comes to the push operation I get:
$ docker push $TEST_IMAGE
The push refers to repository [registry.gitlab.com/myhost/haproxy]
d77ab2f42dd4: Preparing
c70258f465dd: Preparing
96b45c1aa07c: Preparing
28587e66f3e8: Preparing
21b59fc0e3a3: Preparing
9c46f426bcb7: Preparing
9c46f426bcb7: Waiting
denied: access forbidden
ERROR: Job failed: exit code 1
The runner is on my own server, and I'm pushing to gitlab.com
I have also checked on my local machine, executing in terminal commands like in the script - login, build and push - and everything works, but if I run locally with the runner, register it and get the job, I also get an access forbidden error.
So I think the problem is in runner, bo what.
I compared the behaviour on a few versions of gitlab-runner from 10.6 to newest 11.0
Any ideas?
So the problem was wrong registry address - it should be registry.gitlab.com
It misled me that it shows "Loged in" in terminal even without "registry" prefix, so the best solution is using build in variables during login in gitlab-ci.yml:
docker login -u gitlab-ci-token -p $CI_JOB_TOKEN $CI_REGISTRY
Gitlab provides a .gitlab-ci.yml template for building and publishing images to its own registry (click "new file" in one of your project, select .gitlab-ci.yml and docker). The file looks like this and it works out of the box :)
# This file is a template, and might need editing before it works on your project.
# Official docker image.
image: docker:latest
services:
- docker:dind
before_script:
- docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY
build-master:
stage: build
script:
- docker build --pull -t "$CI_REGISTRY_IMAGE" .
- docker push "$CI_REGISTRY_IMAGE"
only:
- master
build:
stage: build
script:
- docker build --pull -t "$CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG" .
- docker push "$CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG"
except:
- master
But by default, this will publish to gitlab's registry. How can we publish to docker hub instead?
No need to change that .gitlab-ci.yml at all, we only need to add/replace the environment variables in project's pipeline settings.
1. Find the desired registry url
Using hub.docker.com won't work, you'll get the following error:
Error response from daemon: login attempt to https://hub.docker.com/v2/ failed with status: 404 Not Found
Default docker hub registry url can be found like this:
docker info | grep Registry
Registry: https://index.docker.io/v1/
index.docker.io is what I was looking for.
2. Set the environment variables in gitlab settings
I wanted to publish gableroux/unity3d images using gitlab-ci, here's what I used in Gitlab's project > Settings > CI/CD > Variables
CI_REGISTRY_USER=gableroux
CI_REGISTRY_PASSWORD=********
CI_REGISTRY=docker.io
CI_REGISTRY_IMAGE=index.docker.io/gableroux/unity3d
CI_REGISTRY_IMAGE is important to set.
It defaults to registry.gitlab.com/<username>/<project>
regsitry url needs to be updated so use index.docker.io/<username>/<project>
Since docker hub is the default registry when using docker, you can also use <username>/<project> instead. I personally prefer when it's verbose so I kept the full registry url.
This answer should also cover other registries, just update environment variables accordingly. 🙌
To expand on the GabLeRoux's answer,
I had issues on the pushing stage of the GitLab CI build:
denied: requested access to the resource is denied
By changing my CI_REGISTRY to docker.io (remove the index.) I was able to successfully push.