I have a very basic integration configured for Gitlab-CI but it fails almost at the beginning when it has to clone the code.
My integration is this:
image: node:latest
stages:
- build
- test
cache:
paths:
- node_modules/
- dist/
build-prod:
stage: build
script:
- npm install
- npm run build-prod
artifacts:
paths:
- node_modules/
- dist/
test_with_karma:
stage: test
script: ng test
And the error that I get is this:
Running with gitlab-runner 11.7.0 (8bb608ff)
on fakehost 2eaf11ea
Using Docker executor with image node:latest ...
Pulling docker image node:latest ...
Using docker image sha256:8c67bfd7b95bdc535edc4a4144f5392b0f73efd6385fbcb47747d028d7059359 for node:latest ...
Running on runner-2eaf11ea-project-56-concurrent-0 via fakehost...
Cloning repository...
Cloning into '/builds/redacted/frontend'...
remote: You are not allowed to download code from this project.
fatal: unable to access 'https://gitlab-ci-token:xxxxxxxxxxxxxxxxxxxx#working-domain.com/redacted/frontend.git/': The requested URL returned error: 403
/bin/bash: line 65: cd: /builds/redacted/frontend: No such file or directory
ERROR: Job failed: exit code 1
What is the problem here?
Check if this is covered by gitlab-org/gitlab-ce issue 39469
YAY - it works for me. This problem seems to have multiple solutions.
The one that worked for me is #44855
To summarize. Being an Administrator on Gitlab does not mean you have the "access" to do whatever you want to do in Gitlab.
"Unable to access" permissions applies to the person who is logged into Gitlab and running the job.
To fix the problem - the person / account running the job must be a member (master) of the project.
This will apply to private projects.
It is not necessary to make a private project Public even though that appears to fix the problem. GITLAB suggests you must have https for the project to work you can use http.
SOLUTION - add your account to the project even if you are the Administrator
And:
Conrad has described it correctly.
You need to have rights to the project to run pipeline, however, as administrator, you can start any pipeline.
I've got the case when the user being Admin in Gitlab could push his commit from command line, although theoretically having no rights to project - and the pipeline has failed.
This inconsistency need to be fixed, either Admin user should not be able to push/start pipeline, having no rights for it, or he should authomatically be granted all rights to all projects. I'd prefer the first one, because it separates gitlab administration from project rights. Sometimes I prefer not having full rights, just like working as non-root under Linux.
Related
I'm building an apptainer/singularity multi-stage recipe in a gitlab CI environment.
The first step of the recipe is built from an image hosted in a private registry, whereas the second built from an image hosted on dockerhub. Something like this:
# First stage
BootStrap: docker
Registry: <my_private_registry>
From: <my_image>
Stage: base
%files
...
%post
...
# Second stage
BootStrap: docker
Registry: index.docker.io
From: continuumio/miniconda3
Stage: final
%files from base
...
%post
...
Since the first registry is private, in the gitlab CI instance I'm setting the variables APPTAINER_DOCKER_USERNAME and APPTAINER_DOCKER_PASSWORD, as suggested here for CI/CD workflow.
This allows to build the first stage of the recipe succesfully.
Unfortunately, when the build of the second stage starts, it fails with:
> FATAL: While performing build: conveyor failed to get: unable to retrieve auth token: invalid username/password: unauthorized: incorrect username or password
I think because the credentials for my private registry are passed to dockerhub in the second stage.
How can I login to different registries in multi-stage builds?
Any idea about how to deal with this problem?
I found a way to accomplish what I wanted. The fact was that environment variables overrides other login modes.
So I deleted the APPTAINER_DOCKER_USERNAME and APPTAINER_DOCKER_PASSWORD environment variables and, using this method, I added the following before_script field to my .gitlab-ci.yaml:
apptainer:
stage: deploy
image:
name: kaczmarj/apptainer:1.1.3
entrypoint: [""]
tags:
- privileged
before_script:
- echo "$DOCKER_REGISTRY_TOKEN" | apptainer remote login --username <my_username> --password-stdin docker://$CI_REGISTRY
This way, both the private registry (stored in $CI_REGISTRY) and the public
one (dockerhub) are available.
So I have a quite simple gitlab-ci.yml script:
test stage:
stage: build
artifacts:
paths:
- result/
script:
…
So the problem is when it gets to the “Uploading artifacts for successful job”, it prints “Missing /usr/local/bin/gitlab-runner. Uploading artifacts is disabled”.
Tried to change an owner and a group of the gitlab-runner file to “gitlab-runner”, even gave 777 rights, but nothing helped.
Any ideas where I’m wrong?
If you are using gitlab latest version, it will be installed in /usr/bin/gitlab-runner, but you are trying to use /usr/local/bin/gitlab-runner.
I am trying to add bitbucket pipelines to deploy my angular application to one of our servers.
I configured SSH on my server and I could fetch the Host's finger print under Known hosts in Bitbucket
Below is my YAML file.
image: node:14
pipelines:
branches:
master:
- step:
name: Building Test angular application
caches:
- node
script:
- echo "npm install in progress.."
- npm install
- echo "Installing angular/cli..."
- npm install -g #angular/cli
- echo "Starting the Build process.."
- ng build
artifacts:
- dist/** # Save build for next steps
- step:
name: "Deployment"
script:
- pipe: atlassian/scp-deploy:0.3.3
variables:
USER: $USER
SERVER: $SERVER
REMOTE_PATH: '/c/testscp/'
LOCAL_PATH: 'dist/*'
SSH_KEY: $MY_SSH_KEY
The first step is running fine without any issues and I can see dist folder being added to the artifact however the second step is failing with the below error.
scp -rp -i /opt/atlassian/pipelines/agent/ssh/id_rsa_tmp dist/TestPipelineApplication <<USER>>#<<SERVERIP>>:/c/testscp/
Warning: Identity file /opt/atlassian/pipelines/agent/ssh/id_rsa_tmp not accessible: No such file or directory.
Load key "/root/.ssh/pipelines_id": invalid format
Permission denied, please try again.
Permission denied, please try again.
<<USER>>#<<SERVERIP>>: Permission denied (publickey,password,keyboard-interactive).
I never configured pipelines before so I am not completely sure what I am missing here.
Also, I looked into the below documentation but of no luck
https://bitbucket.org/atlassian/scp-deploy/src/1.0.1/README.md
Any help or suggestions are greatly appreciated.
On my Gitlab CI / CD, I have a terraform code that requires Python installed to use an external module.
When running terraform plan via Gitlab pipelines, I get the following error:
module.notify_slack.module.lambda.data.aws_caller_identity.current[0]: Refreshing state...
Error: can't find external program "python3"
on .terraform/modules/notify_slack.lambda/terraform-aws-lambda-1.6.0/package.tf line 3, in data "external" "archive_prepare":
3: data "external" "archive_prepare" {
ERROR: Job failed: exit code 1
What image do I need to use that contains Terraform and Python? Will I need to create my own docker image?
I know this is a bit of an old post, but I'll share my solution in case anyone else stumbles upon this problem too.
Choose an existing python image and install terraform manually - this seems to me to be the easiest solution, if pragmatism is important to you.
This is the relevant section of my .gitlab-ci.yml file:
default:
image: python:latest
before_script:
- python -V # Display version for debugging purposes only
- apt-get update -y
- apt-get install unzip wget -y
- wget https://releases.hashicorp.com/terraform/${TERRAFORM_VERSION}/terraform_${TERRAFORM_VERSION}_linux_amd64.zip
- unzip terraform_${TERRAFORM_VERSION}_linux_amd64.zip
- mv terraform /usr/local/bin/
- terraform --version # Display version for debugging purposes only
The environment variable was set up in the GitLab CI/CD settings, otherwise just change it for the specific version of Terraform you want.
I was pleasantly surprised at the speed in which this installation takes place, as this clearly isn't an optimal way to do it - the best performing runner will probably use your own custom image with all of your required dependencies pre-installed - I'll leave you to decide whether its worth it for your own purposes. Nonetheless, this solution doesn't appear to be prohibitively slow.
Gitlab provides a .gitlab-ci.yml template for building and publishing images to its own registry (click "new file" in one of your project, select .gitlab-ci.yml and docker). The file looks like this and it works out of the box :)
# This file is a template, and might need editing before it works on your project.
# Official docker image.
image: docker:latest
services:
- docker:dind
before_script:
- docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY
build-master:
stage: build
script:
- docker build --pull -t "$CI_REGISTRY_IMAGE" .
- docker push "$CI_REGISTRY_IMAGE"
only:
- master
build:
stage: build
script:
- docker build --pull -t "$CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG" .
- docker push "$CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG"
except:
- master
But by default, this will publish to gitlab's registry. How can we publish to docker hub instead?
No need to change that .gitlab-ci.yml at all, we only need to add/replace the environment variables in project's pipeline settings.
1. Find the desired registry url
Using hub.docker.com won't work, you'll get the following error:
Error response from daemon: login attempt to https://hub.docker.com/v2/ failed with status: 404 Not Found
Default docker hub registry url can be found like this:
docker info | grep Registry
Registry: https://index.docker.io/v1/
index.docker.io is what I was looking for.
2. Set the environment variables in gitlab settings
I wanted to publish gableroux/unity3d images using gitlab-ci, here's what I used in Gitlab's project > Settings > CI/CD > Variables
CI_REGISTRY_USER=gableroux
CI_REGISTRY_PASSWORD=********
CI_REGISTRY=docker.io
CI_REGISTRY_IMAGE=index.docker.io/gableroux/unity3d
CI_REGISTRY_IMAGE is important to set.
It defaults to registry.gitlab.com/<username>/<project>
regsitry url needs to be updated so use index.docker.io/<username>/<project>
Since docker hub is the default registry when using docker, you can also use <username>/<project> instead. I personally prefer when it's verbose so I kept the full registry url.
This answer should also cover other registries, just update environment variables accordingly. 🙌
To expand on the GabLeRoux's answer,
I had issues on the pushing stage of the GitLab CI build:
denied: requested access to the resource is denied
By changing my CI_REGISTRY to docker.io (remove the index.) I was able to successfully push.