Dependencies in package.json:
"dependencies": {
"my-repo": "git+ssh://github.com/org-name/my-repo.git"
},
GitHub Actions:
name: Test
on: [push, pull_request]
jobs:
test:
name: Test
runs-on: ubuntu-18.04
steps:
- name: Checkout
uses: actions/checkout#v2
- name: Setup Node.js
uses: actions/setup-node#v2
with:
node-version: 12
registry-url: 'https://npm.pkg.github.com'
scope: '#org-name'
- uses: webfactory/ssh-agent#v0.4.1
with:
ssh-private-key: ${{ secrets.DEPLOY_KEY }}
- name: Install dependencies
run: yarn
- name: Test
run: yarn test
The DEPLOY_KEY in the GitHub Actions secrets is the private key and I've added the corresponding public key as a deploy key in the dependency repo.
I generated the key with ssh-keygen -m PEM -t rsa -b 4096 -C "ssh://github.com/org-name/my-repo.git" -f ./deploykey -q -N ""
Here's the failure I see in the GitHub Actions output:
Exit code: 128
Command: git
Arguments: ls-remote --tags --heads ssh://github.com/org-name/my-repo.git
Directory: /home/runner/work/auth-package/auth-package
Output:
Warning: Permanently added the RSA host key for IP address '140.82.112.4' to the list of known hosts.
runner#github.com: Permission denied (publickey).
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
Really losing my mind on this one!
I don't know If I'm getting it right but to access to a different repo you need an access token. Therefore you need a Github Application with a private key secret. https://github.com/settings/apps
You need those three env variables in your workflow:
- uses: actions/checkout#v2
- name: get secrets
env:
PRIVATE_KEY: ${{ secrets.PRIVATE_KEY }}
APP_ID: ${{ secrets.APP_ID }}
INSTALLATION_ID: ${{ secrets.INSTALLATION_ID }}
Then create your JWT (check your JWT here: https://jwt.io/) to create the access token via REST API
run: |
PEM=$PRIVATE_KEY
GITHUB_APP_ID=$APP_ID
NOW=$( date +%s )
IAT="${NOW}"
EXP=$((${NOW} + 600))
HEADER_RAW='{"alg":"RS256"}'
HEADER=$( echo -n "${HEADER_RAW}" | openssl base64 | tr -d '=' | tr '/+' '_-' | tr -d '\n' )
PAYLOAD_RAW='{"iat":'"${IAT}"',"exp":'"${EXP}"',"iss":'"${GITHUB_APP_ID}"'}'
PAYLOAD=$( echo -n "${PAYLOAD_RAW}" | openssl base64 | tr -d '=' | tr '/+' '_-' | tr -d '\n' )
HEADER_PAYLOAD="${HEADER}"."${PAYLOAD}"
SIGNATURE=$( openssl dgst -sha256 -sign <(echo -n "${PEM}") <(echo -n "${HEADER_PAYLOAD}") | openssl base64 | tr -d '=' | tr '/+' '_-' | tr -d '\n' )
JWT="${HEADER_PAYLOAD}"."${SIGNATURE}"
Then start your API CALL:
ACCESS_TOKEN=$(curl -sS -X POST \
-H "Authorization: Bearer "$JWT"" \
-H "Accept: application/vnd.github.v3+json" \
https://github.com/api/v3/app/installations/"$INSTALLATION_ID"/access_tokens | grep -o '"token": "[^"]*' | grep -o '[^"]*$')
Then start with your Git Clone command:
git clone https://x-access-token:"$ACCESS_TOKEN"#github.com/../repo.git
cd repo
git config --global user.email "<email>"
git config --global user.name "<name>"
git branch upload
git checkout upload
git commit -m "update"
git push --set-upstream origin upload
Then you can clone other repos in your current one for example. Be aware to grant the app access to the repos.
Related
Here is a github workflow that I would like to use to create a release.
name: make release
on:
workflow_dispatch:
jobs:
try-to-create-a-tag:
runs-on: ubuntu-latest
steps:
- name: Create release
run: curl \
-X POST \
-H "Accept:application/vnd.github+json" \
-H "Authorization:token ${ secrets.GITHUB_TOKEN }" \
https://api.github.com/repos/.../releases \
-d '{"tag_name":"tag_test11","target_commitish":"main","name":"tag_test11","body":"Description of the release","draft":false,"prerelease":false,"generate_release_notes":false}'
The workflow runs without error, however I get the following message from curl:
authorization:token ${ secrets.GITHUB_TOKEN }: bad substitution
I have also tried using double braces, however this didn't work either. How can I substitute the variable into the curl command?
NOTE: the error is that one must use a | right after run: when splitting over multiple lines.
You have to use double brackets:
${{ secrets.GITHUB_TOKEN }}
Documentation:
https://docs.github.com/en/actions/security-guides/encrypted-secrets#example-using-bash
jobs:
try-to-create-a-tag:
runs-on: ubuntu-latest
steps:
- name: Create release
run: |
curl \
-X POST \
-H "Accept:application/vnd.github+json" \
-H "Authorization:token ${{ secrets.GITHUB_TOKEN }}" \
https://api.github.com/repos/.../releases \
-d '{"tag_name":"tag_test11","target_commitish":"main","name":"tag_test11","body":"Description of the release","draft":false,"prerelease":false,"generate_release_notes":false}'
Below is my first gitlab-ci.yml script for a static website. It does exactly what I need. It does not require a build process per Angular or React. Does anyone see room for improvement? Any glaring newbie mistakes? Are the exit commands necessary or will it automatically log off when the script terminates? Also, is it necessary to remove the deployment keys at the end of each deployment section?
- build
- deploy_staging
- deploy_production
build:
image: alpine
stage: build
before_script:
- apk add zip
script:
- zip -r website.zip * -x "composer.json" -x "composer.lock" -x "gruntfile.js" -x "package-lock.json" -x "package.json" -x "Read Me" -x "_/" -x "deploy_production.sh" -x "deploy_staging.sh" -x "README.md" -x "Read Me Custom.txt" -x "gitlab-ci.yml"
artifacts:
paths:
- website.zip
deploy_to_staging:
image: alpine
stage: deploy_staging
before_script:
- apk add unzip openssh-client
- eval $(ssh-agent -s)
- echo "$DEPLOYMENT_KEY" | tr -d '\r' | ssh-add -
- mkdir -p ~/.ssh
- chmod 700 ~/.ssh
- ssh-keyscan -H "$DEPLOYMENT_SERVER" >> ~/.ssh/known_hosts
- chmod 644 ~/.ssh/known_hosts
script:
- scp website.zip "$DEPLOYMENT_LOGIN":"$DEPLOYMENT_PATH"
- ssh -p 2222 "$DEPLOYMENT_LOGIN" "
cd temp;
rm website.zip;
cd ../staging;
bash -O extglob -c 'rm -rf !(website.zip)';
unzip website.zip;
"cp website.zip ../../temp/";
rm website.zip;
exit; "
rm -f ~/.ssh/id_rsa
only:
- main
deploy_to_production:
image: alpine
stage: deploy_production
before_script:
- apk add unzip openssh-client
- eval $(ssh-agent -s)
- echo "$DEPLOYMENT_KEY" | tr -d '\r' | ssh-add -
- mkdir -p ~/.ssh
- chmod 700 ~/.ssh
- ssh-keyscan -H "$DEPLOYMENT_SERVER" >> ~/.ssh/known_hosts
- chmod 644 ~/.ssh/known_hosts
script:
- ssh -p 2222 "$DEPLOYMENT_LOGIN" "
cp temp/website.zip portal/;
cd portal;
bash -O extglob -c 'rm -rf !(website.zip)';
unzip website.zip;
rm website.zip;
exit; "
rm -f ~/.ssh/id_rsa
when: manual
only:
- main
The scripts looks pretty straight forward, and it does what it should do. But there are some things you should consider.
you rely on the fact that NO deployment pipeline is run before you execute your life deployment. But theoretically there is the chance, that the zip on the server in the temp folder is not coming from the same pipeline. When eg. Another pipeline executed the staging call already. This way you would deploy the newer package, although you execute the old pipeline. Hence that i recommend to upload again, for safety.
your script contains some duplication, which results in errors, when you need to adapt those duplicated code. I added an example of inheritance for you.
Use environments. GitLab has a pretty nice feature called environments, where you have an overview of existing environments and what is deployed to which environment, by which pipeline. https://docs.gitlab.com/ee/ci/yaml/#environment
Use resourcegroups to prevent parallel job executions to the same environment. https://docs.gitlab.com/ee/ci/yaml/#resource_group
Additionally something to consider on a later stage is adding real releases and tagging to your project - but this is an own topic overall :)
Disclaimer: i am not a pro either, but those are the changes and considerations i would take into account :)
stages:
- build
- deploy_staging
- deploy_production
build:
image: alpine
stage: build
before_script:
- apk add zip
script:
- zip -r website.zip * -x "composer.json" -x "composer.lock" -x "gruntfile.js" -x "package-lock.json" -x "package.json" -x "Read Me" -x "_/" -x "deploy_production.sh" -x "deploy_staging.sh" -x "README.md" -x "Read Me Custom.txt" -x "gitlab-ci.yml"
artifacts:
paths:
- website.zip
.deploy:
image: alpine
before_script:
- apk add unzip openssh-client
- eval $(ssh-agent -s)
- echo "$DEPLOYMENT_KEY" | tr -d '\r' | ssh-add -
- mkdir -p ~/.ssh
- chmod 700 ~/.ssh
- ssh-keyscan -H "$DEPLOYMENT_SERVER" >> ~/.ssh/known_hosts
- chmod 644 ~/.ssh/known_hosts
script:
- scp website.zip "$DEPLOYMENT_LOGIN":"$DEPLOYMENT_PATH"
- ssh -p 2222 "$DEPLOYMENT_LOGIN" "
cd $DEPLOYMENT_PATH;
bash -O extglob -c 'rm -rf !(website.zip)';
unzip website.zip;
rm website.zip;
exit; "
after_script:
- rm -f ~/.ssh/id_rsa
only:
- main
deploy_to_staging:
stage: deploy_staging
variables:
DEPLOYMENT_PATH: "../staging"
extends: .deploy # inheritance to reduce duplicated code
environment:
name: staging
resource_group: staging
deploy_to_production:
stage: deploy_production
variables:
DEPLOYMENT_PATH: "portal"
extends: .deploy # inheritance to reduce duplicated code
environment:
name: production
resource_group: production
when: manual
I have been trying to build my project and deployment to a remote server using gitlab CI runner and using this link as reference
https://www.digitalocean.com/community/tutorials/how-to-set-up-a-continuous-deployment-pipeline-with-gitlab-ci-cd-on-ubuntu-18-04
After runing the pipeline, the publish stage is giving error about the docker tagging
$ docker build -t $TAG_COMMIT -t $TAG_LATEST .
invalid argument "/patch-9:64a25b49" for "-t, --tag" flag: invalid reference format
I have tried changing the docker build tagging in different formats but still could not find out why the error.
I have tried changing the tagging
TAG_LATEST: ${CI_REGISTRY_IMAGE}/${CI_COMMIT_REF_NAME}:latest
TAG_COMMIT:${CI_REGISTRY_IMAGE}/${CI_COMMIT_REF_NAME}:${CI_COMMIT_SHORT_SHA}
but I am still get the error
$ cd $GOPATH/src/$REPO/$NAMESPACE/$PROJECT
$ docker build -t $TAG_COMMIT -t $TAG_LATEST .
invalid argument "/patch-10:fbf4855b" for "-t, --tag" flag: invalid reference format
See 'docker build --help'.
Can anyone help me solve this problem?
My .gitlab-ci.yml file looks
image: golang:1.15.3
variables:
REPO: github.com
NAMESPACE: daniel
PROJECT: danapp
TAG_LATEST: $CI_REGISTRY_IMAGE/$CI_COMMIT_REF_NAME:latest
TAG_COMMIT: $CI_REGISTRY_IMAGE/$CI_COMMIT_REF_NAME:$CI_COMMIT_SHORT_SHA
before_script:
- mkdir -p $GOPATH/src/$REPO/$NAMESPACE/$PROJECT
- cp -r -v $CI_PROJECT_DIR $GOPATH/src/github.com/daniel
- cd $GOPATH/src/$REPO/$NAMESPACE/$PROJECT
stages:
- build
- publish
- deploy
compile:
stage: build
script:
- go build -race -ldflags "-extldflags '-static'" -o $CI_PROJECT_DIR/danapp
artifacts:
paths:
- danapp
publish:
image: docker:latest
stage: publish
services:
- docker:dind
script:
- docker build -t $TAG_COMMIT -t $TAG_LATEST .
- docker login -u gitlab-ci-token -p $CI_BUILD_TOKEN $CI_REGISTRY
- docker push $TAG_COMMIT
- docker push $TAG_LATEST
deploy:
image: alpine:latest
stage: deploy
tags:
- deployment
before_script:
- apk update && apk add openssh-client
- eval $(ssh-agent -s)
- echo "$SSH_PRIVATE_KEY" | tr -d '\r' | ssh-add -
- mkdir -p ~/.ssh
- chmod 700 ~/.ssh
script:
- chmod og= $SSH_PRIVATE_KEY
- apk update && apk add openssh-client
- ssh -i $SSH_PRIVATE_KEY -o StrictHostKeyChecking=no admin#192.168.x.x "docker login -u gitlab-ci-token -p $CI_BUILD_TOKEN $CI_REGISTRY"
- ssh -i $SSH_PRIVATE_KEY -o StrictHostKeyChecking=no admin#192.168.x.x "docker pull $TAG_COMMIT"
- ssh -i $SSH_PRIVATE_KEY -o StrictHostKeyChecking=no admin#192.168.x.x "docker container rm -f danapp || true"
- ssh -i $SSH_PRIVATE_KEY -o StrictHostKeyChecking=no admin#192.168.x.x "docker run -d -p 20005:20005 --name danapp $TAG_COMMIT"
environment: stagging
only:
- master
I suggest you change your $CI_COMMIT_REF_NAME to $CI_COMMIT_REF_SLUG
Maybe this solved.
I added a deploy key with write access to my GitLab repository. My .gitlab-ci.yml file contains:
- git clone git#gitlab.domain:user/repo.git
- git checkout master
- git add myfile.pdf
- git commit -m "Generated PDF file"
- git push origin master
The deploy key works when cloning the repository.
Pushing is not possible, even if the deploy key has write access.
remote: You are not allowed to upload code.
fatal: unable to access 'https://gitlab-ci-token:xxxxxxxxxxxxxxxxxxxx#domain/user/repo.git/': The requested URL returned error: 403
I just encountered the same problem and saw this question without answer, so there is my solution.
Problem
The problem is caused by the fact that the remote url used by git to push the code is in the form http(s)://gitlab-ci-token:xxxxxxxxxxxxxxxxxxxx#git.mydomain.com/group/project.git.
This url is using http(s) protocol so git doesn't use the ssh deploy key that you setup.
Solution
The solution is to change the push url of the remote origin so it matches ssh://git#git.mydomain.com/group/project.git.
The easiest way to do so is to use the predefined variable CI_REPOSITORY_URL.
Here is an example of code doing so by using sed:
# Change url from http(s) to ssh
url_host=$(echo "${CI_REPOSITORY_URL}" | sed -e 's|https\?://gitlab-ci-token:.*#|ssh://git#|g')
echo "${url_host}"
# ssh://git#git.mydomain.com/group/project.git
# Set the origin push url to the new one
git remote set-url --push origin "${url_host}"
Also, those using docker executor may want to verify the SSH host key as suggested by the gitlab documentation on deploy keys for docker executor.
So I give a more complete example for docker executor.
The code is mainly from gitlab documentation on ssh deploy keys.
In this example, the private deploy key is stored inside a variable named SSH_PRIVATE_KEY.
create:push:pdf:
before_script:
- 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )'
- eval $(ssh-agent -s)
- echo "${SSH_PRIVATE_KEY}" | tr -d '\r' | ssh-add - > /dev/null
- mkdir -p ~/.ssh
- chmod 700 ~/.ssh
- git config --global user.email "email#example.com"
- git config --global user.name "User name"
- gitlab_hostname=$(echo "${CI_REPOSITORY_URL}" | sed -e 's|https\?://gitlab-ci-token:.*#||g' | sed -e 's|/.*||g')
- ssh-keyscan "${gitlab_hostname}" >> ~/.ssh/known_hosts
- chmod 644 ~/.ssh/known_hosts
script:
- git checkout master
- git add myfile.pdf
- git commit -m "Generated PDF file"
- url_host=$(echo "${CI_REPOSITORY_URL}" | sed -e 's|https\?://gitlab-ci-token:.*#|ssh://git#|g')
- git remote set-url --push origin "${url_host}"
- git push origin master
I have Bamboo 6.6.0 and pipelines in YAML format,
Pipeline needs some secrets: passwords, configs, and ssh keys.
Bamboo Global variables are ok for passwords, but how can I store ssh keys and config files in a secure way in Bamboo?
The only decent way I've found so far is to encrypt ssh key with base64, put it into Bamboo Global variable (it MUST has secret word in name to be not visible in GUI and logs) and decrypt it during pipeline execution.
Example:
---
project:
key: LAMBDA
plan:
key: LAMBDA
name: GaaS Example Lambda Scripted
stages: #TODO: Clean workspace
- jobs:
- dockerImage: kagarlickij/example-lambda
scripts:
- echo "Creating .ssh dir.." && mkdir ~/.ssh
- echo "Exporting SSH key from Bamboo Global vars to env var.." && export BITBUCKET_SSH_KEY=${bamboo.bitbucket_ssh_key_secret}
- echo "Decrypting SSH key.." && if [ -f ~/.ssh/id_rsa ]; then rm -f ~/.ssh/id_rsa ; fi && echo $BITBUCKET_SSH_KEY | base64 --decode > ~/.ssh/id_rsa && chmod 400 ~/.ssh/id_rsa
- echo "Adding BitBucket to known_hosts.." && ssh-keyscan bitbucket.org >> ~/.ssh/known_hosts