git-secret: gpg: [don't know]: partial length invalid for packet type 20 in the gitlab ci job - gitlab-ci

I have a trouble with git secret in the gitlab ci jobs.
What I done:
init, add users, add files, hide them using git secret
create a job where I want to reveal files:
git secret:
stage: init
before_script:
- sh -c "echo 'deb https://gitsecret.jfrog.io/artifactory/git-secret-deb git-secret main' >> /etc/apt/sources.list"
- wget -qO - 'https://gitsecret.jfrog.io/artifactory/api/gpg/key/public' | apt-key add -
- apt-get update && apt-get install -y git-secret
script:
- echo $GPG_PRIVATE_KEY | tr ',' '\n' > ./pkey.gpg
- export GPG_TTY=$(tty)
- gpg --batch --import ./pkey.gpg
- git secret reveal -p ${GPG_PASSPHRASE}
Result logs:
...
$ gpg --batch --import ./pkey.gpg
gpg: directory '/root/.gnupg' created
gpg: keybox '/root/.gnupg/pubring.kbx' created
gpg: /root/.gnupg/trustdb.gpg: trustdb created
gpg: key SOMEKEY: public key "Email Name <ci#email.com>" imported
gpg: key SOMEKEY: secret key imported
gpg: Total number processed: 1
gpg: imported: 1
gpg: secret keys read: 1
gpg: secret keys imported: 1
$ git secret reveal -p ${GPG_PASSPHRASE}
gpg: [don't know]: partial length invalid for packet type 20
git-secret: abort: problem decrypting file with gpg: exit code 2: /path/to/decrypted/file
I don't understand where the problem. What mean packet type 20? And length of what?
Locally it revealed fine. Command git secret whoknows shows that email on the ci env can decrypt. Passphrase checked and passed to the job.

For me, the problem was the GnuPG versions being different between the encryption machine (v2.3) and the decryption side (v2.2).
After I downgraded it to v2.2 (due to v2.3 not yet being available on Debian), the problem went away.

This is a common problem with the format of the keys.
Since you're using GitLab CI, you should get advantage of the File type in the CI/CD Variables instead of storing the value of the GPG Key as a Variable type.
First of all, forget about generating the armor in one line with the piped | tr '\n' ',' and get the proper multiline armor.
Second, add it to your GitLab CI Variables with type "File", add an empty line at the end and then delete it (this seems stupid but will save you headaches, since it seems to be a problem when copying directly from the shell to the textbox in GitLab).
Third, import directly the file in your keychain:
gpg --batch --import $GPG_PRIVATE_KEY

Related

Load key "/root/.ssh/pipelines_id": invalid format

I am using bitbucket pipeline to deploy app on a ec2-server.
here is my bitbucket-pipelines.yaml file
image: atlassian/default-image:3
pipelines:
branches:
dev:
- step:
name: automated deployment
script:
- pipe: atlassian/scp-deploy:1.2.1
variables:
USER: 'ubuntu'
SERVER: $SERVER_IP
REMOTE_PATH: '/home/ubuntu/utags-test/server'
LOCAL_PATH: '${BITBUCKET_CLONE_DIR}/*'
- pipe: atlassian/ssh-run:0.4.1
variables:
SSH_USER: 'ubuntu'
SERVER: $SERVER_IP
COMMAND: 'cd /home/ubuntu/utags-test/server;docker pull paranjay1/utags-paranjay:dev;docker-compose down;docker-compose up -d'
SSH_KEY: $SERVER_PRIVATE_KEY
DEBUG: 'true'
services:
- docker
error while running pipeline
Build setup13s
pipe: atlassian/scp-deploy:1.2.1
....
....
Digest: sha256:b9111f61b5824ca7ed1cb63689a6da55ca6d6e8985eb778c36a5dfc2ffe776a8
Status: Downloaded newer image for bitbucketpipelines/scp-deploy:1.2.1
INFO: Configuring ssh with default ssh key.
INFO: Adding known hosts...
INFO: Appending to ssh config file private key path
INFO: Applied file permissions to ssh directory.
✔ Deployment finished.
pipe: atlassian/ssh-run:0.4.1
....
....
Digest: sha256:b8ff5416420ef659869bf1ea6e95502b8fa28ccd5e51321e4832d9d81fdefc18
Status: Downloaded newer image for bitbucketpipelines/ssh-run:0.4.1
INFO: Executing the pipe...
INFO: Using passed SSH_KEY
INFO: Executing command on 13.235.33.118
ssh -A -tt -i /root/.ssh/pipelines_id -o StrictHostKeyChecking=no -p 22 ubuntu#13.235.33.118 bash -c 'cd /utags-test/server;docker pull paranjay1/utags-paranjay:dev;docker-compose down;docker-compose up -d'
Load key "/root/.ssh/pipelines_id": invalid format
Load key "/root/.ssh/pipelines_id": invalid format
ubuntu#13.235.33.118: Permission denied (publickey,gssapi-keyex,gssapi-with-mic).
✖ Execution failed.
I already installed docker and docker-compose on my ec2-server
I generated the keys on bitbucket in sshkey section and added bitbucket public key to my authorized_keys file on ec2-server
$SERVER_PRIVATE_KEY contains the ec2-server private key
$SERVER_IP contains my ec2-server public IP
HOW CAN I SOLVE THIS ISSUE and what might be the cause of this error?
atlassian/ssh-run pipe documentation states the alternative SSH_KEY should be base64 encoded. My bet is you missed that info bit.
An base64 encoded alternate SSH_KEY to use instead of the key configured in the Bitbucket Pipelines admin screens (which is used by default). This should be encoded as per the instructions given in the docs for using multiple ssh keys.
Another good question would be: why aren't you using the ssh key provided by the pipeline instead?
You can use repository SSH key, so you won't need to encode it.
bitbucket.com/.../admin/addon/admin/pipelines/ssh-keys
Then remove SSH key variable and it defaultly uses repository ssh key.
you actually don't need to use "SSH_KEY: $SERVER_PRIVATE_KEY" in your pipe. you can use the default keys available in your bitbucket_repo > repository_settings > ssh_key. you can generate a key here. the generated public key should be in the remote server's "/home/ubuntu/.ssh/authorized_key" file. add your remote servers public-IP to the known host and fetch fingerprint.
but if you want to use a different ssh key then you have to add "SSH_KEY: $SERVER_PRIVATE_KEY" in your pipe. where,
$SERVER_PRIVATE_KEY - local machine's private_IP encoded to base64
you have to use $base64 -w 0 < my_ssh_key command to encode your key to base64

GitLab CI failing with SSH error loading key invalid format

I am starting to get into GitLab CI for my company. We have a PrestaShop, and I want automatic deployment to the web server after a Git push.
Unit testing will come later. At the moment I just need it to deal with putting a copy of the "/app" folder in the web root of the web server.
So this is what I have got...
before_script:
- apt-get update -qq
- apt-get install -qq git
- 'which ssh-agent || ( apt-get install -qq openssh-client )'
- eval $(ssh-agent -s)
- ssh-add <(echo "$SSH_PRIVATE_KEY")
- mkdir -p ~/.ssh
- '[[ -f /.dockerenv ]] && echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config'
deploy_test:
type: deploy
environment:
name: test
url: [test server domain]
script:
- ssh [user]#[server] -p [port] "cd [repo folder] && git checkout master && git pull origin master && exit"
- ssh [user]#[server] -p [port] "rsync -rzvh [repo /app folder] [web server root path]"
only:
- master
Recently, gitlab-runner has started failing with the error Error loading key "/dev/fd/63": invalid format.
Can you help me to solve that error?
FYI, I have my personal private key set as $SSH_PRIVATE_KEY environment var in GitLab - the public on the web server of course. SSH is enabled on the web server which has WHM and cPanel. And I pre-checked out a copy of master via cPanel on the web server into the [repo folder].
Originally, OpenSSH used the PKCS #1 format for RSA private keys. This format is not very secure, so newer versions have moved to a different format for storing private keys which is specific to OpenSSH. This is more secure, but it's not backwards compatible.
While it is possible to convert the keys with ssh-keygen, it would be far better for you to create a new key that you used only for deployments. That's a best practice because it separates your personal key from the deployments and means that if one is compromised, the other is not affected.
Since you'd need to create a new key anyway, you'd be better off using an Ed25519 key. Mozilla and others recommend this format of key because it is fast, secure, and easy to make constant time. You can create such a key with ssh-keygen -t ed25519 -f deployment-key, where deployment-key and deployment-key.pub will be the private and public keys.
If you're using CentOS 7 on the server, it does indeed support Ed25519 keys if it have been appropriately updated with patches, and whatever you're using on GitLab should also support it. You'll need to add the new public key to the remote server as with your personal key.
If you really want to continue to use this key, you should be able to export it with ssh-keygen -e -m PEM.
Did you check the $SSH_PRIVATE_KEY run on protected branches and tags pipeline only? if so you need to add your branch into protected. Setting->Repository->Protected Branch. or unchecked the option in Setting->CI/CD->Variables

How can I extract secrets using GitHub Actions?

I have a fairly basic scenario. I made a dedicated ssh key for this purpose and added it to my repository secrets.
Code gets pushed to master
GitHub action uploads it to server using ssh by doing echo "${{ secrets.SSH_KEY }}" > key.
After that I can use this key to connect to my server e.g. ssh -i key devops#myserver.com lsb_release -a
The problem is that for some reason GitHub actions cannot write it to file, it writes characters *** instead of the actual secret value into the file. Therefore obviously I cannot connect to my server.
How can I connect with ssh using this secret? Is there a way to connect without using a file? Can someone who did this common scenario using GitHub actions shed some light?
GitHub Actions should be able to write a secret to a file this way. The reason you see the stars is that the log is filtered, so if a secret would be logged, it's replaced in the log with three asterisks instead. This is a security measure against an accidental disclosure of secrets, since logs are often publicly available.
However, it's a good idea to avoid writing the secret to the log anyway if possible. You can write your command like this so you don't write the secret to the log:
- run: 'echo "$SSH_KEY" > key'
shell: bash
env:
SSH_KEY: ${{secrets.SSH_KEY}}
All you'll see in the log is echo "$SSH_KEY" > key, not the secret or any asterisks.
Note that you do want quotes here, since the > character is special to YAML.
If this doesn't work to log into your server, there's likely a different issue; this technique does work for writing secrets in the general case.
I found a way to do this. We used GatsbyJS for a project and it relies on a
.env.production file for the env variables. I tried to pass them as
env: to the github action, but that didn't work and they were ignored.
Here is what I did. I base 64 encoded the .env.production file:
base64 -i .env.production
Added the output to an env variable in github action. Then in my action I do:
echo ${{ secrets.ENV_PRODUCTION_FILE }} | base64 -d > .env.production
This way the contents of my .env.production file ended being written to the machine that executes the github action.
I used sed in a GHA to replace a TOKEN in a file like this :
run: |-
sed -i "s/TOKEN/${{secrets.MY_SECRET}}/g" "thefile"
The file looks like this:
credentials "app.terraform.io" {
token = "TOKEN"
}
encode it and decode back
- run: 'echo "$SSH_KEY" | base64'
shell: bash
env:
SSH_KEY: ${{ secrets.PRIVATE_KEY }}
and decode it back echo "<encoded string>" | base64 -d
Here is how to solve your actual problem of securely logging into an SSH server using a secret stored in GitHub Actions, named GITHUB_ACTIONS_DEPLOY.
Let's call this "beep", because it will cause an audible bell on the server you login to. Maybe you use this literally ping a server in your house when somebody pushes code to your repo.
- name: Beep
# if: github.ref == 'refs/heads/XXXX' # Maybe limit only this step to some branches
run: |
eval $(ssh-agent)
ssh-add - <<< "$SSH_KEY"
echo "* ssh-rsa XXX" >> /tmp/known_hosts # Get from your local ~/.ssh/known_hosts or from ssh-keyscan
ssh -o UserKnownHostsFile=/tmp/known_hosts user#example.com "echo '\a'"
env:
SSH_KEY: ${{ secrets.PMT_GITHUB_ACTIONS_DEPLOY }}
If, actually you are using SSH as part of a rsync push task, here is how to do that:
- name: Publish
if: github.ref == 'refs/heads/XXX'
run: |
eval $(ssh-agent)
ssh-add - <<< "$SSH_KEY"
echo "* ssh-rsa XXX" >> /tmp/known_hosts
rsync $FROM user#server:
env:
SSH_KEY: ${{ secrets.GITHUB_ACTIONS_DEPLOY }}
RSYNC_RSH: "ssh -o UserKnownHostsFile=/tmp/known_hosts"
You can decode a secret by looping through it with python shell, like this:
- name: Set env as secret
env:
MY_VAL: ${{ secrets.SUPER_SECRET }}
run: |
import os
data = open("file", "w")
for q in (os.getenv("MY_VAL")):
print q
data.write(q)
shell: python
This will both print each character to stdout and store them in file called file. stdout will have an output like this, while file should have the secret string saved inside.
s
e
c
r
e
t
The good solution is to use gpg for encrypting the key, adding it to a repo and decrypting it on the server using passphrase. The passprase should be stored as github project secret of course.
More info how I did it here: https://help.github.com/en/actions/automating-your-workflow-with-github-actions/creating-and-using-encrypted-secrets
echo ${{secrets.AWS_ACCESS_KEY_ID}} | sed 's/./& /g'

Why does importing the GPG key for the mono repo fail?

When following the steps to setup mono on the following site it is failing to import the GPG key for the repo.
https://www.mono-project.com/download/stable/#download-lin-centos
This is happening on CentOS machines running both 6.x and 7.x.
rpm --import "https://keyserver.ubuntu.com/pks/lookup?op=get&search=0x3FA7E0328081BFF6A14DA29AA6A19B38D3D831EF"
error: https://keyserver.ubuntu.com/pks/lookup?op=get&search=0x3FA7E0328081BFF6A14DA29AA6A19B38D3D831EF: key 1 not an armored public key.
This appears to be due to a missing newline at the end of the key file. If you open the key with vi and save it, without making any changes (this is one way to ensure there is a newline at the end of the file), the import works.
curl -v "https://keyserver.ubuntu.com/pks/lookup?op=get&search=0x3FA7E0328081BFF6A14DA29AA6A19B38D3D831EF" -okey
vi key
# don't modify, just save it with ":wq"
rpm --import key
Another way to add the newline to the end of the file: https://unix.stackexchange.com/a/31955
sed -i -e '$a\' key
see https://github.com/mono/mono/issues/15955
I used this workaround to then download. See https://github.com/mono/mono/issues/16025
rpm --import https://download.mono-project.com/repo/xamarin.gpg
su -c 'curl https://download.mono-project.com/repo/centos7-stable.repo | tee /etc/yum.repos.d/mono-centos7-stable.repo’

Getting GitLab CI to clone private repositories

I have GitLab & GitLab CI set up to host and test some of my private repos. For my composer modules under this system, I have Satis set up to resolve my private packages.
Obviously these private packages require an ssh key to clone them, and I have this working in the terminal - I can run composer install and get these packages, so long as I have the key added with ssh-add in the shell.
However, when running my tests in GitLab CI, if a project has any of these dependencies the tests will not complete as my GitLab instance needs authentication to get the deps (obviously), and the test fails saying Host key verification failed.
My question is how do I set this up so that when the runner runs the test it can authenticate to gitlab without a password? I have tried putting a password-less ssh-key in my runners ~/.ssh folder, however the build wont even add the key, "eval ssh-agent -s" followed by ssh-add seems to fail saying the agent isn't running...
See also other solutions:
git submodule permission (see Marco A.'s answer)
job token and override repo in git config (see a544jh's answer)
Here a full howto with SSH keys:
General Design
generating a pair of SSH keys
adding the private one as a secure environment variable of your project
making the private one available to your test scripts on GitLab-CI
adding the public one as a deploy key on each of your private dependencies
Generating a pair of public and private SSH keys
Generate a pair of public and private SSH keys without passphrase:
ssh-keygen -b 4096 -C "<name of your project>" -N "" -f /tmp/name_of_your_project.key
Adding the private SSH key to your project
You need to add the key as a secure environment variable to your project as
following:
browse https://<gitlab_host>/<group>/<project_name>/variables
click on "Add a variable"
fill the text field Key with SSH_PRIVATE_KEY
fill the text field Value with the private SSH key itself
click on "Save changes"
Exposing the private SSH key to your test scripts
In order to make your private key available to your test scripts you need to add
the following to your .gitlab-ci.yml file:
before_script:
# install ssh-agent
- 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )'
# run ssh-agent
- eval $(ssh-agent -s)
# add ssh key stored in SSH_PRIVATE_KEY variable to the agent store
- ssh-add <(echo "$SSH_PRIVATE_KEY")
# disable host key checking (NOTE: makes you susceptible to man-in-the-middle attacks)
# WARNING: use only in docker container, if you use it with shell you will overwrite your user's ssh config
- mkdir -p ~/.ssh
- echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config
Code Snippet comes from GitLab documentation
Adding the public SSH key as a deploy key to all your private dependencies
You need to register the public SSH key as deploy key to all your private
dependencies as following:
browse https://<gitlab_host>/<group>/<dependency_name>/deploy_keys
click on "New deploy key"
fill the text field Title with the name of your project
fill the text field Key with the public SSH key itself
click on "Create deploy key"
If you don't want to fiddle around with ssh keys or submodules, you can override the repo in git's configuration to authenticate with the job token instead (in gitlab-ci.yml):
before_script:
- git config --global url."https://gitlab-ci-token:${CI_JOB_TOKEN}#gitlab.example.com/group/repo.git".insteadOf git#gitlab.example.com:group/repo.git
I'm posting this as an answer since others weren't completely clear and/or detailed IMHO
Starting from GitLab 8.12+, assuming the submodule repo is in the same server as the one requesting it, you can now:
Set up the repo with git submodules as usual (git submodule add git#somewhere:folder/mysubmodule.git)
Modify your .gitmodules file as follows
[submodule "mysubmodule"]
path = mysubmodule
url = ../../group/mysubmodule.git
where ../../group/mysubmodule.git is a relative path from your repository to the submodule's one.
Add the following lines to gitlab-ci.yml
variables:
GIT_SUBMODULE_STRATEGY: recursive
to instruct the runner to fetch all submodules before the build.
Caveat: if your runner seems to ignore the GIT_SUBMODULE_STRATEGY directive, you should probably consider updating it.
(source: https://docs.gitlab.com/ce/ci/git_submodules.html)
The currently accepted answer embeds Gitlab-specific requirements into my .gitmodules file. This forces a specific directory layout for local development and would complicate moving to another version control platform.
Instead, I followed the advice in Juddling's answer. Here's a more complete answer.
My .gitmodules files has the following contents:
[submodule "myproject"]
url = git#git.myhost.com:mygroup/myproject.git
In my gitlab-ci.yml I have the following:
build:
stage: build
before_script:
- git config --global url."https://gitlab-ci-token:${CI_JOB_TOKEN}#git.myhost.com/".insteadOf "git#git.myhost.com:"
- git submodule sync && git submodule update --init
The trailing / and : are critical in the git config line, since we are mapping from SSH authentication to HTTPS. This tripped me up for a while with "Illegal port number" errors.
I like this solution because it embeds the Gitlab-specific requirements in a Gitlab-specific file, which is ignored by everything else.
I used deploy tokens to solve this issue, as setting up SSH keys for a test runner seems a little long winded.
git clone http://<username>:<deploy_token>#gitlab.example.com/tanuki/awesome_project.git
The deploy tokens are per project and are read only.
One way to solve this without changing the git repository's structure is to perform the following steps:
1. get ssh host keys
Get the ssh host keys of the server that you are running on. For gitlab.com:
run ssh-keyscan gitlab.com > known_hosts
check that ssh-keygen -lf known_hosts agrees with the fingerprints reported here.
copy the content of the known_hosts and paste it on a variable called SSH_KNOWN_HOSTS on the repository.
This step is only needed once.
2. configure the job to use ssh
before_script:
- git config --global url."https://gitlab-ci-token:${CI_JOB_TOKEN}#gitlab.com".insteadOf "git#gitlab.com:"
- mkdir -p ~/.ssh
- chmod 700 ~/.ssh
- echo "$SSH_KNOWN_HOSTS" > ~/.ssh/known_hosts
- chmod 644 ~/.ssh/known_hosts
The "ssh://git#gitlab.com" bit may be different if you are trying to do git clone git#gitlab.com: or pip install -e git+ssh://git#gitlab.com/...; adjust it accordingly to your needs.
At this point, your CI is able to use ssh to fetch from another (private) repository.
3. [Bonus DRY]
Use this trick to write it generically:
.enable_ssh: &enable_ssh |-
git config --global url."https://gitlab-ci-token:${CI_JOB_TOKEN}#gitlab.com".insteadOf "ssh://git#gitlab.com"
mkdir -p ~/.ssh
chmod 700 ~/.ssh
echo "$SSH_KNOWN_HOSTS" > ~/.ssh/known_hosts
chmod 644 ~/.ssh/known_hosts
and enable it on jobs that need it
test:
stage: test
before_script:
- *enable_ssh
script:
- ...
If your CI runner is running on a container model, you need to use the deploy key. doc: https://docs.gitlab.com/ee/user/project/deploy_tokens/#git-clone-a-repository
git clone https://<username>:<deploy_token>#gitlab.example.com/tanuki/awesome_project.git
Create your deploy token
Add your token in CI pipeline Variable
make sure your container has the git and change the git URL by insteadOf
image: docker:latest
before_script:
- apk add --no-cache curl jq python3 py3-pip git
- git config --global url."https://gitlab-ci-token:${CI_JOB_TOKEN}#gitlab.example.come/".insteadOf 'git#gitlab.example.come:'
for replace URL: https://docs.gitlab.com/ee/user/project/working_with_projects.html#authenticate-git-fetches
I had a scenario where I had to use my ssh key in 3 different scripts, so I put the ssh key stuff in a single shell script and called it first, before the other 3 scripts. This ended up not working, I think due to the ssh-agent not persisting between shell scripts, or something to that effect. I ended up actually just outputting the private key into the ~/.ssh/id_rsa file, which will for sure persist to other scripts.
.gitlab-ci.yml
script:
- ci/init_ssh.sh
- git push # or whatever you need ssh for
ci/init_ssh.sh
# only run in docker:
[[ ! -e /.dockerenv ]] && exit 0
mkdir -p ~/.ssh
echo "$GITLAB_RUNNER_SSH_KEY" > ~/.ssh/id_rsa
chmod 400 ~/.ssh/id_rsa
echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > /.ssh/config
It works like a charm!
If you are using an alpine-based image (maybe docker:latest or docker:dind), your before_script might look like this:
before_script:
- apk add --no-cache openssh-client git
- mkdir -p /.ssh && touch /.ssh/known_hosts
- ssh-keyscan gitlab.com >> /.ssh/known_hosts
- echo $SSH_KEY | base64 -d >> /.ssh/id_rsa && chmod 600 /.ssh/id_rsa
- git clone git#git.myhost.com:mygroup/myproject.git
Adding this to .gitlab-ci.yml did the trick for me.
(as mentioned here: https://docs.gitlab.com/ee/user/project/new_ci_build_permissions_model.html#dependent-repositories)
before_script:
echo -e "machine gitlab.com\nlogin gitlab-ci-token\npassword ${CI_JOB_TOKEN}" > ~/.netrc
(I tried setting up SSH_PRIVATE_KEY as mentioned in one of the answers above, it won't work)
Gitlab 15.9.0 introduces an update to the pre-defined variable CI_JOB_TOKEN. Now you can control other projects' access to your private repository, see the release note and documentation.
Once access is granted, you can clone private repositories by adding this line to your job's scripts or before_scripts.
git clone https://gitlab-ci-token:${CI_JOB_TOKEN}#gitlab.example.com/<namespace>/<project>
Unfortunately, this still does not play nicely with the submodule integration with Gitlab CI/CD. Instead, I do this in my projects.
# .gitlab-ci.yml
default:
before_script:
- |
cat << EOF > ~/.gitconfig
[url "https://gitlab-ci-token:${CI_JOB_TOKEN}#gitlab.example.com/<namespace>/<project>.git"]
insteadOf = git#gitlab.example.com/<namespace>/<project>.git
EOF
- git submodule update --init --recursive
And this is what my .gitmodules would look like
[submodule "terraform-eks"]
path = modules/<project>
url = git#gitlab.example.com/<namespace>/<project>.git
branch = main
Hope this help!
Seems there is finally a reasonable solution.
In short as of GitLab 8.12 all you need to do is use relative paths in the .submodules, and the git submodule update --init will simply work