I am using bitbucket pipeline to deploy app on a ec2-server.
here is my bitbucket-pipelines.yaml file
image: atlassian/default-image:3
pipelines:
branches:
dev:
- step:
name: automated deployment
script:
- pipe: atlassian/scp-deploy:1.2.1
variables:
USER: 'ubuntu'
SERVER: $SERVER_IP
REMOTE_PATH: '/home/ubuntu/utags-test/server'
LOCAL_PATH: '${BITBUCKET_CLONE_DIR}/*'
- pipe: atlassian/ssh-run:0.4.1
variables:
SSH_USER: 'ubuntu'
SERVER: $SERVER_IP
COMMAND: 'cd /home/ubuntu/utags-test/server;docker pull paranjay1/utags-paranjay:dev;docker-compose down;docker-compose up -d'
SSH_KEY: $SERVER_PRIVATE_KEY
DEBUG: 'true'
services:
- docker
error while running pipeline
Build setup13s
pipe: atlassian/scp-deploy:1.2.1
....
....
Digest: sha256:b9111f61b5824ca7ed1cb63689a6da55ca6d6e8985eb778c36a5dfc2ffe776a8
Status: Downloaded newer image for bitbucketpipelines/scp-deploy:1.2.1
INFO: Configuring ssh with default ssh key.
INFO: Adding known hosts...
INFO: Appending to ssh config file private key path
INFO: Applied file permissions to ssh directory.
✔ Deployment finished.
pipe: atlassian/ssh-run:0.4.1
....
....
Digest: sha256:b8ff5416420ef659869bf1ea6e95502b8fa28ccd5e51321e4832d9d81fdefc18
Status: Downloaded newer image for bitbucketpipelines/ssh-run:0.4.1
INFO: Executing the pipe...
INFO: Using passed SSH_KEY
INFO: Executing command on 13.235.33.118
ssh -A -tt -i /root/.ssh/pipelines_id -o StrictHostKeyChecking=no -p 22 ubuntu#13.235.33.118 bash -c 'cd /utags-test/server;docker pull paranjay1/utags-paranjay:dev;docker-compose down;docker-compose up -d'
Load key "/root/.ssh/pipelines_id": invalid format
Load key "/root/.ssh/pipelines_id": invalid format
ubuntu#13.235.33.118: Permission denied (publickey,gssapi-keyex,gssapi-with-mic).
✖ Execution failed.
I already installed docker and docker-compose on my ec2-server
I generated the keys on bitbucket in sshkey section and added bitbucket public key to my authorized_keys file on ec2-server
$SERVER_PRIVATE_KEY contains the ec2-server private key
$SERVER_IP contains my ec2-server public IP
HOW CAN I SOLVE THIS ISSUE and what might be the cause of this error?
atlassian/ssh-run pipe documentation states the alternative SSH_KEY should be base64 encoded. My bet is you missed that info bit.
An base64 encoded alternate SSH_KEY to use instead of the key configured in the Bitbucket Pipelines admin screens (which is used by default). This should be encoded as per the instructions given in the docs for using multiple ssh keys.
Another good question would be: why aren't you using the ssh key provided by the pipeline instead?
You can use repository SSH key, so you won't need to encode it.
bitbucket.com/.../admin/addon/admin/pipelines/ssh-keys
Then remove SSH key variable and it defaultly uses repository ssh key.
you actually don't need to use "SSH_KEY: $SERVER_PRIVATE_KEY" in your pipe. you can use the default keys available in your bitbucket_repo > repository_settings > ssh_key. you can generate a key here. the generated public key should be in the remote server's "/home/ubuntu/.ssh/authorized_key" file. add your remote servers public-IP to the known host and fetch fingerprint.
but if you want to use a different ssh key then you have to add "SSH_KEY: $SERVER_PRIVATE_KEY" in your pipe. where,
$SERVER_PRIVATE_KEY - local machine's private_IP encoded to base64
you have to use $base64 -w 0 < my_ssh_key command to encode your key to base64
Related
I have a trouble with git secret in the gitlab ci jobs.
What I done:
init, add users, add files, hide them using git secret
create a job where I want to reveal files:
git secret:
stage: init
before_script:
- sh -c "echo 'deb https://gitsecret.jfrog.io/artifactory/git-secret-deb git-secret main' >> /etc/apt/sources.list"
- wget -qO - 'https://gitsecret.jfrog.io/artifactory/api/gpg/key/public' | apt-key add -
- apt-get update && apt-get install -y git-secret
script:
- echo $GPG_PRIVATE_KEY | tr ',' '\n' > ./pkey.gpg
- export GPG_TTY=$(tty)
- gpg --batch --import ./pkey.gpg
- git secret reveal -p ${GPG_PASSPHRASE}
Result logs:
...
$ gpg --batch --import ./pkey.gpg
gpg: directory '/root/.gnupg' created
gpg: keybox '/root/.gnupg/pubring.kbx' created
gpg: /root/.gnupg/trustdb.gpg: trustdb created
gpg: key SOMEKEY: public key "Email Name <ci#email.com>" imported
gpg: key SOMEKEY: secret key imported
gpg: Total number processed: 1
gpg: imported: 1
gpg: secret keys read: 1
gpg: secret keys imported: 1
$ git secret reveal -p ${GPG_PASSPHRASE}
gpg: [don't know]: partial length invalid for packet type 20
git-secret: abort: problem decrypting file with gpg: exit code 2: /path/to/decrypted/file
I don't understand where the problem. What mean packet type 20? And length of what?
Locally it revealed fine. Command git secret whoknows shows that email on the ci env can decrypt. Passphrase checked and passed to the job.
For me, the problem was the GnuPG versions being different between the encryption machine (v2.3) and the decryption side (v2.2).
After I downgraded it to v2.2 (due to v2.3 not yet being available on Debian), the problem went away.
This is a common problem with the format of the keys.
Since you're using GitLab CI, you should get advantage of the File type in the CI/CD Variables instead of storing the value of the GPG Key as a Variable type.
First of all, forget about generating the armor in one line with the piped | tr '\n' ',' and get the proper multiline armor.
Second, add it to your GitLab CI Variables with type "File", add an empty line at the end and then delete it (this seems stupid but will save you headaches, since it seems to be a problem when copying directly from the shell to the textbox in GitLab).
Third, import directly the file in your keychain:
gpg --batch --import $GPG_PRIVATE_KEY
The company I work for has a private gitlab server that only supports ssh protocol when cloning a repository.
Inside this server, I have a gitlab-ci.yml file that uses docker executor to run some scripts.
The script's execution fails because it pulls the repository with https at its early stage. It generates this error message: fatal: unable to access 'https://gitlab.mycompany.com/path/to/the/repository/my_repo.git/': SSL certificate problem: unable to get local issuer certificate.
Where can I configure gitlab runner so that it uses ssh to clone the repository?
Here's the full execution log.
Running with gitlab-runner 12.7.1 (003fe500)
on my Group Runner Yh_yL3A2
Using Docker executor with image www.mycompany.com/path/to/the/image:1.0 ...
Pulling docker image www.mycompany.com/path/to/the/image:1.0 ...
Using docker image sha256:474e110ba44ddfje8ncoz4c44e91f2442547281192d4a82b88capmi9047cd8cb for www.mycompany.com/path/to/the/image:1.0 ...
Running on runner-Yh_yL3A2-project-343-concurrent-0 via b55d8c5ba21f...
Fetching changes...
Initialized empty Git repository in /path/to/the/repository/.git/
Created fresh repository.
fatal: unable to access 'https://gitlab.mycompany.com/path/to/the/repository/my_repo.git/': SSL certificate problem: unable to get local issuer certificate
ERROR: Job failed: exit code 1
Here's my .gitlab-ci.yml
image: www.mycompany.com/path/to/the/image:1.0
before_script:
- eval $(ssh-agent -s)
# Reference: https://docs.gitlab.com/ee/ci/ssh_keys/
# Add the SSH key stored in SSH_PRIVATE_KEY variable to the agent store
# We're using tr to fix line endings which makes ed25519 keys work
# without extra base64 encoding.
# https://gitlab.com/gitlab-examples/ssh-private-key/issues/1#note_48526556
#
- echo "$SSH_PRIVATE_KEY" | tr -d '\r' | ssh-add -
#
# Create the SSH directory and give it the right permissions
#
- mkdir -p ~/.ssh
- chmod 700 ~/.ssh
- echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config
stages:
- deploy
deploy:
stage: deploy
tags:
- infra
only:
refs:
- master
script:
- /bin/sh run.sh
I cannot find an option to specify whether the docker executor should use ssh or https to clone the repository.
I have a fairly basic scenario. I made a dedicated ssh key for this purpose and added it to my repository secrets.
Code gets pushed to master
GitHub action uploads it to server using ssh by doing echo "${{ secrets.SSH_KEY }}" > key.
After that I can use this key to connect to my server e.g. ssh -i key devops#myserver.com lsb_release -a
The problem is that for some reason GitHub actions cannot write it to file, it writes characters *** instead of the actual secret value into the file. Therefore obviously I cannot connect to my server.
How can I connect with ssh using this secret? Is there a way to connect without using a file? Can someone who did this common scenario using GitHub actions shed some light?
GitHub Actions should be able to write a secret to a file this way. The reason you see the stars is that the log is filtered, so if a secret would be logged, it's replaced in the log with three asterisks instead. This is a security measure against an accidental disclosure of secrets, since logs are often publicly available.
However, it's a good idea to avoid writing the secret to the log anyway if possible. You can write your command like this so you don't write the secret to the log:
- run: 'echo "$SSH_KEY" > key'
shell: bash
env:
SSH_KEY: ${{secrets.SSH_KEY}}
All you'll see in the log is echo "$SSH_KEY" > key, not the secret or any asterisks.
Note that you do want quotes here, since the > character is special to YAML.
If this doesn't work to log into your server, there's likely a different issue; this technique does work for writing secrets in the general case.
I found a way to do this. We used GatsbyJS for a project and it relies on a
.env.production file for the env variables. I tried to pass them as
env: to the github action, but that didn't work and they were ignored.
Here is what I did. I base 64 encoded the .env.production file:
base64 -i .env.production
Added the output to an env variable in github action. Then in my action I do:
echo ${{ secrets.ENV_PRODUCTION_FILE }} | base64 -d > .env.production
This way the contents of my .env.production file ended being written to the machine that executes the github action.
I used sed in a GHA to replace a TOKEN in a file like this :
run: |-
sed -i "s/TOKEN/${{secrets.MY_SECRET}}/g" "thefile"
The file looks like this:
credentials "app.terraform.io" {
token = "TOKEN"
}
encode it and decode back
- run: 'echo "$SSH_KEY" | base64'
shell: bash
env:
SSH_KEY: ${{ secrets.PRIVATE_KEY }}
and decode it back echo "<encoded string>" | base64 -d
Here is how to solve your actual problem of securely logging into an SSH server using a secret stored in GitHub Actions, named GITHUB_ACTIONS_DEPLOY.
Let's call this "beep", because it will cause an audible bell on the server you login to. Maybe you use this literally ping a server in your house when somebody pushes code to your repo.
- name: Beep
# if: github.ref == 'refs/heads/XXXX' # Maybe limit only this step to some branches
run: |
eval $(ssh-agent)
ssh-add - <<< "$SSH_KEY"
echo "* ssh-rsa XXX" >> /tmp/known_hosts # Get from your local ~/.ssh/known_hosts or from ssh-keyscan
ssh -o UserKnownHostsFile=/tmp/known_hosts user#example.com "echo '\a'"
env:
SSH_KEY: ${{ secrets.PMT_GITHUB_ACTIONS_DEPLOY }}
If, actually you are using SSH as part of a rsync push task, here is how to do that:
- name: Publish
if: github.ref == 'refs/heads/XXX'
run: |
eval $(ssh-agent)
ssh-add - <<< "$SSH_KEY"
echo "* ssh-rsa XXX" >> /tmp/known_hosts
rsync $FROM user#server:
env:
SSH_KEY: ${{ secrets.GITHUB_ACTIONS_DEPLOY }}
RSYNC_RSH: "ssh -o UserKnownHostsFile=/tmp/known_hosts"
You can decode a secret by looping through it with python shell, like this:
- name: Set env as secret
env:
MY_VAL: ${{ secrets.SUPER_SECRET }}
run: |
import os
data = open("file", "w")
for q in (os.getenv("MY_VAL")):
print q
data.write(q)
shell: python
This will both print each character to stdout and store them in file called file. stdout will have an output like this, while file should have the secret string saved inside.
s
e
c
r
e
t
The good solution is to use gpg for encrypting the key, adding it to a repo and decrypting it on the server using passphrase. The passprase should be stored as github project secret of course.
More info how I did it here: https://help.github.com/en/actions/automating-your-workflow-with-github-actions/creating-and-using-encrypted-secrets
echo ${{secrets.AWS_ACCESS_KEY_ID}} | sed 's/./& /g'
I am trying to write a YAML pipeline script to deploy files that have been altered from my bitbucket repository to my remote server using ssh keys. The document that I have in place at the moment was copied from bitbucket itself and has errors:
pipelines:
default:
- step:
name: Deploy to test
deployment: test
script:
- pipe: atlassian/sftp-deploy:0.3.1
- variables:
USER: $USER
SERVER: $SERVER
REMOTE_PATH: $REMOTE_PATH
LOCAL_PATH: $LOCAL_PATH
I am getting the following error
Configuration error
There is an error in your bitbucket-pipelines.yml at [pipelines > default > 0 > step > script > 1]. To be precise: Missing or empty command string. Each item in this list should either be a single command string or a map defining a pipe invocation.
My ssh public and private keys are setup in bitbucket along with the fingerprint and host. The variables have also been setup.
How do I go about setting up my YAML deploy script to connect to my remote server via ssh and transfer the files?
Try to update the variables section become:
- variables:
- USER: $USER
- SERVER: $SERVER
- REMOTE_PATH: $REMOTE_PATH
- LOCAL_PATH: $LOCAL_PATH
Here is am example about how to set variables: https://confluence.atlassian.com/bitbucket/configure-bitbucket-pipelines-yml-792298910.html#Configurebitbucket-pipelines.yml-ci_variablesvariables
Your directive - step has to be intended.
I have bitbucket-pipelines.yml like that (using rsync instead of ssh):
# This is a sample build configuration for PHP.
# Check our guides at https://confluence.atlassian.com/x/e8YWN for more examples.
# Only use spaces to indent your .yml configuration.
# -----
# You can specify a custom docker image from Docker Hub as your build environment.
image: php:7.2.1-fpm
pipelines:
default:
- step:
script:
- apt-get update
- apt-get install zip -y
- apt-get install unzip -y
- apt-get install libgmp3-dev -y
- curl -sS https://getcomposer.org/installer | php -- --install-dir=/usr/local/bin --filename=composer
- composer install
- cp .env.example .env
#- vendor/bin/phpunit
- pipe: atlassian/rsync-deploy:0.2.0
variables:
USER: $DEPLOY_USER
SERVER: $DEPLOY_SERVER
REMOTE_PATH: $DEPLOY_PATH
LOCAL_PATH: '.'
I suggest to use their online editor in repository for editing bitbucket-pipelines.yml, it checks all formal yml structure and you can't commit invalid file.
Even if you check file on some other yaml editor, it may look fine, but not necessary according to bitbucket specification. Their online editor does fine job.
Also, I suggest to visit their community on atlasian community as it's very active, sometimes their staff members are providing answers.
However, I struggle with plenty dependencies needed to run tests properly. (actual bitbucket-pipelines.yml is becoming bigger and bigger).
Maybe there is some nicely prepared Docker image for this job.
I'm trying to automatically deploy my app to digital ocean through bitbucket pipelines. Here are the steps my deployment is following:
connect to the remote digital ocean droplet using ssh
clone my repository by running a git clone with ssh
launch my application with docker-compose
I have successfully setup ssh access to my remote. I have also configured ssh access to my repository and can successfully execute git clone from my remote server.
However, in the pipeline, while connection to the remote server is successfull, the git clone command fails with the following error.
git#bitbucket.org: Permission denied (publickey).
fatal: Could not read from remote repository.
Anybody has an idea of what is going on here?
Here is my bitbucket-pipelines.yml
image: atlassian/default-image:latest
pipelines:
default:
- step:
deployment: production
script:
- cat deploy.sh | ssh $USER_NAME#$HOST
- echo "Deploy step finished"
And the deployment script deploy.sh
#!/usr/bin/env sh
git clone git#bitbucket.org:<username>/<my_repo>.git
cd my_repo
docker-compose up -d
Logs for the git clone ssh commands within the droplet and from the pipeline
Git uses the default ssh key by default.
You can overwrite the SSH command used by git, by setting the GIT_SSH_COMMAND environment variable. You can add the -i argument to use a different SSH key.
export GIT_SSH_COMMAND="ssh -i ~/.ssh/<key>"
git clone git#bitbucket.org:<username>/<my_repo>.git
From the git documentation:
GIT_SSH
GIT_SSH_COMMAND
If either of these environment variables is set then git fetch and git push will use the specified command instead of ssh when they need to connect to a remote system. The command-line parameters passed to the configured command are determined by the ssh variant. See ssh.variant option in git-config[1] for details.
$GIT_SSH_COMMAND takes precedence over $GIT_SSH, and is interpreted by the shell, which allows additional arguments to be included. $GIT_SSH on the other hand must be just the path to a program (which can be a wrapper shell script, if additional arguments are needed).
Usually it is easier to configure any desired options through your personal .ssh/config file. Please consult your ssh documentation for further details.