This is my first ever question at stackoverflow, so I hope it'll adhere to the community guidelines:
I've build a docker image based on an already existing image which has the sole purpose of running duplicity in an container to backup files and folders to an Amazon S3 bucket in Europe.
Duplicity worked for a couple of days when being run manually inside a container resulting from the image. Now I moved on to run containers via unit files on the host with CoreOS and things don't work anymore - but the command also won't work it I run it manually inside a duplicity container..
The run command:
docker run --rm --env-file=<my backup env file>.env --name=<container image> -v <cache container>:/home/duplicity/.cache/duplicity -v <docker volume with gpg keys>:/home/duplicity/.gnupg --volumes-from <docker container of interest> gymnae/duplicity
the env-file contains the following:
PASSPHRASE=<my super secret passphrase>
AWS_ACCESS_KEY_ID=<my aws access key id>
AWS_SECRET_ACCESS_KEY=<my aws access key>
SOURCE_PATH=<where does the data come from>
REMOTE_URL=s3://s3.eu-central-1.amazonaws.com/<my bucket>
PARAMS_CLEAN="--remove-older-than 3M --force --extra-clean"
ENCRYPT_KEY=<derived from the gpg key>
And the init.sh, which is called on docker run, looks like this:
#!/bin/sh
duplicity \
--verbosity 8 \
--s3-use-ia \
--s3-use-new-style \
--s3-use-server-side-encryption \
--s3-european-buckets \
--allow-source-mismatch \
--ssl-no-check-certificate \
--s3-unencrypted-connection \
--volsize 150 \
--gpg-options "--no-tty" \
--encrypt-key $ENCRYPT_KEY \
--sign-key $ENCRYPT_KEY \
$SOURCE_PATH \
$REMOTE_URL
I tried with -i, -it, -t and just -d - but the result is always the same:
===== Begin GnuPG log =====
gpg: using "<supersecret>" as default secret key for signing
gpg: signing failed: Not a tty
gpg: [stdin]: sign+encrypt failed: Not a tty
===== End GnuPG log =====
GPG error detail: Traceback (most recent call last):
File "/usr/bin/duplicity", line 1532, in <module>
with_tempdir(main)
File "/usr/bin/duplicity", line 1526, in with_tempdir
fn()
File "/usr/bin/duplicity", line 1380, in main
do_backup(action)
File "/usr/bin/duplicity", line 1508, in do_backup
incremental_backup(sig_chain)
File "/usr/bin/duplicity", line 662, in incremental_backup
globals.backend)
File "/usr/bin/duplicity", line 425, in write_multivol
at_end = gpg.GPGWriteFile(tarblock_iter, tdp.name, globals.gpg_profile, globals.volsize)
File "/usr/lib/python2.7/site-packages/duplicity/gpg.py", line 356, in GPGWriteFile
file.close()
File "/usr/lib/python2.7/site-packages/duplicity/gpg.py", line 241, in close
self.gpg_failed()
File "/usr/lib/python2.7/site-packages/duplicity/gpg.py", line 226, in gpg_failed
raise GPGError(msg)
GPGError: GPG Failed, see log below:
===== Begin GnuPG log =====
gpg: using "<supersecret>" as default secret key for signing
gpg: signing failed: Not a tty
gpg: [stdin]: sign+encrypt failed: Not a tty
===== End GnuPG log =====
This Not a tty error while gpg tries to sign is weird.
It didn't seem to be a problem before, or I did some crazy typing on a late night shift that it worked once, but now it just doesn't want to work anymore.
For anyone who struggles from the same problem, I found the answer thanks to the developr of duply
https://sourceforge.net/p/ftplicity/bugs/76/#74c5
In short, you need to add GPG_OPTS='--pinentry-mode loopback' starting with gpg 2.1 and add allow-loopback-pinentry to .gnupg/gpg-agent.conf
This brought me much closer to a working setup.
Related
I have a trouble with git secret in the gitlab ci jobs.
What I done:
init, add users, add files, hide them using git secret
create a job where I want to reveal files:
git secret:
stage: init
before_script:
- sh -c "echo 'deb https://gitsecret.jfrog.io/artifactory/git-secret-deb git-secret main' >> /etc/apt/sources.list"
- wget -qO - 'https://gitsecret.jfrog.io/artifactory/api/gpg/key/public' | apt-key add -
- apt-get update && apt-get install -y git-secret
script:
- echo $GPG_PRIVATE_KEY | tr ',' '\n' > ./pkey.gpg
- export GPG_TTY=$(tty)
- gpg --batch --import ./pkey.gpg
- git secret reveal -p ${GPG_PASSPHRASE}
Result logs:
...
$ gpg --batch --import ./pkey.gpg
gpg: directory '/root/.gnupg' created
gpg: keybox '/root/.gnupg/pubring.kbx' created
gpg: /root/.gnupg/trustdb.gpg: trustdb created
gpg: key SOMEKEY: public key "Email Name <ci#email.com>" imported
gpg: key SOMEKEY: secret key imported
gpg: Total number processed: 1
gpg: imported: 1
gpg: secret keys read: 1
gpg: secret keys imported: 1
$ git secret reveal -p ${GPG_PASSPHRASE}
gpg: [don't know]: partial length invalid for packet type 20
git-secret: abort: problem decrypting file with gpg: exit code 2: /path/to/decrypted/file
I don't understand where the problem. What mean packet type 20? And length of what?
Locally it revealed fine. Command git secret whoknows shows that email on the ci env can decrypt. Passphrase checked and passed to the job.
For me, the problem was the GnuPG versions being different between the encryption machine (v2.3) and the decryption side (v2.2).
After I downgraded it to v2.2 (due to v2.3 not yet being available on Debian), the problem went away.
This is a common problem with the format of the keys.
Since you're using GitLab CI, you should get advantage of the File type in the CI/CD Variables instead of storing the value of the GPG Key as a Variable type.
First of all, forget about generating the armor in one line with the piped | tr '\n' ',' and get the proper multiline armor.
Second, add it to your GitLab CI Variables with type "File", add an empty line at the end and then delete it (this seems stupid but will save you headaches, since it seems to be a problem when copying directly from the shell to the textbox in GitLab).
Third, import directly the file in your keychain:
gpg --batch --import $GPG_PRIVATE_KEY
Sometimes I get errors when I download files from a cloud with python-swiftclient, like this one:
Error downloading object 'uploads/1/image.png': Object GET failed: https://orbit.brightbox.com/v1/acc-12345/uploads/1/image.png 500 Internal Error b'An error occurred'
To search for the all errors and re-download failed files I would want to save output of the swift command to a file
I tried to do the following ways:
swift-cli -A https://orbit.brightbox.com/v1/acc-12345 \
-U user -K secret download uploads 2>&1 | tee uploads.log
# and
swift-cli -A https://orbit.brightbox.com/v1/acc-12345 \
-U user -K secret download uploads > uploads.log
But this didn't work. man swift describes -o option
For a single object download, you may use the -o [--output]
option to redirect the output to a specific file or if "-" then just redirect to stdout or with --no-download actually not to write anything to disk.
but when I try to download a directory with -o option if fails with
-o option only allowed for single file downloads
How can I save log to a file when I download a directory with swift CLI?
Actually redirecting output to a file works with swift-client:
swift-cli -A https://orbit.brightbox.com/v1/acc-12345 \
-U user -K secret download uploads > uploads.log
I was confused because after I started the command above, in another terminal window I did
tail -f uploads.log
But it didn't give me any output (like I was seeing when I was running the download command without redirection).
Seems like that swift-client writes to a file in batches and I needed to wait about a minute until tail -f dumps into the console a hundred of lines like this
uploads/documents/1/image.png [auth 0.000s, headers 0.390s, total 14.361s, 0.034 MB/s]
After using gsutil for more than 1 year I suddenly have this error:
.....
At destination listing 8350000...
At destination listing 8360000...
CommandException: Caught non-retryable exception - aborting rsync
.....
I tried to locate the files with this sync problem but I am not able to do so. Is there a "skip error" option of is there a way I can have gsutil more verbose?
My command line is like this:
gsutil -V -m rsync -d -r -U -P -C -e -x -x 'Download/*' /opt/ gs://mybucket1/kraanloos/
I have created a script to split the problem. This gives me more info for a solution
!#/bin/bash
array=(
3ware
AirTime
Amsterdam
BigBag
Download
guide
home
Install
Holding
Multimedia
newsite
Overig
Trak-r
)
for i in "${array[#]}"
do
echo Processing : $i
PROCESS="/usr/bin/gsutil -m rsync -d -r -U -P -C -e -x 'Backup/*' /opt/$i/ gs://mybucket1/kraanloos/$i/"
echo $PROCESS
$PROCESS
echo ""
echo ""
done
I've been struggling with the same problem the last few days. One way to make it super verbose is to put the -D flag before the rsync argument, as in:
gsutil -D rsync ...
By doing that, I found that my problem is due to having # characters in filenames, as in this question.
In my case, it was because of a broken link to a directory.
As blambert said, use the -D option to see exactly what file causes the problem.
I had struggled with this problem as well. I figured it out now.
you need to re-authenticate your Google Cloud SDK Shell and set a target project again.
It seems like rsync will not show the correct error message.
try cp instead, it will guide you to authentic and set the correct primary project
gsutil cp OBJECT_LOCATION gs://DESTINATION_BUCKET_NAME/
after that, your gsutil rsync should run fine.
Thanks to maybeg, I've managed to backup my data from home to an external server. (An amazon one)
As i don't want to backup company datas to Amazon, i tried with an internal backup server.
I then used this command. (I have my own key)
docker run -d --name volumerize
-v /MyFolder/Keys/:/MyFolder/Keys/
-v jenkins_volume:/source:ro
-v backup_volume:/backup
-e 'VOLUMERIZE_SOURCE=/source'
-e "VOLUMERIZE_TARGET=scp://myuser#mybackupserver/home/myuser/"
-e 'VOLUMERIZE_DUPLICITY_OPTIONS=--ssh-options "-i /MyFolder/Keys/myuserkey"'
-e 'PASSPHRASE="mypassphrase"' blacklabelops/volumerize
When using duplicity backup command, inside or outside the container, i have the following error
/usr/lib/python2.7/site-packages/paramiko/ecdsakey.py:200: DeprecationWarning: signer and verifier have been deprecated. Please use sign and verify instead.
signature, ec.ECDSA(self.ecdsa_curve.hash_object())
BackendException: ssh connection to myuser#mybackupserver:22 failed: not a valid OPENSSH private key file
Strangely, inside or outside the volumerize container, the following is running properly.
ssh -i /MyFolder/Keys/myuserkey myuser#mybackupserver
key_load_public: invalid format
Enter passphrase for key '/MyFolder/Keys/myuser':
[myuser#mybackupserver ~]$
Editing backup file for example is giving me the following :
#!/bin/bash
set -o errexit
source /etc/volumerize/stopContainers
duplicity $# --allow-source-mismatch --archive-dir=/volumerize-cache --ssh-options "-i /MyFolder/Keys/myuserkey" /source scp://myuser#mybackupserver/home/myuser/
source /etc/volumerize/startContainers
I've tried to check env variables inside the container, please find below what i have : (Note that passphrase has been added as env variable as found here)
HOSTNAME=b68f0e1a2d45
TERM=xterm
BLACKLABELOPS_HOME=/var/blacklabelops
GOOGLE_DRIVE_CREDENTIAL_FILE=/credentials/googledrive.cred
VOLUMERIZE_HOME=/etc/volumerize
VOLUMERIZE_SOURCE=/source
DOCKERIZE_VERSION=v0.5.0
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/etc/volumerize
VOLUMERIZE_TARGET=scp://myuser#mybackupserver/home/myuser/
PWD=/etc/volumerize
VOLUMERIZE_DUPLICITY_OPTIONS=--ssh-options "-i /MyFolder/Keys/myuserkey"
VOLUMERIZE_CACHE=/volumerize-cache
GPG_TTY=/dev/console
SHLVL=1
HOME=/root
no_proxy=*.local, 169.254/16
GOOGLE_DRIVE_SETTINGS=/credentials/cred.file
PASSPHRASE="mypassphrase"
_=/usr/bin/env
Can someone point me in the right direction ?
Regards,
pierre
Edit1 :
I tried to compare both private key file (Amazon and Company) using
openssl rsa -in yourkey.pem -check and both says
RSA key ok
writing RSA key
-----BEGIN RSA PRIVATE KEY-----
....
-----END RSA PRIVATE KEY-----
Edit2 :
1 . Had a look without any success at duplicity-backendexception
For information, Paramiko version is 2.2.1
Connection is successful using the following python script.
import paramiko
import StringIO
f = open('/MyFolder/Keys/myuserkey','r')
s = f.read()
keyfile = StringIO.StringIO(s)
mykey = paramiko.RSAKey.from_private_key(keyfile,password='mypassphrase')
ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
ssh.connect('mybackupserver',username='mouser',pkey=mykey)
stdin, stdout, stderr = ssh.exec_command('uptime')
stdout.readlines()
[u' 12:35:27 up 3 days, 1:42, 0 users, load average: 1.59, 3.10, 3.00\n']
try the pexpect+scp:// backend (more on available ssh backends can be found in the duplicity manpage http://duplicity.nongnu.org/duplicity.1.html ).
it uses the command line ssh binaries. maybe the error is different or more detailed there?
the error on
ssh -i /MyFolder/Keys/myuserkey myuser#mybackupserver
key_load_public: invalid format
does not seem normal. try to provide the public key in the proper format or not at all.
..ede/duply.net
I have a self-hosted gitlab and I would like to install a package hosted there using ssh.
I tried:
pip install git+ssh://git#<my_domain>:se7entyse7en/<project_name>.git
Here's the output:
Downloading/unpacking git+ssh://git#<my_domain>:se7entyse7en/<project_name>.git
Cloning ssh://git#<my_domain>:se7entyse7en/<project_name>.git to /var/folders/3r/v7swlvdn2p7_wyh9wj90td2m0000gn/T/pip-4_JdRU-build
ssh: Could not resolve hostname <my_domain>:se7entyse7en: nodename nor servname provided, or not known
fatal: Could not read from remote repository.
Please make sure you have the correct access rights and the repository exists.
Update:
I tried to upload it on gitlab.com and after having uploaded the repo I tried to install it by running:
pip install git+ssh://git#gitlab.com:loumarvincaraig/<project_name>.git
but the nothing changed. In particular here's the content of pip.log:
/Users/se7entyse7en/Envs/test/bin/pip run on Mon Nov 17 22:14:51 2014
Downloading/unpacking git+ssh://git#gitlab.com:loumarvincaraig/<project_name>.git
Cloning ssh://git#gitlab.com:loumarvincaraig/<project_name>.git to /var/folders/3r/v7swlvdn2p7_wyh9wj90td2m0000gn/T/pip-91JVFi-build
Found command 'git' at '/usr/local/bin/git'
Running command /usr/local/bin/git clone -q ssh://git#gitlab.com:loumarvincaraig/<project_name>.git /var/folders/3r/v7swlvdn2p7_wyh9wj90td2m0000gn/T/pip-91JVFi-build
Complete output from command /usr/local/bin/git clone -q ssh://git#gitlab.com:loumarvincaraig/<project_name>.git /var/folders/3r/v7swlvdn2p7_wyh9wj90td2m0000gn/T/pip-91JVFi-build:
Cleaning up...
Command /usr/local/bin/git clone -q ssh://git#gitlab.com:loumarvincaraig/<project_name>.git /var/folders/3r/v7swlvdn2p7_wyh9wj90td2m0000gn/T/pip-91JVFi-build failed with error code 128 in None
Exception information:
Traceback (most recent call last):
File "/Users/se7entyse7en/Envs/test/lib/python2.7/site-packages/pip/basecommand.py", line 134, in main
status = self.run(options, args)
File "/Users/se7entyse7en/Envs/test/lib/python2.7/site-packages/pip/commands/install.py", line 236, in run
requirement_set.prepare_files(finder, force_root_egg_info=self.bundle, bundle=self.bundle)
File "/Users/se7entyse7en/Envs/test/lib/python2.7/site-packages/pip/req.py", line 1092, in prepare_files
self.unpack_url(url, location, self.is_download)
File "/Users/se7entyse7en/Envs/test/lib/python2.7/site-packages/pip/req.py", line 1231, in unpack_url
return unpack_vcs_link(link, loc, only_download)
File "/Users/se7entyse7en/Envs/test/lib/python2.7/site-packages/pip/download.py", line 410, in unpack_vcs_link
vcs_backend.unpack(location)
File "/Users/se7entyse7en/Envs/test/lib/python2.7/site-packages/pip/vcs/__init__.py", line 240, in unpack
self.obtain(location)
File "/Users/se7entyse7en/Envs/test/lib/python2.7/site-packages/pip/vcs/git.py", line 111, in obtain
call_subprocess([self.cmd, 'clone', '-q', url, dest])
File "/Users/se7entyse7en/Envs/test/lib/python2.7/site-packages/pip/util.py", line 670, in call_subprocess
% (command_desc, proc.returncode, cwd))
InstallationError: Command /usr/local/bin/git clone -q ssh://git#gitlab.com:loumarvincaraig/<project_name>.git /var/folders/3r/v7swlvdn2p7_wyh9wj90td2m0000gn/T/pip-91JVFi-build failed with error code 128 in None
I don't know why, but by running the following command it worked (slash instead of : after <my_domain>):
pip install git+ssh://git#<my_domain>/se7entyse7en/<project_name>.git
# ^
# slash instead of :
Yes. This is the default use:
pip install git+ssh://git#<my_domain>:22/<project_group>/<project_name>.git
The use of the colon by itself implies the default ssh port number 22. Because you can control the port number of your server, the port number could be different. Git enables customisation by not providing :22/ or / only.