I try to do full backup on Vertica database. When I execute command:
/opt/vertica/bin/vbr.py --debug 3 --task backup --config-file vertica_backup.ini
I am getting following error:
/bin/sh: 1: ulimit: Illegal option -u
Traceback (most recent call last):
File "/opt/vertica/bin/vbr.py", line 2526, in backup
prepareAll()
File "/opt/vertica/bin/vbr.py", line 1888, in prepareAll
configCheck()
File "/opt/vertica/bin/vbr.py", line 506, in configCheck
concurrency_upperboud = int(subprocess.Popen(['ulimit -u'],
shell=True, stdout= subprocess.PIPE).communicate()[0].strip())
ValueError: invalid literal for int() with base 10: ''
backup failed unexpectedly!
My vertica_backup.ini file:
[Misc]
snapshotName = vertica_backup
restorePointLimit = 1
passwordFile = vertica
[Database]
dbName = dwh_vertica
dbUser = dbadmin
[Transmission]
[Mapping]
v_dwh_vertica_node0001 = vertica1:/home/dbadmin/backups
It's Debian Wheezy:
Linux vertica1 3.16.0-0.bpo.4-amd64 #1 SMP Debian 3.16.7-ckt11-1~bpo70+1 (2015-06-08) x86_64 GNU/Linux
/bin/sh points to /bin/sh -> dash
I've found workaround, in the Vertica backup file /opt/vertica/bin/vbr.py I have changed ulimit -u to ulimit -n.
Related
I have an scp function as follows:
def scp_to38():
child = pexpect.spawn(f"scp {tar_file_path} bbmprd#10.82.20.38:/app2/upload/")
child.expect(f"bbmprd#10.82.20.38's password:")
child.sendline("password")
child.interact()
print(f'{tar_file_path} SENT TO 38:/app2/upload/')
and it is throwing me an error saying a problem with child.interact()
Traceback (most recent call last):
File "scan_for_upload.py", line 39, in <module>
scp_to38()
File "scan_for_upload.py", line 26, in scp_to38
child.interact()
File "/app/anaconda3/envs/python37-1/lib/python3.7/site-packages/pexpect/pty_spawn.py", line 788, in interact
mode = tty.tcgetattr(self.STDIN_FILENO)
termios.error: (25, 'Inappropriate ioctl for device')
I am running the script in a nohup process, I believe that is causing this issue. Because when I run the script in the console, it works fine. How do I interact with the scp password in a nohup process using pexpect module? (I can't install any other external modules in my environment)
interact() is for manual interaction with the spawned process and it requires to be running on a tty but nohup provides no tty. You can just replace child.interact() with child.expect(pexpect.EOF, timeout=None).
i had meet a problem when i run snakemake in cluster system, "missingoutputfile" and i had searched trying to solve the problem, maybe because it is "run" derived, not "shell", and it is a bug, in the new verision you had solve this problem, when i update it (verison 5.32.0). but another problem is raising up.
Error in rule predict_plasforest:
jobid: 34
output: linear_plasmid_genome/DP-Sample058-S54-adapter-phix-moving-sickle-sss_contigs_1kb.csv
log: log/isolating-linear-contig/DP-Sample058-S54-adapter-phix-moving-sickle-sss_linear_plasforest.out, log/isolating-linear-contig/DP-Sample058-S54-adapter-phix-moving-sickle-sss_linear_plasforest.err (check log file(s) for error message)
conda-env: /home/projects/ku_00041/apps/wanli/F_pipeline/conda_envs/60d0848d
shell:
export PATH=$PATH:/home/projects/ku_00041/apps/wanli/F_pipeline/db/blast/bin;cp /home/projects/ku_00041/apps/wanli/F_pipeline/db/plasforest/plasmid_refseq.* .;cp /home/projects/ku_00041/apps/wanli/F_pipeline/db/plasforest/plasforest.sav .;python3 /home/projects/ku_00041/apps/wanli/F_pipeline/db/plasforest/PlasForest.py -i assmebly_res/DP-Sample058-S54-adapter-phix-moving-sickle-sss_contigs_1kb.fasta -r -b -f --threads 30 -o linear_plasmid_genome/DP-Sample058-S54-adapter-phix-moving-sickle-sss_contigs_1kb.csv 2>log/isolating-linear-contig/DP-Sample058-S54-adapter-phix-moving-sickle-sss_linear_plasforest.err >log/isolating-linear-contig/DP-Sample058-S54-adapter-phix-moving-sickle-sss_linear_plasforest.out
(one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)
cluster_jobid: 30135295
when i see the log file:
Traceback (most recent call last):
File "/home/projects/ku_00041/apps/wanli/F_pipeline/db/plasforest/PlasForest.py", line 26, in <module>
from sklearn.ensemble import RandomForestClassifier
ModuleNotFoundError: No module named 'sklearn'
but in the conda-env, it has already install sklean. and when i activate this env and reun the same command, it is working.
did you know how to solve this problem?
This is my first ever question at stackoverflow, so I hope it'll adhere to the community guidelines:
I've build a docker image based on an already existing image which has the sole purpose of running duplicity in an container to backup files and folders to an Amazon S3 bucket in Europe.
Duplicity worked for a couple of days when being run manually inside a container resulting from the image. Now I moved on to run containers via unit files on the host with CoreOS and things don't work anymore - but the command also won't work it I run it manually inside a duplicity container..
The run command:
docker run --rm --env-file=<my backup env file>.env --name=<container image> -v <cache container>:/home/duplicity/.cache/duplicity -v <docker volume with gpg keys>:/home/duplicity/.gnupg --volumes-from <docker container of interest> gymnae/duplicity
the env-file contains the following:
PASSPHRASE=<my super secret passphrase>
AWS_ACCESS_KEY_ID=<my aws access key id>
AWS_SECRET_ACCESS_KEY=<my aws access key>
SOURCE_PATH=<where does the data come from>
REMOTE_URL=s3://s3.eu-central-1.amazonaws.com/<my bucket>
PARAMS_CLEAN="--remove-older-than 3M --force --extra-clean"
ENCRYPT_KEY=<derived from the gpg key>
And the init.sh, which is called on docker run, looks like this:
#!/bin/sh
duplicity \
--verbosity 8 \
--s3-use-ia \
--s3-use-new-style \
--s3-use-server-side-encryption \
--s3-european-buckets \
--allow-source-mismatch \
--ssl-no-check-certificate \
--s3-unencrypted-connection \
--volsize 150 \
--gpg-options "--no-tty" \
--encrypt-key $ENCRYPT_KEY \
--sign-key $ENCRYPT_KEY \
$SOURCE_PATH \
$REMOTE_URL
I tried with -i, -it, -t and just -d - but the result is always the same:
===== Begin GnuPG log =====
gpg: using "<supersecret>" as default secret key for signing
gpg: signing failed: Not a tty
gpg: [stdin]: sign+encrypt failed: Not a tty
===== End GnuPG log =====
GPG error detail: Traceback (most recent call last):
File "/usr/bin/duplicity", line 1532, in <module>
with_tempdir(main)
File "/usr/bin/duplicity", line 1526, in with_tempdir
fn()
File "/usr/bin/duplicity", line 1380, in main
do_backup(action)
File "/usr/bin/duplicity", line 1508, in do_backup
incremental_backup(sig_chain)
File "/usr/bin/duplicity", line 662, in incremental_backup
globals.backend)
File "/usr/bin/duplicity", line 425, in write_multivol
at_end = gpg.GPGWriteFile(tarblock_iter, tdp.name, globals.gpg_profile, globals.volsize)
File "/usr/lib/python2.7/site-packages/duplicity/gpg.py", line 356, in GPGWriteFile
file.close()
File "/usr/lib/python2.7/site-packages/duplicity/gpg.py", line 241, in close
self.gpg_failed()
File "/usr/lib/python2.7/site-packages/duplicity/gpg.py", line 226, in gpg_failed
raise GPGError(msg)
GPGError: GPG Failed, see log below:
===== Begin GnuPG log =====
gpg: using "<supersecret>" as default secret key for signing
gpg: signing failed: Not a tty
gpg: [stdin]: sign+encrypt failed: Not a tty
===== End GnuPG log =====
This Not a tty error while gpg tries to sign is weird.
It didn't seem to be a problem before, or I did some crazy typing on a late night shift that it worked once, but now it just doesn't want to work anymore.
For anyone who struggles from the same problem, I found the answer thanks to the developr of duply
https://sourceforge.net/p/ftplicity/bugs/76/#74c5
In short, you need to add GPG_OPTS='--pinentry-mode loopback' starting with gpg 2.1 and add allow-loopback-pinentry to .gnupg/gpg-agent.conf
This brought me much closer to a working setup.
I have a self-hosted gitlab and I would like to install a package hosted there using ssh.
I tried:
pip install git+ssh://git#<my_domain>:se7entyse7en/<project_name>.git
Here's the output:
Downloading/unpacking git+ssh://git#<my_domain>:se7entyse7en/<project_name>.git
Cloning ssh://git#<my_domain>:se7entyse7en/<project_name>.git to /var/folders/3r/v7swlvdn2p7_wyh9wj90td2m0000gn/T/pip-4_JdRU-build
ssh: Could not resolve hostname <my_domain>:se7entyse7en: nodename nor servname provided, or not known
fatal: Could not read from remote repository.
Please make sure you have the correct access rights and the repository exists.
Update:
I tried to upload it on gitlab.com and after having uploaded the repo I tried to install it by running:
pip install git+ssh://git#gitlab.com:loumarvincaraig/<project_name>.git
but the nothing changed. In particular here's the content of pip.log:
/Users/se7entyse7en/Envs/test/bin/pip run on Mon Nov 17 22:14:51 2014
Downloading/unpacking git+ssh://git#gitlab.com:loumarvincaraig/<project_name>.git
Cloning ssh://git#gitlab.com:loumarvincaraig/<project_name>.git to /var/folders/3r/v7swlvdn2p7_wyh9wj90td2m0000gn/T/pip-91JVFi-build
Found command 'git' at '/usr/local/bin/git'
Running command /usr/local/bin/git clone -q ssh://git#gitlab.com:loumarvincaraig/<project_name>.git /var/folders/3r/v7swlvdn2p7_wyh9wj90td2m0000gn/T/pip-91JVFi-build
Complete output from command /usr/local/bin/git clone -q ssh://git#gitlab.com:loumarvincaraig/<project_name>.git /var/folders/3r/v7swlvdn2p7_wyh9wj90td2m0000gn/T/pip-91JVFi-build:
Cleaning up...
Command /usr/local/bin/git clone -q ssh://git#gitlab.com:loumarvincaraig/<project_name>.git /var/folders/3r/v7swlvdn2p7_wyh9wj90td2m0000gn/T/pip-91JVFi-build failed with error code 128 in None
Exception information:
Traceback (most recent call last):
File "/Users/se7entyse7en/Envs/test/lib/python2.7/site-packages/pip/basecommand.py", line 134, in main
status = self.run(options, args)
File "/Users/se7entyse7en/Envs/test/lib/python2.7/site-packages/pip/commands/install.py", line 236, in run
requirement_set.prepare_files(finder, force_root_egg_info=self.bundle, bundle=self.bundle)
File "/Users/se7entyse7en/Envs/test/lib/python2.7/site-packages/pip/req.py", line 1092, in prepare_files
self.unpack_url(url, location, self.is_download)
File "/Users/se7entyse7en/Envs/test/lib/python2.7/site-packages/pip/req.py", line 1231, in unpack_url
return unpack_vcs_link(link, loc, only_download)
File "/Users/se7entyse7en/Envs/test/lib/python2.7/site-packages/pip/download.py", line 410, in unpack_vcs_link
vcs_backend.unpack(location)
File "/Users/se7entyse7en/Envs/test/lib/python2.7/site-packages/pip/vcs/__init__.py", line 240, in unpack
self.obtain(location)
File "/Users/se7entyse7en/Envs/test/lib/python2.7/site-packages/pip/vcs/git.py", line 111, in obtain
call_subprocess([self.cmd, 'clone', '-q', url, dest])
File "/Users/se7entyse7en/Envs/test/lib/python2.7/site-packages/pip/util.py", line 670, in call_subprocess
% (command_desc, proc.returncode, cwd))
InstallationError: Command /usr/local/bin/git clone -q ssh://git#gitlab.com:loumarvincaraig/<project_name>.git /var/folders/3r/v7swlvdn2p7_wyh9wj90td2m0000gn/T/pip-91JVFi-build failed with error code 128 in None
I don't know why, but by running the following command it worked (slash instead of : after <my_domain>):
pip install git+ssh://git#<my_domain>/se7entyse7en/<project_name>.git
# ^
# slash instead of :
Yes. This is the default use:
pip install git+ssh://git#<my_domain>:22/<project_group>/<project_name>.git
The use of the colon by itself implies the default ssh port number 22. Because you can control the port number of your server, the port number could be different. Git enables customisation by not providing :22/ or / only.
I try to restore single file or dir by duplicity from amazon s3, but but I get an errors.
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: none
Traceback (most recent call last):
File "/usr/bin/duplicity", line 1251, in <module>
with_tempdir(main)
File "/usr/bin/duplicity", line 1244, in with_tempdir
fn()
File "/usr/bin/duplicity", line 1198, in main
restore(col_stats)
File "/usr/bin/duplicity", line 538, in restore
restore_get_patched_rop_iter(col_stats)):
File "/usr/bin/duplicity", line 560, in restore_get_patched_rop_iter
backup_chain = col_stats.get_backup_chain_at_time(time)
File "/usr/lib/python2.6/dist-packages/duplicity/collections.py", line 934, in get_backup_chain_at_time
raise CollectionsError("No backup chains found")
CollectionsError: No backup chains found
What I do wrong?
Here how I doing backups
export PASSPHRASE=****
export AWS_ACCESS_KEY_ID=****
export AWS_SECRET_ACCESS_KEY=****
GPG_KEY=****
BACKUP_SIM_RUN=1
LOGFILE="/var/log/s3-backup.log"
DAILYLOGFILE="/var/log/s3-backup-daily.log"
# The source of your backup
SOURCE=/home/u54433
# The destination
DEST=s3+http://**********
trace () {
stamp=`date +%Y-%m-%d_%H:%M:%S`
echo "$stamp: $*" >> ${DAILYLOGFILE}
}
cat /dev/null > ${DAILYLOGFILE}
trace "removing old backups..."
duplicity remove-older-than 2M --force --sign-key=${GPG_KEY} ${DEST} >> ${DAILYLOGFILE} 2>&1
trace "start backup files..."
duplicity --sign-key=${GPG_KEY} --exclude="**/logs" --s3-european-buckets --s3-use-new-style ${SOURCE} ${DEST} >> ${DAILYLOGFILE} 2>&1
cat "$DAILYLOGFILE" >> $LOGFILE
export PASSPHRASE=
export AWS_ACCESS_KEY_ID=
export AWS_SECRET_ACCESS_KEY=
Use --s3-use-new-style option in all duplicity calls.
I had the same problem as you did. I added the missing option to "duplicity remove-older-than" and everything works great now.
It is better to remove the S3 bucket form amazon and try to recreate full backup this might resolve the issue.
Also
You can see the below link
https://answers.launchpad.net/duplicity/+question/107074
For anyone coming back to this question looking for a definitive answer, #shaikh-systems link leads to the realization that there is some issue in Duplicity/AWS communication of IAM sub-account keys. To restore, I got it to work by using/exporting my primary account key/secret. I'm using duplicity 0.6.21.