I'm trying to create a EMR cluster version 5.30.0 by using bootstrap script and I got an error messages
Terminated with errors On the master instance (xxxxxxx), bootstrap action 1 returned a non-zero
Anyone know the cause of this problem?
Cluster info
Release label:emr-5.30.0
Hadoop distribution:Amazon 2.8.5
Applications: Hive 2.3.6, Spark 2.4.5, Livy 0.7.0, ZooKeeper 3.4.14
bootstrap script
#!/bin/bash -xe
sudo python3 -m pip install matplotlib pandas spark-nlp
error message
Collecting matplotlib
Downloading https://files.pythonhosted.org/packages/ad/62/7b662284352867a86acfb636761ba351723fc3a235efd8397578d903413d/matplotlib-3.5.3-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.whl (11.2MB)
Collecting pandas
Downloading https://files.pythonhosted.org/packages/99/f0/f99700ef327e51d291efdf4a6de29e685c4d198cbf8531541fc84d169e0e/pandas-1.3.5.tar.gz (4.7MB)
Complete output from command python setup.py egg_info:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/mnt/tmp/pip-build-j8xr47sb/pandas/setup.py", line 18, in <module>
import numpy
ModuleNotFoundError: No module named 'numpy'
I tried to change commands in bootstrap script. such as below.
#!/bin/bash -xe
sudo pip3 install -U \
matplotlib \
pandas
Related
everyone i'm using oracle arm/amd processor servers with ubuntu 22.04
i have searching for an solution for this over 1 week more than 100 hours
but i had no luck to figure out what is the problem
so if you know about this please help me!
what is problem :
here's a simply code with pyperclip
import pyperclip
string = 'testing'
pyperclip.copy(string)
a = pyperclip.paste()
print()
whenever i execute this code directly in the cli , it is okay to run
However whenever i run this code with schedulers like Crontab or Systemd.timer
(xclip && xsel is already installed)
i got this error and cannot run the code like below
Traceback (most recent call last):
File "/home/ubuntu/web_selenium/pytest.py", line 10, in <module>
pyperclip.copy(string)
File "/usr/lib/python3/dist-packages/pyperclip/__init__.py", line 689, in lazy_load_stub_copy
return copy(text)
File "/usr/lib/python3/dist-packages/pyperclip/__init__.py", line 358, in __call__
raise PyperclipException(EXCEPT_MSG)
pyperclip.PyperclipException:
Pyperclip could not find a copy/paste mechanism for your system.
For more information, please visit https://pyperclip.readthedocs.io/en/latest/index.html#not-implemented-error
i tried most of solutions in stackoverflow, mostly it was about setting path for interpreter and packages
of course i installed xclip and xsel as well / everything is good when i run code directly through cli
but only when i run code or shell script through Crontab , Systemd.timer, this error is generated
what i have tried
1. run pyperclip code through shell script with various python env PATH
i tried to lay all the possible env path like /usr/bin:/usr/local/bin .......
and even my python pakages path as .profile and ~./bashrc - failed
2. set new venv and install all the modules, run code - failed
3. run crontab as root - failed
4. install new ubuntu and switch server ubuntu amd -> ubuntu aarch64 - failed
5. reinstall module - failed
6. set shebang inside of python script ( whereis python3 - inside of python file - #!/usr/bin/python3.10) - failed
7. set shell into crontab (SHELL=/bin/bash) - failed
now i'm thinking this is the problem of pyperclip
pleaese help me!
i had meet a problem when i run snakemake in cluster system, "missingoutputfile" and i had searched trying to solve the problem, maybe because it is "run" derived, not "shell", and it is a bug, in the new verision you had solve this problem, when i update it (verison 5.32.0). but another problem is raising up.
Error in rule predict_plasforest:
jobid: 34
output: linear_plasmid_genome/DP-Sample058-S54-adapter-phix-moving-sickle-sss_contigs_1kb.csv
log: log/isolating-linear-contig/DP-Sample058-S54-adapter-phix-moving-sickle-sss_linear_plasforest.out, log/isolating-linear-contig/DP-Sample058-S54-adapter-phix-moving-sickle-sss_linear_plasforest.err (check log file(s) for error message)
conda-env: /home/projects/ku_00041/apps/wanli/F_pipeline/conda_envs/60d0848d
shell:
export PATH=$PATH:/home/projects/ku_00041/apps/wanli/F_pipeline/db/blast/bin;cp /home/projects/ku_00041/apps/wanli/F_pipeline/db/plasforest/plasmid_refseq.* .;cp /home/projects/ku_00041/apps/wanli/F_pipeline/db/plasforest/plasforest.sav .;python3 /home/projects/ku_00041/apps/wanli/F_pipeline/db/plasforest/PlasForest.py -i assmebly_res/DP-Sample058-S54-adapter-phix-moving-sickle-sss_contigs_1kb.fasta -r -b -f --threads 30 -o linear_plasmid_genome/DP-Sample058-S54-adapter-phix-moving-sickle-sss_contigs_1kb.csv 2>log/isolating-linear-contig/DP-Sample058-S54-adapter-phix-moving-sickle-sss_linear_plasforest.err >log/isolating-linear-contig/DP-Sample058-S54-adapter-phix-moving-sickle-sss_linear_plasforest.out
(one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)
cluster_jobid: 30135295
when i see the log file:
Traceback (most recent call last):
File "/home/projects/ku_00041/apps/wanli/F_pipeline/db/plasforest/PlasForest.py", line 26, in <module>
from sklearn.ensemble import RandomForestClassifier
ModuleNotFoundError: No module named 'sklearn'
but in the conda-env, it has already install sklean. and when i activate this env and reun the same command, it is working.
did you know how to solve this problem?
i used tensorflow ver0.12.0. i wanna use ver 0.10 so i run pip install tensorflow
then when i use tensorflow ,
`Traceback (most recent call last):
File "image_zooms_training.py", line 6, in
from keras.models import Sequential
File "/home/satan/anaconda3/envs/py27/lib/python2.7/site-packages/keras/init.py", line 2, in
from . import backend
........
ImportError: libcudnn.so.6: cannot open shared object file: No such file or directory
Failed to load the native TensorFlow runtime.
`
this error is happend .
Please tell me how to solve.
Thank you for your help.
enviroment
ubuntu14.04
python2.7
download libcudnn.so.6 in https://developer.nvidia.com/cudnn.
put it to /usr/local/cuda/lib64/ folder
and use next commond
sudo chmod u=rwx,g=rx,o=rx libcudnn.so.6.5.18
sudo ln -s libcudnn.so.6.5.18 libcudnn.so.6
sudo ln -s libcudnn.so.6 libcudnn.so
Change directory to
/home/username/anaconda3/envs/tensorflow/lib/
DCOS provides the following installation command to install the cli tool:
mkdir -p dcos && cd dcos &&
curl -O https://downloads.dcos.io/dcos-cli/install.sh &&
bash ./install.sh . https://mydomain.co &&
source ./bin/env-setup
However, when I run this on my Mac, it fails with the following:
mkdir -p dcos && cd dcos &&
| → curl -O https://downloads.dcos.io/dcos-cli/install.sh &&
| → bash ./install.sh . https://mydomain.co &&
| → source ./bin/env-setup
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 5163 100 5163 0 0 6460 0 --:--:-- --:--:-- --:--:-- 6469
Installing DC/OS CLI from PyPI...
New python executable in /Users/petr/dcos/bin/python
Installing setuptools, pip, wheel...done.
Traceback (most recent call last):
File "/Users/petr/dcos/bin/dcos", line 7, in <module>
from dcoscli.main import main
File "/Users/petr/dcos/lib/python2.7/site-packages/dcoscli/main.py", line 7, in <module>
from dcos import config, constants, emitting, errors, http, subcommand, util
File "/Users/petr/dcos/lib/python2.7/site-packages/dcos/config.py", line 10, in <module>
from dcos import constants, jsonitem, subcommand, util
File "/Users/petr/dcos/lib/python2.7/site-packages/dcos/subcommand.py", line 12, in <module>
from subprocess import PIPE
ImportError: cannot import name PIPE
When I run the same command on Ubuntu, it works just fine. Any idea what the root cause is?
Most of the issues I've seen around installing the DC/OS CLI on OS X had a root cause related to the prerequisites not met. That said, in DC/OS 1.8 we introduced binaries for the CLI (exactly to avoid situations you're describing).
I have a self-hosted gitlab and I would like to install a package hosted there using ssh.
I tried:
pip install git+ssh://git#<my_domain>:se7entyse7en/<project_name>.git
Here's the output:
Downloading/unpacking git+ssh://git#<my_domain>:se7entyse7en/<project_name>.git
Cloning ssh://git#<my_domain>:se7entyse7en/<project_name>.git to /var/folders/3r/v7swlvdn2p7_wyh9wj90td2m0000gn/T/pip-4_JdRU-build
ssh: Could not resolve hostname <my_domain>:se7entyse7en: nodename nor servname provided, or not known
fatal: Could not read from remote repository.
Please make sure you have the correct access rights and the repository exists.
Update:
I tried to upload it on gitlab.com and after having uploaded the repo I tried to install it by running:
pip install git+ssh://git#gitlab.com:loumarvincaraig/<project_name>.git
but the nothing changed. In particular here's the content of pip.log:
/Users/se7entyse7en/Envs/test/bin/pip run on Mon Nov 17 22:14:51 2014
Downloading/unpacking git+ssh://git#gitlab.com:loumarvincaraig/<project_name>.git
Cloning ssh://git#gitlab.com:loumarvincaraig/<project_name>.git to /var/folders/3r/v7swlvdn2p7_wyh9wj90td2m0000gn/T/pip-91JVFi-build
Found command 'git' at '/usr/local/bin/git'
Running command /usr/local/bin/git clone -q ssh://git#gitlab.com:loumarvincaraig/<project_name>.git /var/folders/3r/v7swlvdn2p7_wyh9wj90td2m0000gn/T/pip-91JVFi-build
Complete output from command /usr/local/bin/git clone -q ssh://git#gitlab.com:loumarvincaraig/<project_name>.git /var/folders/3r/v7swlvdn2p7_wyh9wj90td2m0000gn/T/pip-91JVFi-build:
Cleaning up...
Command /usr/local/bin/git clone -q ssh://git#gitlab.com:loumarvincaraig/<project_name>.git /var/folders/3r/v7swlvdn2p7_wyh9wj90td2m0000gn/T/pip-91JVFi-build failed with error code 128 in None
Exception information:
Traceback (most recent call last):
File "/Users/se7entyse7en/Envs/test/lib/python2.7/site-packages/pip/basecommand.py", line 134, in main
status = self.run(options, args)
File "/Users/se7entyse7en/Envs/test/lib/python2.7/site-packages/pip/commands/install.py", line 236, in run
requirement_set.prepare_files(finder, force_root_egg_info=self.bundle, bundle=self.bundle)
File "/Users/se7entyse7en/Envs/test/lib/python2.7/site-packages/pip/req.py", line 1092, in prepare_files
self.unpack_url(url, location, self.is_download)
File "/Users/se7entyse7en/Envs/test/lib/python2.7/site-packages/pip/req.py", line 1231, in unpack_url
return unpack_vcs_link(link, loc, only_download)
File "/Users/se7entyse7en/Envs/test/lib/python2.7/site-packages/pip/download.py", line 410, in unpack_vcs_link
vcs_backend.unpack(location)
File "/Users/se7entyse7en/Envs/test/lib/python2.7/site-packages/pip/vcs/__init__.py", line 240, in unpack
self.obtain(location)
File "/Users/se7entyse7en/Envs/test/lib/python2.7/site-packages/pip/vcs/git.py", line 111, in obtain
call_subprocess([self.cmd, 'clone', '-q', url, dest])
File "/Users/se7entyse7en/Envs/test/lib/python2.7/site-packages/pip/util.py", line 670, in call_subprocess
% (command_desc, proc.returncode, cwd))
InstallationError: Command /usr/local/bin/git clone -q ssh://git#gitlab.com:loumarvincaraig/<project_name>.git /var/folders/3r/v7swlvdn2p7_wyh9wj90td2m0000gn/T/pip-91JVFi-build failed with error code 128 in None
I don't know why, but by running the following command it worked (slash instead of : after <my_domain>):
pip install git+ssh://git#<my_domain>/se7entyse7en/<project_name>.git
# ^
# slash instead of :
Yes. This is the default use:
pip install git+ssh://git#<my_domain>:22/<project_group>/<project_name>.git
The use of the colon by itself implies the default ssh port number 22. Because you can control the port number of your server, the port number could be different. Git enables customisation by not providing :22/ or / only.