I am trying to install tensorflow on in a Docker Container on a RB Pi Zero.
I get some weird hash-error, see below. What can I try next?
root#123456:/# sudo pip3 install --no-cache-dir tensorflow
Collecting tensorflow
Downloading https://www.piwheels.org/simple/tensorflow/tensorflow-1.14.0-cp35-none-linux_armv6l.whl (94.2MB)
100% |████████████████████████████████| 94.2MB 888kB/s
THESE PACKAGES DO NOT MATCH THE HASHES FROM THE REQUIREMENTS FILE. If you have updated the package versions, please update the hashes. Otherwise, examine the package contents carefully; someone may have tampered with them.
tensorflow from https://www.piwheels.org/simple/tensorflow/tensorflow-1.14.0-cp35-none-linux_armv6l.whl#sha256=cba22b6d9a3e7a92c07e142bd5256c9773fd20c18090cb1d222357d3b3028655:
Expected sha256 cba22b6d9a3e7a92c07e142bd5256c9773fd20c18090cb1d222357d3b3028655
Got 65c83ef17cd950cf40d021070f3e7e1fa99499a99815c15495920ddc3440a98f
No space issues:
root#123456:/# df -h /
Filesystem Size Used Avail Use% Mounted on
overlay 29G 9.8G 19G 36% /
I've tried to remove files in 'rm -rf /var/lib/apt/lists/partial' and performed apt-get update&upgrade, but with same result. What can I try next?
Update
The hashes were fixed, so the hash mismatch warning should be gone now.
Original answer
Unfortunately, this is a common issue with https://www.piwheels.org when large wheels are uploaded. If you take a close look at the simple URLs of tensorflow wheels, you'll notice that the wheels
tensorflow-1.14.0-cp36-none-linux_armv7l.whl
tensorflow-1.14.0-cp36-none-linux_armv6l.whl
tensorflow-1.14.0-cp35-none-linux_armv7l.whl
tensorflow-1.14.0-cp35-none-linux_armv6l.whl
tensorflow-1.14.0-cp34-none-linux_armv7l.whl
tensorflow-1.14.0-cp34-none-linux_armv6l.whl
all have the same sha256 hash in download links, which means the hashes are just wrong. The workaround is to download the wheels to and install from disk:
$ wget https://www.piwheels.org/simple/tensorflow/tensorflow-1.14.0-cp35-none-linux_armv6l.whl
$ pip install tensorflow-1.14.0-cp35-none-linux_armv6l.whl
I have also reported wrong hashes here, so the issue will be fixed sooner or later.
Related
I try to convert caffe model. I am using coremltools v5.
this is my code
import coremltools
caffe_model = ('oxford102.caffemodel', 'deploy.prototxt')
labels = 'flower-labels.txt'
coreml_model = coremltools.converters.caffe.convert(
caffe_model,
class_labels=labels,
image_input_names='data'
)
coreml_model.save('FlowerClassifier.mlmodel')
I convert using below command
python3 convert-script.py
And i get an error message like below.
error message
Does anybody face this problem and have solution on it?
I just came across this as I was having the same problem. The caffe support is not available in the newer versions of coremltools API. To make this code run an older version of coremltools (such as 3.4) must be used, which requires using Python 2.7 - which is best done in a virtual environment.
I assume you've solved your issue already, but I added this in case anyone else stumbles onto this question.
There are several solutions according to your case:
I had the same issue on my M1 Mac. You can resolve the same by duplicating your Terminal, and running it with Rosetta.(This worked for me)
cd ~/.virtualenvs/<your venv name here>/bin
mkdir bk; cp python bk; mv -f bk/python .;rmdir bk
codesign -s - --preserve-metadata=identifier,entitlements,flags,runtime -f python
Fore more solutions and issue you can watch this issue on github
I had the same error running python 3.7
In the virtualenv, solution is to run:
pip install coremltools==3.0
Don't have to change python versions and just rerun the script
Original question:
I am installing virtualenvwrapper on Ubuntu 18.04. Here is what I have tried so far:
From https://virtualenvwrapper.readthedocs.io/en/latest/#:~:text=virtualenvwrapper%20is%20a%20set%20of,introducing%20conflicts%20in%20their%20dependencies:
$ pip install virtualenvwrapper
...
$ export WORKON_HOME=~/Envs
$ mkdir -p $WORKON_HOME
$ source /usr/local/bin/virtualenvwrapper.sh
Error: bash: /usr/local/bin/virtualenvwrapper.sh: No such file or directory
Ok, so I went looking for virtualenvwrapper.sh. Eventually I found it:
joanna#joanna-X441BA:~/.local/bin$ ls
...
virtualenv-clone
virtualenvwrapper_lazy.sh
virtualenvwrapper.sh
...
Tried again with the new path: joanna#joanna-X441BA:/$ source /home/.local/bin/virtualenvwrapper.sh
bash: /home/.local/bin/virtualenvwrapper.sh: No such file or directory
Agh, ok. Searched StackOverflow, followed the instructions in Issue installing Virtualenvwrapper on Ubuntu 18.04?:
export WORKON_HOME=$HOME/.virtualenv
export VIRTUALENVWRAPPER_LOG_DIR="$WORKON_HOME"
export VIRTUALENVWRAPPER_HOOK_DIR="$WORKON_HOME"
export VIRTUALENVWRAPPER_PYTHON=/usr/bin/python3
I checked that this is in fact where my python3 installation lives. Same result though - No such file or directory.
Also tried sudo apt-get update and it successfully updated a bunch of stuff. But still No such file or directory.
Following this article, I also tried
export VIRTUALENVWRAPPER_PYTHON=/usr/local/bin/python
export VIRTUALENVWRAPPER_VIRTUALENV=/usr/local/bin/virtualenv
(with the paths corrected for my filesystem setup.)
I don't understand why Bash is saying "No such file or directory" when it is in fact there (inexact error message that could be improved?), or more importantly how to fix this problem.
Note: This article warns me not to use sudo with pip because "If you think you need to use sudo, you're probably trying to modify a distribution-owned file", which is apparently Very Bad. I have also seen several articles warning that I should really use python -m pip install (or python3 -m pip install?) instead of plain pip install, because plain pip install can cause unintended side effects. I am not an expert by any means in these matters, but avoiding side effects sounds good to me.
Solution:
I finally got it to work! I kept playing around with it and double-checking that all the paths were correct and I did everything in the right order. I also had a stray installation of Python 3.8 that it seems I had installed in one of my folders that I use for code (my understanding is that Python installations should automatically go into one of the root folders like /usr/bin). On advice from my mentor I used the file manager to delete the stray Python 3.8, which may have helped with the virtualenvwrapper problem. After deleting the stray Python 3.8 I ran these commands and it finally worked!!!!
export WORKON_HOME=$HOME/.virtualenv
export VIRTUALENVWRAPPER_LOG_DIR="$WORKON_HOME"
export VIRTUALENVWRAPPER_HOOK_DIR="$WORKON_HOME"
export VIRTUALENVWRAPPER_PYTHON=/usr/bin/python3
source ~/.local/bin/virtualenvwrapper.sh
My advice to readers of this question who are stumped with similar problems: triple-check the paths you are using by cd-ing into the folders, and also check for spelling errors in what you are typing. Your computer's file system may be set up slightly different from mine so don't blindly copy-paste the file paths I used. This will save you a lot of time and frustration. Also note that the file location of ~ is NOT the same as /home, that's one mistake I made.
I am trying to set up PyBOSSA on an AWS EC2 instance running Ubuntu 18.04 LTS. I am following the official instructions and have encountered three errors so far.
sudo apt-get install -y git postgresql postgresql-all postgresql-server-dev-all libpq-dev python-psycopg2 libsasl2-dev libldap2-dev libssl-dev python-virtualenv python-dev build-essential libjpeg-dev libssl-dev libffi-dev dbus libdbus-1-dev libdbus-glib-1-dev libldap2-dev libsasl2-dev python-pip python3-pip redis-server
cd ~
git clone --recursive https://github.com/Scifabric/pybossa
cd pybossa
virtualenv -p python3 env (I'm using Python3 explicitly as my system also has Python 2.7 installed).
source env/bin/activate
pip install -U pip
pip install -r ~/pybossa/requirements.txt
At this point, I start getting error messages... I have copied the stdout and stderr into a file, which I have uploaded here.
I'm not sure if the errors there are what have caused my later errors, but I pushed on through the instructions anyway in hopes it'd work...
cp settings_local.py.tmpl settings_local.py
cp alembic.ini.template alembic.ini
redis-server contrib/sentinel.conf --sentinel
I noted that the Redis server version was 4.0.9 (the instructions say it needs to be v2.6 or greater).
The output from starting the Redis server was as follows:
30284:X 30 Mar 03:09:22.004 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
30284:X 30 Mar 03:09:22.004 # Redis version=4.0.9, bits=64, commit=00000000, modified=0, pid=30284, just started
30284:X 30 Mar 03:09:22.004 # Configuration loaded
...I gather that's ok...
rqscheduler --host 127.0.0.1
This command wasn't installed on my system. I tried to use apt to install it, but there was nothing there. I also tried apt install rq rqscheduler rq-scheduler - nothing found. I then Googled and found the website for rq-scheduler, and found that I could install it by running pip install rq-scheduler
That installed correctly. Nonetheless, running the command rqscheduler --host 127.0.0.1 in the terminal still failed: rqscheduler: command not found.
Knowing that it was a Python package, I wondered if maybe I needed to prepend python3 onto the start of the command: python3 rqscheduler --host 127.0.0.1. Response: python3: can't open file 'rqscheduler': [Errno 2] No such file or directory.
I also tried pip3 install rq-scheduler (which installed fine) and then running the command, but encountered the same error.
I would appreciate knowing how to get that running, but for the purpose of this test, I skipped setting up Regis and the scheduler, and continued with the PyBOSSA instructions:
sudo su postgres
createuser -d -P pybossa
(Password set)
createdb pybossa -O pybossa
exit
python3 cli.py db_create
...and then I got this error:
File "cli.py", line 162
'''SELECT id, created FROM task_run WHERE created LIKE ('\x%')''')
^
SyntaxError: (unicode error) 'unicodeescape' codec can't decode bytes in position 54-55: truncated \xXX escape
I instead tried python cli.py db_create, just in case it'd work, and got a different error:
python cli.py db_create
ValueError: invalid \x escape
So I'm seeing three separate issues:
Installing the PyBOSSA-required Python packages.
The issue with the rqscheduler command.
The error when starting the PyBOSSA server.
What do these errors mean?
1 ) For the installation, try this:
virtualenv env
source env/bin/activate
sudo apt install python3-pip
pip3 install -r requirements.txt
Which ended with no error.
2) Try :
pip install rq-scheduler==0.9.1
or
pip3 install rq-scheduler==0.9.1
3) The \ char need to be escaped (like \\) in python.
So you may alter the cli.py line 162 (using text editor) from:
'''SELECT id, created FROM task_run WHERE created LIKE ('\x%')''')
To:
'''SELECT id, created FROM task_run WHERE created LIKE ('\\x%')''')
But it will be better to be fixed by dev on github ...
CONCLUSION
According to official documentation,
PYBOSSA for python 3 We’ve finally migrated PYBOSSA to python 3. We’re
not going to merge into master until we test it in production a bit
more, so please, help us by testing it. All you have to do is
basically, check out the python3 branch (migrate-python3) and run it.
Then, any bug, issue you find, you just report it and we will be happy
to help you.
The PYBOSSA python3 version is freshly migrated so finaly is not very stable ... I expect that it will be better to use the PYBOSSA python2.7 branch and follow exactly the documentation.
And according to official github account they try to make money with support (?...)
Get professional support You can hire us to help you with your PYBOSSA
project or server (specially for python 2.7). Go to our website, and
contact us.
The issue has now been fixed for the master branch (https://github.com/Scifabric/pybossa/pull/1986). You can fetch the new code and use it.
This is about a tensorboard which is built from source, not about pip-installed one.
I could successfully build it.
$ git clone https://github.com/tensorflow/tensorboard.git
$ cd tensorboard/
$ bazel build //tensorboard
tensorflow/tensorboard$ bazel build //tensorboard
Starting local Bazel server and connecting to it...
......................................
: (log messages here)
Target //tensorboard:tensorboard up-to-date:
bazel-bin/tensorboard/tensorboard
INFO: Elapsed time: 326.553s, Critical Path: 187.92s
INFO: 619 processes: 456 linux-sandbox, 12 local, 151 worker.
INFO: Build completed successfully, 1268 total actions
Then yes I can run it as documented in tensorboard/README.md, and it works.
$ ./bazel-bin/tensorboard/tensorboard --logdir path/to/logs
The problem is, I'd like to run it as if installed via pip like this:
$ tensorboard --logdir path/to/logs
But as far as I looked for, no script provided to create .whl file so that we can locally-pip-install it, unlike tensorflow provides one like this.
$ bazel-bin/tensorflow/tools/pip_package/build_pip_package /tmp/tensorflow_pkg
$ sudo pip install /tmp/tensorflow_pkg/tensorflow-1.8.0-py2-none-any.whl
So... can anybody show how to do that? Making packaging script would solve this, but it should exist somewhere as long as tensorboard is provided via pip anyway. :)
My workaround so far is not clean enough:
$ ln -s /my/build/folder/tensorboard/bazel-bin/tensorboard/tensorboard ~/bin
$ ln -s /my/build/folder/tensorboard/bazel-bin/tensorboard/tensorboard.runfiles ~/bin
I appreciate your suggestions, thanks!
Update July-21:
Thanks to W JC, I found instruction is already there in tensorboard/pip_package/BUILD.
# rm -rf /tmp/tensorboard
# bazel run //tensorboard/pip_package:build_pip_package
# pip install -U /tmp/tensorboard/*py2*.pip
Though unfortunately it shows error in my environment, and I guess it's local issue maybe because I'm using anaconda.
But basically the problem was resolved. It should basically work as long as running on supported environment.
It seems there exists an script in the /tensorboard/pip_packages try to build wheels
bazel run //tensorboard/pip_package:build_pip_package ./ did generate the wheel out, but in the folder where bazel-bin points to. In my case, it's generated at ~/.cache/bazel/_bazel_peijia/b64ba42719633ff75eec6880decefcd3/execroot/org_tensorflow_tensorboard/bazel-out/k8-fastbuild/bin/tensorboard/pip_package/build_pip_package.runfiles/org_tensorflow_tensorboard/tensorboard-2.10.0a0-py3-none-any.whl
PROBLEM: I am attempting to run a spark-submit script from my local machine to a cluster of machines. The work done by the cluster uses numpy. I currently get the following error:
ImportError:
Importing the multiarray numpy extension module failed. Most
likely you are trying to import a failed build of numpy.
If you're working with a numpy git repo, try `git clean -xdf` (removes all
files not under version control). Otherwise reinstall numpy.
Original error was: cannot import name multiarray
DETAIL:
In my local environment I have setup a virtualenv that includes numpy as well as a private repo I use in my project and other various libraries. I created a zip file (lib/libs.zip) from the site-packages directory at venv/lib/site-packages where 'venv' is my virtual environment. I ship this zip to the remote nodes. My shell script for performing the spark-submit looks like this:
$SPARK_HOME/bin/spark-submit \
--deploy-mode cluster \
--master yarn \
--conf spark.pyspark.virtualenv.enabled=true \
--conf spark.pyspark.virtualenv.type=native \
--conf spark.pyspark.virtualenv.requirements=${parent}/requirements.txt \
--conf spark.pyspark.virtualenv.bin.path=${parent}/venv \
--py-files "${parent}/lib/libs.zip" \
--num-executors 1 \
--executor-cores 2 \
--executor-memory 2G \
--driver-memory 2G \
$parent/src/features/pi.py
I also know that on the remote nodes there is a /usr/local/bin/python2.7 folder that includes a python 2.7 install.
so in my conf/spark-env.sh I have set the following:
export PYSPARK_PYTHON=/usr/local/bin/python2.7
export PYSPARK_DRIVER_PYTHON=/usr/local/bin/python2.7
When I run the script I get the error above. If I screen print the installed_distributions I get a zero length list []. Also my private library imports correctly (which says to me it is actually accessing my libs.zip site-packages.). My pi.py file looks something like this:
from myprivatelibrary.bigData.spark import spark_context
spark = spark_context()
import numpy as np
spark.parallelize(range(1, 10)).map(lambda x: np.__version__).collect()
EXPECTATION/MY THOUGHTS:
I expect this to import numpy correctly especially since I know numpy works correctly in my local virtualenv. I suspect this is because I'm not actually using the version of python that is installed in my virtualenv on the remote node. My question is first, how do I fix this and second how do I use my virtualenv installed python on the remote nodes instead of the python that is just manually installed and currently sitting on those machines? I've seen some write-ups on this but frankly they are not well written.
With --conf spark.pyspark.{} and export PYSPARK_PYTHON=/usr/local/bin/python2.7 you set options for your local environment / your driver. To set options for the cluster (executors) use the following syntax:
--conf spark.yarn.appMasterEnv.PYSPARK_PYTHON
Furthermore, I guess you should make your virtualenv relocatable (this is experimental, however). <edit 20170908> This means that the virtualenv uses relative instead of absolute links. </edit>
What we did in such cases: we shipped an entire anaconda distribution over hdfs.
<edit 20170908>
If we are talking about different environments (MacOs vs. Linux, as mentioned in the comment below), you cannot just submit a virtualenv, at least not if your virtualenv contains packages with binaries (as is the case with numpy). In that case I suggest you create yourself a 'portable' anaconda, i.e. install Anaconda in a Linux VM and zip it.
Regarding --archives vs. --py-files:
--py-files adds python files/packages to the python path. From the spark-submit documentation:
For Python applications, simply pass a .py file in the place of instead of a JAR, and add Python .zip, .egg or .py files to the search path with --py-files.
--archives means these are extracted into the working directory of each executor (only yarn clusters).
However, a crystal-clear distinction is lacking, in my opinion - see for example this SO post.
In the given case, add the anaconda.zip via --archives, and your 'other python files' via --py-files.
</edit>
See also: Running Pyspark with Virtualenv, a blog post by Henning Kropp.