Colab does not allow to downgrade TensorFlow and says that only versions 2 is available.
here is code and output:
!pip install tensorflow-gpu==1.15.2
import tensorflow as tf
print(tf.__version__)
output:
Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/
ERROR: Could not find a version that satisfies the requirement tensorflow-gpu==1.15.2 (from versions: 2.2.0, 2.2.1, 2.2.2, 2.2.3, 2.3.0, 2.3.1, 2.3.2, 2.3.3, 2.3.4, 2.4.0, 2.4.1, 2.4.2, 2.4.3, 2.4.4, 2.5.0, 2.5.1, 2.5.2, 2.5.3, 2.6.0, 2.6.1, 2.6.2, 2.6.3, 2.6.4, 2.6.5, 2.7.0rc0, 2.7.0rc1, 2.7.0, 2.7.1, 2.7.2, 2.7.3, 2.7.4, 2.8.0rc0, 2.8.0rc1, 2.8.0, 2.8.1, 2.8.2, 2.8.3, 2.8.4, 2.9.0rc0, 2.9.0rc1, 2.9.0rc2, 2.9.0, 2.9.1, 2.9.2, 2.9.3, 2.10.0rc0, 2.10.0rc1, 2.10.0rc2, 2.10.0rc3, 2.10.0, 2.10.1, 2.11.0rc0, 2.11.0rc1, 2.11.0rc2, 2.11.0)
ERROR: No matching distribution found for tensorflow-gpu==1.15.2
2.9.2
I tried this code by creating some new projects in Colab but I got no new results
To downgrade tensorflow in google colab you must downgrade the version of python because python3.8 is not compatible with tensorflow1.x. So, Downgrade the version of python to python3.7 and install tensorflow1.x. This will work.
1.At first, Install python==3.7 version.
!sudo apt-get update -y
!sudo apt-get install python3.7
#change alternatives
!sudo update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.8 1
!sudo update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.7 2
#check python version
!python --version
The above code will install python3.7 version, but it is not mapped to colab kernel, so we must map newly installed python.
# install pip for python==3.7
!sudo apt-get install python3.7-distutils
!wget https://bootstrap.pypa.io/get-pip.py
!python get-pip.py
# install colab dependencies
!python -m pip install ipython ipython_genutils ipykernel jupyter_console prompt_toolkit httplib2 astor
# link to the old google package
!ln -s /usr/local/lib/python3.8/dist-packages/google \
/usr/local/lib/python3.7/dist-packages/google
Let's check version of python again, it will be python==3.7.16:
!python --version
Now, it's time to install tensorflow 1.x.
!pip install tensorflow==1.x
I hope that it will help to fix your issue.Thank you!
Related
I want to install tensorflow-text with version of 2.8.x.
First when I use:
pip install tensorflow-text',
I can only installed the version of 2.6.x.
And then when I use:
pip install -U "tensorflow-text==2.8.*" ,
it said that:
ERROR: Could not find a version that satisfies the requirement
tensorflow-text==2.8.* (from versions: 0.1.0rc2, 0.1.0, 1.0.0b2,
1.15.0rc0, 1.15.1, 2.0.0rc0, 2.0.1, 2.1.0rc0, 2.2.0rc2, 2.2.0, 2.2.1, 2.3.0rc1, 2.3.0, 2.4.0b0, 2.4.0rc0, 2.4.0rc1, 2.4.1, 2.4.2, 2.4.3, 2.5.0rc0, 2.5.0, 2.6.0rc0, 2.6.0) ERROR: No matching distribution found for tensorflow-text==2.8.*
I have been finished install Tensorflow env step by step from "https://developer.apple.com/metal/tensorflow-plugin/"
Tf is working!numpy is working! scipy is working!
but when i import sklearn package, have an error message like this:
ImportError: dlopen(/Users/mecilmeng/miniforge3/envs/tf/lib/python3.9/site-packages/scipy/spatial/qhull.cpython-39-darwin.so, 0x0002): Library not loaded: #rpath/liblapack.3.dylib
Referenced from: /Users/mecilmeng/miniforge3/envs/tf/lib/python3.9/site-packages/scipy/spatial/qhull.cpython-39-darwin.so
Reason: tried: '/Users/mecilmeng/miniforge3/envs/tf/lib/liblapack.3.dylib' (no such file), '/Users/mecilmeng/miniforge3/envs/tf/lib/liblapack.3.dylib' (no such file), '/Users/mecilmeng/miniforge3/envs/tf/lib/python3.9/site-packages/scipy/spatial/../../../../liblapack.3.dylib' (no such file), '/Users/mecilmeng/miniforge3/envs/tf/lib/liblapack.3.dylib' (no such file), '/Users/mecilmeng/miniforge3/envs/tf/lib/liblapack.3.dylib' (no such file), '/Users/mecilmeng/miniforge3/envs/tf/lib/python3.9/site-packages/scipy/spatial/../../../../liblapack.3.dylib' (no such file), '/Users/mecilmeng/miniforge3/envs/tf/bin/../lib/liblapack.3.dylib' (no such file), '/Users/mecilmeng/miniforge3/envs/tf/bin/../lib/liblapack.3.dylib' (no such file), '/usr/local/lib/liblapack.3.dylib' (no such file), '/usr/lib/liblapack.3.dylib' (no such file)
How to fix it?
pip list
Package Version
------------------------ -------------------
absl-py 0.10.0
aiohttp 3.8.1
aiosignal 1.2.0
anyio 3.5.0
appnope 0.1.2
argon2-cffi 20.1.0
astunparse 1.6.3
async-generator 1.10
async-timeout 4.0.1
attrs 21.4.0
Babel 2.9.1
backcall 0.2.0
beniget 0.3.0
bleach 4.1.0
blinker 1.4
Bottleneck 1.3.2
brotlipy 0.7.0
cached-property 1.5.2
cachetools 4.2.2
certifi 2021.10.8
cffi 1.15.0
charset-normalizer 2.0.4
click 8.0.3
cryptography 3.4.7
cycler 0.11.0
Cython 0.29.28
debugpy 1.5.1
decorator 5.1.1
defusedxml 0.7.1
dill 0.3.4
entrypoints 0.3
flatbuffers 2.0
fonttools 4.25.0
frozenlist 1.2.0
gast 0.4.0
google-auth 1.33.0
google-auth-oauthlib 0.4.1
google-pasta 0.2.0
googleapis-common-protos 1.54.0
grpcio 1.42.0
h5py 3.1.0
idna 3.3
importlib-metadata 4.8.2
ipykernel 6.4.1
ipython 7.31.1
ipython-genutils 0.2.0
jedi 0.18.1
Jinja2 3.0.2
joblib 1.1.0
json5 0.9.6
jsonschema 3.2.0
jupyter-client 7.1.2
jupyter-core 4.9.1
jupyter-server 1.13.5
jupyterlab 3.2.1
jupyterlab-pygments 0.1.2
jupyterlab-server 2.10.3
keras 2.8.0
Keras-Preprocessing 1.1.2
kiwisolver 1.3.1
libclang 13.0.0
Markdown 3.3.4
MarkupSafe 2.0.1
matplotlib 3.5.0
matplotlib-inline 0.1.2
mistune 0.8.4
multidict 5.2.0
munkres 1.1.4
nbclassic 0.2.6
nbclient 0.5.3
nbconvert 6.3.0
nbformat 5.1.3
nest-asyncio 1.5.1
networkx 2.6.3
notebook 6.4.6
numexpr 2.8.1
numpy 1.22.2
oauthlib 3.1.1
opencv-python 4.5.5.62
opt-einsum 3.3.0
packaging 21.3
pandas 1.3.5
pandocfilters 1.5.0
parso 0.8.3
pexpect 4.8.0
pickleshare 0.7.5
Pillow 9.0.1
pip 21.2.4
ply 3.11
prometheus-client 0.13.1
promise 2.3
prompt-toolkit 3.0.20
protobuf 3.19.1
ptyprocess 0.7.0
pyasn1 0.4.8
pyasn1-modules 0.2.8
pybind11 2.9.1
pycparser 2.21
Pygments 2.11.2
PyJWT 2.1.0
pyOpenSSL 21.0.0
pyparsing 3.0.4
pyrsistent 0.18.0
PySocks 1.7.1
python-dateutil 2.8.2
pythran 0.9.11
pytz 2021.3
pyzmq 22.3.0
requests 2.27.1
requests-oauthlib 1.3.0
rsa 4.7.2
scikit-learn 1.0.2
scipy 1.7.1
Send2Trash 1.8.0
setuptools 58.0.4
six 1.15.0
sklearn 0.0
sniffio 1.2.0
tensorboard 2.8.0
tensorboard-data-server 0.6.1
tensorboard-plugin-wit 1.6.0
tensorflow-datasets 4.5.2
tensorflow-macos 2.8.0
tensorflow-metadata 1.6.0
tensorflow-metal 0.3.0
termcolor 1.1.0
terminado 0.13.1
testpath 0.5.0
tf-estimator-nightly 2.8.0.dev2021122109
threadpoolctl 2.2.0
tornado 6.1
tqdm 4.62.3
traitlets 5.1.1
typing-extensions 3.7.4.3
urllib3 1.26.8
wcwidth 0.2.5
webencodings 0.5.1
websocket-client 0.58.0
Werkzeug 2.0.2
wheel 0.35.1
wrapt 1.12.1
yarl 1.6.3
zipp 3.7.0
You can install using Rosetta2 Mode.
To work in Rosetta Mode:
If Rosetta 2 is not installed by default in your M1 Mac, then open the pre-installed Terminal app and run the following command:
/usr/sbin/softwareupdate --install-rosetta --agree-to-license
Rosetta allows us to use apps built for Mac with intel chip.
Several CLI tools do not have native versions built for the new M1 architecture.
Enabling them on your native M1 Mac terminal can be frustrating.
Follow these steps to enable Rosetta:
Select the app(Terminal) in the Finder.
Right click on the app(Terminal) and select Get Info.
In General, check the Open using Rosetta check-box.
Close the Terminal Info.
Now when you quit the terminal and open it again.
If you haven't installed Rosetta yet, then it would prompt you to install it.
If the popup shows up, then click on Install button, then enter your user name and password to allow installation to proceed.
Close the Terminal and open again.
Now we have a special terminal that can install tools with Rosetta translation.
To verify that you are using a Rosetta terminal, run the following command and it should output i386:
arch
The native terminal without Rosetta would output arm64 for the above command.
Moving forward, all commands we ask you to execute should be done in Rosetta enabled terminal.
Uninstall arm64 brew
If you have installed brew in the past from the native terminal, it is likely that you have an arm64 build of brew. Having two different builds of brew can cause major problems as the packages with different builds will not be compatible with each other.
To avoid this problem you need to uninstall your current installation of arm64 brew.
You can check which build you have by running the following command:
which brew
If your installation of brew is the Intel build, then the command should output /usr/local/bin/brew. If that is the case you can skip installing brew and just update your current installation by running brew update.
If your output is /opt/homebrew then your installation of brew is the arm64 build.
You need to uninstall the arm64 build of brew by running the following command from the native terminal:
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/uninstall.sh)"
Install Intel brew
Install Homebrew, which is the package manager:
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
Once done, run the below command to ensure that we make use of the HEAD revision:
git -C $(brew --repository homebrew/core) checkout master
Now verify the installation of the brew command:
which brew
The command should output /usr/local/bin/brew, which is the expected path.
The current default version of Python running on Google Colab is 3.7, but I need 3.9 for my notebooks to work.
How can I update Google Colab's Python version to 3.9 (or greater)?
In Google Colab you have a Debian-based Linux, and you can do whatever you can on a Debian Linux. Upgrading Python is as easy as upgrading it on your own Linux system.
Detect the current python version in Colab:
!python --version
#Python 3.8.16
Install new python version
Let's first install and upgrade to Python 3.9:
#install python 3.9
!sudo apt-get update -y
!sudo apt-get install python3.9
#change alternatives
!sudo update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.8 1
!sudo update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.9 2
#check python version
!python --version
#3.9.16
Port Colab kernel to the new installed python
As mentioned in the comments, the above commands just add a new python version to your google colab and update the default python for commandline usage. But your runtime packages such as sys are still running on the previous python version. The following commands need to be executed as well, to update the sys version.
# install pip for new python
!sudo apt-get install python3.9-distutils
!wget https://bootstrap.pypa.io/get-pip.py
!python get-pip.py
# credit of these last two commands blongs to #Erik
# install colab's dependencies
!python -m pip install ipython ipython_genutils ipykernel jupyter_console prompt_toolkit httplib2 astor
# link to the old google package
!ln -s /usr/local/lib/python3.8/dist-packages/google \
/usr/local/lib/python3.9/dist-packages/google
Now you can restart runtime and check the sys version. Note that in the new python version you have to install every packages, such as pandas, tensorflow, etc. from scratch.
Also, note that you can see a list of installed Python versions and switch between them at any time with this command:
(If nothing changed after installation, use this command to select python version manually)
!sudo update-alternatives --config python3
#after running, enter the row number of the python version you want.
It's also possible to update the kernel without going through ngrok or conda with some creative package installation.
Raha's answer suggesting making a link between the default google package and the newly installed Python version is the trick that makes this work because, at least with Python 3.9, the version of pandas (0.24.0) that the google package requires fails to build.
Here's the code I used to install and switch my Colab kernel to Python 3.9:
#install python 3.9 and dev utils
#you may not need all the dev libraries, but I haven't tested which aren't necessary.
!sudo apt-get update -y
!sudo apt-get install python3.9 python3.9-dev python3.9-distutils libpython3.9-dev
#change alternatives
!sudo update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.8 1
!sudo update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.9 2
#Check that it points at the right location
!python3 --version
# install pip
!curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
!python3 get-pip.py --force-reinstall
#install colab's dependencies
!python3 -m pip install ipython ipython_genutils ipykernel jupyter_console prompt_toolkit httplib2 astor
# link to the old google package
!ln -s /usr/local/lib/python3.8/dist-packages/google \
/usr/local/lib/python3.9/dist-packages/google
# There has got to be a better way to do this...but there's a bad import in some of the colab files
# IPython no longer exposes traitlets like this, it's a separate package now
!sed -i "s/from IPython.utils import traitlets as _traitlets/import traitlets as _traitlets/" /usr/local/lib/python3.9/dist-packages/google/colab/*.py
!sed -i "s/from IPython.utils import traitlets/import traitlets/" /usr/local/lib/python3.9/dist-packages/google/colab/*.py
If Google updates from Python 3.8, you'll have to change the path to the default package.
Then go the Runtime menu and select Restart runtime. It should reconnect and choose the updated version of Python as the default kernel. You can check that it worked with:
#check python version
import sys
print(sys.version)
!python3 --version
!python --version
To use another python version in google colab, you need to:
1- Installing Anaconda.
2- Adding (fake) google colab library.
3- Starting Jupyterlab.
4- Accessing it with ngrok.
# install Anaconda3
!wget -qO ac.sh https://repo.anaconda.com/archive/Anaconda3-2020.07-Linux-x86_64.sh
!bash ./ac.sh -b
# a fake google.colab library
!ln -s /usr/local/lib/python3.6/dist-packages/google \
/root/anaconda3/lib/python3.8/site-packages/google
# start jupyterlab, which now has Python3 = 3.8
!nohup /root/anaconda3/bin/jupyter-lab --ip=0.0.0.0&
# access through ngrok, click the link
!pip install pyngrok -q
from pyngrok import ngrok
print(ngrok.connect(8888))
you can also use:
# Install the python version
!apt-get install python3.9
# Select the version
!python3.9 setup.py
another way is to use a virtual environment with your desired python version:
virtualenv env --python=python3.9
Update 24.12.2022 - Unfortunately, the method does not work anymore.
This worked for me (copied from GitHub), I successfully installed Python 3.10.
#The code below installs 3.10 (assuming you now have 3.8) and restarts environment, so you can run your cells.
import sys #for version checker
import os #for restart routine
if '3.10' in sys.version:
print('You already have 3.10')
else:
#install python 3.10 and dev utils
#you may not need all the dev libraries, but I haven't tested which aren't necessary.
!sudo apt-get update -y
!sudo apt-get install python3.10 python3.10-dev python3.10-distutils libpython3.10-dev
!sudo apt-get install python3.10-venv binfmt-support #recommended in install logs of the command above
#change alternatives
!sudo update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.8 1
!sudo update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.10 2
# install pip
!curl -sS https://bootstrap.pypa.io/get-pip.py | python3.10
!python3 get-pip.py --force-reinstall
#install colab's dependencies
!python3 -m pip install setuptools ipython ipython_genutils ipykernel jupyter_console prompt_toolkit httplib2 astor
#minor cleanup
!sudo apt autoremove
#link to the old google package
!ln -s /usr/local/lib/python3.8/dist-packages/google /usr/local/lib/python3.10/dist-packages/google
#this is just to verify if 3.10 folder was indeed created
!ls /usr/local/lib/python3.10/
#restart environment so you don't have to do it manually
os.kill(os.getpid(), 9)
In addition to Kaveh's answer, I added the following code. (This colab python version is python 3.8 and I tried to downgrade to python 3.7)
!pip install google-colab==1.0.0
# install colab's dependencies
!python -m pip install ipython==7.9.0 ipython_genutils==0.2.0 ipykernel==5.3.4 jupyter_console==6.1.0 prompt_toolkit==2.0.10 httplib2==0.17.4 astor==0.8.1 traitlets==5.7.1 google==2.0.3
This way, I solved the crashing runtime error.
Simple as that: -
!wget -O mini.sh https://repo.anaconda.com/miniconda/Miniconda3-py39_4.9.2-Linux-x86_64.sh
!chmod +x mini.sh
!bash ./mini.sh -b -f -p /usr/local
!conda install -q -y jupyter
!conda install -q -y google-colab -c conda-forge
!python -m ipykernel install --name "py39" --user
Source: https://colab.research.google.com/drive/1m47aWKayWTwqJG--x94zJMXolCEcfyPS?usp=sharing#scrollTo=r3sLiMIs8If3
When attempting to install tensorflow on an ARM M1 MacBookPro, I am seeing the following issues:
% pip install tensorflow
ERROR: Could not find a version that satisfies the requirement tensorflow (from versions: none)
ERROR: No matching distribution found for tensorflow
After searching around, I found a package called tensorflow-macos which produces the following issues:
clang: error: the clang compiler does not support 'faltivec', please use -maltivec and include altivec.h explicitly
error: Command "gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -arch arm64 -arch x86_64 -g -DNPY_INTERNAL_BUILD=1 -DHAVE_NPY_CONFIG_H=1 -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE=1 -D_LARGEFILE64_SOURCE=1 -DNO_ATLAS_INFO=3 -DHAVE_CBLAS -Ibuild/src.macosx-10.9-universal2-3.9/numpy/core/src/umath -Ibuild/src.macosx-10.9-universal2-3.9/numpy/core/src/npymath -Ibuild/src.macosx-10.9-universal2-3.9/numpy/core/src/common -Inumpy/core/include -Ibuild/src.macosx-10.9-universal2-3.9/numpy/core/include/numpy -Inumpy/core/src/common -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/src/npysort -I/Users/thing/dev/SOTAMoon/venv/include -I/Library/Frameworks/Python.framework/Versions/3.9/include/python3.9 -Ibuild/src.macosx-10.9-universal2-3.9/numpy/core/src/common -Ibuild/src.macosx-10.9-universal2-3.9/numpy/core/src/npymath -c numpy/core/src/multiarray/descriptor.c -o build/temp.macosx-10.9-universal2-3.9/numpy/core/src/multiarray/descriptor.o -MMD -MF build/temp.macosx-10.9-universal2-3.9/numpy/core/src/multiarray/descriptor.o.d -faltivec -I/System/Library/Frameworks/vecLib.framework/Headers -std=c99" failed with exit status 1
----------------------------------------
ERROR: Failed building wheel for numpy
Failed to build numpy
ERROR: Could not build wheels for numpy which use PEP 517 and cannot be installed directly
----------------------------------------
WARNING: Discarding https://files.pythonhosted.org/packages/a7/81/20d5d994c91ed8347efda90d32c396ea28254fd8eb9e071e28ee5700ffd5/h5py-3.1.0.tar.gz#sha256=1e2516f190652beedcb8c7acfa1c6fa92d99b42331cbef5e5c7ec2d65b0fc3c2 (from https://pypi.org/simple/h5py/) (requires-python:>=3.6). Command errored out with exit status 1: /Users/thing/dev/SOTAMoon/venv/bin/python3 /private/var/folders/8k/z291bhgd5gs06tp9b4j6_bb40000gn/T/pip-standalone-pip-8qa87uy0/__env_pip__.zip/pip install --ignore-installed --no-user --prefix /private/var/folders/8k/z291bhgd5gs06tp9b4j6_bb40000gn/T/pip-build-env-bxy_nyoo/normal --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- 'numpy==1.12; python_version == "3.6"' 'numpy==1.19.3; python_version >= "3.9"' 'numpy==1.14.5; python_version == "3.7"' pkgconfig 'Cython>=0.29; python_version < "3.8"' 'Cython>=0.29.14; python_version >= "3.8"' 'numpy==1.17.5; python_version == "3.8"' Check the logs for full command output.
ERROR: Could not find a version that satisfies the requirement h5py~=3.1.0 (from tensorflow-macos) (from versions: 2.2.1, 2.3.0b1, 2.3.0, 2.3.1, 2.4.0b1, 2.4.0, 2.5.0, 2.6.0, 2.7.0rc2, 2.7.0, 2.7.1, 2.8.0rc1, 2.8.0, 2.9.0rc1, 2.9.0, 2.10.0, 3.0.0rc1, 3.0.0, 3.1.0, 3.2.0, 3.2.1, 3.3.0)
ERROR: No matching distribution found for h5py~=3.1.0
Unfortunately I couldn't work out a solution to the 'faltivec' issue. I also found a tensorflow package made by Apple that seems specifically geared for M1's, however going by this tutorial produces the following issues:
% pip install --upgrade --force --no-dependencies https://github.com/apple/tensorflow_macos/releases/download/v0.1alpha3/tensorflow_addons_macos-0.1a3-cp38-cp38-macosx_11_0_arm64.whl https://github.com/apple/tensorflow_macos/releases/download/v0.1alpha3/tensorflow_macos-0.1a3-cp38-cp38-macosx_11_0_arm64.whl
ERROR: tensorflow_addons_macos-0.1a3-cp38-cp38-macosx_11_0_arm64.whl is not a supported wheel on this platform.
After this, I am a bit stuck. I have upgraded pip to 21.1.3, and my python version is Python 3.9.6.
Tensorflow-macos and Tensorflow-metal Install
Currently, to harness the M1 GPU you need to install Tensorflow-macos and TensorFlow-metal as opposed to Tensorflow, the install steps are detailed here, they can be summarized as follows using mini-forge:
conda create -n tf python=3.8 -y
conda activate tf
conda install -c apple tensorflow-deps -y
# Navigate the issue with conda environments
# built against pre-macOS 11 SDK use (Just in case)
# this solves the Intel / M1 install issues:
SYSTEM_VERSION_COMPAT=0 pip install tensorflow-macos
SYSTEM_VERSION_COMPAT=0 pip install tensorflow-metal
to test all is well we can run:
#!/usr/bin/env bash
PYCMD=$(cat <<EOF
import tensorflow as tf
tf.config.list_physical_devices()
with tf.device('/GPU'):
a = tf.random.normal(shape=(2,), dtype=tf.float32)
b = tf.nn.relu(a)
print(a)
print(b)
EOF
)
python3 -c "$PYCMD" 2>/dev/null
the expected output for my test machine:
>./val.sh
Metal device set to: Apple M1
systemMemory: 16.00 GB
maxCacheSize: 5.33 GB
tf.Tensor([-0.4044231 0.08157467], shape=(2,), dtype=float32)
tf.Tensor([0. 0.08157467], shape=(2,), dtype=float32)
ok, everything is (assumed) to be working...
Side-Note: Package Install Issues:
One thing to note is that that packages that rely on tensorflow will try to install tensorflow as they will likely not have an explicit dependency on tensorflow-macos or tensorflow-metal.
When you install tensorflow dependent packages or code you'll likely have to review the requirements.txt file before running:
python3 pip install -r requirements.txt
The requirements.txt will try to install the tensorflow package, not the tensorflow-macosx or tensorflow-metal packages which will result in a host of issues if you run the requirements.txt file unchanged. That and explicit versions of packages == which can also cause problems.
Hopefully, this points you in the right direction in thinking about the hurdles involved in getting python to work with the Apple-M1 laptops using the TensorFlow metal plugin to harness the M1 GPU features.
Stay safe and well.
try
python3 -m pip install tensorflow-macos
python3 -m pip install tensorflow-metal
you may also need:
brew install hdf5
export HDF5_DIR=$(brew --prefix hdf5)
I'm going over these example from google-cloud Coursera courses, and although they worked till a few weeks ago, I can't install tf.transform or apache_beam on Datalab anymore.
https://github.com/GoogleCloudPlatform/training-data-analyst/blob/master/courses/machine_learning/feateng/tftransform.ipynb
https://github.com/GoogleCloudPlatform/training-data-analyst/blob/master/courses/machine_learning/deepdive/06_structured/4_preproc_tft.ipynb
When installing tensorflow_transform I get the following errors:
%bash
pip install --upgrade --force tensorflow_transform==0.6.0
twisted 18.7.0 requires PyHamcrest>=1.9.0, which is not installed.
datalab 1.1.3 has requirement six==1.10.0, but you'll have six 1.11.0 which is incompatible.
gapic-google-cloud-pubsub-v1 0.15.4 has requirement oauth2client<4.0dev,>=2.0.0, but you'll have oauth2client 4.1.2 which is incompatible.
proto-google-cloud-pubsub-v1 0.15.4 has requirement oauth2client<4.0dev,>=2.0.0, but you'll have oauth2client 4.1.2 which is incompatible.
apache-airflow 1.9.0 has requirement bleach==2.1.2, but you'll have bleach 1.5.0 which is incompatible.
apache-airflow 1.9.0 has requirement funcsigs==1.0.0, but you'll have funcsigs 1.0.2 which is incompatible.
google-cloud-monitoring 0.28.0 has requirement google-cloud-core<0.29dev,>=0.28.0, but you'll have google-cloud-core 0.25.0 which is incompatible.
proto-google-cloud-datastore-v1 0.90.4 has requirement oauth2client<4.0dev,>=2.0.0, but you'll have oauth2client 4.1.2 which is incompatible.
pandas-gbq 0.3.0 has requirement google-cloud-bigquery>=0.28.0, but you'll have google-cloud-bigquery 0.25.0 which is incompatible.
googledatastore 7.0.1 has requirement httplib2<0.10,>=0.9.1, but you'll have httplib2 0.11.3 which is incompatible.
googledatastore 7.0.1 has requirement oauth2client<4.0.0,>=2.0.1, but you'll have oauth2client 4.1.2 which is incompatible.
Cannot uninstall 'dill'. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall.
The tensorflow version on my Datalab instance was 1.4.
I had to add this one line of code to update tensorflow to 1.10.1
%bash
pip install --upgrade --force-reinstall pip==10.0.1
pip install tensorflow==1.10.1
pip install tensorflow_transform
my environment:
apache-airflow==1.9.0
apache-beam==2.6.0
tensorflow==1.10.1
tensorflow-metadata==0.9.0
tensorflow-tensorboard==0.4.0rc3
tensorflow-transform==0.8.0
The current version of Datalab uses TensorFlow 1.8, so please change the notebook cell in question to:
%bash
pip uninstall -y google-cloud-dataflow
pip install --upgrade --force tensorflow_transform==0.8.0 apache-beam[gcp]
I've updated and checked in the two notebooks linked above.
Another problem might be that you are using Python 2. Datalab by default now uses Python 3 and your pip install (above) happens in Python 3 even if the kernel is Python 2 because %%bash opens up a new shell in which the conda activate of Python 2 has not happened.
To make sure the pip install happens in Python 2, change your pip install of apache-beam[gcp] as follows:
%%bash
source activate py2env
conda install -y dill pytz # do this for all the distutils complaints
pip uninstall -y google-cloud-dataflow
pip install --upgrade --force tensorflow_transform==0.8.0 apache-beam[gcp]