I work with Tensorflow and PyTorch and manage each project that I work on inside of a Conda environment. This allows me to share projects and update my stack without breaking old projects. I am trying to get Conda to manage Cuda within an environment as well. Is this possible? I want old projects to be maintained whenever I update to newer versions of Cuda and CUDnn. Is there a way for Anaconda do this?
Related
I followed the instructions on the official website to download the TensorFlow. I chose to create a virtual environment as the instruction shown for macOS. My question is that if I need to activate the virtual environment each time before I use TensorFlow?
For example, I want to use tensor flow on Jupiter notebook and that means I need to install Jupiter and other required packages like Seaborn/pandas as well on the virtual environment. However I already downloaded anaconda and basically, it has all the packages I need.
Besides, will it make a difference if I download it with conda?
Well, if you downloaded the packages (like you said TensorFlow and Seaborn) in the base Conda environment which is the default environment that anaconda provides on installation, then to use what it has, you need to run whatever program/IDE like Jupyter lab from it. So you would open Anaconda Prompt and then type in jupyter lab and it would open up a new socket and you can edit with your installed python libraries from Conda.
Otherwise in IDE's VSCode you can simply set the python interpreter to that from Conda.
However, if you install the libraries and packages you need using pip on your actual python installation not Conda, then there is no need for any activation. Everything will run right out of the box. You don't need to select the interpreter in IDE's like VSCode.
Bottom line, if you know what libraries you need and don't mind running pip install package-name every time you need a package, stick with pip.
If you don't like to that sort of 'low level' stuff then use Anaconda or Miniconda.
1.Create a new environment through conda create --name tftest. (You can replace tftest with e.g. the name of your current project.)
2.Activate that new environment through activate tftest.
3.Install TF into this environment through conda install tensorflow.
4.Ensure that you're in the right environment through where python (which should produce a path containing "tftest").
5.Run Python through python.
6.import tensorflow as tf in a shell in that environment.
Thanks to great community as I found this thanks to another post!!!
Starting with version 1.6.0, prebuild binaries need AVX instructions.
There are some bug reports by people who tried to use the precompiled binaries but whose doesn't support AVX instructions and got the same error as you posted here:
https://github.com/tensorflow/tensorflow/issues/17761
https://github.com/tensorflow/tensorflow/issues/17386
Maybe you have this problem? If yes, you may have to build tensorflow from sources or downgrade to tensorflow 1.5.1.
How come we need to install tensorflow as a separate environment?
If we do it this way, many common libraries are not available when tensorflow is activated.
Most of the common libraries such as matplotlib, panda, etc. are not within tensorflow environment. So we have to install again to use them.
So why not just install under root so we don't have to re-install all those libraries under the new environment?
Thanks.
Upon trying to install Tensorflow for conda environment, I encountered with the following error message, without any progress:
tensorflow-1.1.0-cp35-cp35mwin_amd64.whl is not a supported wheel on this platform
Have you tried uninstalling and re-installing TensorFlow using pip within your Conda environment? I.e.:
pip uninstall tensorflow
Followed by:
pip install tensorflow
If it doesn't work, the issue may be with your Python installation. TensorFlow only supports 64-bit Python 3.5+ on Windows (see more info here).
Perhaps you have Python's default installation, which comes in a 32-bit version. If that's the case, you can download the 64-bit Python 3.5 or later from here to run in your Conda environment and then you should be able to install/run TensorFlow without any issues.
Make sure that the Python version installed in the Environment is 3.5 not 3.6. Since 3.6 was released Conda automatically sets that version as default for python 3. However, it is still not supported by Tensorflow.
You can work using tensorflow library along with other essential libraries using the Dockerfile. Using Docker for environment are a good way to run experiments in reproducible manner as in this blog
You can also try using datmo in order setup environment and track machine learning projects for making it reproducible using datmo CLI tool.
I am looking into using Tensorflow for my research soon, and looked at the online documentation for installing with Conda https://www.tensorflow.org/versions/r0.11/get_started/os_setup.html#anaconda-installation.
It suggested creating a new environment, and installing Tensorflow in it, and the installing other python packages afterwards.
But I already have an existing environment with lots of packages I need, and I'm wondering if its safe to add Tensorflow into that environment?
Also, I have a question about how this installation with conda works. I know that Conda will create a distinct set of folders containing the libraries needed for each environment, but if I install Tensorflow, what happens to all the base low level C++ and CUDA libraries that get compiled? Do they reside in my Conda environment's folder, or are they in some system wide libraries closer to my root?
PS: I'm using Ubuntu 16.04, and have a GPU that I want to run Tensorflow on.
Thank you.
But I already have an existing environment with lots of packages I need, and I'm wondering if its safe to add Tensorflow into that environment?
conda has this awesome feature called "revisions". You can show your current environment with
conda list --revisions
which allows you to revert changes to your conda environment. This allows you to install new packages with confidence that if something breaks you can always revert it later. See this page for more info: https://www.continuum.io/blog/developer/advanced-features-conda-part-2. tl;dr: conda install --revisions <revision_number>
what happens to all the base low level C++ and CUDA libraries that get compiled
Are you talking about the libraries that get compiled when you are trying to run your code? Or the C++/CUDA libraries? If you're talking about the C++/CUDA libs then conda is not compiling them, but merely installing a pre-compiled binary into a specific location that gets picked up. If you're talking about your code, then where exactly those files live would seem to depend on where you put them.