My main Python environment on OSX Big Sur is the base env from Anaconda.
Now, I regularly need to install packages that are not available via conda or conda-forge or not in the latest version.
I understand that there are several different approaches:
Using the pip package installed by conda and follow the preferred way of using pip within conda i.e. to add those packages at the end of a environment.yaml file and to delete and re-generate the env whenever a new package needs to be added, in order to avoid conflicts from the interplay between conda and pip.
Forking the latest available recipe as described here, updating the version number and details, and using my local version of the recipe.
Simply installing the packages via pip, npm (Node JS package manager), apm (Atom package manager) from within the activated conda environment.
Which approach is best? Is it safe to use pip with conda in the way described in #1?
#2 is more effort than the other two. But I am looking for a generic approach that will not mess up my installation.
Related
While I am in a conda environment, the 'conda list' and 'pip freeze' show different number of libraries. For example, 'tensorflow-gpu' is listed in 'pip freeze', but not in 'conda list'. If I want to use tensorflow-gpu in this environment, should I run pip install tensorflow-gpu to install it again, or not necessary?
I think when you are using the conda environment. The conda list is going to show all the general packages that shared by the same conda environment. And the reason why 'tensorflow-gpu' is listed in 'pip freeze', but not in 'conda list', is because you used pip install to installed 'tensorflow-gpu'(could be you or the IDE). In this case, 'tensorflow-gpu' is only exists under this python project I believe. Actually, there is an official document about this topic.
Issues may arise when using pip and conda together. When combining
conda and pip, it is best to use an isolated conda environment. Only
after conda has been used to install as many packages as possible
should pip be used to install any remaining software. If modifications
are needed to the environment, it is best to create a new environment
rather than running conda after pip. When appropriate, conda and pip
requirements should be stored in text files.
Use pip only after conda Install as many requirements as possible with
conda then use pip.
Pip should be run with --upgrade-strategy only-if-needed (the
default).
Do not use pip with the --user argument, avoid all users installs.
And here is the link.
previously, I installed the tensorflow 1.13 in my machine.
There are some projects depending on different version of tensorflow and I do not want to mixed up different version of tensowflow.
So I just tried create a env called tf2.0 and used pip to install tensorflow 2.0.0b1 in that specific virtual environment.
However, after I ran 'pip install tensorflow-gpu==2.0.0b1` in that "tf2.0" conda environment, I found that it takes effect globally, which mean I have to use tensorflow-gpu 2.0.0b1 even when that virtual env "tf2.0" disactivated.
I wish I could use tensorflow 1.13 when virtual env is deactivated.
It's hard to troubleshoot the described conditions without more details (exact commands run, showing PATH before and after and post activation, etc.). Nevertheless, you can try switching to following the most recent recommendations for mixing Conda and Pip. Namely, avoid installing things ad hoc, which is prone to using the wrong pip and clobbering packages, but instead define a YAML file and always create the whole env in one go.
As a minimal example:
my_env.yaml
name: my_env
channels:
- defaults
dependencies:
- python
- pip
- pip:
- tensorflow-gpu==2.0.0b1
which can be created with conda env create -f my_env.yaml. Typically, it is best to include everything possible in the "non-pip" section of dependencies.
It is mostly that you used a wrong pip. To make sure you are using correct pip, it is usually a good practice to do
python -m pip install —user PACKAGE_NAME
Given that you have conda, pip should be the last resort.
Conda channel conda-forge most likely has the latest package version you are looking for.
conda install -c conda-forge PACKAGE_NAME
If you have to use pip, make sure you are in an environment and that environment has its own pip.
conda create -n test python=3.7
conda activate test
python -m pip install PACKAGE_NAME
From your described problem, I can guess that your environment is not activated in which you are trying to install the tensorflow2.0
Please make sure to activate the environment after making it.
so after creating the environment do this-
conda activate tf2.0
make sure you see this
(tf2.0) C:\Users\XYZ>
And then you install your tensorflow.
I don't have sudo access to the remote pc where cuda is already installed. Now, I have to install tensorflow-gpu on that system. Please give me the step by step guide to install it without sudo.
Operating System : Ubuntu 18.04
I had to do this before. Basically, I installed miniconda (you can also use anaconda, same thing and installation works without sudo), and installed everything using conda.
Create my environment and activate it:
conda create --name myenv python=3.6.8
conda actiavate myenv
Install the CUDA things and Tensorflow
conda install cudatoolkit=9.0 cudnn=7.1.2 tensorflow-gpu
Depending on your system, you may need to change version numbers.
Not sure how familiar you are with conda - it is basically a package-manager/repository and environment manager like pip/venv with the addition that it can handle non-python things as well (such as cudnn for example). As a note - if a package is not availabe through conda, you can still use pip as a fallback.
Untested with pip
I previously tried to do it without conda and using pip (I ended up failing due to some version conflicts, got frustrated with the process and moved to conda). It gets a little more complicated since you need to manually install it. So first, download cudnn from nvidia and unpack it anywhere you want. Then, you need to add it to the LD_LIBRARY_PATH:
export LD_LIBRARY_PATH=/path/to/cuda/lib64:/path/to/cudnn/lib64/:${LD_ LIBRARY_PATH}
In order to install matplotlib in Debian 9 should I issue
sudo apt-get install python3-matplotlib
or
pip3 install matplotlib?
What is the preferred way? Let me add more detail.
'man pip3' says
pip is a Python package installer, recommended for installing Python
packages which are not available in the Debian archive.
matplotlib confirms this
To install Matplotlib at the system-level, we recommend that you use
your distribution's package manager. This will guarantee that
Matplotlib's dependencies will be installed as well.
According to this I shall run apt-get. However, I had a similar case with numpy. 'import numpy' worked after 'pip3 install numpy'.
Using
Debian GNU/Linux 9.6 (stretch)
$ python3 -V
Python 3.5.3
The key quote from man pip3 is:
recommended for installing Python packages which are not available in the Debian archive.
You can check whether a package is available in the Debian archive by issuing an apt search command, eg.
apt search matplotlib
If your system already has all the necessary dependencies then installing a package with pip will work (as you experienced with pip install numpy).
If you do not have the necessary dependencies your distribution’s package manager will install them automatically. However, this is not true of pip: you would need to resolve any dependency issues manually. In the case of matplotlib, you can find a list of the dependencies you'd need to install manually here.
Essentially, using your system package manager to install a package takes away the extra work of ensuring you have all the necessary dependencies for that package.
pip comes into its own if you desire to work with virtual environments.
For the first time I'v installed tensorflow with conda installation. Then I actually work with a seq2seq model. After that I have again installed the tensorflow with the pip installation. But now the libraries are very different. All the old scripts are misplaced etc. Why is that ? Why I didn't face this when I was working with coda instillation
It has been claimed that Tensorflow installed with Conda performs a lot faster than a Pip installation, for example:
https://towardsdatascience.com/stop-installing-tensorflow-using-pip-for-performance-sake-5854f9d9eb0c
Conda also installs all of the package dependencies automatically, which Pip does not, as far as I'm aware.
https://www.anaconda.com/blog/developer-blog/tensorflow-in-anaconda/
Pip and conda install to two different locations. You should try to stick to one or the other. I would recommend uninstalling the conda version and sticking to pip but it's up to you how to proceed.
Update 01-02-2019: It seems that conda is now the faster and preferred way to install tensorflow. Note this may change again in the future.