I've setup a conda environment and used conda install numpy. This includes the mkl package, which is quite large, 100s of MBs. Conda installs these into Library/bin in the environment when on Windows. In another conda environment I installed numpy+mkl from Christoph Gohlke's binaries. What's the difference between these two numpy builds that are both linked against MKL? Is there a preferred version to use?
The install from Christoph's binaries don't put all the mkl_* DLLs in Library/bin. Christoph's website says all the MKL libraries are included in the numpy.core folder after the pip install. I checked and there are a few DLL files in there, but none with the mkl_* prefix.
The conda version ends up using 100s of MBs more than the other install so I'm curious what benefits there are to using the Conda version.
Related
While I am in a conda environment, the 'conda list' and 'pip freeze' show different number of libraries. For example, 'tensorflow-gpu' is listed in 'pip freeze', but not in 'conda list'. If I want to use tensorflow-gpu in this environment, should I run pip install tensorflow-gpu to install it again, or not necessary?
I think when you are using the conda environment. The conda list is going to show all the general packages that shared by the same conda environment. And the reason why 'tensorflow-gpu' is listed in 'pip freeze', but not in 'conda list', is because you used pip install to installed 'tensorflow-gpu'(could be you or the IDE). In this case, 'tensorflow-gpu' is only exists under this python project I believe. Actually, there is an official document about this topic.
Issues may arise when using pip and conda together. When combining
conda and pip, it is best to use an isolated conda environment. Only
after conda has been used to install as many packages as possible
should pip be used to install any remaining software. If modifications
are needed to the environment, it is best to create a new environment
rather than running conda after pip. When appropriate, conda and pip
requirements should be stored in text files.
Use pip only after conda Install as many requirements as possible with
conda then use pip.
Pip should be run with --upgrade-strategy only-if-needed (the
default).
Do not use pip with the --user argument, avoid all users installs.
And here is the link.
My main Python environment on OSX Big Sur is the base env from Anaconda.
Now, I regularly need to install packages that are not available via conda or conda-forge or not in the latest version.
I understand that there are several different approaches:
Using the pip package installed by conda and follow the preferred way of using pip within conda i.e. to add those packages at the end of a environment.yaml file and to delete and re-generate the env whenever a new package needs to be added, in order to avoid conflicts from the interplay between conda and pip.
Forking the latest available recipe as described here, updating the version number and details, and using my local version of the recipe.
Simply installing the packages via pip, npm (Node JS package manager), apm (Atom package manager) from within the activated conda environment.
Which approach is best? Is it safe to use pip with conda in the way described in #1?
#2 is more effort than the other two. But I am looking for a generic approach that will not mess up my installation.
For scientific computing use cases, it is amazing that Anaconda distribution has MKL optimization for libraries like numpy, scipy etc.
I am wondering what if I use Miniconda instead, and install numpy via conda command, does it ship the same benefits of MLK optimization as what Anaconda does ?
Yes, it ship the same benefits of MLK optimization as what Anaconda does.
These Miniconda installers contain the conda package manager and Python. Once Miniconda is installed, you can use the conda command to install any other packages and create environments, etc.
For example:
$ conda install numpy
...
$ conda create -n py3k anaconda python=3
...
Note: If you already have Miniconda or Anaconda installed, and you just want to upgrade, you should not use the installer. Just use conda update.
For instance:
$ conda update conda
will update conda.
Yes, it is exactly the same package as would come with the Anaconda installer.
Note that you can also install NumPy from the conda-forge channel, if you wanted to use OpenBLAS instead of MKL.
I created a new conda environment as follows
conda create -n NAME python=3.5
Added edit, of course activate the environment like so
activate NAME
Installed these libraries all at once
conda install numpy scipy pandas scikit-learn jupyter matplotlib
relevant libary versions that conda installed
when importing pandas ge
ImportError: Missing required dependencies ['numpy']
Can anyone replicate this installation of env on windows? I am thinking one of the files version are causing this and cannot find a solution, barring an externality.
Have uninstalled and installed pandas, numpy, etc all libraries, have also tried installing with pip even, nothing is working. Have even created another environment from scratch same error is occurring.
Conda should have installed a number of libraries when you build python 3.5 in the new environment. If you run "conda list" in the command line of the newly built environment it will show the packages you have by default. Try to build the environment with "conda create -n py35 python=3.5 anaconda" so it uses the anaconda python and will auto install packages.
For the first time I'v installed tensorflow with conda installation. Then I actually work with a seq2seq model. After that I have again installed the tensorflow with the pip installation. But now the libraries are very different. All the old scripts are misplaced etc. Why is that ? Why I didn't face this when I was working with coda instillation
It has been claimed that Tensorflow installed with Conda performs a lot faster than a Pip installation, for example:
https://towardsdatascience.com/stop-installing-tensorflow-using-pip-for-performance-sake-5854f9d9eb0c
Conda also installs all of the package dependencies automatically, which Pip does not, as far as I'm aware.
https://www.anaconda.com/blog/developer-blog/tensorflow-in-anaconda/
Pip and conda install to two different locations. You should try to stick to one or the other. I would recommend uninstalling the conda version and sticking to pip but it's up to you how to proceed.
Update 01-02-2019: It seems that conda is now the faster and preferred way to install tensorflow. Note this may change again in the future.