I cannot upgrade my package from private gitlab.
I share a project on a gitlab hosted in my company local network.
Until now, I used to simply upgrade my package with
> pip install -U myPackage
But now, pip tells me my package is up-to-date:
> pip install -U myPackage
Requirement already satisfied: myPackage in <some_windows_path> (1.16.5)
I tried to figure it out and found that pip does not search on the gitlab whereas pip show indicates the right home-page.
So I tried to force pip to take into account the gitlab with some options (--extra-index-url, --find-links).
I also tried to reinstall it from void with (git+https).
In both case I get SSL error.
Since I have no issue with git, I retrieve the certificate used by Git (git config -l) and tried to specified the certificate to pip (--cert).
But it was one more fail.
> uname -a
Windows_NT <some_machine_name> 6 3 x86
> python -V
Python 3.10.4
> pip -V
pip 22.2.2 from <some_windows_path> (python 3.10)
> git -v
git version 2.37.3.windows.1
The issue is linked to SSL certification.
Since I have no issue with Git, I guess we can exclude proxy as a cause.
I must miss something here but what?
> pip install --dry-run --client-cert <windows_path_of_git_config_http.sslcainfo> -U git+https:<gitlab_url>
Collecting git+https:<gitlab_url>
Cloning https:<gitlab_url> to c:\users\me\appdata\local\temp\pip-req-build-m6p4_4ve
Running command git clone --filter=blob:none --quiet https:<gitlab_url> 'C:\Users\me\AppData\Local\Temp\pip-req-build-m6p4_4ve'
fatal: unable to access 'https:<gitlab_url>': SSL certificate problem: unable to get local issuer certificate
error: subprocess-exited-with-error
× git clone --filter=blob:none --quiet https:<gitlab_url> 'C:\Users\me\AppData\Local\Temp\pip-req-build-m6p4_4ve' did not run successfully.
│ exit code: 128
╰─> See above for output.
note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error
× git clone --filter=blob:none --quiet https:<gitlab_url> 'C:\Users\me\AppData\Local\Temp\pip-req-build-m6p4_4ve' did not run successfully.
│ exit code: 128
╰─> See above for output.
note: This error originates from a subprocess, and is likely not a problem with pip.
I'm trying to run this notebook on Google Cloud Datalab: https://github.com/GoogleCloudPlatform/training-data-analyst/blob/master/courses/machine_learning/feateng/feateng.ipynb
While it was perfectly working yesterday, today running the first block results in cffi.error.VerificationError (undefined symbol: SSLv2_client_method).
Can you advise on how I can fix this?
I've tried to do the same from a different GC profile and the problem remains.
I've also tried to fix 'pip install' as described here: pip install fails with "connection error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:598)". Doing this in Datalab causes the same error; doing this in Cloud shell outside Datalab doesn't change anything.
This has been answered on github by #ekuuni:
https://github.com/GoogleCloudPlatform/training-data-analyst/issues/313
%%bash
source activate py2env
conda install -y pytz
conda update -y pyopenssl
pip uninstall -y google-cloud-dataflow
pip install --upgrade apache-beam[gcp]
I needed these 2 more lines for it to work.
pip install tensorflow_transform
pip install tensorflow==1.9.0
A good workaround is using virtual environments rather than Datalab so you don't have to deal with version updates:
https://cloud.google.com/dataflow/docs/quickstarts/quickstart-python
I have successfully installed the following on python 2.7
apache-beam==2.7.0
tensorflow==1.11.0
tensorflow-transform==0.8.0
I updated my mac to high sierra, and now I can't install pycurl. It fails with this message : Curl is configured to use SSL, but we have not been able to determine which SSL backend it is using. Please see PycURL documentation for how to specify the SSL backend manually.
I searched on the documentation and the web and I found some solution that not fix my problem. the most popular is this one :
pip uninstall pycurl
export PYCURL_SSL_LIBRARY=openssl
pip install pycurl
here is the complete error
A solution similar to the one you found worked for me when issued from within my virtualenv. I use Homebrew as a package manager on macOS High Sierra, and Pipenv to manage my project dependencies and virtualenv. The error emerged after adding the PyVimeo API Library, which has PycURL as a dependency, to my project.
The generated errors were, first,
src/pycurl.c:137:4: warning: #warning "libcurl was compiled with SSL
support, but configure could not determine which library was used;
thus no SSL crypto locking callbacks will be set, which may cause
random crashes on SSL requests" [-Wcpp]
then,
ImportError: pycurl: libcurl link-time ssl backend (openssl) is
different from compile-time ssl backend (none/other)
As mentioned in the PycURL docs, the solution was to "tell [PycURL's] setup.py what SSL backend is used." Setting the environment variables recommended in the output of brew info openssl, alone, did not solve the problem.
Then I found a tangentially related Github issue comment and tried the following from within my project's virtualenv:
(env)$ pip uninstall pycurl
(env)$ pip install --upgrade pip
(env)$ export LDFLAGS=-L/usr/local/opt/openssl/lib
(env)$ export CPPFLAGS=-I/usr/local/opt/openssl/include
(env)$ export PYCURL_SSL_LIBRARY=openssl
(env)$ pip install pycurl
The install command gave this output:
Collecting pycurl Using cached
https://files.pythonhosted.org/packages/e8/e4/0dbb8735407189f00b33d84122b9be52c790c7c3b25286826f4e1bdb7bde/pycurl-7.43.0.2.tar.gz
Building wheels for collected packages: pycurl Running setup.py
bdist_wheel for pycurl ... done Stored in directory:
/Users/me/Library/Caches/pip/wheels/d2/85/ae/ebf5ff0f1378a69d082b4863df492bf54c661bf6306a2bd
Successfully built pycurl
tuspy 0.2.1 has requirement pycurl==7.43.0,
but you'll have pycurl 7.43.0.2 which is incompatible. Installing
collected packages: pycurl Successfully installed pycurl-7.43.0.2
I noted the (somewhat petty?) tuspy error and trudged on. This time, my script ran without PycURL complaining.
Pip always fails ssl even when I do pip install dedupe or pip install --trusted-host pypi.python.org dedupe
The output is always the same no matter what:
Collecting dedupe
Retrying (Retry(total=4, connect=None, read=None,
redirect=None, status=None)) after connection broken by
'SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate
verify failed (_ssl.c:777)'),)': /simple/dedupe/
Retrying...
skipping
Could not find a version that satisfies the requirement dedupe (from versions: ) No matching distribution found for dedupe
So I uninstalled anaconda and reinstalled it. Same thing.
Do you think the problem is that my _ssl.c file (which I have no idea where it is) must be corrupt or something? Why would pip need to reference that if I'm telling it to bypass ssl verification anyway?
It may be related to the 2018 change of PyPI domains.
Please ensure your firewall/proxy allows access to/from:
pypi.org
files.pythonhosted.org
So you could give a try to something like:
$ python -m pip install --trusted-host files.pythonhosted.org --trusted-host pypi.org --trusted-host pypi.python.org [--proxy ...] [--user] <packagename>
Please see $ pip help install for the --user option description (omit if in a virtualenv).
The --trusted-host option doesn't actually bypass SSL/TLS, but allows to mark host as trusted when (and only when) it does not have valid (or any) HTTPS. It shouldn't really matter with PiPY because pypi.org (formerly pypi.python.org) does use HTTPS and there is CDN in front of it which always enforces TLSv1.2 handshake requirement regardless of the connecting pip client options.. But if you had your own local mirrors of pypi.org with HTTP-only access, then --trusted-host could be handy. Oh, and if you are behind a proxy, please also make sure to also specify: --proxy [user:passwd#]proxyserver:port
Some corporate proxies may even go as far as to replace the certificates of HTTPS connections on the fly. And if your system clock is out of sync, it could break SSL verification process as well.
If firewall / proxy / clock isn't a problem, then check SSL certificates being used in pip's SSL handshake. In fact, you could just get a current cacert.pem (Mozilla's CA bundle from curl) and try it using the pip option --cert:
$ pip --cert ~/cacert.pem install --user <packagename>
where --cert argument is system path to your alternate CA bundle in PEM format. (regarding the --user option, please see below).
Or, it's possible to create a custom config ~/.pip/pip.conf and point the option at a valid system cert (or your cacert.pem) as a workaround, for example:
[global]
cert = /etc/pki/tls/external-roots/ca_bundle.pem
(or another pem file)
It's even possible to manually replace the original cacert.pem found in pip with your trusty CA bundle (if your pip is very old for example). Older pip versions knew to fallback between pip/_vendor/requests/cacert.pem and system stores like /etc/ssl/certs/ca-certificates.crt or /etc/pki/tls/certs/ca-bundle.crt in case of cert issues, but in recent pip it's no longer the case, as it seems to rely solely on pip/_vendor/certifi/cacert.pem
Basically, pip package uses requests which uses urllib3 which, among other things, verifies SSL certificates; and all of them are shipped (vendored) within pip, along with the certifi package (also included, since pip 9.0.2) that provides current CA bundle (cacert.pem file) required for TLS verification. Requests itself uses urllib3 and certifi internally, and before 9.0.2, pip used cacert.pem from requests or the system. What it all means is that actually updating pip may help fix the CERTIFICATE_VERIFY_FAILED error, particularly if the OS and pip were deployed long ago:
The OP used anaconda, so they could try:
$ conda update pip - because issues can arise if conda and pip are both used together in the same environment. If there's no pip version update available, they could try:
$ conda config --add channels conda-forge; conda update pip
Alternatively, it's possible to use conda alone to directly install / manage python packages: it is a tool completely separate from pip, but provides similar features in terms of package and venv management. Its packages come not from PyPI, but from anaconda's own repositories.
The problem is, if you mix both and run conda after pip, the former can overwrite and break packages (and their dependencies) installed via pip, and render it all unusable. So it's recommended to only use one or the other, or, if you have to, use only pip after conda (and no conda after pip), and only in isolated conda environments.
On normal Linux Python installations without conda:
If you are using a version of pip supplied by your OS distribution, then use vendor-supplied upgrades for a system-wide pip update:
$ sudo apt-get install python-pip or: $ sudo yum install python27-pip
Some updates may not be readily available because distros usually lag behind PyPI. In this case, it's possible to upgrade pip at your user level (right in your $HOME dir), or inside a virtualenv, like:
$ python -m pip install --user --trusted-host files.pythonhosted.org --trusted-host pypi.org --trusted-host pypi.python.org --upgrade pip
(omit --user if in a virtualenv)
The --user switch will upgrade pip only for the current user (in your home ~/.local/lib/) rather than for the whole OS, which is a good practice to avoid interfering with the system python packages. It's enabled by default in a pip distributed in recent Ubuntu/Fedora versions. Be aware of how to solve ImportError if you don't use this option and happen to overwrite the OS-level system pip.
Alternatively (also at a user level) you could try:
$ curl -LO https://bootstrap.pypa.io/get-pip.py && python get-pip.py --user
The PyPA script contains a wrapper that extracts the .pem SSL bundle from pip._vendor.certifi.
Otherwise, if still no-go, try running pip with -vvv option to add verbosity to the output and check if there is now another SSLError caused by tlsv1 alert protocol version.
This worked for me, try this:
pip install --trusted-host=pypi.org --trusted-host=files.pythonhosted.org --user {name of whatever I'm installing}
My way is a simplification of #Alex C's answer:
python -m pip install --trusted-host pypi.python.org --trusted-host files.pythonhosted.org --trusted-host pypi.org --upgrade pip
I experienced the same issue because I have Zscaler (a cloud security software) installed and was causing:
URL host for python packages being blocked
invalid SSL certificate warnings popping up
SSL inspection certificate not trusted
As mentioned by others, the below will fix individual package installations. pypi.python.org is not required since it has been replaced by pypi.org.
pip install --trusted-host pypi.org --trusted-host files.pythonhosted.org <package to install>
I permanently fixed the issue by creating pip.ini file (pip.conf in Unix) and adding the below:
[global]
trusted-host = pypi.python.org
pypi.org
files.pythonhosted.org
See pip configuration files for how to locate your pip.ini, or where to put it if you need to create one.
The error above or one like it was caused by the virtual machine (VM) not be time synchronized, my guest Ubuntu VM was several days in the past.
I ran this commend to get the VM to pick up the correct network time:
sudo timedatectl set-ntp on
This makes the Ubuntu guest OS get the network time. (You may have to provide a network time source... I used this article: Digital Ocean - How to set time on Ubuntu)
Check the time is correct:
timedatectl
Re-run the failing pip command.
When doing:
$ sudo pypy -m easy_install lxml
The response is:
Searching for lxml
[...snip...]
ERROR: /bin/sh: 1: xslt-config: not found
** make sure the development packages of libxml2 and libxslt are installed **
Using build configuration of libxslt
/usr/lib/pypy/lib-python/2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'bugtrack_url'
warnings.warn(msg)
warning: no files found matching '*.txt' under directory 'src/lxml/tests'
src/lxml/lxml.etree.c:8:22: fatal error: pyconfig.h: No such file or directory
compilation terminated.
error: Setup script exited with error: command 'cc' failed with exit status 1
At the same time, sudo pip install lxml works fine.
What's going on?
Thanks.
sudo apt-get install python-dev fixed it for me on ubuntu 13.04
$yum install python-lxml or apt-get install python-lxml
this solved mine.
I've stumbled with this trouble a couple of times.
Short answer
Python2: $ python2.7 setup.py clean build --with-cython install
Python3: $ pip-3.3 install lxml
Long answer
The hypothesis is that pip install lxml should work in every environment, regardless if you are using Python2 or Python3.
There's also Cython to be considered: You will certainly enjoy lxml compiled with Cython due to relevant performance gains.
For reasons unknown to me, the compilation on Python2 does not find Cython.
To be more precise and absolutely explicit about this matter, both commands below DO NOT employ Cython:
# DO NOT use these commands. I repeat: DO NOT use these commands.
$ pip-2.7 install lxml
$ easy_install-2.7 install lxml
So, when using Python2 you have only one alternative, as far as I know, which is: compile from sources, Luke!
# install build environment and dependencies
$ kernel_release=$( uname -r )
$ sudo apt-get install linux-headers-${kernel_release} build-essential -y
$ sudo apt-get install libxml2-dev libxslt1-dev -y
# Download from github and compile from sources
$ git clone --branch lxml-3.2.4 https://github.com/lxml/lxml
$ python2.7 setup.py clean build --with-cython install
I've handled this problem by installed Ubuntu package pypy-dev.