Lxml import issues when using Scrapy - scrapy

I am trying to use Scrapy with Anaconda/Miniconda on Windows 10. Installation goes fine, but trying to actually run Scrapy gives the following error:
Traceback (most recent call last):
File "C:\ProgramData\Miniconda3\Scripts\scrapy-script.py", line 6, in <module>
from scrapy.cmdline import execute
File "C:\ProgramData\Miniconda3\lib\site-packages\scrapy\__init__.py", line 34, in <module>
from scrapy.spiders import Spider
File "C:\ProgramData\Miniconda3\lib\site-packages\scrapy\spiders\__init__.py", line 10, in <module>
from scrapy.http import Request
File "C:\ProgramData\Miniconda3\lib\site-packages\scrapy\http\__init__.py", line 11, in <module>
from scrapy.http.request.form import FormRequest
File "C:\ProgramData\Miniconda3\lib\site-packages\scrapy\http\request\form.py", line 11, in <module>
import lxml.html
File "C:\ProgramData\Miniconda3\lib\site-packages\lxml\html\__init__.py", line 53, in <module>
from .. import etree
ImportError: DLL load failed: The specified module could not be found.
I have tried reinstalling Scrapy, lxml, and Anaconda itself (this time, I'm using a clean install of Miniconda), as well as downloading unofficial lxml build from https://www.lfd.uci.edu/~gohlke/pythonlibs/, as suggested in one of the answers on Stack Overflow, but the problem persists. I have also done this on an Amazon AWS EC2 instance started from scratch, but I'm getting the same issue.
It seems to be something relatively common, but I couldn't find an answer that would work for me. What's an appropriate way to address this? Is it just about lxml, or is there something else causing this problem?

Related

using snakemake.utils in profile without using conda-not-block-search-path-envvars

I just updated from snakemake 5.28 to 5.32, and now from snakemake.utils import read_job_properties throws an import error for my job submission script in my snakemake profile:
Traceback (most recent call last):
File "/ebio/abt3_projects/software/dev/ll_pipelines/llmgqc/bin/ll_pipeline_utils/profiles/sge/sge-submit.py", line 7, in <module>
from snakemake.utils import read_job_properties
ModuleNotFoundError: No module named 'snakemake'
Using --conda-not-block-search-path-envvars prevents this import error, but I'd rather not have to use --conda-not-block-search-path-envvars every time I run snakemake.
I don't see anything special that the SLURM snakemake profile uses to get around this import issue, so I really don't know what to do besides using --conda-not-block-search-path-envvars all of the time.

beautifulSoup 4 works on python2.6?

I'm trying to do a web scraping project using bs4, but on the remote server they have python 2.6.6 installed.
Can I work with bs4 on python2.6.6?
I already tried but got these errors:
Traceback (most recent call last):
File "/home/infoforense/public_html/citius-scraping.py", line 16, in <module>
from bs4 import BeautifulSoup
File "/usr/lib/python2.6/site-packages/bs4/__init__.py", line 34, in <module>
from .builder import builder_registry, ParserRejectedMarkup
File "/usr/lib/python2.6/site-packages/bs4/builder/__init__.py", line 7, in <module>
from bs4.element import (
File "/usr/lib/python2.6/site-packages/bs4/element.py", line 12, in <module>
import soupsieve
File "/usr/lib/python2.6/site-packages/soupsieve/__init__.py", line 29, in <module>
from .__meta__ import __version__, __version_info__ # noqa: F401
File "/usr/lib/python2.6/site-packages/soupsieve/__meta__.py", line 190, in <module>
__version__ = __version_info__._get_canonical()
File "/usr/lib/python2.6/site-packages/soupsieve/__meta__.py", line 146, in _get_canonical
ver = "{}.{}.{}".format(self.major, self.minor, self.micro)
ValueError: zero length field name in format
I am the author of the failing dependency soupsieve, and can verify that soupsieve does not support Python 2.6.
It is possible that that an earlier version of Beautiful Soup (before 4.7) does fully support Python 2.6, but I'm not sure.
Python 2.6 is such an old version that even the Python team has stopped supporting it.
You could try installing older versions of Beautiful Soup and see if one of them allow you to move forward, but I can confirm that the latest versions, at least 4.7 and above, will not work on Python 2.6.

How to fix "calibration_pb2 from 'object_detection.protos' " error (Windows)

I've tried to run the below code but it always gives a set of errors. I tried searching the answers but none work for my code, there are two files named 'object_detection' one in the research folder and other in the object_detection-0.1-py3.7.egg folder which might be causing the error but i tried to change the path but the errors still persist
I'm trying to execue this command:
C:\tensorflow1\models\research\object_detection>python train.py --
logtostderr --train_dir=training/ --
pipeline_config_path=training/faster_rcnn_inception_v2_pets.config
but have the following errors coming up:
Traceback (most recent call last):
1. File "train.py", line 51, in <module>
from object_detection.builders import model_builder
2. File "C:\Users\Swayam\mypython\lib\site-packages\object_detection-
0.1-
py3.7.egg\object_detection\builders\model_builder.py", line 27, in
<module>
from object_detection.builders import post_processing_builder
3. File "C:\Users\Swayam\mypython\lib\site-packages\object_detection-
0.1-
py3.7.egg\object_detection\builders\post_processing_builder.py",
line 2
2, in <module>
from object_detection.protos import post_processing_pb2
4. File "C:\Users\Swayam\mypython\lib\site-packages\object_detection-
0.1-
py3.7.egg\object_detection\protos\post_processing_pb2.py", line 15,
in
<module>
from object_detection.protos import calibration_pb2 as
object__detection_dot_protos_dot_calibration__pb2
5. ImportError: cannot import name 'calibration_pb2' from
'object_detection.protos' (C:\Users\Swayam\mypython\lib\site-
packages\object_detection-0.1-
py3.7.egg\object_detection\protos\__init__.py)
I've tried using the
protoc object_detection/protos/*.proto --python_out=.
command but it brings up errors too.
Also, the environment is not made in conda, could that be the cause of the error? Though all the necessary installations are present in the existing virtual environment.
Try this Solution:
Check if the file
"calibration_pb2.py"
is located in the following path, in your case may be this one:
C:\Users\Swayam\mypython\lib\site-packages\object_detection-0.1-
py3.7.egg\object_detection\protos\
If not, just copy it from your working path:
C:\tensorflow1\models\research\object_detection\protos\
If it works, I sugggest you try to copy all the *pb2.py files into the path mentioned above.
you just compile this
protoc --python_out=. .\object_detection\protos\anchor_generator.proto .\object_detection\protos\argmax_matcher.proto .\object_detection\protos\bipartite_matcher.proto .\object_detection\protos\box_coder.proto .\object_detection\protos\box_predictor.proto .\object_detection\protos\eval.proto .\object_detection\protos\faster_rcnn.proto .\object_detection\protos\faster_rcnn_box_coder.proto .\object_detection\protos\grid_anchor_generator.proto .\object_detection\protos\hyperparams.proto .\object_detection\protos\image_resizer.proto .\object_detection\protos\input_reader.proto .\object_detection\protos\losses.proto .\object_detection\protos\matcher.proto .\object_detection\protos\mean_stddev_box_coder.proto .\object_detection\protos\model.proto .\object_detection\protos\optimizer.proto .\object_detection\protos\pipeline.proto .\object_detection\protos\post_processing.proto .\object_detection\protos\preprocessor.proto .\object_detection\protos\region_similarity_calculator.proto .\object_detection\protos\square_box_coder.proto .\object_detection\protos\ssd.proto .\object_detection\protos\ssd_anchor_generator.proto .\object_detection\protos\string_int_label_map.proto .\object_detection\protos\train.proto .\object_detection\protos\keypoint_box_coder.proto .\object_detection\protos\multiscale_anchor_generator.proto .\object_detection\protos\graph_rewriter.proto .\object_detection\protos\calibration.proto
it will resolve the issue

Twisted work with Python 3.3?

i use py3.3.2 and Twisted 15 on win7 and get this error
Traceback (most recent call last):
File "C:\Users\coder\Desktop\s.py", line 1, in <module>
from twisted.internet import protocol, reactor, endpoints
File "D:\Python33\lib\site-packages\twisted\internet\protocol.py", line 18, in <module>
from twisted.internet import interfaces, error, defer
File "D:\Python33\lib\site-packages\twisted\internet\defer.py", line 29, in <module>
from twisted.python import lockfile, log, failure
File "D:\Python33\lib\site-packages\twisted\python\lockfile.py", line 52, in <module>
_open = file
NameError: name 'file' is not defined
>>>
Twisted does not currently have continuous integration for Windows on Python 3; the only supported platform with Python 3 test coverage is Ubuntu. Please file a bug at https://twistedmatrix.com/.
In the meanwhile, please use Python 2.7 for Windows to use Twisted there.

cannot import name PyMCObjects when import pymc

I have pymc installed, and I am now trying to import it in my code. However,
First when I do import pymc, I got the error message:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "//anaconda/lib/python2.7/site-packages/pymc-2.3.3-py2.7-macosx-10.9-x86_64.egg/pymc/__init__.py", line 30, in <module>
from .CommonDeterministics import *
File "//anaconda/lib/python2.7/site-packages/pymc-2.3.3-py2.7-macosx-10.9-x86_64.egg/pymc/CommonDeterministics.py", line 21, in <module>
from .utils import safe_len, stukel_logit, stukel_invlogit, logit, invlogit, value, find_element
File "//anaconda/lib/python2.7/site-packages/pymc-2.3.3-py2.7-macosx-10.9-x86_64.egg/pymc/utils.py", line 14, in <module>
from . import flib
ImportError: numpy.core.multiarray failed to import
which can be fixed by manually import numpy.core.multiarray, but then, when I try again to import pymc, I got another error message:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "//anaconda/lib/python2.7/site-packages/pymc-2.3.3-py2.7-macosx-10.9-x86_64.egg/pymc/__init__.py", line 30, in <module>
from .CommonDeterministics import *
File "//anaconda/lib/python2.7/site-packages/pymc-2.3.3-py2.7-macosx-10.9-x86_64.egg/pymc/CommonDeterministics.py", line 13, in <module>
from . import PyMCObjects as pm
ImportError: cannot import name PyMCObjects
I googled around, and the solutions to similar problems are either:
In case that's related to the installation of pymc, force using the 32-bit instead of 64. However, this answer is for Windows users, and I have OS X 10.9 (Mavericks).
In other cases, it's usually due to a circular dependency (for example, here), which I don't think is the case here.
My guess is that something is wrong with the package flib, but I couldn't figure out what. I would appreciate any answer/hint on this problem. Thanks in advance!
Update (July-4-2014)
Thanks, Chris! I rebuilt my pymc using the conda build by doing conda install -c https://conda.binstar.org/tobeplugged pymc, and now when I do import pymc, a different error message appears:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "//anaconda/lib/python2.7/site-packages/pymc-2.3.3-py2.7-macosx-10.9-x86_64.egg/pymc/__init__.py", line 44, in <module>
from . import CommonDeterministics
ImportError: cannot import name CommonDeterministics
As Chris pointed out (see comments below), the problem(s) is being looked at here. The problem seems to have to do with the version of gcc (version 4.9). I will try to rebuild my gcc with the older version (4.8.2) later today to see if that will solve the problem for OS X 10.9 (Mavericks). Otherwise, I believe people can wait a little bit for the pymc team to solve this problem, too.