Scrapy - url open error [Errno 61] Connection refused - scrapy

I am trying to deploy a project in Osx (Osx El Capitan 10.11.6), but I get the following error.
Deploying to project "webofscience" in
http://localhost:6800/addversion.json
Deploy failed:
Could they be administrator permissions ?, add some line in the host file ??
I can not think what I could do to solve it. Anyone have any suggestions on how to fix the error ??
source code: https://github.com/damlaozdemir/webofsciencescrapy
Console error
webofsciencescrapy-master sudo scrapyd-deploy local -p webofscience
Password:
Packing version 1518114723
Deploying to project "webofscience" in http://localhost:6800/addversion.json
Deploy failed: <urlopen error [Errno 61] Connection refused>
webofsciencescrapy-master
Solution:
I had 2 problems,
(1) The first one that my scrapyd installation was damaged. I threw the following error.
blablabla
File "/Library/Python/2.7/site-packages/twisted/protocols/tls.py", line 63, in <module>
from twisted.internet._sslverify import _setAcceptableProtocols
File "/Library/Python/2.7/site-packages/twisted/internet/_sslverify.py", line 38, in <module>
TLSVersion.TLSv1_1: SSL.OP_NO_TLSv1_1,
exceptions.AttributeError: 'module' object has no attribute 'OP_NO_TLSv1_1'
I solved it in this link https://github.com/scrapy/scrapy/issues/2473 summary
sudo pip install -U pip
sudo pip install --upgrade scrapy
sudo pip install --upgrade twisted
sudo pip install --upgrade pyopenssl
sudo pip install Twisted==16.4.1
(2) The second problem is 'Deploy failed: ', for this problem it's necessary run the command scrapyd (layer 8 problem).

Related

Jenkins : Pycharm : I keep getting a message to upgrade pip

Im trying to run automated test on jenkins cloud version : so i get this error :
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/tmp/pip-build-0739wibs/cryptography/setup.py", line 14, in <module>
from setuptools_rust import RustExtension
ModuleNotFoundError: No module named 'setuptools_rust'
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-0739wibs/cryptography/
You are using pip version 9.0.3, however version 22.0.3 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Import chromedriver on the env variable.
Import python path on the env variable.
/tmp/jenkins3940904514869226010.sh: line 16: pytest: command not found
Build step 'Execute shell' marked build as failure
Finished: FAILURE
I do `python3 -m pip install --upgrade pip
I get Requirement already satisfied: pip in ./venv/lib/python3.8/site-packages (22.0.3)
`
I have fixed it by adding this line on Execute shell
pip3 install --upgrade pip

How to fix geckodriver error when running InstaPy on raspberry pi OS?

I installed InstaPy recently and downloaded one standard script on the internet. I know that there is nothing wrong with opensource code and on forums it says to download geckodriver. However, I haven't found a suitable solution to install geckodriver so can you guys let me know the best way to fix this?
Error:
>>> %Run basic_follow_unfollow_activity.py
InstaPy Version: 0.6.13
._. ._. ._. ._. ._. ._. ._. ._.
Workspace in use: "/home/pi/InstaPy"
Error, unable to determine correct filename for 32bit linux
Traceback (most recent call last):
File "/home/pi/Desktop/basic_follow_unfollow_activity.py", line 36, in <module>
headless_browser=False)
File "/home/pi/.local/lib/python3.7/site-packages/instapy/instapy.py", line 338, in __init__
geckodriver_log_level,
File "/home/pi/.local/lib/python3.7/site-packages/instapy/browser.py", line 122, in set_selenium_local_session
driver_path = geckodriver_path or get_geckodriver()
File "/home/pi/.local/lib/python3.7/site-packages/instapy/browser.py", line 38, in get_geckodriver
sym_path = gdd.download_and_install()[1]
File "/home/pi/.local/lib/python3.7/site-packages/webdriverdownloader/webdriverdownloader.py", line 177, in download_and_install
show_progress_bar=show_progress_bar)
File "/home/pi/.local/lib/python3.7/site-packages/webdriverdownloader/webdriverdownloader.py", line 129, in download
download_url = self.get_download_url(version, os_name=os_name, bitness=bitness)
File "/home/pi/.local/lib/python3.7/site-packages/webdriverdownloader/webdriverdownloader.py", line 324, in get_download_url
raise RuntimeError(info_message)
RuntimeError: Error, unable to determine correct filename for 32bit linux
I tried following solutions:
This link on forum to install geckodriver - https://www.raspberrypi.org/forums/viewtopic.php?t=167292
Installed firefox ESR in terminal - sudo apt-get install firefox-esr
I've had similar problems to get the geckodriver running on my Pi. If i remember correctly there was a problem with the ARM support. I ended up switching to chromedriver.
you can easily install it with:
sudo apt install chromium-chromedriver
after that you can use it without any executable_path like this:
from selenium import webdriver
driver = webdriver.Chrome()
Installing Geckodrive instruction is given on instapy documentation through this link - https://github.com/InstaPy/instapy-docs/blob/master/How_Tos/How_to_Raspberry.md
GeckoDriver releases can be found in: https://github.com/mozilla/geckodriver/releases. The latest ARM release as of 2019-08-16 is v0.23.
Follow these steps:
1) wget https://github.com/mozilla/geckodriver/releases/download/v0.23.0/geckodriver-v0.23.0-arm7hf.tar.gz
2) tar -xvzf geckodriver-v*
3) chmod +x geckodriver
4) sudo cp geckodriver /usr/local/bin/
Ensure that /usr/local/bin is in your shell PATH
Install InstaPy
1) sudo apt-get install python3-pip
2) python3 -m pip install --user instapy
3) sudo reboot (optional)
Finishing up the Firefox installation
1) sudo pip3 install future
2) sudo apt-get install xvfb
3) sudo pip3 install pyvirtualdisplay

Error intsalling tensorflow via pip install

WARNING: Failed to write executable - trying to use .deleteme logic
ERROR: Could not install packages due to an EnvironmentError: [WinError 2] The system cannot find the file specified: 'c:\python38\Scripts\chardetect.exe' -> 'c:\python38\Scripts\chardetect.exe.deleteme'
I was trying to install Tensorflow via command pip install tensorflow. and after downloading everything at the end found this error..can anyone tell me why this happened?
According to https://www.codegrepper.com/ you can use these two:
In Windows
python -m pip install -U pip --user
In Linux
pip install -U pip --user

How to fix "ImportError: No module named error"

I installed elastalert from git clone https://github.com/Yelp/elastalert.git and integrated with slack tool but when I try to run elastalert using command python -m elastalert.elastalert --verbose --rule example_frequency.yaml I am getting this error:
Traceback (most recent call last):
File "runpy.py", line 174, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File "runpy.py", line 72, in _run_code
exec code in run_globals
File "/usr/local/lib/python2.7/dist-packages/elastalert-0.2.1-py2.7.egg/elastalert/elastalert.py", line 29, in <module>
from . import kibana
File "/usr/local/lib/python2.7/dist-packages/elastalert-0.2.1-py2.7.egg/elastalert/kibana.py", line 4, in <module>
import urllib.error
ImportError: No module named error
I expect output of getting alerts in slack tool
I just have the same problem. I used python 2.7 to install elastalert.
So I remove elastalert and reinstall it with python36
pip uninstall elastalert
sudo yum install python36 gcc python36-devel -y
sudo yum install python36-setuptools -y
sudo easy_install-3.6 pip
cd elastalert/
pip3.6 install "setuptools>=11.3"
python3.6 setup.py install
pip3.6 install -r requirements.txt
pip3.6 install elastalert
python36 -m elastalert.elastalert
Hope useful for you or someone that need

uninstall virtualenv in mac

I try to install tensorflow via virtualenv in mac, the reference is here:
for I have python already, so I install virtualenv by:
sudo pip install --upgrade virtualenv
build a new virtualenv environment in ~/development project/tensorflow, not in ~/tensorflow as is said in the reference:
virtualenv --system-site-packages ~/development project/tensorflow
cd ~/development project/tensorflow
active virtualenv by:
source bin/activate # if use bash,and I choose this line to run
when I am in virtualenv and run this:
(tensorflow)$ pip install --upgrade <$url_to_binary.whl>
it comes out an error:
zsh: parse error near '\n'
then I check here to help solve it
it says that <$url_to_binary.whl> should be replaced by https://storage.googleapis.com/tensorflow/mac/tensorflow-0.5.0-py2-none-any.whl, and I do so. And problem gone.
but, when I try to test tensorflow by cd some dir in tensorflow like:
(tensorflow)$ cd tensorflow/models/image/mnist
the dir is not exist.
so I guess the url I modified cause it, and I try to reinstall virtualenv and tensorflow, to make my install clean.
tensorflow is successfully uninstalled, but when I remove virtualenv, it booms:
Exception: Traceback (most recent call last): File
"/usr/local/lib/python2.7/site-packages/pip/basecommand.py", line 215,
in main
status = self.run(options, args) File "/usr/local/lib/python2.7/site-packages/pip/commands/uninstall.py",
line 76, in run
requirement_set.uninstall(auto_confirm=options.yes) File "/usr/local/lib/python2.7/site-packages/pip/req/req_set.py", line 346,
in uninstall
req.uninstall(auto_confirm=auto_confirm) File "/usr/local/lib/python2.7/site-packages/pip/req/req_install.py", line
754, in uninstall
paths_to_remove.remove(auto_confirm) File "/usr/local/lib/python2.7/site-packages/pip/req/req_uninstall.py",
line 115, in remove
renames(path, new_path) File "/usr/local/lib/python2.7/site-packages/pip/utils/init.py", line
267, in renames
shutil.move(old, new) File "/usr/local/Cellar/python/2.7.12_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py",
line 303, in move
os.unlink(src) OSError: [Errno 13] Permission denied: '/usr/local/lib/python2.7/site-packages/virtualenv-15.1.0.dist-info/DESCRIPTION.rst'
I delete dir tensorflow
for now I could not uninstall virtualenv neither install it, but it still exist....that's really wired.
If I run sudo pip install --upgrade virtualenv again, it says:
The directory '/Users/Calvino/Library/Caches/pip/http' or its parent
directory is not owned by the current user and the cache has been
disabled. Please check the permissions and owner of that directory. If
executing pip with sudo, you may want sudo's -H flag. The directory
'/Users/Calvino/Library/Caches/pip' or its parent directory is not
owned by the current user and caching wheels has been disabled. check
the permissions and owner of that directory. If executing pip with
sudo, you may want sudo's -H flag. Requirement already up-to-date:
virtualenv in /usr/local/lib/python2.7/site-packages
My question is that how could I remove virtualenv and tensorflow thoroughly and reinstall them.
I would be very appreciate if anyone could help.
have been two years but anyways yo need to uninstall with >
//Reinstal python 2 or 3 with new SO update
brew update
sudo installer -pkg /Library/Developer/CommandLineTools/Packages/macOS_SDK_headers_for_macOS_10.14.pkg -target /
brew reinstall python#3
Reference
then reinstall tensorflow with>
https://www.tensorflow.org/install/pip
Now, you just need to run:
(tensorflow)$ pip install --upgrade tensorflow # for Python 2.7
(tensorflow)$ pip3 install --upgrade tensorflow # for Python 3.n
And remove by: (this is for virtualenv, you change the path, following path also should be changed to ~/development/project/tensorflow)
$ rm -r ~/tensorflow
If you install tensorflow with native pip:
$ pip uninstall tensorflow
$ pip3 uninstall tensorflow
More information: https://www.tensorflow.org/install/install_mac