I'm not good at English to express, Sorry.
I want to know how to debug scrapy with vs2017.
I work well with vs2015,here is the way:
import os
os.system("run.cmd")
and run.cmd:
scrapy crawl weiqn --nolog
pause
Thanks.
Haha,now I found the way.
It is the same as Pycharm debugger.
Hear is pycharm:
enter image description here
Related
I am trying to do an END-to-END project for BostonHousing on VS Code. I have installed the requirements already from requirements.txt and also chose the Python interpreter as that of my virtual environment version still the error pops like below for Jupyter notebook. The flask file is fine. Please guide me on what's wrong here.
enter link description here
you may got your answer here.check and read carefully.
Import "[module]" could not be resolvedPylance (reportMissingImports)
I tried to use replit online platform for web scraping using selenium in Edge browser. replit automatically install selenium 4.1.3 but I couldn't find msedge-selenium-tools to support this version.
How can I install selenium (3.141) to support msedge-selenium-tools?
Any help will be highly appreciated.
Thank You
Bibek
I don't know much about this online IDE, but I did some testing and found that it seems to be able to control the version of the referenced package via a configuration in the pyproject.toml file.
First, uninstall selenium 4.1.3 in installed Packages. When installing edge-selenium-tools package, you will find that it bundles selenium 3.141 installed with edge-selenium-tools. Second, add this line in pyproject.toml file.
selenium = "^3.141.0"
Something like this:
In addition, for more questions related to IDE replit, I suggest you post your questions in its community, I think there may be someone with more relevant experience who can answer your questions.
I install selenium through Python pip and write a simple file called crawlTest.py, it works well.Later, I install Pycharm and try to auto complete method from selenium, but it doesn't work.
I guess Pycharm didn't know where I install selenium and an external library is needed.But I try by myself and search on google and StackOverflow, I didn't get proper solution.
I have read the flowing posts, but still not work. It is very useful and
time-saving for those who use selenium and Pycharm if some one could give me .
here is post list I hava viewed:
How to get PyCharm to auto-complete code in methods?
PyCharm doesn't auto-completes very simple cases such as: import pickle
I install selenium by python pip, and it works with python3.5，but not python 2.7. so I changed my project interpreter as python 3.5 in Py . Now it works!
for use auto-complete Pycharm feture go to "File\Setting\Project:" add new interpreter and select you instaleed python directory.
-Mac OSX 10.9 Mavericks -Python 2.7 -matplotlib 1.3
Hi,
I tried running matplotlib with the TKAgg backend, but ran into some issues, so I decided to try GTKAgg instead. I can do "import gtk", so I know GTK is properly installed on my machine. However, when I try "import matplotlib.pyplot as plt", I get the following error:
ImportError: No module named _backend_gdk
I went into my matplotlib/backends directory and couldn't find that module. It seems that I need to install the gtk dev package/headers for it to work properly. I've googled around and can't seem to find a good answer to how to go about doing this on Mac OS. Here are my questions:
1) I'm a bit unclear on the interplay between the gtk headers, gtk and matplotlib. Could someone briefly explain what exactly the headers do?
2) What is the best way to download this gtk dev package for Mac OS?
3) Will I need to uninstall and reinstall matplotlib so that it recognizes the headers?
Thank you
Mika
I am using phantomjs to build a web crawler. My current crawler is a Python script using selenium, which requires a running Firefox browser. While Selenium is good for debugging scripts (as I can use firebug to inspect the webpage), it cannot be deployed on linux servers without GUI. So I am trying to translate my Python script to phantomjs.
To debug phantomjs scripts, I save the page source html and render a png screenshot to disk, and open it in Firefox to inspect the source page. I am wondering if there is a better way to do this, e.g. a plugin for phantomjs, etc.
This little hack is using a simple technique: It grabs the screen as PhantomJS or CasperJS sees it with captureBase64('png') and then it is POSTing the image into the receiving server which then sends it via socket.io to the browser which displays it is as inline image.
The source code is available on github:
https://github.com/maciejjankowski/flaming-octo-puss
I'm not sure if PhantomJS has it, but CasperJS does (and the latter adds some extra functions to PhantomJS)
and use remote debugging as others suggest
As for most of the JS scripts, it's not so trivial to debug phantomjs script, because there is no IDE/compiler behind this.
First, I higly suggest you to read the Troubleshooting section.
As said by torazaburo, your best option is to use Remote Debugging : scripts can be run in a Web Inspector interface-like console : step-in, step-out, break points, local variables... many classic debugger features are available. If you're familliar with Chrome developpers tools, it's pretty the same.
Though not a solution to your Phantomjs troubles,I think Selenium with Python bindings can be used very efficiently as a headless scraper in a Linux environment.
You can use PyVirtualDisplay, a Python wrapper for Xvfb and Xephyr to fake a display.PyVirtualDisplay needs Xvfb as a dependency. On Ubuntu, first install Xvfb:
sudo apt-get install xvfb
then install PyVirtualDisplay from Pypi:
pip install pyvirtualdisplay
Sample Selenium script in Python in a headless mode with PyVirtualDisplay:
#!/usr/bin/env python
from pyvirtualdisplay import Display
from selenium import webdriver
display = Display(visible=0, size=(800, 600))
display.start()
# now Firefox will run in a virtual display.
# you will not see the browser.
browser = webdriver.Firefox()
browser.get('http://www.google.com')
print browser.title
browser.quit()
display.stop()