When I am trying to use Selenium I am getting an error as:
nvalidArgumentException: Message: invalid argument: user data directory is already in use, please specify a unique value for --user-data-dir argument, or don't use --user-data-dir
my code :
driver = webdriver.Chrome('chromedriver\chromedriver.exe')
# request url
driver.get('https://jamalon.com/ar/best-seller-books-2019')
#give it some time
sleep(2)
## retrive , download html page
html = driver.page_source
#close
driver.close()
i try to do same solution here but i get same error :(
InvalidArgumentException: Message: invalid argument: user data directory is already in use error using --user-data-dir to start Chrome using Selenium
First if you're using Pycharm then keep your chromedriver.exe in the location where your current python file is located.
Then use the below code:
from selenium import webdriver
import time
driver = webdriver.Chrome()
# request url
driver.get('https://jamalon.com/ar/best-seller-books-2019')
#give it some time
time.sleep(2)
## retrive , download html page
html = driver.page_source
print(html)
#close
driver.close()
Output:
Related
I am trying to use this code that works on my local machine on python anywhere and i want to understand if it is even possible:
from selenium import webdriver
from bs4 import BeautifulSoup
import time
# Initialize webdriver
driver = webdriver.Chrome(executable_path="/Users/matteo/Downloads/chromedriver")
# Navigate to website
driver.get("https://apnews.com/article/prince-harry-book-meghan-royals-4141be64bcd1521d1d5cf0f9b65e20b5")
time.sleep(5)
# Parse page source
soup = BeautifulSoup(driver.page_source, "html.parser")
# Find desired elements using Beautiful Soup
elements = soup.find_all("p")
# Print element text
for element in elements:
print(element.text)
# Close webdriver
driver.quit()
Do i need to have installed chrome to make that work or is chromium enough? Because when i run that code on my local machine a chrome page opens up. How does that work on python anywhere? Would it crush?
I am wondering if the code i am using only works if someone is on a GUI with Chrome installed or if it can work on python anywhere too.
The short answer is no. ChromeDriver must have chrome installed. You can run your tests headless for time save, but chrome still must be installed.
Selenium requires a driver to interface with the chosen browser. Firefox, for example, requires geckodriver, which needs to be installed before the below examples can be run.
after installing chrome driver for selenium in path directory
from time import sleep
from selenium import webdriver
# this is tested on Firefox or you can use "webdriver.Chrome()"
browser = webdriver.Firefox()
browser.get(‘https://www.facebook.com/')
sleep(5)
browser.close()
This error message...
SyntaxError: invalid character in identifier
...implies that there is a syntax error at line 5 i.e. in the url string you have passed through get() method.
As #cruisepandey correctly pointed out, you need to pass the url_string within single quotes i.e. '...':
browser.get('https://www.facebook.com/')
or within double quotes i.e. "...":
browser.get("https://www.facebook.com/")
I'm new at Python and I'm trying to use selenium webdriver to do web scraping from a webpage. When I run my code, I have not problems in obtaining the results I need, but when someone else tries to run the code from an executable it shows the error: selenium.common.exceptions.webdriverexception: message: 'chromedriver.exe' unexpectedly exited.status code was: 1. The path where I'm saving chromedriver.exe is a public repository. Can someone help me with this please? Here is the piece of code I'm using:
from selenium import webdriver
url= "https://www.byma.com.ar/obligaciones-negociables/"
driver = webdriver.Chrome(executable_path=r'\\path\\chromedriver.exe')
driver.implicitly_wait(30)
driver.get(url)
time.sleep(2)
You do not need to setup a driver like this :
driver = webdriver.Chrome(executable_path=r'\\path\\chromedriver.exe')
do this :
This is a pre - requisite
Installation
pip install chromedriver-autoinstaller
Usage:
Just type import chromedriver_autoinstaller in the module you want to use chromedriver.
Example
from selenium import webdriver
import chromedriver_autoinstaller
chromedriver_autoinstaller.install() # Check if the current version of chromedriver exists
# and if it doesn't exist, download it automatically,
# then add chromedriver to path
driver = webdriver.Chrome()
driver.get("http://www.python.org")
assert "Python" in driver.title
and the export this changes to your executables
I want to collect the information of a webpage using chromedriver. How do I install it and use it?
You have to install selenium first if you don't have it already. Then to use selenium:
from selenium.webdriver import Chrome
url="URL of the webpage you want to read"
setting up the driver
webdriver = "path of the chromedriver.exe file saved in your pc"
driver.get(url)
using css selector
y = driver.find_element_by_css_selector('css selector of the data you want to read from the webpage').text
print(y)
You don't install the chromedriver - you download the .exe (from here) and use the path to it in webdriver.Chrome(). This getting started page has a comprehensive guide:
from selenium import webdriver
driver = webdriver.Chrome('/path/to/chromedriver') # refers to the path where you saved the exe
driver.get('http://www.google.com/');
time.sleep(5) # Let the user actually see something!
search_box = driver.find_element_by_name('q')
search_box.send_keys('ChromeDriver')
search_box.submit()
time.sleep(5) # Let the user actually see something!
driver.quit()
Note: download the .exe that matches with your version of chrome!
(In Help > About Google Chrome)
As mentioned by #Patha_Mondal, you need to download the driver and select the elements you want to read. However, as your original question asks "How to use selenium in pandas to read a webpage?", I would say instead consider using Scrapy along with Selenium to create a ".csv" file from the Webpage Data.
Read the ".csv" data into pandas using pandas.read_csv() .
The data from the Webpage might not be clean or properly formatted. Using Scrapy to create a dataset out of it would be beneficial for reading it into pandas. Avoid using pandas directly in the same script as Selenium and Scrapy.
Hope it Helped.
When i tried to take screenshot of a webpage using selenium in python, i get error message selenium.common.exceptions.TimeoutException: Message: timeout: Timed out receiving message from renderer: 10.000.
Code:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
indi_url = 'http://www.google.com'
options = Options()
options.add_argument("disable-infobars")
options.add_argument("--start-maximized")
options.add_argument("--disable-popup-blocking")
options.add_argument("disable-popup-blocking")
options.add_argument("--disable")
driver = webdriver.Chrome(options=options)
driver.get(indi_url)
driver.implicitly_wait(30)
driver.save_screenshot("new.png")
Error message:
I'm using Chrome version 73, chromedriver version 73.
Note: code was working fine (ie.screenshot)in lower version of chrome and chrome driver.
Help me out in fixing this issue for new version of chrome driver.
Thanks in advance
As the error shows, your filename for screenshot does not match the template extensions .png
Here is an example how to make a screenshot.
Java:
File scrFile = ((TakesScreenshot)driver).getScreenshotAs(OutputType.FILE);
FileUtils.copyFile(scrFile, new File(".\\Screenshots\\example_screenshot.png"));
Python:
driver.save_screenshot("screenshot.png")
This error message...
UserWarning: name used for saved screenshot does not match file type. It should end with a .png extension
"type. It should end with a .png extension", UserWarning)
...implies that the Selenium-Python client encountered an issue while invoking get_screenshot_as_file() method.
get_screenshot_as_file()
get_screenshot_as_file() saves a screenshot of the current window to a PNG image file. Returns False if there is any IOError, else returns True. Use full paths in your filename.
Args:
filename: The full path you wish to save your screenshot to. This should end with a .png extension.
Usage:
driver.get_screenshot_as_file('/Screenshots/foo.png')
Defination:
if not filename.lower().endswith('.png'):
warnings.warn("name used for saved screenshot does not match file "
"type. It should end with a `.png` extension", UserWarning)
png = self.get_screenshot_as_png()
try:
with open(filename, 'wb') as f:
f.write(png)
except IOError:
return False
finally:
del png
return True
Analysis
As per the snapshot of the error stack trace:
You have used the command as:
driver.get_screenshot_as_file('new.jpeg')
The issues were:
The filename didn't end with .png
The desired full path of your filename wasn't provided.
Even if you desire to use save_screenshot() this method in-turn invokes get_screenshot_as_file(filename)
Solution
Create a directory within your project as Screenshots and provide the absolute path of the filename you desire for the screenshot while invoking either of the methods as follows:
driver.get_screenshot_as_file("./Screenshots/YakeshrajM.png")
driver.save_screenshot("./Screenshots/YakeshrajM.png")
Update
Currently GAed Chrome v73 have some issues and you may like to downgrade to Chrome v72. You can find a couple of relevant discussions in:
Getting Timed out receiving message from renderer: 600.000 When we execute selenium scripts using Jenkins windows service mode
Timed out receiving message from renderer: 10.000 while capturing screenshot using chromedriver and chrome through Jenkins on Windows