I'm new at Python and I'm trying to use selenium webdriver to do web scraping from a webpage. When I run my code, I have not problems in obtaining the results I need, but when someone else tries to run the code from an executable it shows the error: selenium.common.exceptions.webdriverexception: message: 'chromedriver.exe' unexpectedly exited.status code was: 1. The path where I'm saving chromedriver.exe is a public repository. Can someone help me with this please? Here is the piece of code I'm using:
from selenium import webdriver
url= "https://www.byma.com.ar/obligaciones-negociables/"
driver = webdriver.Chrome(executable_path=r'\\path\\chromedriver.exe')
driver.implicitly_wait(30)
driver.get(url)
time.sleep(2)
You do not need to setup a driver like this :
driver = webdriver.Chrome(executable_path=r'\\path\\chromedriver.exe')
do this :
This is a pre - requisite
Installation
pip install chromedriver-autoinstaller
Usage:
Just type import chromedriver_autoinstaller in the module you want to use chromedriver.
Example
from selenium import webdriver
import chromedriver_autoinstaller
chromedriver_autoinstaller.install() # Check if the current version of chromedriver exists
# and if it doesn't exist, download it automatically,
# then add chromedriver to path
driver = webdriver.Chrome()
driver.get("http://www.python.org")
assert "Python" in driver.title
and the export this changes to your executables
Related
I trying to use selenium in jupyter notebook on Firefox. I have installed the geckodriver in /usr/bin directory .When I run my code it says:
WebDriverException: Message: 'geckodriver' executable needs to be in PATH.
if I run the command %env PATH following output is displayed:
$PATH:/usr/bin/
from selenium import webdriver
browser = webdriver.Firefox()
I have even tried
browser=webdriver.Firefox(executable_path=r'/usr/bin/geckodriver')
Can somebody help me solve this?
I hope you are working with geckodriver on the local machine and providing the absolute path for the driver to run, This can be done automatically using webdriver-manager python
Advantages of using it are-
You don't have to give the path every time to run code.
Makes the code portable and easy to run on any machine without checking for geckodriver or chromedriver.
pip install webdriver-manager
Now the above code in the question can be changed to work simply with,
from selenium import webdriver
from webdriver_manager.firefox import GeckoDriverManager
driver = webdriver.Firefox(executable_path=GeckoDriverManager().install())
If the above doesn't help, try doing this.
autoinstaller for geckodriver-
pip install geckodriver-autoinstaller
from selenium import webdriver
import geckodriver_autoinstaller
geckodriver_autoinstaller.install() #checks for driver or install it. driver =
webdriver.Firefox() driver.get("python.org")
this worked fine for me, otherwise, try to give a full path after downloading the latest geckodriver binary file.
Try to implement this with a Maven project so you can add dependencies in the pom.xml file. so you dont have to give the path each time. Refer this article here.Its in java but will help.
I have a windows64 bit machine and I downloaded the chromedriver on the exact location it is supposed to be and my file path is this:
'C:\Users\username\AppData\Local\Google\Chrome\Application\chrome.exe\chromedriver'
Then I wrote this code:
import selenium
from selenium import webdriver
driver=webdriver.Chrome('C:\Users\pushp\AppData\Local\Google\Chrome\Application\chrome.exe\chromedriver')
However, I am getting a file not found error and also this message -
'Message: 'chromedriver' executable needs to be in PATH.'
How do I fix this?
driver =webdriver.Chrome(r'C:\Users\pushp\AppData\Local\Google\Chrome\Application\chrome.exe\chromedriver.exe')
Just need to add .exe
Set it like the following
import selenium
from selenium import webdriver
webdriver.Chrome(executable_path=r"C:\Users\pushp\AppData\Local\Google\Chrome\Application\chromedriver.exe")
Or
webdriver.Chrome(executable_path="C:\\Users\\pushp\\AppData\\Local\\Google\\Chrome\\Application\\chromedriver.exe")
You can also install pip install webdriver-manager then run the following code which will install the correct webdriver for you
from selenium import webdriver
from webdriver_manager.chrome import ChromeDriverManager
driver = webdriver.Chrome(ChromeDriverManager().install())
First, machine and package specs:
I am running:
ChromeDriver version 75.0.3770.140
Selenium: version '3.141.0'
WSL (linux subsystem) of windows 10
I am trying to run a chromebrowser through selenium. I found: these commands, to use selenium through google chrome.
I have a test directory, with only the chromedriver binary file, and the script, in it. The location of the directory is: /home/kela/test_dir/
I ran the code:
import selenium
from selenium import webdriver
from bs4 import BeautifulSoup
from selenium.webdriver.firefox.options import Options
from selenium.webdriver.firefox.firefox_binary import FirefoxBinary
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities
options = Options()
options.binary_location='/home/kela/test_dir/chromedriver'
driver = webdriver.Chrome(chrome_options = options,executable_path='/home/kela/test_dir/chromedriver')
The output from this code is:
selenium.common.exceptions.SessionNotCreatedException: Message: session not created: No matching capabilities found
Can anyone explain why I need capabilities when the same script works for others without capabilities? I did try adding:
chrome_options.add_argument('--headless')
chrome_options.add_argument('--no-sandbox')
but I got the same error. So I'm not sure what capabilities I need to add (considering it works for others without it?)
Edit 1: Addressing DebanjanB's comments below:
Chromedriver is in the expected location. I am using windows 10. From here, the expected location is C:\Program Files (x86)\Google\Chrome\Application\chrome.exe; and this is where it is on my machine (I copied and pasted this location from the chrome Properties table).
ChromeDriver is having executable permission for non-root users.
I definitely have Google Chrome v75.0 installed (I can see that the Product version 75.0.3770.100)
I am running the script as a non-root user, as my bash command line ends with a $ and not # (i.e kela:~/test_dir$ and not kela:~/test_dir#)
Edit 2: Based on DebanjanB's answer below, I am very close to having it working, but just not quite.
The code:
import selenium
from selenium import webdriver
from bs4 import BeautifulSoup
from selenium.webdriver.firefox.options import Options
from selenium.webdriver.firefox.firefox_binary import FirefoxBinary
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities
from selenium.webdriver.chrome.options import Options
options = Options()
options.binary_location='/c/Program Files (x86)/Google/Chrome/Application/chrome.exe'
driver = webdriver.Chrome(options=options)
driver.get('http://google.com/')
Produces a dialog box that reads:
Google Chrome cannot read and write to it's data directory: /tmp/.com/google.Chrom.gyw63s
So then I double checked my Chrome permissions and I should be able to write to Chrome:
Also, I can see that /tmp/ has a bunch of .com dirs in it:
.com.google.Chrome.4jnWme/ .com.google.Chrome.FdNyKP/ .com.google.Chrome.VAcWMQ/ .com.google.Chrome.ZbkRx0/ .com.google.Chrome.iRrceF/
.com.google.Chrome.A2QHHB/ .com.google.Chrome.G7Y51c/ .com.google.Chrome.WD8BtK/ .com.google.Chrome.cItmhA/ .com.google.Chrome.pm28hN/
However, since that seemed to be more of a warning than an error, I clicked 'ok' to close the dialog box, and a new tab does open in the browser; but the URL is just 'data:,'. The same thing happens if I remove the line 'driver.get('http://google.com')' from the script, so I know the warning/issue is with the line:
driver = webdriver.Chrome(chrome_options = options,executable_path='/home/kela/test_dir/chromedriver')
For example, from here, I tried adding:
options.add_argument('--profile-directory=Default')
But the same warning pops up.
Edit 3:
As edit 3 was starting to veer into a different question than specifically being addressed here, I started a new question here.
This error message...
selenium.common.exceptions.SessionNotCreatedException: Message: session not created: No matching capabilities found
...implies that the ChromeDriver was unable to initiate/spawn a new WebBrowser i.e. Chrome Browser session.
binary_location
binary_location set/get(s) the location of the Chrome (executable) binary and is defined as:
def binary_location(self, value):
"""
Allows you to set where the chromium binary lives
:Args:
- value: path to the Chromium binary
"""
self._binary_location = value
So as per your code trials, options.binary_location='/home/kela/test_dir/chromedriver' is incorrect.
Solution
If Chrome is installed at the default location, you can safely remove this property. Incase Chrome is installed at a customized location you need to use the options.binary_location property to point to the Chrome installation.
You can find a detailed discussion in Selenium: WebDriverException:Chrome failed to start: crashed as google-chrome is no longer running so ChromeDriver is assuming that Chrome has crashed
Effectively, you code block will be:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
options = Options()
options.binary_location=r'C:\Program Files (x86)\Google\Chrome\Application\chrome.exe'
driver = webdriver.Chrome(options=options, executable_path='/home/kela/test_dir/chromedriver.exe')
driver.get('http://google.com/')
Additionally, ensure the following:
ChromeDriver is having executable permission for non-root users.
As you are using ChromeDriver v75.0 ensure that you have the recommended version of the Google Chrome v75.0 as:
---------ChromeDriver 75.0.3770.8 (2019-04-29)---------
Supports Chrome version 75
Execute the Selenium Test as non-root user.
I am trying to use chromedriver with Selenium on Windows 10 but I get the following error:
Traceback (most recent call last):
File "scrape.py", line 4, in <module>
driver = webdriver.Chrome()
File "C:\Python37\lib\site-packages\selenium\webdriver\chrome\webdriver.py", line 73, in __init__
self.service.start()
File "C:\Python37\lib\site-packages\selenium\webdriver\common\service.py", line 83, in start
os.path.basename(self.path), self.start_error_message)
selenium.common.exceptions.WebDriverException: Message: 'chromedriver' executable needs to be in PATH. Please see https://sites.google.com/a/chromium.org/chromedriver/home
And here's my test script:
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
driver = webdriver.Chrome()
driver.get("http://www.python.org")
assert "Python" in driver.title
elem = driver.find_element_by_name("q")
elem.clear()
elem.send_keys("pycon")
elem.send_keys(Keys.RETURN)
assert "No results found." not in driver.page_source
driver.close()
I've tried many things. I'll detail below.
I've attempted to add the path to chromedriver to PATH. Image here:
This works fine because I can run chromedriver from the commandline:
C:\Users\KraftWurk>chromedriver
Starting ChromeDriver 74.0.3729.6 (255758eccf3d244491b8a1317aa76e1ce10d57e9-refs/branch-heads/3729#{#29}) on port 9515
Only local connections are allowed.
Please protect ports used by ChromeDriver and related test frameworks to prevent access by malicious code.
But still, when I run my scripts I get the error that chromedriver needs to be in PATH ... it is, so not sure what's going on there.
I've read the following: Python Selenium Chrome Webdriver
I've attempted to provide the path as suggested using:
driver = webdriver.Chrome(executable_path=r"C:\drivers\chromedriver.exe")
I still get the same warning.
I'm not quite sure what's going on. I'm using Python 3.7 on Windows 10. Selenium 3.141.0 and Chromedriver 74.0.3729.6
To eliminates lot of manual works and incompatibility issues, I would suggest you to go for WebDriverManager as it automatically downloads required binary and we do not need to set any path.
It supports browsers such as Chrome, Firefox, PhantomJS, Microsoft Edge, or Internet Explorer.
How do we use this in our project?
Only setup required is to install this package using ‘pip’.
pip install webdriver_manager
That’s it! We are all set. Just import this module in your python project and start using it.
For Chrome:
from webdriver_manager.chrome import ChromeDriverManager
from selenium import webdriver
driver = webdriver.Chrome(executable_path=ChromeDriverManager().install())
driver.get("http://www.google.com/")
print driver.title
driver.quit()
For Firefox:
from webdriver_manager.firefox import GeckoDriverManager
from selenium import webdriver
driver = webdriver.Firefox(executable_path=GeckoDriverManager().install())
driver.get("http://www.google.com/")
print driver.title
driver.quit()
For Edge:
from webdriver_manager.microsoft import EdgeDriverManager
from selenium import webdriver
driver = webdriver.Edge(executable_path=EdgeDriverManager().install())
driver.get("http://www.google.com/")
print driver.title
driver.quit()
For IE:
from webdriver_manager.microsoft import IEDriverManager
from selenium import webdriver
driver = webdriver.Ie(executable_path=IEDriverManager().install())
driver.get("http://www.google.com/")
print driver.title
driver.quit()
webdriver_manager, by default, tries to download the latest version of a given driver binary. To use a specific version of driver, pass the driver version like below.
webdriver.Chrome(executable_path=ChromeDriverManager("2.42").install())
I have completed code for geckodriver and was wondering how to make geckodriver headless now. I saw a post previously with the following text:
from selenium import webdriver
from selenium.webdriver.firefox.options import Options
options = Options()
options.add_argument("--headless")
driver = webdriver.Firefox(firefox_options=options,
executable_path="C:\\Utility\\BrowserDrivers\\geckodriver.exe")
print("Firefox Headless Browser Invoked")
driver.get('http://google.com/')
driver.quit()
I don't understand where the download for options came from under webdriver. When I downloaded geckodriver, all that came with it was the executable file. Any help is greatly appreciated!!
Works for me. Steps I used: (1) Open a command prompt and navigate to the folder containing geckodriver.exe. (2) Start geckodriver.exe without any options from a command prompt. (3) Open another command prompt and type python and press the Return key. (4) Copy/paste the following code into your your python session.
from selenium import webdriver
from selenium.webdriver.firefox.options import Options
from selenium.webdriver import Firefox
options = Options()
options.add_argument("--headless")
# Don't put the path to geckodriver in the following. But the firefox executable
# must be in the path. If not, include the path to firefox, not geckodriver below.
driver = Firefox(firefox_options=options)
print("Firefox Headless Browser Invoked")
driver.get('http://google.com/')
# Print the first 300 characters on the page.
print(driver.page_source[:300])
driver.quit()