When the ChromeDriver version does not match my current chrome version, i upgrade chromedriver by the following code:
from selenium import webdriver
from webdriver_manager.chrome import ChromeDriverManager
driver = webdriver.Chrome(ChromeDriverManager().install())
Then i use selenium for scraping the website data, but still i got some errors. Anyone can help me with this issue? Appreciate.
import time
import pandas as pd
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
options = Options()
options.headless = True
driver = webdriver.Chrome(options=options)
driver.get("https://www.binance.com/cn/futures/funding-history/0")
time.sleep(5)
The errors took place when the above code is ran which has been attached.
You may try to include the below code to get the latest version automatically
through PIP :
pip install chromedriver-autoinstaller
Usage
Just type import chromedriver_autoinstaller in the module you want to use chromedriver.
Example
from selenium import webdriver
import chromedriver_autoinstaller
chromedriver_autoinstaller.install() # Check if the current version of chromedriver exists
# and if it doesn't exist, download it automatically,
# then add chromedriver to path
driver = webdriver.Chrome()
driver.get("http://www.python.org")
assert "Python" in driver.title
Read more about auto upgrade here
Related
Trying to run selenium script with targeted chrome profile. But once I run the script, it won't start with the targeted profile but with a new profile. Here's my code:
# import selenium common driver
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.keys import Keys
import time
# for specified chrome profile
from selenium.webdriver.chrome.options import Options
# wait page until targeted element loaded
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.common.exceptions import TimeoutException
# undetectable module
import undetected_chromedriver.v2 as uc # use pip install undetected-chromedriver
if __name__ == '__main__':
options = uc.ChromeOptions()
# another way to set profile is the below (which takes precedence if both variants are used
options.add_argument(r'--user-data-dir=C:\Users\Fadli\AppData\Local\Google\Chrome\User Data\Profile 4')
# just some options passing in to skip annoying popups
options.add_argument(r'--no-first-run --no-service-autorun --password-store=basic')
driver = uc.Chrome(options=options) # version_main allows to specify your chrome version instead of following chrome global version
driver.get('https://nowsecure.nl')
Try the below line if it help
# another way to set profile is the below (which takes precedence if both variants are used
options.add_argument(r'--user-data-dir=C:\Users\Fadli\AppData\Local\Google\Chrome\User Data')
options.add_argument(r'--profile-directory=Profile 4')
you can check it here for more reference
I have successfully installed Selenium in Databricks and can import the Python selenium and webdriver. On my local computer once I run the Selenium get command a separate browser windows open where I can see what Selenium is doing.
However, when running the same script on Databricks there is unfortunately no window opening. I am wondering if this is at all possible. I found some options like
browser.execute_script('''window.open("http://bings.com","_blank");''')
or
add_experimental_option
but these options did not work.
I am currently using the following options on Databricks:
from selenium import webdriver
from webdriver_manager.chrome import ChromeDriverManager
from webdriver_manager.utils import ChromeType
import pandas as pd
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument('--no-sandbox')
chrome_options.add_argument('--headless')
chrome_options.add_argument('--disable-dev-shm-usage')
# installed driver
chrome_driver = "/tmp/chromedriver1/chromedriver"
driver = webdriver.Chrome(executable_path=chrome_driver, options=chrome_options)
Has someone an idea how this could work?
Thanks in advance!
No it won't open since you are using headless browser. See the below line from your code :-
chrome_options.add_argument('--headless')
Having said that, it will execute the Python-Selenium bindings instruction, you have. You can try to print the page title like this :
driver = webdriver.Chrome(executable_path=chrome_driver, options=chrome_options)
driver.get("https://www.google.com")
print(driver.title)
should print google.com title in the console.
I'm new at Python and I'm trying to use selenium webdriver to do web scraping from a webpage. When I run my code, I have not problems in obtaining the results I need, but when someone else tries to run the code from an executable it shows the error: selenium.common.exceptions.webdriverexception: message: 'chromedriver.exe' unexpectedly exited.status code was: 1. The path where I'm saving chromedriver.exe is a public repository. Can someone help me with this please? Here is the piece of code I'm using:
from selenium import webdriver
url= "https://www.byma.com.ar/obligaciones-negociables/"
driver = webdriver.Chrome(executable_path=r'\\path\\chromedriver.exe')
driver.implicitly_wait(30)
driver.get(url)
time.sleep(2)
You do not need to setup a driver like this :
driver = webdriver.Chrome(executable_path=r'\\path\\chromedriver.exe')
do this :
This is a pre - requisite
Installation
pip install chromedriver-autoinstaller
Usage:
Just type import chromedriver_autoinstaller in the module you want to use chromedriver.
Example
from selenium import webdriver
import chromedriver_autoinstaller
chromedriver_autoinstaller.install() # Check if the current version of chromedriver exists
# and if it doesn't exist, download it automatically,
# then add chromedriver to path
driver = webdriver.Chrome()
driver.get("http://www.python.org")
assert "Python" in driver.title
and the export this changes to your executables
I have a windows64 bit machine and I downloaded the chromedriver on the exact location it is supposed to be and my file path is this:
'C:\Users\username\AppData\Local\Google\Chrome\Application\chrome.exe\chromedriver'
Then I wrote this code:
import selenium
from selenium import webdriver
driver=webdriver.Chrome('C:\Users\pushp\AppData\Local\Google\Chrome\Application\chrome.exe\chromedriver')
However, I am getting a file not found error and also this message -
'Message: 'chromedriver' executable needs to be in PATH.'
How do I fix this?
driver =webdriver.Chrome(r'C:\Users\pushp\AppData\Local\Google\Chrome\Application\chrome.exe\chromedriver.exe')
Just need to add .exe
Set it like the following
import selenium
from selenium import webdriver
webdriver.Chrome(executable_path=r"C:\Users\pushp\AppData\Local\Google\Chrome\Application\chromedriver.exe")
Or
webdriver.Chrome(executable_path="C:\\Users\\pushp\\AppData\\Local\\Google\\Chrome\\Application\\chromedriver.exe")
You can also install pip install webdriver-manager then run the following code which will install the correct webdriver for you
from selenium import webdriver
from webdriver_manager.chrome import ChromeDriverManager
driver = webdriver.Chrome(ChromeDriverManager().install())
I am trying to use chromedriver with Selenium on Windows 10 but I get the following error:
Traceback (most recent call last):
File "scrape.py", line 4, in <module>
driver = webdriver.Chrome()
File "C:\Python37\lib\site-packages\selenium\webdriver\chrome\webdriver.py", line 73, in __init__
self.service.start()
File "C:\Python37\lib\site-packages\selenium\webdriver\common\service.py", line 83, in start
os.path.basename(self.path), self.start_error_message)
selenium.common.exceptions.WebDriverException: Message: 'chromedriver' executable needs to be in PATH. Please see https://sites.google.com/a/chromium.org/chromedriver/home
And here's my test script:
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
driver = webdriver.Chrome()
driver.get("http://www.python.org")
assert "Python" in driver.title
elem = driver.find_element_by_name("q")
elem.clear()
elem.send_keys("pycon")
elem.send_keys(Keys.RETURN)
assert "No results found." not in driver.page_source
driver.close()
I've tried many things. I'll detail below.
I've attempted to add the path to chromedriver to PATH. Image here:
This works fine because I can run chromedriver from the commandline:
C:\Users\KraftWurk>chromedriver
Starting ChromeDriver 74.0.3729.6 (255758eccf3d244491b8a1317aa76e1ce10d57e9-refs/branch-heads/3729#{#29}) on port 9515
Only local connections are allowed.
Please protect ports used by ChromeDriver and related test frameworks to prevent access by malicious code.
But still, when I run my scripts I get the error that chromedriver needs to be in PATH ... it is, so not sure what's going on there.
I've read the following: Python Selenium Chrome Webdriver
I've attempted to provide the path as suggested using:
driver = webdriver.Chrome(executable_path=r"C:\drivers\chromedriver.exe")
I still get the same warning.
I'm not quite sure what's going on. I'm using Python 3.7 on Windows 10. Selenium 3.141.0 and Chromedriver 74.0.3729.6
To eliminates lot of manual works and incompatibility issues, I would suggest you to go for WebDriverManager as it automatically downloads required binary and we do not need to set any path.
It supports browsers such as Chrome, Firefox, PhantomJS, Microsoft Edge, or Internet Explorer.
How do we use this in our project?
Only setup required is to install this package using ‘pip’.
pip install webdriver_manager
That’s it! We are all set. Just import this module in your python project and start using it.
For Chrome:
from webdriver_manager.chrome import ChromeDriverManager
from selenium import webdriver
driver = webdriver.Chrome(executable_path=ChromeDriverManager().install())
driver.get("http://www.google.com/")
print driver.title
driver.quit()
For Firefox:
from webdriver_manager.firefox import GeckoDriverManager
from selenium import webdriver
driver = webdriver.Firefox(executable_path=GeckoDriverManager().install())
driver.get("http://www.google.com/")
print driver.title
driver.quit()
For Edge:
from webdriver_manager.microsoft import EdgeDriverManager
from selenium import webdriver
driver = webdriver.Edge(executable_path=EdgeDriverManager().install())
driver.get("http://www.google.com/")
print driver.title
driver.quit()
For IE:
from webdriver_manager.microsoft import IEDriverManager
from selenium import webdriver
driver = webdriver.Ie(executable_path=IEDriverManager().install())
driver.get("http://www.google.com/")
print driver.title
driver.quit()
webdriver_manager, by default, tries to download the latest version of a given driver binary. To use a specific version of driver, pass the driver version like below.
webdriver.Chrome(executable_path=ChromeDriverManager("2.42").install())