I've got a Lambda Function for headless chrome + python selenium deployed with Serverless framework that runs fine locally but crashes on lambda.
Some basic details:
(Driver info: chromedriver=2.41.578700 (2f1ed5f9343c13f73144538f15c00b370eda6706),platform=Linux 4.14.231-180.360.amzn2.x86_64 x86_64)
Chromium Version: 89xx
selenium==3.141.0
Here is how i'm invoking it with selenium:
options = Options()
options.binary_location = '/opt/headless-chromium'
options.add_argument('--headless')
options.add_argument('--no-sandbox')
options.add_argument('--single-process')
options.add_argument("--remote-debugging-port=9222")
options.add_argument('--disable-dev-shm-usage')
#'/opt/chromedriver' not found
driver = webdriver.Chrome('/opt/chromedriver', chrome_options=options)
driver.get('https://www.neaminational.org.au/')
body = f"Headless Chrome Initialized, Page title: {driver.title}"
driver.close();
driver.quit();
response = {
"statusCode": 200,
"body": body
}
I'm getting the cryptic Message: unknown error: Chrome failed to start: exited abnormally
(chrome not reachable)
(The process started from chrome location /opt/headless-chromium is no longer running, so ChromeDriver is assuming that Chrome has crashed.).
Now i've tested this on my ubuntu 18 (same chromium binary, same chrome driver, same install selenium version) and it's working fine... so my issue must be with compatibility with the lambda amz linux environment.
Can anyone give me some idea on how i could troubleshoot this? Seems silly to stumble around trying different versions when they all seem compatible with eachother locally.
Any insight appreciated greatly!
I found this to be really helpful:
https://www.youtube.com/watch?v=jWqbYiHudt8
https://github.com/soumilshah1995/Selenium-on-AWS-Lambda-Python3.7
The versions are the following:
RUNTIME=python3.7
SELENIUM_VER=3.141.0
CHROME_BINARY_VER=v1.0.0-55 # based on Chromium 69.0.3497.81
CHROMEDRIVER_VER=2.43 # supports Chrome v69-71
Credits go to Soumil Nitin Shah.
Best,
Ramón
Related
I'm running a simple CI pipeline on GitLab for a Selenium script headlessly + using webdriver_manager to handle chrome driver binary.
This part is passed:
Get LATEST chromedriver version for None google-chrome
There is no [linux64] chromedriver for browser None in cache
Trying to download new driver from https://chromedriver.storage.googleapis.com/100.0.4896.60/chromedriver_linux64.zip
Driver has been saved in cache [/root/.wdm/drivers/chromedriver/linux64/100.0.4896.60]
But after that I'm getting this error:
WebDriverException: Message: Service /root/.wdm/drivers/chromedriver/linux64/100.0.4896.60/chromedriver unexpectedly exited. Status code was: 127`
What is the problem? Seems like webdriver_manager has a problem by running in CI.
Here is a simple script for reproduce:
from selenium import webdriver
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.chrome.service import Service
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument("--headless")
service = Service(executable_path=ChromeDriverManager().install())
driver = webdriver.Chrome(service=service, options=chrome_options)
driver.get("http://google.com")
driver.find_element('name', 'q').send_keys("Wikipedia")
This is one of the pipelines:
https://gitlab.com/mmonfared/test/-/jobs/2350697126
This is a sample project:
https://gitlab.com/mmonfared/test
I've also opened an issue in webdriver_manager github repo, no answers yet:
https://github.com/SergeyPirogov/webdriver_manager/issues/363
This error message...
WebDriverException: Message: Service /root/.wdm/drivers/chromedriver/linux64/100.0.4896.60/chromedriver unexpectedly exited. Status code was: 127`
...implies that you are executing your tests as the root user.
Deep Dive
As per Chrome doesn't start or crashes immediately
A common cause for Chrome to crash during startup is running Chrome as
root user (administrator) on Linux. While it is possible to work
around this issue by passing --no-sandbox flag when creating your
WebDriver session, such a configuration is unsupported and highly
discouraged. Please configure your environment to run Chrome as a
regular user instead.
Solution
Execute your tests as a non-root user.
System:
OS: Ubuntu 20.04.2 LTS
Python: 3.8.5
selenium: 3.141.0
ChromeDriver: 90.0.4430.24
Brave: Version 1.24.86 Chromium: 90.0.4430.212 (Official Build) (64-bit)
I am trying to get Selenium to work with Brave.
I am using the script from here.
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
options = Options()
options.binary_location = '/snap/bin/brave'
driver_path = '/usr/local/bin/chromedriver'
drvr = webdriver.Chrome(options=options, executable_path=driver_path)
drvr.get('https://stackoverflow.com')
I keep getting this error
selenium.common.exceptions.WebDriverException: Message: unknown error: DevToolsActivePort file doesn't exist
And I have tried all the answers in stackflow
I tried all different combinations of these arguments but nothing worked:(
options.add_argument("start-maximized")
options.add_argument("disable-infobars")
options.add_argument("--disable-extensions")
options.add_argument("--disable-gpu")
options.add_argument("--disable-dev-shm-usage")
options.add_argument("--no-sandbox")
options.add_argument('--remote-debugging-port=9222')
options.add_argument('--headless')
Can anyone help me out?
I was able to get it working by changing the remote-debugging-port to a different port.
Here is my code that worked.
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
options = Options()
#driver_path = '/usr/local/bin/operadriver'
driver_path = '/usr/local/bin/chromedriver'
brave_path = '/snap/bin/brave'
options.binary_location = brave_path
options.add_argument('--remote-debugging-port=9224') #NOT 9222
drvr = webdriver.Chrome(options=options, executable_path=driver_path)
drvr.get('https://stackoverflow.com')
drvr.quit
I did see a tip here about using an opera driver, which i found here, but it did not solve my error.
The issue was that after running the script once the process was not killed and was still running in the background so the port was taken and failed on all subsequent runs until I changed the port (or killed the process). The question is why is close() not working in brave when headless, it is working for chrome!
I am trying to use chrome but I need it to be hidden from the user.
So I tried to use "headless" with ChromeOptions
ChromeOptions options = new ChromeOptions();
options.addExtensions(new File("C://temp//1.14.12_0.crx"));
options.addArguments("disable-infobars");
options.addArguments("headless");
but then I crashed with the following exception:
[1524948093.974][WARNING]: chrome quit unexpectedly, leaving behind temporary directories for debugging:
[1524948093.974][SEVERE]: Port leaked: 12545
Exception in thread "main" org.openqa.selenium.WebDriverException: unknown error: failed to wait for extension background page to load: chrome-extension://cjpalhdlnbpafiamejdnhcphjbkeiagm/background.html
from unknown error: page could not be found: chrome-extension://cjpalhdlnbpafiamejdnhcphjbkeiagm/background.html
(Driver info: chromedriver=2.37.544315 (730aa6a5fdba159ac9f4c1e8cbc59bf1b5ce12b7),platform=Windows NT 10.0.16299 x86_6
I made some searching and found out that it might be related to the fact that I also load an ad blocker extension, so I temporarily commented the extension loading and the chrome was loaded HIDDEN.
But I still need this extension to work for me when chrome is hidden.
Is there any solution?
Chrome Version - 66
Chrome Driver version - 2.37
Thanks!
So, this one made the trick:
options.addArguments("load-extension=.//resources//1.14.12_0.crx");
instead of
options.addExtensions(new File(".//resources//1.14.12_0.crx"));
Enjoy!
In [12]: from selenium import webdriver
In [13]: chrome_options = webdriver.ChromeOptions()
In [14]: chrome_options.add_argument('--no_sandbox')
In [15]: chrome_options.add_argument('--privileged')
In [16]: browser = webdriver.Chrome('/home/jeremie/Downloads/chromedriver', chrome_o
...: ptions=chrome_options)
In [17]: browser.get('http://localhost:8000')
When I ran the last line, I got
WebDriverException: Message: chrome not reachable
(Session info: chrome=58.0.3029.81)
(Driver info: chromedriver=2.30.477691 (6ee44a7247c639c0703f291d320bdf05c1531b57),platform=Linux 4.8.0-32-generic x86_64)
I tried to change another version of the chromedriver, but I got other issues. I tried to fix my problem with other question from SE, but nothing solved my problem. What could I do to fix that problem?
check you have the execution right on the chrome bin. Is possbile the chrome browser installed not by your account?
Give a try not to add arguments to chromeOptions
Give a try with lower down your chromedriver version
I guess you are not setProperty()
driver = webdriver.Chrome('C:\\Users\\Downloads\\chromedriver_win32\\chromedriver.exe')
driver.get("https://stackoverflow.com/")
There can be multiple reasons for seeing the WebDriverException as chrome not reachable.
First and foremost, we have to ensure that our Selenium version, chromedriver version and Chrome version are compatible. You can find the compatibility information on the Downloads page of ChromeDriver individually for each releases.
Consider removing all the dangling chromedriver instances from you system. If possible make a system restart.
Periodically run CCleaner to wipe away all the previous execution leftovers.
When you initialized the webdriver instance, you mentioned:
browser = webdriver.Chrome('/home/jeremie/Downloads/chromedriver', chrome_options=chrome_options)
Instead, while we mention the absolute path of the chromedriver binary we must also provide the argument executable_path. So we may need to change to:
driver = webdriver.Chrome(chrome_options=chrome_options, executable_path='/home/jeremie/Downloads/chromedriver')
driver.get("https://stackoverflow.com/")
if __name__ == '__main__':
driver=webdriver.Firefox(executable_path=r'/home/saurabh/Saurabh/LearnPython/Automation/geckodriver');
After running the above code I am getting an error as :
selenium.common.exceptions.WebDriverException: Message: Unable to find a matching set of capabilities
I don't see any significant error in your code as such.
It is to be noted that the current Selenium-Python binding is unstable with geckodriver and looks to be Architecture specific. You can find the github discussion and merge here. So you may additionally need to pass the absolute path of the firefox binary as firefox_binary argument while initializing the webdriver
Here is your own code with a simple tweak which opens the Mozilla Firefox browser:
from selenium import webdriver
from selenium.webdriver.firefox.firefox_binary import FirefoxBinary
if __name__ == '__main__':
binary = FirefoxBinary('C:\\Program Files\\Mozilla Firefox\\firefox.exe')
driver = webdriver.Firefox(firefox_binary=binary, executable_path="C:\\path\\to\\geckodriver.exe")
Make sur you are pointing to \path\to\FirefoxPortable\App\Firefox64\firefox.exe and not just \path\to\FirefoxPortable\FirefoxPortable.exe