I'm running a Selenium webscraper in Java on a Linux server through SSH, but the browser crashes upon termination of the session despite various attempts to instruct the program to run in the background. Specifically, I've used screen, tmux, nohup, disown, and setsid (as here https://www.tecmint.com/keep-remote-ssh-sessions-running-after-disconnection/) with similar, bad results. I would like to try a VNC session, but so far not enabled on the server.
With screen for example:
I type screen, run the program, and normal output is produced
I detach the screen with Ctrl+a+d, normal output continues to be produced
I end the SSH session. Output ceases and (upon logging back in and returning to the screen session), I get the error message
Exiting due to channel error
Exiting due to channel error
Exiting due to channel error
Exiting due to channel error
Exception in thread "main" org.openqa.selenium.WebDriverException: Failed to decode response from marionette
I've tried this a couple of times, so it's not a coincidence that it crashes at the exact time of ending the SSH session. What could be going wrong and how can I try to diagnose the issue?
System information:
OS: Red Hat Enterprise Linux, Linux 3.10.0-1127.10.1.el7.x86_64
Java: 1.8.0_252
Firefox version: Mozilla Firefox 78.3.1esr
Firefox boot options: "--marionette" "-headless" "-foreground" "-no-remote" "-profile"
Gecko driver version: 0.27.0
Selenium version: 3.141.59m revision e82be7d358
SSH client: MobaXTerm, personal version 10.4
Through sheer dumb trial and error luck I tried swapping MobaXTerm for PuTTY (continuing to use screen) and the problem went away. I have no idea why this works. The universe is a complex place which surpasses mere human comprehension.
Related
I am using a very complex setup to test various non-public webpages. I use jenkins to run the python-selenium tests within a dockerimage. That way, I am completely independent of the jenkins environment and can create my own environment. In this environment I have the following software installed:
Ubuntu 16.04.3
Firefox: Mozilla Firefox 57.0.1
geckodriver: 0.18.0
nosetests: 1.3.7
selenium: 3.8.0
When running the tests, which mostly succeed, I see in the geckodriver.log output messages like
(firefox:55): GConf-WARNING **: Client failed to connect to the D-BUS daemon:
/usr/bin/dbus-launch terminated abnormally without any error message
My questions:
What does this message mean?
Could that be an indication of the reason why sometimes the tests are failing?
If so, how to fix it?
The error you are seeing is :
(firefox:55): GConf-WARNING **: Client failed to connect to the D-BUS daemon:
/usr/bin/dbus-launch terminated abnormally without any error message
So it is clear Firefox attempts to connect to D-BUS daemon and fails as dbus-launch gets terminated abnormally.
dbus-launch
dbus-launch is basically the utility to start a message bus by firefox through a shell script. It would normally be called from a user's login scripts. dbus-launch launches a session bus instance and print the address and pid of that instance to standard output.
You can read more about dbus-launch from the Linux man page
Root Cause
This error can arise if you use su (root), sudo, suedit, gksu. The main reason is DBUS_SESSION_BUS_ADDRESS retains its value when you su instead of picking up the value in /root/.dbus/session-bus.
Here you will find a detailed discussion on GConf Error: No D-BUS daemon running?! How to reinstall or fix?.
Another possible reason may be the base Firefox Browser version may be an older version on which updates were taken to reach the current version.
Solution :
There are a couple of solutions available to address this ubuntu related issue as follows :
Before you start Firefox you have to type export $(dbus-launch)
However this may result into another error with NSS_USE_SHARED_DB. So you have to use export NSS_USE_SHARED_DB=ENABLED as well.
The most convenient way would be to put all the configuration with in .bashrc file :
export $(dbus-launch)
export NSS_USE_SHARED_DB=ENABLED
firefox &
This discussion speaks about the solution in details.
If dbus-launch is not installed on your system you have to install dbus-x11 package which contains the dbus-launch program.
An effective solution would be to uninstall the older base version of Firefox Browser and install a recent released GA version of Firefox Browser.
Best Approach
The issue with dbus-launch was addressed properly by both Ubuntu and Mozila. To overcome this error you need to follow the below mentioned steps :
Keep your Ubuntu os updated with the Latest Patch Releases and updates.
Always use the latest released version of Selenium-Python client, WebDriver variant e.g. GeckoDriver and Web Browser, e.g. Firefox Browser.
Clean and Build the Project Workspace afresh through your IDE before and after executing your Test Suite.
Clear the Browser Cache before and after the execution of your Tests.
If you have to uninstall any of the Web Client variants (e.g. Mozilla Firefox) you can use Revo Uninstallar with Moderate Memory Scan so that the stale registry settings are discarded.
Use CCleaner tool regularly to wipe away the OS chores including the stale rust_mozprofile directories.
What does this message mean?
DBus is a message bus system for interprocess commutation. There is an open geckodriver issue on a similar if not the same subject:
request to geckodriver fails with no meaningful log entry when there is no access to $HOME/.mozilla or $HOME/.cache
Could that be an indication of the reason why sometimes the tests are failing?
The warning should not really affect the tests but it's difficult to speculate about your intermittent test failures without seeing what is actually happening in your tests.
If so, how to fix it?
Here are some things to try:
upgrade geckodriver to the latest stable version (currently 0.19.1)
update Firefox to the latest nightly version (currently 58)
try this answer
try this answer
For me the solution was to install dbus-x11
apt install dbus-x11
I use Selenium to do the web-scraping. However, my Firefox browser always crashes after a while (it is very intensive the task of scraping).
The error message from Firefox:
Firefox had a problem and crashed. We’ll try to restore your tabs and
windows when it restarts.
The error message from Python:
selenium.common.exceptions.WebDriverException: Message: Tried to run
command without establishing a connection
I'd like to know if there's a way to deal with that. Ideally, I want something like after Firefox crashes, Selenium would open a new tab and continue from where it crashed.
i tried try: except:pass this type of code and they didn't work out.
I am trying to run google chrome browser for headless testing in RHEL7 containers running in Openshift cluster. The idea is to use the browser for selenium automation.
The test runs perfectly in my local machine with Docker containers, I was able to get the screenshots - no issues. However, once I deploy them into OpenShift google chrome browser doesn't respond on time. Webdriver throws the below error.
> Webdriver error:
> {"message":"Unexpected data in simpleCallback.","data":"<html><body><h1>504 Gateway Time-out</h1>\nThe server didn't respond in time.\n</body></html>\n"}
Based on the message I failed to figure out the problem, I did run the google-chrome through CLI and below is what I get from within the container.
google-chrome-stable --no-sandbox --headless --disable-gpu
--window-size=1920,1080 --proxy-server=http://myproxy.net:9999 --screenshot="/tmp/sc1.png" http://www.mytestingsite.com
Fontconfig warning: "/etc/fonts/fonts.conf", line 86: unknown element "blank"
[1001/164944.431186:ERROR:gpu_process_transport_factory.cc(1007)] Lost UI shared context.
[1001/164944.450572:ERROR:zygote_host_impl_linux.cc(259)] Failed to adjust OOM score of renderer with pid 117: Permission denied (13)
Couple of things I suspect could be a problem, the permission denied error thrown by the non-privileged user or it is something else. I really need chrome to throw some verbose logging to figure this out. I tried enable '--enable-logging=v=1' doesn't help.
Could someone shed some light on how to debug this issue or any ideas what could cause such a thing in OpenShift platform?
I'm having troubles running Ruby/RSpec tests against Safari 11 when I'm trying to run tests via ssh manually or via Jenkins (where machine where Safari exists is remote slave).
When executing tests, I'm getting following error:
Selenium::WebDriver::Error::WebDriverError:
unable to connect to safaridriver 127.0.0.1:7050
What is weird is that I'm able to run tests when I'm logged to the Mac machine directly. This leads me to the conclusion that there could be some permission which, by default, disables execution from ssh session but not sure why?
Also, to my knowledge, Safari Driver is part of Safari 10+ and as such is not installed anymore as extension
Update: I've found out that safaridriver executable that should be spawned by tests cannot be spawned from some reason when I login via ssh.
Example:
/usr/bin/safaridriver -p 7050
It will just terminate with non-zero exit code while running same command directly on machine will run safaridriver in foreground. Since I expect Jenkins to be running test job on this machine, my idea to overcome this issue would be to launch safaridriver on this machine (by cron or launchctl) and then use this instance to connect to it with my tests. However, so far, I was not able to make my tests (Selenium/Capybara) re-use existing safaridriver instead of always trying to spawn new one on different port. Any idea on this would also be greatly appreciated.
This is my environment:
OS: MacOS Sierra 10.12.6
Browser: Safari 11.0.2
Thanks in advance
There is a possible workaround to do this. First you need to create a Automator Workflow or Apple Script that launches the SafariDriver
Then save this as a application. Let assume we name it SafariDriver7050.
Then from the SSH session you need to execute
open /Applications/SafariDriver7050.app
This will actually launch SafariDriver in the logged in session and it should work for you.
The caveat being shutting it down, you will need to first kill the SafariDriver7050 app and then you need to kill the safaridriver process. The order matters, else it will create a error dialog on UI
Edit-1:
As you suggested, it would be even easier to do this, when you wrap your test as a app and then it will be automatically be able to launch SafariDriver without any issues. The key to issue is using open command in a SSH session
Edit-2
Why does SafariDriver not work in SSH? Well if you look at the linux counterpart
In case of linux we can use the DISPLAY environment variable to launch an app in an existing display or we can use something like XVFB to launch the browser in a virtual display. That is the concept that most frameworks use in case of linux machines.
But Mac doesn't have such kind of feature, which is why this workaround is needed. Now why it doesn't have that, I am not sure. There may be some other workaround that I may not be aware of, so anyone who has valuable info, can help improve this part of the answer
For my case:
On OSX host side was created Automator app as described above and modified a little bit:
security unlock-keychain -p your_host_password /Users/$USER/Library/Keychains/login.keychain-db
safaridriver --enable
safaridriver -p 7050
On CCI/Jenkins side:
open /Applications/StartSafariDriver7050.app/
run pytest cmd
osascript -e 'quit app "StartSafariDriver7050"'
pkill safaridriver
In Python:
def safaridriver():
return webdriver.Safari(desired_capabilities=Caps.SAFARI, port=7050)
We started a new Selenium environment with Selenium 3.4, using a grid configuration with a Linux-based hub and a Windows Server-based node. InternetExplorerDriver started throwing NoSuchWindowException when run remotely. I got the following output:
org.openqa.selenium.support.ui.ExpectedConditions findElement
WARNING: WebDriverException thrown by findElement(By.xpath: //h2[starts-with(#class, 'headline')])
org.openqa.selenium.NoSuchWindowException: Unable to find elements on closed window (WARNING: The server did not provide any stacktrace information)
[many more stack trace lines]
Caused by: org.openqa.selenium.remote.ScreenshotException: Screen shot has been taken
at org.openqa.selenium.remote.ErrorHandler.throwIfResponseFailed(ErrorHandler.java:141)
... 43 more
Ultimately it repeats the NoSuchWindowException over and over afterward.
We're running IE 11, so I ensured that Enhanced Protected Mode was disabled, the proper registry key fixes had been applied, and so on as described at SeleniumHQ (https://github.com/SeleniumHQ/selenium/wiki/InternetExplorerDriver#required-configuration), as well as many answers here on StackOverflow. Chrome works fine, Firefox works fine, IE even works fine as long as it is run locally. I can't find any more information on this.
How do I stop this exception from occurring?
For us, it was a matter of the Windows Server running the node (we had been running it on a Windows 7 machine previously). The node had to be run as an administrator. We ultimately created a shortcut and set it to be run as an administrator through the Shortcut tab -> Advanced button.