Selenium GeckoDriver failing with blank page - selenium

I'm getting a blank page when trying to verify the driver url. I have this set up in debug logging mode. This is the FirefoxWebDriver.
UPDATE: If I take out all webdriver options then the tests work on a server but do not work in a Docker container.
FINAL UPDATE: Switched over to Chrome and found success, but with some important caveats.
[TestMethod]
public void NavigatesToContainerSite()
{
WebDriverWait webDriverWait = new WebDriverWait(_driver, TimeSpan.FromSeconds(30));
_driver.Navigate().GoToUrl("http://www.example.com:8083/");
string url = _driver.Url;
Assert.AreEqual("http://www.example.com:8083/", url);
}
[ClassInitialize]
public static void SetupOnce(TestContext context)
{
_options = new FirefoxOptions();
Proxy p = new Proxy();
p.SocksProxy = proxy url;
p.SocksVersion = 5;
p.Kind = ProxyKind.Manual;
_options.Proxy = p;
_options.PageLoadStrategy = PageLoadStrategy.None;
_options.LogLevel = FirefoxDriverLogLevel.Debug;
_options.AddArguments("--no-sandbox", "--disable-software-rasterizer","--headless" ,"--whitelisted-ips=\"\"",
"--disable-infobars","--disable-gpu","--disable-dev-shm-usage","--disable-extensions");
_driver = new FirefoxDriver(_options);
}
Here is the logging aspect of the request. If I just create a passing test that navigates to the url it seems to work but when I try to get something off the page (url, page elements) it throws exception.
Question: does the 200 OK indicate the driver.navigate succeeded or does it just mean the request for the webdriver session succeeded?
I just put the example.com url in this code example to hide my real url.
"1662816403483\twebdriver::server\tDEBUG\t-> POST /session/b29b42e9-cef5-4ef3-a281-8aafb21ea8cc/url {\"url\":\"http://www.example.com:8083/?username=owen.charles\"}",
"1662816403484\tMarionette\tDEBUG\t0 -> [0,2,\"WebDriver:Navigate\",{\"url\":\"http://www.example.com:8083/"}]",
"1662816403487\tMarionette\tDEBUG\t0 <- [1,2,null,{\"value\":null}]",
"1662816403487\twebdriver::server\tDEBUG\t<- 200 OK {\"value\":null}",
"1662816403541\twebdriver::server\tDEBUG\t-> POST /session/b29b42e9-cef5-4ef3-a281-8aafb21ea8cc/url {\"url\":\"http://www.example.com:8083/"}",
"1662816403542\tMarionette\tDEBUG\t0 -> [0,3,\"WebDriver:Navigate\",{\"url\":\"http://www.example.com:8083/"}]",
"1662816403543\tMarionette\tDEBUG\t0 <- [1,3,null,{\"value\":null}]",
"1662816403543\twebdriver::server\tDEBUG\t<- 200 OK {\"value\":null}",
"1662816403580\twebdriver::server\tDEBUG\t-> GET /session/b29b42e9-cef5-4ef3-a281-8aafb21ea8cc/url ",
"1662816403581\tMarionette\tDEBUG\t0 -> [0,4,\"WebDriver:GetCurrentURL\",{}]",
"1662816403581\tMarionette\tDEBUG\t0 <- [1,4,null,{\"value\":\"about:blank\"}]",
"1662816403581\twebdriver::server\tDEBUG\t<- 200 OK {\"value\":\"about:blank\"}",
"1662816403657\twebdriver::server\tDEBUG\t-> DELETE /session/b29b42e9-cef5-4ef3-a281-8aafb21ea8cc ",
"1662816403658\tMarionette\tDEBUG\t0 -> [0,5,\"Marionette:Quit\",{\"flags\":[\"eForceQuit\"]}]",
"1662816403658\tMarionette\tINFO\tStopped listening on port 49197",
This method doesn't produce an error:
[TestMethod]
public void NavigatesToContainerSite()
{
WebDriverWait webDriverWait = new WebDriverWait(_driver, TimeSpan.FromSeconds(30));
_driver.Navigate().GoToUrl("http://www.example.com:8083");
Assert.AreEqual("one", "one");
}

No one answered this but I did come up with a solution so I will just post it here. I had enough issues with the GeckoWebDriver that I switched to ChromeWebDriver. But the solution I arrived at may resolve the Gecko issue as well.
Steps I took:
Downloaded Chrome, major version 105
Incorporated ChromeWebDriver, major version 105 (these two have to match!)
Set two arguments when I instantiate the ChromeWebDriver in code
ChromeOptions options = new ChromeOptions();
options.AddArguments("--no-sandbox","--headless");
return new ChromeDriver(options);
Then it came to changes in the dockerfile for deploying this in a container. Because I have a Windows Docker host, I must deploy to a Windows container. I found out that Chrome requires about four or five fonts, and their various iterations, for its default installation to work. The Windows container only has one font in the installation. If you pull up Chrome settings, you'll probably see these fonts. One is for serif fonts, another sans-serif, etc. These fonts are:
arial.ttf
arialbd.ttf
arialbi.ttf
ariali.ttf
arialn.ttf
arialnb.ttf
arialni.ttf
ariblk.ttf
consola.ttf
consolab.ttf
consolaz.ttf
times.ttf
timesbi.ttf
timesi.ttf
I had to manually install these fonts into the windowsservercore image of Windows.
The resulting dockerfile looks like this:
FROM mcr.microsoft.com/dotnet/sdk:6.0-windowsservercore-ltsc2019
COPY ./test /test
WORKDIR /test
SHELL ["powershell"]
RUN "Add-WindowsFeature Web-WebSockets"
RUN Fonts/Add-Font.ps1 Fonts/Fonts
RUN ["msiexec","/i","chrome.msi","/qn"]
ENTRYPOINT ["dotnet", "test", "--logger:trx"]
In addition, I had to allocate 2g of shared memory in my Docker run statement.
docker run --shm-size="2g"
I found that the Selenium tests kick off about 5 instances of the browser and the web driver, sometimes taking up 600 MB of RAM. This will disappear if you use the driver.quit method after test is complete.
And voila, everything green.

Related

How to get Chrome Dev Tools logs with Selenium C#?

My aim is getting the transfer size of page. Explained below image.
How can I get it with C# code using selenium NuGet package? I searched chrome dev tools.
https://chromedevtools.github.io/devtools-protocol/tot/Network/
I see there is Network.dataReceived property. As I understand I need to get this.
My code
using OpenQA.Selenium.Chrome;
using OpenQA.Selenium;
ChromeOptions options = new ChromeOptions();
//Following Logging preference helps in enabling the performance logs
options.SetLoggingPreference("performance", LogLevel.All);
//Based on your need you can change the following options
options.AddUserProfilePreference("intl.accept_languages", "en-US");
options.AddUserProfilePreference("disable-popup-blocking", "true");
options.AddArgument("test-type");
options.AddArgument("--disable-gpu");
options.AddArgument("no-sandbox");
options.AddArgument("start-maximized");
options.LeaveBrowserRunning = true;
//Creating Chrome driver instance
IWebDriver driver = new ChromeDriver(options);
var url = " https://...............";
driver.Navigate().GoToUrl(url);
//Extracting the performance logs
var logs = driver.Manage().Logs.GetLog("performance");
for (int i = 0; i < logs.Count; i++)
{
Console.WriteLine((i+1) + " - " + logs[i].Message);
}
I found some code that possibly worked in old versions. I am using Chrome 105 version(as a nuget package, and selenium web dirver 4.0.0) and Chrome should be 105 version. In C# I can not find the below code, this is Java version:
for (LogEntry entry : driver.manage().logs().get(LogType.PERFORMANCE)) {
if(entry.getMessage().contains("Network.dataReceived")) {
Matcher dataLengthMatcher = Pattern.compile("encodedDataLength\":(.*?),").matcher(entry.getMessage());
dataLengthMatcher.find();
}
Even if we can do this with C# code, I dont believe driver.Manage().Logs.GetLog("performance") returns the same with the listed lines in dev tools network section. Is there nay option about that?

Cross browser test

I have to do login to an application with two different browsers (IE & FF) and hence I tried to do cross browser test. When the URL gets passed to IE, I am getting link as "Continue to this website not recommended", whereas the same link will not be displayed in FF. In the below if I use this statement driver.findElement(By.id("overridelink")).click(); then it works fine in IE but failing in firefox since there wont be any link in FF.
please let me know the possible where one script should run in both the browsers. Below is the script which am trying
#Test
#Parameters("browser")
public void verifypagetitle(String browsername) {
if(browsername.equalsIgnoreCase("IE"))
{
System.setProperty("webdriver.ie.driver",
"D:\\2.53.1\\IEDriverServer.exe");
driver = new InternetExplorerDriver();
}
else if (browsername.equalsIgnoreCase("firefox"))
{
System.setProperty("webdriver.firefox.bin",
"C:\\Program Files\\Mozilla Firefox\\firefox.exe");
driver = new FirefoxDriver();
}
driver.get("URL");
driver.manage().window().maximize();
driver.findElement(By.id("overridelink")).click();
driver.findElement(By.id("j_username")).sendKeys("userid");
driver.findElement(By.id("j_password")).sendKeys("pwd");
driver.findElement(By.id("j_password")).submit();
This is most likely caused by the IE security level. In my experience if I change it to Low, the warning is not present. You can try and see if this will solve the issue in your case.
Another solution is to add conditional logic to check the browser being used. Something like this:
Capabilities cap = ((RemoteWebDriver) browserDriver).getCapabilities();
String browsername = cap.getBrowserName();
// This block to find out if it is IE
if ("internet explorer".equalsIgnoreCase(browsername)) {
driver.findElement(By.id("overridelink")).click();
}

How to disable 'This type of file can harm your computer' pop up

I'm using selenium chromedriver for automating web application.
In my application, I need to download xml files. But when I download xml file, I get 'This type of file can harm your computer' pop up. I want to disable this pop up using selenium chromedriver and I want these type of files to be downloaded always. How can this be done?
Selenium version : 2.47.1
Chromedriver version : 2.19
UPDATE it's long standing Chrome bug from 2012.
The problem with XML files started to happen to me as of Chrome 47.0.2526.80 m.
After spending maybe 6 hours trying to turn off every possible security option I tried a different approach.
Ironically, it seems that turning on the Chrome option "Protect you and your device from dangerous sites" removes the message "This type of file can harm your computer. Do you want to keep file.xml anyway?"
I am using 'Ruby' with 'Watir-Webdriver' where the code looks like this:
prefs = {
'safebrowsing' => {
'enabled' => true,
}
}
b = Watir::Browser.new :chrome, :prefs => prefs
Starting the browser like this, with safebrowsing option enabled, downloads the xml files without the message warning. The principle should be the same for Selenium with any programming language.
#####
Edited: 13-04-2017
In latest version of Google Chrome the above solution is not enough. Additionally, it is necessary to start the browser with the following switch:
--safebrowsing-disable-download-protection
Now, the code for starting the browser would look something like this:
b = Watir::Browser.new :chrome, :prefs => prefs, :switches => %w[--safebrowsing-disable-download-protection]))
I am posting below the complete code that got file download working for me:
Hope it helps :-) I am using Java-Selenium
System.setProperty("webdriver.chrome.driver", "C:/chromedriver/chromedriver.exe");
String downloadFilepath = "D:/MyDeskDownload";
HashMap<String, Object> chromePrefs = new HashMap<String, Object>();
chromePrefs.put("profile.default_content_settings.popups", 0);
chromePrefs.put("download.default_directory", downloadFilepath);
chromePrefs.put("safebrowsing.enabled", "true");
ChromeOptions options = new ChromeOptions();
options.setExperimentalOption("prefs", chromePrefs);
DesiredCapabilities cap = DesiredCapabilities.chrome();
cap.setCapability(CapabilityType.ACCEPT_SSL_CERTS, true);
cap.setCapability(ChromeOptions.CAPABILITY, options);
WebDriver driver = new ChromeDriver(cap);
Following Python code works for me
chromeOptions = webdriver.ChromeOptions()
prefs = {'safebrowsing.enabled': 'false'}
chromeOptions.add_experimental_option("prefs", prefs)
driver = webdriver.Chrome(chrome_options=chromeOptions)
The accepted answer stopped working after a recent update of Chrome. Now you need to use the --safebrowsing-disable-extension-blacklist and --safebrowsing-disable-download-protection command-line switches. This is the WebdriverIO config that works for me:
var driver = require('webdriverio');
var client = driver.remote({
desiredCapabilities: {
browserName: 'chrome',
chromeOptions: {
args: [
'disable-extensions',
'safebrowsing-disable-extension-blacklist',
'safebrowsing-disable-download-protection'
],
prefs: {
'safebrowsing.enabled': true
}
}
}
});
Note that I am also disabling extensions, because they generally interfere with automated testing, but this is not strictly needed to fix the problem with downloading XML and JavaScript files.
I found these switches by reading through this list. You can also see them in the Chromium source.
I came across this recently, using Katalon Studio, Chrome version 88.
Thankfully just the enabling safebrowsing did the trick. You access the settings via "Project Settings" and you navigate to "Desired Capabilities" -> "Web UI" -> "Chrome".
If you don't have a "prefs" setting add it and set the type to Dictionary. Then in the Value add a boolean named "safebrowsing.enabled" and set the value to "true".
Result might look like:
(And you can the default directory setting has nothing to do with this example).
I used all of suggested chrome options in C# and only when my internet connected worked for me but when internet disconnected none of them work for me.
(I'm not sure.may be chrome safebrowsing need to internet connection)
By using older version of chrome(version71) and chromedriver(version 2.46) and after downloading,i saw downloaded XML file name constains 'Unconfirmed' with 'crdownload' extension
and parsing XML file wouldn't work properly.
Finally, creating wait with Thread.Sleep(1000) solved my problem.
IWebDriver Driver;
//chromedriver.exe version2.46 path
string path = #"C:\cd71";
ChromeDriverService driverService = ChromeDriverService.CreateDefaultService(path, "chromedriver.exe");
ChromeOptions options = new ChromeOptions();
// options.AddArgument("headless");
options.AddArgument("--window-position=-32000,-32000");
options.AddUserProfilePreference("download.default_directory", #"c:\xmlFiles");
options.AddUserProfilePreference("download.prompt_for_download", false);
options.AddUserProfilePreference("disable-popup-blocking", "true");
options.AddUserProfilePreference("safebrowsing.enabled", "true");
Driver = new ChromeDriver(driverService, options);
try
{
Driver.Navigate().GoToUrl(url);
Thread.Sleep(1000);
//other works like: XML parse
}
catch
{
}
For context, I had a .csv with a list of .swf flash documents and their respective urls. Since the files could only be accessed after a valid login, I couldn't use a simple requests based solution.
Just like .xml, downloading a .swf triggers a similar prompt.
None of the answers worked for me.
So I stripped away all the extra arguments and settings the above answers proposed and just made the chrome instance headless.
options.add_argument("--headless")
prefs = {
"download.default_directory": "C:\LOL\Flash",
"download.prompt_for_download": False,
"download.directory_upgrade": True,
}
options.add_experimental_option("prefs",prefs)
The only workaround that works for me:
Use argument with path to chrome profile
chromeOptions.add_argument(chrome_profile_path)
Search for file download_file_types.pb in chrome profile folder.
In my case ..chromedriver\my_profile\FileTypePolicies\36\download_file_types.pb
Backup this file and then open with any hexeditor (you can use oline).
Search for the filetype you want to download, i.e. xml and change it to anything i.e. xxx
Im using Google Version 80.0.3987.122 (Official Build) (32-bit) and ChromeDriver 80.0.3987.106. Getting the same error even after adding the below while downloading a .xml file.
$ChromeOptions = New-Object OpenQA.Selenium.Chrome.ChromeOptions
$ChromeOptions.AddArguments(#(
"--disable-extensions",
"--ignore-certificate-errors"))
$download = "C:\temp\download"
$ChromeOptions.AddUserProfilePreference("safebrowsing.enabled", "true");
$ChromeOptions.AddUserProfilePreference("download.default_directory", $download);
$ChromeOptions.AddUserProfilePreference("download.prompt_for_download", "false");
$ChromeOptions.AddUserProfilePreference("download.directory_upgrade", "true");
$ChromeDriver = New-Object OpenQA.Selenium.Chrome.ChromeDriver($chromeOptions)

How to handle favicon.ico in headless selenium with xvfb

I have the various scenarios of selenium test script which is properly running on selenium web driver with firefox browser. when i running them in headless mode some of scenarios are running but some of them are not running some time but most of time it fails and throw the error like
onhandlernotfound/favicon.ico
and
pagenotfound/favicon.ico[onhandlernotfound/favicon.ico]
the screenshot attached - first and second.
Give me the solution as soon as possible
my test case failed and get the error that element is not currently visible so may not be interacted with
element is not currently visible so may not be interacted with
You can try to get favicon internet-path and try to download it(externally, not via the selenium).
If favicon will be not accessible -- there will be no ability to download. If its is accessible -- its will be shown in any of browser (in case of enabled images) =)
Maybe this is not the best way, but this will work.
As you didnt write your language, I can give you code to download image for c#:
string webPath = ""; /*favicon internet path*/
if (webPath != string.Empty)
{
try
{
System.Net.WebRequest request = System.Net.WebRequest.Create(webPath);
System.Net.WebResponse response = request.GetResponse();
System.IO.Stream responseStream = response.GetResponseStream();
Bitmap bitmapImg = new Bitmap(responseStream);
return bitmapImg;
}
catch (System.Net.WebException)
{
}
}

Selenium IDE: Verify broken link 404 not found?

Any one have any idea whether Selenium IDE able to verify any broken link in page??
Short answer: yes, but probably not the best idea.
Selenium might not be the best program to do this - especially if you use only Selenium IDE.
Multiple programs exist with the sole purpose of testing broken links (do a little search), hell, even the W3C has a webapp for that.
That said, you could take the output from Selenium IDE formatted to your favourite language, program a Selenium test case with more advanced Unit Testing classes, include a while loop which would check for the existence of more links, and visit them / check headers, if you really want to do this through Selenium.
In short, you can't (yet). See a more full writeup of the situation at: How do I ask Selenium IDE to check a HTTP Status Code (e.g. 2XX, 404, 500)
Basically the selenium maintainers decided this function did not belong in selenium because error codes are read by machines not humans.
This sounds like a job for a simple spider script, for example using the wget on Linux
--spider
When invoked with this option, Wget will behave as a Web spider,
which means that it will not download the pages, just check that
they are there. For example, you can use Wget to check your book‐
marks:
wget --spider --force-html -i bookmarks.html
This feature needs much more work for Wget to get close to the
functionality of real web spiders.
You can use 404 status code to verify using REST web services.
Use gson and apache jars to get it.
public class Testing{
public static void main(String[] args) {
try{
WebDriver driver = new FirefoxDriver();
driver.manage().timeouts().implicitlyWait(10, TimeUnit.SECONDS);
driver.manage().window().maximize();
driver.navigate().to("https://www.amazon.in/");
List<WebElement> links = driver.findElements(By.tagName("a"));
System.out.println("Number of links : " + links.size());
for(WebElement e : links) {
String linkText = e.getAttribute("href");
System.out.println("Link -->>" +linkText);
if(linkText!=null && !linkText.isEmpty()) {
HttpPost post = new HttpPost(linkText);
HttpClient client = HttpClientBuilder.create().build();
HttpResponse res = client.execute(post);
String s = res.getStatusLine().toString();
if(s.contains("404")) {
System.out.println("Navigated");
//your code to handle logic
} else {
//your code to handle logic with other response code
}
}
}
} catch (Exception e) {
System.out.println(e.getStackTrace());
}
}
}