How to get Chrome Dev Tools logs with Selenium C#? - selenium

My aim is getting the transfer size of page. Explained below image.
How can I get it with C# code using selenium NuGet package? I searched chrome dev tools.
https://chromedevtools.github.io/devtools-protocol/tot/Network/
I see there is Network.dataReceived property. As I understand I need to get this.
My code
using OpenQA.Selenium.Chrome;
using OpenQA.Selenium;
ChromeOptions options = new ChromeOptions();
//Following Logging preference helps in enabling the performance logs
options.SetLoggingPreference("performance", LogLevel.All);
//Based on your need you can change the following options
options.AddUserProfilePreference("intl.accept_languages", "en-US");
options.AddUserProfilePreference("disable-popup-blocking", "true");
options.AddArgument("test-type");
options.AddArgument("--disable-gpu");
options.AddArgument("no-sandbox");
options.AddArgument("start-maximized");
options.LeaveBrowserRunning = true;
//Creating Chrome driver instance
IWebDriver driver = new ChromeDriver(options);
var url = " https://...............";
driver.Navigate().GoToUrl(url);
//Extracting the performance logs
var logs = driver.Manage().Logs.GetLog("performance");
for (int i = 0; i < logs.Count; i++)
{
Console.WriteLine((i+1) + " - " + logs[i].Message);
}
I found some code that possibly worked in old versions. I am using Chrome 105 version(as a nuget package, and selenium web dirver 4.0.0) and Chrome should be 105 version. In C# I can not find the below code, this is Java version:
for (LogEntry entry : driver.manage().logs().get(LogType.PERFORMANCE)) {
if(entry.getMessage().contains("Network.dataReceived")) {
Matcher dataLengthMatcher = Pattern.compile("encodedDataLength\":(.*?),").matcher(entry.getMessage());
dataLengthMatcher.find();
}
Even if we can do this with C# code, I dont believe driver.Manage().Logs.GetLog("performance") returns the same with the listed lines in dev tools network section. Is there nay option about that?

Related

Selenium GeckoDriver failing with blank page

I'm getting a blank page when trying to verify the driver url. I have this set up in debug logging mode. This is the FirefoxWebDriver.
UPDATE: If I take out all webdriver options then the tests work on a server but do not work in a Docker container.
FINAL UPDATE: Switched over to Chrome and found success, but with some important caveats.
[TestMethod]
public void NavigatesToContainerSite()
{
WebDriverWait webDriverWait = new WebDriverWait(_driver, TimeSpan.FromSeconds(30));
_driver.Navigate().GoToUrl("http://www.example.com:8083/");
string url = _driver.Url;
Assert.AreEqual("http://www.example.com:8083/", url);
}
[ClassInitialize]
public static void SetupOnce(TestContext context)
{
_options = new FirefoxOptions();
Proxy p = new Proxy();
p.SocksProxy = proxy url;
p.SocksVersion = 5;
p.Kind = ProxyKind.Manual;
_options.Proxy = p;
_options.PageLoadStrategy = PageLoadStrategy.None;
_options.LogLevel = FirefoxDriverLogLevel.Debug;
_options.AddArguments("--no-sandbox", "--disable-software-rasterizer","--headless" ,"--whitelisted-ips=\"\"",
"--disable-infobars","--disable-gpu","--disable-dev-shm-usage","--disable-extensions");
_driver = new FirefoxDriver(_options);
}
Here is the logging aspect of the request. If I just create a passing test that navigates to the url it seems to work but when I try to get something off the page (url, page elements) it throws exception.
Question: does the 200 OK indicate the driver.navigate succeeded or does it just mean the request for the webdriver session succeeded?
I just put the example.com url in this code example to hide my real url.
"1662816403483\twebdriver::server\tDEBUG\t-> POST /session/b29b42e9-cef5-4ef3-a281-8aafb21ea8cc/url {\"url\":\"http://www.example.com:8083/?username=owen.charles\"}",
"1662816403484\tMarionette\tDEBUG\t0 -> [0,2,\"WebDriver:Navigate\",{\"url\":\"http://www.example.com:8083/"}]",
"1662816403487\tMarionette\tDEBUG\t0 <- [1,2,null,{\"value\":null}]",
"1662816403487\twebdriver::server\tDEBUG\t<- 200 OK {\"value\":null}",
"1662816403541\twebdriver::server\tDEBUG\t-> POST /session/b29b42e9-cef5-4ef3-a281-8aafb21ea8cc/url {\"url\":\"http://www.example.com:8083/"}",
"1662816403542\tMarionette\tDEBUG\t0 -> [0,3,\"WebDriver:Navigate\",{\"url\":\"http://www.example.com:8083/"}]",
"1662816403543\tMarionette\tDEBUG\t0 <- [1,3,null,{\"value\":null}]",
"1662816403543\twebdriver::server\tDEBUG\t<- 200 OK {\"value\":null}",
"1662816403580\twebdriver::server\tDEBUG\t-> GET /session/b29b42e9-cef5-4ef3-a281-8aafb21ea8cc/url ",
"1662816403581\tMarionette\tDEBUG\t0 -> [0,4,\"WebDriver:GetCurrentURL\",{}]",
"1662816403581\tMarionette\tDEBUG\t0 <- [1,4,null,{\"value\":\"about:blank\"}]",
"1662816403581\twebdriver::server\tDEBUG\t<- 200 OK {\"value\":\"about:blank\"}",
"1662816403657\twebdriver::server\tDEBUG\t-> DELETE /session/b29b42e9-cef5-4ef3-a281-8aafb21ea8cc ",
"1662816403658\tMarionette\tDEBUG\t0 -> [0,5,\"Marionette:Quit\",{\"flags\":[\"eForceQuit\"]}]",
"1662816403658\tMarionette\tINFO\tStopped listening on port 49197",
This method doesn't produce an error:
[TestMethod]
public void NavigatesToContainerSite()
{
WebDriverWait webDriverWait = new WebDriverWait(_driver, TimeSpan.FromSeconds(30));
_driver.Navigate().GoToUrl("http://www.example.com:8083");
Assert.AreEqual("one", "one");
}
No one answered this but I did come up with a solution so I will just post it here. I had enough issues with the GeckoWebDriver that I switched to ChromeWebDriver. But the solution I arrived at may resolve the Gecko issue as well.
Steps I took:
Downloaded Chrome, major version 105
Incorporated ChromeWebDriver, major version 105 (these two have to match!)
Set two arguments when I instantiate the ChromeWebDriver in code
ChromeOptions options = new ChromeOptions();
options.AddArguments("--no-sandbox","--headless");
return new ChromeDriver(options);
Then it came to changes in the dockerfile for deploying this in a container. Because I have a Windows Docker host, I must deploy to a Windows container. I found out that Chrome requires about four or five fonts, and their various iterations, for its default installation to work. The Windows container only has one font in the installation. If you pull up Chrome settings, you'll probably see these fonts. One is for serif fonts, another sans-serif, etc. These fonts are:
arial.ttf
arialbd.ttf
arialbi.ttf
ariali.ttf
arialn.ttf
arialnb.ttf
arialni.ttf
ariblk.ttf
consola.ttf
consolab.ttf
consolaz.ttf
times.ttf
timesbi.ttf
timesi.ttf
I had to manually install these fonts into the windowsservercore image of Windows.
The resulting dockerfile looks like this:
FROM mcr.microsoft.com/dotnet/sdk:6.0-windowsservercore-ltsc2019
COPY ./test /test
WORKDIR /test
SHELL ["powershell"]
RUN "Add-WindowsFeature Web-WebSockets"
RUN Fonts/Add-Font.ps1 Fonts/Fonts
RUN ["msiexec","/i","chrome.msi","/qn"]
ENTRYPOINT ["dotnet", "test", "--logger:trx"]
In addition, I had to allocate 2g of shared memory in my Docker run statement.
docker run --shm-size="2g"
I found that the Selenium tests kick off about 5 instances of the browser and the web driver, sometimes taking up 600 MB of RAM. This will disappear if you use the driver.quit method after test is complete.
And voila, everything green.

How to enable IE mode in Chromium Edge Browser in selenium C#?

I want to automate a website in Edge which is require IE mode to be enabled. How can launch Edge in IE mode in selenium?
Below code which I currently use launches Edge in non IE mode, which won't display the website properly.
Dim edgeDriverService = Microsoft.Edge.SeleniumTools.EdgeDriverService.CreateChromiumService()
Dim edgeOptions = New Microsoft.Edge.SeleniumTools.EdgeOptions()
edgeOptions.PageLoadStrategy = PageLoadStrategy.Normal
edgeOptions.UseChromium = True
Dim driver As IWebDriver = New Microsoft.Edge.SeleniumTools.EdgeDriver(edgeDriverService, edgeOptions)
driver.Navigate().GoToUrl("http://example.com")
Tried using edgeOptions.AddAdditionalCapability("ie.edgechromium", True)but it didn't work
You could refer to the section Automating Internet Explorer mode in this article about how to use IE mode in Edge Chromium in Selenium C#.
You could refer to the following steps:
Download the latest version of IEDriverServer from Selenium site. Here I use 32 bit Windows IE version 3.150.1.
Make some preparations to use IEDriver according to this.
Create a C# console project using Visual Studio.
Install Selenium.WebDriver 3.141.0 nuget package from Nuget package manager.
Add the code below to the project and modify the paths to your owns in the code:
static void Main(string[] args)
{
var dir = "{FULL_PATH_TO_IEDRIVERSERVER}";
var driver = "IEDriverServer.exe";
if (!Directory.Exists(dir) || !File.Exists(Path.Combine(dir, driver)))
{
Console.WriteLine("Failed to find {0} in {1} folder.", dir, driver);
return;
}
var ieService = InternetExplorerDriverService.CreateDefaultService(dir, driver);
var ieOptions = new InternetExplorerOptions{};
ieOptions.AddAdditionalCapability("ie.edgechromium", true);
ieOptions.AddAdditionalCapability("ie.edgepath", "{FULL_PATH_TO_MSEDGE.EXE}");
var webdriver = new InternetExplorerDriver(ieService, ieOptions, TimeSpan.FromSeconds(30));
webdriver.Url = "http://www.example.com";
}
Run the project to test:
Notes:
Make sure to close all the Edge browser tabs and window before running the code.
Use full paths in the code. For example: ieOptions.AddAdditionalCapability("ie.edgepath", #"C:\Program Files (x86)\Microsoft\Edge\Application\msedge.exe");.

Unable to install WebExtension with Selenium

I'm trying to test my firefox webextension but firefox refuses to install it because it doesn't have the install.rdf file. But that file is not used anymore by webextensions.
const firefox = require('selenium-webdriver/firefox');
const webdriver = require('selenium-webdriver');
require('geckodriver');
let profile = new firefox.Profile();
profile.addExtension(process.cwd() + '/build/firefox/');
profile.setPreference('extensions.firebug.showChromeErrors', true);
let options = new firefox.Options().setProfile(profile);
let _driver = new webdriver.Builder()
.forBrowser('firefox')
.setFirefoxOptions(options)
.build();
Error: ENOENT: no such file or directory, open '/dev/webext/build/firefox/install.rdf'
Is there a setting that I need to enable to tell it it's a webextension?
The WebExtension API is not yet supported by Selenium v3.4.0 . The method Profile::addExtension only works for a legacy addon where install.rdf is present at the root.
To test a web extension, you can either use a profile where the extension is already installed, or you can implement the custom command available with GeckoDriver since v0.17.0:
var webdriver = require('selenium-webdriver');
var Command = require('selenium-webdriver/lib/command').Command;
function installWebExt(driver, extension) {
let cmd = new Command('moz-install-web-ext')
.setParameter('path', path.resolve(extension))
.setParameter('temporary', true);
driver.getExecutor()
.defineCommand(cmd.getName(), 'POST', '/session/:sessionId/moz/addon/install');
return driver.schedule(cmd, 'installWebExt(' + extension + ')');
}
var driver = new webdriver.Builder()
.forBrowser('firefox')
.build();
installWebExt(driver, "C:\\temp\\extension.zip");
This is an issue with FirefoxDriver. This issue is already logged in both SeleniumHQ/selenium#4184 and
mozilla/geckodriver#759
GeckoDriver says that
A workaround for the time being would be to use the add-on endpoints
geckodriver 0.17.0 provides to get an extension installed from the
local disk.
https://github.com/mozilla/geckodriver/blob/release/src/marionette.rs#L66
So you have to use the geckodriver endpoints to do that. I have already mentioned on how to use the endpoints here

How to disable 'This type of file can harm your computer' pop up

I'm using selenium chromedriver for automating web application.
In my application, I need to download xml files. But when I download xml file, I get 'This type of file can harm your computer' pop up. I want to disable this pop up using selenium chromedriver and I want these type of files to be downloaded always. How can this be done?
Selenium version : 2.47.1
Chromedriver version : 2.19
UPDATE it's long standing Chrome bug from 2012.
The problem with XML files started to happen to me as of Chrome 47.0.2526.80 m.
After spending maybe 6 hours trying to turn off every possible security option I tried a different approach.
Ironically, it seems that turning on the Chrome option "Protect you and your device from dangerous sites" removes the message "This type of file can harm your computer. Do you want to keep file.xml anyway?"
I am using 'Ruby' with 'Watir-Webdriver' where the code looks like this:
prefs = {
'safebrowsing' => {
'enabled' => true,
}
}
b = Watir::Browser.new :chrome, :prefs => prefs
Starting the browser like this, with safebrowsing option enabled, downloads the xml files without the message warning. The principle should be the same for Selenium with any programming language.
#####
Edited: 13-04-2017
In latest version of Google Chrome the above solution is not enough. Additionally, it is necessary to start the browser with the following switch:
--safebrowsing-disable-download-protection
Now, the code for starting the browser would look something like this:
b = Watir::Browser.new :chrome, :prefs => prefs, :switches => %w[--safebrowsing-disable-download-protection]))
I am posting below the complete code that got file download working for me:
Hope it helps :-) I am using Java-Selenium
System.setProperty("webdriver.chrome.driver", "C:/chromedriver/chromedriver.exe");
String downloadFilepath = "D:/MyDeskDownload";
HashMap<String, Object> chromePrefs = new HashMap<String, Object>();
chromePrefs.put("profile.default_content_settings.popups", 0);
chromePrefs.put("download.default_directory", downloadFilepath);
chromePrefs.put("safebrowsing.enabled", "true");
ChromeOptions options = new ChromeOptions();
options.setExperimentalOption("prefs", chromePrefs);
DesiredCapabilities cap = DesiredCapabilities.chrome();
cap.setCapability(CapabilityType.ACCEPT_SSL_CERTS, true);
cap.setCapability(ChromeOptions.CAPABILITY, options);
WebDriver driver = new ChromeDriver(cap);
Following Python code works for me
chromeOptions = webdriver.ChromeOptions()
prefs = {'safebrowsing.enabled': 'false'}
chromeOptions.add_experimental_option("prefs", prefs)
driver = webdriver.Chrome(chrome_options=chromeOptions)
The accepted answer stopped working after a recent update of Chrome. Now you need to use the --safebrowsing-disable-extension-blacklist and --safebrowsing-disable-download-protection command-line switches. This is the WebdriverIO config that works for me:
var driver = require('webdriverio');
var client = driver.remote({
desiredCapabilities: {
browserName: 'chrome',
chromeOptions: {
args: [
'disable-extensions',
'safebrowsing-disable-extension-blacklist',
'safebrowsing-disable-download-protection'
],
prefs: {
'safebrowsing.enabled': true
}
}
}
});
Note that I am also disabling extensions, because they generally interfere with automated testing, but this is not strictly needed to fix the problem with downloading XML and JavaScript files.
I found these switches by reading through this list. You can also see them in the Chromium source.
I came across this recently, using Katalon Studio, Chrome version 88.
Thankfully just the enabling safebrowsing did the trick. You access the settings via "Project Settings" and you navigate to "Desired Capabilities" -> "Web UI" -> "Chrome".
If you don't have a "prefs" setting add it and set the type to Dictionary. Then in the Value add a boolean named "safebrowsing.enabled" and set the value to "true".
Result might look like:
(And you can the default directory setting has nothing to do with this example).
I used all of suggested chrome options in C# and only when my internet connected worked for me but when internet disconnected none of them work for me.
(I'm not sure.may be chrome safebrowsing need to internet connection)
By using older version of chrome(version71) and chromedriver(version 2.46) and after downloading,i saw downloaded XML file name constains 'Unconfirmed' with 'crdownload' extension
and parsing XML file wouldn't work properly.
Finally, creating wait with Thread.Sleep(1000) solved my problem.
IWebDriver Driver;
//chromedriver.exe version2.46 path
string path = #"C:\cd71";
ChromeDriverService driverService = ChromeDriverService.CreateDefaultService(path, "chromedriver.exe");
ChromeOptions options = new ChromeOptions();
// options.AddArgument("headless");
options.AddArgument("--window-position=-32000,-32000");
options.AddUserProfilePreference("download.default_directory", #"c:\xmlFiles");
options.AddUserProfilePreference("download.prompt_for_download", false);
options.AddUserProfilePreference("disable-popup-blocking", "true");
options.AddUserProfilePreference("safebrowsing.enabled", "true");
Driver = new ChromeDriver(driverService, options);
try
{
Driver.Navigate().GoToUrl(url);
Thread.Sleep(1000);
//other works like: XML parse
}
catch
{
}
For context, I had a .csv with a list of .swf flash documents and their respective urls. Since the files could only be accessed after a valid login, I couldn't use a simple requests based solution.
Just like .xml, downloading a .swf triggers a similar prompt.
None of the answers worked for me.
So I stripped away all the extra arguments and settings the above answers proposed and just made the chrome instance headless.
options.add_argument("--headless")
prefs = {
"download.default_directory": "C:\LOL\Flash",
"download.prompt_for_download": False,
"download.directory_upgrade": True,
}
options.add_experimental_option("prefs",prefs)
The only workaround that works for me:
Use argument with path to chrome profile
chromeOptions.add_argument(chrome_profile_path)
Search for file download_file_types.pb in chrome profile folder.
In my case ..chromedriver\my_profile\FileTypePolicies\36\download_file_types.pb
Backup this file and then open with any hexeditor (you can use oline).
Search for the filetype you want to download, i.e. xml and change it to anything i.e. xxx
Im using Google Version 80.0.3987.122 (Official Build) (32-bit) and ChromeDriver 80.0.3987.106. Getting the same error even after adding the below while downloading a .xml file.
$ChromeOptions = New-Object OpenQA.Selenium.Chrome.ChromeOptions
$ChromeOptions.AddArguments(#(
"--disable-extensions",
"--ignore-certificate-errors"))
$download = "C:\temp\download"
$ChromeOptions.AddUserProfilePreference("safebrowsing.enabled", "true");
$ChromeOptions.AddUserProfilePreference("download.default_directory", $download);
$ChromeOptions.AddUserProfilePreference("download.prompt_for_download", "false");
$ChromeOptions.AddUserProfilePreference("download.directory_upgrade", "true");
$ChromeDriver = New-Object OpenQA.Selenium.Chrome.ChromeDriver($chromeOptions)

Selenium 2 chrome driver fails with WebException inside HttpCommandExecutor

I am trying to setup an automated test environment using
- TestStack.Seleno v0.8.2
- TestStack.BDDfy v4.0.0
- Selenium .NET WebDriver 2.43.0.0
- Chrome v38
- ChromeDriver v2.9
While I am able to establish initial session hand shake between chrome driver and chrome browser, subsequent calls to actual web application via chrome driver are failing with timeout exceptions.
Here is the code to instantiate a SelenoHost object :
var options = new ChromeOptions();
Instance.Run(configure => configure
.WithWebServer(new InternetWebServer(String.Format("http://{0}/portal", IISServerHost)))
.WithRemoteWebDriver(() => BrowserFactory.Chrome(options))
.UsingLoggerFactory(new ConsoleFactory()));
If I debug the above method call, it fails inside SelenoApplication Initialize method :
public void Initialize()
{
_initialised = true;
_logger.Debug("Starting Webserver");
WebServer.Start();
_logger.Debug("Browsing to base URL");
Browser.Navigate().GoToUrl(WebServer.BaseUrl); >>> this line fails inside HttpCommandExecutor.CreateResponse() method
}
Not able to figure out what obvious am i missing out here.
BTW, web application is hosted on IIS 7.5 and is configured for windows authentication.
Launching chrome with 'no-sandbox' resolved the issue.
Here is how the final configuration looks like :
var options = new ChromeOptions();
options.AddArgument("ignore-certificate-errors");
options.AddArgument("no-sandbox");
var driverService = ChromeDriverService.CreateDefaultService();
driverService.EnableVerboseLogging = (ChromeDriverVerboseLogigng == "true");
driverService.LogPath = ChromeDriverLogPath;
_SelenoHostLazy.Value.Run(configure => configure
.WithWebServer(new InternetWebServer(String.Format("http://localhost/portal", IISServerHost)))
.WithRemoteWebDriver(() => new ChromeDriver(driverService, options))
.UsingLoggerFactory(new ConsoleFactory()));