Unable to load resource due to SSL certificates using Selenium Webdriver with BrowserMob proxy to capture har file - ssl

I'm using Selenium Webdriver (Chromedriver) in Java, along with BrowserMob Proxy Server to capture HTTP traffic in a har file. I recently encountered a problem where sections of the website would not load, and I've narrowed it down to this error:
"Failed to load resource https://ajax.googleapis.com/ajax/libs/jquery/1.8.2/jquery.min.js".
It seems that when using the proxy server, the Selenium driver can't access the SSL certificate for the https link. Here is a snippet of the code I am using:
ProxyServer server = new ProxyServer(4040);
server.start();
Proxy proxy = server.seleniumProxy();
DesiredCapabilities capabilities = new DesiredCapabilities();
capabilities.setCapability(CapabilityType.PROXY, proxy);
driver = new ChromeDriver(capabilities);
server.newHar("myHar");
Har har = server.getHar();
server.stop();
I have tried adding "capabilities.setCapability(CapabilityType.ACCEPT_SSL_CERTS, true);" and this solves the above problem, but only if I delete "capabilities.setCapability(CapabilityType.PROXY, proxy);" and then I am no longer able to capture the har file.
I tried switching to a firefox driver and setting up a firefox profile, but either I am not doing it properly or it won't work for my purposes either.
I have also tried setting up the cybervillainsCA certificate that comes with BrowserMob proxy in Trusted Root Certifaction Authorities, but that hasn't solved the issue either.
Does anyone know how to solve this in a way that I can collect HTTP traffic as well as successfully load the resource mentioned above?

What I ended up doing was scrapping the proxy entirely, and instead switched to using a Firefox profile with the Firebug and NetExport extensions in order to export the har file. This ended up significantly slowing down the tests, and exports a har for each page visited rather than for the entire session.
Code:
//Firefox profile
FirefoxProfile ffProfile = new FirefoxProfile();
ffProfile.addExtension(new File("firebug-1.11.4-fx.xpi"));
ffProfile.addExtension(new File("netExport-0.9b3.xpi"));
//Set default Firefox preferences
ffProfile.setPreference("app.update.enabled", false);
String domain = "extensions.firebug.";
//Set default Firebug preferences
ffProfile.setPreference(domain + "currentVersion", "1.11.4");
ffProfile.setPreference(domain + "allPagesActivation", "on");
ffProfile.setPreference(domain + "defaultPanelName", "net");
ffProfile.setPreference(domain + "net.enableSites", true);
//Set default NetExport preferences
ffProfile.setPreference(domain + "netexport.alwaysEnableAutoExport", true);
ffProfile.setPreference(domain + "netexport.autoExportToFile", true);
ffProfile.setPreference(domain + "netexport.showPreview", false);
ffProfile.setPreference(domain + "netexport.defaultLogDir", "string file path");
//WebDriver, instantiated outside the method
driver = new FirefoxDriver(ffProfile);
s = new WebDriverBackedSelenium(driver, "http://www.google.ca/");
I collected the har file after each page as follows:
HarFileReader r = new HarFileReader();
HarFileWriter w = new HarFileWriter();
int count = 1;
String allHars = "";
String harFolderPath = "file path for har";
File dir = new File(harFolderPath);
for (File child : dir.listFiles()) {
HarLog log = r.readHarFile(child);
File f = new File(harFolderPath + "\\test"+count+".txt");
w.writeHarFile(log, f);
allHars = allHars + readFileAsString(f.getPath());
count++;
}
FileUtils.cleanDirectory(dir);

Related

Selenium Proxy IP Address Configuration IP Not Correct

I have paid/rented a proxy server in brazil
String proxyAddress = "myusername:myuserpass123#196.18.199.51:15464"
proxy.setAutodetect(false);
proxy.setHttpProxy(proxyAddress);
proxy.setSslProxy(proxyAddress);
chromeOptions.setCapability(CapabilityType.PROXY, proxy);
WebDriver webDriver = new ChromeDriver(chromeOptions);
I m running the webdriver on my local computer and I am in Indonesia. When the chrome browser opens up, I can debug and made sure that capabilities were set correctly: I can see the manual proxy setting set to the correct address string above.
However, when webdriver opens https://api.ipify.org/?format=json, it still returns my IP in Indonesia. What am I Missing here? My expectation is because I had configured webdriver to be proxied by a server in Brazil, https://api.ipify.org/?format=json should return Brazilian IP address?
Using Selenium 4 BiDirectional API (https://www.selenium.dev/documentation/webdriver/bidirectional/bidi_api/)
Register Basic Auth.
Some applications make use of browser authentication to secure pages. With Selenium, you can automate the input of basic auth credentials whenever they arise.
//C#
//Console App .NET 6
using OpenQA.Selenium;
using OpenQA.Selenium.Chrome;
Proxy proxy = new Proxy();
var proxyAddress = "address:port";
proxy.HttpProxy = proxyAddress;
proxy.SslProxy = proxyAddress;
ChromeOptions options = new ChromeOptions();
options.Proxy = proxy;
IWebDriver driver = new ChromeDriver(options);
NetworkAuthenticationHandler handler = new NetworkAuthenticationHandler()
{
UriMatcher = (d) => d.Host.Contains("your-domain.com"), // or set it `true` to enable proxy everywhere
Credentials = new PasswordCredentials("admin", "password")
};
INetwork networkInterceptor = driver.Manage().Network;
networkInterceptor.AddAuthenticationHandler(handler);
await networkInterceptor.StartMonitoring();
driver.Navigate().GoToUrl("https://api.ipify.org/?format=json");
await networkInterceptor.StopMonitoring();

Unable to install WebExtension with Selenium

I'm trying to test my firefox webextension but firefox refuses to install it because it doesn't have the install.rdf file. But that file is not used anymore by webextensions.
const firefox = require('selenium-webdriver/firefox');
const webdriver = require('selenium-webdriver');
require('geckodriver');
let profile = new firefox.Profile();
profile.addExtension(process.cwd() + '/build/firefox/');
profile.setPreference('extensions.firebug.showChromeErrors', true);
let options = new firefox.Options().setProfile(profile);
let _driver = new webdriver.Builder()
.forBrowser('firefox')
.setFirefoxOptions(options)
.build();
Error: ENOENT: no such file or directory, open '/dev/webext/build/firefox/install.rdf'
Is there a setting that I need to enable to tell it it's a webextension?
The WebExtension API is not yet supported by Selenium v3.4.0 . The method Profile::addExtension only works for a legacy addon where install.rdf is present at the root.
To test a web extension, you can either use a profile where the extension is already installed, or you can implement the custom command available with GeckoDriver since v0.17.0:
var webdriver = require('selenium-webdriver');
var Command = require('selenium-webdriver/lib/command').Command;
function installWebExt(driver, extension) {
let cmd = new Command('moz-install-web-ext')
.setParameter('path', path.resolve(extension))
.setParameter('temporary', true);
driver.getExecutor()
.defineCommand(cmd.getName(), 'POST', '/session/:sessionId/moz/addon/install');
return driver.schedule(cmd, 'installWebExt(' + extension + ')');
}
var driver = new webdriver.Builder()
.forBrowser('firefox')
.build();
installWebExt(driver, "C:\\temp\\extension.zip");
This is an issue with FirefoxDriver. This issue is already logged in both SeleniumHQ/selenium#4184 and
mozilla/geckodriver#759
GeckoDriver says that
A workaround for the time being would be to use the add-on endpoints
geckodriver 0.17.0 provides to get an extension installed from the
local disk.
https://github.com/mozilla/geckodriver/blob/release/src/marionette.rs#L66
So you have to use the geckodriver endpoints to do that. I have already mentioned on how to use the endpoints here

Selenium: Unable to create folder and file for download in the Jenkins pipeline

The test is to download a file by clicking the link for download template. When I execute the script in my local machine, it is working perfectly. Able to create the download folder and the file downloaded is stored in the newly created "download" folder.
But when I integrate it to the jenkins pipeline, there is no folder created and no file downloaded.
Note: In the jenkins pipeline, the script is executed using chrome in the selenium grid. Please refer to the configuration in the chrome driver.
HashMap<String, Object> chromePrefs = new HashMap<String, Object>();
chromePrefs.put("profile.default_content_settings.popups", 0);
chromePrefs.put("download.default_directory", System.getProperty("user.dir") + "\\src\\main\\resources\\downloads");
ChromeOptions options = new ChromeOptions();
HashMap<String, Object> chromeOptionsMap = new HashMap<String, Object>();
options.setExperimentalOption("prefs", chromePrefs);
options.addArguments("--test-type");
DesiredCapabilities cap = DesiredCapabilities.chrome();
cap.setCapability(ChromeOptions.CAPABILITY, chromeOptionsMap);
cap.setCapability(CapabilityType.ACCEPT_SSL_CERTS, true);
cap.setCapability(ChromeOptions.CAPABILITY, options);
driver = new RemoteWebDriver(new URL(seleniumGridUrl), cap);
If you are not going to check the contents of the downloaded file in your test, I'd like to suggest using a kind of by-pass approach. Using HTTP library, you may just execute Head http call to the resource, you need to download. In the response you'll get the file type and its size. In your test you simply assert against them to validate correctness of accessed for downloading file.
I think the issue you get is connected somehow with the permissions, that does not allow creating folders / files on the environment, where test is executed.

Unable to Fetch Network Traffic Logs via BrowserMob Proxy with Selenium Webdriver

I am not able to Fetch Network Logs using Browsermob Proxy when set Http proxy , it will just create a har file but i am not able to see any logs inside the file
Below is my Code :-
String strFilePath = "./PerformanceLogs/PerformanceLogs.har";
String noProxy = "localhost, 127.0.0.1";
LegacyProxyServer server = new BrowserMobProxyServer();
server.start();
server.setCaptureHeaders(true);
server.setCaptureContent(true);
Proxy proxy = server.seleniumProxy().setHttpProxy(PROXY)
.setFtpProxy(PROXY)
.setSslProxy(PROXY)
.setNoProxy(noProxy);
DesiredCapabilities capabilities = DesiredCapabilities.firefox();
capabilities.setCapability(CapabilityType.PROXY, proxy);
WebDriver driver = new FirefoxDriver(capabilities);
server.newHar("NetworkLogs");
driver.navigate().to("https://url-");
driver.findElements(xpath="").click();
pMonitor = new Prism_Selenium_Monitor_Tab(driver);
pMonitor.prism_monitor_credentialDeploy(URL,excelfile[0], excelfile[4]);
Har har = server.getHar();
File harFile = new File(strFilePath);
har.writeTo(harFile);
server.stop();
can anyone please me in this?
Thanks in advance
For some reason you're starting LegacyProxy, try starting BrowserMobProxyServer, i.e.
BrowserMobProxyServer server = new BrowserMobProxyServer();
Default proxy type is HTTP, i.e. try to delete this part:
.setNoProxy(noProxy);
It's actually shouldn't be a problem, but in default case there is no need to name your HAR, i.e. this code is redundant:
server.newHar("NetworkLogs");
you may just:
server.newHar();

Browsermob proxy server crashes websites

trying to use browsermob proxy server by falling code:
final int port = 9000;
server = new ProxyServer(port);
server.start();
final DesiredCapabilities dc = new DesiredCapabilities();
dc.setCapability(CapabilityType.PROXY, server.seleniumProxy());
setName("test");
FirefoxBinary binary = new FirefoxBinary(new File("C:\\Program Files (x86)\\ff21\\firefox.exe"));
File profileDir = new File("C:\\Users\\arno\\Documents\\profiles\\firefox21.default");
FirefoxProfile profile = new FirefoxProfile(profileDir);
driver = new FirefoxDriver(binary, profile, dc);//;
server.newHar("monitis");
but it crashes the website: see the capture .
I'm using the fallowing code to find out what is going on(getting the url and the http status )
Har har = server.getHar();
for(HarEntry entry : har.getLog().getEntries()){
System.out.println(entry.getRequest().getUrl() +": " + entry.getResponse().getStatus());
}
and it gives this result:
mysite/files/js/numeral.min.js: 200
mysite/js/94842541.js: 200
mysite/files/css/page-home.min.css?v=6: -999
mysite/files/css/ui.min.css?v=18: -999
mysite/files/js/ui.min.js?also=jquery.selectric.min.js,jquery.checkradios.min.js,index.min.js&v=53: -999
Try using the non-deprecated BrowserMobProxyServer class instead of the legacy ProxyServer implementation. Make sure you're using the latest version of BMP and the browsermob-core-littleproxy module as well.
it turned out, that in Firefox settings, in network settings tab, I need to set up the option "auto detect proxy server for this network"