trying to use browsermob proxy server by falling code:
final int port = 9000;
server = new ProxyServer(port);
server.start();
final DesiredCapabilities dc = new DesiredCapabilities();
dc.setCapability(CapabilityType.PROXY, server.seleniumProxy());
setName("test");
FirefoxBinary binary = new FirefoxBinary(new File("C:\\Program Files (x86)\\ff21\\firefox.exe"));
File profileDir = new File("C:\\Users\\arno\\Documents\\profiles\\firefox21.default");
FirefoxProfile profile = new FirefoxProfile(profileDir);
driver = new FirefoxDriver(binary, profile, dc);//;
server.newHar("monitis");
but it crashes the website: see the capture .
I'm using the fallowing code to find out what is going on(getting the url and the http status )
Har har = server.getHar();
for(HarEntry entry : har.getLog().getEntries()){
System.out.println(entry.getRequest().getUrl() +": " + entry.getResponse().getStatus());
}
and it gives this result:
mysite/files/js/numeral.min.js: 200
mysite/js/94842541.js: 200
mysite/files/css/page-home.min.css?v=6: -999
mysite/files/css/ui.min.css?v=18: -999
mysite/files/js/ui.min.js?also=jquery.selectric.min.js,jquery.checkradios.min.js,index.min.js&v=53: -999
Try using the non-deprecated BrowserMobProxyServer class instead of the legacy ProxyServer implementation. Make sure you're using the latest version of BMP and the browsermob-core-littleproxy module as well.
it turned out, that in Firefox settings, in network settings tab, I need to set up the option "auto detect proxy server for this network"
Related
I have paid/rented a proxy server in brazil
String proxyAddress = "myusername:myuserpass123#196.18.199.51:15464"
proxy.setAutodetect(false);
proxy.setHttpProxy(proxyAddress);
proxy.setSslProxy(proxyAddress);
chromeOptions.setCapability(CapabilityType.PROXY, proxy);
WebDriver webDriver = new ChromeDriver(chromeOptions);
I m running the webdriver on my local computer and I am in Indonesia. When the chrome browser opens up, I can debug and made sure that capabilities were set correctly: I can see the manual proxy setting set to the correct address string above.
However, when webdriver opens https://api.ipify.org/?format=json, it still returns my IP in Indonesia. What am I Missing here? My expectation is because I had configured webdriver to be proxied by a server in Brazil, https://api.ipify.org/?format=json should return Brazilian IP address?
Using Selenium 4 BiDirectional API (https://www.selenium.dev/documentation/webdriver/bidirectional/bidi_api/)
Register Basic Auth.
Some applications make use of browser authentication to secure pages. With Selenium, you can automate the input of basic auth credentials whenever they arise.
//C#
//Console App .NET 6
using OpenQA.Selenium;
using OpenQA.Selenium.Chrome;
Proxy proxy = new Proxy();
var proxyAddress = "address:port";
proxy.HttpProxy = proxyAddress;
proxy.SslProxy = proxyAddress;
ChromeOptions options = new ChromeOptions();
options.Proxy = proxy;
IWebDriver driver = new ChromeDriver(options);
NetworkAuthenticationHandler handler = new NetworkAuthenticationHandler()
{
UriMatcher = (d) => d.Host.Contains("your-domain.com"), // or set it `true` to enable proxy everywhere
Credentials = new PasswordCredentials("admin", "password")
};
INetwork networkInterceptor = driver.Manage().Network;
networkInterceptor.AddAuthenticationHandler(handler);
await networkInterceptor.StartMonitoring();
driver.Navigate().GoToUrl("https://api.ipify.org/?format=json");
await networkInterceptor.StopMonitoring();
I try to open a Tor Browser using Selenium on Ubuntu 18. I have tried lots of examples but with no success.
proxyIP = "127.0.0.1"
proxyPort = "9050"
profileTor = '/etc/tor/' # torrc
binary = os.path.expanduser("~/.local/share/torbrowser/tbb/x86_64/tor-browser_en-US/Browser/firefox")
firefox_binary = FirefoxBinary(binary)
firefox_profile = FirefoxProfile(profileTor)
proxy_address = "127.0.0.1:9050"
proxy = Proxy({
'proxyType': ProxyType.MANUAL,
'httpProxy': proxy_address,
})
driver = webdriver.Firefox(firefox_binary = firefox_binary,firefox_profile=firefox_profile, proxy = proxy)
A blank Tor Browser window opens but after a while I get an error as:
selenium.common.exceptions.WebDriverException: Message: connection refused.
I have also tried an alternative to firefox binary the:
start-tor-browser
which opens a working Tor Browser and showing some index.
The script however stops and I cannot visit another page using Selenium unless I do it manually.
I have also tried the:
profile.default
as some examples suggest but I get an error:
Unable to start Tor. The torrc file is missing and could not be created.
To open a Tor Browser using Selenium you can start the Tor daemon first and then open the Tor Browser and you can use the following solution:
Sample WindowsOS style Code Block:
from selenium import webdriver
from selenium.webdriver.firefox.firefox_profile import FirefoxProfile
import os
torexe = os.popen(r'C:\Users\AtechM_03\Desktop\Tor Browser\Browser\TorBrowser\Tor\tor.exe')
profile = FirefoxProfile(r'C:\Users\AtechM_03\Desktop\Tor Browser\Browser\TorBrowser\Data\Browser\profile.default')
profile.set_preference('network.proxy.type', 1)
profile.set_preference('network.proxy.socks', '127.0.0.1')
profile.set_preference('network.proxy.socks_port', 9050)
profile.set_preference("network.proxy.socks_remote_dns", False)
profile.update_preferences()
driver = webdriver.Firefox(firefox_profile= profile, executable_path=r'C:\Utility\BrowserDrivers\geckodriver.exe')
driver.get("http://check.torproject.org")
I am not able to Fetch Network Logs using Browsermob Proxy when set Http proxy , it will just create a har file but i am not able to see any logs inside the file
Below is my Code :-
String strFilePath = "./PerformanceLogs/PerformanceLogs.har";
String noProxy = "localhost, 127.0.0.1";
LegacyProxyServer server = new BrowserMobProxyServer();
server.start();
server.setCaptureHeaders(true);
server.setCaptureContent(true);
Proxy proxy = server.seleniumProxy().setHttpProxy(PROXY)
.setFtpProxy(PROXY)
.setSslProxy(PROXY)
.setNoProxy(noProxy);
DesiredCapabilities capabilities = DesiredCapabilities.firefox();
capabilities.setCapability(CapabilityType.PROXY, proxy);
WebDriver driver = new FirefoxDriver(capabilities);
server.newHar("NetworkLogs");
driver.navigate().to("https://url-");
driver.findElements(xpath="").click();
pMonitor = new Prism_Selenium_Monitor_Tab(driver);
pMonitor.prism_monitor_credentialDeploy(URL,excelfile[0], excelfile[4]);
Har har = server.getHar();
File harFile = new File(strFilePath);
har.writeTo(harFile);
server.stop();
can anyone please me in this?
Thanks in advance
For some reason you're starting LegacyProxy, try starting BrowserMobProxyServer, i.e.
BrowserMobProxyServer server = new BrowserMobProxyServer();
Default proxy type is HTTP, i.e. try to delete this part:
.setNoProxy(noProxy);
It's actually shouldn't be a problem, but in default case there is no need to name your HAR, i.e. this code is redundant:
server.newHar("NetworkLogs");
you may just:
server.newHar();
Using Selenium and the web driver to drive phantomjs as part of a web scraping project. From the "Known Issues" for Windows at PhantomJS, it is suggested that the proxy-type be set to "none" to speed up network performance.
I tried the following:
PhantomJSOptions options = new PhantomJSOptions();
options.AddAdditionalCapability("proxy", "{proxyType:none}");
However, this sets proxy to a string and I think I need a json object. Can someone show me how to do this correctly?
Use the driver service to set options:
var phantomJSDriverService = PhantomJSDriverService.CreateDefaultService(phantomJsDir);
phantomJSDriverService.ProxyType = "none";
var driver = new PhantomJSDriver(phantomJSDriverService);
I'm using Selenium Webdriver (Chromedriver) in Java, along with BrowserMob Proxy Server to capture HTTP traffic in a har file. I recently encountered a problem where sections of the website would not load, and I've narrowed it down to this error:
"Failed to load resource https://ajax.googleapis.com/ajax/libs/jquery/1.8.2/jquery.min.js".
It seems that when using the proxy server, the Selenium driver can't access the SSL certificate for the https link. Here is a snippet of the code I am using:
ProxyServer server = new ProxyServer(4040);
server.start();
Proxy proxy = server.seleniumProxy();
DesiredCapabilities capabilities = new DesiredCapabilities();
capabilities.setCapability(CapabilityType.PROXY, proxy);
driver = new ChromeDriver(capabilities);
server.newHar("myHar");
Har har = server.getHar();
server.stop();
I have tried adding "capabilities.setCapability(CapabilityType.ACCEPT_SSL_CERTS, true);" and this solves the above problem, but only if I delete "capabilities.setCapability(CapabilityType.PROXY, proxy);" and then I am no longer able to capture the har file.
I tried switching to a firefox driver and setting up a firefox profile, but either I am not doing it properly or it won't work for my purposes either.
I have also tried setting up the cybervillainsCA certificate that comes with BrowserMob proxy in Trusted Root Certifaction Authorities, but that hasn't solved the issue either.
Does anyone know how to solve this in a way that I can collect HTTP traffic as well as successfully load the resource mentioned above?
What I ended up doing was scrapping the proxy entirely, and instead switched to using a Firefox profile with the Firebug and NetExport extensions in order to export the har file. This ended up significantly slowing down the tests, and exports a har for each page visited rather than for the entire session.
Code:
//Firefox profile
FirefoxProfile ffProfile = new FirefoxProfile();
ffProfile.addExtension(new File("firebug-1.11.4-fx.xpi"));
ffProfile.addExtension(new File("netExport-0.9b3.xpi"));
//Set default Firefox preferences
ffProfile.setPreference("app.update.enabled", false);
String domain = "extensions.firebug.";
//Set default Firebug preferences
ffProfile.setPreference(domain + "currentVersion", "1.11.4");
ffProfile.setPreference(domain + "allPagesActivation", "on");
ffProfile.setPreference(domain + "defaultPanelName", "net");
ffProfile.setPreference(domain + "net.enableSites", true);
//Set default NetExport preferences
ffProfile.setPreference(domain + "netexport.alwaysEnableAutoExport", true);
ffProfile.setPreference(domain + "netexport.autoExportToFile", true);
ffProfile.setPreference(domain + "netexport.showPreview", false);
ffProfile.setPreference(domain + "netexport.defaultLogDir", "string file path");
//WebDriver, instantiated outside the method
driver = new FirefoxDriver(ffProfile);
s = new WebDriverBackedSelenium(driver, "http://www.google.ca/");
I collected the har file after each page as follows:
HarFileReader r = new HarFileReader();
HarFileWriter w = new HarFileWriter();
int count = 1;
String allHars = "";
String harFolderPath = "file path for har";
File dir = new File(harFolderPath);
for (File child : dir.listFiles()) {
HarLog log = r.readHarFile(child);
File f = new File(harFolderPath + "\\test"+count+".txt");
w.writeHarFile(log, f);
allHars = allHars + readFileAsString(f.getPath());
count++;
}
FileUtils.cleanDirectory(dir);