Some js plugins can indeed do. How?
js_code = f"""
var prefs = Components.classes["#mozilla.org/preferences-service;1"].getService(Components.interfaces.nsIPrefBranch);
prefs.setIntPref("network.proxy.type", 1);
prefs.setCharPref("network.proxy.http", "${host}");
prefs.setIntPref("network.proxy.http_port", "${port}");
prefs.setCharPref("network.proxy.ssl", "${host}");
prefs.setIntPref("network.proxy.ssl_port", "${port}");
prefs.setCharPref("network.proxy.ftp", "${host}");
prefs.setIntPref("network.proxy.ftp_port", "${port}");
"""
It is not effict。
I have a more hardcore approach, provided you don't use Headless mode. And they don't care about the speed. You can load a fully fledged proxy plug-in before instantiating WebDriver. LIke SwitchyOmega, when SwitchyOmage is first started, you can choose to go to the second window, and this window is the configuration page of SwitchyOmage, and set its proxy. Then selenium gets the url of the configuration page, closes the tag, and manually launches the plug-in (because Selenium doesn't control it), but later when you want to change the proxy, you can use Selenium to execute JS to open a new tag and go to the URL of the configuration page. Change the proxy IP and port number using selenium control elements. Hahaha, very hardcore!
Related
I am working on selenium web driver. I want to open existing browser window via selenium script(Java).
There is no built-in method in Selenium that supports this feature directly. However, you can try the following:
driver.get("link");
driver.find_element_by_link_text('new link name').click().switch_to.window(driver.window_handles[0]);
Just switch to handle for self.You can also try storing handles in variables if you like.
The whole post can be found here.
Please refer to the attached screenshot, I need to save data from 'Save all as HAR' option automatically.
I doubt that selenium provides a way to interact with the inspector. You might have to do something like inject Javascript into the page that re-submits the request and then saves the response/request data on the page in a hidden element. You could then simply get the text from that hidden element via selenium.
Hope that helps
No, Selenium is not the tool you want to use to do this. JMeter may be useful, or this site suggests using a proxy server to do this for you.
Selenium does not give you the ability to track network traffic. It does however allow you to configure a proxy that the browser will use when it is started up.
S to get around your problem you will need to use something like the BrowserMobProxy:
BrowserMobProxy browserMobProxy = new BrowserMobProxyServer();
browserMobProxy.start();
Proxy proxy = ClientUtil.createSeleniumProxy(browserMobProxy, InetAddress.getLoopbackAddress());
DesiredCapabilities capabilities = DesiredCapabilities.firefox();
capabilities.setCapability(CapabilityType.PROXY, proxy);
WebDriver driver = new FirefoxDriver(capabilities);
You can then perform some actions with Selenium and when you want to collect a Har file perform:
browserMobProxy.getHar();
For more information about the BrowserMobProxy have a look at the BrowserMobProxy Homepage
There are many selenium webdriver binding package of Golang.
However, I don't want to control browser throught server.
How can I control browser with Golang and selenium without selenium server?
You can try github.com/fedesog/webdriver which says in its documentation:
This is a pure go library and doesn't require a running Selenium driver.
I would characterize the Selenium webdriver as a client rather than a server. Caveat: I have used the Selenium webdriver (Chrome version) from .Net and I am assuming it is similar for Go.
The way Selenium works is that you will launch an instance of it from within code, and it creates a live version of the selected browser (i.e. Chrome) and your program retains control over it. Then you write code to tell the browser to navigate to a page, inspect the response, and interact with the browser by filling out form data, clicking on buttons, etc. You can see what is happening on the browser as the code runs, so it is easy to troubleshoot when the interaction doesn't go as planned.
I have used Selenium to upload tens of thousands of records to a website that has no API and only a graphical user interface. Give it a chance.
I am using Selenium webdriver with firefox. I am wondering if there is a setting i can change such that it to only requesting resources from certain domains. (Specifically i want it only to request content which is on the same domain as the webpage itself).
My current set up, written in Python, is:
from selenium import webdriver
firefox_profile = webdriver.FirefoxProfile()
## Here, I change various default setting in Firefox, and install a couple of monitoring extensions
driver = webdriver.Firefox(firefox_profile)
driver.get(web_address)
What i want to do, is if i specify the web address wwww.domain.com, then to only load content served by domain.com, and not e.g. all the tracking content hosted by other domains that would typically be requested. Hoping could be achieved by a change to the profile settings in firefox, or via an extension.
Note - there is a similar question (without an answer) - Restricting Selenium/Webdriver/HtmlUnit to a certain domain - but it is four years old, and i think Selenium has evolved a lot since then.
With thanks to Vicky, (who's approach of using Proxy settings i followed - although directly from Selenium), the code below will change the proxy settings in firefox such that it will not connect to a domain except that on the white-list.
I suspect several setting changes are unnecessary and can be omitted for most purposes. Code in Python.
from selenium import webdriver
firefox_profile = webdriver.FirefoxProfile()
## replace desired_domain.com below with whitelisted domain. Separate domains by comma.
firefox_profile.set_preference("network.proxy.no_proxies_on","localhost,127.0.0.1,desired_domain.com")
firefox_profile.set_preference("network.proxy.backup.ftp","0.0.0.0")
firefox_profile.set_preference("network.proxy.backup.ftp_port",1)
firefox_profile.set_preference("network.proxy.backup.socks","0.0.0.0")
firefox_profile.set_preference("network.proxy.backup.socks_port",1)
firefox_profile.set_preference("network.proxy.backup.ssl","0.0.0.0")
firefox_profile.set_preference("network.proxy.backup.ssl_port",1)
firefox_profile.set_preference("network.proxy.ftp","0.0.0.0")
firefox_profile.set_preference("network.proxy.ftp_port",1)
firefox_profile.set_preference("network.proxy.http","0.0.0.0")
firefox_profile.set_preference("network.proxy.http_port",1)
firefox_profile.set_preference("network.proxy.socks","0.0.0.0")
firefox_profile.set_preference("network.proxy.socks_port",1)
firefox_profile.set_preference("network.proxy.ssl","0.0.0.0")
firefox_profile.set_preference("network.proxy.ssl_port",1)
firefox_profile.set_preference("network.proxy.type",1)
firefox_profile.set_preference("network.proxy.share_proxy_settings",True)
driver = webdriver.Firefox(firefox_profile)
driver.get(web_address_desired)
I think it is still impossible in selenium.But you can still achieve this by using proxies like browsermob. Webdriver integrates well with browsermob proxy.
Sample pseudeocode in java
//LittleProxy-powered 2.1.0 release
LegacyProxyServer server = new BrowserMobProxyServer();
server.start(0);
// Blacklist websites
server.blacklistRequests("https?://.*\\.blocksite\\.com/.*", 410);//these sites will be blocked
/// get the Selenium proxy object
Proxy proxy = ClientUtil.createSeleniumProxy(server);
// configure it as a desired capability
DesiredCapabilities capabilities = new DesiredCapabilities();
capabilities.setCapability(CapabilityType.PROXY, proxy);
// initialize the driver with the capabilities ;
Webdriver driver = new FirefoxDriver(capabilities);
Hope this helps you.Kindly get back if you need any further help
I need to build a Python scraper to scrape data from a website where content is only displayed after a user clicks a link bound with a Javascript onclick function, and the page is not reloaded. I've looked into Selenium in order to do this and played around with it a bit, and it seems Selenium opens a new Firefox web browser everytime I instantiate a driver:
>>> driver = webdriver.Firefox()
Is this open browser required, or is there a way to get rid of it? I'm asking because the scraper is potentially part of a web app, and I'm afraid if multiple users start using it, I will have a bunch of browser windows open on my server.
Yes, selenium automates web browsers.
You can add this at the bottom of your python code to make sure the browser is closed at the end:
driver.quit()