I am pretty new to using Selenium and it's webdrivers. I have a need to enable DoH (dns over https) together with an option for selecting which DoH server to connect to in chrome driver in Selenium.
I have been researching online and have gone through recommended switches available here: https://peter.sh/experiments/chromium-command-line-switches/
as well as seen a similar post here: How to disable dns over https in selenium for disabling DoH (I don't even have DoH enabled by default in first place in chromedriver), but haven't figured out yet to how to get it enabled in the headless mode.
I also looked at the switches available for firefox driver but still don't see any right away available switches to use for the same.
Any help would be appreciated.
Thanks!
fbw
To enable DoH you need to do the following:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
local_state = {
"dns_over_https.mode": "automatic",
"dns_over_https.templates": "",
}
options = Options()
options.add_experimental_option('localState', local_state)
driver = webdriver.Chrome(options=options)
This will turn on the DoH which looks like this in browser settings on the chrome://settings/security page:
Also you can set "dns_over_https.mode": "automatic" which will set the secure option of DoH configuration:
Unfortunately I failed to figure out ho to use "dns_over_https.templates": "". Documentation says about it:
String containing a space-separated list of DNS over HTTPS templates
to use in secure mode or automatic mode. If no templates are specified
in automatic mode, we will attempt discovery of DoH servers associated
with the configured insecure resolvers.
I'm not familiar with DoH, so this description tells me nothing. I don't know what a DoH template is. I hope you know what they are talking about.
Related
I have a Selenoid Firefox container running some tests but new browser versions are giving me issues.
Since Firefox 103.0 Cross-site tracking cookies are blocked by default and I am looking for a way to add a website exception in Privacy & Security > Cookies and Site Data using Firefox capabilities / Options but nothing seems to do the trick.
A good example of website that causes issues is https://teams.microsoft.com where firefox will now ask you if you "Really want to accept cookies from there ?" which I am trying to bypass by setting website exception in advance.
Done it by selecting a profile at geckodriver start instead of letting it generate one: Add args: ["-profile", "/binary/nameoffirefoxprofile"] to capabilities pref in your code. For this you need to get a firefox profile folder and copy it to your container first. If this works it will replace the generated rust_mozprofile_someID of geckodriver.
I've been trying to use Robot Framework to write some cross browser tests.
One of the requirements is that i need to use a proxy to access the website I am testing. Right now, I am trying to launch the safari browser and get it to go through the proxy to reach the website, but I seem to have an issue.
Here is the Robot framework keyword
# ${MY_PROXY} is a variable located elsewhere in the file
Open Safari
${desired_capabilities} = Evaluate selenium.webdriver.DesiredCapabilities.SAFARI
sys, selenium.webdriver
${safari_proxy} = Create Dictionary proxyType MANUAL httpProxy ${MY_PROXY}
sslProxy ${MY_PROXY}
Set To Dictionary ${desired_capabilities} proxy ${safari_proxy}
Create Webdriver Safari desired_capabilities=${desired_capabilities}
So far, i've been receiving this error
SessionNotCreatedException: Message: Capability 'proxy' could not be honored.
Currently using robotframework-seleniumlibrary 4.5.0 with selenium 3.141.0
Does the safari webdriver allow proxies? I can't seem to find much on this topic
I am using Selenium to open a web site, login and copy some information from one web site. However it is happening on my work station and have a display monitor.
my IT team wants to move this process to a virtual server which does not have a monitor.
1.Will this work - even if we install Chrome of Firefox on the server
2. Can we Chrome - headless to make this happen
3. Any other way - we can think of using Xserver
Please let me know.
No . To run your script you don't need to have monitor. You can access your virtual machine through remote connection and you can start the execution from that machine. Once the execution started, you can close the remote desktop session and execution will continue to run on remote machine or virtual server.
I hope this helps. Please let me know if you have any further questions.
1.Will this work - even if we install Chrome or Firefox on the server - Yes it will work
2.Can we Chrome - headless to make this happen - If you are going to use virtual server just for execution,then you don't need to run in headless mode. Headless execution is needed for environments where you don't need a visible UI shell. Below code will help you run your script in headless mode
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
options = Options()
options.add_argument('--headless')
options.add_argument('--disable-gpu') # Last I checked this was necessary.
driver = webdriver.Chrome("/usr/local/bin/chromedriver", chrome_options=options)
driver.get("https://google.com")
#code to extract the details
driver.quit()
3.Any other way - we can think of using Xserver - Not sure
Chrome headless should solve your problem here -- I've done this in the past with some of my automation and had success.
Just remember to use ChromeOptions to add the --headless flag. You may need to tweak some other ChromeOptions as well -- I also had to add --disable-gpu and --window-size=1920,1200 to get mine working just right.
I am running non http url inside of my tests (so migration to https wont be easy) and because of that I am getting an warning inside of browser.
How to disable "Non secure" warning inside of Chrome during selenium tests?
I've tried to play with arguments but nothing works
args: [
'start-maximized',
'disable-webgl',
'blacklist-webgl',
'blacklist-accelerated-compositing',
'disable-accelerated-2d-canvas',
'disable-accelerated-compositing',
'disable-accelerated-layers',
'disable-accelerated-plugins',
'disable-accelerated-video',
'disable-accelerated-video-decode',
'disable-gpu',
'disable-infobars',
'test-type',
'disable-extensions',
'allow-running-insecure-content',
'disable-web-security',
'ignore-certificate-errors',
'ignore-gpu-blacklist',
'no-default-browser-check',
'no-first-run',
'disable-default-apps'
]
The issue is that I need to resize windows to 420x800 but because of warning browser can't do that.
"Not Secure" SSL Error
As per Fix “Not Secure” SSL Error on Chrome Browser | Remove Warning with the release of Chrome 68, Google started showing all the HTTP sites as Not Secure on Chrome Browser.
Treatment of HTTP pages
This feature can be turned On / Off by accessing the page at chrome://flags/#enable-mark-http-as and setting the following attribute:
Mark non-secure origins as non-secure: Changes the UI treatment for HTTP pages on Mac, Windows, Linux, Chrome OS, Android
Default
Enabled
Enabled (mark as actively dangerous)
Enabled (mark with a Non Secure warning and dangerous on form edits)
Disabled
Using Selenium to disable this feature you need to use the ChromeOption --allow-running-insecure-content as follows:
Python:
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument("start-maximized")
chrome_options.add_argument('disable-infobars')
chrome_options.add_argument('--allow-running-insecure-content')
driver = webdriver.Chrome(chrome_options=chrome_options, executable_path=r'C:\Utility\BrowserDrivers\chromedriver.exe')
driver.get("http://www.legislation.vic.gov.au/")
This does not work for chrome on android devices. It's a bad idea for companies to tell users what and what they can not look at. Tech giants like Google have gone too far and the government is letting it happen.
I am using Selenium webdriver with firefox. I am wondering if there is a setting i can change such that it to only requesting resources from certain domains. (Specifically i want it only to request content which is on the same domain as the webpage itself).
My current set up, written in Python, is:
from selenium import webdriver
firefox_profile = webdriver.FirefoxProfile()
## Here, I change various default setting in Firefox, and install a couple of monitoring extensions
driver = webdriver.Firefox(firefox_profile)
driver.get(web_address)
What i want to do, is if i specify the web address wwww.domain.com, then to only load content served by domain.com, and not e.g. all the tracking content hosted by other domains that would typically be requested. Hoping could be achieved by a change to the profile settings in firefox, or via an extension.
Note - there is a similar question (without an answer) - Restricting Selenium/Webdriver/HtmlUnit to a certain domain - but it is four years old, and i think Selenium has evolved a lot since then.
With thanks to Vicky, (who's approach of using Proxy settings i followed - although directly from Selenium), the code below will change the proxy settings in firefox such that it will not connect to a domain except that on the white-list.
I suspect several setting changes are unnecessary and can be omitted for most purposes. Code in Python.
from selenium import webdriver
firefox_profile = webdriver.FirefoxProfile()
## replace desired_domain.com below with whitelisted domain. Separate domains by comma.
firefox_profile.set_preference("network.proxy.no_proxies_on","localhost,127.0.0.1,desired_domain.com")
firefox_profile.set_preference("network.proxy.backup.ftp","0.0.0.0")
firefox_profile.set_preference("network.proxy.backup.ftp_port",1)
firefox_profile.set_preference("network.proxy.backup.socks","0.0.0.0")
firefox_profile.set_preference("network.proxy.backup.socks_port",1)
firefox_profile.set_preference("network.proxy.backup.ssl","0.0.0.0")
firefox_profile.set_preference("network.proxy.backup.ssl_port",1)
firefox_profile.set_preference("network.proxy.ftp","0.0.0.0")
firefox_profile.set_preference("network.proxy.ftp_port",1)
firefox_profile.set_preference("network.proxy.http","0.0.0.0")
firefox_profile.set_preference("network.proxy.http_port",1)
firefox_profile.set_preference("network.proxy.socks","0.0.0.0")
firefox_profile.set_preference("network.proxy.socks_port",1)
firefox_profile.set_preference("network.proxy.ssl","0.0.0.0")
firefox_profile.set_preference("network.proxy.ssl_port",1)
firefox_profile.set_preference("network.proxy.type",1)
firefox_profile.set_preference("network.proxy.share_proxy_settings",True)
driver = webdriver.Firefox(firefox_profile)
driver.get(web_address_desired)
I think it is still impossible in selenium.But you can still achieve this by using proxies like browsermob. Webdriver integrates well with browsermob proxy.
Sample pseudeocode in java
//LittleProxy-powered 2.1.0 release
LegacyProxyServer server = new BrowserMobProxyServer();
server.start(0);
// Blacklist websites
server.blacklistRequests("https?://.*\\.blocksite\\.com/.*", 410);//these sites will be blocked
/// get the Selenium proxy object
Proxy proxy = ClientUtil.createSeleniumProxy(server);
// configure it as a desired capability
DesiredCapabilities capabilities = new DesiredCapabilities();
capabilities.setCapability(CapabilityType.PROXY, proxy);
// initialize the driver with the capabilities ;
Webdriver driver = new FirefoxDriver(capabilities);
Hope this helps you.Kindly get back if you need any further help