I use the following code to run Tor controlled by Selenium in Lubuntu:
from selenium import webdriver
from selenium.webdriver.firefox.firefox_profile import FirefoxProfile
torexe = os.popen(r'/home/sergey/.local/share/torbrowser/tbb/x86_64/tor-browser_en-US/Browser/firefox')
profile = FirefoxProfile(r"/home/sergey/.local/share/torbrowser/tbb/x86_64/tor-browser_en-US/Browser/TorBrowser/Data/Browser/profile.default")
profile.set_preference('network.proxy.type', 1)
profile.set_preference('network.proxy.socks', '127.0.0.1')
profile.set_preference('network.proxy.socks_port', 9050)
profile.set_preference("network.proxy.socks_remote_dns", False)
profile.update_preferences()
driver = webdriver.Firefox(firefox_profile=profile, executable_path=r'/usr/local/bin/geckodriver')
driver.get("https://www.google.com")
...
It works, but in somewhat strange manner. Running it results in opening two windows, one window is Tor's while the other one being that of Firefox. Selenium controls the FF window only. The Tor window just sits there idly.
It is not to say that makes life unbearable, as I have said every thing works, but I am merely curious to know how to «make it completely right» (by this, I mean executing Selenium script in the only window of Tor).
I figured it out. In order to launch Tor using Selenium please run the following code (it still launches Firefox, this time without any Tor windows, but it uses the Tor service):
from selenium import webdriver
profile = webdriver.FirefoxProfile()
profile.set_preference('network.proxy.type', 1)
profile.set_preference('network.proxy.socks', '127.0.0.1')
profile.set_preference('network.proxy.socks_port', 9050)
profile.set_preference("network.proxy.socks_remote_dns", False)
profile.update_preferences()
driver = webdriver.Firefox(profile)
You may want to add the following three lines to verify the Tor service provides new IP for you Firefox instance:
driver.get("http://icanhazip.com")
current_IP = driver.find_element_by_css_selector("body > pre:nth-child(1)")
print(current_IP.get_attribute("innerHTML"))
This site icanhazip.com allows one to see his external IP without necessity to pass the abominable captcha test.
Related
I have this code:
async def acess_all():
def acess_localhost():
option = Options()
option.add_argument("--no-sandbox")
option.debugger_Address="127.0.0.1:8081"
driver = webdriver.Chrome(options=option)
driver.get("http://127.0.0.1:8081")
wait = WebDriverWait(driver, 5)
wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, 'a.item'))).click()
try:
t = threading.Thread(target=get_infos)
t.start()
os.system("chromium --remote-debugging-port=8081 https://google.com")
except:
print("Error!")
What I need:
Guys, as you can see, this code opens the Chrome browser on the Google page, and my Selenium code opens the browser on localhost because it's accessing where the remote-debbuging address points to, but I can't access the console tab of devtools, I need to access this tab using Selenium and run a javascript command to copy the cookie in json format, but I can't, how can I access the devtools console tab?
I believe I've figured out a way to run a javascript command in the console and get a return value (using selenium, as in the question).
Note: I'm running this on a Windows computer, but the idea should remain the same on a different operating system. You might have to tweak a few things in this code. Also, all Chrome sessions have to be closed beforehand to get this to work.
Ok, unless I interpreted your question wrong (tell me if I did), this is the basic order of the things you want you want to happen when:
Open the regular chrome browser (by regular, I mean not selenium, but regular chrome.exe or Google Chrome.app) to google.com and set the debugging port (which I assume is what you're doing when you run this command: chromium --remote-debugging-port=8081 https://google.com)
Open Chromedriver (Selenium) and go to the locally-hosted debugger window at 127.0.0.1:8081
Select the "Google" window option in the list of available windows at 127.0.0.1:8081
Once the devtools window opens, move to the Console tab
Finally, run some Javascript in the Console's input box and somehow get a return value for a cookie
You already did the first few items (1-3) in your code but you needed help figuring out the rest. I think I've found a way to do it.
So assuming that you already opened the google.com window on the Chrome browser and the 127.0.0.1:8081 window on localhost, all you need to do now is access the Console.
Here's what my chromedriver (selenium) browser screen looks like at this point, just for reference.
We'll start by waiting until a specific element (devtools-elements-breadcrumbs) has loaded on the page. We wait for this so we are sure that the page is loaded enough to navigate. I found this element by looking at the driver page_source. Here's how we wait for it:
wait.until(EC.presence_of_element_located((By.TAG_NAME, "devtools-elements-breadcrumbs")))
Once the breadcrumb element is located we can move to the console window by sending a shortcut to the chromedriver browser telling it to move right one tab in the Inspector window (to Console). The command is Control + ] (Windows/Linux) or Command + ] (Mac), to go to the next panel. This is how we send that shortcut to the selenium window (once again, using Windows):
click_console = ActionChains(driver)
click_console.key_down(Keys.CONTROL).send_keys(']').key_up(Keys.CONTROL).perform()
or on a Mac,
click_console = ActionChains(driver)
click_console.key_down(Keys.COMMAND).send_keys(']').key_up(Keys.COMMAND).perform()
After moving to the Console, we now have to wait for the console area to load, so we wait for all the CodeMirror class elements (once again, found using driver.page_source)
# wait for console area to open
wait.until(EC.presence_of_all_elements_located((By.CLASS_NAME, "CodeMirror")))
Now the last thing to do is to send the javascript command to the console. In this example, I'm searching for the APISID cookie.
cookie_name = "SEARCH_SAMESITE"
# javascript find the cookie
# https://stackoverflow.com/a/59603055/11073064
js_command = '(\"; \" + document.cookie).split(\"; {}\").pop().split(\';\').shift();'.format(cookie_name)
# send the command to the selenium webbrowser
send_js_command = ActionChains(driver)
send_js_command.send_keys(js_command, Keys.ENTER).perform()
and to get the value outputted in the console after you run that command:
# wait for return value span tag to be found
element = wait.until(EC.presence_of_element_located((By.CSS_SELECTOR, 'span.object-value-string.source-code')))
value = element.text
driver.close()
Here's the full code I used (on Windows).
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
import threading
from selenium.webdriver.chrome.options import Options
from selenium.webdriver import ActionChains
from selenium.webdriver.common.keys import Keys
import os
def start_chrome_browser_with_debug():
# this is the path to your regular google chrome browser, not selenium chromedriver
# note the chrome.exe, not chromedriver.exe.
# point the path to your regular chrome browser
os.system('"C:\Program Files (x86)\Google\Chrome\Application\chrome.exe" --remote-debugging-port=8081 https://google.com')
cookie_name = "SEARCH_SAMESITE"
js_command = '(\"; \" + document.cookie).split(\"; {}\").pop().split(\';\').shift();'.format(cookie_name)
chrome_browser = threading.Thread(target=start_chrome_browser_with_debug)
chrome_browser.start()
option = Options()
option.add_argument("--no-sandbox")
option.debugger_Address = "127.0.0.1:8081"
driver = webdriver.Chrome(options=option)
driver.get("http://127.0.0.1:8081")
wait = WebDriverWait(driver, 5)
wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, 'a.item'))).click()
wait.until(EC.presence_of_element_located((By.TAG_NAME, "devtools-elements-breadcrumbs")))
click_console = ActionChains(driver)
click_console.key_down(Keys.CONTROL).send_keys(']').key_up(Keys.CONTROL).perform()
wait.until(EC.presence_of_all_elements_located((By.CLASS_NAME, "CodeMirror")))
send_js_command = ActionChains(driver)
send_js_command.send_keys(js_command, Keys.ENTER).perform()
element = wait.until(EC.presence_of_element_located((By.CSS_SELECTOR, 'span.object-value-string.source-code')))
value = element.text
driver.close()
os.system('taskkill /F /IM chrome.exe /T')
print(value)
Actually, you can execute the javascript command by selenium itself :
driver.execute_script("some javascript code here");
You can't use Selenium to interact with the dev tools console, however you don't need it to get the cookies, just use get_cookies() from the webdriver and convert it to json using json.dumps()
import json
cookies = driver.get_cookies()
js = json.dumps(cookies)
I couldn't make this work on a Mac using only Python and Selenium. However, I found a solution that works if I use pyautogui in addition.
I'm working on a file, duals7.svg, saved locally. I want to be able to open it, play with it and watch what happens at the Javascript developer console in Chrome.
The script below does exactly what I want:
# chrome.py
# https://www.browserstack.com/guide/python-selenium-to-run-web-automation-test
# https://www.tutorialspoint.com/how-do-i-pass-options-to-the-selenium-chrome-driver-using-python
# https://stackoverflow.com/questions/36311191/how-to-open-chrome-developer-console-in-selenium-webdriver-using-java
# https://towardsdatascience.com/controlling-the-web-with-python-6fceb22c5f08
from selenium import webdriver
from selenium.webdriver.chrome.options import Options as ChromeOptions
import pyautogui
op = webdriver.ChromeOptions ()
op.add_argument ('--allow-file-access-from-files')
op.add_argument ('--auto-open-devtools-for-tabs')
driver = webdriver.Chrome (options=op)
driver.get ('file:///Users/murray/projects/warp/dual7.svg')
console = (850, 175)
pyautogui.moveTo (console [0], console [1], duration = 0.5)
pyautogui.click (console [0], console [1])
(I don't think the pyautogui.moveTo command is needed. Including it allows me to watch the mouse move, which I like.)
When I run it (python chrome.py), I see my svg "app" in Chrome with the Javascript dev console open:
The coordinates of the console button were determined by trial and error. Please forgive the aesthetics of the app: it'll look nice when it's done.
When I interact with my "app" I see my console.log () debugging messages.
If you're interested, the app is being modified from
/* elliptic-mesh.js
*
* generates a structured mesh bounded by four splines by
* solving the elliptic Laplace equation
*
* For more info see:
* http://www.particleincell.com/2012/online-meshing-example/
* http://www.particleincell.com/2012/bezier-splines/
*
* Lubos Brieda, Particle In Cell Consulting LLC, 2012
* you may freely use this algorithm in your codes but whenever possible
* please include a link/reference to the source article
*/
My version generalizes Dr. Brieda's code a bit, adds some debugging statements, and will add save and load options.
In Chrome, shift-escape gives you Chrome's task manager.
In the task manager, you can see various stats, including the "memory footprint" of a tab.
I'd like to get that value in Selenium. Is there a way to do so?
Apparently there is a chrome.processes API which looks like it could be useful, if it could be accessed from Selenium.
You can execute arbitrary JavaScript code from Selenium therefore you can access i.e. Window.Performance object in general and Windows.Performance.Memory object in particular
Example code:
from selenium import webdriver
driver = webdriver.Chrome()
driver.get('https://accounts.seetest.io/signup')
print(driver.execute_script("return window.performance.memory"))
driver.quit()
Example output:
{'jsHeapSizeLimit': 2197815296, 'totalJSHeapSize': 23449360, 'usedJSHeapSize': 14905688}
Is there any way to switch from headless to headed in the middle of a task in Selenium?
For example, I'm running some workflow headless, but in the middle I want the user to be able to push a button to see what the actual browser window looks like. How do I do that?
Not possible.
But it is possible to hide browser window from the screen :) :
driver.Manage().Window.Position =new Point(-2000, 0);
Unfortunately, switching between UI (headed) and HeadLess execution through selenium is not possible.
For example, Selenium creates an instance of browser of type WebDriver (either Firefox/Chrome/IE/HtmlUnit). Selenium can operate only on the instance it has created, and during creation time it will be defined if it is a UI test (Firefox/Chrome/IE) or Headless (HtmlUnit).
So, the design of Selenium prevents to switch between headless/UI execution in the middle of the test, as per your requirement.
Using xvfb you can do this. You can develop your tests using any driver, so, after configured the xvfb, you just need to use
xvfb-run tests
to run headless and if you don't want, you can run the tests without xvfb parameter.
A linux machine is required to use xvfb.
I've done this for one of my daily tasks - signin into firewall. It was a 2FA process, which required combo of browser interaction and a (random generated) token. For the first part a simple powershell script was running in headless mode, so when the site prompts for your token, it turns IE to visible and waits for user input. If such is provided, again switches to headless and finishes the process. The user is informed for success/fail of the operation with dialog message box.
You can start with such simple IE automation and optimize your .ps1 script later. The solution I was using back in the days:
$ie = new-object -ComObject "InternetExplorer.Application"
$requestUri = http://-your-site-here
# switch the browser headless mode on/off
#$ie.visible = $true
$ie.silent = $true
$ie.navigate($requestUri)
while($ie.Busy) { Start-Sleep -Milliseconds 100 }
# get DOM in order to interact
$doc = $ie.Document
$userInputId = "your-uname-id-here"
$doc.getElementsByTagName("input") | % {
if ($_.id -ne $null){
if ($_.id.Contains($userInputId)) { $_.value = "your-username-here" }
}
}
# can call JS events on elements
# like $btn.click()
Write-Verbose "Login Complete"
the popup window is only happening if I use the Fire Fox browser otherwise, is there a way to fix this problem? I have to enter userid/password every time the i use FF as my browser.
currently, I am entering every time i run my test which is very painful but looking to make it more automated....
I have goggled and found two links here and here but no avail
http://username:password#xyz.com
This worked for me (xyz.com being the site name)
After spending hours reading I finally found the solution which works pretty well and I hope this will help others too. - Enjoy!!
First - follow this steps:
1) Open the FireFox browser
2) Type the following `about:config`
3) Look for `network.http.phishy-userpass-length` if you don't find then create a new Integer key
Create a new Integer key (right-click->New->Integer): `network.http.phishy-userpass-length` with value `255`
Second: You need to create a Firefox driver with the following:
FirefoxProfile profile = new FirefoxProfile();
profile.SetPreference("network.http.phishy-userpass-length", 255);
profile.SetPreference("network.automatic-ntlm-auth.trusted-uris", "YOUR HOST ADDRESS HERE");
_driver = new FirefoxDriver(profile);
let me know if you have any questions.
If this is a windows user account & password, then you need to enable the integrated windows login by setting
network.negotiate-auth.delegation-uris: MyIISServer.domain.com
network.automatic-ntlm-auth.trusted-uris: MyIISServer.domain.com
network.automatic-ntlm-auth.allow-proxies: True
network.negotiate-auth.allow-proxies: True
in the Firefox profile that WebDriver starts. Once you have the profile created and saved (run "Firefox -P" when no other instances are running to select a profile), you can do this in the code:
File profileDir = new File("C:/wherever/SeleniumFirefoxProfile");
FirefoxProfile profile = new FirefoxProfile(profileDir);
profile.setEnableNativeEvents(true);
driver = new FirefoxDriver(profile);
I have had to handle these a few times, but my approach is using a script outside Selenium. Are you working on Windows?
Basically what you do is this:
1) Prior to loading the page, clicking the URL, etc that causes that dialog to appear:
-- Launch an asynchronous script to handle the login
2) Then load the page, click the link, etc
-- Selenium will block until your asynch script completes
The async script:
-- Sleep for a few seconds
-- Activate the dialog
-- Send the username
-- Send a TAB
-- Send the password
-- Send a TAB
-- Send the Enter Key
If you are working on windows, I can post sample scripts to handle this. I've done it with Java and C#, but I would guess that basically the same thing would work regardless of how you are writing your tests (unless you are strictly using the FF plugin, in which case this won't work).
Let me know if you'd like more details.
You can use a FF plugin "autoauth". Download this plugin and create Firefox instance by the following way:
FirefoxProfile firefoxProfile = new ProfilesIni().getProfile("default");
File pluginAutoAuth = new File("D:\\autoauth-2.1-fx+fn.xpi");
firefoxProfile.addExtension(pluginAutoAuth);
driver = new FirefoxDriver(firefoxProfile);
I used "autoauth-2.1-fx+fn.xpi"
How could I start a Selenium browser (like Firefox) minimized? I want the command browser.start(mySettings) to start a browser minimized.
I have an alternate solution that may meet your needs. You can set the position of the WebDriver to be outside of your view. That way, it'll be out of sight while it runs. (It may start at a visible location, but it will only be there for an instant.)
FirefoxDriver driver = new FirefoxDriver();
driver.manage().window().setPosition(new Point(-2000, 0));
Your question does not say that why you want to run your test cases in minimized browser but unfortunately selenium do not provide any built-in function for the same.
Normally when we want to run test cases with maximized browser we use driver.manage().window().maximize();
No doubt there are several ways to minimize your window through code by using Java key event by using keyboard shortcuts for minimimzing window or by using JavaScriptExecuter but that too depend on which OS and language you are working.
One more thing you can try is HtmlUnitDriver.By using this you cant even see the browser, so that may also serve your purpose if you have a case of not opening the browser while execution of test cases.
Dimension windowMinSize = new Dimension(100,100);
driver.manage().window().setSize(windowMinSize);
For C# you can minimize window easily, also with a built in way.
See: https://www.selenium.dev/documentation/en/webdriver/browser_manipulation
Screenshot:
Without knowing your motive for minimizing the browser and assuming that you are using the WebDriver drivers (Selenium v2) and don't want a UI to pop up, one could use the lightweight browser HtmlUnitDriver.
After you define the driver
driver = webdriver.Chrome(r"FULL PATH TO YOUR CHROMEDRIVER")
Do
driver.minimize_window()
And then call the site
driver.get(r'SITE YOU WANT TO SELECT')
It will minimize.
When Using Python to Move the FireFox Browser off screen:
driver = webdriver.FireFox()
driver.set_window_position(-2000,0)
driver.set_window_position(2000,2000)
or
(x,y) values more than monitor resolution
Later if u want to see the window again
driver.maximize_window()
The workarounds mentioned in the post did not work for NodeWebKit browser, so as a workaround i had to use native C# code as mentioned below:
public static void MinimiseNWKBrowser(IWebDriver d)
{
var body = UICommon.GetElement(By.TagName("body"), d);
body.Click();
string alt = "%";
string space = " ";
string down = "{DOWN}";
string enter = "{ENTER}";
SendKeys.SendWait(alt + space);
for(var i = 1; i <= 5; i++)
{
SendKeys.SendWait(down);
}
SendKeys.SendWait(enter);
}
So this workaround basically uses "ALT+SPACE" to bring up the browser action menu to select "MINIMIZE" from the options and presses "ENTER"
In php we can use JavaScript command to minimize the browser window.
$this->selenium->getEval("Minimize();");
and similar command for java :
browser.getEval("Minimize();");
The best way to minimize your browser is to use shortcut using Robot class.
Shortcut: Alt+Space+N (for Windows)
Robot robot=new Robot();
robot.keyPress(KeyEvent.VK_ALT);
robot.keyPress(KeyEvent.VK_SPACE);
robot.keyPress(KeyEvent.VK_N);
robot.keyRelease(KeyEvent.VK_ALT);
robot.keyRelease(KeyEvent.VK_SPACE);
robot.keyRelease(KeyEvent.VK_N);
By using above code you can minimize your browser.
Selenium doesn't have a built-in method to minimize the Browsing Context. Minimizing the browser while the Test Execution is In Progress would be against the best practices as Selenium may loose the focus over the Browsing Context and an exception may raise during the Test Execution. You can find a relevant detailed discussion in How to execute tests with selenium webdriver while browser is minimized
However, to mimic the functionality of minimizing the Browsing Context you can use the following steps:
Set the dimension of the Browsing Context to [0, 0]
Set the position of the top left corner of the Browsing Context just below the Viewport
Code Block (Java):
driver.navigate().to("https://www.google.com/");
Point p = driver.manage().window().getPosition();
Dimension d = driver.manage().window().getSize();
driver.manage().window().setSize(new Dimension(0,0));
driver.manage().window().setPosition(new Point((d.getHeight()-p.getX()), (d.getWidth()-p.getY())));
You can use:
driver.manage().window().maximize();
For example code snippet with chrome driver:
System.setProperty("webdriver.chrome.driver", "C://chromedriver.exe");
driver = new ChromeDriver();
baseUrl = "chrome://newtab/";
driver.manage().window().maximize().timeouts().implicitlyWait(30, TimeUnit.SECONDS);
for Firefox use just "firefordriver.exe"
driver.manage().window().minimize();
This should help minimize the window. You can also use "maximize" in place of "minimize" to maximize the window.