Python Selenium with Tor Browser (Ubuntu) - selenium

I try to open a Tor Browser using Selenium on Ubuntu 18. I have tried lots of examples but with no success.
proxyIP = "127.0.0.1"
proxyPort = "9050"
profileTor = '/etc/tor/' # torrc
binary = os.path.expanduser("~/.local/share/torbrowser/tbb/x86_64/tor-browser_en-US/Browser/firefox")
firefox_binary = FirefoxBinary(binary)
firefox_profile = FirefoxProfile(profileTor)
proxy_address = "127.0.0.1:9050"
proxy = Proxy({
'proxyType': ProxyType.MANUAL,
'httpProxy': proxy_address,
})
driver = webdriver.Firefox(firefox_binary = firefox_binary,firefox_profile=firefox_profile, proxy = proxy)
A blank Tor Browser window opens but after a while I get an error as:
selenium.common.exceptions.WebDriverException: Message: connection refused.
I have also tried an alternative to firefox binary the:
start-tor-browser
which opens a working Tor Browser and showing some index.
The script however stops and I cannot visit another page using Selenium unless I do it manually.
I have also tried the:
profile.default
as some examples suggest but I get an error:
Unable to start Tor. The torrc file is missing and could not be created.

To open a Tor Browser using Selenium you can start the Tor daemon first and then open the Tor Browser and you can use the following solution:
Sample WindowsOS style Code Block:
from selenium import webdriver
from selenium.webdriver.firefox.firefox_profile import FirefoxProfile
import os
torexe = os.popen(r'C:\Users\AtechM_03\Desktop\Tor Browser\Browser\TorBrowser\Tor\tor.exe')
profile = FirefoxProfile(r'C:\Users\AtechM_03\Desktop\Tor Browser\Browser\TorBrowser\Data\Browser\profile.default')
profile.set_preference('network.proxy.type', 1)
profile.set_preference('network.proxy.socks', '127.0.0.1')
profile.set_preference('network.proxy.socks_port', 9050)
profile.set_preference("network.proxy.socks_remote_dns", False)
profile.update_preferences()
driver = webdriver.Firefox(firefox_profile= profile, executable_path=r'C:\Utility\BrowserDrivers\geckodriver.exe')
driver.get("http://check.torproject.org")

Related

Selenium starts the browser but shows an error as Message: Can not connect to the Service

The code starts the browser, stops at this step (line 5), and after a while throws an error:
selenium.common.exceptions.WebDriverException: Message: Can not connect to the Service C:\Program Files\Mozilla Firefox\firefox.exe
from selenium import webdriver
from selenium.webdriver.firefox.service import Service
s = Service(r'C:\Program Files\Mozilla Firefox\firefox.exe')
driver = webdriver.Firefox(service=s)
driver.get('http://www.google.com')
myPageTitle = driver.title
print(myPageTitle)
driver.quit()
Firefox - 95.0.2
Selenium - 4.1.0
I tried with chrome, same problem
Does anyone know what the problem is and how to solve it?
As an argument to Service() instead of the firefox executable, you need to pass the absolute location of the GeckoDriver executable which can be downloaded from mozilla/geckodriver page.
So your effective code block will be:
from selenium import webdriver
from selenium.webdriver.firefox.service import Service
s = Service(r'C:\path\to\geckodriver.exe')
driver = webdriver.Firefox(service=s)
driver.get('http://www.google.com')
myPageTitle = driver.title
print(myPageTitle)
driver.quit()

selenium timeout on Heroku

I tried to build a flask+selenium web on Heroku , so I went through a lot of step , I got my website setup , but when I tried to call an API that include selenium , I got an error from Selenium:
raise TimeoutException(message, screen, stacktrace)
selenium.common.exceptions.TimeoutException: Message:
Here's the part of my selenium python code , I tried to login to instagram.
chrome_options = webdriver.ChromeOptions()
chrome_options.binary_location = os.environ.get("GOOGLE_CHROME_BIN")
chrome_options.add_argument("--headless")
chrome_options.add_argument('window-size=1400,800')
chrome_options.add_argument("--disable-dev-shm-usage")
chrome_options.add_argument("--no-sandbox")
browser = webdriver.Chrome(executable_path=os.environ.get("CHROMEDRIVER_PATH"), chrome_options=chrome_options)
ig_username = os.environ.get("IG_USERNAME")
ig_password = os.environ.get("IG_PASSWORD")
web = 'https://www.instagram.com';
browser.get(web)
print ("Headless Chrome with custom window-size")
size = browser.get_window_size()
print("Window size: width = {}px, height = {}px".format(size["width"], size["height"]))
try:
wait = WebDriverWait(browser,10)
ig_account_ele = wait.until(EC.element_to_be_clickable((By.XPATH, //*[#id="loginForm"]/div/div[1]/div/label/input)))
print(ig_account_ele)
ig_account_ele.send_keys(ig_username)
The code will return timeout error at first element that I am trying to search.I even tried it on google.com and other element like span, it still won't work.
It's all fine when I run the code on local!!! ,but when it's deployed to Heroku, it will return timeout error or unable to locate element when I test the code with find_element_by_id,
so I assume there must be something wrong with selenium , but I don't have any way to solve it, seems it is also hard to see what's going on with chromedirver when running headless mode on Heroku...

Selenium Google login works locally but not in CircleCI

I'd like to run an end-to-end test of logging into our website using Selenium. We use Auth0 and the only available login mechanism is through Google social login. I wrote a script using Python Selenium (version 3.141.0), pytest, and selenium/standalone-chrome:87.0 Docker image which works correctly on my local machine, Mac OS 10.15.4.
However, it gets stuck at some point when I try to run it on CircleCI.
I use ubuntu-1604:202007-01 image in CircleCI
How I set up remote driver (tried a lot of arguments/commands..):
#pytest.fixture(scope="function")
def browser(remote_webdriver_url):
options = webdriver.ChromeOptions()
options.add_argument('--disable-popup-blocking')
options.add_argument('--disable-web-security')
options.add_argument('--allow-running-insecure-content')
options.add_argument('--start-maximized')
options.add_argument('-incognito')
options.add_experimental_option("useAutomationExtension", False)
options.add_experimental_option("excludeSwitches", ["enable-automation"])
browser = webdriver.Remote(
command_executor=remote_webdriver_url,
desired_capabilities=DesiredCapabilities.CHROME,
options=options)
return browser
My docker-compose.yml
version: '3.1'
services:
selenium-chrome:
image: selenium/standalone-chrome:87.0
# added the envvar as I found something about this in Selenium forums, it has no effect.
environment:
DBUS_SESSION_BUS_ADDRESS: /dev/null
shm_size: 2g
restart: 'no'
ports:
- "4444:4444"
My test code:
import os
from selenium.webdriver.common.by import By
from selenium.webdriver.remote.webelement import WebElement
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
def test_logging_in(browser: WebElement, url):
auth0_title = "Sign In with Auth0"
browser.get(url)
assert browser.title == auth0_title
# Log in
auth_login_button_class_name = 'auth0-lock-social-button-text'
_ = WebDriverWait(browser, 20).until(EC.presence_of_element_located((By.CLASS_NAME, auth_login_button_class_name)))
sign_in_button = browser.find_element_by_class_name(auth_login_button_class_name)
browser.implicitly_wait(5)
#
sign_in_button.click()
# Wait until we're redirected to Google's login page
_ = WebDriverWait(browser,20).until(EC.title_contains('Google'))
# Type in the email address and go to the next page
email_input = browser.find_element_by_tag_name('input')
email_input.send_keys(os.environ.get('E2E_TEST_DEVELOPMENT_USER_EMAIL'))
first_next_button = browser.find_element_by_id("identifierNext")
first_next_button.click()
# Wait until the password entry screen is loaded
browser.get_screenshot_as_file('/tmp/scr.png')
_ = WebDriverWait(browser, 20).until(EC.presence_of_element_located((By.ID, "forgotPassword"))) ##### THIS IS WHERE I GET THE TIMEOUT
# Put the password in
password_input = browser.find_element_by_xpath("//input[#name='password']")
password_input.send_keys(os.environ.get('E2E_TEST_DEVELOPMENT_USER_PASSWORD'))
second_next_button = browser.find_element_by_id("passwordNext")
second_next_button.click()
# Wait until the login is successful by observing the logout button
logout_icon_class_name = "bp3-icon-log-out"
_ = WebDriverWait(browser, 20).until(EC.presence_of_element_located((By.CLASS_NAME, logout_icon_class_name)))
assert browser.title == 'My page title'
sign_out_button = browser.find_element_by_class_name(logout_icon_class_name)
sign_out_button.click()
def test_teardown(browser):
browser.close()
browser.quit()
The test times out after clicking on the first button after typing in the email. I got screenshots from the run in CI, and it does seem to be stuck loading (see the Google's progress bar at the top, and the fact that it's more white-ish color), see the screenshot:
I also took a screenshot before clicking on the "Next" button, to show the contrast:
After having spent a long time on this and trying many things, I'm about to give up. Any ideas why this works locally but not in CI environment?

AttributeError: module 'selenium.webdriver' has no attribute 'switch_to_alert'

I am making a simple crawler that can open a site and when a pop up appears, it should close it. but the following command isn't working.
from selenium import webdriver
browser = webdriver.Chrome(executable_path=r"C:\Program Files\chromedriver.exe")
url = "https://www.bnbaccessories.com/"
browser.get(url)
alert = webdriver.switch_to_alert().dismiss()
innerHTML = browser.execute_script("return document.body.innerHTML")
browser.implicitly_wait(50)
browser.close()
Use this
alert = browser.switch_to.alert.dismiss()
instead
webdriver.switch_to_alert().dismiss()
driver instance name is browser not webdriver

Browsermob proxy server crashes websites

trying to use browsermob proxy server by falling code:
final int port = 9000;
server = new ProxyServer(port);
server.start();
final DesiredCapabilities dc = new DesiredCapabilities();
dc.setCapability(CapabilityType.PROXY, server.seleniumProxy());
setName("test");
FirefoxBinary binary = new FirefoxBinary(new File("C:\\Program Files (x86)\\ff21\\firefox.exe"));
File profileDir = new File("C:\\Users\\arno\\Documents\\profiles\\firefox21.default");
FirefoxProfile profile = new FirefoxProfile(profileDir);
driver = new FirefoxDriver(binary, profile, dc);//;
server.newHar("monitis");
but it crashes the website: see the capture .
I'm using the fallowing code to find out what is going on(getting the url and the http status )
Har har = server.getHar();
for(HarEntry entry : har.getLog().getEntries()){
System.out.println(entry.getRequest().getUrl() +": " + entry.getResponse().getStatus());
}
and it gives this result:
mysite/files/js/numeral.min.js: 200
mysite/js/94842541.js: 200
mysite/files/css/page-home.min.css?v=6: -999
mysite/files/css/ui.min.css?v=18: -999
mysite/files/js/ui.min.js?also=jquery.selectric.min.js,jquery.checkradios.min.js,index.min.js&v=53: -999
Try using the non-deprecated BrowserMobProxyServer class instead of the legacy ProxyServer implementation. Make sure you're using the latest version of BMP and the browsermob-core-littleproxy module as well.
it turned out, that in Firefox settings, in network settings tab, I need to set up the option "auto detect proxy server for this network"