I have a page that takes some time to load (30-60 seconds), so I have setup an implicit wait, and would like to be able to refresh the page if it doesn't finish loading the first time.
My code does not end up refreshing the page in Chrome (the loading spinner doesn't start and the page doesn't change), nor does it print out the status of the Javascript call - I don't know why. I'm running Chrome 68, with chromedriver 2.42 and Python 3.6.
I've replaced the actual URL and timeout with sample values below to make it reproducible:
from selenium import webdriver
from selenium.common.exceptions import TimeoutException
import os
mypath = os.path.dirname(os.path.abspath(__file__))
chrome_path = mypath+"/chromedriver"
driver = webdriver.Chrome(executable_path=chrome_path)
driver.set_page_load_timeout(2)
try:
driver.get("https://www.bbcnews.com")
except TimeoutException:
print("Timeout, retrying...")
state = driver.execute_script("var state = document.readyState; return state;")
print(state)
driver.refresh("https://www.bbcnews.com")
Related
Recently, I made an auto-login script using Selenium on my Windows for my ISP's page. (I used Firefox's Geckodriver) :
`
from selenium import webdriver
from time import sleep
from selenium.webdriver.common.by import By
from selenium.webdriver.firefox.options import Options
from selenium.common.exceptions import NoSuchElementException
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException
import warnings
warnings.filterwarnings("ignore")
url = "..."
un = "..."
pwd = "..."
options = Options()
options.binary_location = r"C:\\Program Files\\Mozilla Firefox\\firefox.exe" #my firefox binary was custom installed elsewhere
driver=webdriver.Firefox(executable_path = r'C:\\geckodriver.exe' , options=options)
driver.get(url)
# .get() generally waits for a page to load, but cannot tell why the page is taking time to load
# router login pages generally load within a second
# if they don't, then there is generally a problem with the ISP
delay = 10
print ("A period of 10 seconds will be given for the site to load in case of a slow connection.")
print ("...")
sleep (6)
try:
WebDriverWait(driver, delay).until(EC.presence_of_element_located((By.ID, 'templatemo_wrapper'))) #common ID b/w log in & logged in page
except TimeoutException:
print ("Loading took too much time! Maybe your connection is not right")
try :
driver.find_element(By.NAME, "Username").send_keys(un)
driver.find_element(By.NAME, "Password").send_keys(pwd)
driver.find_element(By.ID, "submit_btn").click()
print("Done Logging In !!!")
sleep (6)
except NoSuchElementException :
print ("Already Logged In.")
sleep (6)
My next target is to automate my ISP login page on Android. But I cannot figure out how to do that. I made a simple application using Kivy (since Kivy is based on Python).
from kivymd.app import MDApp
from kivymd.uix.screen import Screen
from kivymd.uix.dialog import MDDialog
from kivymd.uix.button import MDRoundFlatButton, MDFlatButton
from kivy.lang import Builder
screen_helper = """
Screen:
BoxLayout:
orientation:"vertical"
MDTopAppBar:
title: "Login App"
elevation: 5
MDLabel:
"""
class Loginapp(MDApp):
def build(self):
self.theme_cls.primary_palette = "Blue"
screen = Screen()
button = MDRoundFlatButton(text="Login", pos_hint={"center_x": 0.5, "center_y": 0.5},
on_release=self.showresult)
titlebar = Builder.load_string(screen_helper)
screen.add_widget(button)
screen.add_widget(titlebar)
return screen
def showresult(self, obj):
# the connection STATUS would go to the MDDialog(text portion)
close_btn = MDFlatButton(text="Close", on_release=self.closedialog)
self.dlg = MDDialog(title="Connection Status :", text = ""
size_hint=(0.7, 1), buttons=[close_btn])
self.dlg.open()
def closedialog(self, obj):
self.dlg.dismiss()
Loginapp().run()
`
Here are the images of my App :
Login Page of App &
Result Page of App
The problem lies here :
In my Windows script, I had to specify my geckodriver location, my Python version etc. But how do I do that on Android (I use Chrome on Android)? Also, will the packages that worked on Windows work on Android despite using Kivy?
I must specify here that I am currently learning Kivy / Selenium. Really a beginner here.
I could not find any conclusive result on googling :(
Thanks to anyone who helps me out here :)
I tried to build a flask+selenium web on Heroku , so I went through a lot of step , I got my website setup , but when I tried to call an API that include selenium , I got an error from Selenium:
raise TimeoutException(message, screen, stacktrace)
selenium.common.exceptions.TimeoutException: Message:
Here's the part of my selenium python code , I tried to login to instagram.
chrome_options = webdriver.ChromeOptions()
chrome_options.binary_location = os.environ.get("GOOGLE_CHROME_BIN")
chrome_options.add_argument("--headless")
chrome_options.add_argument('window-size=1400,800')
chrome_options.add_argument("--disable-dev-shm-usage")
chrome_options.add_argument("--no-sandbox")
browser = webdriver.Chrome(executable_path=os.environ.get("CHROMEDRIVER_PATH"), chrome_options=chrome_options)
ig_username = os.environ.get("IG_USERNAME")
ig_password = os.environ.get("IG_PASSWORD")
web = 'https://www.instagram.com';
browser.get(web)
print ("Headless Chrome with custom window-size")
size = browser.get_window_size()
print("Window size: width = {}px, height = {}px".format(size["width"], size["height"]))
try:
wait = WebDriverWait(browser,10)
ig_account_ele = wait.until(EC.element_to_be_clickable((By.XPATH, //*[#id="loginForm"]/div/div[1]/div/label/input)))
print(ig_account_ele)
ig_account_ele.send_keys(ig_username)
The code will return timeout error at first element that I am trying to search.I even tried it on google.com and other element like span, it still won't work.
It's all fine when I run the code on local!!! ,but when it's deployed to Heroku, it will return timeout error or unable to locate element when I test the code with find_element_by_id,
so I assume there must be something wrong with selenium , but I don't have any way to solve it, seems it is also hard to see what's going on with chromedirver when running headless mode on Heroku...
I am trying to scrape this website using python's BeautifulSoup package and for automating the user flow I am using selenium. As this website requires authentication to access this page, I am trying to log in first using selenium webdriver. Here is my code:
from bs4 import BeautifulSoup
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import WebDriverWait
from selenium.common.exceptions import TimeoutException
def configure_driver():
# Add additional Options to the webdriver
chrome_options = Options()
# add the argument and make the browser Headless.
chrome_options.add_argument("--headless")
# Instantiate the Webdriver: Mention the executable path of the webdriver you have downloaded
# For linux/Mac
# driver = webdriver.Chrome(options = chrome_options)
# For windows
driver = webdriver.Chrome(executable_path="/home/<user_name>/Downloads/chromedriver_linux64/chromedriver",
options = chrome_options)
return driver
def getLinks(driver):
# Step 1: Go to pluralsight.com, category section with selected search keyword
driver.get(f"https://www.coursera.org/learn/competitive-data-science/supplement/RrrDR/additional-material-and-links")
# wait for the element to load
try:
WebDriverWait(driver, 5).until(lambda s: s.find_element_by_class_name("_ojjigd").is_displayed())
except TimeoutException:
print("TimeoutException: Element not found")
return None
email = driver.find_element_by_name('email')
print(str(email))
password = driver.find_element_by_name('password')
email.send_keys("username") # provide some actual username
password.send_keys("password") # provide some actual password
form = driver.find_element_by_name('login')
print(form.submit())
WebDriverWait(driver, 10)
print(driver.title)
soup = BeautifulSoup(driver.page_source)
# Step 3: Iterate over the search result and fetch the course
divs = soup.findAll('div', args={'class': 'item-box-content'})
print(len(divs))
# create the driver object.
driver = configure_driver()
getLinks(driver)
# close the driver.
driver.close()
Now after doing form.submit() it is expected to log in and change the page, right? But it is simply staying in the same page, so I cannot access the contents of the authenticated page. Someone please help.
That is because there is no name attribute.
instead of this :
form = driver.find_element_by_name('login')
Use this :
wait.until(EC.element_to_be_clickable((By.XPATH, "//button[text()='Login']"))).click()
I tried this code on my local, seems to be working fine
driver.maximize_window()
wait = WebDriverWait(driver, 30)
driver.get("https://www.coursera.org/learn/competitive-data-science/supplement/RrrDR/additional-material-and-links")
wait.until(EC.element_to_be_clickable((By.ID, "email"))).send_keys("some user name")
wait.until(EC.element_to_be_clickable((By.ID, "password"))).send_keys("some user name")
wait.until(EC.element_to_be_clickable((By.XPATH, "//button[text()='Login']"))).click()
Since login button is in a form so .submit() should work too.
wait.until(EC.element_to_be_clickable((By.XPATH, "//button[text()='Login']"))).submit()
This works too.
I'd like to run an end-to-end test of logging into our website using Selenium. We use Auth0 and the only available login mechanism is through Google social login. I wrote a script using Python Selenium (version 3.141.0), pytest, and selenium/standalone-chrome:87.0 Docker image which works correctly on my local machine, Mac OS 10.15.4.
However, it gets stuck at some point when I try to run it on CircleCI.
I use ubuntu-1604:202007-01 image in CircleCI
How I set up remote driver (tried a lot of arguments/commands..):
#pytest.fixture(scope="function")
def browser(remote_webdriver_url):
options = webdriver.ChromeOptions()
options.add_argument('--disable-popup-blocking')
options.add_argument('--disable-web-security')
options.add_argument('--allow-running-insecure-content')
options.add_argument('--start-maximized')
options.add_argument('-incognito')
options.add_experimental_option("useAutomationExtension", False)
options.add_experimental_option("excludeSwitches", ["enable-automation"])
browser = webdriver.Remote(
command_executor=remote_webdriver_url,
desired_capabilities=DesiredCapabilities.CHROME,
options=options)
return browser
My docker-compose.yml
version: '3.1'
services:
selenium-chrome:
image: selenium/standalone-chrome:87.0
# added the envvar as I found something about this in Selenium forums, it has no effect.
environment:
DBUS_SESSION_BUS_ADDRESS: /dev/null
shm_size: 2g
restart: 'no'
ports:
- "4444:4444"
My test code:
import os
from selenium.webdriver.common.by import By
from selenium.webdriver.remote.webelement import WebElement
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
def test_logging_in(browser: WebElement, url):
auth0_title = "Sign In with Auth0"
browser.get(url)
assert browser.title == auth0_title
# Log in
auth_login_button_class_name = 'auth0-lock-social-button-text'
_ = WebDriverWait(browser, 20).until(EC.presence_of_element_located((By.CLASS_NAME, auth_login_button_class_name)))
sign_in_button = browser.find_element_by_class_name(auth_login_button_class_name)
browser.implicitly_wait(5)
#
sign_in_button.click()
# Wait until we're redirected to Google's login page
_ = WebDriverWait(browser,20).until(EC.title_contains('Google'))
# Type in the email address and go to the next page
email_input = browser.find_element_by_tag_name('input')
email_input.send_keys(os.environ.get('E2E_TEST_DEVELOPMENT_USER_EMAIL'))
first_next_button = browser.find_element_by_id("identifierNext")
first_next_button.click()
# Wait until the password entry screen is loaded
browser.get_screenshot_as_file('/tmp/scr.png')
_ = WebDriverWait(browser, 20).until(EC.presence_of_element_located((By.ID, "forgotPassword"))) ##### THIS IS WHERE I GET THE TIMEOUT
# Put the password in
password_input = browser.find_element_by_xpath("//input[#name='password']")
password_input.send_keys(os.environ.get('E2E_TEST_DEVELOPMENT_USER_PASSWORD'))
second_next_button = browser.find_element_by_id("passwordNext")
second_next_button.click()
# Wait until the login is successful by observing the logout button
logout_icon_class_name = "bp3-icon-log-out"
_ = WebDriverWait(browser, 20).until(EC.presence_of_element_located((By.CLASS_NAME, logout_icon_class_name)))
assert browser.title == 'My page title'
sign_out_button = browser.find_element_by_class_name(logout_icon_class_name)
sign_out_button.click()
def test_teardown(browser):
browser.close()
browser.quit()
The test times out after clicking on the first button after typing in the email. I got screenshots from the run in CI, and it does seem to be stuck loading (see the Google's progress bar at the top, and the fact that it's more white-ish color), see the screenshot:
I also took a screenshot before clicking on the "Next" button, to show the contrast:
After having spent a long time on this and trying many things, I'm about to give up. Any ideas why this works locally but not in CI environment?
I'm trying to set MutationObserver for observing page mutation while loading.
In order to do that, MutationObserver should be configured before page loading.
With selenium-chromedriver, couldn't find the way to inject JS for such purpose.
I know chrome extension can do that but extensions won't work on headless mode.
That's the problem.
It's possible via the DevTool API by calling Page.addScriptToEvaluateOnNewDocument
from selenium import webdriver
from selenium.webdriver.remote.webdriver import WebDriver
import json
def send(driver, cmd, params={}):
resource = "/session/%s/chromium/send_command_and_get_result" % driver.session_id
url = driver.command_executor._url + resource
body = json.dumps({'cmd': cmd, 'params': params})
response = driver.command_executor._request('POST', url, body)
if response['status']:
raise Exception(response.get('value'))
return response.get('value')
def add_script(driver, script):
send(driver, "Page.addScriptToEvaluateOnNewDocument", {"source": script})
WebDriver.add_script = add_script
# launch Chrome
driver = webdriver.Chrome()
# add a script which will be executed when the page starts loading
driver.add_script("""
if (window.self === window.top) { // if main document
console.log('add script');
}
""")
# load a page
driver.get("https://stackoverflow.com/questions")
We can now use execute_cdp_cmd(cmd, cmd_args) to execute Chrome Devtools Protocol command in Selenium
from selenium import webdriver
driver = webdriver.Chrome()
driver.execute_cdp_cmd(
"Page.addScriptToEvaluateOnNewDocument",
{
"source": """// Your JavaScript here"""
}
)
driver.get("https://stackoverflow.com")
driver.quit()
The argument for "source" is just a string. So you can actually write your script in a .js file (for syntax highlighting) and read it using Python