i just start with selenium and have problems with webscraping from a betting website that keeps updating it's data. Because elements keeps updating i get sometimes stale element. I think this is because the elements are no longer attached to the DOM, but i don't now how i can't avoid this. How can I fix my code to handle the issue and then figure out a solution to it instead of re running the code?
Here is the code i have:
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import time
options = Options()
options.headless = True
website = 'https://www.betcity.nl/sportsbook#event/1007785387'
s = Service('C:/chromedriver.exe')
options.add_argument('window-size=1920x1080')
driver = webdriver.Chrome(service=s, options=options)
driver.get(website)
time.sleep(10)
box = driver.find_element(By.CLASS_NAME, 'KambiBC-scroller-wrapper')
rows = WebDriverWait(box, 50).until(EC.presence_of_all_elements_located((By.XPATH, "//ul[contains(#class, 'KambiBC-betoffer-categories-filter')]//a[contains(text(), 'Doelpuntenmaker')]")))
rows[0].click()
time.sleep(10)
scorer1 = []
boxs = driver.find_elements(By.CLASS_NAME, 'KambiBC-outcomes-list__column')
for box in boxs:
rows = box.find_elements(By.XPATH, ".//h4[contains(#class, 'KambiBC-outcomes-list__row-header KambiBC-outcomes-list__row-header--participant')]//span")
for n, odd in enumerate(rows):
scorer1.append(odd.text)
After years of getting annoyed by stale element exceptions I gave up fighting with them and now just have resilient functions (or c# extension methods) that I pass my element locators to and they retry the operation for a specified time until it works.
It isn't very sophisticated but it saves a lot of pain.
c# example:
public static void WaitClickElement(this IWebDriver driver, By by, int millis = 2000)
{
TestTimer timer = new TestTimer(millis);
while(!timer.Expired())
{
try
{
driver.FindElement(by).Click();
break;
}
catch(Exception e)
{
if(timer.Expired()) {
throw new Exception("Could not click element in specified time: " + by.ToString() + "\n" + e.Message);
}
}
}
}
Related
Path = r"C:\WebDriverEdge\chromedriver.exe"
service = Service(Path)
options = Options()
options.add_argument('--user-data-dir=C:\\Users\\Admin\\AppData\\Local\\Google\\Chrome\\User Data\\')
options.add_argument("--profile-directory=Profile 1")
#connect to driver
driver = webdriver.Chrome(service = service, options = options)
driver.get("https://open.spotify.com/search")
x_path = '//*[#id="main"]/div/div[2]/div[1]/header/div[3]/div/div/form/input'
search = driver.find_element(By.XPATH, x_path)
action = webdriver.ActionChains(driver)
action.move_to_element(search).send_keys("Let me").perform()
I try to click on search bar at Spotify and use it to search. My problem is when I already login my code get error "unable to find element" but without sign in I can fill the search bar easily.
I don't know why. Is any one run into this before? Thanks in advance
p/s: XPath still the same
I'd rather use this form[role='search'] input css with explicit wait like below:
driver.get("https://open.spotify.com/search")
driver.maximize_window()
wait = WebDriverWait(driver, 20)
wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, "form[role='search']"))).send_keys('Let me')
Imports:
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
how handle auto suggest in "from" and "destination" box for this website "https://www.goibibo.com/" in selenium.
please help
I tired using the basic method but unable to get the X path of the auto suggestion drop down
Unable to click on the drop down
package basic;
import java.util.List;
import java.util.concurrent.TimeUnit;
import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.WebElement;
import org.openqa.selenium.chrome.ChromeDriver;
import org.openqa.selenium.support.ui.ExpectedConditions;
import org.openqa.selenium.support.ui.WebDriverWait;
public class goibibo {
public static void main(String[] args) throws InterruptedException {
// TODO Auto-generated method stub
WebDriver driver = new ChromeDriver();
driver.get("https://www.goibibo.com/");
new WebDriverWait(driver, 20)
.until(ExpectedConditions.elementToBeClickable(By.xpath("//input[#id='gosuggest_inputSrc']")))
.sendKeys("Mum");
List<WebElement> myList = new WebDriverWait(driver, 20).until(
ExpectedConditions.visibilityOfAllElementsLocatedBy(By.xpath("//*[#id=\"react-autosuggest-1\"]")));
for (WebElement element : myList) {
if (element.getText().contains("Mumbai"))
;
element.click();
}
}
}
Chrome Browser
First how to Find XPATH of auto populate box in Chrome Browser open your website than click on Inspect element and click on source Tab now, click for opening your auto populate box and Press **F8** **Key for pause debugger**. Then click on your Element tab and you can easily get your xpath refer below snap for more information. so it will freeze your HTML.
Now click on Elements an Create your own xpath.
Fire Fox Browser
Second how to find xpath of Auto Populate box in Firefox - Open your Firefox and Right click and click on inspect elements on your website. there is option of animation so it will open all your DOM Expanded like below image. so by reading this dom structure you can create easily your XPATH.
Not how to find Elements from auto populate box. Refer below code snippet for that.
package com.software.testing;
import java.util.List;
import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.WebElement;
import org.openqa.selenium.chrome.ChromeDriver;
import org.openqa.selenium.support.ui.ExpectedConditions;
import org.openqa.selenium.support.ui.WebDriverWait;
public class Testingclass extends DriverFactory {
private static WebDriver driver = null;
public static void main(String[] args) throws InterruptedException {
System.setProperty("webdriver.chrome.driver", "your driver path");
driver = new ChromeDriver();
driver.get("https://www.goibibo.com/");
new WebDriverWait(driver, 20)
.until(ExpectedConditions.elementToBeClickable(By.xpath("//input[#id='gosuggest_inputSrc']")))
.sendKeys("A");
Thread.sleep(1000);
List<WebElement> myList = new WebDriverWait(driver, 20).until(ExpectedConditions.presenceOfAllElementsLocatedBy(
By.xpath("//div[#class='dib marginL10 pad0 textOverflow width90']/div/span")));
for (int i = 0; i < myList.size(); i++) {
System.out.println(myList.get(i).getText());
if (myList.get(i).getText().equals("Ahmedabad")) {
myList.get(i).click();
break;
}
}
}
}
Don't forgot to use break after your conditional statement else it
will thrown an exception.
So you can try one solution please find the below screenshot,
As you can see in screenshot if i type M in text box then dropdown shows the record respect to letter 'M' and if you see in source the <ul> which is dynamic as you see just below <input> so you need to handle that dropdown by it's locator it is dynamic hence first you need to pass some text in text box and after that you need to select the element from the drop down using Select in selenium you use selectByVisibleText("") or what ever or you can use List<Element> you can store all the respected sources (Mumbai, Mysore ,etc)coming from dropdown and use it wisely
new WebDriverWait(driver, 20).until(ExpectedConditions.elementToBeClickable(By.xpath("//input[#id='gosuggest_inputSrc'"]))).sendKeys("M");
List<WebElement> myList = new WebDriverWait(driver, 20).until(ExpectedConditions.visibilityOfAllElementsLocatedBy(By.xpath("Xpath of the dynamic drop down")));
for (WebElement element:myList) {
if(element.getText().contains("Mumbai"));
element.click();
}
I gave you an idea let me know if you need any further help
I have automated it through selenium with python. Its collecting all suggested cities in a list and then clicking the required one.
from selenium import webdriver
driver = webdriver.Chrome()
driver.maximize_window()
driver.get("https://www.goibibo.com/")
driver.implicitly_wait(3)
listCity = []
driver.find_element_by_xpath("//input[#id='gosuggest_inputSrc']").send_keys("JA")
cities = driver.find_elements_by_xpath("//div[#class='mainTxt clearfix']//preceding-sibling::span")
for city in cities:
listCity.append(city.text)
for city in cities:
if "Jagdalpur" in city.text:
city.click()
break
print(listCity)
print(len(listCity))
Use below code it will work
Webelement ele=driver.findelement()
Actions ob = new Actions(driver);
ob.moveToElement(ele);
ob.click(ele);
Action action = ob.build();
action.perform();
how handle auto suggest in "from" and "destination" box for this website "https://www.goibibo.com/" in selenium.
please help
I tired using the basic method but unable to get the X path of the auto suggestion drop down
Unable to click on the drop down
package basic;
import java.util.List;
import java.util.concurrent.TimeUnit;
import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.WebElement;
import org.openqa.selenium.chrome.ChromeDriver;
import org.openqa.selenium.support.ui.ExpectedConditions;
import org.openqa.selenium.support.ui.WebDriverWait;
public class goibibo {
public static void main(String[] args) throws InterruptedException {
// TODO Auto-generated method stub
WebDriver driver = new ChromeDriver();
driver.get("https://www.goibibo.com/");
new WebDriverWait(driver, 20)
.until(ExpectedConditions.elementToBeClickable(By.xpath("//input[#id='gosuggest_inputSrc']")))
.sendKeys("Mum");
List<WebElement> myList = new WebDriverWait(driver, 20).until(
ExpectedConditions.visibilityOfAllElementsLocatedBy(By.xpath("//*[#id=\"react-autosuggest-1\"]")));
for (WebElement element : myList) {
if (element.getText().contains("Mumbai"))
;
element.click();
}
}
}
Chrome Browser
First how to Find XPATH of auto populate box in Chrome Browser open your website than click on Inspect element and click on source Tab now, click for opening your auto populate box and Press **F8** **Key for pause debugger**. Then click on your Element tab and you can easily get your xpath refer below snap for more information. so it will freeze your HTML.
Now click on Elements an Create your own xpath.
Fire Fox Browser
Second how to find xpath of Auto Populate box in Firefox - Open your Firefox and Right click and click on inspect elements on your website. there is option of animation so it will open all your DOM Expanded like below image. so by reading this dom structure you can create easily your XPATH.
Not how to find Elements from auto populate box. Refer below code snippet for that.
package com.software.testing;
import java.util.List;
import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.WebElement;
import org.openqa.selenium.chrome.ChromeDriver;
import org.openqa.selenium.support.ui.ExpectedConditions;
import org.openqa.selenium.support.ui.WebDriverWait;
public class Testingclass extends DriverFactory {
private static WebDriver driver = null;
public static void main(String[] args) throws InterruptedException {
System.setProperty("webdriver.chrome.driver", "your driver path");
driver = new ChromeDriver();
driver.get("https://www.goibibo.com/");
new WebDriverWait(driver, 20)
.until(ExpectedConditions.elementToBeClickable(By.xpath("//input[#id='gosuggest_inputSrc']")))
.sendKeys("A");
Thread.sleep(1000);
List<WebElement> myList = new WebDriverWait(driver, 20).until(ExpectedConditions.presenceOfAllElementsLocatedBy(
By.xpath("//div[#class='dib marginL10 pad0 textOverflow width90']/div/span")));
for (int i = 0; i < myList.size(); i++) {
System.out.println(myList.get(i).getText());
if (myList.get(i).getText().equals("Ahmedabad")) {
myList.get(i).click();
break;
}
}
}
}
Don't forgot to use break after your conditional statement else it
will thrown an exception.
So you can try one solution please find the below screenshot,
As you can see in screenshot if i type M in text box then dropdown shows the record respect to letter 'M' and if you see in source the <ul> which is dynamic as you see just below <input> so you need to handle that dropdown by it's locator it is dynamic hence first you need to pass some text in text box and after that you need to select the element from the drop down using Select in selenium you use selectByVisibleText("") or what ever or you can use List<Element> you can store all the respected sources (Mumbai, Mysore ,etc)coming from dropdown and use it wisely
new WebDriverWait(driver, 20).until(ExpectedConditions.elementToBeClickable(By.xpath("//input[#id='gosuggest_inputSrc'"]))).sendKeys("M");
List<WebElement> myList = new WebDriverWait(driver, 20).until(ExpectedConditions.visibilityOfAllElementsLocatedBy(By.xpath("Xpath of the dynamic drop down")));
for (WebElement element:myList) {
if(element.getText().contains("Mumbai"));
element.click();
}
I gave you an idea let me know if you need any further help
I have automated it through selenium with python. Its collecting all suggested cities in a list and then clicking the required one.
from selenium import webdriver
driver = webdriver.Chrome()
driver.maximize_window()
driver.get("https://www.goibibo.com/")
driver.implicitly_wait(3)
listCity = []
driver.find_element_by_xpath("//input[#id='gosuggest_inputSrc']").send_keys("JA")
cities = driver.find_elements_by_xpath("//div[#class='mainTxt clearfix']//preceding-sibling::span")
for city in cities:
listCity.append(city.text)
for city in cities:
if "Jagdalpur" in city.text:
city.click()
break
print(listCity)
print(len(listCity))
Use below code it will work
Webelement ele=driver.findelement()
Actions ob = new Actions(driver);
ob.moveToElement(ele);
ob.click(ele);
Action action = ob.build();
action.perform();
this is my test:
public void Add_Store()
{
StepDescription = " Create a Store";
Thread.Sleep(15000);
Store_Link.Click();
Thread.Sleep(15000);
Create_Store_Button.Click();
System.Threading.Thread.Sleep(Convert.ToInt32("5000"));
}
I want to remove all the sleep; where should i add the wait now? i want to make it generic; so it make sense to add in the element locator itself?
the element locator looks as below:
[Find(ControlType = ElementType.Hyperlink, Properties = "XPath=//a[starts-with(#id, 'Stores')]")]
private Element Store_Link { get; set; }
since i want to make the wait generic, can you guide me how to remove the sleep and add the wait?
As you have tagged Python :
You can use explicit wait as #Ishita has mentioned. Though you want to have generic wait , that can be used every where.
For generic, you may create instance of WebDriverWait in setup/fixtures level.Something like :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
wait = WebDriverWait(webDriverInstance, timeInSec)
#wait = WebDriverWait(driver, 30)
wait.until(EC.visibility_of_element_located((By.ID, 'Some id')))
or may be somewhere else in your code :
wait.until(EC.element_to_be_clickable((By.ID, 'Some id')))
Now you can use this wait reference, in your code where ever it has the accessibility.
For Reference, you can use this link Selenium_explicit_wait_Python
and the code you have shared look likes in C# :
You can use this code in this case :
WebDriverWait wait = new WebDriverWait(driver, TimeSpan.FromSeconds(Time));
wait.Until(ExpectedConditions.ElementToBeClickable(locator));
Note that we can bind conditions with EC (ExpectedConditions).
Hope this will help.
you can use Explicit wait instead of sleep,
Python:
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
WebDriverWait(driver, 10).until(EC.visibility_of_element_located((By.XPATH, ""))).click()
//EC.presence_of_element_located
//EC.element_to_be_clickable
C#:
WebDriverWait wait = new WebDriverWait(Driver, TimeSpan.FromSeconds(10));
wait.Until(ExpectedConditions.ElementToBeClickable(webElement)).Click();
There are many types of wait.
One of the best Solution for creating JS wait Framework for project:
Blog Link : https://www.swtestacademy.com/selenium-wait-javascript-angular-ajax/
& Github link : https://github.com/swtestacademy/JSWaiter
I'm trying to use Selenium to log into ESPN. A solution that used to work is detailed here. In order to log in I need to find the frame that has the username and password fields and switch to that frame. Unfortunately, that numerical index of that frame isn't always the same. I decided to just try them all, but as soon as I've switched to one frame, switching to the next fails with selenium.common.exceptions.StaleElementReferenceException: Message: The element reference of <iframe class="ob-pifr"> stale: either the element is no longer attached to the DOM or the page has been refreshed. So I'm looking for
A way to switch frames without the StaleElementReferenceException
A way to check whether a frame is the one I want before I switch to it
Some other solution, though I'd prefer something introspective to a magic (and presumably fragile) formula like "It's always the third frame from the end)
Here's some sample code that leads to the StaleElementReferenceException:
from time import sleep
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.common.exceptions import NoSuchElementException
driver = webdriver.Firefox(executable_path = '/usr/local/bin/geckodriver')
driver.get("http://games.espn.go.com/ffl/signin")
WebDriverWait(driver, 1000).until(EC.presence_of_all_elements_located((By.XPATH,"(//iframe)")))
elem = None
frms = driver.find_elements_by_xpath("(//iframe)")
print("Found {} frames", len(frms)) # Varies from 6 to 8
for count, frm in enumerate(frms):
print("Trying frm[{}]".format(count))
driver.switch_to.frame(frm)
sleep(2)
try:
# The command below will fail the second time around with
# `either the element is no longer attached to the DOM or the page has been refreshed`
elem = driver.find_element_by_xpath("(//input)[1]")
except NoSuchElementException:
pass
else:
break
Frame id is disneyid-iframe which opens the login popup So first you need to switch into it
driver.switch_to_frame(driver.find_element_by_id("disneyid-iframe"))
and then perform sendkeys like
driver.find_element_by_xpath("//input[#type='email']").send_keys("emailid")
driver.find_element_by_xpath("//input[#type='password']").send_keys("password")
Other way to switch in frame is ExplicitWait. It wait until frame available once there then it get switched in
wait = WebDriverWait(driver, 60)
wait.until(EC.frame_to_be_available_and_switch_to_it((By.ID, "disneyid-iframe")))
Your Final code will be :
from time import sleep
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.common.exceptions import NoSuchElementException
driver = webdriver.Firefox(executable_path = '/usr/local/bin/geckodriver')
driver.get("http://games.espn.go.com/ffl/signin")
wait = WebDriverWait(driver, 60)
wait.until(EC.frame_to_be_available_and_switch_to_it((By.ID, "disneyid-iframe")))
driver.find_element_by_xpath("//input[#type='email']").send_keys("emailid")
driver.find_element_by_xpath("//input[#type='password']").send_keys("password")
driver.find_element_by_xpath("//button[#type='submit']").click()
Note : Please check syntax as per Python.