Having trouble finding element on page with xpath - selenium

I am trying to access the data as shown below.
I am using this command to capture the information but to no avail. Does anyone have any tips on where I'm going wrong?
Code trials:
posts = firefox.find_elements_by_xpath(//*[#id="flotGagueValue0"])
print(posts)
for post in posts:
print(post)

You are looking for ".text".
posts = firefox.find_elements_by_xpath(//*[#id="flotGagueValue0"]) print(posts) for post in posts: print(post.text)
I would write it like this for readability:
posts = firefox.find_elements_by_xpath(//*[#id="flotGagueValue0"])
for post in posts:
print(post.text)

The desired element is an Angular element so to extract the text 98.72 you have to induce WebDriverWait for the visibility_of_element_located() and you can use either of the following solutions:
Using CSS_SELECTOR and text attribute:
print(WebDriverWait(firefox, 20).until(EC.visibility_of_element_located((By.CSS_SELECTOR, "div.panel-content div.singlestat-panel > div span#flotGagueValue0"))).text)
Using XPATH and get_attribute():
print(WebDriverWait(firefox, 20).until(EC.visibility_of_element_located((By.XPATH, "//div[#class='panel-content']//div[#class='singlestat-panel']/div//span[#id='flotGagueValue0']"))).get_attribute("innerHTML"))
Note : You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

Related

select element in selenium using partial xcode or partial css selector

I am trying to select a certain element in a webpage in selenium. I know that the element's name looks like person_xxxxx with xxxx being random numbers. I would like to know if it is possible to select this element using xpath or css selector. So far I have tried:
cartes_profils=liste_profils_shadow.find_elements_by_xpath('//*[contains(#id,"person_")]')
which is deprecated and doesn't work
cartes_profils=liste_profils_shadow.find_elements(by=By.CSS_SELECTOR,value='input[id^="#person_"]')
which runs but doesn't select the desired elements
cartes_profils=liste_profils_shadow.find_elements(by=By.XPATH,value=("//*[contains(#id, 'person_')]"))
which returns an "invalid selector" error
PS: I know that there are similar topics already answered but they are all a million years old and the solutions do not work for me
You should ideally be using starts-with for xpath. Also, yes you are right
find_elements_by_*** is deprecated in selenium 4. You should use find_element(By.XPATH, "")
so your effective xpath would be:
//*[starts-with(#id,'person_')]
use it like
driver.find_element(By.XPATH, "//*[starts-with(#id,'person_')]")
and equivalent CSS would be: (you are fairly close here)
input[id^="person_"]
use it like
driver.find_element(By.CSS_SELECTOR, "input[id^="person_"]")
My recommendation would be explicit waits:
element = WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//*[starts-with(#id,'person_')]")))
You'll have to import these for explicit waits:
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

Web-scraping dynamic website with user input using Selenium and Python

As A Swimmer, I am trying to pull times from a table that can be accessed after the User Inputs their name or other optional fields. The website dynamically generates this data. Below is my current code which does not factor in user inputs.
I am very confused about how selenium's automation works and how to find the right text field for it to read my results and for the rest of my code to extract the table.
Can anyone give some advice on how to proceed?
Any help is appreciated and thanks in advance.
This Is My Current Code:
from selenium import webdriver
from bs4 import BeautifulSoup
import pandas as pd
options = webdriver.ChromeOptions()
options.add_argument('--headless')
options.add_argument('--no-sandbox')
options.add_argument('--disable-dev-shm-usage')
site = 'https://www.swimming.org.nz/results.html'
wd = webdriver.Chrome( "C:\\Users\\joseph\\webscrape\\chromedriver.exe")
wd.get(site)
html = wd.page_source
df = pd.read_html(html)
df[1].to_csv('Results.csv')
To start with you need to send a character sequence to the Swimmer field.
To send a character sequence to the Swimmer field as the elements are within an iframe so you have to:
Induce WebDriverWait for the desired frame to be available and switch to it.
Induce WebDriverWait for the desired element to be clickable.
You can use either of the following Locator Strategies:
Using CSS_SELECTOR:
driver.get("https://www.swimming.org.nz/results.html")
WebDriverWait(driver, 20).until(EC.frame_to_be_available_and_switch_to_it((By.CSS_SELECTOR,"iframe#iframe")))
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "input[id^='x-MS_FIELD_MEMBER']"))).send_keys("Joseph Zhang")
Using XPATH:
driver.get("https://www.swimming.org.nz/results.html")
WebDriverWait(driver, 20).until(EC.frame_to_be_available_and_switch_to_it((By.XPATH,"//iframe[#id='iframe']")))
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//input[starts-with(#id, 'x-MS_FIELD_MEMBER')]"))).send_keys("Joseph Zhang")
Note : You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
Browser Snapshot:
References
You can find a couple of relevant discussions in:
Switch to an iframe through Selenium and python
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element while trying to click Next button with selenium
selenium in python : NoSuchElementException: Message: no such element: Unable to locate element

How can I click on the third element in this list using selenium? I have tried everything and nothing works

I am running a webscraper and I am not able to click on the third element. I am not sure what to do as I have tried googling and running several types of code.
Below is a screenshot of the html and my code. I need the third element in the list to be clicked on. It is highlighted in the screenshot. I am not sure what to do with the css and data-bind
here is the code for max bed options. I also need to get the 2 beds just like we did for min bed options
thanks!!
According to the picture the following should work:
driver.find_element_by_xpath('//span[#id="bedsMinMaxRangeControl"]//li[#data-value="2"]').click()
But we need to see the entire page HTML to give a correct answer.
Also don't forget to use delays/ waits there.
UPD
For the new question the code will be:
driver.find_element_by_xpath('//span[#id="bedsMinMaxRangeControl"]//ul[contains(#class,"maxBedOptions")]//li[#data-value="2"]').click()
Here you should also use the appropriate data-value that has values from -1 up to 3
You can use css_selector with the data-value attribute.
locator = ".dropdownContent .minBedOptions li[data-value='2']"
WebDriverWait(driver, 10).until((EC.element_to_be_clickable, (By.CSS_SELECTOR, locator))).click()
I used WebDriverWait so make sure to import it...
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
If it's just a list using li tags as it seems from the shared snap, You could probably write the easiest xpath as :
//li[contains(text(), '2 Beds')]
and use it like this :
driver.find_element_by_xpath("//li[contains(text(), '2 Beds')]").click()
or if you want to use xpath in conjunction of WebDriverWait, use it like this :
wait = WebDriverWait(driver, 10)
element = wait.until(EC.element_to_be_clickable((By.XPATH, "//li[contains(text(), '2 Beds')]")))
Import :
from selenium.webdriver.support import expected_conditions as EC
Now let's talk about when we do not want to be dependent on 2 Beds text in XPATH, cause if text changes in the UI, we'd have to change the locator in Selenium-Python bindings.
A good way to do is to :
Make use of data-value attribute : //li[#data-value = '2']
Make use of ul and li tags : //ul[contains(#class, 'minBedOptions')/li[#data-value = '2']]

Selenium access table by xpath

i'm fairly new to selenium and i'm building a scraper to extract info from a table.
I'm able to acces the table body by ID with no problem but when I try to access it's children they are not found.
the inspector shows the xpath for the first cell as //*[#id="tb_list"]/tr[1]/td[1] but
find_element_by_xpath(//*[#id="tb_list"]/tr[1]/td[1])
can't find it.
I also tried the following to no avail.
table = driver.find_element_by_id("tb_list")
table.find_element_by_xpath(".//tr[1]/td[1]")
it's able to find tb_list but fails to locate the children
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":".//tr[1]/td[1]"}
Everywhere I looked people suggest one of these 2 methods, what am I doing wrong? The table is dynamically populated from a database, could this be an issue?
I'm using python and the chrome web driver,
I'm hesitant to give a snippet of the html as the site is not publicly available and i dont own it.
[1] indicates first descendent. So the xpath:
//*[#id="tb_list"]/tr[1]/td[1]
can be optimized as:
//*[#id="tb_list"]/tr/td
Effectively the line of code would be:
driver.find_element(By.XPATH, "//*[#id='tb_list']/tr/td")
Ideally, to locate the element you need to induce WebDriverWait for the visibility_of_element_located() and you can use either of the following Locator Strategies:
Using XPATH:
element = WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//*[#id='tb_list']/tr/td")))
Note : You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

find_element_by_class_name in selenium giving error

I am trying to automate a button click using selenium but it is giving me the error. The html code of the page is:
The code i am trying is:
create_team=driver.find_element_by_class_name('ts-btn ts-btn-fluent ts-btn-fluent-secondary ts-btn-fluent-with-icon join-team-button')
create_team.click()
I am getting the following error:
driver.find_element_by_class_name() only accepts one className, it's not built to handle multiple classNames, reference - (How to locate an element with multiple class names?), THIS SEEMS TO BE UP FOR DEBATE
Use driver.find_element_by_css_selector('ts-btn.ts-btn-fluent.ts-btn-fluent-secondary.ts-btn-fluent-with-icon.join-team-button')
With driver.find_element_by_css_selector you can chain multiple classNames together using a dot(.) between each className in the selector.
To click on the element you can use either of the following Locator Strategies:
Using css_selector:
driver.find_element_by_css_selector("button.ts-btn.ts-btn-fluent.ts-btn-fluent-secondary.ts-btn-fluent-with-icon.join-team-button[data-tid='tg-discover-team']").click()
Using xpath:
driver.find_element_by_xpath("//button[#class='ts-btn ts-btn-fluent ts-btn-fluent-secondary ts-btn-fluent-with-icon join-team-button' and #data-tid='tg-discover-team']").click()
As the desired element is an Angular element, ideally to click on the element you need to induce WebDriverWait for the element_to_be_clickable() and you can use either of the following Locator Strategies:
Using CSS_SELECTOR:
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button.ts-btn.ts-btn-fluent.ts-btn-fluent-secondary.ts-btn-fluent-with-icon.join-team-button[data-tid='tg-discover-team']"))).click()
Using XPATH:
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//button[#class='ts-btn ts-btn-fluent ts-btn-fluent-secondary ts-btn-fluent-with-icon join-team-button' and #data-tid='tg-discover-team']"))).click()
Note: You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
create_team=driver.find_element_by_class_name('ts-btn.ts-btn-fluent.ts-btn-fluent-secondary ts-btn-fluent-with-icon.join-team-button')
create_team.click()
You have to replace space with . as space indicate multiple class
You can use xpath or css also:
create_team=driver.find_element_by_xpath("//*[#class='ts-btn.ts-btn-fluent.ts-btn-fluent-secondary ts-btn-fluent-with-icon.join-team-button]')
create_team.click()
create_team=driver.find_element_by_css_selector("[class='ts-btn.ts-btn-fluent.ts-btn-fluent-secondary ts-btn-fluent-with-icon.join-team-button]')
create_team.click()
More to answer
If you check the exception from the by_class_name:
You can see that it is using css_class locator under the hood ( You can see it add . in frontautomatically)
Working example:
from selenium import webdriver
import time
from selenium.webdriver.support import expected_conditions as EC
driver = webdriver.Chrome()
driver.get("https://stackoverflow.com/questions/65579491/find-element-by-class-name-in-selenium-giving-error/65579606?noredirect=1#comment115946541_65579606")
time.sleep(5)
elem = driver.find_element_by_class_name('overflow-x-auto.ml-auto.-secondary.grid.ai-center.list-reset.h100')
print(elem.get_attribute("outerHTML"))