I have an html which looks as following:
<div class="v-window-outerheader"><div class="v-window-maximizebox" tabindex="0" role="button" aria-label="maximize button" id="38_window_maximizerestore"></div>
<div class="v-window-closebox" tabindex="0" role="button" aria-label="close button" id="38_window_close"></div>
This code is for a message box which pops up on logging into the website.
I have been using the following code to click on the X on top right to close the message pop:
from selenium import webdriver
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait, Select
from selenium.webdriver.common.keys import Keys
path_to_chromedriver = r'C:\chromedriver' # change path as needed
browser = webdriver.Chrome(executable_path=path_to_chromedriver)
wait = WebDriverWait(browser, 10)
browser.get("https://ftrcenter.pjm.com/ftrcenter/pages/secure/")
wait.until(EC.presence_of_element_located((By.XPATH, '//*[#id="37_window_close"]'))).click()
But, the ID number 37_window_close changes every so often. Today the id is 38_window_close
Rest of the html code remains same. How do I modify my code so that my code doesn't break due to this change in ID?
The pop up box appears as following, where X is on top right to close it:
Use aria-label="close button" attribute.
wait.until(EC.presence_of_element_located((By.XPATH, '//*[#aria-label="close button"]'))).click()
Or class attribute.
wait.until(EC.presence_of_element_located((By.XPATH, '//*[#class="v-window-closebox"]'))).click()
Or use contains with id
wait.until(EC.presence_of_element_located((By.XPATH, '//*[contains(#id,"_window_close")]'))).click()
For dynamic IDs, You can match the substring instead of the exact string.
Use Below XPath:
//div[contains(#id,"_window_close")]
OR CSS:
div[id*="_window_close"]
Hope this helps :)
Related
How to extract the text which is displayed as part of the link inside the span marker.
<span class="pull-left w-100 font30 medium_blue_type mb10"><a href='/XLY'>XLY</a></span> <span class="w-100">Largest Allocation</span>
Output:
XLY
I've tried several approaches, among all, using
elems = driver.find_elements_by_class_name("span.pull-left.w-100.font30.medium_blue_type.mb10")
elems = driver.find_element_by_xpath('.//span[#class = "pull-left w-100 font30 medium_blue_type mb10"]')
but can't get it working. The website is https://www.etf.com/stock/TSLA.
EDIT:
Is it possible to do it without opening the window in the browser, e.g. using "headless" option?
op = webdriver.ChromeOptions()
op.add_argument('headless')
driver = webdriver.Chrome(CHROME_DRIVER_PATH, options=op)
If you prefer to have a text-based locators, you can use the below:
//span[text()='Largest Allocation']/../span
You should click on the cookies I understand button first.
Make use of explicit waits.
So your effective code would be:
driver = webdriver.Chrome(driver_path)
driver.maximize_window()
wait = WebDriverWait(driver, 30)
driver.get("https://www.etf.com/stock/TSLA")
try:
wait.until(EC.element_to_be_clickable((By.LINK_TEXT, "I Understand"))).click()
print("Clicked on I understand button")
except:
pass
txt = wait.until(EC.visibility_of_element_located((By.XPATH, "//span[text()='Largest Allocation']/../span"))).text
print(txt)
Imports:
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
Output:
Clicked on I understand button
XLY
Process finished with exit code 0
If you are looking for locators not based on text, use the below line of code:
txt = wait.until(EC.visibility_of_element_located((By.XPATH, "(//span[contains(#class,'medium_blue_type')]//a)[2]"))).text
There are several possible problems here:
Maybe you are missing a delay
The locator you are using may be not unique
I can see here you are extracting the attribute value from the returned web element
The web element can be inside iframe etc.
Based on currently shared information you can try adding a wait and extracting the web element value as following:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
wait = WebDriverWait(driver, 20)
href = wait.until(EC.visibility_of_element_located((By.XPATH, "//span[#class = "pull-left w-100 font30 medium_blue_type mb10"]"))).get_attribute("href")
Use the following xpath to identify the href link.
//div[./span[text()='Largest Allocation']]//a
You need to induce some delay to get the element.
Use WebDriverWait() and wait for visibility of the element.
To get the text:
print(WebDriverWait(driver,10).until(EC.visibility_of_element_located((By.XPATH, "//div[./span[text()='Largest Allocation']]//a"))).text)
To get the href:
print(WebDriverWait(driver,10).until(EC.visibility_of_element_located((By.XPATH, "//div[./span[text()='Largest Allocation']]//a"))).get_attribute("href"))
you need to import below libraries.
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
I'm using Selenium and need to click on a button called Income Statement at http://www.tradingview.com/screener. Could anyone help me to find the button name and let me know if the code where I'm trying to implement it is correct. Any help much appreciated. Thanks!!
url = 'http://www.tradingview.com/screener'
driver = webdriver.Firefox()
driver.get(url)
button_element = driver.find_element_by_xpath("//input[#name='Income Statement']")
button_element.click()
There are 2 problems here:
You have to add wait / delay before accessing that element.
This should preferably be done by expected conditions explicit waits.
You are using a wrong locator.
This should work better:
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
url = 'http://www.tradingview.com/screener'
driver = webdriver.Firefox()
wait = WebDriverWait(driver, 20)
driver.get(url)
wait.until(EC.visibility_of_element_located((By.XPATH, "//div[#data-set='income_statement']"))).click()
Your xpath does not locate the right element in the HTMLDOM.
so instead of this:
//input[#name='Income Statement']
use this:
//div[#data-set='income_statement']
or
a CSS like this:
div[data-set = 'income_statement']
explanation:
See this is a outerHTML:
<div class="tv-screener-toolbar__favorites-item js-favorite-item" data-set="income_statement">
Income Statement
</div>
As we can see it is a div tag, so we are using //div and also it has data-set attribute income_statement and when you write the XPath, it is able to locate the right node with 1/1 uniqueness.
Also It's a best practise to use ExplicitWait:
wait = WebDriverWait(driver, 30)
try:
wait.until(EC.element_to_be_clickable((By.XPATH, "//div[#data-set='income_statement']"))).click()
print('Clicked on the button')
except:
print('Could not click ')
pass
Imports:
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
I am trying to create a bot as a side project. So the automated flow will be the "bot" opens up my browser, then the intended link, log in and select items from a dropdown button.
So far i managed to open my browser, open my link and log in automatically with my email address. But after logging in, i am unable to select what i wanted under the dropdown button and i am not even sure why.
I have try to find my element by id and it returns a "TypeError: 'module' object is not callable".
Currently, the code i have is:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.common.exceptions import TimeoutException
import selenium.webdriver.support.ui as Select
import time
driver = webdriver.Chrome()
driver.maximize_window() # For maximizing window
driver.implicitly_wait(20) # gives an implicit wait for 20 seconds
driver.get("https://connect.com/my-dashboard")
login = driver.find_element_by_id('Email')
login.send_keys('user#email.com')
# click the 'Next' button
driver.find_element_by_class_name('move-right').click()
# select option under dropdown button
select = Select(driver.find_element_by_id('siteId'))
select.select_by_visible_text('Entity A')
in the end, it returns:
TypeError: 'module' object is not callable
I have clicked into "Investigate" for the button and it is as follows:
<p-dropdown _ngcontent-anw-c314="" appdropdownposition="" formcontrolname="siteSelector" styleclass="fixed-width small-medium" dropdownicon="pwc-icon icon-caret-down" inputid="siteId" optionlabel="label" class="dashboard-controls__quick-nav u-m-r-1 ng-tns-c66-2 ng-untouched ng-pristine ng-valid"><div class="ng-tns-c66-2 fixed-width small-medium p-dropdown p-component"><div class="p-hidden-accessible ng-tns-c66-2"><input type="text" readonly="" aria-haspopup="listbox" class="ng-tns-c66-2" id="siteId"></div><!----><span class="ng-tns-c66-2 p-dropdown-label p-inputtext p-placeholder ng-star-inserted">- Select -</span><!----><!----><!----><div role="button" aria-haspopup="listbox" class="p-dropdown-trigger ng-tns-c66-2"><span class="p-dropdown-trigger-icon ng-tns-c66-2 pwc-icon icon-caret-down"></span></div><!----></div></p-dropdown>
<div class="ng-tns-c66-2 fixed-width small-medium p-dropdown p-component"><div class="p-hidden-accessible ng-tns-c66-2"><input type="text" readonly="" aria-haspopup="listbox" class="ng-tns-c66-2" id="siteId"></div><!----><span class="ng-tns-c66-2 p-dropdown-label p-inputtext p-placeholder ng-star-inserted">- Select -</span><!----><!----><!----><div role="button" aria-haspopup="listbox" class="p-dropdown-trigger ng-tns-c66-2"><span class="p-dropdown-trigger-icon ng-tns-c66-2 pwc-icon icon-caret-down"></span></div><!----></div>
<div class="p-hidden-accessible ng-tns-c66-2"><input type="text" readonly="" aria-haspopup="listbox" class="ng-tns-c66-2" id="siteId"></div>
<input type="text" readonly="" aria-haspopup="listbox" class="ng-tns-c66-2" id="siteId">
I have no idea how else to do this, I only knew how to work on dropdown with "id", not sure if i can do it with html or "class".
Or i should put a waiting time. Please help !
Your issue is in this line selenium.webdriver.support.ui as Select
You assign Select alias to the module import selenium.webdriver.support.ui. This is not Select class so you cannot deal with this alias as you would do with Select class.
I would suggest to have such import (to avoid confusions):
import selenium.webdriver.support.ui as uiClasses
and then use:
select = uiClasses.Select(driver.find_element_by_id('siteId'))
If you pay attention to the HTML the you've shared, Select wouldn't work.
Pro tip :
Select will only work if you have a HTML with Select and options tag.
Solution
Try to click directing with Explicit waits.
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.ID, "siteId"))).click()
time.sleep(3)
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//*[text()='Entity A']"))).click()
Imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
I am trying to write a Python program that uses Selenium to click a button to go to the next page if the button is clickable. This is because I am web scraping from varying amounts of pages.
I have tried to use a while loop that checks the href attribute, but the code doesn't click the button, nor does it return an error. If I simply write button.click(), but without a while loop or conditional check for the href attribute, then the program clicks the button correctly.
My code also has a while loop condition of "variable is not None". Is this a valid usage of "is not"? My logic is for the program to click the button to go to the next page if there is an href available from the to click.
Code:
import requests
from bs4 import BeautifulSoup
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
import time
import numpy as np
import pandas as pd
PATH = "C:\Program Files (x86)\chromedriver.exe"
wd = webdriver.Chrome(PATH)
wd.get("https://profiles.ucr.edu/app/home/search;name=;org=Physics%20and%20Astronomy;title=;phone=;affiliation=Faculty")
time.sleep(1)
button = wd.find_element_by_xpath("""//a[#aria-label='Next page']""")
#<a tabindex="0" aria-label="Next page" class="ng-star-inserted" style=""> Next <span class="show-for-sr">page</span></a>
href_data = button.get_attribute('href')
while (href_data is not None):
time.sleep(0.5)
button.click()
href_data = button.get_attribute('href')
Would anyone here be willing to assist me with this? I understand that Selenium requires the user to download a webdriver, so I apologize for any difficulties with testing my code.
Thank you, ExactPlace441
To loop until all pages were clicked.
wd.get('https://profiles.ucr.edu/app/home/search;name=;org=Physics%20and%20Astronomy;title=;phone=;affiliation=Faculty')
wait=WebDriverWait(wd, 10)
while True:
try:
wait.until(EC.element_to_be_clickable((By.XPATH, "//a[#aria-label='Next page']"))).click()
time.sleep(5)
except:
break
Import
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
I faced the same problem then I used gecko driver(selenium Firefox) instead of Chrome. My code was working perfectly in selenium Firefox but same code was not working in selenium Chrome. Without while loop I hadn't any problem to click on button in selenium Chrome browser but it was not working when added while loop. After using gecko driver(selenium Firefox) my problem was solved. Here is an example of while loop that you can use. It will clicking on button until the button disappeared or reach the last page.
i = 1
try:
while i < 2:
button_element = driver.find_element_by_xpath("give your button xpath")
button_element.click() #Our loop will continuing until our button xpath disappeared from web page
except:
pass #when the button xpath will disappeared it will ignore the error and jump to the next section of our code.
Here I modified your code:
import requests
from bs4 import BeautifulSoup
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException
import time
import numpy as np
import pandas as pd
driver = webdriver.Firefox()
driver.maximize_window()
url = "https://profiles.ucr.edu/app/home/search;name=;org=Physics%20and%20Astronomy;title=;phone=;affiliation=Faculty"
driver.get(url)
timeout = 20
# This container collect data from first page
containers = WebDriverWait(driver, timeout).until(EC.visibility_of_all_elements_located((By.XPATH,'//div[#class="column ng-star-inserted"]' )))
for container in containers:
name = container.find_element_by_css_selector('.header-details h5') #we are srcaping name from each page
print(name.text)
i = 1
try:
while i < 2: #Now it will look for “next page button” in every page and continuing click on “next page button” until it will reach the last page.
next_page_button = driver.find_element_by_xpath("//li[#class='pagination-next ng-star-inserted']")
next_page_button.click()
#our this container2 start collect data from second page to last page
containers = WebDriverWait(driver, timeout).until(EC.visibility_of_all_elements_located((By.XPATH,'//div[#class="column ng-star-inserted"]' )))
for container in containers:
name = container.find_element_by_css_selector('.header-details h5') #we are srcaping name from each page
print(name.text)
time.sleep(3)
except:
pass #if any page don't have “next page button” then our code will be end without any error.
I'd like to click the button 'Annual' at a page that is by default set on 'Quarterly'. There are two links that are basically called the same, except that one has data-ptype="Annual" so I tryed to copy the xpath to click the button (also tried other options but none did work).
However, I get the AttributeError: 'list' object has no attribute 'click'. I read a lot of similar posts, but wasn't able to fix my problem.. so I assume that javascript event must be called/clicked/performed somehow differnt.. idk Im stuck
from selenium import webdriver
link = 'https://www.investing.com/equities/apple-computer-inc-balance-sheet'
driver = webdriver.Firefox()
driver.get(link)
elm = driver.find_elements_by_xpath("/html/body/div[5]/section/div[8]/div[1]/a[1]").click()
The html is the following:
<a class="newBtn toggleButton LightGray" href="javascript:void(0);" data-type="rf-type-button" data-ptype="Annual" data-pid="6408" data-rtype="BAL">..</a>
you need to use find_element_by_xpath not find_elements_by_xpath that return a list
driver.find_element_by_xpath("/html/body/div[5]/section/div[8]/div[1]/a[1]").click()
Also i think is better to use Waits for example.
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.firefox.options import Options
options = Options()
options.add_argument("--window-size=1920,1080")
driver = webdriver.Firefox(firefox_options=options)
path = "/html/body/div[5]/section/div[8]/div[1]/a[1]"
try:
element = WebDriverWait(driver, 5).until(
EC.element_to_be_clickable((By.XPATH, path)))
element.click()
finally:
driver.quit()
I would still suggest you to go with linkText over XPATH. Reason this xpath : /html/body/div[5]/section/div[8]/div[1]/a[1] is quite absolute and can be failed if there is one more div added or removed from HTML. Whereas chances of changing the link Text is very minimal.
So, Instead of this code :
elm = driver.find_elements_by_xpath("/html/body/div[5]/section/div[8]/div[1]/a[1]").click()
try this code :
annual_link = driver.find_element_by_link_text('Annual')
annual_link.click()
and yes #Druta is right, use find_element for one web element and find_elements for list of web element. and it is always good to have explicit wait.
Create instance of explicit wait like this :
wait = WebDriverWait(driver,20)
and use the wait reference like this :
wait.until(EC.elementToBeClickable(By.LINK_TEXT, 'Annual'))
UPDATE:
from selenium import webdriver
link = 'https://www.investing.com/equities/apple-computer-inc-balance-sheet'
driver = webdriver.Firefox()
driver.maximize_window()
wait = WebDriverWait(driver,40)
driver.get(link)
driver.execute_script("window.scrollTo(0, 200)")
wait.until(EC.element_to_be_clickable((By.LINK_TEXT, 'Annual')))
annual_link = driver.find_element_by_link_text('Annual')
annual_link.click()
print(annual_link.text)
make sure to import these :
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
As per the documentation find_elements_by_xpath(xpath) returns a List with elements if any was found or else an empty list if not. Python's List have no click() method associated with it. Instead find_element_by_xpath(xpath) method have the click() method associated with it. So you have to use find_element_by_xpath(xpath) method inducing a waiter through WebDriverWait inconjunction with expected_conditions set as element_to_be_clickable(locator) as follows:
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//a[#class='newBtn toggleButton LightGray' and #data-type='rf-type-button']"))).click()
Note : You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
Notice that find_elements_by_xpath is plural it returns a list of elements. Not just one. The list can contain none, exactly one, or more elements.
You can for example click the first match with:
driver.find_elements_by_xpath("/html/body/div[5]/section/div[8]/div[1]/a[1]")[0].click()
or iterate through the list and click all these elements, or you can use the find_element_by_xpath (which returns a single element, if it can be found):
driver.find_element_by_xpath("/html/body/div[5]/section/div[8]/div[1]/a[1]").click()
For me, it was not working, and tried a hell lot of tricks, and none worked. Some people recommended driver.implicitly_wait(10) instead of time.sleep(10) which didn't work. so please try giving time.sleep(10) both above and below the .click() code line, and check if it works or not.