Trying to get the download link from this website to download the CSV file straight to a pandas dataframe with read_csv, however, having trouble scraping the link (from the href code).
To get to the csv download button, an initial click on the "Download Data" button is needed to open up another window.
The python code I have put together so far is:
driver = webdriver.Chrome(executable_path=r'/usr/local/bin/chromedriver')
URL = 'https://statisticsnz.shinyapps.io/covid_19_dashboard/'
driver.get(URL)
try:
element = WebDriverWait(driver, 15).until(
EC.presence_of_element_located((By.XPATH, '//*[#id="download_data-show"]'))
)
element.click()
element = WebDriverWait(driver, 15).until(
EC.presence_of_element_located((By.XPATH, '//*[#id="download_data-downloadData"]'))
)
outer = element.get_attribute('outerHTML')
finally:
driver.quit()
link = outer.find('a',href=re.compile('download_data-downloadData')).get('href') #I don't think this code line will work, but essentially trying to find the link to download with
df = pd.read_csv(URL + link)
The outer object returns the following html code and is missing the link that I need (href field is empty) to use directly in the read_csv file:
In[]: print(outer)
Out[]: <a id="download_data-downloadData" class="btn btn-default shiny-download-link shiny-bound-output" href="" target="_blank" download="">
<i class="fa fa-download"></i>
Download
</a>
When I inspect the 'Download' button from chrome, it gives me the following html code with the link in the href field:
<a id="download_data-downloadData" class="btn btn-default shiny-download-link shiny-bound-output" href="session/c199b49c2d09d042041ee7682c9c5922/download/download_data-downloadData?w=6e59d758" target="_blank" download="">
<i class="fa fa-download"></i>
Download
</a>
Any ideas why the link isn't showing when I scrape through Selenium, or maybe there is a better way of doing this?
Try to get the link directly of your download element like this:
try:
element = WebDriverWait(driver, 15).until(
EC.presence_of_element_located((By.XPATH, '//*[#id="download_data-show"]'))
)
element.click()
element = WebDriverWait(driver, 15).until(
EC.presence_of_element_located((By.XPATH, '//*[#id="download_data-downloadData"]'))
)
link = element.get_attribute('href')
finally:
driver.quit()
df = pd.read_csv(link)
Related
I have a "clear all" button which is an anchor. The HTML structure is like this:
<div class="form-row toggle-closed" id="ProductFilters" style="display: block;">
<div class="form-row__filter">
<ul class="form-row__filter__bg-display">
<li class="filter__group__item__small">
<a id="ProductFiltersFilterText" class="f-right" data-select-all="ProductFilters" href="#">clear all</a>
</li>
</ul>
</div>
</div>
Then in the Selenium test I'm trying to find the a tag using this:
SeleniumHelper.ExpandFilterSection(_webDriver, "#ProductFilters");
var clearAllButton = _webDriver.FindElement(By.CssSelector("div.form-row__filter>ul>li>#ProductFiltersFilterText"));
clearAllButton.Click();
THen I started debugging, in the automated Chrome window I could see that by executing ExpandFilterSection, the filter was expanded, the "clear all" button was exposed, then a bug said:
OpenQA.Selenium.ElementNotVisibleException: 'element not visible'
How ever in the Autos I see:
It seems that the "clear all" button is found, why does it say "element not visible"? Functionality of the button is trigger by JavaScript though.
To click() on the element with text as clear all you have to induce WebDriverWait for the desired ElementToBeClickable() and you can use either of the following Locator Strategies:
linkText:
SeleniumHelper.ExpandFilterSection(_webDriver, "#ProductFilters");
new WebDriverWait(driver, TimeSpan.FromSeconds(20)).Until(ExpectedConditions.ElementToBeClickable(By.LinkText("clear all"))).Click();
cssSelector:
SeleniumHelper.ExpandFilterSection(_webDriver, "#ProductFilters");
new WebDriverWait(driver, TimeSpan.FromSeconds(20)).Until(ExpectedConditions.ElementToBeClickable(By.CssSelector("div#ProductFilters>div.form-row__filter>ul.form-row__filter__bg-display>li.filter__group__item__small>a#ProductFiltersFilterText"))).Click();
xpath:
SeleniumHelper.ExpandFilterSection(_webDriver, "#ProductFilters");
new WebDriverWait(driver, TimeSpan.FromSeconds(20)).Until(ExpectedConditions.ElementToBeClickable(By.XPath("//div[#id='ProductFilters']/div[#class='form-row__filter']/ul[#class='form-row__filter__bg-display']/li[#class='filter__group__item__small']/a[#id='ProductFiltersFilterText']"))).Click();
I want to click on group add button of Facebook, but I have not found it possible using an xpath.
Here is the html for the button:
<li>
<a
class="_42ft _4jy0 _3-8i _4jy3 _517h _51sy"
role="button"
href="#"
ajaxify="/ajax/groups/members/add_get.php?group_id=1168192579894018&refresh=1"
rel="dialog">
<i class="_3-8_ img sp__lkuGKPb9f- sx_e8790e"></i>
Add
</a>
</li>
This is how I have tried to click on it:
JavascriptExecutor jse1 = (JavascriptExecutor)driver;
jse1.executeScript("document.getElementById('gbqfb').click();");
You should try using xpath as below :-
driver.findElement(By.xpath(".//a[contains(.,'Add')]")).click();
Or
driver.findElement(By.xpath(".//a[normalize-space() = 'Add']")).click();
I don't see the id that you are using.
You can try with the following css selector:
a[href*='groups/members/add'][href*='1168192579894018']
The first href assure that you are clicking the add for the group,
the second href assures that you are clicking add for the right group
I don't know how the page is looking, if you have only one add then you can remove the second href.
If if you can get the selector with css then for sure you can get it with xpath.
I recommend using css.
I'm trying to select below element in a webpage.
.active > b:nth-child(1)
This is my code :
timbro = browser.find_element_by_css_selector('.active > b:nth-child(1)')
hover = ActionChains(browser).move_to_element(timbro)
hover.perform()
This is part of the error I'm getting:
Message: Unable to locate element:
{"method":"css selector","selector":".active > b:nth-child(1)"}"
what I'm trying to do is activate a drop down menu so that I can click on another link.
HTML Snippet:
<li>
<b>Menu iniziale</b>
<ul style="display: block;" class="sub-links">
<li>
Anagrafica
</li>
<li>
Fine sessione
</li>
<li>
Home
</li>
<li>
Timbro
</li>
</ul>
</li>
You should try using WebDriverWait to wait until presence of element as below :-
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
wait = WebDriverWait(browser, 5)
menuIniziale = wait.until(EC.presence_of_element_located((By.LINK_TEXT, "Menu iniziale")))
hover = ActionChains(browser).move_to_element(menuIniziale).move_to_element(wait.until(EC.presence_of_element_located((By.LINK_TEXT, "Timbro"))))
hover.click().build().perform()
Edited1 :- If unfortunately LINK_TEXT does not work try using XPATH as below :-
menuIniziale = wait.until(EC.presence_of_element_located((By.XPATH, ".//a[contains(.,'Menu iniziale')]")))
hover = ActionChains(browser).move_to_element(menuIniziale).move_to_element(wait.until(EC.presence_of_element_located((By.XPATH, ".//a[contains(.,'Timbro')]"))))
hover.click().build().perform()
Or try using CSS_SELECTOR as below :-
menuIniziale = wait.until(EC.presence_of_element_located((By.CSS_SELECTOR, "a.main-link")))
hover = ActionChains(browser).move_to_element(menuIniziale).move_to_element(wait.until(EC.presence_of_element_located((By.CSS_SELECTOR, "ul.sub-links > li:nth-child(4)"))))
hover.click().build().perform()
Edited2 : If this element is inside an iframe, you need to switch that iframe before finding element as below :-
wait.until(EC.frame_to_be_available_and_switch_to_it(("frame name or id")))
#Now after successfully switching to frame do any one of the above steps
Edited3 :- If you are now able to open menu but not able to select subMenu try as below :-
wait.until(EC.frame_to_be_available_and_switch_to_it(("frame name or id")))
menuIniziale = wait.until(EC.presence_of_element_located((By.CSS_SELECTOR, "a.main-link")))
hover = ActionChains(browser).move_to_element(menuIniziale)
hover.build().perform()
subMenu = wait.until(EC.visibility_of_element_located((By.XPATH, ".//a[contains(.,'Timbro')]")))
subMenu.click()
Edited4: Final code
menuIniziale = wait.until(EC.presence_of_element_located((By.XPATH, ".//a[contains(.,'Menu iniziale')]")))
hover = ActionChains(browser).move_to_element(menuIniziale)
hover.perform()
subMenu = wait.until(EC.visibility_of_element_located((By.XPATH, ".//a[contains(.,'Timbro')]")))
subMenu.click()
hover = ActionChains(browser).move_to_element(subMenu)
hover.click().perform()
I am trying to find the field name "SavePLButton". The Html code behind this page is:
<ul class="button-bar">
<li class="first">
<a href="#" id="SavePLButton" type="button" name="SavePLButton" value="Save" onclick="formSubmit('SAVEEXIT');">
<i class="icon-save"/>
Save
</a>
</li>
The C# code that I am using is:
var Submit = Driver.Instance.FindElement(By.CssSelector("i.icon-save"));
Submit.Click();
I have also tried:
var Submit = Driver.Instance.FindElement(By.Id("SavePLButton"));
Submit.Click();
It is unable to find the fieldname. Can someone please help. Thank you.
Try using WebDriverWait to wait until element visible as below :-
var wait = new WebDriverWait(driver, TimeSpan.FromSeconds(10));
var Submit = wait.Until(ExpectedConditions.ElementIsVisible(By.Id("SavePLButton")));
Submit.Click();
Note :- make sure before try this element is not inside any frame
Hope it helps.....:)
I want to go to http://ted.com/talks, click the "See All Topics" in the "Topics" dropdown, and then click the a random letter heading, like "C" or "D-E". However, I don't know how to find the element in the modal popup for that specific letter heading.
This is what the letter heading elements look like when I click "Inspect Element":
<li class="topic-select__range">
<a class = "topic-select__range__link" href="#" data-index="0">A-B</a>
</li>
<li class="topic-select__range">
<a class = "topic-select__range__link" href="#" data-index="1">C</a>
</li>
<li class="topic-select__range">
<a class = "topic-select__range__link" href="#" data-index="2">D-E</a>
</li>
... etc.
My program can get all the way to See All Topics just fine but gets Cannot Locate Element Error when I try to click on the letter headers. This is what my code snippet looks like so far:
# Each header is assigned a number. A header is picked at random, and clicked.
# [ERROR: Cannot locate element.]
random_letter = random.randint(0, 8)
topics_window = browser.find_element_by_class_name("select-modal__content topic-select") # error here, when program tries to find modal popup or tags within it
letter_headers = topics_window.find_element_by_class_name("topic-select__range__link")
letter_headers[random_letter].click()
I couldn't find a way to access modal JavaScript content with selenium. I have instead found a way to do so using web API.