python webscraping with selenium: trouble locating modal element - selenium

I want to go to http://ted.com/talks, click the "See All Topics" in the "Topics" dropdown, and then click the a random letter heading, like "C" or "D-E". However, I don't know how to find the element in the modal popup for that specific letter heading.
This is what the letter heading elements look like when I click "Inspect Element":
<li class="topic-select__range">
<a class = "topic-select__range__link" href="#" data-index="0">A-B</a>
</li>
<li class="topic-select__range">
<a class = "topic-select__range__link" href="#" data-index="1">C</a>
</li>
<li class="topic-select__range">
<a class = "topic-select__range__link" href="#" data-index="2">D-E</a>
</li>
... etc.
My program can get all the way to See All Topics just fine but gets Cannot Locate Element Error when I try to click on the letter headers. This is what my code snippet looks like so far:
# Each header is assigned a number. A header is picked at random, and clicked.
# [ERROR: Cannot locate element.]
random_letter = random.randint(0, 8)
topics_window = browser.find_element_by_class_name("select-modal__content topic-select") # error here, when program tries to find modal popup or tags within it
letter_headers = topics_window.find_element_by_class_name("topic-select__range__link")
letter_headers[random_letter].click()

I couldn't find a way to access modal JavaScript content with selenium. I have instead found a way to do so using web API.

Related

Right way of using asp-page and asp-route in asp.net core razor pages between _layout and index page

Below is the file structure
I have default Index page under Pages folder, which has layout as _LayoutHarman.cshmtl
Code : Pages/Shared/_LayoutHarman.chtml
header menu : pages are in subfolder.ie category folder in this case
<a asp-route-cat_slug="electronic" asp-page="./category/Index">Electronic</a>
<a asp-route-cat_slug="beauty-care" asp-page="./category/index" >Beauty Care</a>
<a asp-route-cat_slug="toy" asp-page="./category/index" >Toy</a>
footer menu : pages are on root folder
<a asp-page="./Contact" >Contact</a>
<a asp-page="./terms" >Terms</a>
<a asp-page="./privacy" >Privacy</a>
Code : Pages/category/Index.cshtml
#page "{cat_slug?}/{pageIndex:int?}"
#model bList.Pages.category.IndexModel
#{
}
<nav aria-label="Page navigation example">
<ul class="pagination text-center">
#if(Model.prev_no!=0){
<li><a asp-page="category" asp-route-cat_slug="#Model.cat_url" asp-route-pageIndex="#Model.prev_no"> Previous</a></li>
}
#if (Model.no_more_record != "YES")
{
<li><a asp-page="category" asp-route-cat_slug="#Model.cat_url" asp-route-pageIndex="#Model.next_no">Next</a></li>
}
</ul>
</nav>
Here Next/ previous button generate url as follow
https://localhost:8080/category/toy/1
https://localhost:8080/category/toy/2
https://localhost:8080/category/toy/3
respective on selected category
Issue : When i visit on category page and click on prev or next button and then try to click on link Contact,Terms,Privacy i.e (which is on _LayoutHarman.cshtml) or on header menu then href become blank.
Edited One:
Code: On _LayoutHarman.cshtml
HeaderMenu:
<a href="./category/toy" >toy</a>
Footer Menu
<a asp-page="./Contact" >Contact</a>
<a asp-page="./terms" >Terms</a>
Code: On Category/Index.html page
Prev</li>
Next
Now on Next/ Prev button click , header menu generates url as
https://localhost:44382/category/toy/category/toy hence error page occurs. but for footer menu contact/term/privacy works properly
I did reproduce your problem, you need to delete the point(.) in asp-page attribute in your _LayoutHarman.cshmtl.
Changed like this:
<a asp-route-cat_slug="electronic" asp-page="/category/Index">Electronic</a>
<a asp-route-cat_slug="beauty-care" asp-page="/category/index" >Beauty Care</a>
<a asp-route-cat_slug="toy" asp-page="/category/index" >Toy</a>
<a asp-page="/Contact" >Contact</a>
<a asp-page="/terms" >Terms</a>
<a asp-page="/privacy" >Privacy</a>
And you don't need to add asp-page="category" in Pages/category/Index.cshtml, just delete this attribute in this page.
Here is my test result:
Do you really need all this extra server side processing, asp-route-cat_slug asp-page?
Can't you just use plain old fashion HTML hyperlink? As for the dynamic parameters maybe you can set only them...
View category
Just an idea to think on...
Cheers

Puppeteer (1.14.0) unable to find anchor tag by "id" although JQuery can find it

Node v10.15.3 (64-bit)
Puppeteer v1.14.0
Chromium v 75.0.3738.0 (Developer Build) (64-bit)
I am at the very last page of a site automation tool and the last button I need to press (which is actually an anchor tag with an "id" attribute) cannot be found by Puppeteer. The code is:
await page2.click('#____bs');
The error I receive is:
Error: Node is either not visible or not an HTMLElement
The HTML code is here:
<a role="button" ct="Button" rel="tooltip" title="Post"
aria-label="Please Select Post" data-toggle="cso-button" id="____bq"
class="cso-btn cso-btn cso-action cso-corner cso-btn-primary "
href="javascript:void(0)"
data-ctl-options="{'actiontype';:'ACTION','value':'Click'}"
style="display: inline-block;">Post</a>
While in debug mode with this page up, I can use JQuery to find the element easily:
$("#____bq").text()
"Post"
Therefore, why can't Puppeteer find it? Is there an alternate way I could try to click this link?
Could it also be that the control (the anchor) is outside the visible edge of the screen? I wouldn't think that would matter since I'm assuming Puppeteer is looking at the HTML and not the screen itself but doesn't hurt to ask.
Thanks in advance.
UPDATE: Here's the entire HTML that contains the anchor I'm trying to select:
<div class="row split">
<div class="clsFP-expand clsBG-expand cso-hidden cso-cont-50r cso-pad5r">
<span class="cso-pad5-ie">
<a role="button" ct="Button" rel="tooltip" title="Cancel" aria-label="Please Select Cancel" data-toggle="cso-button" id="____bo" class="cso-btn cso-btn cso-action cso-corner cso-btn-grey " href="javascript:void(0)" data-ctl-options="{"actiontype":"ACTION","value":"Click"}" >Cancel</a>
</span>
<span class="cso-pad5-ie">
<a role="button" ct="Button" rel="tooltip" title="Post" aria-label="Please Select Post" data-toggle="cso-button" id="____bs" class="cso-btn cso-btn cso-action cso-corner cso-btn-primary inactive " href="javascript:void(0)" data-ctl-options="" >Post</a>
<a role="button" ct="Button" rel="tooltip" title="Post" aria-label="Please Select Post" data-toggle="cso-button" id="____bq" class="cso-btn cso-btn cso-action cso-corner cso-btn-primary " href="javascript:void(0)" data-ctl-options="{"actiontype":"ACTION","value":"Click"}" >Post</a>
</span>
</div>
</div>
It's not that Puppeteer is not being able to get the element. That's a validation coming from Puppeteer (see https://github.com/GoogleChrome/puppeteer/blob/5ee21d97e796263857b9f3fbeaf63366179d346f/lib/JSHandle.js#L203).
According to the doc:
This method fetches an element with selector, scrolls it into view if needed, and then uses page.mouse to click in the center of the element. If there's no element matching selector, the method throws an error.
That means that Chromium should be able to scroll to that element, and that element should be visible in order to be able to click on it.

trouble making in xpath for "ul" html code in selenium webdriver

HTML code:
<div id="routingPanel" class="">
<div id="routingPanelRight">
<ul id="routingList" class="ui-sortable">
<li class="ui-menu-item ui-draggable" style="display: list-item;" role="presentation" data-type="srl" data-id="15">
<a class="ui-corner-all" tabindex="-1">AS-HTTS-US-LAN-SW</a>
<span class="fa fa-trash"/>
<span class="type">[srl]</span>
</li>
<li class="ui-menu-item ui-draggable" style="display: list-item;" role="presentation" data-type="queue" data-id="119">
<a class="ui-corner-all" tabindex="-1">AS-EMEA-NORTH</a>
<span class="fa fa-trash"/>
<span class="type">[queue]</span>
</li></ul></div></div>
I need to click on a button which is having the span class"fa fa-trash" but it is inside li class. And i have list on buttons on the page with li class changing.
I am giving testdata from excel file so i can't use the direct value.
i tried to use this xpath
.//*[#id='routingList']/li[5]/span[1] //testdata1
.//*[#id='routingList']/li[2]/span[1] //testdata2
where li value changes everytime from excel file.
WebDriverWait wait = new WebDriverWait(driver, 15);
wait.until(ExpectedConditions.visibilityOfElementLocated((By.xpath("//ul[#id='routingList']/li/span[1]")))).click();
List<WebElement> options = driver.findElements(By.xpath("//ul[#id='routingList']/li/span[1]"));
for (WebElement option : options) {
if(testData.equals(option.getText()))
option.click();
Tried above code but it is deleting only one from the list ,where i have passed two more testdata that needs to be deleted.
Need suggestions Please
According to the information you gave me in comments, I think the problem is that you are trying to get a text from an element that doesn't contain text.
Let's say your testData is AS-HTTS-US-LAN-SW. In the HTML you provided and the xpath you mentioned, you are selecting an autoclosing tag <span class="fa fa-trash"/>. Once this tag is selected, you are trying to get the text inside of it, and there is none.
<ul id="routingList" class="ui-sortable">
<li class="ui-menu-item ui-draggable" style="display: list-item;" role="presentation" data-type="srl" data-id="15">
===========================
<a class="ui-corner-all" tabindex="-1">AS-HTTS-US-LAN-SW</a> ----> The text is contained here
<span class="fa fa-trash"/> ---> No text in that tag
===========================
<span class="type">[srl]</span>
</li>
</ul>
So, basically, you have to modify a little bit your xpath from : //ul[#id='routingList']/li/span[1] to : //ul[#id='routingList']/li/a to get the text, and then go back to the parent node to find your button with : ../span[contains(#class, 'fa fa-trash')]
WebDriverWait wait = new WebDriverWait(driver, 15);
wait.until(ExpectedConditions.visibilityOfElementLocated((By.xpath("//ul[#id='routingList']/li/span[1]")))) // removed the click here because you were clicking on the first element of the list
List<WebElement> options = driver.findElements(By.xpath("//ul[#id='routingList']/li/a"));
for (WebElement option : options) {
if(testData.equals(option.getText()))
option.findElement(By.xpath("../span[contains(#class, 'fa fa-trash')]")).click();
Tell me if it helped
I know you already accepted an answer but there's a more efficient way to do this. You can specify the text you are looking for as part of the XPath. So, you do a single search instead of looping through all the options which can be a performance hit if there are many options. Also, with something like this you are likely to use it more than once so put it in a function.
In this case, the function would take in the string you are looking for and then click the appropriate element.
public void selectRegion(String regionName)
{
driver.findElement(By.xpath("//a[.='" + regionName + "']/following-sibling::span[#class='fa fa-trash']")).click();
}
and you would call it like
selectRegion(testData);
The function looks for an A tag that contains the desired text and then clicks the sibling SPAN with class fa fa-trash.

Selenium: Click on element in div container doesn't work

Fyi: I'm using the selenium package for R, the selection code is equal to javascript or python so I'm asking a general selenium question.
I have a container which I have to make visible by a click, this works.
Then I try to select an element inside this container, I think I find it correctly but the click on the element only makes the popup window disappear.
Example code:
<div class="dateRanges" style="top: 275.313px; display: block;">
<a class="top dateOption CUSTOM" id="id32" href="javascript:;">
Benutzerdefinierter Zeitraum
</a>
<a class="dateOption TODAY" id="id33" href="javascript:;">
Heute
</a>
...
</div>
Element I try to find is "top dateOption CUSTOM".
My different tries which all failed:
remDr$findElement(using = 'xpath','//a[contains(#class,"top")]')$clickElement()
remDr$findElement(value = '//a[#class = "top dateOption CUSTOM"]')$clickElement()
remDr$findElements(using = 'css selector','a[class="top dateOption CUSTOM"]')[[1]]$clickElement()
I don't get any error message, the click seems to happen as the popup disappears, but the effect of the button does not.
I tried to save the object, wait a few seconds and click afterwards with no different effect.
I also tried different approaches with "idc4" with the same outcome.
Would appreciate any help, thank you.

VB.NET ~ Need to know how to click on this href

I have this link (not a button) that I want to click on ...I don't want to navigate to the link via webbrowser1.navigate("") because the link changes but the href and inner text is always the same.
Here is the HTML of the link
Should this be put into the webbrowser or behind a button
<ul id="nav">
<li>
<li>
<li>
Submit a New Story
</li>
<li>
</ul>
Thanks guys.
I'm not sure what is the result you're seeking for, but if you want to trigger a click as if the client would have clicked it, you could achieve it by obtaining the WebBrowser.Documentproperty which returns a HtmlDocument class, you can retrieve that a tag then trigger a click on it.
The All property should be a good start:
var aTag = d.All.Where(a => a.TagName == "a");