Webscraping selenium looop - selenium

I am still handling with a view problems. I am finally able to load the whole page with a view window.scrollBy(..) commands...
The problem I am facing now is that I would like to filter all headlines out of a text. The text I would like to screen is shown with this command:
[Code]
main = driver.find_element_by_id("mrt-node-quoteNewsStream-0-Stream")
print(main.text)
That works well and all the result is shown. Within these results I am now wanna filter, as mentioned above all headlines. This should work wit the following code:
articles = main.find_elements_by_tag_name("mrt-node-quoteNewsStream-0-Stream") # li
for mrt-node-quoteNewsStream-0-Stream in articles:
header = article.find_element_by_class_name("M(0)")
print(header.text)
Unfortunately it shows me the following syntax error message:
File "", line 7
for mrt-node-quoteNewsStream-0-Stream in articles:
^
SyntaxError: can't assign to operator
Line 7 is the one with the following one:
for mrt-node-quoteNewsStream-0-Stream in articles:
Any help is highly appreciated. Thanks

This line of code returns a list of elements:
articles = main.find_elements_by_tag_name("mrt-node-quoteNewsStream-0-Stream")
So articles is a list of WebElement and each of them looks like:
<selenium.webdriver.remote.webelement.WebElement (session="04a9fac269c3a9cb724cc72769aed4e0", element="1b8ee8d0-b26a-4c67-be10-615286a4d427")>
As per your second set of code trials, mrt-node-quoteNewsStream-0-Stream is the tag_name and you must not look for the tag_name within the element.

Related

How can I get the link of all the posts in the Instagram profile with Selenium?

I'm trying to get the links of all the posts in an instagram profile.
How can I get to the href="/p/CX067tNhZ8i/" in the photo.
What I'm trying to do is find the href= blabla of all posts.
All your posts are in class="v1Nh3 kIKUG _bz0w".
I tried to get the hraf= blabla value from this class with the get_attribute command, but it didn't work.
Thank you for your help.
browser.get("https://www.instagram.com/lightning.mcqueen34/")
links = []
elements = browser.find_element_by_xpath('//*[#id="react-root"]/div/div/section/main/div/div[4]/article/div[1]/div/div[1]/div[3]')
for i in elements:
links.append(i.get_attribute('href'))
I thought this would work but the elements value is not a list . It gave an error.
This should work:elements = browser.find_elements_by_tag_name('a')
Below answer will not work in all cases, dependant on how the DOM of the page is loaded.
Replace this line:
elements = browser.find_element_by_xpath('//*[#id="react-root"]/div/div/section/main/div/div[4]/article/div[1]/div/div[1]/div[3]')
With:
elements = browser.find_element_by_xpath("//a[#href]")
This will let you retreive all links with a href from the page.
Try to change XPath first to get the DIV class or ID after trying this //a[#href] Xpath to get all HREF.

Selenium finding elements returns incorrect elements

I'm using Selenium to try and get some elements on a web page but I'm having trouble getting the ones I want. I'm getting some, but they're not the ones I want.
So what I have on my page are five divs that look like this:
<div class="membershipDetails">
Inside each one is something like this:
<div class="membershipDetail">
<h3>
VIP Membership
</h3>
</div>
They DO all have this same link, but they don't have the same text ('VIP Membership' would be replaced by something else)
So the first thing was to get all the divs above in a list. This is the line I use:
listElementsMembership = driver.find_elements_by_css_selector(div[class^='membershipDetail'])
This gives me five elements, just as I would expect. I checked the 'class' attribute name and they are what I would expect. At this point I should say that they aren't all EXACTLY the same name 'membershipDetail'. Some have variations. But I can see that I have all five.
The next thing is to go through these elements and try and get that element which contains the href ('VIP Membership').
So I did that like this:
for elem in listElementsMembership:
elemDetailsLink = elem.find_element_by_xpath('//a[contains(#href,"EditMembership")]')
Now this does return something, but it always got me the element from the FIRST of the five elements. It's as if the 'elem.find_element_by_xpath' line is going up a level first before finding these hrefs. I kind of confirmed this by switching this to a 'find_elements_by_xpath' (plural) and getting, you guessed it, five elements.
So is this line:
elemDetailsLink = elem.find_element_by_xpath('//a[contains(#href,"EditMembership")]')
going up a level before getting its results? If it is, now can I make it not do that and just restrict itself to the children?
If you are trying to find element with in an element use a . in the xpath like below:
listElementsMembership = driver.find_elements_by_css_selector(div[class^='membershipDetail'])
for elem in listElementsMembership:
elemDetailsLink = elem.find_element_by_xpath('.//a') # Finds the "a" tag with respect to "elem"
Suppose if you are looking for VIP Membership:
listElementsMembership = driver.find_elements_by_css_selector(div[class^='membershipDetail'])
for elem in listElementsMembership:
value = elem.find_element_by_xpath('.//a').get_attribute("innerText")
if "VIP Membership" in value:
print(elem.find_element_by_xpath('.//a').get_attribute("innerText"))
And if you dont want iterate over all the five elements try to use xpath like below: (As per the HTML you have shared)
//div[#class='membershipDetail']//a[text()='VIP Membership']
Or
//div[#class='membershipDetail']//a[contains(text(),'VIP Membership')]
You've few mistake in that css selector.
Quotes are missing.
^ is for starts-with, not sure if you really need that. In case it's partial matching please use * instead of ^
Also, I do not see any logic for the below statement in your code attempt.
The next thing is to go through these elements and try and get that
element which contains the href ('VIP Membership').
Code :
listElementsMembership = driver.find_elements_by_css_selector("div[class*='membershipDetail']")
for ele in listElementsMembership:
e = ele.find_element(By.XPATH, ".//descendant::a")
if "VIP Membership" in e.get_attribute('href'):
print(e.text, e.get_attribute('href'))
You can give an index using a square bracket like this.
elemDetailsLink = elem.find_element_by_xpath('(//a[contains(#href,"EditMembership")])[1]')
If you are trying to get an element using XPath, the index should start with 1, not 0.

Unable to access the element in the frame

I am still dipping my toes in selenium , I am SAP guy working on Selenium automation POC. my requirement is to click on the drop down and select a value from the droplist.
I have extensively looked at the previous posts but could not find any answers. I have tried all the suggestions from the post but nothing seems to be working for me.
Please can you help me how to access this drop down value.
HTML code is attached in the pic along with the element that I am trying to access
My selenium code :
driver.switchTo().frame(driver.findElement(By.name("SMFrame")));
System.out.println("TExt" + driver.findElement(By.xpath("//div[#class='file-type']")).getText());
Error:
error -- > no such element: Unable to locate element: {"method":"xpath","selector":"//div[#class='file-type']"}
Image
Assuming you are correctly getting into the iframe
Right now you are selecting a div above the actual dropdown element (the select-dropdown tag). You may want to try changing your xpath to search for the tag like so:
//div[#class='file-dropdown']/select-dropdown
Also you should be creating a dropdown object like this. Then calling a method on it to select the dropdown option you are looking for:
Select dropdown = new Select(driver.findElement(By.xpath("//div[#class='file-dropdown']/select-dropdown");
System.out.println(dropdown.selectByVisibleText("text that you are looking for in the options"));
This is a good explanation of how to do it: https://www.javatpoint.com/selenium-webdriver-handling-drop-downs

i am not able find dynamic element in android app

i have dynamic elements in my android app and there is nothing to find it out .Please check screen shot of my app and attributes and let me how can we do it.
I have tried with the given attributes
I just want ti find these elements and send keys
Try find element by xpath with this value:
//*[contains(#class,'android.widget.EditText')]
It will return the first element if there are many elements with same class.
If you want other element with class EditText, you can add count like this:
(//*[contains(#class,'android.widget.EditText')])[2]
The above for second element.
I tried this way of finding element:
List name = driver.findElements(By.xpath("android.widget.EditText"));
name.get(0).click();
But i am getting bellow error:-
java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
Please let me know what is the issue?
I found one of my dynamic element through this method :-
findElementByXPath("//android.widget.FrameLayout[#index='0']/android.widget.EditText[#index='0']")
but for second element also , these details are same, so if i am putting same method to find second element , it search the first one only.
Than i printed size of the element with this find method
Size is coming like this
(996, 89)
Please let me know how can i find second element , because it has same details.

I.click()-selector in CodeceptJS - how to find first button with specific innerHTML

I have various buttons and several buttons with the same name "Start". I need to click on the first found button with this name (innerHTML).
With jQuery this works with :
$('button:contains(Start):first').click()
How does it work with I.click()-Selector in CodeceptJS? I can't find the right syntax and always getting:
"invalid selector: An invalid or illegal selector was specified"
Here is the API for this function: https://github.com/Codeception/CodeceptJS/blob/master/docs/webapi/click.mustache
The only working solution I found is:
I.click('//button[1]');
But this solution is confusing, because you need to know the exactly number in the order of this element - and I have a lot of buttons with different names. Also this not allows me to search by innerHTML such as "Start".
You could use the I.executeScript like this:
I.executeScript("var elements = document.getElementsByName('Start');elements[0].click();"); or
I.executeScript("var elements =
document.querySelector(\"button[name*='Start']\");elements[0].click();");
You need using XPath for that
//button[1][contains(text(), 'Start')]
locate("//button[contains(text(), 'Start')]").first()
or
locate("//button[contains(text(), 'Start')]").at(1)
Works fine.