Extracting information from same page popup in python Selenium webscraping - selenium

Note : I'm experienced in python however just starting out in selenium and webscraping. Please excuse if this is a bad question or if my fundamentals in selenium seem amiss. I could not find an answer in hours of searching hence i ask here
Goal: To extract the "About the Business" information found in Yelp pages of businesses
Some pages have their about the business information within a Read More button based popup (eg : https://www.yelp.com/biz/and-pizza-bethesda-bethesda)
Some pages do not have their business information in a Read More button based popup (eg : https://www.yelp.com/biz/pneuma-fashions-upper-marlboro-3 )
Problem: Unable to navigate to the About the Business popup that appears after clicking the Read More button and extract the text present in it.
Attempts as of now: From googling I had found explanations on how to handle alert popups or window popups. However the code doesnt work. The popup that emerges when clicking Read More button does not cause change in window_handles
import re
# getting all sections of the page
result=driver.find_elements_by_tag_name("section")
About = None
for sec in result:
if sec.text.startswith("About the Business"):
# this pertains only to the About the business section
main_page=driver.current_window_handle
print(main_page) # Returns the current handle
sec.find_element_by_tag_name("button").click()
popup=None
for handle in driver.window_handles: # is an iterable with only one handle
# The only handle present is the main_page handle
print(handle)
if handle!=main_page:
popup = handle
print(popup) # returns None
driver.switch_to.window(popup) # Throws error because popup=None
# THE FOLLOWING SECTION IS NOT EXECUTED BECAUSE OF THE ERROR ABOVE
#////////////////////////////////////////////////////
button_contents=driver.find_elements_by_tag_name("p")
for b in button_contents:
print(b.text) # intended to print text contents
close=driver.find_element_by_tag_name("button")
close.click()
driver.switch_to.window(main_page)
Please help
Thank you to everyone who reads this question and provides advice and answers

That is a custom pop-up so you won't need to switch to it. I suggest to study about getting relative xpath . Use loop to navigate to your urls and include below code
driver.get(your_URL)
readMoreBtnXpath= "//h4[text()='About the Business']/ancestor::section//button"
aboutTheBusinessSec = "//h4[text()='About the Business']/ancestor::section"
fromTheBusinessSec = "((//h2[text()='From the business']/parent::div/following-sibling::div//div)[5]/div)[last()]/preceding-sibling::div"
try:
driver.find_element(By.XPATH, readMoreBtnXpath).click()
button_contents = driver.find_elements(By.XPATH, fromTheBusinessSec)
for b in button_contents:
print(b.text)
except:
print(driver.find_element(By.XPATH, aboutTheBusinessSec).text)

One thing that u should know is that the pop-up is not displayed in a new window. It is instead displayed in the same page itself. Here is the complete code to extract the text from the pop-up:
from selenium import webdriver
driver = webdriver.Chrome()
driver.get('https://www.yelp.com/biz/and-pizza-bethesda-bethesda')
try:
driver.find_element_by_xpath('//*[#id="wrap"]/div[3]/div/div[4]/div/div/div[2]/div/div/div[1]/div/div[1]/section[5]/div[2]/button').click()
p1 = driver.find_element_by_xpath('//*[#id="modal-portal-container"]/div[2]/div/div/div/div[2]/div/div[2]/div/div[2]/div/div/div[1]/p').text
p2 = driver.find_element_by_xpath('//*[#id="modal-portal-container"]/div[2]/div/div/div/div[2]/div/div[2]/div/div[2]/div/div/div[2]/p[2]').text
print("Specialties --",p1)
print("History --",p2)
except:
print('Read more button not found')
Output:
Specialties -- Award-winning pizza: Named one of Fast Company's "World's Most Innovative Companies" in 2018, third-place in the Washington Post Express's of "Best Fast Casual" in 2018, third place in the Washington City Paper's "Best Gluten-Free Menu" in 2018 and won its "Best Pizza in D.C." in 2017, 11th on TripAdvisor's "Best Fast Casual Restaurants -- United States" in 2018.
History -- Since 2012, we've built pizza shops with an edge to their craft pies, beverages and shop design, created an environment where ALL of our Tribe can thrive, supported our local communities and now we'll text you back, if you want. Started with a pizza shop. Became a culture. That's &pizza.
Edit:
Since this doesn't work with this website, replace the first find_element_by_xpath with:
driver.find_element_by_xpath("//div[#class='lemon--div__373c0__1mboc border-color--default__373c0__3-ifU']/button[.='Read more']").click()
This works for both the websites.

Related

RSelenium: Entering search term - trouble with css selector

I am trying to trigger a search on this site. First, I want to enter a search term, then click the search button. I am able to to do the second step, however I am unfortunately unable to access the search field. Below my attempt.
Start RSelenium
link_to_page<-"https://www.cec.ro/sucursale"
library(RSelenium)
rD <- rsDriver(browser = "firefox", port = 483L, verbose = F)
remDr <- rD[["client"]]
# Navigate to site, and wait
remDr$navigate(link_to_page)
Sys.sleep(5)
#Search for element by its id
remDr$findElement(using="css", "#edit-localitate--_jjPr3WukFY")
Error: Summary: NoSuchElement
Detail: An element could not be located on the page using the given search parameters.
class: org.openqa.selenium.NoSuchElementException
Further Details: run errorDetails method
There is apparently something wrong with the css selector. I checked, it's not nested in an iframe or so, but mabe it's related to the 'form' element it is nested in? Grateful for any hint. Many thanks.
The error is NoSuchElement which indicates an element could not be located on the page using the given search parameter
The second part of the classname i.e. _jjPr3WukFY is dynamically generated and is bound to change sooner/later. They may change next time you access the application afresh or even while next application startup. So can't be used in locators.
Solution
You need to consider any of the other attributes which is static in nature. Example:
remDr$findElement(using="css", "button[id^edit-localitate]")
or
remDr$findElement(using="xpath", "//button[starts-with(#id, 'edit-localitate')]")

How to find Xpath of any "city" of variable dropdown?

I installed Chrome addon "Selectors Hub".
I opened site: spicejet.com
I choose some random city of "from" dropdown.
With help of "Selectors Hub" chrome addon, I grabbed the Xpath code of
that city:
//div[#class='css-1dbjc4n r-14lw9ot r-z2wwpe r-vgw6uq r-156q2ks r-urutk0 r-8uuktl r-136ojw6']//div[11]
While validating this Xpath code in console, it shows 0 matches.
This website is built on ReactJs as front-end, if I am not wrong, and finding the elements of a ReactJs website is a bit challenging; adding to that, if you rely on some locator finding tool, it's gets more difficult. It's always better you build your own locator strategy than rely on tools, especially for websites built with React, Vue, etc.
Having said that, the strategy here is to find the relatively narrowed down relative locator, and then since you are looking for a random selection of city, collect all the cities first, then apply random to it. Here is what it figured:
I collected cities, but along with it came some unwanted items (courtesy my relative locator), and I check them and if they are picked up, I pass them out, and only when a city is selected by random, I click on it. Check this code:
import random
driver.get("https://www.spicejet.com/")
time.sleep(10)
driver.find_element(By.XPATH, "//div[#data-testid='to-testID-destination']").click()
time.sleep(2)
cities = driver.find_elements(By.XPATH, "//div[#data-testid='to-testID-destination']//div[#data-focusable='true']")
print(len(cities))
x = random.choice(cities)
if x.text in ['To', 'India', 'International']:
pass
else:
print(x.text)
x.click()
time.sleep(5)
driver.quit()
Output:
Pakyong
Pakyong Airport
PYG
Process finished with exit code 0

Scraping a page every time it changes

Hi I am currently looking to scrape a [age such as this "https://www.tennis24.com/match/ABiALWlt/#match-statistics;0" every time it changes score. currently i have the ability to scrape it using selenium and BS using the below code
from selenium import webdriver
Chrom_path = r"C:\Users\Dan1\Desktop\chromedriver.exe"
driver = webdriver.Chrome(Chrom_path)
driver.get("https://www.tennis24.com/match/zVrM3ySQ/#match-statistics;0")
data = driver.find_elements_by_class_name("statTextGroup")
for d in data:
sub_data = d.find_elements_by_xpath(".//*")
assert len(sub_data)==3
for s_d in sub_data:
print(s_d.get_attribute('class')[19:], s_d.get_attribute('innerText'))
but I have no idea how to automate it so that once the score at the top of the page located here"Medical timeout6 : 6 ( 0 : 0 )" changes, the scraper scrapes the new data. the change to monitor though is only visible when the match is in play and not always there.
if you need anymore info please let me know and ill be happy to add it
You can scrape in a while loop the "scoreboard"-class and when this is not the same as the old value of this then this value changed and you can scrape the other things you wanted.
Hope it helped

What is the functioning of fireevent() in HP QTP / UFT?

I am learning HP UFT.
Recently I came across fireevent and I tried to implement it on the website of Flipkart. I was trying to use firevent("onmouseover") for the link Men on the homepage of the website.
I used ChildObjects to find out the Link and WebElement (in two different tests), first Highlighted it and then used object.fireevent("onmouseover") as well as object.fireevent("OnClick"). The OnClick is working and it is showing the link as selected (i.e. the dotted box covering the link when we press tab), but it is not showing the Menu Under Men Section.
I had googled and bingged a lot. But was unable to find the exact working of FireEvent in QTP/UFT.
Please Help me solving the above problem as well as some tutorials on FireEvent.
EDIT: I am using IE 11 for testing.
Motti has already answered the technical definition, but I shall attempt to give you a solution to your functional issue.
In my experience .FireEvent often doesn't work as you would expect. An alternative for something like onmouseover is to simulate user behaviour a bit more closely by actually moving the mouse to the desired location. In our framework we have a little extension function to do just that, a pared-down version of which is shown here:
Sub My_MouseOver(objSender)
Dim absX : absX = objSender.GetROProperty("abs_x")
Dim absY : absY = objSender.GetROProperty("abs_y")
Dim width : width = objSender.GetROProperty("width")
Dim height : height = objSender.GetROProperty("height")
Dim x : x = (absX + (width / 2))
Dim y : y = (absY + (height / 2))
Dim deviceReplay : Set deviceReplay = CreateObject("Mercury.DeviceReplay")
deviceReplay.MouseMove x, y
Reporter.ReportEvent micDone, "A step name", "A useful step description"
End Sub
RegisterUserFunc "Link", "MouseOver", "My_MouseOver"
RegisterUserFunc "WebButton", "MouseOver", "My_MouseOver"
RegisterUserFunc "WebElement", "MouseOver", "My_MouseOver"
As an example you can then do as follows to bring up the "ELECTRONICS" menu overlay on flipkart.com (obviously substitute your own Browser and Page definitions):
Browser("Flipkart").Page("Main Nav").Link("xpath:=//a[#data-tracking-id='electronics']").MouseOver
In the original version there's various extra bits of error handling and custom reporting so it tells you what you clicked on, but the essence is the same. It locates the object on screen, calculates the centre and moves the mouse there. You might want to wait a small amount of time for the menu overlay to appear after doing so before calling .Click on one of the newly-displayed sub-elements.
I found a solution to my problem and it is working perfectly.
Setting.WebPackage("ReplayType") = 2
object.FireEvent "onmouseover"
Setting.WebPackage("ReplayType") = 1
In this case, the object will be:
Browser("name:=Online Shopping.*").Page("name:=Online Shopping.*").Link("innertext:=Men")
I have tried this and it is working fine. I guess, we do not need any alternatives. But I really don't know is, Ctrl+Space is not working for this in UFT. Don't know the reason.
This actually depends on what browser you're using.
Warning: There are exceptions to the information presented in this answer and it also may change in the future. The answer is meant to give a basic understanding but don't depend on it to be true without checking the behaviour for your specific version/use-case.
Originally QTP's FireEvent was supposed to call IE's non-standard fireEvent method.
On Firefox and Chrome this is implemented using the standard dispatchEvent. You should check which events the web site expects to get.
Things get complicated if you mix the event models (the standard DOM level 2 and Microsofts) as explained in this blog post.

Click a Link working in Selenium Ide but not working in Webdriver

I am testing a scenario for the site "https://www.freecrm.com/index.html"
login credentials [ john2013 / john2013 ]
Scenario :
1 open the site https://www.freecrm.com/index.html
2 login with valid credentials
3 click on the "New Contacts" link
4 Add new contacts
using Selenium ide i am able to login and click on the "New Contact" link , but when i am trying to do the same thing using Webdriver [ java] i am not able to click the "New Contact" link
the code i have written is given below
driver.findElement(By.name("username")).clear()
driver.findElement(By.name("username")).sendKeys("john2013");
driver.findElement(By.name("password")).clear();
driver.findElement(By.name("password")).sendKeys("john2013");
driver.findElement(By.cssSelector("input[type=\"image\"]")).click();
The code up to the above is working fine but clicking the "New Contact" link
driver.findElement(By.xpath("//div[#class='noprint']/span[#class='headertext']/a[3]/")).click();
is not working though the same xpath is working in IDE.
i have tried with expected condition option , sleep but nothing is working.
can any one help me in this regard.
The problem is that your page uses a frameset. You should tell your webdriver which frame to use when it will search for your element. So try this:
driver.findElement(By.name("username")).clear()
driver.findElement(By.name("username")).sendKeys("john2013");
driver.findElement(By.name("password")).clear();
driver.findElement(By.name("password")).sendKeys("john2013");
driver.findElement(By.cssSelector("input[type=\"image\"]")).click();
//switch the driver to use one of the frames on your page. Potentially wait for a bit till the page is loaded
driver.switchTo().frame("mainpanel");
driver.findElement(By.xpath("//*[text()='New Contact']")).click();
Please change the xpath of link to
xpath=//a[contains(text(),'<whatevertext>')]
Incase if the same text link is already present in some part o the page we can proceed with occurences. Example if it is repeating second time:
xpath=(//a[contains(text(),'<whatevertext>')])[2]
If Nothing is working the best way is through java script executor.
First get the webelement of the link.
Then use the following code"
webelement=driver.findElement(by.xpath("=//a[contains(text(),'')]
"))
(JavaScriptExecutor)driver.executescript("argument[0].click();",webelement)
Actually Speaking //div[#class='noprint']/span[#class='headertext']/a[3]/ is absolute XPath.
as it ends with a[3] the position will vary always so it is advisable to use relative XPath/CSS
Suggested CSS:css=.noprint > .headertext > a.classname
Another Suggested CSS: css=.noprint > .headertext > a[attribute='value']
So we can use either of the above format.