Push a button to load reviews - selenium

I'm trying to scrape reviews from this site:
https://www.bbb.org/sacramento/business-reviews/heating-and-air-conditioning/elite-heating-air-conditioning-in-elk-grove-ca-47012326/reviews-and-complaints
But the content of the reviews isn't been loaded with by scrapy.
I tried then to use selenium to push the button and load the content:
url = 'https://www.bbb.org/losangelessiliconvalley/business-reviews/plumbers/bryco-plumbing-in-chatsworth-ca-13096711/reviews-and-complaints'
driver_1 = webdriver.Firefox()
driver_1.get(url)
content = driver_1.page_source
REVIEWS_BUTTON = '//*[#class="button orange first"]'
button = driver_1.find_element_by_xpath(REVIEWS_BUTTON)
button.click()
But selenium isn't able to find the button from the above xapth, I'm getting the following error:
selenium.common.exceptions.NoSuchElementException: Message: Unable to locate element: {"method":"xpath","selector":"//*[#class=\"button orange first\"]"}

Your button located inside an iframe, so you need to switch to it first and then handle the button:
REVIEWS_BUTTON = '//*[#class="button orange first"]'
driver_1.switch_to_frame('the_iframe')
button = driver_1.find_element_by_xpath(REVIEWS_BUTTON)
button.click()
driver.switch_to_default_content()

Related

Webdriver Selenium not loading new page after click()

I´m using selenium to scrape a webpage and it finds the elements on the main page, but when I use the click() function, the driver never finds the elements on the new page. I used beautifulSoup to see if it´s getting the html, but the html is always from the main. (When I see the driver window it shows that the page is opened).
html = driver.execute_script('return document.documentElement.outerHTML')
soup = bs.BeautifulSoup(html, 'html.parser')
print(soup.prettify)
I´ve used webDriverWait() to see if it´s not loading but even after 60 seconds it never does,
element = WebDriverWait(driver, 60).until(EC.presence_of_element_located((By.ID, "ddlProducto")))
also execute_script() to check if by clicking the button using javascript loads the page, but it returns None when I print a variable saving the new page.
selectProducto = driver.execute_script("return document.getElementById('ddlProducto');")
print(selectProducto)
Also used chwd = driver.window_handles and driver.switch_to_window(chwd[1]) but it says that the index is out of range.
chwd = driver.window_handles
driver.switch_to.window(chwd[1])

Selenium & webdriver : click one 1 specific button

I wish to click on a specific button of my page using webdriver from Selenium
However, all the buttons have the same class name and they do not have Ids.
As it is the first button I have tried this :
button = driver.find_element(By.CLASS_NAME,"actionButton-0-2-74[0]").click()
then I tried with XPATH to click directly on the link doing this :
buttonbutton = driver.find_element(By.XPATH,"//input[#xmlns='http://www.w3.org/2000/svg']")
but the message error shows >
NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":"//input[#xmlns='http://www.w3.org/2000/svg']"}
the html is in the picture below
enter image description here

Playwright: Download via Print to PDF?

I'm seeking to scrape a web page using Playwright.
I load the page, and click the download button with Playwright successfully. This brings up a print dialog box with a printer selected.
I would like to select "Save as PDF" and then click the "Save" button.
Here's my current code:
with sync_playwright() as p:
browser = p.chromium.launch(headless=True)
playwright_page = browser.new_page()
got_error = False
try:
playwright_page.goto(url_to_start_from)
print(playwright_page.title())
html = playwright_page.content()
except Exception as e:
print(f"Playwright exception: {e}")
got_error = True
if not got_error:
soup = BeautifulSoup(html, 'html.parser')
#download pdf
with playwright_page.expect_download() as download_info:
playwright_page.locator("text=download").click()
download = download_info.value
path = download.path()
download.save_as(DOWNLOADED_PDF_FOLDER)
browser.close()
Is there a way to do this using Playwright?
Thanks very much to #KJ in the comments, who suggested that with headless=True, Chromium won't even put up a print dialog box in the first place.

Locating elements in section with selenium

I'm trying to enter text into a field (the subject field in the image) in a section using Selenium .
I've tried locating by Xpath , ID and a few others but it looks like maybe I need to switch context to the section. I've tried the following, errors are in comments after lines.
from selenium.webdriver import Firefox
from selenium.webdriver.firefox.options import Options
opts = Options()
browser = Firefox(options=opts)
browser.get('https://www.linkedin.com/feed/')
sign_in = '/html/body/div[1]/main/p/a'
browser.find_element_by_xpath(sign_in).click()
email = '//*[#id="username"]'
browser.find_element_by_xpath(email).send_keys(my_email)
pword = '//*[#id="password"]'
browser.find_element_by_xpath(pword).send_keys(my_pword)
signin = '/html/body/div/main/div[2]/div[1]/form/div[3]/button'
browser.find_element_by_xpath(signin).click()
search = '/html/body/div[8]/header/div[2]/div/div/div[1]/div[2]/input'
name = 'John McCain'
browser.find_element_by_xpath(search).send_keys(name+"\n")#click()
#click on first result
first_result = '/html/body/div[8]/div[3]/div/div[1]/div/div[1]/main/div/div/div[1]/div/div/div/div[2]/div[1]/div[1]/span/div/span[1]/span/a/span/span[1]'
browser.find_element_by_xpath(first_result).click()
#hit message button
msg_btn = '/html/body/div[8]/div[3]/div/div/div/div/div[2]/div/div/main/div/div[1]/section/div[2]/div[1]/div[2]/div/div/div[2]/a'
browser.find_element_by_xpath(msg_btn).click()
sleep(10)
## find subject box in section
section_class = '/html/body/div[3]/section'
browser.find_element_by_xpath(section_class) # no such element
browser.switch_to().frame('/html/body/div[3]/section') # no such frame
subject = '//*[#id="compose-form-subject-ember156"]'
browser.find_element_by_xpath(subject).click() # no such element
compose_class = 'compose-form__subject-field'
browser.find_element_by_class_name(compose_class) # no such class
id = 'compose-form-subject-ember156'
browser.find_element_by_id(id) # no such element
css_selector= 'compose-form-subject-ember156'
browser.find_element_by_css_selector(css_selector) # no such element
wind = '//*[#id="artdeco-hoverable-outlet__message-overlay"]
browser.find_element_by_xpath(wind) #no such element
A figure showing the developer info for the text box in question is attached.
How do I locate the text box and send keys to it? I'm new to selenium but have gotten thru login and basic navigation to this point.
I've put the page source (as seen by the Selenium browser object at this point) here.
The page source (as seen when I click in the browser window and hit 'copy page source') is here .
Despite the window in focus being the one I wanted it seems like the browser object saw things differently . Using
window_after = browser.window_handles[1]
browser.switch_to_window(window_after)
allowed me to find the element using an Xpath.

Navigation error in Selenium: Link is not clickable at point (...). Other element would receive the click [duplicate]

This question already has answers here:
Can not click on a Element: ElementClickInterceptedException in Splinter / Selenium
(7 answers)
ElementClickInterceptedException: Message: element click intercepted Element is not clickable error clicking a radio button using Selenium and Python
(1 answer)
ElementClickInterceptedException: Message: element click intercepted: Element <label> is not clickable with Selenium and Python
(2 answers)
Closed 2 years ago.
I would like to have Selenium navigate thorught the items of a navbar of this page, then retrieve its contents and pass them to beatifulsoup.
My code works fine for the first page, after which I get the following error:
ElementClickInterceptedException: Message: element click intercepted: Element ... is not clickable at point (77, 73). Other element would receive the click: <div class="fc-dialog-overlay"></div>
Previous answers have not helped.
Here is my code. I would truly appreciate any help.
webdriver.ChromeOptions()
options.add_argument('--ignore-certificate-errors')
options.add_argument('--incognito')
options.add_argument('--headless')
driver = webdriver.Chrome(options=options)
target_page = 'https://www.rollingstone.com/music/music-lists/500-greatest-songs-of-all-time-151127/'
driver.get(target_page)
header = driver.find_element_by_id('pmc-gallery-list-nav-bar-render')
items = header.find_elements_by_tag_name('a')
songs = []
for item in items:
driver.maximize_window()
time.sleep(5)
item.click()
page_source = driver.page_source
soup = BeautifulSoup(page_source)
song_all = soup.find_all('h2', {'class':'c-gallery-vertical-album__title'})
for song in song_all:
songs.append(strip_extra_chars(song.get_text()))
driver.quit()
Let's comment options.add_argument('--headless') and see how program runs.
You need to identify which element overlap your link.
For example, I tried to run your code and this dialog apears:
In this case, to disable notification you can add:
prefs = {"profile.default_content_setting_values.notifications": 2}
options.add_experimental_option("prefs", prefs)
But this dialog doesn't match .fc-dialog-overlay locator so probably your problem with another element. You need to find it.
Another solution, you can click using js driver.execute_script()