Selenium: Can't access cookie banner html - selenium

I've been trying to access the html code of cookie banners using Selenium. For some websites, I can see the cookie banner html in the Firefox Web-Inspector, however, I cannot access it via Selenium.
For example https://faz.net. Here, driver.page_source does not contain the html code of the cookie banner and I also can't access it's elements via driver.find_elements (e.g. the "ZUSTIMMEN" - button. "zustimmen" means "to accept").
What I've tried so far:
from selenium import webdriver
driver = webdriver.Firefox()
driver.implicitly_wait(20)
driver.get("https://faz.net")
print(driver.page_source) # page source does not contain the button "ZUSTIMMEN"
print(driver.find_elements_by_xpath('//button[text()="ZUSTIMMEN"]'))
driver.execute_script("arguments[0].click();", WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, '//button[text()=ZUSTIMMEN"]'))))
What am I doing wrong?

That button ZUSTIMMEN is in iframe. You need to switch the driver focus to iframe like below :
driver.get("https://faz.net")
wait = WebDriverWait(driver, 10)
wait.until(EC.frame_to_be_available_and_switch_to_it((By.CSS_SELECTOR, "iframe[id^='sp_message_iframe']")))
wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button[title='ZUSTIMMEN']"))).click()
Imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
once you are done with iframe, you can switch to default content like this :
driver.switch_to.default_content()

That element is inside an iframe.
You have to switch to the iframe in order to access the element.
Like this:
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
driver = webdriver.Firefox()
driver.get("https://faz.net")
WebDriverWait(driver, 10).until(EC.frame_to_be_available_and_switch_to_it((By.XPATH,"//iframe[contains(#title,'SP')]")))
Now you can click on the cookie button in order to close it with
wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "[title='ZUSTIMMEN']"))).click()

Related

Selenium - Having Problems finding a nz-select form element, when trying to click

I am currently trying to automate the input into a form on a website, but i cant seem to find a way to select the dropdown.
On website:
immosuche.degewo.de/de/properties/W1400-40660-0750-0902.html
You'll need to click on Kontaktieren.
In HTML:
I'm currently trying to find it by xpath this way:
driver.findElement(By.xpath("/html/body/el-root/div/el-listing-application/form/div[2]/div[1]/nz-form-item/nz-form-control/div/span/nz-select/div")).click();
But i always get this exception:
Exception in thread "main" org.openqa.selenium.NoSuchElementException: no such element: Unable to locate element: {"method":"xpath","selector":"/html/body/el-root/div/el-listing-application/form/div[2]/div[1]/nz-form-item/nz-form-control/div/span/nz-select/div"}
(Session info: chrome=108.0.5359.126)
For documentation on this error, please visit: https://selenium.dev/exceptions/#no_such_element
Does any one have any idea how i could click it without getting the Exeption?
Those elements are inside an iframe. To access elements inside it you need to switch into that iframe first.
The following code sample is working:
import time
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
options = Options()
options.add_argument("start-maximized")
webdriver_service = Service('C:\webdrivers\chromedriver.exe')
driver = webdriver.Chrome(options=options, service=webdriver_service)
wait = WebDriverWait(driver, 10)
url = "https://immosuche.degewo.de/de/properties/W1400-40660-0750-0902.html"
driver.get(url)
wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, ".expose__header-functions a[href='#kontakt']"))).click()
time.sleep(3)
wait.until(EC.frame_to_be_available_and_switch_to_it((By.TAG_NAME, "iframe")))
wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "[formcontrolname='salutation']"))).click()
wait.until(EC.element_to_be_clickable((By.XPATH, "//li[contains(.,'Herr')]"))).click()
wait.until(EC.element_to_be_clickable((By.ID, "firstName"))).send_keys('Prophet')
wait.until(EC.element_to_be_clickable((By.ID, "lastName"))).send_keys('Mozes')
wait.until(EC.element_to_be_clickable((By.ID, "email"))).send_keys('mymail#mail.com')
wait.until(EC.element_to_be_clickable((By.ID, "formly_2_input_numberPersonsTotal_0"))).send_keys('5')
wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, ".form-actions [type='submit']"))).click()
The result screenshot is:
The request is sent.
When finished working inside the iframe don't forget to switch to the default content with:
driver.switch_to.default_content()
The <nz-select> element is within an iframe so you have to:
Induce WebDriverWait for the frameToBeAvailableAndSwitchToIt.
Induce WebDriverWait for the desired elementToBeClickable.
You can use the following locator strategies:
driver.get("https://immosuche.degewo.de/de/properties/W1400-40660-0750-0902.html");
new WebDriverWait(driver, Duration.ofSeconds(10), Duration.ofSeconds(10)).until(ExpectedConditions.elementToBeClickable(By.cssSelector("button#cookie-consent-submit"))).click();
new WebDriverWait(driver, Duration.ofSeconds(10), Duration.ofSeconds(10)).until(ExpectedConditions.elementToBeClickable(By.xpath("//p[contains(., 'degewo Marzahner Wohnungsgesellschaft mbH')]//following::div[1]//span[text()='Kontaktieren']"))).click();
new WebDriverWait(driver, Duration.ofSeconds(10)).until(ExpectedConditions.frameToBeAvailableAndSwitchToIt(By.cssSelector("iframe[src^='https://app.wohnungshelden.de/public/listings']")));
WebElement elem = new WebDriverWait(driver, Duration.ofSeconds(10), Duration.ofSeconds(10)).until(ExpectedConditions.elementToBeClickable(By.xpath("//nz-select[#nzplaceholder='Bitte auswählen']//div[#nz-select-top-control]")));
((JavascriptExecutor)driver).executeScript("arguments[0].click();", elem);
Browser Snapshot:

Navigate in python with selenium

I have an error on my code because it does not manage to click on a cookie button. I have already tried several techniques and none of them work. I have tried implicitwait and explicitwait nothing works.
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
driver = webdriver.Chrome(executable_path="ressources/chromedriver.exe")
driver.get("https://vu.fr/gWDI")
WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.CSS_SELECTOR,
"button.ncss-btn-primary-dark"))).click()
driver.quit()

Click on button in selenium

I want to click on the list but they will give me time out error these is the page link https://www.s-ge.com/de/members-map
This is code:
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium import webdriver
from time import sleep
PATH="C:\Program Files (x86)\chromedriver.exe"
url='https://www.s-ge.com/de/members-map'
driver =webdriver.Chrome(PATH)
driver.get(url)
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button#m-view-tabs__button is-activ"))).click()
Clicking on accept cookies is optional, it's not required.
You should launch the browser in full screen so that Selenium can have the list button in its view port.
If you observe even when you try to click manually on the list button, you are scrolling a little bit and then performing a click. You should automate that scrolling part as well.
Surprised to see that none of the answer are using CSS_SELECTOR.
Code:
driver_path = r'C:\\Users\\***\\***\\chromedriver.exe'
driver = webdriver.Chrome(driver_path)
driver.maximize_window()
wait = WebDriverWait(driver, 20)
driver.get("https://www.s-ge.com/de/members-map")
list_button = WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button[data-target-tab='list']")))
driver.execute_script("arguments[0].click();", list_button)
Imports:
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
Could you try to use this XPath locator instead?
//button[contains(#data-target-tab, 'list')]
Either the locator is not correct, or the element is not clickable.
In case the element is not clickable, try to use js click.
element = driver.find_element_by_xpath("//button[contains(#data-target-tab, 'list')]")
def js_click(self, element):
driver.execute_script("arguments[0].click();", element)
Try with xpath
driver.find_element_by_xpath("//button[#class='m-view-tabs__button is-active']").click()

Selenium scraping JS loaded pages

I'm trying to scrape some of the loaded JS data from https://surviv.io/stats/player787, such as the number of total kills. Could someone tell me how I can scrape the js loaded data with selenium. Thanks.
EDIT: Here is some of the code
from selenium import webdriver
browser = webdriver.Firefox()
browser.get('https://surviv.io/stats/player787')
b = browser.find_element_by_tag_name('tr')
The 'tr' which contains the data that I want is not grabbed by selenium
To get the count of kills.Induce WebDriverWait and visibility_of_all_elements_located
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium import webdriver
browser = webdriver.Firefox()
browser.get('https://surviv.io/stats/player787')
allkills = WebDriverWait(browser,20).until(EC.visibility_of_all_elements_located((By.XPATH,"//div[#class='card-mode-stat-name' and text()='KILLS']/following-sibling::div[1]")))
for item in allkills:
print(item.text)
The reason it's not finding it is because the page isn't fully rendered. You can add a wait with selenium so will not move on until the specified element is rendered first.
Also, if it's in a <table> tag, let pandas do the parsing for you (it uses beautifulsoup under the hood to pull out the <table>, <th>, <tr>, and <td> tags, returns them as a list of dataframes once you get the rendered html source:
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.common.exceptions import TimeoutException
import pandas as pd
browser = webdriver.Chrome('C:/chromedriver_win32/chromedriver.exe')
browser.get('https://surviv.io/stats/player787')
delay = 3 # seconds
WebDriverWait(browser, delay).until(EC.presence_of_element_located((By.CLASS_NAME, 'player-stats-overview')))
df = pd.read_html(browser.page_source)[0]
print (df.loc[0,'Kills'])
browser.close()
Output:
18884
print (df)
Wins Kills Games K/G
0 638 18884 8896 2.1
You could avoid the overhead of a browser and simply mimic the POST request the page makes.
import requests
headers = {'content-type': 'application/json; charset=UTF-8'}
data = {"slug":"player787","interval":"all","mapIdFilter":"-1"}
r = requests.post('https://surviv.io/api/user_stats', headers=headers, json=data)
data = r.json()
desired_stats = ['wins', 'kills', 'games', 'kpg']
for stat in desired_stats:
print(stat, ': ' , data[stat])
For OP:
View of payload in network tab visible when you click on the appropriate xhr indicated by the url in my answer (you need to scroll down to see the payload info)
To scrape the values 652, 19152, 8926, 2.1, etc from JS loaded pages you you have to induce WebDriverWait for the visibility_of_all_elements_located() and you can use either of the following Locator Strategies:
Using CSS_SELECTOR:
driver.get('https://surviv.io/stats/player787')
print([my_elem.get_attribute("innerHTML") for my_elem in WebDriverWait(driver, 5).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "table.player-stats-overview td")))])
Using XPATH:
driver.get('https://surviv.io/stats/player787')
print([my_elem.get_attribute("innerHTML") for my_elem in WebDriverWait(driver, 5).until(EC.visibility_of_all_elements_located((By.XPATH, "//table[#class='player-stats-overview']//td")))])
Console Output:
['652', '19152', '8926', '2.1']
Note : You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

How to click on the element with tooltip as Export as CSV using Selenium and Python

I'm trying to click an "export as csv" button on this webpage using selenium https://monitoringpublic.solaredge.com/solaredge-web/p/site/public?name=Alva%20Court%20E5&locale=en_GB#/dashboard (the button is next to "Power and Energy" title). Once I run the program, the site pops up but the download button is not clicked, resulting on Timeout Exception
However the code works with the following site that I found from another StackOverflow question https://www.rotowire.com/football/injury-report.php (although once I run the program and the site pops up, I have to manually accept the cookies in order for the file to be downloaded but that's another issue).
My question is why does the second link work but the first does not?
Here is the code:
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait as wait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
browser = webdriver.Chrome("C:/Path/chromedriver.exe")
url = 'https://monitoringpublic.solaredge.com/solaredge-web/p/site/public? name=Alva%20Court%20E5&locale=en_GB#/dashboard'
browser.get(url)
button = wait(browser, 20).until(EC.element_to_be_clickable((By.CLASS_NAME, "is-csv")))
button.click()
browser.close()
To invoke click() on the element with tooltip text as Export as CSV you have to induce WebDriverWait for the element_to_be_clickable() and you can use the following Locator Strategy:
Using XPATH:
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//p[#class='x-panel-header-text' and text()='Power and Energy']//following::button[1]"))).click()
Note : You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
Browser Snapshot:
For Power and Energy selector is #power_energy_panel button[class*=export].
For Comparative Energy is #se-comparative-energy-panel button[class*=export].
url = "https://monitoringpublic.solaredge.com/solaredge-web/p/site/public?name=Alva%20Court%20E5&locale=en_GB#/dashboard"
browser.get(url)
button = WebDriverWait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "#power_energy_panel button[class*=export]")))
button.click()
The class name is incorrect. try with following class name,
button = wait(browser, 20).until(EC.element_to_be_clickable((By.CLASS_NAME, "se-button-btn export_csv_btn")))
button.click()