Capturing fully loaded webpage - selenium

To automate the updating process of some websites, I want to compare a before and after capture of a webpage and check for differences.
I tried fully capturing a webpage in Python using Selenium, but some elements that have animations on them are not included in the capture. In the code example below I tried capturing webflow.com. See the result, and it's missing elements: Result of capture.
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.common.by import By
options = webdriver.ChromeOptions()
options.headless = True
options.binary_location = "/usr/bin/google-chrome"
driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()), options=options)
URL = 'https://webflow.com/'
driver.get(URL)
S = lambda X: driver.execute_script('return document.body.parentNode.scroll'+X)
driver.set_window_size(S('Width'),S('Height'))
driver.find_element(By.TAG_NAME, "body").screenshot('screenshot.png')
driver.quit()
I tried different ways to make the driver scroll over the page, but that did not change the output. How can I make all the elements visible in a screen capture? I am open for different solutions than Selenium.
Note: the animations are not only webflow animations, so I would like it to work for different animations as well.

Related

Selenium TimeoutException when trying to click Cookies Agree

I am trying to crawl some articles from a website and before doing so, I need to click the "Cookies Agree" using Selenium in Python.
But unfortunately, I keep getting either TimeoutException or NoSuchElementException!
I've figured out that the click button is within iframe, so I've switched to it and clicked the consent button.
homepage = 'link'
driver.maximize_window()
driver.get(homepage)
driver.implicitly_wait(5)
driver.switch_to.frame('location')
try:
consent = wait(driver, 10).until(EC.element_to_be_clickable((By.CLASS_NAME, 'classname')))
consent.click()
except TimeoutException :
print('timeoutexception')
driver.switch_to.default_content()
iframe
consent click button
But still I just can't get through the TimeoutException error.
What have I done wrong....?!
You mess up wait and driver objects. They are different. Switch to iframe and wait for your button.
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
# switch to frame here
wait = WebDriverWait(driver, 20)
wait.until(EC.element_to_be_clickable(
(By.CSS_SELECTOR, '.message-component.message-button.no-children:nth-of-type(2)')))
consent = driver.find_element_by_css_selector('.message-component.message-button.no-children:nth-of-type(2)')
consent.click()
There are two .message-component.message-button.no-children css locators and you need the second one.
To find iframe use (this is bulletproof):
driver.switch_to.frame(driver.find_element_by_xpath("//iframe[contains(#id,'sp_message_iframe_')]"))

How to switch to nested framesets using Python Selenium?

I need to reach a target frame with Xpath = html/frameset/frameset[1]/frame[2].
The HTML tree of the web site is as follows (the target frame is marked with a red *):
The web page is a chat room and I want to get the input box and the send button.
The following code logs into the chat room and tries to get to the target frame, but it fails there (Selenium hangs while trying to switch to that frame):
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
driver = webdriver.Chrome('/home/yky/Downloads/chromedriver')
driver.get('http://ip131.ek21.com/oaca_1/?ot=1')
### Log into chat room:
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, '//*[#id="mlogin"]/form/ul/li[1]/input'))).send_keys("UserOne")
driver.find_element_by_xpath('//*[#id="mlogin"]/form/div/span').click()
time.sleep(6)
### NONE of these works:
#driver.switch_to.frame(driver.find_element_by_css_selector("frame[name='ta']"))
#driver.switch_to.frame("ta")
#driver.switch_to_frame(driver.find_element_by_xpath('html/frameset/frameset[1]/frame[2]'))
#driver.switch_to.frame(2);
inputbox = driver.find_element_by_name("says_temp")
sendbutton = driver.findElement(By.xpath("//input[#value='送出']"));
I find the suggestion to ignore framesets to be very beguiling, as it doesn't make sense when the framesets contain other frames as a tree structure. Also, the problem is that the first frameset does not contain any frame; it just contains another frameset. So there is no way for Selenium to switch to the first frameset.
Please help!!

How to use selenium and Python to click 'accept cookies' button in popup, how can I move on when popup has no x

I am new to using the selenium module. I have started with some simple tutorials which go ok until I get a popup.
Because the popup does not have an x, I am not able to apply other advice I have found online.
How to close pop up window in Selenium
However I have tried to inspect the code of the popup window and I can see that maybe I have a couple of options, close by referencing the link text 'Accept Cookies', or close by the button id which is "onetrust-accept-btn-handler"
This is the code I have so far.
import selenium
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
import time
PATH = "C:\Program Files (x86)\chromedriver.exe"
driver = webdriver.Chrome(PATH)
driver.get("https://www.reed.co.uk/jobs/senior-insight-analyst/42347955")
driver.implicitly_wait(10)
link = driver.find_element_by_link_text("onetrust-accept-btn-handler")
link.click()
Trying
link = driver.find_element_by_link_text("onetrust-accept-btn-handler")
and
link = driver.find_element_by_link_text("Accept cookies")
Result in errors
NoSuchElementException: Message: no such element: Unable to locate element: {"method":"link text","selector":"onetrust-accept-btn-handler"}
or
NoSuchElementException: Message: no such element: Unable to locate element: {"method":"link text","selector":"Accept cookies"}
I am wondering if there is any better command than link text if the button id is known, or am I missing another step in the process because it is a pop up? Any help appreciated. Thank you.
Please use the explicit wait so that your popup window can come up and your selenium script can detect the element and click on it.
Use the below code -
WebDriverWait(driver, 15).until(EC.element_to_be_clickable((By.ID, 'onetrust-accept-btn-handler'))).click()
Mark it as answer if it resolves your problem.
Thank you Swaroop Humane and Dev for your answers. The answer above works, I also needed to add in three more lines of import code before the solution managed to click the Accept cookies button in the pop up. https://selenium-python.readthedocs.io/waits.html
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
Final code below.
import selenium
from selenium.webdriver.common.by import By
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import time
PATH = "C:\Program Files (x86)\chromedriver.exe"
driver = webdriver.Chrome(PATH)
driver.get("https://www.reed.co.uk/jobs/senior-insight-analyst/42347955")
#driver.implicitly_wait(10)
#link = driver.find_element_by_link_text("onetrust-accept-btn-handler")
#link.click()
WebDriverWait(driver, 15).until(EC.element_to_be_clickable((By.ID, 'onetrust-accept-btn-handler'))).click()
Then this happened.
I probably need to figure out a way to run selenium while I am signed into this webservice, rather than let it run a new browser every time. Any ideas?

How to click on the button with text as Export as per the url through Selenium and Python?

I am stuck and I would love your help. I am not a Python genius, so apologies for the language. I need to click on a button (Export) on this website https://www.fec.gov/data/filings/?data_type=processed&committee_id=C00097485. The button should drive to the bottom of the page where a link to an Excel file appears. Now, I have used this code:
text="//button[#type='button' and contains(.,'Export')]"
driver = webdriver.Firefox()
driver.get("https://www.fec.gov/data/filings/data_type=processed&committee_id=C00142711")
time.sleep(5)
button=driver.find_element_by_xpath(text)
button.click
The script runs fine, no error messages. The website appears, but the 'click' doesnt take place.
I also tried:1) the "wait driver until element is clickable", 2) the ActionChain to move the cursor, 3) to substitute click with sendKeys.
There is no Iframe. I tried also on Chrome. I am using a pc with Windows 10.
What am I doing wrong??? Considering that with other websites, the click function works perfectly fine!
Click is a method so it should be button.click(). You are missing parens.
Also, it would be better if you used WebDriverWait instead of .sleep(), e.g. button = WebDriverWait(driver, 20).until( EC.element_to_be_clickable((By.XPATH, text)));
As per the url the button with text as Export is a JavaScript enabled element, so you need to induce _WebDriverWait_for the element to be clickable and you can use the following solution:
Code Block:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
driver = webdriver.Firefox()
driver.get("https://www.fec.gov/data/filings/?data_type=processed&committee_id=C00097485")
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button.js-export.button.button--cta.button--export"))).click()
Browser Snapshot:

How to accept all the alerts of a website?

When browsing webpages, I get sometimes alerts. How can I accept each alert that appears ? I have no idea what is the number of alerts that are in a given webpage.
This link alerts, let us make a test:
from selenium import webdriver
from selenium.webdriver.common import alert
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait
from selenium.common.exceptions import TimeoutException
class AlertsManager:
def alertsManager(self,url):
self.url_to_visit=url
self.driver=webdriver.Firefox()
self.driver.get(self.url_to_visit)
try:
while WebDriverWait(self.driver,1).until(EC.alert_is_present()):
self.alert=self.driver.switch_to_alert()
self.driver.switch_to_alert().accept()
except TimeoutException:
pass
print("Continue what you want here ...")
if __name__=='__main__':
AM=AlertsManager()
url="http://htmlite.com/JS006.php"
AM.alertsManager(url)
You can set capabilities. However the required capabilities may not be implemented on some specific browsers, test it by your self (As my knowledge, it only works with FF).
capabilities.setCapability(CapabilityType.UNEXPECTED_ALERT_BEHAVIOUR,UnexpectedAlertBehaviour.ACCEPT);