I am trying to crawl some articles from a website and before doing so, I need to click the "Cookies Agree" using Selenium in Python.
But unfortunately, I keep getting either TimeoutException or NoSuchElementException!
I've figured out that the click button is within iframe, so I've switched to it and clicked the consent button.
homepage = 'link'
driver.maximize_window()
driver.get(homepage)
driver.implicitly_wait(5)
driver.switch_to.frame('location')
try:
consent = wait(driver, 10).until(EC.element_to_be_clickable((By.CLASS_NAME, 'classname')))
consent.click()
except TimeoutException :
print('timeoutexception')
driver.switch_to.default_content()
iframe
consent click button
But still I just can't get through the TimeoutException error.
What have I done wrong....?!
You mess up wait and driver objects. They are different. Switch to iframe and wait for your button.
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
# switch to frame here
wait = WebDriverWait(driver, 20)
wait.until(EC.element_to_be_clickable(
(By.CSS_SELECTOR, '.message-component.message-button.no-children:nth-of-type(2)')))
consent = driver.find_element_by_css_selector('.message-component.message-button.no-children:nth-of-type(2)')
consent.click()
There are two .message-component.message-button.no-children css locators and you need the second one.
To find iframe use (this is bulletproof):
driver.switch_to.frame(driver.find_element_by_xpath("//iframe[contains(#id,'sp_message_iframe_')]"))
Related
Before this, I have automated a login into the website howe page. I was stuck trying to make it so that it clicks on the image to navigate to another webpage.
shop_button = driver.find_element(By.XPATH, "/html/body/footer/div/div[2]/a/img")
shop_button.click()
This is the code for the image hyperlink
You can use the below xpath
//p[text()='Shop']//preceding-sibling::img
this should locate the img node, however p tag which has Shop as a text has to be unique in nature.
If it's unique, you can click on it like:
Code trial 1:
time.sleep(5)
driver.find_element(By.XPATH, "//p[text()='Shop']//preceding-sibling::img").click()
Code trial 2:
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//p[text()='Shop']//preceding-sibling::img"))).click()
Imports:
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
PS: time.sleep(5) is just for visualization purpose, you should not use it ideally if code 2 works fine.
You want to redirect to another webpage by clicking on a image. According to the code snippet, I can see that Image tag is inside the tag, you can click on that.
Can you try this ?
shop_button = driver.find_element(By.XPATH,"/html/body/footer/div/div[2]/a")
shop_button.click()
Do let me know if this works.
I was using Selenium find_element_by_xpath to click the cookie popup "Accept" button. But it was failing to find the button. So, I tried to locate the button using querySelector in the dev console. Also, the querySelector failed to find the button but after clicking inspect on the button the querySelector able to find the button.
Also, searching the xpath in dev elements just showing 1 of 1.
Why this is happening ? How do I click the "Accept" button using selenium ?
Website link: https://www.transfermarkt.com/
The xpath: //*[#id="notice"]/div[3]/div[2]/button
After inspect on the button.
That element is inside an iframe, so to access it you will first have to switch to that iframe.
You didn't mention what language are you using so I will give my solution in Python. This can be done with other languages as well with some little syntax changes.
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
wait = WebDriverWait(driver, 20)
wait.until(EC.frame_to_be_available_and_switch_to_it((By.CSS_SELECTOR,'iframe[title="SP Consent Message"]')))
wait.until(EC.visibility_of_element_located((By.XPATH, '//button[#title="ACCEPT ALL"]'))).click()
When finished working inside this iframe you will have to switch back to the default content with
driver.switch_to.default_content()
I am new to using the selenium module. I have started with some simple tutorials which go ok until I get a popup.
Because the popup does not have an x, I am not able to apply other advice I have found online.
How to close pop up window in Selenium
However I have tried to inspect the code of the popup window and I can see that maybe I have a couple of options, close by referencing the link text 'Accept Cookies', or close by the button id which is "onetrust-accept-btn-handler"
This is the code I have so far.
import selenium
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
import time
PATH = "C:\Program Files (x86)\chromedriver.exe"
driver = webdriver.Chrome(PATH)
driver.get("https://www.reed.co.uk/jobs/senior-insight-analyst/42347955")
driver.implicitly_wait(10)
link = driver.find_element_by_link_text("onetrust-accept-btn-handler")
link.click()
Trying
link = driver.find_element_by_link_text("onetrust-accept-btn-handler")
and
link = driver.find_element_by_link_text("Accept cookies")
Result in errors
NoSuchElementException: Message: no such element: Unable to locate element: {"method":"link text","selector":"onetrust-accept-btn-handler"}
or
NoSuchElementException: Message: no such element: Unable to locate element: {"method":"link text","selector":"Accept cookies"}
I am wondering if there is any better command than link text if the button id is known, or am I missing another step in the process because it is a pop up? Any help appreciated. Thank you.
Please use the explicit wait so that your popup window can come up and your selenium script can detect the element and click on it.
Use the below code -
WebDriverWait(driver, 15).until(EC.element_to_be_clickable((By.ID, 'onetrust-accept-btn-handler'))).click()
Mark it as answer if it resolves your problem.
Thank you Swaroop Humane and Dev for your answers. The answer above works, I also needed to add in three more lines of import code before the solution managed to click the Accept cookies button in the pop up. https://selenium-python.readthedocs.io/waits.html
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
Final code below.
import selenium
from selenium.webdriver.common.by import By
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import time
PATH = "C:\Program Files (x86)\chromedriver.exe"
driver = webdriver.Chrome(PATH)
driver.get("https://www.reed.co.uk/jobs/senior-insight-analyst/42347955")
#driver.implicitly_wait(10)
#link = driver.find_element_by_link_text("onetrust-accept-btn-handler")
#link.click()
WebDriverWait(driver, 15).until(EC.element_to_be_clickable((By.ID, 'onetrust-accept-btn-handler'))).click()
Then this happened.
I probably need to figure out a way to run selenium while I am signed into this webservice, rather than let it run a new browser every time. Any ideas?
I can't find the correct xpath to use for retweet and like on this page: https://twitter.com/snowfulls/status/1198269659465818115
Also, I need help finding the xpath for the second retweet button that comes up to confirm the retweet.
Is there a way to find xpaths automatically?
To answer your first question -- no, there is not a way to find XPaths automatically unless you use some kind of scanner tool. These XPaths are not always accurate though. The best approach is to use an XPath browser extension helper that will allow you to test out XPath expressions on a page in real-time. That is what I have used to help develop my solution.
To click the "Like" button on a tweet, you can use the following code:
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
# ensure the above references are added to use WebDriverWait correctly
# wait for the element to exist
like_button = WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.XPATH, "//div[#aria-label='Like']")))
# click the like button
like_button.click()
To click the retweet button, similarly:
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
# ensure the above references are added to use WebDriverWait correctly
# wait for the element to exist
retweet_button = WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.XPATH, "//div[#aria-label='Retweet']")))
# click the retweet button
retweet_button.click()
# now, confirm the retweet:
retweet_confirm = WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.XPATH, "//div[#data-testid='retweetConfirm']")))
# click the retweet confirm button
retweet_confirm.click()
The above solution uses a few simple extensions of Selenium library -- mainly, WebDriverWait and ExpectedConditions class. WebDriverWait allows us to wait up to a specified time for a condition to occur. This goes hand-in-hand with ExpectedConditions class, which measures the status of elements on the page to confirm whether or not a WebElement meets a certain condition.
So, WebDriverWait(driver, 10).until(EC.presence_of_element_located means "Wait up to 10 seconds for the presence of a WebElement" -- this WebElement then gets specified in the locator strategy, By.XPath, "....".
Hope this explanation helps a bit.
I am stuck and I would love your help. I am not a Python genius, so apologies for the language. I need to click on a button (Export) on this website https://www.fec.gov/data/filings/?data_type=processed&committee_id=C00097485. The button should drive to the bottom of the page where a link to an Excel file appears. Now, I have used this code:
text="//button[#type='button' and contains(.,'Export')]"
driver = webdriver.Firefox()
driver.get("https://www.fec.gov/data/filings/data_type=processed&committee_id=C00142711")
time.sleep(5)
button=driver.find_element_by_xpath(text)
button.click
The script runs fine, no error messages. The website appears, but the 'click' doesnt take place.
I also tried:1) the "wait driver until element is clickable", 2) the ActionChain to move the cursor, 3) to substitute click with sendKeys.
There is no Iframe. I tried also on Chrome. I am using a pc with Windows 10.
What am I doing wrong??? Considering that with other websites, the click function works perfectly fine!
Click is a method so it should be button.click(). You are missing parens.
Also, it would be better if you used WebDriverWait instead of .sleep(), e.g. button = WebDriverWait(driver, 20).until( EC.element_to_be_clickable((By.XPATH, text)));
As per the url the button with text as Export is a JavaScript enabled element, so you need to induce _WebDriverWait_for the element to be clickable and you can use the following solution:
Code Block:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
driver = webdriver.Firefox()
driver.get("https://www.fec.gov/data/filings/?data_type=processed&committee_id=C00097485")
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button.js-export.button.button--cta.button--export"))).click()
Browser Snapshot: