I'm trying to webscrape this page https://www.bumeran.com.pe/empleos-publicacion-menor-a-7-dias.html. However, I'm having trouble clicking on the next-page button to scrape all the pages. I've tried doing this:
from selenium import webdriver
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
options = webdriver.ChromeOptions()
options.add_argument('--no-sandbox')
options.add_argument('--disable-dev-shm-usage')
PATH = "C:\Program Files (x86)\chromedriver.exe"
wd = webdriver.Chrome(PATH, options=options)
wd.maximize_window()
wd.get("https://www.bumeran.com.pe/empleos-publicacion-menor-a-7-dias.html")
def pag_sig():
element_present = EC.visibility_of_element_located((By.XPATH, '//*[#class="Pagination__NextPage-sc-el3mid-4 gSlsBf"]/i'))
boton = WebDriverWait(wd,5).until(element_present)
boton.click()
page_max=127 #I get this with another piece of code
for i in range(page_max):
pag_sig()
However, this does not work properly. Sometimes my scraper clicks on other links rather than the button itself. I've tried adding waits but it doesn't work. How could I approach this problem?
May its because of not scrolling down. Try the below xpath and code once.
#xpath - //div[#id='listado-avisos']//button[#class='Pagination__NextPage-sc-el3mid-4 gSlsBf']
driver.get("https://www.bumeran.com.pe/empleos-publicacion-menor-a-7-dias.html")
for i in range (5):
nextoption = driver.find_element_by_xpath("//div[#id='listado-avisos']//button[#class='Pagination__NextPage-sc-el3mid-4 gSlsBf']")
driver.execute_script("arguments[0].scrollIntoView(true);", nextoption)
driver.execute_script("window.scrollBy(0,-300)")
nextoption.click()
time.sleep(3)
You can click on the button instead of directly clicking on the i below can help, I hope
button = WebDriverWait(driver,20).until(EC.element_to_be_clickable((By.XPATH,"//button[#class='Pagination__NextPage-sc-el3mid-4 gSlsBf']")))
button.click()
Related
I am working on a web scraping project at the moment. The website I am trying to get data from is not the easiest to work with. I am using Selenium, and have worked my way through most of the items I want to select.
The website: http://crashinformationky.org/AdvancedSearch
I can get the website to open, and select different properties. But when I select Data, then try and "click" on Today, to change the date. Nothing works.
I have tried using the xpath, css selectore, link text, partial link text, nothing works.
Here is my most recent attempt.
WebDriverWait(driver, timeout=5).until(driver.find_element(By.CSS_SELECTOR, '#QueryPanel-cond-2 > div:nth-child(4) > a'))
date_one = driver.find_element(By.CSS_SELECTOR, '#QueryPanel-cond-2 > div:nth-child(4) > a')
date_one.click()
date_one_enter = driver.find_element(By.XPATH,'//*[#id="dp1647307835636"]')
date_one_enter.send_keys('01/01/2016')
Welcome Kyle!
As a personal preference I like to use Full XPATH's when selecting items with Selenium. I was able to click on TODAY, clear the field, send a new date and hit ENTER. See below:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as ec
from selenium.webdriver.support.wait import WebDriverWait
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.keys import Keys
options = Options()
options.add_argument('--no-sandbox')
options.add_argument('--disable-extensions')
options.add_argument('--disable-dev-shm-usage')
options.add_argument('--ignore-certificate-errors')
options.add_argument('--enable-logging')
service = Service(ChromeDriverManager().install())
driver = webdriver.Chrome(service=service, options=options)
url = "http://crashinformationky.org/AdvancedSearch"
driver.get(url)
WebDriverWait(driver, 30).until(ec.visibility_of_element_located((By.XPATH, "/html/body/div[1]/div[3]/form/div[2]/div/div[1]/div[2]/div[2]/div/div[3]/a"))).click()
WebDriverWait(driver, 30).until(ec.visibility_of_element_located((By.XPATH, "/html/body/div[10]/div/div[3]"))).click()
WebDriverWait(driver, 30).until(ec.visibility_of_element_located((By.XPATH, "/html/body/div[1]/div[3]/form/div[2]/div/div[1]/div[2]/div[2]/div/div[2]/div[2]/div/div[3]/a"))).click()
field = driver.find_element(By.XPATH, "/html/body/div[1]/div[3]/form/div[2]/div/div[1]/div[2]/div[2]/div/div[2]/div[2]/div/div[3]/input")
field.send_keys(Keys.COMMAND + "a")
field.send_keys(Keys.DELETE)
field.send_keys('01/01/2016')
field.send_keys(Keys.ENTER)
Do note that I am using a MAC, so when I send the Keys.COMMAND that would be Keys.CONTROL for Windows.
I want to click on the list but they will give me time out error these is the page link https://www.s-ge.com/de/members-map
This is code:
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium import webdriver
from time import sleep
PATH="C:\Program Files (x86)\chromedriver.exe"
url='https://www.s-ge.com/de/members-map'
driver =webdriver.Chrome(PATH)
driver.get(url)
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button#m-view-tabs__button is-activ"))).click()
Clicking on accept cookies is optional, it's not required.
You should launch the browser in full screen so that Selenium can have the list button in its view port.
If you observe even when you try to click manually on the list button, you are scrolling a little bit and then performing a click. You should automate that scrolling part as well.
Surprised to see that none of the answer are using CSS_SELECTOR.
Code:
driver_path = r'C:\\Users\\***\\***\\chromedriver.exe'
driver = webdriver.Chrome(driver_path)
driver.maximize_window()
wait = WebDriverWait(driver, 20)
driver.get("https://www.s-ge.com/de/members-map")
list_button = WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button[data-target-tab='list']")))
driver.execute_script("arguments[0].click();", list_button)
Imports:
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
Could you try to use this XPath locator instead?
//button[contains(#data-target-tab, 'list')]
Either the locator is not correct, or the element is not clickable.
In case the element is not clickable, try to use js click.
element = driver.find_element_by_xpath("//button[contains(#data-target-tab, 'list')]")
def js_click(self, element):
driver.execute_script("arguments[0].click();", element)
Try with xpath
driver.find_element_by_xpath("//button[#class='m-view-tabs__button is-active']").click()
Been searching for a while and tried all the solutions present but none appear to be working. I created a "slide show" that will first log in, then alternate between tabs. All of that is working but i cannot get rid of the
"Chrome is being controlled by automated test software" bar. Any advise?
Code
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
usernameStr = 'test'
passwordStr = 'test'
browser = webdriver.Chrome()
#first tab
browser.get(('www.testwebsite.com?'))
# fill in username and hit the next button
username = browser.find_element_by_id('username')
username.send_keys(usernameStr)
password = WebDriverWait(browser, 10).until(
EC.presence_of_element_located((By.ID, 'password')))
password.send_keys(passwordStr)
nextButton = browser.find_element_by_class_name('emp-submit')
nextButton.click()
#second tab
browser.execute_script("window.open('about:blank', 'tab2');")
browser.switch_to.window("tab2")
browser.get('www.testwebsite.com')
Try this:
from selenium import webdriver
options = webdriver.ChromeOptions()
options.add_experimental_option("useAutomationExtension", False)
options.add_experimental_option("excludeSwitches",["enable-automation"])
driver_path = '/Users/myuser/Downloads/chromedriver'
driver = webdriver.Chrome(executable_path=driver_path, chrome_options=options)
driver.get('https://google.com')
driver.close()
When you open Chrome Browser in through ChromeDriver this infobar containing the notification is embedded as follows:
Chrome is being controlled by automated test software
Browser snapshot without the argument disable-infobars:
But if you add the argument disable-infobars through an instance of ChromeOptions you can get rid of this infobar as follows:
Code Block:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
options = Options()
options.add_argument('start-maximized')
options.add_argument('disable-infobars')
driver = webdriver.Chrome(chrome_options=options, executable_path=r'C:\WebDrivers\chromedriver.exe')
driver.get('https://www.google.com/')
Browser snapshot applying the argument disable-infobars:
THIS IS WORKING WITH LATEST RELEASE.
Try it by changing driver path under "service_obj"
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
chrome_options = webdriver.ChromeOptions()
chrome_options.add_experimental_option("useAutomationExtension", False)
chrome_options.add_experimental_option("excludeSwitches",["enable-automation"])
service_obj = Service(r"C:\Users\Documents\Sublime_srk\drivers\chromedriver_win32\chromedriver.exe")
driver = webdriver.Chrome(options=chrome_options,service=service_obj)
driver.get("https://www.google.co.in/")
Click on the "x" to close the bar. It doesn't work initially, but then maximize and restore the window and it should disappear. This is for anyone who doesn't trigger their tests through Java.
Info
I want to automate the web-server http://bioinfo.unipune.ac.in/IRESPred/IRESPred.html using python selenium.
html code
<input name="fileToUpload" type="file" class="btn btn-default btn-file btn-sm" xpath="1">
code
from selenium import webdriver
import time
driver = webdriver.Edge(r"C:\Users\Amardeep\Downloads\msedgedriver.exe")
driver.maximize_window()
driver.get("http://bioinfo.unipune.ac.in/IRESPred/IRESPred.html")
time.sleep(2)
button = driver.find_element_by_xpath("//input[#name = 'fileToUpload']").click()
output error
selenium.common.exceptions.InvalidArgumentException: Message: invalid argument
Tried solutions
I used different x-paths to click the button which opens a window to upload a file-path but button is not clickable. I used find element by name,class name,css selector but it also not worked. How should I solve this problem.
Try below code - (I'm using Chromedriver and you are using Edge, so modify code accordingly)
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver import ActionChains
from selenium.webdriver.common.by import By
import time
driver = webdriver.Chrome()
driver.maximize_window()
wait = WebDriverWait(driver, 5)
action = ActionChains(driver)
driver.get("http://bioinfo.unipune.ac.in/IRESPred/IRESPred.html")
time.sleep(2)
Upload_btn = wait.until(EC.presence_of_element_located((By.XPATH, "//input[#name='fileToUpload']")))
action.move_to_element(Upload_btn).click().perform()
I'm able to click on a button using the above code, let me know if you have any issues or you face any error message.
I'm trying to click an "export as csv" button on this webpage using selenium https://monitoringpublic.solaredge.com/solaredge-web/p/site/public?name=Alva%20Court%20E5&locale=en_GB#/dashboard (the button is next to "Power and Energy" title). Once I run the program, the site pops up but the download button is not clicked, resulting on Timeout Exception
However the code works with the following site that I found from another StackOverflow question https://www.rotowire.com/football/injury-report.php (although once I run the program and the site pops up, I have to manually accept the cookies in order for the file to be downloaded but that's another issue).
My question is why does the second link work but the first does not?
Here is the code:
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait as wait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
browser = webdriver.Chrome("C:/Path/chromedriver.exe")
url = 'https://monitoringpublic.solaredge.com/solaredge-web/p/site/public? name=Alva%20Court%20E5&locale=en_GB#/dashboard'
browser.get(url)
button = wait(browser, 20).until(EC.element_to_be_clickable((By.CLASS_NAME, "is-csv")))
button.click()
browser.close()
To invoke click() on the element with tooltip text as Export as CSV you have to induce WebDriverWait for the element_to_be_clickable() and you can use the following Locator Strategy:
Using XPATH:
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//p[#class='x-panel-header-text' and text()='Power and Energy']//following::button[1]"))).click()
Note : You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
Browser Snapshot:
For Power and Energy selector is #power_energy_panel button[class*=export].
For Comparative Energy is #se-comparative-energy-panel button[class*=export].
url = "https://monitoringpublic.solaredge.com/solaredge-web/p/site/public?name=Alva%20Court%20E5&locale=en_GB#/dashboard"
browser.get(url)
button = WebDriverWait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "#power_energy_panel button[class*=export]")))
button.click()
The class name is incorrect. try with following class name,
button = wait(browser, 20).until(EC.element_to_be_clickable((By.CLASS_NAME, "se-button-btn export_csv_btn")))
button.click()