Threading and Selenium - selenium

I'm trying to make multiple tabs in Selenium and open a page on each tab simultaneously. Here is the code.
CHROME_DRIVER_PATH = "C:/chromedriver.exe"
from selenium import webdriver
import threading
driver = webdriver.Chrome(CHROME_DRIVER_PATH)
links = ["https://www.google.com/",
"https://stackoverflow.com/",
"https://www.reddit.com/",
"https://edition.cnn.com/"]
def open_page(url, tab_index):
driver.switch_to_window(handles[tab_index])
driver.get(url)
return
# open a blank tab for every link in the list
for link in range(len(links)-1 ): # 1 less because first tab is already opened
driver.execute_script("window.open();")
handles = driver.window_handles # get handles
all_threads = []
for i in range(0, len(links)):
current_thread = threading.Thread(target=open_page, args=(links[i], i,))
all_threads.append(current_thread)
current_thread.start()
for thr in all_threads:
thr.join()
Execution goes without errors, and from what I understand this should logically work correctly. But, the effect of the program is not as I imagined. It only opens one page at a time, sometimes it doesn't even switch the tab... Is there a problem that I'm not aware of in my code or threading doesn't work with Selenium?

There is no need in switching to new window to get URL, you can try below to open each URL in new tab one by one:
links = ["https://www.google.com/",
"https://stackoverflow.com/",
"https://www.reddit.com/",
"https://edition.cnn.com/"]
# Open all URLs in new tabs
for link in links:
driver.execute_script("window.open('{}');".format(link))
# Closing main (empty) tab
driver.close()
Now you can handle (if you want) all the windows from driver.window_handles as usual

Related

Rotating Proxies and Selenium - Close driver and re-run script with new IP

I am running an automated script that needs to change IP every 5-6 refreshes. I understand that the it cannot be changed dynamically as it needs to re-open the webdriver with new options. I have a list of proxies in a text file and I wish to use these IPs at random every 5-6 times the driver refreshes. It would normally get stuck in a try catch statement that looks for a keyword and if it does not find the keyword it will refresh every 5 seconds until this word is found.
def find_link_by_word_in_href(driver, words):
for word in words:
try:
return driver.find_element(By.XPATH, f"//*[contains(#href,'{word}')]")
except NoSuchElementException:
pass
while True:
element = find_link_by_word_in_href(driver, ['dadsa', 'daasd', 'asdsad'])
if element is not None:
element.click()
break
else:
driver.refresh()
time.sleep(5)
I wish to break this once it hits 5 refreshes and restart the script choosing a new IP from the text file. Can anybody help or point me in the correct direction to solve this?

trying to open full text when webscraping using selenium

I am trying to open the full text on a page when using selenium.
I got the following code:
import selenium
from selenium import webdriver as wb
from selenium.webdriver.common.by import By
import time
webD=wb.Chrome("C:\Program Files (x86)\chromedriver.exe")
webD.get('https://www.flashscore.com/')
webD.maximize_window() # For maximizing window
webD.implicitly_wait(2) # gives an implicit wait for 20 seconds
webD.find_element_by_id('onetrust-reject-all-handler').click()
matchpages = webD.find_elements(By.CLASS_NAME, "preview-ico.icon--preview")
Open_full_text = webD.find_elements(By.CLASS_NAME, "previewShowMore.showMore")
for matchpages in matchpages:
matchpages.click()
time.sleep(5)
for Open_full_text in Open_full_text:
Open_full_text.click()
However, When I try to let this script click on the open full text button, it does not open up completely.
What is the reason for this, or how can this be adjusted? Is it ok to use the for loop in this scenario, or do I have to use another method?
Thanks very much!

Selenium's WebDriver.execute_script() returns 'None'

My program is having trouble getting an existing class from a webpage using Selenium. It seems that my WebDriver.execute_script function is not working.
import urllib
from selenium import webdriver
#Path to the chromedriver is definitely working fine.
path_to_chromedriver = 'C:\Users\Ben\Desktop\Coding\FreeFoodFinder\chromedriver.exe'
browser = webdriver.Chrome(executable_path = path_to_chromedriver)
url = 'http://www.maidservicetexas.com/'
browser.implicitly_wait(30)
browser.get(url)
content = browser.execute_script("document.getElementsByClassName('content')");
#Just printing the first character of the returned content's toString for now. Don't want the whole thing yet.
#Only ever prints 'N', the first letter of 'None'...so obviously it isn't finding the jsgenerated content even after waiting.
print content
My program returns 'None,' which tells me that the javascript function is not returning a value/being executed. Chrome's web dev tools tell me that 'content' is certainly a valid class name. The webpage isn't even dynamically generated (my eventual goal is to scrape dynamic content, which is why I make my WebDriver wait for 30 seconds before running the script.)
Return the value:
content = browser.execute_script("return document.getElementsByClassName('content');");

Explicit in selenium

I am using Selenium to get div value, but the fallowing code is not waiting for the page, just for URL. I used time.sleep, which is very primitive and totally not flexible. I want to change it on the explicit, but I am not too experienced in Python and I have a problem with that.
The website name has been changed just in case :
def repeat():
import wx
while True:
botloc = driver.find_element_by_id('botloc').text
print botloc
botX,botY = map(int,botloc.split(','))
print botX
print botY
wx.Yield()
def checker():
if driver.current_url == 'logged.example.com':
time.sleep(5)
repeat()
else:
checker()
checker()
How can I replace time.sleep with something flexible to wait the shortest time as possible after the page will be loaded? How to use explicit correctly with my code?
I know that's possible with using an element from the website, but I can't write anything sensible, I just need an example.
Is possibility to use element_by_id('botloc') for wait till it will be visible then start repeat() ?
How can i replace time.sleep with something flexible to wait shortest
time as possible after the page will be loaded?
I suppose you use get(url) to load the page. Generally you don't have to do anything, WebDriver automatically waits until page is being loaded. So you can remove time.sleep(). However there are some issues reported when loading the page using get with firefox driver, because of that you will have to wait for some target element which is supposed to be in the loaded page as mentioned below.
How to use explicit correctly with my code?
Have you checked Selenium webdriver documentation ? you can wait for botloc element explicitly as below
//assuming you have a valid webdriver reference
//Ex: DEFAULT_WAIT = 10 means
//waits up to 10 seconds before throwing a TimeoutException or if it finds the element will return it in 0 - 10 seconds.
element = WebDriverWait(webdriver, DEFAULT_WAIT).until(EC.presence_of_element_located((By.ID, "botloc")))
Refer this page for more information

Cucumber + Capybara tests to ensure a new window is opened

I have the following lines in my feature file:
Given I have website "www.google.co.uk"
When I click the website "www.google.co.uk"
Then "www.google.co.uk" page is opened in a new window
I am struggling to find a way to test that the webpage is definately opened in a new window.
Currently I have been using this in my step def:
Then /url "([^"]*)" is opened in new window/ do |url|
browser = page.driver.browser
current_id = browser.window_handle
tab_id = page.driver.find_window(url)
browser.switch_to.window tab_id
page.driver.browser.close
browser.switch_to.window current_id
end
but this still passes the test if the webpage is loaded on the same page, I want it to fail if the webpage is loaded on the same window/tab.
Any suggestions?
Many thanks
I see no assertions in your test.
My approach would be to test size of window_handles array after performing clicking action on the link, since before clicking the size should equal 1 and after the clicking window_handles should equal 2.
assert page.driver.browser.window_handles.size == 2
Imho, good enough, since if the webpage is loaded in the same tab, the size will be 1 and the test will fail.