From which place Selenium Webdriver usually gets title - using driver.title ?
from Page Source
or from DOM structure
title
title returns the title of the current page.
Usage:
title = driver.title
Defination:
def title(self):
"""Returns the title of the current page.
:Usage:
title = driver.title
"""
resp = self.execute(Command.GET_TITLE)
Details: When you invoke driver.title the HTTP GET request is invoked through the /session/{session id}/title URI Template.
NOTE: This command returns the document title of the current top-level browsing context, equivalent to calling document.title.
The remote end steps are:
If the current top-level browsing context is no longer open, return error with error code no such window.
Handle any user prompts and return its value if it is an error.
Let title be the result of calling the algorithm for getting the title attribute of the current top-level browsing context's active document.
Return success with data title.
Related
I´m using selenium to scrape a webpage and it finds the elements on the main page, but when I use the click() function, the driver never finds the elements on the new page. I used beautifulSoup to see if it´s getting the html, but the html is always from the main. (When I see the driver window it shows that the page is opened).
html = driver.execute_script('return document.documentElement.outerHTML')
soup = bs.BeautifulSoup(html, 'html.parser')
print(soup.prettify)
I´ve used webDriverWait() to see if it´s not loading but even after 60 seconds it never does,
element = WebDriverWait(driver, 60).until(EC.presence_of_element_located((By.ID, "ddlProducto")))
also execute_script() to check if by clicking the button using javascript loads the page, but it returns None when I print a variable saving the new page.
selectProducto = driver.execute_script("return document.getElementById('ddlProducto');")
print(selectProducto)
Also used chwd = driver.window_handles and driver.switch_to_window(chwd[1]) but it says that the index is out of range.
chwd = driver.window_handles
driver.switch_to.window(chwd[1])
I'm having some issues in finding Selenium logs of actions performed on the browser.
My situation is the following:
I have a canvas with some clickable elements (it's a map).
I click on a specific point through actions.moveByOffset(pointToClick.getX(), pointToClick.getY()).click().build().perform(). Sometimes the point gets clicked, sometimes it seems not and I need to figure out what exactly happens.
How can I make sure that the click has indeed been performed? I set up the LoggingPreferences in the WebDriver configuration class
LoggingPreferences logPrefs = new LoggingPreferences();
logPrefs.enable(LogType.PERFORMANCE, Level.ALL);
logPrefs.enable(LogType.PROFILER, Level.ALL);
logPrefs.enable(LogType.BROWSER, Level.ALL);
logPrefs.enable(LogType.CLIENT, Level.ALL);
logPrefs.enable(LogType.DRIVER, Level.ALL);
logPrefs.enable(LogType.SERVER, Level.ALL);
capabilities.setCapability(CapabilityType.LOGGING_PREFS, logPrefs);
but some of them (server, performance) are empty, while the browser logs contain data similar to what I see in the browser console. Since those are all the LogTypes available, I'm not even sure I'm looking at the correct direction. Do you have any suggestion?
Thank you for your help!
There are a few advanced libraries built-in to Selenium such as EventFiringWebDriver, AbstractEventListener, etc, which can be used to log all Selenium actions being performed.
Relevant Java Docs can be found here: https://www.selenium.dev/selenium/docs/api/java/org/openqa/selenium/support/events/EventFiringWebDriver.html
https://www.selenium.dev/selenium/docs/api/java/org/openqa/selenium/support/events/EventFiringDecorator.html
https://www.selenium.dev/selenium/docs/api/java/org/openqa/selenium/support/events/WebDriverListener.html
I also have a working Python SeleniumBase example from here, which performs actions that get recorded and printed out. To run that example, you'll first need to pip install seleniumbase, then run the test with pytest: pytest test_event_firing.py -s.
from selenium.webdriver.support.events import EventFiringWebDriver
from selenium.webdriver.support.events import AbstractEventListener
from seleniumbase import BaseCase
class MyListener(AbstractEventListener):
def before_navigate_to(self, url, driver):
print("Before navigating to: %s" % url)
def after_navigate_to(self, url, driver):
print("After navigating to: %s" % url)
def before_find(self, by, value, driver):
print('Before find "%s" (by = %s)' % (value, by))
def after_find(self, by, value, driver):
print('After find "%s" (by = %s)' % (value, by))
def before_click(self, element, driver):
print('Before clicking on element with text: "%s"' % element.text)
def after_click(self, element, driver):
print("Click complete!")
class EventFiringTests(BaseCase):
def test_event_firing_webdriver(self):
self.driver = EventFiringWebDriver(self.driver, MyListener())
print("\n* EventFiringWebDriver example *")
self.open("https://xkcd.com/1862/")
self.click("link=About")
self.open("https://store.xkcd.com/search")
self.type('input[name="q"]', "xkcd book\n")
self.open("https://xkcd.com/1822/")
The output of that prints the Selenium actions that were detected via the EventListener:
* EventFiringWebDriver example *
Before navigating to: https://xkcd.com/1862/
After navigating to: https://xkcd.com/1862/
Before find "About" (by = link text)
After find "About" (by = link text)
Before find "About" (by = link text)
After find "About" (by = link text)
Before clicking on element with text: "About"
Click complete!
Before navigating to: https://store.xkcd.com/search
After navigating to: https://store.xkcd.com/search
Before find "input[name="q"]" (by = css selector)
After find "input[name="q"]" (by = css selector)
Before navigating to: https://xkcd.com/1822/
After navigating to: https://xkcd.com/1822/
Similar solutions exist for the other Selenium language bindings.
I'm trying to enter text into a field (the subject field in the image) in a section using Selenium .
I've tried locating by Xpath , ID and a few others but it looks like maybe I need to switch context to the section. I've tried the following, errors are in comments after lines.
from selenium.webdriver import Firefox
from selenium.webdriver.firefox.options import Options
opts = Options()
browser = Firefox(options=opts)
browser.get('https://www.linkedin.com/feed/')
sign_in = '/html/body/div[1]/main/p/a'
browser.find_element_by_xpath(sign_in).click()
email = '//*[#id="username"]'
browser.find_element_by_xpath(email).send_keys(my_email)
pword = '//*[#id="password"]'
browser.find_element_by_xpath(pword).send_keys(my_pword)
signin = '/html/body/div/main/div[2]/div[1]/form/div[3]/button'
browser.find_element_by_xpath(signin).click()
search = '/html/body/div[8]/header/div[2]/div/div/div[1]/div[2]/input'
name = 'John McCain'
browser.find_element_by_xpath(search).send_keys(name+"\n")#click()
#click on first result
first_result = '/html/body/div[8]/div[3]/div/div[1]/div/div[1]/main/div/div/div[1]/div/div/div/div[2]/div[1]/div[1]/span/div/span[1]/span/a/span/span[1]'
browser.find_element_by_xpath(first_result).click()
#hit message button
msg_btn = '/html/body/div[8]/div[3]/div/div/div/div/div[2]/div/div/main/div/div[1]/section/div[2]/div[1]/div[2]/div/div/div[2]/a'
browser.find_element_by_xpath(msg_btn).click()
sleep(10)
## find subject box in section
section_class = '/html/body/div[3]/section'
browser.find_element_by_xpath(section_class) # no such element
browser.switch_to().frame('/html/body/div[3]/section') # no such frame
subject = '//*[#id="compose-form-subject-ember156"]'
browser.find_element_by_xpath(subject).click() # no such element
compose_class = 'compose-form__subject-field'
browser.find_element_by_class_name(compose_class) # no such class
id = 'compose-form-subject-ember156'
browser.find_element_by_id(id) # no such element
css_selector= 'compose-form-subject-ember156'
browser.find_element_by_css_selector(css_selector) # no such element
wind = '//*[#id="artdeco-hoverable-outlet__message-overlay"]
browser.find_element_by_xpath(wind) #no such element
A figure showing the developer info for the text box in question is attached.
How do I locate the text box and send keys to it? I'm new to selenium but have gotten thru login and basic navigation to this point.
I've put the page source (as seen by the Selenium browser object at this point) here.
The page source (as seen when I click in the browser window and hit 'copy page source') is here .
Despite the window in focus being the one I wanted it seems like the browser object saw things differently . Using
window_after = browser.window_handles[1]
browser.switch_to_window(window_after)
allowed me to find the element using an Xpath.
I am attempting to write a test that is able to do the following:
1. Navigate to a website.
2. Navigate to a page under the menu.
3. Once in that page, assert that the image I want is displayed under a section labeled "SECTION".
Here is my code (approach 1):
public void test1() throws Exception {
WebElement compare_image = driver.findElement(By.linkText("URL link where the image is located"));
driver.get("website URL");
WebElement image = driver.findElement(By.cssSelector("cssSelector for image from FireFox -> inspect element -> copy CSS selector"));
assertEquals(image, compare_image); }
I am very new to Selenium and QA automation, so any detailed help would be appreciated as my google searches so far are coming up short. It is giving me an element not present exception for the findElement call, but I don't know why as I tried all the Bys I could get from inspect element.
Am I approaching this correctly? If not, what can I do differently?
If you want to check image is present or not under a section then you have to create a webelement for that section.
WebElement section= driver.findElement(By.xpath("//img(#class=‘Section')"));
Now create an image element under section element.
WebElement image= section.findElement(By.xpath("//img(#class=‘Test Image')"));
Now check image is exist or not.
boolean imagePresent = image.isDisplayed();
Now assert on boolean result.
assertTrue(imagePresent, “No image is exist”);
Note: Please take care of locators for section and Image as you didn’t provide Html for it. Code will work perfectly.
I am trying to scrape this website as part of my lesson to learn Kotlin and Web scraping with jsoup.
What I am trying to scrape is the Jackpot $1,000,000 est. values.
The below code was something that I wrote after searching and checking out a couple of tutorials online, but it won't even give me $1,000,000 (which was what this code was trying to scrape).
Jsoup.connect("https://online.singaporepools.com/lottery/en/home")
.get()
.run {
select("div.slab__text slab__text--highlight").forEachIndexed { i, element ->
val titleAnchor = element.select("div")
val title = titleAnchor.text()
println("$i. $title")
}
}
My first thought is that maybe this website is using JavaScript. That's why it was not successful.
How should I be going about scraping it?
I was able to scrape what you were looking for from this page on that same site.
Even if it's not what you want, the procedure may help someone in the future.
Here is how I did that:
First I opened that page
Then I opened the Chrome developer tools by pressing CTRL+
SHIFT+i or
by right-clicking somewhere on page and selecting Inspect or
by clicking ⋮ ➜ More tools ➜ Developer tools
Next I selected the Network tab
And finally I refreshed the page with F5 or with the refresh button ⟳
A list of requests start to appear (network log) and after, say, a few seconds, all requests will complete executing. Here, we want to look for and inspect a request that has a Type like xhr. We can filter requests by clicking the filter icon and then selecting the desired type.
To inspect a request, click on its name (first column from left):
Clicking on one of the XHR requests, and then selecting the Response tab shows that the response contains exactly what we are looking for. And it is HTML, so jsoup can parse it:
Here is that response (if you want to copy or manipulate it):
<div style='vertical-align:top;'>
<div>
<div style='float:left; width:120px; font-weight:bold;'>
Next Jackpot
</div>
<span style='color:#EC243D; font-weight:bold'>$8,000,000 est</span>
</div>
<div>
<div style='float:left; width:120px; font-weight:bold;'>
Next Draw
</div>
<div class='toto-draw-date'>Mon, 15 Nov 2021 , 9.30pm</div>
</div>
</div>
By selecting the Headers tab (to the left of the Response tab), we see the Request URL is https://www.singaporepools.com.sg/DataFileArchive/Lottery/Output/toto_next_draw_estimate_en.html?v=2021y11m14d21h0m and the Request Method is GET and agian the Content-Type is text/html.
So, with the URL and the HTTP method we found, here is the code to scrape that HTML:
val document = Jsoup
.connect("https://www.singaporepools.com.sg/DataFileArchive/Lottery/Output/toto_next_draw_estimate_en.html?v=2021y11m14d21h0m")
.userAgent("Mozilla")
.get()
val targetElement = document
.body()
.children()
.single()
val phrase = targetElement.child(0).text()
val prize = targetElement.select("span").text().removeSuffix(" est")
println(phrase) // Next Jackpot $8,000,000 est
println(prize) // $8,000,000
Here is another solution for parsing a dynamic page with Selenium and jsoup.
We first get and store the page with Selenium and then parse it with jsoup.
Just make sure to download the browser driver and move its executable file to your classpath.
I downloaded the Chrome driver version 95 and placed it along my Kotlin .kts script.
System.setProperty("webdriver.chrome.driver", "chromedriver.exe")
val result = File("output.html")
// OR FirefoxDriver(); download its driver and set the appropriate system property above
val driver = ChromeDriver()
driver.get ("https://www.singaporepools.com.sg/en/product/sr/Pages/toto_results.aspx")
result.writeText(driver.pageSource)
driver.close()
val document = Jsoup.parse(result, "UTF-8")
val targetElement = document
.body()
.children()
.select(":containsOwn(Next Jackpot)")
.single()
.parent()!!
val phrase = targetElement.text()
val prize = targetElement.select("span").text().removeSuffix(" est")
println(phrase) // Next Jackpot $8,000,000 est
println(prize) // $8,000,000
Another version of code for getting the target element:
val targetElement = document
.body()
.selectFirst(":containsOwn(Next Jackpot)")
?.parent()!!
I only used the following dependencies:
org.seleniumhq.selenium:selenium-java:4.0.0
org.jsoup:jsoup:1.14.3
See the standalone script file. It can be executed with Kotlin runner from command line like this:
kotlin my-script.main.kts