Hi I am looking for an explanation on how selenium searches for an element on a website. For us, we use inspect element to find the id, name, xpath, etc. of an element, and then put it in selenium for some action to be done. How does selenium find the element we tell it to find? Does it ctrl shift J like us and inspect element?
Note: I am not looking for how to code selenium to find element.
Html pages are just documents with elements structured like in a tree.
In general
Selenium uses element locators to find things. Locators work lazily. When you look up an element Selenium first checks if its cached. If not, it uses SearchContext which finds all elements within the current context (eg. DOM element) using a given mechanism, for example by XPathEvaluator.
SearchContext runs findElement() if you are looking for one element or findElements() if you are looking for more than one.
In simple terms, findElement() tries to run JavaScript script to find the element asynchronously. If it can’t, it tries to find it directly by using an interestingly called method – xpathWizardry, i.e. by using XPathEvaluator evaluation.
XPath
When you use XPath (XML Path Language) in Selenium, this is just a way to navigate through hierarchical structure of an XML-like document, such as html.
XPath uses a non-XML syntax to provide a flexible way of pointing to different parts of an XML document.
Internally selenium uses W3 XPathEvaluator, which evaluates XPath expressions.
You can study XPathEvaluator source code here.
Search Context
The SearchContext is a topmost interface present in the Selenium WebDriver hierarchy. It has two methods that will be the abstract as SearchContext is an interface.
findElement(): Find the first WebElement using the given method.
WebElement findElement(By by)
Parameters:
by - The locating mechanism
Returns:
The first matching element on the current context
Throws:
NoSuchElementException - If no matching elements are found
findElements(): Find all elements within the current context using the given mechanism.
java.util.List<WebElement> findElements(By by)
Parameters:
by - The locating mechanism to use
Returns:
A list of all WebElements, or an empty list if nothing matches
The browser DOM exposes API like querySelector,querySelectorAll, getElementById, getElementsByClassName, getElementsByName, etc through javascript that can be used to locate elements.
For example :
Navigate to www.bing.com
Press F12 to open developer console.
Enter document.querySelector("#sb_form_q") to locate search box input by css selector. I am using #Id here as css selector.
Enter document.getElementById("sb_form_q") to locate search box input by it's Id
Enter document.getElementsByClassName("sb_form_q")[0] to locate search box input by it's class name
Enter document.getElementsByName("q")[0] to locate search box input by it's name
All of above should return "<input id="sb_form_q" class="sb_form_q" name="q" type="search" maxlength="1000" autocomplete="off" aria-label="Enter your search term" autofocus="" aria-controls="sw_as" aria-autocomplete="both" aria-owns="sw_as" aria-activedescendant="sa_5004">" same result.
Selenium uses these DOM API to retrieve the elements. However, selenium might use these DOM API via some other mechanism (e.g C ++) and not by executing javascript for faster execution. XPath lookup is something not supported directly by browser DOM API. Selenium probably provides it's own implementation for XPath lookup or rely on some browser polyfill for this functionality.
Related
I was using selenium IDE to automate testing on a web page that has dynamic xpath in it.
I noticed selenium IDE was capturing the xpath fine the first time playing it. Then after closing the browser and opening, of course the xpath has changed, but the target saved was the old xpath.
Is there a way to handle this in selenium?
I know I can use the .contains method but can i apply that to the target?
Picture of selenium IDE firefox extension
To identify the dynamic elements you can construct dynamic locators. As couple of examples:
Using a css for a <span> tag with id attribute starting with abc:
span[id^='abc']
Using a css for a <span> tag with class attribute containing pqr:
span[class*='pqr']
Using a xpath for a <span> tag with value attribute ending with xyz:
span[value$='xyz']
Using a xpath for a <span> tag with id attribute starting with abc:
//span[starts-with(#id, 'abc')]
Using a xpath for a <span> tag with class attribute containing pqr:
//span[contains(#class, 'pqr')]
Explanation of the dynamic CSS_SELECTOR
The wildcards are defined as follows:
^ : To indicate an attribute value starts with
* : To indicate an attribute value contains
$ : To indicate an attribute value ends with
References
You can find a couple of relevant detailed discussions in:
Java Selenium webdriver expression finding dynamic element by ccs that starts with and ends with
There are few elements in an UI page without ID. I will download a particular version and then save all the current element tags and give to script as input, since few elements are not having id, this is causing script failures.
How can I locate the element without using Xpath.
Is there any simple way when there is no fixed id.
The short answer is "no." Sorry. All of the usual ways (by id, class, etc.) are relying on the same css information to locate elements. Xpath just shows all the ugly plumbing out in public. I don't think xpath has ever been described as "simple" but there is usually a way, using xpath, to find any element.
Xpath can be intimidating. Start with a plugin that will generate the xpath for you, once you click on an element. Usually the xpath generated will be extremely long and inefficient, but with practice you can see what can be trimmed and what is crucial. And to do that, also use a plugin that will "check" your xpath to see if it can find the element. Once you can find it (and ONLY the element you want) try trimming it to see if you can still find it with the abbreviated xpath locator.
reference ImageDon't be afraid of Xpaths. It's relatively easy to grab an Xpath using the Google Chrome browser. Navigate to your page and open Developer tools. Right-click on the particular tag for which you need an Xpath. Copy -> Xpath
i want to identify the web element for the selenium script using any plug in example in firefox we use a firepath . is plug in for chrome to identify the webelement .i have tried with developers tool but using developers tool not getting clear locator?
In most browsers just right-click on chosen element on webpage and choose 'Inspect'/'Inspect element' etc and the developer tools would open with marked HTML code for that element.
If that html code didn't show you unique ID for that element it means that you need to:
use xpath to find element by unique text inside (dont use xpath is slow)
or css selector to find it by unique attribute of that element
or asks dev team to add proper unique id/attibute to that element, and thats the best option
The documentation on code.google.com describes the elementIdElement functionality as "Search for an element on the page, starting from the identified element". Does this mean the search is done for every element following that element throughout the rest of the web page or only for dependents of that element?
If it is the former, then how would I construct the "value" entry if the "using" parameter is "css selector" and I want to find a descendant of the current element's sibling? I thought the value would be "+ div .classname", but this doesn't seem to work.
One way you can do it is by emulating Selenium ByChained, by calling one wait after another:
browser
.useXpath()
.url('http://www.google.com')
.waitForElementVisible('.//body', 1000)
.waitForElementVisible(".//div[contains(#class, 'classVar')]")
.click('button')
The correct xpath string for navigating to an element's parent's following sibling for the existence of another element with a given class is
'(//div[#id="currentElementID"]/../following-sibling::*[1]//div[contains(#class,"classToLookFor")])[1]'
In this example, I know the current element's ID so that's what I use, but the '//div[#id="currentElementID"]' can be replaced with whatever you need to navigate to the starting element. Also, this example assumes the element I'm looking for is a div.
In my WebDriver project, I have planned to add Jsoup to get 'parent' and 'siblings' and few other features. I need to find an element through Jsoup and click its parent using WebDriver. That means I need to convert a Joup element object to WebElement object. Please let me know how I can do this, if this is feasible.
If it is not possible to integrate Jsoup and WebDriver in such way, then please discuss on how I can get parents and all using WebDriver.
Also, is it possible to list ALL possible elements present under a particular WebElement?
It's quite interesting we're doing the similar approach, integrating JSoup and Selenium WebDriver. I can understand your issue especially dealing with some dynamic website based on some Javascript framework which has no stable IDs or attributes.
Our solution looks like the following, and hopefully it could be some advice for you:
webDriver.getPageSource() to get the current HTML source
use JSoup to parse this HTML source, and leverage Jsoup selector (which is much more powerful than Selenium) to locate the target element
get parents or siblings of this element
write an iteration function to get element xPath, such as //body/div[2]/form[1]/input[3]
webDriver.findElement(By.xpath(...)) to locate element in selenium context
EDITED
The idea of the iteration function is:
first check the tag of your parent node, if it is body, then iteration ends
if not , then use getSiblings to check the index of the node among all the nodes with same tag, e.g, the 3rd div, then equals to div[3]
iterate to your parent node, and do the same procedures
Once you get the xpath of the child node, and parent node, just replace parent node xpath to be empty string inside the child node xpath, finally you can get the relative xpath.
You can use xpath selectors to select parent and child elements
Related questions
Select parent using xpath
XML xpath, get the parent element till a specific element
Getting child nodes using xpath?
What about running findElements with xpath : .//* on your particular element? Also, look into xpath parent::* and following-sibling::*. For the particular case I understand, there is no need for Jsoup.