Cant get element generated by javaScript with Selenium - selenium

I want to bypass a website which uses cloudflare's botdetection and DDos-protection. I am using selenium. when the page load request is sent, the page starts loading and an element appears on the screen.
the html looks like this:
the white box is a span named 'mark' and the hidden blue box is a 'input' tag with type 'checkbox'.
I tried to get the elements it with usual ways like driver.find_elements(By.CLASS_NAME, "class") also xpath, cssSelector and JsPath but didnt work. note I have waited manually to element fully appear on the screen and the problem is not about waiting for element to be loaded.
because it was generated by js so I tried element = driver.execute_script("return document.querySelector('$cssSelector')") pattern. also with xpath and JsPath. they also didnt work and elements were not found.
the code:
markSpanJsPath = driver.execute_script("return document.querySelector('#cf-stage > div.ctp-checkbox-container > label > span')")
if markSpanJsPath:#kkk
print('found markSpanJsPath')
driver.execute_script("arguments[0].click();", markSpanJsPath)
print('found markSpanJsPath js click')
markSpanXpath = driver.find_elements(By.XPATH,'//*[#id="cf-stage"]/div[6]/label/span')
if markSpanXpath:
print('found markSpanXpath')
driver.execute_script("arguments[0].click();", markSpanXpath)
print('found markSpanXpath js click')
printed nothing.
so how to click on one of 'mark span' or 'checkbox' to pass the 'human verification'?

First of all it should be find_element() not find_elements()
use below xpath to identify and click.
markSpanXpath = driver.find_element(By.XPATH,'//label[#class="ctp-checkbox-label"]//span[#class="mark"]')
markSpanXpath.click()
if above click doesn't work try java scripts.
driver.execute_script("arguments[0].click();", markSpanXpath)

Related

AttributeError: 'WebElement' object has no attribute 'Click' error trying to Click on a link using Selenium Python

So I am in the process of trying to reach a search page which involves clicking on a clickable link on the bottom of the page. My code seems to be able to find the link or at least not throw an error when attempting to, however I get the error "AttributeError: 'WebElement' object has no attribute 'Click'" even though the element is physically clickable on the page.
Here is the code and website.
driver = webdriver.Edge(r'C:/Users/User/Desktop/Anaconda/edgedriver_win32/msedgedriver')
driver.get("https://www.canada.ca/en/environment-climate-change/services/species-risk-public-registry/cosewic-assessments-status-reports")
#click on the "Search COSEWIC status reports" button
element = WebDriverWait(driver, 10).until(
EC.presence_of_element_located((By.LINK_TEXT, "Search COSEWIC status reports"))
)
link = driver.find_element_by_link_text("Search COSEWIC status reports");
link.Click();
If I am wrong about this element being clickable please let me know. To be clear I am trying to click on the link "Search COSEWIC status reports found at the bottom of the webpage "https://www.canada.ca/en/environment-climate-change/services/species-risk-public-registry/cosewic-assessments-status-reports"
Update
I have found a workaround but the question still remains.
I have run into another attribute that needs clicking and it doesn't seem to have attributes 'id' or anything easy to identify by.
<span data-v-7ee3c58f="" class="name-primary">COSEWIC Status Appraisal Summary on the Pacific Water Shrew <em>Sorex bendirii</em> in Canada</span>
I have tried copying the XPath to this element and the id within the XPath but they don't seem to work. this is the first result on the page. "https://species-registry.canada.ca/index-en.html#/documents?documentTypeId=18&sortBy=documentTypeSort&sortDirection=asc&pageSize=10&keywords=pacific%20water%20shrew"
Your language bing is python so instead of Click() you need to use click()
Additionally, to click on a clickable element instead of presence_of_element_located() you need to induce WebDriverWait for the element_to_be_clickable() and you can use either of the following Locator Strategies:
driver = webdriver.Edge(r'C:/Users/User/Desktop/Anaconda/edgedriver_win32/msedgedriver')
driver.get("https://www.canada.ca/en/environment-climate-change/services/species-risk-public-registry/cosewic-assessments-status-reports")
#click on the "Search COSEWIC status reports" button
WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.LINK_TEXT, "Search COSEWIC status reports"))).click()

WebDriverWait.until.expected_conditions.presence_of_element_located not waiting for reloaded DOM

I have an app with 2 buttons. Button A takes the app to a new page with no buttons, then returns to the page with 2 buttons. My automated test is meant to click on Button A, wait while the app heads to the new pages and returns, then click on Button B.
The code:
el05a = WebDriverWait(driver, 120).until(
expected_conditions.presence_of_element_located((By.ID, "id_of_button_a"))
)
el05a.click()
el05b = WebDriverWait(driver, 120).until(
expected_conditions.presence_of_element_located((By.ID, "id_of_button_b"))
)
el05b.click()
However, I receive a StaleElementReferenceException about button B not being in DOM anymore.
Obviously, button B is not gonna be in the DOM while the app is at the new page, but why does my code not know to wait until the presence of button B is located? I thought presence_of_element_located means the code would be on hold until the element is located.
I know this could "technically" be patched with a time.sleep module but I'm trying to avoid that.
As per your query it seems likes as your checking presence_of_element_located and which only check for it presence and not the visibility of the element.
Try replacing the presence_of_element_located with visibility_of_element_located.
There is difference between visibility_of_element_located and presence_of_element_located.
1) visibility_of_element_located
Checking that an element is present on the DOM of a page and visible. Basically it tests if the element we are looking for is present as well as visible on the page.
2) presence_of_element_located
Checking that an element is present on the DOM of a page. Basically it tests if the element we are looking for is present somewhere on the page.
Code:
el05a = WebDriverWait(driver, 120).until(
expected_conditions. visibility_of_element_located((By.ID, "id_of_button_a"))
)
el05a.click()
el05b = WebDriverWait(driver, 120).until(
expected_conditions. visibility_of_element_located((By.ID, "id_of_button_b"))
)
el05b.click()
visibility_of_element_located: Returns the WebElement once it is located and visible.
An expectation for checking that an element is present on the DOM of a page and visible. Visibility means that the element is not only displayed but also has a height and width that is greater than 0.
presence_of_element_located: Returns the WebElement if element is present on DOM and not even visible.
An expectation for checking that an element is present on the DOM of a page. This does not necessarily mean that the element is visible.
Please change it from
expected_conditions.presence_of_element_located((By.ID, "id_of_button_a"))
to
expected_conditions.visibility_of_element_located((By.ID, "id_of_button_a"))

XPATH for Dynamic Drop Down Arrows based on Selenium GUI

I am new to this GUI Scripting.... My requirement is I want to click on the drop down arrow, & inside there are some elemets with tree structure to select the particular node/alarm..
So Intially I need to crack the XPATH for the drop down arrow of "COUNTERS" enter image description here
enter image description here
Tired Xpath by using absolute Xpath as below :
WebDriverWait(self.driver, 70).until(
EC.element_to_be_clickable((By.XPATH,
"/html/body/div/pm-application/epic-layout/div/div/div/div[5]/div[4]/epic-layout-panel/div/div/div/pm-reporting/epic-layout/div/div/div/div[3]/div[4]/epic-layout-panel/div/pm-tree-wrapper/div/div/div[1]/div[2]/div[1]/div[2]/div/div[2]/div/div[1]/div/div/div[3]/span/i")))
Also with Relative paths as well :
WebDriverWait(self.driver, 70).until(
EC.element_to_be_clickable((By.XPATH, "//div[#id='1483f603-ed09-6586-16e8-fc074ea8f908']//span//i[#class='icon tree-node-arrow collapsed']")))
WebDriverWait(self.driver, 70).until(
EC.element_to_be_clickable((By.XPATH,
"//*[#class='virtual-tree']//*[#class='icon tree-node-arrow collapsed']")))
WebDriverWait(self.driver, 70).until(
EC.element_to_be_clickable((By.XPATH,
"//i[#class='icon tree-node-arrow expanded']")))
But still I am facing below issue , Please help me to resolve this issues
AttributeError: 'str' object has no attribute 'find_element'
Please read about CSS selectors and Xpath selectors. Both of them don't like white or empty spaces.
It is a good practice to give Selenium the element actual tag, and not just asterisk sign.
Here is an example of the find element using CSS selector:
"div[class*='tree-node']"
This selector will return all the div elements that their class attribute contains the phrase "tree-node".
Using xpath you can find element by their attributes as well, or by the text written inside them:
"//div[contains(text(),'333')]"
This selector will give all div elements that have the text 333 inside them

Selecting elements using xpath

So very new here to Selenium but I'm having trouble selecting the element I want from this website. In this case, I got the x_path using Chrome's 'copy XPath tool.' Basically, I'm looking to extract the CID text (in this case 4004) from the website, but my code seems to be unable to do this. Any help would be appreciated!
I have also tried using the CSS selector method as well but it returns the same error.
chrome_options = Options()
chrome_options.add_argument("--headless")
chrome_options.binary_location = '/Applications/Google Chrome Canary.app/Contents/MacOS/Google Chrome Canary'
driver= webdriver.Chrome()
chem_name = "D008294"
url = "https://pubchem.ncbi.nlm.nih.gov/#query=" + chem_name
driver.get(url)
elements = driver.find_elements_by_xpath('//*[#id="collection-results-container"]/div/div/div[2]/ul/li/div/div/div/div[2]/div[2]/div[2]/span/a/span/span')
driver.close()
print(elements.text)
As of now, this is the error I receive: 'list' object has no attribute 'text'
Here is the xpath that you can use.
//span[.='Compound CID']//following-sibling::a/descendant::span[2]
Why your script did not worked: I 2 issues in your code.
elements = driver.find_elements_by_xpath('//*[#id="collection-results-container"]/div/div/div[2]/ul/li/div/div/div/div[2]/div[2]/div[2]/span/a/span/span')
driver.close() # <== don't close the browser until you are done with all your steps on the browser or elements
print(elements.text) # <== you can not get text from list (python will through error here
How to fix it:
CID = driver.find_element_by_xpath("//span[.='Compound CID']//following-sibling::a/descendant::span[2]").text # <== returning the text using find_element (not find_elements)
driver.close()
print(CID) # <== now you can print `CID` though browser closed as the value already stored in variable.
Function driver.find_elements_by_xpath return list of Element. You should loop to get text of each element,
Like this:
for ele in print(elements.text):
print(ele.text)
Or if you want to match first Element, use driver.find_element_by_xpath function instead.
Using xpath provided chrome is always does not work as expected. First you have to know how to write xpath and verify it chrome console.
see these links, which helps you to know about xpaths
https://www.guru99.com/xpath-selenium.html
https://www.w3schools.com/xml/xpath_syntax.asp
In this case, first find the span contains text Compound CID and move to parent span the down to child a/span/span. something like //span[contains(text(),'Compound CID']/parent::span/a/span/span.
And also you need to findelement which return single element and get text from it. If you use findelements then it will return list of elements, so you need to loop and get text from those elements.
xpath: //a[contains(#href, 'compound')]/span[#class='breakword']/span
you can use the "href" as your attribute reference since I noticed that it has unique value for each component.
Example:
href="https://pubchem.ncbi.nlm.nih.gov/substance/53790330"
href="https://pubchem.ncbi.nlm.nih.gov/compound/4004"

HTML code - how to find out absolute position on the page?

Is there any browser engine or plugin which would give user information about position of given HTML element ? I want to know where is element located e.g. left corner or center of the page.
It should not be a huge problem as Firefox and Chrome marks you elements within page as you go through html code in Developer Tools > "Element tab".
Example of highlighted element : http://imgur.com/mUHd51q we see that selected element is currently in the centre of the screen - how to get this information programatically ?
Selenium-webdriver can give you information about any DOM element you want:
d = Selenium::WebDriver.for :chrome
d.get "http://www.google.com"
elem = d.find_element(:name, "btnI")
elem.location
=> #<struct Selenium::WebDriver::Point x=532, y=356>