I have an online calculator that i want to automate some operations for, like subtraction, division, etc. but the thing is that there are no elements since it is a canvas app calculator (link below). I'm wondering how can i click the buttons so im able to automate some operations?
The online calculator im trying to automate:
https://www.online-calculator.com/full-screen-calculator/
The canvas HTML code
<canvas id="canvas" width="335" height="434" style="position: absolute; display: block; background-color: rgb(255, 255, 255); width: 167.939px; height: 217px;"></canvas>
My Selenium-Java code
driver.get("https://www.online-calculator.com/full-screen-calculator/");
driver.manage().window().maximize();
WebElement li = driver.findElements(By.tagName("iframe")).get(0);
driver.switchTo().frame(li);
WebElement canv = driver.findElements(By.tagName("canvas")).get(0);
System.out.println(canv.getSize());
System.out.println(canv.getLocation());
try {
Actions builder = new Actions(driver);
builder.moveToElement(canv, x, y);
builder.click();
builder.perform();
} catch (Exception e)
{
// do nothing
}
So as you see in above code, I still have to find the x and y of the operations i want to run. For ex, how can i find the coordinates of the buttons 10, 4 and the subtraction operation '-', if I want to calculate '10 - 4' and then validate that it is equal to 6. Any help would be highly appreciated. Thanks :)
Note: The canvas width, height and location will change if the window size changes ((Im thinking later of locking the window size so my tests are not flaky on different screens)).
The <canvas> element is within an <iframe>. So to invoke click() on the elements within the <canvas> you have to:
Induce WebDriverWait for the desired frame to be available and switch to it.
Induce WebDriverWait for the desired element to be clickable.
You can use the following solution:
Code Block:
driver.get("https://www.online-calculator.com/full-screen-calculator/")
new WebDriverWait(driver, 10).until(ExpectedConditions.frameToBeAvailableAndSwitchToIt(By.id("fullframe")));
WebElement canvas = new WebDriverWait(driver, 10).until(ExpectedConditions.elementToBeClickable(By.id("canvas")));
HTML5 Canvas
The element is only a container for graphics and is a rectangular area on an HTML page. By default, a canvas has no border and no content. However, an id attribute (to be referred to in a script), a width and height attribute are specified to define the size of the canvas. To add a border, the style attribute is used. An example:
<canvas id="myCanvas" width="200" height="100" style="border:1px solid #000000;"></canvas>
The HTML canvas is a two-dimensional grid. The upper-left corner of the canvas has the coordinates (0,0).
In the article Automated Testing of HTML5 Canvas Applications with Selenium WebDriver #Aaron Mulder mentions, interacting with the elements within <canvas> is possible using event support of the Actions Class API:
moveToElement(WebElement target, int xOffset, int yOffset): Moves the mouse to an offset from the top-left corner of the element.
new Actions(driver).moveToElement(canvas, xWithinCanvas, yWithinCanvas).click().perform();
But is not 100% reliable as, every mouse down, mouse up, or mouse click happens at the center of the element. So the code above produces a mouse move event to the provided coordinates (x,y), then a mouse move event to the center of the <canvas>, then a mouse down, mouse up, and click at the center of the <canvas>. That should have been fine for a <button> but is not worth for a <canvas>, where you want to be able to click at a specific location.
The workaround, is to dispatch synthesized mouse events using JavaScript as follows:
// pageX and pageY are offsets which you need to know through mouse coordinates.
((JavascriptExecutor)driver).executeScript("var evt = $.Event('click', { pageX: " + x +
", pageY: " + (y + 55) + " } );" +
"$('#myCanvas').trigger(evt);");
However, to click on the elements within the <canvas> you can be at ease using firefox as the mouse move event works well in Firefox and you can avoid using the mouse coordinates as the event processing as follows:
new Actions(driver).moveToElement(
canvas, xWithinCanvas, yWithinCanvas).perform();
((JavascriptExecutor)driver).executeScript("$('#canvas').click();");
This usecase
To automate a substruction operation e.g. 3-1= using Selenium you can use the following solution:
Code Block:
driver.get("https://www.online-calculator.com/full-screen-calculator/");
new WebDriverWait(driver, 10).until(ExpectedConditions.frameToBeAvailableAndSwitchToIt(By.id("fullframe")));
WebElement canvas = new WebDriverWait(driver, 10).until(ExpectedConditions.elementToBeClickable(By.id("canvas")));
//clicking on 3
new Actions(driver).moveToElement(canvas, 0, 0).moveByOffset(0,(255/6)*3).click().build().perform();
//clicking on the substract sign (-)
new Actions(driver).moveToElement(canvas, 0, 0).moveByOffset((174/5)*2,(255/6)*3).click().build().perform();
//clicking on 1
new Actions(driver).moveToElement(canvas, 0, 0).moveByOffset(-(174/5)*4,(255/6)*3).click().build().perform();
//clicking on equals to sign (=)
new Actions(driver).moveToElement(canvas, 0, 0).moveByOffset((174/5)*4,(255/6)*4).click().build().perform();
Execution Video:
Reference
You can find a couple of relevant detailed discussion in:
How to perform Mouse Wheel scrolling over HTML5 Canvas in Selenium?
I think you are facing difficulty to find the coordinates of the element. To find them easily will have some plugins for each browser. For chrome, you can use Page Ruler and for Firefox, you can use MeasureIt. By using these tools you can get the coordinates of the particular element. After that, you can easily click on those elements. Watch the following video for how to use MeasureIt in firefox (follow from 11:45 minutes).
Finding the coordinates of particular Element using Firefox MeasureIt plugin
And your other concern is how to find the coordinates of the size of the screen changes? normally will have some standard screen sizes for each monitor or laptop.
So first, get the size of the screen using selenium and then you can use if the condition for each screen size. Means, if the size is something like 800*1200 then use these coordinates else use some other coordinates for each screen size.
I want to simulate a simple mouse click and drag within an svg element.
I managed to get the coordinates of my starting and ending point, both absolute (window coordinates) and relative to the encapsulating svg element.
Here is the code I am using to simulate the mouse:
Actions builder = new Actions(driver);
builder.moveToElement(area, xStart, yStart);
builder.clickAndHold();
builder.moveToElement(area, xStop, yStop);
builder.release();
Action setFilter = builder.build();
setFilter.perform();
Where area is a WebElement representing my svg and the coordinates are relative to that element. Note that:
area.getLocation(); // returns null
This made me wonder whether the webdriver is able to find that element at all. So I tried with absolute coordinates:
Actions builder = new Actions(driver);
builder.moveByOffset(chart.getLocation().x + xStart, chart.getLocation().y + yStart);
builder.clickAndHold();
builder.moveByOffset(xStop - xStart, yStop - yStart);
builder.release();
Action setFilter = builder.build();
setFilter.perform();
where chart is the div surrounding the svg element (note that the offset between the div position and the svg position is only 10 pixels and is not significant). That didn't work either and I also tried by relative position to the div but still no luck.
What am I doing wrong here?
I got a hack for it to work but it requires the webpage to be opened on the foreground. If you're doing anything else at the same time it may break the test and I cannot say if it would work if ran remotely.
Here's what it looks like:
Robot robert = new Robot();
robert.mouseMove(xStart, yStart);
// full click once to get focus on the window
robert.mousePress(MouseEvent.BUTTON1_MASK);
robert.mouseRelease(MouseEvent.BUTTON1_MASK);
// then set the filter
robert.mousePress(MouseEvent.BUTTON1_MASK);
robert.mouseMove(xStop, yStop);
robert.mouseRelease(MouseEvent.BUTTON1_MASK);
As part of my Selenium test for a login function, I would like to click a button by identifying its coordinates and instructing Selenium to click at those coordinates. This would be done without identifying the element itself (via id, xpath, etc).
I understand there are other more efficient ways to run a click command, but I'm looking to specifically use this approach to best match the user experience. Thanks.
There is a way to do this. Using the ActionChains API you can move the mouse over a containing element, adjust by some offset (relative to the middle of that element) to put the "cursor" over the desired button (or other element), and then click at that location. Here's how to do it using webdriver in Python:
from selenium import webdriver
from selenium.webdriver.common.action_chains import ActionChains
browser = webdriver.Chrome()
elem = browser.find_element_by_selector(".some > selector")
ac = ActionChains(browser)
ac.move_to_element(elem).move_by_offset(x_offset, y_offset).click().perform()
Y'all are much to quick to dismiss the question. There are a number of reasons one might to need to click at a specific location, rather than on an element. In my case I have an SVG bar chart with an overlay element that catches all the clicks. I want to simulate a click over one of the bars, but since the overlay is there Selenium can't click on the element itself. This technique would also be valuable for imagemaps.
In C# API you use actions
var element = driver.FindElement(By...);
new Actions(driver).moveToElement(element).moveByOffset(dx, dy).click().perform();
Although it is best to just use simple Id, CSS, Xpath selectors when possible. But the functionality is there when needed (i.e. clicking elements in certain geographic places for functionality).
I first used the JavaScript code, it worked amazingly until a website did not click.
So I've found this solution:
First, import ActionChains for Python & active it:
from selenium.webdriver.common.action_chains import ActionChains
actions = ActionChains(driver)
To click on a specific point in your sessions use this:
actions.move_by_offset(X coordinates, Y coordinates).click().perform()
NOTE: The code above will only work if the mouse has not been touched, to reset the mouse coordinates use this:
actions.move_to_element_with_offset(driver.find_element_by_tag_name('body'), 0,0))
In Full:
actions.move_to_element_with_offset(driver.find_element_by_tag_name('body'), 0,0)
actions.move_by_offset(X coordinates, Y coordinates).click().perform()
This can be done using Actions class in java
Use following code -
new Actions(driver).moveByOffset(x coordinate, y coordinate).click().build().perform();
Note: Selenium 3 doesn't support Actions class for geckodriver
Also, note that x and y co-ordinates are relative values from current mouse position. Assuming mouse co-ordinates are at (0,0) to start with, if you want to use absolute values, you can perform the below action immediately after you clicked on it using the above code.
new Actions(driver).moveByOffset(-x coordinate, -y coordinate).perform();
If using a commercial add-on to Selenium is an option for you, this is possible: Suppose your button is at coordinates x=123, y=456. Then you can use Helium to click on the element at these coordinates as follows:
from helium.api import *
# Tell Helium about your WebDriver instance:
set_driver(driver)
click(Point(123, 456))
(I am one of Helium's authors.)
This worked for me in Java for clicking on coordinates irrespective on any elements.
Actions actions = new Actions(driver);
actions.moveToElement(driver.findElement(By.tagName("body")), 0, 0);
actions.moveByOffset(xCoordinate, yCoordinate).click().build().perform();
Second line of code will reset your cursor to the top left corner of the browser view and last line will click on the x,y coordinates provided as parameter.
In Selenium Java, you can try it using Javascript:
WebDriver driver = new ChromeDriver();
if (driver instanceof JavascriptExecutor) {
((JavascriptExecutor) driver).executeScript("el = document.elementFromPoint(x-cordinate, y-cordinate); el.click();");
}
Action chains can be a little finicky. You could also achieve this by executing javascript.
self.driver.execute_script('el = document.elementFromPoint(440, 120); el.click();')
I used the Actions Class like many listed above, but what I found helpful was if I need find a relative position from the element I used Firefox Add-On Measurit to get the relative coordinates.
For example:
IWebDriver driver = new FirefoxDriver();
driver.Url = #"https://scm.commerceinterface.com/accounts/login/?next=/remittance_center/";
var target = driver.FindElement(By.Id("loginAsEU"));
Actions builder = new Actions(driver);
builder.MoveToElement(target , -375 , -436).Click().Build().Perform();
I got the -375, -436 from clicking on an element and then dragging backwards until I reached the point I needed to click. The coordinates that MeasureIT said I just subtracted. In my example above, the only element I had on the page that was clickable was the "loginAsEu" link. So I started from there.
If all other methods mentioned on this page failed and you are using python, I suggest you use the mouse module to do it in a more native way instead. The code would be as simple as
import mouse
mouse.move("547", "508")
mouse.click(button='left')
You can also use the keyboard module to simulate keyboard actions
import keyboard
keyboard.write('The quick brown fox jumps over the lazy dog.')
To find the coordinate of the mouse, you can use the follow JavaScript Code.
document.onclick=function(event) {
var x = event.screenX ;
var y = event.screenY;
console.log(x, y)
}
If you don't like screenX, you can use pageX or clientX. More on here
P.S. I come across a website that prevents programmatic interactions with JavaScript/DOM/Selenium. This is probably a robot prevention mechanism. However, there is no way for them to ban OS actions.
WebElement button = driver.findElement(By.xpath("//button[#type='submit']"));
int height = button.getSize().getHeight();
int width = button.getSize().getWidth();
Actions act = new Actions(driver);
act.moveToElement(button).moveByOffset((width/2)+2,(height/2)+2).click();
import pyautogui
from selenium import webdriver
driver = webdriver.Chrome(chrome_options=options)
driver.maximize_window() #maximize the browser window
driver.implicitly_wait(30)
driver.get(url)
height=driver.get_window_size()['height']
#get browser navigation panel height
browser_navigation_panel_height = driver.execute_script('return window.outerHeight - window.innerHeight;')
act_y=y%height
scroll_Y=y/height
#scroll down page until y_off is visible
try:
driver.execute_script("window.scrollTo(0, "+str(scroll_Y*height)+")")
except Exception as e:
print "Exception"
#pyautogui used to generate click by passing x,y coordinates
pyautogui.FAILSAFE=False
pyautogui.moveTo(x,act_y+browser_navigation_panel_height)
pyautogui.click(x,act_y+browser_navigation_panel_height,clicks=1,interval=0.0,button="left")
This is worked for me. Hope, It will work for you guys :)...
I used AutoIt to do it.
using AutoIt;
AutoItX.MouseClick("LEFT",150,150,1,0);//1: click once, 0: Move instantaneous
Pro:
simple
regardless of mouse movement
Con:
since coordinate is screen-based, there should be some caution if the app scales.
the drive won't know when the app finish with clicking consequence actions. There should be a waiting period.
To add to this because I was struggling with this for a while. These are the steps I took:
Find the coordinates you need to click. Use the code below in your console and it will display and alert of the coordinates you need.
document.onclick = function(e)
{
var x = e.pageX;
var y = e.pageY;
alert("User clicked at position (" + x + "," + y + ")")
};
Make sure the element is actually visible on the screen and click it using Actionchains
WebDriverWait(DRIVER GOES HERE, 10).until(EC.element_to_be_clickable(YOUR ELEMENT LOCATOR GOES HERE))
actions = ActionChains(DRIVER GOES HERE)
actions.move_by_offset(X, Y).click().perform()
Selenium::WebDriver has PointerActions module which includes a number of actions you can chain and/or trigger without specifying the element, eg. move_to_location or move_by. See detailed specs here. As you can see, you can only use actual coordinates. I am sure this interface is implemented in other languages / libraries accordingly.
My short reply may echo some other comments here, but I just wanted to provide a link for reference.
You could use the html tag as the element and then use the coordinates you want, in this example below I am using Javascript. But I am able to click on the top left of the screen.
async function clickOnTopLeft() {
const html = driver.wait(
until.elementLocated(By.xpath('/html')),
10000)
const { width, height } = await html.getRect()
const offSetX = - Math.floor(width / 2)
const offsetY = - Math.floor(height / 2)
await driver.actions().move(
{ origin: html, x: offSetX, y: offsetY })
.click().perform()
}
If you can see the source code of page, its always the best option to refer to the button by its id or NAME attribute. For example you have button "Login" looking like this:
<input type="submit" name="login" id="login" />
In that case is best way to do
selenium.click(id="login");
Just out of the curiosity - isnt that HTTP basic authentification? In that case maybe look at this:
http://code.google.com/p/selenium/issues/detail?id=34
Selenium won't let you do this.