Recording Selenium tests for functional Plone test cases - selenium

Question

I'd like to use Selenium for creating simple functional tests against Plone add-ons. The main driver here is that non-programmers can create and understand test cases with some effort, because they see in a web browser what's happening. What is the recommended best practice to
The test case prepares Plone site environment where the test will be run (installs add-ons, mocks up mail host, creates sample content)
How to run Plone functional test case to the point you can start a Selenium recording in a browser and how to open a browser with recording enabled?
How later run the recorded test output from Python code?
Are there other test recording frameworks out there which you combine with Plone? Able to tests against Javascripted UI is a requirement.

Answer 1

My guess is that there are tools that do the individual steps independently, but those tools don't work together exactly as you describe.
In my experience, the quality of recorded tests is so bad that a programmer will have to rewrite them anyway. It only gets worse when you have a lot of JavaScript. If in addition, you have a site that uses AJAX, one of the problems that occurs is that you will sometimes have to wait for a specific element to appear, before doing the next click, and this is where most recorders will fail.
I would also love to hear about a tool that is targeted to end users, and allows them to record and run Plone tests on their own, and if anyone knows about this kind of project, I would really like to get involved in its development.

Answer 2

plone.app.testing comes with seleniumtestlayer since 4.1.
Here is my own more sophisticated helper code:
"""
Some PSE 2012 Selenium notes
* https://github.com/plone/plone.seleniumtesting
* https://github.com/emanlove/pse2012
Selenium WebDriver API
* http://code.google.com/p/selenium/source/browse/trunk/py/selenium/webdriver/remote/webdriver.py
Selenium element match options
* http://code.google.com/p/selenium/source/browse/trunk/py/selenium/webdriver/common/by.py
Selenium element API (after find_xxx())
* http://code.google.com/p/selenium/source/browse/trunk/py/selenium/webdriver/remote/webelement.py
You can do pdb debugging using ``selenium_helper.selenium_error_trapper()`` if you run
tests with ``SELENIUM_DEBUG`` turned on::
SELENIUM_DEBUG=true bin/test -s testspackage -t test_usermenu
Then you'll get debug prompt on any Selenium error.
"""
import os
# Try use ipdb debugger if we have one
try:
import ipdb as pdb
except ImportError:
import pdb
from selenium.webdriver.common.by import By
from selenium.common.exceptions import NoSuchElementException
from selenium.common.exceptions import WebDriverException
from selenium.webdriver.support.wait import WebDriverWait
from plone.app.testing import selenium_layers
SELENIUM_DEBUG = "SELENIUM_DEBUG" in os.environ
class SeleniumTrapper(object):
"""
With statement for break on Selenium errors to ipdb if it has been enabled for this test run.
"""
def __init__(self, driver):
self.driver = driver
def __enter__(self):
pass
def __exit__(self, type, value, traceback):
"""
http://effbot.org/zone/python-with-statement.htm
"""
if isinstance(value, WebDriverException) and SELENIUM_DEBUG:
# This was Selenium exception
print "Selenium API call failed because of browser state error: %s" % value
print "Selenium instance has been bound to self.driver"
pdb.set_trace()
class SeleniumHelper(object):
"""
Selenium convenience methods for Plone.
Command Selenium browser to do common actions.
This mainly curries and delegates to plone.app.testing.selenium_layers helper methods.
More info:
* https://github.com/plone/plone.app.testing/blob/master/plone/app/testing/selenium_layers.py
"""
def __init__(self, testcase, driver=None):
"""
:param testcase: Yout test class instance
:param login_ok_method: Selenium check function to run to see if login success login_ok_method(selenium_helper)
"""
self.testcase = testcase
if driver:
# Use specific Selenium WebDriver instance
self.driver = driver
else:
# plone.app.tesrting selenium layer
self.driver = testcase.layer['selenium']
self.portal = testcase.layer["portal"]
def selenium_error_trapper(self):
"""
Create ``with`` statement context helper which will invoke Python ipdb debugger if Selenium fails to do some action.
If you run test with SELENIUM_DEBUG env var set you'll get dropped into a debugger on error.
"""
return SeleniumTrapper(self.driver)
def reset(self):
"""
Reset Selenium test browser between tests.
"""
def login(self, username, password, timeout=15, poll=0.5, login_cookie_name="__ac", login_url=None):
"""
Perform Plone login using Selenium test browser and Plone's /login_form page.
"""
submit_button_css = '#login_form input[name=submit]'
if not login_url:
# Default Plone login URL
login_url = self.portal.absolute_url() + '/login_form'
with self.selenium_error_trapper():
submit_button = self.open(login_url, wait_until_visible=submit_button_css)
self.find_element(By.CSS_SELECTOR, 'input#__ac_name').send_keys(username)
self.find_element(By.CSS_SELECTOR, 'input#__ac_password').send_keys(password)
submit_button.click()
# Check that we get Plone login cookie before the timeout
waitress = WebDriverWait(self.driver, timeout, poll)
matcher = lambda driver: driver.get_cookie(login_cookie_name) not in ["", None]
waitress.until(matcher, "After login did not get login cookie named %s" % login_cookie_name)
def logout(self, logout_url=None):
"""
Perform logout using Selenium test browser.
:param logout_url: For non-default Plone logout view
"""
if not logout_url:
logout_url = self.portal.absolute_url() + "/logout"
self.open(logout_url)
def get_plone_page_heading(self):
"""
Get Plone main <h1> contents as lowercase.
XXX: Looks like Selenium API returns uppercase if there is text-transform: uppercase?
:return: Empty string if there is no title on the page (convenience for string matching)
"""
try:
title_elem = self.driver.find_element_by_class_name("documentFirstHeading")
except NoSuchElementException:
return ""
if not title_elem:
return ""
return title_elem.text.lower()
def trap_error_log(self, orignal_page=None):
"""
Read error from the site error log and dump it to output.
Makes debugging Selenium tests much more fun when you directly see
the actual errors instead of OHO.
:param orignal_page: Decorate the traceback with URL we tried to access.
"""
# http://svn.zope.org/Zope/trunk/src/Products/SiteErrorLog/SiteErrorLog.py?rev=96315&view=auto
error_log = self.portal.error_log
entries = error_log.getLogEntries()
if len(entries) == 0:
# No errors, yay!
return
msg = ""
if orignal_page:
msg += "Plone logged an error when accessing page %s\n" % orignal_page
# We can only fail on traceback
if len(entries) >= 2:
msg += "Several exceptions were logged.\n"
entry = entries[0]
raise AssertionError(msg + entry["tb_text"])
def is_error_page(self):
"""
Check that if the current page is Plone error page.
"""
return "but there seems to be an error" in self.get_plone_page_heading()
def is_unauthorized_page(self):
"""
Check that the page is not unauthorized page.
..note ::
We cannot distingush login from unauthorized
"""
# require_login <-- auth redirect final target
return "/require_login/" in self.driver.current_url
def is_not_found_page(self):
"""
Check if we got 404
"""
return "this page does not seem to exist" in self.get_plone_page_heading()
def find_element(self, by, target):
"""
Call Selenium find_element() API and break on not found and such errors if running tests in SELENIUM_DEBUG mode.
"""
with self.selenium_error_trapper():
return self.driver.find_element(by, target)
def find_elements(self, by, target):
"""
Call Selenium find_elements() API and break on not found and such errors if running tests in SELENIUM_DEBUG mode.
"""
with self.selenium_error_trapper():
return self.driver.find_elements(by, target)
def click(self, by, target):
"""
Click an element.
:param by: selenium.webdriver.common.by.By contstant
:param target: CSS selector or such
"""
with self.selenium_error_trapper():
elem = self.driver.find_element(by, target)
elem.click()
def open(self, url, check_error_log=True, check_sorry_error=True, check_unauthorized=True, check_not_found=True, wait_until_visible=None):
"""
Open an URL in Selenium browser.
If url does not start with http:// assume it is a site root relative URL.
:param wait_until_visible: CSS selector which must match before we proceed
:param check_error_log: If the page has created anything in Plone error log then dump this traceback out.
:param check_sorry_error: Assert on Plone error response page
:param check_unauthorized: Assert on Plone Unauthorized page (login dialog)
:return: Element queried by wait_until_visible or None
"""
elem = None
# Convert to abs URL
if not (url.startswith("http://") or url.startswith("https://")):
url = self.portal.absolute_url() + url
selenium_layers.open(self.driver, url)
if check_error_log:
self.trap_error_log(url)
if wait_until_visible:
elem = self.wait_until_visible(By.CSS_SELECTOR, wait_until_visible)
# XXX: These should be waited also
if check_sorry_error:
self.testcase.assertFalse(self.is_error_page(), "Got Plone error page for url: %s" % url)
if check_unauthorized:
self.testcase.assertFalse(self.is_unauthorized_page(), "Got Plone Unauthorized page for url: %s" % url)
if check_not_found:
self.testcase.assertFalse(self.is_not_found_page(), "Got Plone not found page for url: %s" % url)
return elem
def wait_until_visible(self, by, target, message=None, timeout=10, poll=0.5):
"""
Wait until some element is visible on the page (assume DOM is ready by then).
Wraps selenium.webdriver.support.wait() API.
http://selenium.googlecode.com/svn/trunk/docs/api/py/webdriver_support/selenium.webdriver.support.wait.html#module-selenium.webdriver.support.wait
"""
if not message:
message = "Waiting for element: %s" % target
waitress = WebDriverWait(self.driver, timeout, poll)
matcher = lambda driver: driver.find_element(by, target)
waitress.until(matcher, message)
elem = self.driver.find_element(by, target)
return elem
(Not on github yet)