Can anybody suggest a good tool to test GWT UI. What is the opinion about using Selenium & Webdriver ?
Any pointers and suggestions are welcome.
Thanks,
SD
Google themselves have an excellent article describing when and how to apply different testing strategies to GWT apps. The basic idea is to use unit-testing whenever you are testing business logic, but to use Selenium for specific end-to-end tests where you want to simulate user interaction.
The most important thing is limit how much GWT-specific testing you need to do since GWTTestCase and Selenium are so slow. The MVP pattern referenced in the Testing Methodologies helps enormously with this. Always ask yourself "Can I make this view any dumber?" Push logic for dynamically changing the UI into presenters that manipulate views so that you can test the logic in JUnit.
Then I'd suggest doing most of the UI testing in Selenium/Webdriver. Those tests should be fairly simple though: "Does X show up when I click Y?" kinds of things. GWTTestCase ends up being the most useful for me for code that needs to run under Java and GWT.
Or, as an alternative to the MVP pattern, you may use the gwt-test-utils framework to write simple tests in java, which runs faster then GWTTestCase tests :-)
You can do bog standard unit testing for your GWT components, if you want to do some layout checking things which can't be done through unit tests (can't give you examples since I don't know GWT nearly at all) then you may use Selenium, however Selenium does have some issues in specific cases, namely selecting elements and very rich content which may prove to be hard to handle since GWT is all about rich content.
Related
I have been reading a lot about test-driven development and decided that I want to give it a go on a small project. For reference, I am currently reading 'Growing Object-Oriented Software, Guided by Tests'.
I understand how to unit test my application and how to unit test certain parts of the UI as well, but I am struggling to set up end-to-end tests. For example, testing that a certain path through my whole application produces the correct output (this is my basic understanding of an end-to-end test).
It's not necessary to simulate click events, but it is necessary to have some sort of connection to the UI.
Am I right in thinking that I need a combination of "Logic" tests (test without launching the app), "Application" tests (test with launching the app) and the asynchronous functionality of something like GHUnit to accomplish this?
EDIT:
After reading some of the answers below, it sounds like I'm looking for functional end-to-end testing, but I think I should give an example of a test as I imagine it.
Start the application.
Call the login function with a test users credentials. (Note: doesn't necessarily need UI automation).
Verify a label on the window says "Logging In...".
After successfully verifying the user, verify the label now says "Welcome, Adam!".
KIF sounds like it could work, as it has steps to check changes in UI elements and it looks like there is a Mac OSX branch also. I'm sure I could also write a small class which constantly polls the UI for changes I expect and times-out after a certain time, but I'm wondering if this the right way to go about it.
However, perhaps I am trying to take what I am reading in 'Growing Object-Oriented Software, Guided by Tests' and trying to apply it too literally to Cocoa.
Another UPDATE:
So I've been reading the advice so far, checked the various places linked to and started to implement something whilst still referencing the book. I think what I'm really trying to get at is the Test-Driven-Development part. What stood out most in said book, was that they described what they wanted to happen from a users perspective first with acceptance tests.
I realise that solid unit testing will be necessary as soon as I start writing methods, but I was keen to write some high-level acceptance tests first, using some of the UI. I have started to write my own application "driver" class, using some similar methods to the GHAsyncTestCase ideas to help me accomplish this. Does this sound correct/useful/necessary?
I really appreciate all the comments so far and they have definitely helped me work out in my own head what I'm trying to do and what various areas of testing there are. I will finish up this question soon, as it is getting rather large, so any final advice is welcome!
I think the key thing that I got from "Growing Object Orientated Software" was to decouple as much as possible from the UI. Without code to look at it's harder to give suggestions but with your revision I'd think that separating the "verify a label says.." bit from the UI. What is setting this message, and can you just test for that event?
The more you can decouple from the UI the more you can unit-test (quicker and easier) rather than integrating other frameworks or drivers of UI elements.
You might be interested in Square's KIF framework: http://corner.squareup.com/2011/07/ios-integration-testing.html
It looks really cool for integration/UI testing.
I believe you can use the Accessibility features to script the UI. I'd check the WWDC 2011 videos for one entitled "Design Patterns to Simplify Mac Accessibility". They did something similar in 2010.
Based on your response to #Norman, I guess you're looking for recommendations that span both functional end-to-end as well as UI-based end-to-end but perhaps a UI automation framework might change your mind? Something intrusive like FoneMonkey might be helpful:
http://www.gorillalogic.com/fonemonkey
If that doesn't work for you, I'd be interested in knowing why & what "gap" you perceive in such UI driven tests versus code-based functional testing?
We are using Selenium for functional testing our AJAX application. Although Selenium has been a great help for us, it still suffers the same problem functional testing always suffered: tests maintenance overhead. We have quite a lot of Java code that isn't easy to maintain when software changes. I know that other testing tools (like BadBoy an others) aren't very different in this respect. My question is: has there been any break-through in the functional testing industry recently? Or maybe there's an on-going promising research in this area?
One thing I know is that using page object pattern can reduce the level of maintenance for selenium tests.
keyword based test automation (Robot Framework)
Currently Selenium is one of the best tool available to Automate functional testing. But one must need to identify the potential use of any tool need to implement on any specific type of application. For AJAX based application if we use Selenium using Javascript language it would be better. Or we can use Selenoid (A replacement for Selenium as, it is easy to maintain) for AJAX based projects.
Using Selenium RC I am planing for creating an automation testing framework for our Java web-application.... please share some way like how to create framework ...which things is important in framework.....
Thanks in advance
-Ravi
There are a few key factors that would greatly increase the maintainability, speed, and general usability of your tests but none are more important than abstracting your element locators out of your tests and into a central repository. I've heard of many ways of doing that but the best option I've seen is using the Page Object pattern. Essentially, each page becomes a java class, with properties representing the elements you want to use. Because you only define this page once in your PageObject and not in your many tests, if you ever need to change an element locator, there's only one place to do it.
Selenium 2 has a wonderful PageObject Factory built in but you can't use that because you want to use Selenium RC. Fortunately The Automated Tester, David Burns, has an excellent article on using Page Objects in C#, it should be close enough to get you started. http://www.theautomatedtester.co.uk/tutorials/selenium/page-object-pattern.htm For more on the Selenium 2 / WebDriver page object implementation see teh Selenium Google Code Wiki http://code.google.com/p/selenium/wiki/PageObjects
Some other really important factors to consider
Use an external framework for Data Driven tests one of the most common questions on the Selenium Google Group is how to test run data driven tests. The answer is almost always to use the data driven features that came with your unit testing tool. TestNG has become really popular with Selemium users because of it's excellent support for Data Driven tests. There's even guidance on the TestNG site for Selenium
Parallelize your tests Because most tests drive a live browser (unless you're using HTMLUnit) tests will take some time. The more you can parallelize your tests across multiple machines the faster they will run. Sauce Labs has an excellent blog series on this
Screenshot on errors less common but also a good idea. Sometimes you need to look at your webpage when there's a problem
I've got analysis paralysis looking at all the different functional testing options for a new grails (v1.3.4) application. I've looked at
WebDriver/Selenium (which I've used before)
WebTest/Canoo
Geb
Tellurium
Grails Functional Test
and there must be others. I think some of the criteria that I would use to make a decision include (in no particular order):
Likely longevity, active development
Can do ajax/javascript
Support for PageObject or similar patterns
Maturity
Headless (eg htmlunit) is ok, especially if it makes things go faster
Good reporting
Support for NTLM credential provider or similar
Compact, robust test scripts
Takes advantage of groovy language
I would be particularly interested to hear from people who have tried more than one framework. Thanks!
I maintain the plugin for WebTest but I'd recommend giving Geb a try. I haven't used it personally on a project yet but I think it will tick off most of your criteria.
It is the most actively developed (IMO) but is quite new. It is built on WebDriver/Selenium so should also be a good fit with your past experience.
WebTest doesn't see a lot of development these days and does not have built in support for page objects. But it does give you great reports. It's downside is it's historical foundation of Ant. It makes it hard to test in a dynamic fashion as your test steps are built up when the test case is executed and then the actual test is run as a second pass.
I have used GFunc as well but the lack of reporting is a real pain and it does not get as much development as Geb. It is far more "groovy" than WebTest though and is a thin wrapper over HtmlUnit allowing you to "roll your own" testing functionality quite easily.
cheers
Lee
Grails Functional Test is HtmlUnit only. I wouldn't suggest coding directly to HtmlUnit these days with the other frameworks that are available that give you page object and better abstraction out of the box.
I've used the webdriver plugin which seems to work pretty well and has page object built in - but the page object support is coupled to the JUnit hierarchy, which makes it hard for me to use because I want to also use Cucumber.
I'm very excited about Geb and am hearing a lot of success stories with it but haven't gotten to finish setting it up myself.
I've started a new role in my life. I was a front end web developer, but I've now been moved to testing web software, or more so, automating the testing of the software. I believe I am to pursue a BDD (Behavior Driven Development) methodology. I am fairly lost as to what to use, and how to piece it together.
The code that is being used/written is in Java to write a web interface for the application to test. I have documentation of the tests to run, but I've been curious how to go about automating it.
I've been directed to Cucumber as one of the "languages" to help with the automation. I have done some research and come across a web site for a synopsis of BDD Tools/Frame works,
8 Best Behavior Driven Development (BDD) Tools and Testing Frameworks. This helped a little but then I got a little confused of how to implement it. It seems that Selenium is a common denominator in a lot of the BDD frameworks for testing a GUI, but it still doesn't seem to help describe what to do.
I then came across the term Functional Testing tool, and I think that confused me even more. Do they all test a GUI?
I think the one that looked like it was all one package was SmartBear TestComplete, and then there is, what seems to be, another similar application by SmartBear called, SmartBear TestLeft, but I think I saw that they still used Cucumber for BDDing it. There a few others that looked like they might work as well, but I guess the other question is what's the cheapest route?
I guess the biggest problem I have is how to make these tests more dynamic, as the UI/browser dimensions can easily change from system to system, and how do I go about writing automation that can handle this, and tie into a BDD methodology?
Does anyone have any suggestions here? Does anybody out there do this?
Thanks in advance.
BDD Architecture
BDD automation typically consists of a few layers:
The natural language steps
The wiring that ties the steps to their definition
The step definitions, which usually access page objects
Page objects, which provide all the capabilities of a page or widget
Automation over the actual code being exercised, often through the GUI.
The wiring between natural language steps and the step definitions is normally done by the BDD tool (Cucumber).
The automation is normally done using the automation tool (Selenium). Sometimes people do skip the GUI, perhaps targeting an API or the MVC layer instead. It depends how complex the functionality in your web page is. If in doubt, give Selenium a try. I've written automation frameworks for desktop apps; the principle's the same regardless.
Keeping it maintainable
To make the steps easy to maintain and change, keep the steps at a fairly high level. This is frequently referred to as "declarative" as opposed to "imperative". For instance, this is too detailed:
When Fred provides his receipt
And his receipt is scanned
And the cashier clicks "Refund to original card"
And the card is inserted...
Think about what the user is trying to achieve:
When Fred gets a refund to his original card
Generally a scenario will have a few Givens or Thens, but typically only one When (unless you have something like users interacting or time passing, where both events are needed to illustrate the behaviour).
Your page objects in this scenario might well be a "RefundPageObject" or perhaps, if that's too large, a "RefundToCardPageObject". This pattern allows multiple scenario steps to access the same capabilities without duplication, which means that if the way the capabilities are exercised changes, you only need to change them in one place.
Different page objects could also be used for different systems.
Getting started
If you're attacking this for the first time, start by getting an empty scenario that just runs and passes without doing anything (make the steps empty). When you've done this, you'll have successfully wired up Cucumber.
Write the production code that would make the scenario run. (This is the other way round from the way you'd normally do it; normally you'd write the scenario code first. I've found this is a good way to get started though.)
When you can run your scenario manually, add the automation directly to the steps (you've only got one scenario at this point). Use your favourite assertion package (JUnit) to get the outcome you're after. You'll probably need to change your code so that you can automate over it easily, by e.g.: giving relevant test ids to elements in your webpage.
Once you've got one scenario running, try to write any subsequent scenarios first; this helps you think about your design and the testability of what you're about to do. When you start adding more scenarios, start extracting that automation out into page objects too.
Once you've got a few scenarios, have a think about how you might want to address different systems. Avoid using lots of "if" statements if you can; those are hard to maintain. Injecting different implementations of page objects is probably better (the frameworks may well support this by now; I haven't used them in a while).
Keep refactoring as you add more scenarios. If the steps are too big, split them up. If the page objects are too big, divide them into widgets. I like to organize my scenarios by user / stakeholder capabilities (normally related to the "when" but sometimes to the "then") then by different contexts.
So to summarize:
Write an empty scenario
Write the code to make that pass manually
Wire up the scenario using your automation tool; it should now run!
Write another scenario, this time writing the automation before the production code
Refactor the automation, moving it out of the steps into page objects
Keep refactoring as you add more scenarios.
Now you've got a fully wired BDD framework, and you're in a good place to keep going while making it maintainable.
A final hint
Think of this as living documentation, rather than tests. BDD scenarios hardly ever pick up bugs in good teams; anything they catch is usually a code design issue, so address it at that level. It helps people work out what the code does and doesn't do yet, and why it's valuable.
The most important part of BDD is having the conversations about how the code works. If you're automating tests for code that already exists, see if you can find someone to talk to about the complicated bits, at least, and verify your understanding with them. This will also help you to use the right language in the scenarios.
See my post on using BDD with legacy systems for more. There are lots of hints for beginners on that blog too.
Since you feel lost as to where to start, I will hint you about some blogs I have written that talks a bit about your problem.
Some categories that may help you:
http://www.thinkcode.se/blog/category/Cucumber
http://www.thinkcode.se/blog/category/Selenium
This, rather long and old post, might give you hints as well:
http://www.thinkcode.se/blog/2012/11/01/cucumberjvm-not-just-for-testing-guis
Notice that versions are dated, but hopefully it can give some ideas as what too look for.
I am not an expert on the test automation but I am currently working on this part. So let me share some idea and hope it will help you at the current stage.
We have used selenium+cucumber+intellij for testing web application. We have used testcomplete+cucumber+intellij for testing java desktop application.
As to the test of web application, we have provided a test mode in our web application, which allows us to get some useful details of the product and the environment; and also allows us to easily trigger events through clicking the button and inputting text into the test panel under test mode.
I hope these are helpful for you.