I'm currently reading the book Growing Object-Oriented Software Guided by Tests which gives a decent introduction into test driven development.
One drawback is that the code developed through the book is based on Swing. For my next project I'd like to use Java FX 2 and I'm wondering if there are any GUI testing frameworks out there for it? The book introduces windowlicker, which allows testing Swing applications only.
There's yet another new testing library for JavaFX called TestFX.
There is a library named JemmyFX. Jemmy itself is a set of libraries and tools to test UI applications (mostly Java-based: Swing, AWT, SWT; but it's being extended to native right now). JemmyFX covers JavaFX for that matter.
JemmyFX can be downloaded as a part of the test branch of OpenJFX 2 (open-sourced part of JavaFX).
For JavaFX 8, the test branch includes support for new controls and other fixes.
There's a new test robot called MarvinFX:
MarvinFX has the goal to easily test JavaFX controls and scenes with a special attention on properties.
This other post shows how to use assertions and rules with MarvinFX.
Another library (which I have not used or tested) is TestComplete.
Automaton is another testing library for JavafX and Swing GUIs born out of the same team that did TestFX
easy tests for Swing and JavaFX applications
written for testers. Only basic coding skills required.
According to Eclipse site, the modern testing tools for JavaFX in the year 2015 are Squish and Jubula . But Squish is commercial. And Jubula is partly free - JavaFX belongs to that free part. And in the Eclipse Mars version already exists the testing distribution. Uses Jubula for UI testing.
Jubula is now totally free, and does indeed support JavaFX.
Related
I found it over a few references that it is recommended to go with the same stack for creating automation suite that is being used for the development of the application, i.e., using Java for writing tests if application is Java based.
Is there any specific reason for the same? Or is there any specific hindrance if not followed.
I would personally say that developers making an application in one language will probably feel more familiar writing the tests in the same language. And Integration tests should preferably be written by the developer who wrote the feature.
So using the same language for your test suite and your application development increases the odds your developers will be able to write and fix tests for features they create and fix themselves.
I think that should be team decision, the same as what testing frameworks to use. But it's totally fine to use for example Spock which is groovy based framework if your application is java based.
JavaFX 2 does not provide validation support (masks, input filtering and so on...). It is difficult to adopt a technology that does not offer basic functionalities. I am trying to implement my own validators, but that is a big pain.
Are there any news in JavaFX 8 about validation? I was trying to find new features, but I don't know where to search, it is difficult to find exactly what will be the changes in JavaFX 8. I need to know about it, because I was considering JavaFX for a new application.
Answer
No, JavaFX 8 will not implement high level validation support.
Opinion
I think it unlikely that comprehensive validation support will ever be in the JavaFX core libraries (this is just my personal opinion).
JavaFX provides a base library upon which other libraries can be implemented. JavaFX 2.2 provides enough support that you could write validation libraries on top of JavaFX (and people have done so).
There are some reasons why you might not want validation support in JavaFX core libraries:
As JavaFX is part of Java SE and the open-jfx project, it must follow the Java release process, which can be quite slow at times for various reasons. 3rd party projects don't have such restrictions.
Things in JavaFX implement just one way of doing things generally, whereas you might wish to have multiple frameworks built on JavaFX, each for different purposes, then choose between the framework which best meets your needs.
Some advanced validation frameworks might make use of things such as Beans Validation, which are currently only part of the Java EE specification. JavaFX is written to work with just Java SE, so the core framework can't make use of anything from Java EE, even though that might be desirable for validation.
Generally, validation is not something done in isolation, but is often part of a wider framework (such as a form filling framework or a complete application platform).
3rd Party Validation Libraries
Some sample 3rd party libraries which perform validation for JavaFX fields are:
JideFX
FXForm2
I was trying to find new features, but I don't know where to search, it is difficult to find exactly what will be the changes in JavaFX 8.
I am sure, when Java 8 is officially released in a couple of weeks, it will come with an announcement summarizing new JavaFX features.
You can find out what all of the new features for JavaFX 8 are by filtering the JavaFX issue tracker on fixed features for Java 8. Here is a link to the result of a JavaFX 8 fixed feature query. Note, the linked query reports only larger features, not minor tweaks.
Hi I have a Swing application to test and I found Sikuli a nice tool to do it, but I am a little worried about the size of the community and if it's being continually developed and it's being used by other companies.
Do you use it?
For what?
Is it stable?
Is it the best tool for the job you needed?
I use it in my company, too.
It can be used quite easily for not too complex gui-tests.
Sikuli was not developed for the last year but development is now increasing again.
Questions in the Sikuli-FAQ section on launchpad are answered fast although the community is not that big.
In my company, Sikuli is used for gui testing which was previously done by human testers.
It saves some time but not everything is automatable with Sikuli, e.g. the OCR functionality is not dependable (but will be updated from tesseract 2.04 to 3 in the near future).
For my job it was the best tool because it is the only open source (=free) tool I found that provides screenshot based automation that can be integrated with other systems like CI-systems and is programmable with Java and Python which makes easy unit testing possible with JUnit or PyUnit.
Hope I could help.
Yes we use it in-house for testing. It is actively supported. I have reported bugs in Sikuli and have had tickets and workarounds suggested within days with the bugs fixed in the next revision.
It is quite stable. The problems I have encountered typically come from not specifying images correctly and the program selecting an incorrect area of the screen.
One of our more unique uses was creating a set of automated bench tests for a legacy embedded system. The system was written in assembly and had no unit testing capabilities. It communicated with a custom legacy PC application. Rather than try to locate the PC source code, reverse engineer the design, and then write some meaningful bench tests, we created a number of Sikuli scripts to interface with the PC app. It saved weeks of development.
Yes, we use it for automating GUI tests. It's used mostly for old systems that were developed with no test driven back end. (ie: no testing api)
We tests some very complex tools including a debugger using Sikuli.
We tend not to use the Sikuli IDE though.
I have been working on Test Automation from last few months and have been using the tool named "Testcomplete". But I have noticed that the tools do not matter a lot in the field of automation. Only thing you expect from an automation tool is the ability of the tool to spit out the recognition strings for the different controls used in the test application.
Apart from this, you will always have to build a automation framework which will serve your needs writing code.
So my question is, Is my thinking that automation tools do not matter a lot in the field of automation correct? In the sense, you can use any tools to get your automation running. Or Do the tools really matter? (Please ignore the costs factor of the tools). Also if I need to learn a new automation tool, then what do I concentrate on? Or how do I go about learing the tool? In short, what exactly does "learning a tool" mean?
My 3 best reasons for choosing which tool to use:
it works. This is important, not all tools work in all scenarios i.e. flash, silver light, adobe air, legacy apps with no automation support, etc.
whole team skills. This includes not only testers, but also developers. Test automation shouldn't be an isolated effort, developers should also collaborate on it. This is far easier when dev and test are using the same language/platform.
price. Doesn't have to be free (but it could), but of course its an important factor.
Personally we use the same test runner as the one for the unit tests. That along with extra third party automation pieces that do the plumbing for you.
Some additional thoughts on why the tool is important:
Community - What's the user community like? Are there a lot of user-generated resources out there to help?
Support - (if vendor) What's customer support like? Do they fix problems quickly? Is it easy to find solutions to common problems?
Extensibility - Often in test automation, you'll need to roll your own or code work-arounds, if the tool does not support a particular type of object in your application. How easy is it to extend the product? What programming language does the tool use? What kind of support do you get from the IDE?
An other piece of advice: sometimes you'll need wrapper classes around certain frameworks. We were using WatiN, which was really good at its time, but it lacked Chrome (it had a small percentage that time) support. The thing that killed WatiN for us was the lack of coping with new FireFox releases: FireFox 8 was out, and we had to run on our tests on FireFox 3.6...
Selenium was the solution, but it had a totally different logic and we already had more than a 1000 tests.
So we had to create a wrapper class around Selenium to "fake" it was WatiN. We had some issues, but we had to rewrite only some special cases... And not all tests.
The point is, sometimes, support for frameworks just cease to exist. But with an own framework focusing on what the test actually does instead of how it works would save you in this situation.
Variety of Test Automation Frameworks and Test Automation Tools are already available in the market. Thus, I would not recommend the built your own Test Automation Framework at all.
As far as selection of automation tools is a concern, I would say it does matter on the following basis:
Support: How much level of support you have when you are choosing an automation tool for your project.
Community: How big community is using that tool and how responsive that community is about sharing knowledge?
Pricing: (Proprietary or Open Source) Last but not the least is the pricing of the automation tool that you are planning to introduce in your project.
QA teams' expertise also matters sometimes. For example, in case your QA team does not have a developer or semi-developer skills vs Not-Technical QA Team, etc.
Regarding the Automation framework, there are many automation frameworks also available in the market already, therefore no need to reinvent the wheel. and selection of automation framework mostly depends on your selection of scripting language.
For example, if you choose python as your scripting language, then you have option to choose UnitTest, PyTest etc. as an automation framework.
In case of Java, you have option of JUnit and TestNG as an automation framework.
and so on, the base on your selection of scripting language.
Finally, when it comes to structuring your automation framework, it solely depends on many things as the following:
Your nature of the project
Single product vs multiple products
and many more...
Check an example of multiple product automation project directory structure. https://github.com/pancht/python-selenium-framework
I hope, in some way, I would have helped you out in giving an answer to your question.
Thanks,
Panchdev Singh Chauhan
What frameworks exist to unit test Objective-C code? I would like a framework that integrates nicely with Apple Xcode.
Xcode includes XCTest, which is similar to OCUnit, an Objective-C unit testing framework, and has full support for running XCTest-based unit tests as part of your project's build process. Xcode's unit testing support is described in the Xcode Overview: Using Unit Tests.
Back in the Xcode 2 days, I wrote a series of weblog posts about how to perform some common tasks with Xcode unit testing:
Unit testing Cocoa frameworks
Debugging Cocoa framework unit tests
Unit testing Cocoa applications
Debugging Cocoa application unit tests
Despite using OCUnit rather than XCTest, the concepts are largely the same.
Finally, I also wrote a few posts on how to write tests for Cocoa user interfaces; the way Cocoa is structured makes it relatively straightforward, because you don't have to spin an event loop or anything like that in most cases.
Trust, but verify.
Unit testing Cocoa user interfaces: Target-Action
Unit testing Cocoa user interfaces: Cocoa Bindings
This makes it possible to do test-driven development for not just your model-level code but also your controller-level and even view-level code.
Check out GHUnit by Gabriel Handford:
"The goals of GHUnit are:
Runs unit tests within XCode, allowing
you to fully utilize the XCode
Debugger. A simple GUI to help you
visualize your tests. Show stack
traces. Be installable as a framework
(for Cocoa apps) with a simple (or
not) target setup; or easy to package
into your iPhone project."
I started using the Google toolbox testing rig for iPhone, and its working out great for me.
google-toolbox-for-mac
Check out OCUnit. Apple's developer network has a great introduction.
Note that the Google Toolbox for Mac (GTM) project simply extends/augments Apple's SenTestingKit framework (which is, itself based on OCUnit). As they say on the project site:
GTM has several enhancement to the
standard SenTestingKit allowing you to
do UI unit testing, automated binding
unit testing, log tracking, and unit
testing on the iPhone, as well as
tools for doing static and dynamic
testing of your code.
Note the following comment about user-interface testing:
GTM has extensive support for user
interface unit tests. It supports
testing both the imaging and/or
internal state of almost all of the
standard Cocoa/UIKit UI objects, and
makes it easy for you to extend this
support to your own UI objects.
See their "Code Verification and Unit Testing" page for instructions on how to use it.
I came to the conclusion that GHUnit is the most advanced testing framework for Objective-C. I have done a roundup of testing frameworks on my blog. It is the most flexible in terms of deployment (iphone, simulator or mac os native) and assert capabilities. Because it is based on GTM, it inherits all of GTM's advantages over SenTestingKit but also adds a lot more. Another bonus is that it is being maintained very actively.
I have conducted effort to integrate OCMock into GHUnit, it works great!. You can get the code on github.
I realize this is an old question, but if you prefer BDD-style testing (rspec, Jasmine, etc.) over xUnit-style testing (Test::Unit, JSUnit, JUnit, etc.), then you may consider checking out Cedar. Cedar brings BDD-style testing to Objective-C, now that the language supports closures.
We're happily using Cedar for our iOS projects at Pivotal Labs, and we're actively working on improving it. Any feedback or suggestions are welcome at cedar-discuss#googlegroups.com
I would also recommend using coverage tools to see which part of the code are covered with unit tests and which are not. Basic line and branch code coverage can be generated with the GCOV tool. If you want to generate nice HTML coverage reports there are LCOV and ZCOV which do just that.
I recommend gh-unit, it has a nice GUI for test results.
http://github.com/gabriel/gh-unit/tree/master
The Unit Testing support bundled within xcode (for its simple setup) combined with ocrunner (for some autotest/Growl goodness) is currently my favorite Obj-C Unit Testing setup.
here is a whole lot of them
List_of_unit_testing_frameworks in Objective-C
Sen:te (the creator of the testing framework included with Xcode) explains how to use OCUnit with an iPhone project: simple-iphone-ipad-unit-test.
Matt Gallagher of Cocoa with Love has a very good article on unit testing.
I would suggest looking into Kiwi, an open source BDD testing framework for iOS:
Kiwi
Check out the project's WIKI to start or get Daniel Steinberg's book "Test Driving iOS Development with Kiwi"
test-driving-ios-development
I use SimpleUnitTest works with iPhone and iPad libs.
http://cbess.blogspot.com/2010/05/simple-iphone-ipad-unit-test.html
It comes with a unit test Xcode template to easily add a unit test class. Wraps GTM.
You can literally drop it into an active project and start adding unit tests within 3 minutes (or less).
Specta is a modern TDD(Test Driven Development)/BDD(Behavior Driven Development) framework which runs on top of XCTest. It supports unit testing for iOS and Mac OS X projects.
I hope u can use 'SenTestKit', from which u can test each and every method.