Hi I have a Swing application to test and I found Sikuli a nice tool to do it, but I am a little worried about the size of the community and if it's being continually developed and it's being used by other companies.
Do you use it?
For what?
Is it stable?
Is it the best tool for the job you needed?
I use it in my company, too.
It can be used quite easily for not too complex gui-tests.
Sikuli was not developed for the last year but development is now increasing again.
Questions in the Sikuli-FAQ section on launchpad are answered fast although the community is not that big.
In my company, Sikuli is used for gui testing which was previously done by human testers.
It saves some time but not everything is automatable with Sikuli, e.g. the OCR functionality is not dependable (but will be updated from tesseract 2.04 to 3 in the near future).
For my job it was the best tool because it is the only open source (=free) tool I found that provides screenshot based automation that can be integrated with other systems like CI-systems and is programmable with Java and Python which makes easy unit testing possible with JUnit or PyUnit.
Hope I could help.
Yes we use it in-house for testing. It is actively supported. I have reported bugs in Sikuli and have had tickets and workarounds suggested within days with the bugs fixed in the next revision.
It is quite stable. The problems I have encountered typically come from not specifying images correctly and the program selecting an incorrect area of the screen.
One of our more unique uses was creating a set of automated bench tests for a legacy embedded system. The system was written in assembly and had no unit testing capabilities. It communicated with a custom legacy PC application. Rather than try to locate the PC source code, reverse engineer the design, and then write some meaningful bench tests, we created a number of Sikuli scripts to interface with the PC app. It saved weeks of development.
Yes, we use it for automating GUI tests. It's used mostly for old systems that were developed with no test driven back end. (ie: no testing api)
We tests some very complex tools including a debugger using Sikuli.
We tend not to use the Sikuli IDE though.
Related
I am new to the Testing Arena. I am working with a very heavy ExtJs application.
And I am looking for the best testing tool.
I came across a bunch of tools, but can't seem to make a decision.
1) Siesta 2) Jasmine 3) Riatest
I want to be able to deploy these tests easily on a CI server.
Siesta and Jasmine can both be used with PhantomJs to automate the tests, but which one is better and easy to use?
As long as I can generate various clicks correctly and capture output, I'm cool.
Any help is appreciated.
Our company is moving from a Java based client to an ExtJS web and mobile application. We use QTP/UFT for Java automation which is slow, buggy, expensive, and cannot get passed the DOM easily so I started investigating Siesta recently. It seems like a viable option in my book but I admit I haven't checked out the other applications.
The initial setup with Siesta took longer than expected but with its event recorder, it makes it a gratifying transition. The recorder still requires debugging. I'm in QA and know how to script using Python, Bash, etc but it's definitely a learning curve to transition from VBScript to ExtJs/Siesta JavaScript. They have an open source version and a free 45 day trial to check out.
I've read about HTML Robot and SmartBear. Here's a post on the Sencha forums that talks about different automation software. Sencha also plans to release some kind of automation involving SenchaCmd during SenchaCon 2015 this April 7 to 9.
You should take a tool which covers your needs and improve the software quality.
Jasmine is good for unit tests without much gui interaction, you should use this to test your domain logic (e.g. stores, models, ...). Jasmine can run on every environment, a simple server with nodejs runtime is enought.
For regression tests the choice is yours. What tool you are comfortable with? Choosing a tool is one part, using it is another. Riatest seems like a windows application, are you able to run this on your CI server?
Evaluate them with your dev team and then make a choice for the long run.
Just new on software testing...
Regarding testing, I think GUI applications are pretty difficult to automate. Some testing involves with interacting particular GUI objects in particular sequence (e.g., clicking buttons). The interface often changes from one window to another. And the timing and sync sometimes also pose an issue (e.g., recording mouse clicks and replaying may screw up).
Is there any solution for testing such applications with less human labour? Thank you for sharing your experience.
Yes, GUI apps are indeed tough to automate. Regardless of the app's technology (Swing, web, WPF, iOS), you first have to focus on automating high-value tests. Moreover, test automation shouldn't be at just the GUI level, it should be a mix across unit, integration, and functional (GUI) tests too.
Are you working on a web app? If so, have a look at great open source tools like Watir or WebDriver. (I'll also pitch Telerik's Test Studio to you; however, for full disclosure I'm their evangelist for that tool.)
Desktop applications (or mobile) bring a lot of challenges to automation, and it's totally dependent on what platform you're working with. Test Studio supports WPF, but you can also look to other commercial and a few free tools. I don't know of any tools for Swing apps, but that lack of knowledge is due to me having been out of that domain for many years. (And maybe I'm so out of it that Swing's not even the normal Java GUI toolset...)
iOS and Android are tough ones to find reliable automation tools for. I know the Frank framework/API will work on iOS (Test Studio has a free recorder in the App Store), but I don't know of any other tools that reliably support the extraordinary matrix of Droid hardware and OS versions.
Regardless of your platform and toolset, you need to learn the basic approaches for dealing with GUI testing: focus on high value tests, learn to avoid duplication through approaches like Page Object Pattern, learn how to deal with synchronization/timing issues in your specific application.
It's a long haul, but if you work carefully it's totally worth it.
(And fun, too, IMO.)
I am in a new project that is looking to include an end to end testing framework.
We want something flexible, I've used Fitnesse before and I think we need something similar to it.
We are also using Hudson CI and are looking for something that would integrate easily with it.
Is there a clear winner?
UPDATED: The system has many components, some of them are web services running on tomcat, there are a couple of NoSQL databases too, but no UI testing is required for the moment.
Please add a comment if further clarification on the project details is needed.
The robot framework is a good keyword driven testing framework that we use for end to end integration testing.
http://code.google.com/p/robotframework/
There is a plugin for jenkins (a fork of hudson) that is very useful. It reportedly works with hudson as well.
Great to read you have decided start using Robot Framework. Its Hudson plugin already has recently got much better (trend charts, available from Hudson directly, ...) and moved to new place:
http://wiki.hudson-ci.org/display/HUDSON/Robot+Framework+Plugin
Can you be more specific, when you say, you want to have an end-to-end testing framework? What kind of application are you developing (standalone, web based, web service, ...)? What language do you program in?
I would also look for previous knowledge in your team or around your team (e.g. other teams). It might be sufficient, to use TestNG or JUnit (I have seen an old paper on UI testing using JUnit). Almost all test frameworks should be fine for Hudson, as long as they can be started in a headless/scripted mode and if they provide the results in a format that Hudson understands (in the worst case HTML is sufficent, though Hudson will only be able to display the pages). So if you have IBM Monitoring tools available, you might want to look at Rational Functional Tester or Rational Performance Tester.
FitNesse is a good tool for functional testing. The business contacts can create the test cases by themselfs (developers have to provide some connector code), which will create more visibility and removes some work from the test engineers. Drawback is, that you can't really do UI testing. If you need a open source plattform for UI, have a look at seam. Even though I like the OSS philosophi, you might need longer ramp up time (higher costs) and possibly more maintainace time for OSS (not true for all OSS). In any case, check how easy it is to get support for your framework. You might even consider paid support for your framework (regardless if open or closed source). Also don't forget, even though you can fix OSS source yourself, you might not have the expertise or time for fixing bugs or adding features.
So give us more info and we can give you better advices.
My experience with jBehave is very good. Recommend.
It's based on BDD - Behavior Driven Development.
I have been working on Test Automation from last few months and have been using the tool named "Testcomplete". But I have noticed that the tools do not matter a lot in the field of automation. Only thing you expect from an automation tool is the ability of the tool to spit out the recognition strings for the different controls used in the test application.
Apart from this, you will always have to build a automation framework which will serve your needs writing code.
So my question is, Is my thinking that automation tools do not matter a lot in the field of automation correct? In the sense, you can use any tools to get your automation running. Or Do the tools really matter? (Please ignore the costs factor of the tools). Also if I need to learn a new automation tool, then what do I concentrate on? Or how do I go about learing the tool? In short, what exactly does "learning a tool" mean?
My 3 best reasons for choosing which tool to use:
it works. This is important, not all tools work in all scenarios i.e. flash, silver light, adobe air, legacy apps with no automation support, etc.
whole team skills. This includes not only testers, but also developers. Test automation shouldn't be an isolated effort, developers should also collaborate on it. This is far easier when dev and test are using the same language/platform.
price. Doesn't have to be free (but it could), but of course its an important factor.
Personally we use the same test runner as the one for the unit tests. That along with extra third party automation pieces that do the plumbing for you.
Some additional thoughts on why the tool is important:
Community - What's the user community like? Are there a lot of user-generated resources out there to help?
Support - (if vendor) What's customer support like? Do they fix problems quickly? Is it easy to find solutions to common problems?
Extensibility - Often in test automation, you'll need to roll your own or code work-arounds, if the tool does not support a particular type of object in your application. How easy is it to extend the product? What programming language does the tool use? What kind of support do you get from the IDE?
An other piece of advice: sometimes you'll need wrapper classes around certain frameworks. We were using WatiN, which was really good at its time, but it lacked Chrome (it had a small percentage that time) support. The thing that killed WatiN for us was the lack of coping with new FireFox releases: FireFox 8 was out, and we had to run on our tests on FireFox 3.6...
Selenium was the solution, but it had a totally different logic and we already had more than a 1000 tests.
So we had to create a wrapper class around Selenium to "fake" it was WatiN. We had some issues, but we had to rewrite only some special cases... And not all tests.
The point is, sometimes, support for frameworks just cease to exist. But with an own framework focusing on what the test actually does instead of how it works would save you in this situation.
Variety of Test Automation Frameworks and Test Automation Tools are already available in the market. Thus, I would not recommend the built your own Test Automation Framework at all.
As far as selection of automation tools is a concern, I would say it does matter on the following basis:
Support: How much level of support you have when you are choosing an automation tool for your project.
Community: How big community is using that tool and how responsive that community is about sharing knowledge?
Pricing: (Proprietary or Open Source) Last but not the least is the pricing of the automation tool that you are planning to introduce in your project.
QA teams' expertise also matters sometimes. For example, in case your QA team does not have a developer or semi-developer skills vs Not-Technical QA Team, etc.
Regarding the Automation framework, there are many automation frameworks also available in the market already, therefore no need to reinvent the wheel. and selection of automation framework mostly depends on your selection of scripting language.
For example, if you choose python as your scripting language, then you have option to choose UnitTest, PyTest etc. as an automation framework.
In case of Java, you have option of JUnit and TestNG as an automation framework.
and so on, the base on your selection of scripting language.
Finally, when it comes to structuring your automation framework, it solely depends on many things as the following:
Your nature of the project
Single product vs multiple products
and many more...
Check an example of multiple product automation project directory structure. https://github.com/pancht/python-selenium-framework
I hope, in some way, I would have helped you out in giving an answer to your question.
Thanks,
Panchdev Singh Chauhan
I'm looking for a code coverage tool that I can use with a BlackBerry application. I'm using J2ME-Unit for Unit Testing and I want to see how much of my code is being covered by my tests.
I've tried using Cobertura for J2ME but after days of wrestling with it I failed to get any results from it. (I believe that the instrumentation is un-done by the RAPC compilation). And despite this message, the project seems to be dead.
I've looked at JInjector but the project seems very incomplete. There is little (if any) documentation and although it claims to be able to work with BlackBerry projects, I haven't seen any places where it has been used for that purpose. I've played with the project quite a bit but to no avail.
I've also tried the "Coverage" view in the BlackBerry JDE, even though I use Eclipse for development. The view stays permanently blank, regardless of clicking "Refresh" and running the application from the JDE.
I've looked at most of the tools on this SO thread, but they won't work with J2ME/BlackBerry projects.
Has anyone had any success with any code coverage tools on the BlackBerry? If so, what tools have you used? How have you used them?
If anyone has managed to get JInjector or Cobertura for J2ME to work with a BlackBerry project, what did you have to do to get it working?
I can't speak for Coberatura or JInjector, because I don't know how they collect test coverage probe data.
What is
critical is how this data is captured (does it need Java runtime support only available in standard Java VMs?) and how it is exported to the test coverage display/report generation tools.
Our SD Java Test Coverage tool instruments your source code; at runtime this produces an array of native Java booleans representing the coverage data, without need for any special VM support. Normally, this array is exported directly to a file, used by the test coverage display mechanism, by a TCVDump method provided with the test coverage tool, as your application exits.
Java (and other programming langauges used) in embedded systems often requires custom methods to extract the test coverage data. You might need to code a special dump procedure (in Java) to write out that boolean array to an accessible place. Our experience with building such custom dump procedures is that they are generally pretty simple (a few dozen lines); the real trick is deciding how/where to put the data, so that it can be easily moved to the target file. Mostly this is just a peculiar pair of copies, the first of which copies the boolean array to some staging location, and the second which writes the staged data into the destination file. (The standard TCVdump method is provided in source form to enable this kind of customization).
While I haven't specifically looked at BlackBerry, if you can write the data anywhere, you can pretty much be assured you can achieve this. We've had success with other embedded hand-set systems, such as Symbian, doing this.
If you want a complete overview of how to generally instrument code for test coverage following this strategy, see this paper: Branch Coverage for Arbitrary Languages Made Easy
I was actively involved with JInjector while working at Google. We were able to use it to successfully obtain code coverage for Blackberry applications. The application lifecycle for Balckberry apps is less predictable than J2ME and we found we had to tweak the application code to ensure the coverage data was gathered. I didn't personally work on the blackberry apps, several other engineers did. I'd hoped we'd create an example blackberry application and make it available on the jinjector site, but events and life got in the way.
If you would be willing to provide a sample blackberry apps with some unit tests, I'd be willing to spend a few hours trying to help you get the code coverage working. I'm not actively working with either J2ME or Blackberry (I'm currently working on Android apps when I have time to experiment with mobile) so I'm quite rusty. I have a day job that doesn't involve much mobile test automation, however I continue to work on ways to improve the test automation for mobile apps e.g. http://code.google.com/p/mwta/downloads/list for Android Test Automation.
I'm julianharty at gmail.com