I work in a small firm whose product is a relatively simple web-based software. For the last several years we have used Selenium IDE and it's record-and-play functionality. It has delivered 98% of what we need. However, now that Selenium IDE will no longer be working several months from now, we must find an alternative. We considered using Selenium WebDriver, but none of us no know programming and we just don't have the time to all learn it quickly enough to maintain our current testing. So, we are looking for something that has at minimum the following features...
Has record-and-play functionality (as opposed to programming)
Can support multiple web browsers (Firefox, Chrome, IE, Safari)
Can generate reports
Will continue to be supported for years to come
Has either a large user community or good company customer service
Database connect would be nice
Hopefully, we can import our existing Selenium IDE scripts, rather than rewrite them.
We are willing to pay if it's good.
There are literally dozens of options for software and I don't have a clear path to selecting one. Some promising software is...
- Katalon Recorder and Studio. It looks like an excellent replacement for Selenium IDE. But it has small community for help
- QTP - A colleague previously worked with this and recommends it. It looks like an excellent alternative for webdriver, as it allows record and play. But it is very expensive.
Any help is appreciated.
As per your feature requirement here are a couple of viable Selenium IDE alternatives :
Protractor :
Pros :
1. Simple installation. Only a couple of commands needed to install Selenium WebDriver and start testing.
2. Uses JavaScript which is easy to learn with limited programming background.
Cons :
1. Though flexible but still evolving. Setting up the project and the reporting plugins needs a bit of expertise.
2. Requires clear understanding of sync/async behavior.
Katalon Studio :
Pros :
1. No experience required for installation.
2. Unzip the package and ready to go.
3. No additional plug-ins are required.
4. The scripting interface allows an user to switch between the keyword-base table and the code editor.
Cons :
1.Though web automation and mobile automation is achievable API testing module needs more buildup.
Selenium Builder :
Pros :
1. Selenium Builder contains a lot of code from Selenium IDE and Webdriver.
2. The project is getting forked into a modern and a legacy version.
Cons :
1. As of version 40, Firefox requires all extensions to be signed and as it contains code from Selenium IDE and Webdriver it is getting rejected from signing.
2. Getting Selenium Builder into an acceptable state for signing means removing the existing code, which will remove support for Selenium 1.x and support for local playback of Selenium 2.x.
So it's a mix-bag situation and you can always pick one up and drop the other one at your disposal.
QTP I believe has now become UFT (Unified Functional Test) which does have a record option. Unfortunately what you lack in programming, you'll have to make up for in scripting knowledge (VBScript from what I worked with). It's not impossible but would probably take more effort to learn rather than implementing your own free version of Java/Selenium WebDriver. UFT is extremely expensive especially for a small firm which is a per license fee. If I remember right my last job had multiple licenses each $20k a piece.
Related
Hi I have a Swing application to test and I found Sikuli a nice tool to do it, but I am a little worried about the size of the community and if it's being continually developed and it's being used by other companies.
Do you use it?
For what?
Is it stable?
Is it the best tool for the job you needed?
I use it in my company, too.
It can be used quite easily for not too complex gui-tests.
Sikuli was not developed for the last year but development is now increasing again.
Questions in the Sikuli-FAQ section on launchpad are answered fast although the community is not that big.
In my company, Sikuli is used for gui testing which was previously done by human testers.
It saves some time but not everything is automatable with Sikuli, e.g. the OCR functionality is not dependable (but will be updated from tesseract 2.04 to 3 in the near future).
For my job it was the best tool because it is the only open source (=free) tool I found that provides screenshot based automation that can be integrated with other systems like CI-systems and is programmable with Java and Python which makes easy unit testing possible with JUnit or PyUnit.
Hope I could help.
Yes we use it in-house for testing. It is actively supported. I have reported bugs in Sikuli and have had tickets and workarounds suggested within days with the bugs fixed in the next revision.
It is quite stable. The problems I have encountered typically come from not specifying images correctly and the program selecting an incorrect area of the screen.
One of our more unique uses was creating a set of automated bench tests for a legacy embedded system. The system was written in assembly and had no unit testing capabilities. It communicated with a custom legacy PC application. Rather than try to locate the PC source code, reverse engineer the design, and then write some meaningful bench tests, we created a number of Sikuli scripts to interface with the PC app. It saved weeks of development.
Yes, we use it for automating GUI tests. It's used mostly for old systems that were developed with no test driven back end. (ie: no testing api)
We tests some very complex tools including a debugger using Sikuli.
We tend not to use the Sikuli IDE though.
I'm trying to get started with Selenium, but it's extremely confusing for anyone who still doesn't know the project.
If I go to the Selenium download page I get 7 download links for things with different names, and no description of what is each one. And then the Selenium 2 download link points to a page with another 8 files.
What does all of that mean? Which ones are deprecated? Which files should I get?
What are the differences between version 1 and 2? Is the version 1 deprecated, or do we still need to mix some modules from version 1 together with version 2?
And then, after searching on Google, I find Which Selenium Tool Should I Use?, but it's a very outdated page (last edited in 2007). It also talks about a limitation of the JavaScript testing interface: Same Origin Policy. Does this limitation still apply nowadays?
What are "Selenium Core", "Selenium Remote Control", "Selenium IDE"?
How about "Selenium (standalone) Server"? Is it the same thing as the Remote Control, or something different? What is the difference between Selenium Remote Control and Selenium Server?
And then I reach an official FAQ that is also confusing, and talks about a WebDriver thing that was going to be part of Selenium. Is it already inside Selenium? Is that FAQ up-to-date? How does it relate with other Selenium parts?
After all, another way to understand this question is: "I've never used Selenium before, I'm completely lost, and the documentation doesn't help." What I need is an introduction of what are all the parts of the project, which ones are old and deprecated, which ones are needed, which ones are interchangeable, which ones work with different browsers...
(maybe it would be a nice idea to have a community wiki answer that tries to be as accurate and detailed as possible)
I'm not much experienced with Selenium, but recently I have started it to automate form submissions while unit testing. For starters, try Selenium IDE. It is a Firefox addon which will help you to record actions and modify/replay them.
Actually more than that. You can perform Accessor operations, which will perfom a series of recorded/configured actions and will store the end results for us to analyze later.
And then there is Assertions, which will actually check the end state of a series of actions with some pre-configured conditions.
One thing I can say you is that Selenium is very handy and easy to learn. You can write your own test cases using plain html!
Here is a great beginner video - http://wiki.openqa.org/download/attachments/400/Selenium+IDE.swf?version=1
After seeing that, read through this tutorial - http://jroller.com/selenium/
And after that, read this reference - http://release.seleniumhq.org/selenium-core/1.0/reference.html
It explains everything you need to know. If you are first to Selenium, you will be amazZzed to see what all things selenium can do.
I have been working on Test Automation from last few months and have been using the tool named "Testcomplete". But I have noticed that the tools do not matter a lot in the field of automation. Only thing you expect from an automation tool is the ability of the tool to spit out the recognition strings for the different controls used in the test application.
Apart from this, you will always have to build a automation framework which will serve your needs writing code.
So my question is, Is my thinking that automation tools do not matter a lot in the field of automation correct? In the sense, you can use any tools to get your automation running. Or Do the tools really matter? (Please ignore the costs factor of the tools). Also if I need to learn a new automation tool, then what do I concentrate on? Or how do I go about learing the tool? In short, what exactly does "learning a tool" mean?
My 3 best reasons for choosing which tool to use:
it works. This is important, not all tools work in all scenarios i.e. flash, silver light, adobe air, legacy apps with no automation support, etc.
whole team skills. This includes not only testers, but also developers. Test automation shouldn't be an isolated effort, developers should also collaborate on it. This is far easier when dev and test are using the same language/platform.
price. Doesn't have to be free (but it could), but of course its an important factor.
Personally we use the same test runner as the one for the unit tests. That along with extra third party automation pieces that do the plumbing for you.
Some additional thoughts on why the tool is important:
Community - What's the user community like? Are there a lot of user-generated resources out there to help?
Support - (if vendor) What's customer support like? Do they fix problems quickly? Is it easy to find solutions to common problems?
Extensibility - Often in test automation, you'll need to roll your own or code work-arounds, if the tool does not support a particular type of object in your application. How easy is it to extend the product? What programming language does the tool use? What kind of support do you get from the IDE?
An other piece of advice: sometimes you'll need wrapper classes around certain frameworks. We were using WatiN, which was really good at its time, but it lacked Chrome (it had a small percentage that time) support. The thing that killed WatiN for us was the lack of coping with new FireFox releases: FireFox 8 was out, and we had to run on our tests on FireFox 3.6...
Selenium was the solution, but it had a totally different logic and we already had more than a 1000 tests.
So we had to create a wrapper class around Selenium to "fake" it was WatiN. We had some issues, but we had to rewrite only some special cases... And not all tests.
The point is, sometimes, support for frameworks just cease to exist. But with an own framework focusing on what the test actually does instead of how it works would save you in this situation.
Variety of Test Automation Frameworks and Test Automation Tools are already available in the market. Thus, I would not recommend the built your own Test Automation Framework at all.
As far as selection of automation tools is a concern, I would say it does matter on the following basis:
Support: How much level of support you have when you are choosing an automation tool for your project.
Community: How big community is using that tool and how responsive that community is about sharing knowledge?
Pricing: (Proprietary or Open Source) Last but not the least is the pricing of the automation tool that you are planning to introduce in your project.
QA teams' expertise also matters sometimes. For example, in case your QA team does not have a developer or semi-developer skills vs Not-Technical QA Team, etc.
Regarding the Automation framework, there are many automation frameworks also available in the market already, therefore no need to reinvent the wheel. and selection of automation framework mostly depends on your selection of scripting language.
For example, if you choose python as your scripting language, then you have option to choose UnitTest, PyTest etc. as an automation framework.
In case of Java, you have option of JUnit and TestNG as an automation framework.
and so on, the base on your selection of scripting language.
Finally, when it comes to structuring your automation framework, it solely depends on many things as the following:
Your nature of the project
Single product vs multiple products
and many more...
Check an example of multiple product automation project directory structure. https://github.com/pancht/python-selenium-framework
I hope, in some way, I would have helped you out in giving an answer to your question.
Thanks,
Panchdev Singh Chauhan
My goal is:
Our customers could generate new web-tests.
Our continuous integration server makes a test-environment deployment; it should execute the tests against it
The test could also be run against some other environment.
(Final acceptance tests should be made by the customer, to test fonts etc, but this would be a great pre-acceptance check for our test-environment. Customers could focus on other things than now.)
Usually some property (like text field id) has changed or something and the tests will break in a few weeks. It seems that recorded tests broke often, so it's better to easily record a new one than trying to maintenance and modify an old test.
Now, I found a whole new approach. Maybe recording is not the right way.
How about, if our customers could make use cases in a human readable own language which the machine would understand and compile to web-recording (with Domain Specific Language, DSL).
This is not sci-fi, it has been already made, so read on. :-)
I have tried to use these automatic web testing frameworks:
Visual Studio web test (Customers can't execute)
Selenium (Works only with Firefox, our customers have IE)
WatiN (.NET version of Watir, recorder seems to be a bit buggy)
HP Quick Test Pro (Not easy enough to make new tests)
None of these have provided actually what I need... but Selenium is the closest one.
Our customers speak Finnish, so in the beginning of a software project, in specification phase, user writes a use case like:
Avaa "OmaLomake"
Syötä "Tuomas" kohtaan "nimi"
Paina "Seuraava"
Translation:
Open "MyForm"
Insert "Tuomas" into field "name"
Press "Next"
Now... This is a human-readable use case, but also it can be compiled to automatic web acceptance test. Open, Insert, into field and Press are keywords, others are values.
What kind of DSL tool would be good for this?
Microsoft is making a new DSL-making-tool in their Oslo-project called MGrammar. It means that you can make a custom language to make it easy for non-technical people to work with machines. (The same basic idea that was (and failed) with Cobol and Visual Basic.)
I found that someone has already made this kind of DSL with MGrammar, but it is for Watin, not Selenium:
http://www.codinginstinct.com/2008/11/creating-watin-dsl-using-mgrammar.html
So the continuous integration server process will be:
Fetch a new version from source control (as usual).
Build, run unit tests and analyze the code (as usual).
Make an installation package and tag version in version control (as usual).
Compile use cases to web tests
Run web tests
Accept/Reject the software :-)
Running a web-test in the continuous integration server usually means a lot of configuration work. So, before I try this, I'm curious, what do you think?
Have you used same kind of setup, and what are your experiences? (What exact environment?)
How about DSL, will it have enough power for use cases or will it be another endless development task? Will the customers ever generate the tests?
First of all, Selenium does work with IE and other browsers as well as Firefox; cross browser support is one of its strengths. Here's the list of supported browsers.
However, if you want a human language-based DSL for writing your tests, take a look at Cucumber - the syntax is almost exactly like your example above. Cucumber already has Finnish language support - see the examples at this link.
Fitnesse and Selenium Integration tools such as Selenesse(http://github.com/marisaseal/selenesse) or Fitnium(http://www.magneticreason.com/tools/fitnium/fitnium.html) can also serve your purpose. However, you need to find answer for who will put the element locators in the test cases written by customers. If customers put the locators using the recorders, it may not be possible to maintain. If customers write the steps and a automation tester/developer can put those locators using regex, custom location strategy, this approach may work.
The TestPlan software uses a specialize language for writing tests. It is highly domain specific and works very well in web environments. It supports the Selenium backend so you gain that compatibility, plus it can run without a browser, for even faster tests. I have used it on some fairly large web projects in the type of setup you are looking for.
Your example script might look like this:
GotoURL /SomePage
Click MyForm
SubmitForm with
%Params:name% Tuomos
%Submit% value:Next
end
That's it. It nicely describes what the user wants to do and is a functioning test. You can combine scripts into units and have custom function as well. So if you really wanted you could write the Finish equivalents to the names.
My project is compatible only with Internet Explorer. I want the test scripts to get generated automatically as it is done in Selenium IDE.
Can i use Selenium RC to test my application? I could not use Selenium IDE as it can be used only with Mozilla Firefox.
seleniumrc works with IE. You can specify the browser and the path to it within the config file.
It can be easily integrated into night builds via ant.
stick to writing the testcase in java.
Selenium RC and Selenium Grid are both really good at running tests against IE. You can see all the browsers that are supported by Selenium here and Selenium is Designed to write for one browser and work in the rest. THere are a few little quirks that wont work in every browser but 99% of the time it will.
Selenium RC works with IE, but is very buggy with IE 6 (to the point of being unusable). Generating the scripts is not trivial and there are many methods of doing it. We have created a Firefox extension that examines objects via introspection to make click recording easy. There are many options out there but your best bet is to write your tests with Firefox/Firebug (or Chrome). They will make object location much simpler and if you are careful the locator strings should still work in IE.
There could be two answer to you question:
Besides Selenium, though it has ample of advantages, I am reading about another tool which uses same API which Selenium use. The only changes in API I have seen so far is it reduces the complexity of functions thus making it more easier and simpler for user who is learning.
The tool is called 'Helium' and it has 50% (and more) less complex functions and code as Selenium has.
The only problem with this tool is it is paid tool for learning purpose and for implementing not-so-big scale project you can use it. But yeah after some time its gonna cost you.
I have implemented some code on Helium. Please let me know , if you face any issue initially or you are thinking to implement it.
Other being, you can use Selenium Builder(http://khyatisehgal.wordpress.com/2014/05/26/selenium-builder-exporting-and-execution/) which is an advanced form of Selenium IDE. It imports your command in different languages and does work more effectively and efficiently as Selenium IDE does(http://khyatisehgal.wordpress.com/2014/05/25/selenium-builder/)
Please let me know , if you have any doubt in any of the tool.
I know Watin is compatible with IE and Firefox. If you want to generate the test code you can use the Watin Test Recorder
This of course is implying that you are using .Net
... Or you could just use the .net bindings that comes along with the latest couple of versions, then you can just run 'em through nUnit.
For ex. Selenium IDE users Katalon Recorder might be a good match. Supports different browsers.