Could someone explain what an automation test is and why I would use it. I read from the wiki page that a tester would create a automation script? What kind of scripting language can be used to do this?
Automation tests are carried out to check the behavior of an application against expected behavior. Normally used in regression testing where you validate that a newer version of the application doesn't hinder any of the previous version's features. These might also be carried along with manual testing.
Coming to the scripting language part, this might help you:- https://softwareengineering.stackexchange.com/questions/19292/best-language-or-tool-for-automating-tedious-manual-tasks
In simple words, If you are doing regression test or testing same piece of code over and over you can automate that manual process. That's called automation testing.
You can use several different scripting languages to achieve this and it's depends on which tool you are using. Some popular automation tools are Selenium, QTP, Loadrunner, Jmeter, SOAP UI etc.
You want to check your login with more than 1000 of users how much time you will spend to run this test case ?
In the same way you want to test you mobile API's before it used by developer how will you test?
There are lots of thing for that you have to go for automation In small application, sites you can work as an tester after that when those app's sites will grow will large data than those product owner will move for automated test cases
Related
I am new to testing. The commonly used terms like Framework and tool confuses me a lot. Can anyone please explain me the difference between a Framework like STAF[software testing automation framework]and Tool like selenium.
Also how to select a tool for a particular framework. What are the criterias used for selection?
Brief explanations are welcomed!!
Tool:
Simply put, a tool is a software. In case of test automation, tools are software that let you automate your tests on an application. There are many test automation tools that you can choose from depending on your requirements. Some examples are Selenium, UFT, Visual Studio CUIT, Jamo Solutions Meux Test, T-Plan Robot, Telerik Test Studio etc.
Often, you'll have to write tests in the tools using a supported programming language. For instance, testers using UFT need to code in VB while those using Visual Studio can code in both VB and C#. However, some testing tools (like Telerik Test Studio) let you write script-less tests where your tests will consist of a bunch of easily understandable keywords, not code.
Framework:
The most popular test automation tools like Selenium and Visual Studio provide all the basic features you require to build your own tests. However, they do not provide ready-made features (like Reporting and Exception Handling) for testing. This requires the creation of a 'Framework' which is nothing but a collection of code written using a tool of one's choice that makes testing an application easy. Simply put, a framework is what you create with a tool (or a collection of tools) to test your application.
A typical framework consists of two parts: test scripts and function libraries. Test Scripts are the pieces of code that need to be executed to perform actions on the application under test (AUT). Function Libraries are classes consisting of important functions that are called by your test scripts. These can include timing functions, reporting/logging functions, exception handling functions, data communication functions etc.
You can also use an external database to pass test data to your test scripts during run-time instead of hard-coding it in your test scripts. Frameworks that employ external databases are called data-driven frameworks. The external database can be of your choice, be it a SQL Server, an XML file or a simple Excel spreadsheet. Data-driven frameworks make use of APIs or include custom-made classes that let you communicate with the database to transfer data.
Another type of framework is the keyword-driven framework. These frameworks are used in long-term test automation projects that require scripting of thousands of test cases. The main objective of these frameworks is to reduce the time taken to script a test case by reusing code that has already been written. They often include very strong function libraries which enable scripting of test cases using just predefined keywords. For example, common actions on an application like login and logout are performed by single line codes like:
Actions.Login();
and
Actions.Logout();
where Actions is a Function Library that consists of the Login() and Logout() functions. This massively reduces the script size and the long-term maintenance requirements of the test script, among other benefits.
Of course, you can either build a test automation framework and use it for your own application or create a generic test automation framework and make it available to the testing community for everyone to use, which is what STAF is.
Selection of testing tools:
To address your second question, there is no straight-forward answer to it. There are a number of criteria that can affect your decision. But in the end, it is all about your requirements and the requirements of your AUT.
If it is a Windows desktop app, you have to use Coded UI Tests in
Visual Studio.
If it is a Web application, you can use Selenium, UFT, Visual Studio or Test Studio.
If it is a mobile app, you can use Appium, Jamo Solutions Meux Test or T-Plan Robot.
If you want to test your mobile app over a large number of devices
and platforms, you can use cloud-based tools like Sauce Labs,
Perfecto Mobile or Device Anywhere.
If you are short on budget, you'll be better off using open source tools over
commercial tools, and so on.
Application Testing is a huge industry now and there is no dearth of testing tools available in the market. You will find the perfect tool for you if you know what you want and do some research on Google.
I will try to answer what I believe people normally use these terms to mean, lets start with the simpler term: A tool.
A tool like selenium is what actually does the automation, it has an API that will work for pretty much anything it covers (in this case websites) but it knows nothing about how the website you want to test works, this means it deals with low level constructs such as elements on a page and clicks.
A framework is normally just wrapping a tool to make it easier to make a test by imparting knowledge of your application, a standard example is login.
Say you want a test that checks when you enter a correct username and password in you get access to application. Using just selenium it would something like:
driver.findElement(By.id("username").sendKeys("MyUsername");
driver.findElement(By.id("password").sendKeys("password123");
driver.findElement(By.id("login").click();
Thats pretty simple, but as you can guess login is going to be used a lot across your tests and so it makes sense to move this into a place that makes it easier to reuse (both from a less code stand point and maintainability). This is where a framework comes into play, normally with selenium it will be page objects (see here)
Base on my understanding:
TOOLS
We "USE" tools to meet our objective (can be own self or your small groups of team).
Example: We use Selenium IDE as a tools helps us to automate some repetitive steps to do certain verification during our smoke test.
FRAMEWORK
We "DESIGN" a framework to meet the organization mission.
Things to consider when we design the framework including:
Maintainability
Reusability
Data Driven
Reporting
Schedule running through CI tools like Jenkins
Example: We design a Test Automation Framework by using WebDriver + Java + TestNG + ANT, to meet the objective to identify our current code base stability, and the test will be trigger and run by jenkins in a daily basis, SSRS report will captured in a daily basis as well each time the test finished. Stakeholders can review the daily code stability report anytime he/she needed.
Hope that can help you :D
is there any tool out there that i can used to set-up run automatically and i was goggling and i found selenium test runner? there are so many tools out there its hard to figured out which is best
I'm using C# and using MSTest as a test framework and I'm looking forward to see if I can get a way from testing in MSTEST
any help?
This is very subjective question. Every requirement will have its own correct answer. Anyhow I will try to address few requirements and will be updating as I learn more.
If you are automating web app browser tests (sans flash player and silverlight) I would say that selenium is the way to go. There are ways to automate flash and silverlight too, but that is answer for another question.
Selenium is anyways an automation too and your choice will rather is of which test framework to select. So here are few options:
1. Integrating with CI tools:
If you want to organize your tests as segregated atomic units and want them to be integrated to some CI server (e.g. TeamCity). I will recommend using NUnit to run your selenium tests.
2. Behavioral Tests
It is a new trend in the software development and how we test our products. Using behavioral (i.e. business specification) like language. In my experience it is also a very good format to write up acceptance tests. You can use selenium with something like Nbehave or SpecFlow
3. Centralize Test management and Execution
Now this might not fit for everyone but I have found FitNesse (and its c# binding) to be very useful in maintaining and executing selenium test cases.
Please note this answer may not be right and is certainly not complete given the scope of the question. I have nevertheless tried provide few pointers.
My question maybe silly, but can anyone coach me?
Except doing some test(most likely white box testing) while coding, after the App was built, do we have some testing tools or special method for doing the test?
All I can imaging for now, is only manual testing the functionality of my App.
Thanks everyone.
Update: Added section 'Automated testing for iOS4'
As a professional tester my suggestion is that you should have a healthy mix of automated and manual testing. The Examples below are in .net but it should be easy to find a tool for whatever technique you are using.
AUTOMATED TESTING
Unit Testing
Use NUnit to test your classes, functions and interaction between them.
http://www.nunit.org/index.php
Automated Functional Testing
If it's possible you should automate a lot of the functional testing. Some frame works have functional testing built into them. Otherwise you have to use a tool for it. If you are developing web sites/applications you might want to look at Selenium.
http://www.peterkrantz.com/2005/selenium-for-aspnet/
Continuous Integration
Use CI to make sure all your automated tests run every time someone in your team makes a commit to the project.
http://martinfowler.com/articles/continuousIntegration.html
Automated testing for iOS4
Automate the testing of your application by scripting touch events using the new UIAutomation Instrument.
Some links:
http://answers.oreilly.com/topic/1646-how-to-use-uiautomation-to-create-iphone-ui-tests/
http://cocoawithlove.com/2008/11/automated-user-interface-testing-on.html
http://alexvollmer.com/posts/2010/07/03/working-with-uiautomation/
MANUAL TESTING
As much as I love automated testing it is, IMHO, not a substitute for manual testing. The main reason being that an automated can only do what it is told and only verify what it has been informed to view as pass/fail. A human can use it's intelligence to find faults and raise questions that appear while testing something else.
Exploratory Testing
ET is a very low cost and effective way to find defects in a project. It take advantage of the intelligence of a human being and a teaches the testers/developers more about the project than any other testing technique i know of. Doing an ET session aimed at every feature deployed in the test environment is not only an effective way to find problems fast, but also a good way to learn and fun!
http://www.satisfice.com/articles/et-article.pdf
Take a look at automated testing tools. Supports automated and manual testing/sending feedback from within the app with annotated screen shots
I suggest you take a look at the iPhoneUnitTests sample code posted by Apple on their developer site.
FoneMonkey is a free and open source functional testing automation tool available for download from Gorilla Logic.
There a number of emerging options for automated functional testing, including Appium, Calabash, Frank, and Zucchini.
Much of testing any application is about understanding what you are testing and areas that should be tested. Some of this comes with experience, but types of things to consider testing about would be:
Functionality
iOS Design Guidelines / UI
Gestures
Connectivity
Types of devices to test on
Audio
Data
Crash reporting
Analytics
There's a big list of areas to cover.
I recommend Kiwi, its used for Behavior Driven Development. By far my favorite testing framework, makes testing much more fun, and test much readable and clear.
https://github.com/allending/Kiwi
I have been working on Test Automation from last few months and have been using the tool named "Testcomplete". But I have noticed that the tools do not matter a lot in the field of automation. Only thing you expect from an automation tool is the ability of the tool to spit out the recognition strings for the different controls used in the test application.
Apart from this, you will always have to build a automation framework which will serve your needs writing code.
So my question is, Is my thinking that automation tools do not matter a lot in the field of automation correct? In the sense, you can use any tools to get your automation running. Or Do the tools really matter? (Please ignore the costs factor of the tools). Also if I need to learn a new automation tool, then what do I concentrate on? Or how do I go about learing the tool? In short, what exactly does "learning a tool" mean?
My 3 best reasons for choosing which tool to use:
it works. This is important, not all tools work in all scenarios i.e. flash, silver light, adobe air, legacy apps with no automation support, etc.
whole team skills. This includes not only testers, but also developers. Test automation shouldn't be an isolated effort, developers should also collaborate on it. This is far easier when dev and test are using the same language/platform.
price. Doesn't have to be free (but it could), but of course its an important factor.
Personally we use the same test runner as the one for the unit tests. That along with extra third party automation pieces that do the plumbing for you.
Some additional thoughts on why the tool is important:
Community - What's the user community like? Are there a lot of user-generated resources out there to help?
Support - (if vendor) What's customer support like? Do they fix problems quickly? Is it easy to find solutions to common problems?
Extensibility - Often in test automation, you'll need to roll your own or code work-arounds, if the tool does not support a particular type of object in your application. How easy is it to extend the product? What programming language does the tool use? What kind of support do you get from the IDE?
An other piece of advice: sometimes you'll need wrapper classes around certain frameworks. We were using WatiN, which was really good at its time, but it lacked Chrome (it had a small percentage that time) support. The thing that killed WatiN for us was the lack of coping with new FireFox releases: FireFox 8 was out, and we had to run on our tests on FireFox 3.6...
Selenium was the solution, but it had a totally different logic and we already had more than a 1000 tests.
So we had to create a wrapper class around Selenium to "fake" it was WatiN. We had some issues, but we had to rewrite only some special cases... And not all tests.
The point is, sometimes, support for frameworks just cease to exist. But with an own framework focusing on what the test actually does instead of how it works would save you in this situation.
Variety of Test Automation Frameworks and Test Automation Tools are already available in the market. Thus, I would not recommend the built your own Test Automation Framework at all.
As far as selection of automation tools is a concern, I would say it does matter on the following basis:
Support: How much level of support you have when you are choosing an automation tool for your project.
Community: How big community is using that tool and how responsive that community is about sharing knowledge?
Pricing: (Proprietary or Open Source) Last but not the least is the pricing of the automation tool that you are planning to introduce in your project.
QA teams' expertise also matters sometimes. For example, in case your QA team does not have a developer or semi-developer skills vs Not-Technical QA Team, etc.
Regarding the Automation framework, there are many automation frameworks also available in the market already, therefore no need to reinvent the wheel. and selection of automation framework mostly depends on your selection of scripting language.
For example, if you choose python as your scripting language, then you have option to choose UnitTest, PyTest etc. as an automation framework.
In case of Java, you have option of JUnit and TestNG as an automation framework.
and so on, the base on your selection of scripting language.
Finally, when it comes to structuring your automation framework, it solely depends on many things as the following:
Your nature of the project
Single product vs multiple products
and many more...
Check an example of multiple product automation project directory structure. https://github.com/pancht/python-selenium-framework
I hope, in some way, I would have helped you out in giving an answer to your question.
Thanks,
Panchdev Singh Chauhan
My goal is:
Our customers could generate new web-tests.
Our continuous integration server makes a test-environment deployment; it should execute the tests against it
The test could also be run against some other environment.
(Final acceptance tests should be made by the customer, to test fonts etc, but this would be a great pre-acceptance check for our test-environment. Customers could focus on other things than now.)
Usually some property (like text field id) has changed or something and the tests will break in a few weeks. It seems that recorded tests broke often, so it's better to easily record a new one than trying to maintenance and modify an old test.
Now, I found a whole new approach. Maybe recording is not the right way.
How about, if our customers could make use cases in a human readable own language which the machine would understand and compile to web-recording (with Domain Specific Language, DSL).
This is not sci-fi, it has been already made, so read on. :-)
I have tried to use these automatic web testing frameworks:
Visual Studio web test (Customers can't execute)
Selenium (Works only with Firefox, our customers have IE)
WatiN (.NET version of Watir, recorder seems to be a bit buggy)
HP Quick Test Pro (Not easy enough to make new tests)
None of these have provided actually what I need... but Selenium is the closest one.
Our customers speak Finnish, so in the beginning of a software project, in specification phase, user writes a use case like:
Avaa "OmaLomake"
Syötä "Tuomas" kohtaan "nimi"
Paina "Seuraava"
Translation:
Open "MyForm"
Insert "Tuomas" into field "name"
Press "Next"
Now... This is a human-readable use case, but also it can be compiled to automatic web acceptance test. Open, Insert, into field and Press are keywords, others are values.
What kind of DSL tool would be good for this?
Microsoft is making a new DSL-making-tool in their Oslo-project called MGrammar. It means that you can make a custom language to make it easy for non-technical people to work with machines. (The same basic idea that was (and failed) with Cobol and Visual Basic.)
I found that someone has already made this kind of DSL with MGrammar, but it is for Watin, not Selenium:
http://www.codinginstinct.com/2008/11/creating-watin-dsl-using-mgrammar.html
So the continuous integration server process will be:
Fetch a new version from source control (as usual).
Build, run unit tests and analyze the code (as usual).
Make an installation package and tag version in version control (as usual).
Compile use cases to web tests
Run web tests
Accept/Reject the software :-)
Running a web-test in the continuous integration server usually means a lot of configuration work. So, before I try this, I'm curious, what do you think?
Have you used same kind of setup, and what are your experiences? (What exact environment?)
How about DSL, will it have enough power for use cases or will it be another endless development task? Will the customers ever generate the tests?
First of all, Selenium does work with IE and other browsers as well as Firefox; cross browser support is one of its strengths. Here's the list of supported browsers.
However, if you want a human language-based DSL for writing your tests, take a look at Cucumber - the syntax is almost exactly like your example above. Cucumber already has Finnish language support - see the examples at this link.
Fitnesse and Selenium Integration tools such as Selenesse(http://github.com/marisaseal/selenesse) or Fitnium(http://www.magneticreason.com/tools/fitnium/fitnium.html) can also serve your purpose. However, you need to find answer for who will put the element locators in the test cases written by customers. If customers put the locators using the recorders, it may not be possible to maintain. If customers write the steps and a automation tester/developer can put those locators using regex, custom location strategy, this approach may work.
The TestPlan software uses a specialize language for writing tests. It is highly domain specific and works very well in web environments. It supports the Selenium backend so you gain that compatibility, plus it can run without a browser, for even faster tests. I have used it on some fairly large web projects in the type of setup you are looking for.
Your example script might look like this:
GotoURL /SomePage
Click MyForm
SubmitForm with
%Params:name% Tuomos
%Submit% value:Next
end
That's it. It nicely describes what the user wants to do and is a functioning test. You can combine scripts into units and have custom function as well. So if you really wanted you could write the Finish equivalents to the names.