is there any tool out there that i can used to set-up run automatically and i was goggling and i found selenium test runner? there are so many tools out there its hard to figured out which is best
I'm using C# and using MSTest as a test framework and I'm looking forward to see if I can get a way from testing in MSTEST
any help?
This is very subjective question. Every requirement will have its own correct answer. Anyhow I will try to address few requirements and will be updating as I learn more.
If you are automating web app browser tests (sans flash player and silverlight) I would say that selenium is the way to go. There are ways to automate flash and silverlight too, but that is answer for another question.
Selenium is anyways an automation too and your choice will rather is of which test framework to select. So here are few options:
1. Integrating with CI tools:
If you want to organize your tests as segregated atomic units and want them to be integrated to some CI server (e.g. TeamCity). I will recommend using NUnit to run your selenium tests.
2. Behavioral Tests
It is a new trend in the software development and how we test our products. Using behavioral (i.e. business specification) like language. In my experience it is also a very good format to write up acceptance tests. You can use selenium with something like Nbehave or SpecFlow
3. Centralize Test management and Execution
Now this might not fit for everyone but I have found FitNesse (and its c# binding) to be very useful in maintaining and executing selenium test cases.
Please note this answer may not be right and is certainly not complete given the scope of the question. I have nevertheless tried provide few pointers.
Related
I found it over a few references that it is recommended to go with the same stack for creating automation suite that is being used for the development of the application, i.e., using Java for writing tests if application is Java based.
Is there any specific reason for the same? Or is there any specific hindrance if not followed.
I would personally say that developers making an application in one language will probably feel more familiar writing the tests in the same language. And Integration tests should preferably be written by the developer who wrote the feature.
So using the same language for your test suite and your application development increases the odds your developers will be able to write and fix tests for features they create and fix themselves.
I think that should be team decision, the same as what testing frameworks to use. But it's totally fine to use for example Spock which is groovy based framework if your application is java based.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 8 years ago.
Improve this question
Guys I am looking for Web Testing Generic Automation Framework which can be used to do automation testing of various web based applications .Looking for C# based framework as that is the language I am more familiar with. But any other language framework will also do and it should not use any proprietary/licensed language.
Framework should have some open source and free of cost license model.
I searched for selenium based framework on Google and SO. But could not come with any which have source code available. It will be good if the framework encapsulates all the functionality provided by Selenium WebDriver and/or Selenium RC and empower the functional tester to create and maintain test in human readable scripts.
Requirements of the framework:
The framework code should avoid hard coding of test steps. My idea is to maintain the test scripts outside the automation framework code , so that they can be easily be modified if needed. The framework should read through the step tables and the data tables and run the test accordingly.
If there is no such framework available now right then we can collectively build such a framework in a open source community model.
P.S.
I have read a little about Hermes Framework and Robot Framework, but not yet tried them, any help is welcome.
The good side of this problem: there are a lot of flexible tools and approaches, you can get together and build a flexible, reliable and robust test automation framework.
The hard part is: yes, there is no “out of box” solution, and you’ll need to find and put together lots of tools in order to solve this test automation puzzle.
What I would recommend:
First you need to choose a unit-test test framework. This is a tool which helps to identify separate methods in code as tests, so you can run them together or separately and get the run results, such as pass or fail.
My personal opinion, is that the testing tool – MS-Test – which ships with Visual Studio 2013 (and also Express Edition) is good enough. Another alternatives are: NUnit or Gallio Icarus
All unit-testing frameworks includes a mechanism for doing assertions inside the test. The capability of assertions class depends on given unit-testing framework. Here, I would like to recommend a popular library which works great for the entire unit testing framework.
This is Fluent Assertions (also available from NuGet repository).
That’s a hard moment. You need to decide: are you going to use the PageObject approach in order to build your test automation framework, or you are going to choose simpler approach, without heavy utilization of the Object Oriented Programming.
Properly designed Page Objects makes your test automation code much maintainable. Utilizing the OOP – you can do a magic in your code: write less to do more. Although, such approach requires more skill.
Here are a good articles on this topic:
Maintainable Automated UI Tests
And this one:
Tips to Avoid Brittle UI Tests
The alternative to the PageObject is a scripted approach. This approach can be also successful and requires less time to start.
Coypu is a good and usable example of such framework for Selenium Web Driver.
All the popular unit-testing frameworks support data-driven tests. The best support is in NUnit – you can run/re-run and see the tests generated for individual data row in the tests tree.
MS-Test supports reading data from different data-sources: text files, excel, mssql etc., but it is not possible to re-run the test for individual data row. Although, there is a hack for this – Ms-Test Rows.
For my data-driven tests, I am using a great library – Linq to Excel
I have a lot more to say. There are so many approaches to build test automation framework – and there is no ready solution yet.
I am trying to build one according to my testing methodology – SWD.Starter .
This project is still on its early development stages. But, at least, probably you’ll find a few tips how to build and organize the test automation code.
I've implemented https://github.com/leblancmeneses/RobustHaven.IntegrationTests based on my prior experience on large projects "trying" to implement full end to end testing.
I've been using this and and have a lot of useful extensions for general selenium, angularjs, and kendo ui work. Since this framework is not obtrusive you could just use these extensions without using anything else.
I'm using this on my latest project and everyone is loving it.
There are a lot of bdd/spec frameworks (specflow, mspec, nspec, storyq) to help wire the behavior of your system to tests.
What I've learned:
make it frictionless for any .net developer/tester to begin writing/running tests.
Most fail here because it requires installing additional pluggins into visual studio.
mine uses the standard nunit
Logically you would think that a feature is a class file and scenarios are [Test] methods - to support some of these frameworks they make each scenario a class file.
use the original spec to create stubs of your tests - hopefully readable code
I used spec flow back in 2010 - so things might have changed. I generated my tests from my bdd document. A year later when I went to add more tests and update existing tests, I felt I wasted a lot of time with ceremony than writing code I really wanted - I stopped using it.
My approach uses t4 to generate stubs - developer has a choice to generate from feature file, for a specific scenario or don't use generated code at all.
how is state shared across steps / nested steps
most use dictionary<string,object> to help you separate data from being hardcoded in your tests accessed from a context object.
mine uses viewmodels and pointers to those viewmodels - if your using something like angularjs you are using viewmodels in your server side display/editor templates and in angularjs controller so why not reuse these in your tests!
start early with CI - make development transparent
My project has ResultDiff that given the nunit testresult.xml file, folder location to your gherkin feature files, and output json file; Read description on why this is important on the screenshot: https://github.com/leblancmeneses/RobustHaven.IntegrationTests#step-5-ci-setup-resultdiff
Example:
Modified means business and developers have a mismatch of Gherkin statements - did something change that we need to talk about?
What is missing? a dashboard to render the .json file created by ResultDiff. It's on my backlog.....
With a centralized dashboard that supports multiple environments(branches of your code) this dashboard will serve all stakeholders (business, developers) what is the status of features being developed.
There is a framework named "omelet" which is built in java on top of testng for selenium,
For cross browser multi-parallel testing , it easily blends with your CI tools and have some cool reporting features with step level reports
Running your test cases on BrowserStack and Grid was never so easy as with omelet with few config changes.
if you want to give it a try then do follow the 5 min tutorial available on the website, there is archetype available on maven central + there are many more features available
Stable version is 1.0.4 and we are currently looking for people to contribute to project.
Documentation over here
Github link
Could someone explain what an automation test is and why I would use it. I read from the wiki page that a tester would create a automation script? What kind of scripting language can be used to do this?
Automation tests are carried out to check the behavior of an application against expected behavior. Normally used in regression testing where you validate that a newer version of the application doesn't hinder any of the previous version's features. These might also be carried along with manual testing.
Coming to the scripting language part, this might help you:- https://softwareengineering.stackexchange.com/questions/19292/best-language-or-tool-for-automating-tedious-manual-tasks
In simple words, If you are doing regression test or testing same piece of code over and over you can automate that manual process. That's called automation testing.
You can use several different scripting languages to achieve this and it's depends on which tool you are using. Some popular automation tools are Selenium, QTP, Loadrunner, Jmeter, SOAP UI etc.
You want to check your login with more than 1000 of users how much time you will spend to run this test case ?
In the same way you want to test you mobile API's before it used by developer how will you test?
There are lots of thing for that you have to go for automation In small application, sites you can work as an tester after that when those app's sites will grow will large data than those product owner will move for automated test cases
I am in a new project that is looking to include an end to end testing framework.
We want something flexible, I've used Fitnesse before and I think we need something similar to it.
We are also using Hudson CI and are looking for something that would integrate easily with it.
Is there a clear winner?
UPDATED: The system has many components, some of them are web services running on tomcat, there are a couple of NoSQL databases too, but no UI testing is required for the moment.
Please add a comment if further clarification on the project details is needed.
The robot framework is a good keyword driven testing framework that we use for end to end integration testing.
http://code.google.com/p/robotframework/
There is a plugin for jenkins (a fork of hudson) that is very useful. It reportedly works with hudson as well.
Great to read you have decided start using Robot Framework. Its Hudson plugin already has recently got much better (trend charts, available from Hudson directly, ...) and moved to new place:
http://wiki.hudson-ci.org/display/HUDSON/Robot+Framework+Plugin
Can you be more specific, when you say, you want to have an end-to-end testing framework? What kind of application are you developing (standalone, web based, web service, ...)? What language do you program in?
I would also look for previous knowledge in your team or around your team (e.g. other teams). It might be sufficient, to use TestNG or JUnit (I have seen an old paper on UI testing using JUnit). Almost all test frameworks should be fine for Hudson, as long as they can be started in a headless/scripted mode and if they provide the results in a format that Hudson understands (in the worst case HTML is sufficent, though Hudson will only be able to display the pages). So if you have IBM Monitoring tools available, you might want to look at Rational Functional Tester or Rational Performance Tester.
FitNesse is a good tool for functional testing. The business contacts can create the test cases by themselfs (developers have to provide some connector code), which will create more visibility and removes some work from the test engineers. Drawback is, that you can't really do UI testing. If you need a open source plattform for UI, have a look at seam. Even though I like the OSS philosophi, you might need longer ramp up time (higher costs) and possibly more maintainace time for OSS (not true for all OSS). In any case, check how easy it is to get support for your framework. You might even consider paid support for your framework (regardless if open or closed source). Also don't forget, even though you can fix OSS source yourself, you might not have the expertise or time for fixing bugs or adding features.
So give us more info and we can give you better advices.
My experience with jBehave is very good. Recommend.
It's based on BDD - Behavior Driven Development.
My goal is:
Our customers could generate new web-tests.
Our continuous integration server makes a test-environment deployment; it should execute the tests against it
The test could also be run against some other environment.
(Final acceptance tests should be made by the customer, to test fonts etc, but this would be a great pre-acceptance check for our test-environment. Customers could focus on other things than now.)
Usually some property (like text field id) has changed or something and the tests will break in a few weeks. It seems that recorded tests broke often, so it's better to easily record a new one than trying to maintenance and modify an old test.
Now, I found a whole new approach. Maybe recording is not the right way.
How about, if our customers could make use cases in a human readable own language which the machine would understand and compile to web-recording (with Domain Specific Language, DSL).
This is not sci-fi, it has been already made, so read on. :-)
I have tried to use these automatic web testing frameworks:
Visual Studio web test (Customers can't execute)
Selenium (Works only with Firefox, our customers have IE)
WatiN (.NET version of Watir, recorder seems to be a bit buggy)
HP Quick Test Pro (Not easy enough to make new tests)
None of these have provided actually what I need... but Selenium is the closest one.
Our customers speak Finnish, so in the beginning of a software project, in specification phase, user writes a use case like:
Avaa "OmaLomake"
Syötä "Tuomas" kohtaan "nimi"
Paina "Seuraava"
Translation:
Open "MyForm"
Insert "Tuomas" into field "name"
Press "Next"
Now... This is a human-readable use case, but also it can be compiled to automatic web acceptance test. Open, Insert, into field and Press are keywords, others are values.
What kind of DSL tool would be good for this?
Microsoft is making a new DSL-making-tool in their Oslo-project called MGrammar. It means that you can make a custom language to make it easy for non-technical people to work with machines. (The same basic idea that was (and failed) with Cobol and Visual Basic.)
I found that someone has already made this kind of DSL with MGrammar, but it is for Watin, not Selenium:
http://www.codinginstinct.com/2008/11/creating-watin-dsl-using-mgrammar.html
So the continuous integration server process will be:
Fetch a new version from source control (as usual).
Build, run unit tests and analyze the code (as usual).
Make an installation package and tag version in version control (as usual).
Compile use cases to web tests
Run web tests
Accept/Reject the software :-)
Running a web-test in the continuous integration server usually means a lot of configuration work. So, before I try this, I'm curious, what do you think?
Have you used same kind of setup, and what are your experiences? (What exact environment?)
How about DSL, will it have enough power for use cases or will it be another endless development task? Will the customers ever generate the tests?
First of all, Selenium does work with IE and other browsers as well as Firefox; cross browser support is one of its strengths. Here's the list of supported browsers.
However, if you want a human language-based DSL for writing your tests, take a look at Cucumber - the syntax is almost exactly like your example above. Cucumber already has Finnish language support - see the examples at this link.
Fitnesse and Selenium Integration tools such as Selenesse(http://github.com/marisaseal/selenesse) or Fitnium(http://www.magneticreason.com/tools/fitnium/fitnium.html) can also serve your purpose. However, you need to find answer for who will put the element locators in the test cases written by customers. If customers put the locators using the recorders, it may not be possible to maintain. If customers write the steps and a automation tester/developer can put those locators using regex, custom location strategy, this approach may work.
The TestPlan software uses a specialize language for writing tests. It is highly domain specific and works very well in web environments. It supports the Selenium backend so you gain that compatibility, plus it can run without a browser, for even faster tests. I have used it on some fairly large web projects in the type of setup you are looking for.
Your example script might look like this:
GotoURL /SomePage
Click MyForm
SubmitForm with
%Params:name% Tuomos
%Submit% value:Next
end
That's it. It nicely describes what the user wants to do and is a functioning test. You can combine scripts into units and have custom function as well. So if you really wanted you could write the Finish equivalents to the names.