We have an application in production and there are slim chances that it would get enhanced further. We are curious whether shall we consider automating same. if yes, how it would be beneficial. Thanks!
You need to answer yourself if automation will benefit you. If you don't see benefits from automation - there is none, other than gathering knowledge and experience while developing automation.
Other non-straight benefits:
- robust smoke tests (if update finally comes)
- automated tests are sort of behavior documentation (take a look at BDD)
Making test-automation have positive ROI is probably the hardest step in whole automation. Without experience on what to automate and how to do it, it's negative in most cases. So gathering such knowledge and experience is benefit, which of course comes with a cost. If you have a gut feeling that this case won't benefit you - it probably won't, and you should look for a different place to gather experience.
Good luck
Automation is always a hot topic in our office. I agree with G. Swiec that ROI is going to be your biggest concern. I personally think there is value in having a robust automation suite for an app. Even if you don't plan on enhancing your app further, the tests could help catch bugs you missed or even if dependencies go wonky. If you have decent tools and the hours to do it, I think it could add value. However, these benefits are quickly outweighed if you don't have the man hours to research and implement an automated testing strategy.
You may want to consider other popular avenues like contract testing. Good contract testing can help relieve the need to write big automated e2e tests altogether!
We are a couple of entrepenours who have developed a cross-browser app and a backend administration system for the app. Or actually, we paid a company to develop it. Now we want it tested profesionally, but we dont want to use the same company for this purpose.
The tests may involve
Integration Testing
Functional Testing
System Testing
Stress Testing
Performance Testing
Usability Testing
For some of the tests, we think that the actual source code is required. We dont feel komfortable giving our source code away "just like that", to unknown parties, so what are the common methods for having external companies testing ones software?
you don't really need to give the source to perform the mentioned tests. you need to provide working environments or provide binaries and instructions how to deploy them. it seems sufficient for 1,2 (i don't know what does 3 means), 4 and 5. It's way too late for usability testing. it should have been done during UI design phase (how do you want to test it right now?)
but those tests are not sufficient. you forgot about penetration testing. and above tests are black box testing and they can show you how the application works.
but if you have any real plans for this application you must be sure it's maintainable. and for this you need the whitebox testing, you have to analyze the code.
you can start with automated analyzing to check the overall quality of the code. but at the end you will still need good programmers to perform the code review. but you don't really have to give them the code. you can invite them to your office and let them review the code on your workstations. unless your idea is so simple and brilliant that only one look at the code is enough to reproduce it. in such case you will need to sign NDA or give some shares to the expert who will take care of the quality
I am testing a silver light 3.0 application. I need to measure its performance and usability kind of things.. i have already searched some of the tools but yet to find the right one.
Could anyone help me in doing the above thing?
if you have developed this with good seperation of concerns, then you should be able to run tests against the backend code that doing the actual work.
So create a test project, create some mocks and run the tests 1,000 times.
The UI, other than functional testing, should not need to be stress tested IMHO.
As for usability, show some people or engage a 3rd party to take a look at the UI and give recomendations on what you could improve, if anything, to make it more usable.
I'm just starting out working on test scripts.I'm going to get a web application created in .net for testing. I have no idea what kind of testing is needed for such kind of applications.
My suggestion is that you should have a healthy mix of automated and manual testing.
AUTOMATED TESTING
Unit Testing
Use NUnit to test your classes, functions and interaction between them.
http://www.nunit.org/index.php
Automated Functional Testing
If it's possible you should automate a lot of the functional testing. Some frame works have functional testing built into them. Otherwise you have to use a tool for it. If you are developing web sites/applications you might want to look at Selenium.
http://www.peterkrantz.com/2005/selenium-for-aspnet/
Continuous Integration
Use CI to make sure all your automated tests run every time someone in your team makes a commit to the project.
http://martinfowler.com/articles/continuousIntegration.html
MANUAL TESTING
As much as I love automated testing it is, IMHO, not a substitute for manual testing. The main reason being that an automated can only do what it is told and only verify what it has been informed to view as pass/fail. A human can use it's intelligence to find faults and raise questions that appear while testing something else.
Exploratory Testing
ET is a very low cost and effective way to find defects in a project. It take advantage of the intelligence of a human being and a teaches the testers/developers more about the project than any other testing technique i know of. Doing an ET session aimed at every feature deployed in the test environment is not only an effective way to find problems fast, but also a good way to learn and fun!
http://www.satisfice.com/articles/et-article.pdf
There are a lot of small things that you may need to check (assuming you're doing manual testing):
Check for the exact location/alignment of items
Check whether all hyperlinks are working as expected
Check by clicking a button (submitting the form) multiple time
Check for security aspects (google for xss or cross site scripting)
Check for fonts etc. (if they're different from standards)
Hope this helps.
A web application should go through below test's
1) Functionality Testing
2) Usability testing
3) Interface testing
4) Compatibility testing
5) Performance testing
6) Security testing
for complete guide
Read this Article
Recently I've came up with the question is it worth at all to spent development time to generate automatic unit test for web based projects? I mean it seems useless at some point because at some point those projects are oriented on interactions with users/clients, so you cannot anticipate the whole possible set of user action so you be able to check the correctness of content showed. Even regression test can hardly be done. So I'm very eager to know to know the opinion of other experienced developers.
Selenium have a good web testing framework
http://seleniumhq.org/
Telerik are also in the process of developing one for web app testing.
http://www.telerik.com/products/web-ui-test-studio.aspx
You cannot anticipate the whole
possible set of user action so you be
able to check the correctness of
content showed.
You can't anticipate all the possible data your code is going to be handed, or all the possible race conditions if it's threaded, and yet you still bother unit testing. Why? Because you can narrow it down a hell of a lot. You can anticipate the sorts of pathological things that will happen. You just have to think about it a bit and get some experience.
User interaction is no different. There are certain things users are going to try and do, pathological or not, and you can anticipate them. Users are just inputting particularly imaginative data. You'll find programmers tend to miss the same sorts of conditions over and over again. I keep a checklist. For example: pump Unicode into everything; put the start date after the end date; enter gibberish data; put tags in everything; leave off the trailing newline; try to enter the same data twice; submit a form, go back and submit it again; take a text file, call it foo.jpg and try to upload it as a picture. You can even write a program to flip switches and push buttons at random, a bad monkey, that'll find all sorts of fun bugs.
Its often as simple as sitting someone down who's unfamiliar with the software and watching them use it. Fight the urge to correct them, just watch them flounder. Its very educational. Steve Krug refers to this as "Advanced Common Sense" and has an excellent book called "Don't Make Me Think" which covers cheap, simple user interaction testing. I highly recommend it. It's a very short and eye opening read.
Finally, the client themselves, if their expectations are properly prepared, can be a fantastic test suite. Be sure they understand its a work in progress, that it will have bugs, that they're helping to make their product better, and that it absolutely should not be used for production data, and let them tinker with the pre-release versions of your product. They'll do all sorts of things you never thought of! They'll be the best and most realistic testing you ever had, FOR FREE! Give them a very simple way to report bugs, preferably just a one button box right on the application which automatically submits their environment and history; the feedback box on Hiveminder is an excellent example. Respond to their bugs quickly and politely (even if its just "thanks for the info") and you'll find they'll be delighted you're so responsive to their needs!
Yes, it is. I just ran into an issue this week with a web site I am working on. I just recently switched-out the data access layer and set up unit tests for my controllers and repositories, but not the UI interactions.
I got bit by a pretty obvious bug that would have been easily caught if I had integration tests. Only through integration tests and UI functionality tests do you find issues with the way different tiers of the application interact with one another.
It really depends on the structure and architecture of your web application. If it contains an application logic layer, then that layer should be easy to unit test with automating tools such as Visual Studio. Also, using a framework that has been designed to enable unit testing, such as ASP.NET MVC, helps alot.
If you're writing a lot of Javascript, there have been a lot of JS testing frameworks that have come around the block recently for unit testing your Javascript.
Other than that, testing the web tier using something like Canoo, HtmlUnit, Selenium, etc. is more a functional or integration test than a unit test. These can be hard to maintain if you have the UI change a lot, but they can really come in handy. Recording Selenium tests is easy and something you could probably get other people (testers) to help you create and maintain. Just know that there is a cost associated with maintaining tests, and it needs to be balanced out.
There are other types of testing that are great for the web tier - fuzz testing especially, but a lot of the good options are commercial tools. One that is open source and plugs into Rails is called Tarantula. Having something like that at the web tier is a nice to have run in a continuous integration process and doesn't require much in the form of maintenance.
Unit tests make sense in TDD process. They do not have much value if you don't do test-first development. However the acceptance test are a big thing for quality of the software. I'd say that acceptance test is a holy grail of the development. Acceptance tests show whether the application satisfies the requirements. How do I know when to stop developing the feature --- only when all my acceptance test pass. Automation of acceptance testing a big thing because I do not have to do it all manualy each time I make changes to the application. After months of development there can be hundreds of test and it becomes unfeasible (sometime impossible) to run all the test manually. Then how do I know if my application still works?
Automation of acceptance tests can be implemented with use of xUnit test frameworks, which makes a confusion here. If I create an acceptance test using phpUnit or httpUnit is it a unit test? My answer is no. It does not matter what tool I use to create and run test. Acceptance test is the one that show whether the features is working IAW requirements. Unit test show whether a class (or function) satisfies the developer's implementation idea. Unit test has no value for the client (user). Acceptance test has a lot of value to the client (and thus to developer, remember Customer Affinity)
So I strongly recommend creating automated acceptance tests for the web application.
The good frameworks for the acceptance test are:
Sahi (sahi.co.in)
Silenium
Simpletest (I't a unit-test framework for php, but includes the browser object that can be used for acceptance testing).
However
You have mentioned that web-site is all about user interaction and thus test automation will not solve the whole problem of usability. For example: testing framework shows that all tests pass, however the user cannot see the form or link or other page element due to accidental style="display:none" in the div. The automated tests pass because the div is present in the document and test framework can "see" it. But the user cannot. And the manual test would fail.
Thus, all web-applications needs manual testing. The automated test can reduce the test workload drastically (80%), but manual test are as well significant for the quality of the resulting software.
As for the Unit testing and TDD -- it make the code quality. It is beneficial to the developers and for the future of the project (i.e. for projects longer that a couple of month). However TDD requires skill. If you have the skill -- use it. If you don't consider gaining the skill, but mind the time it will take to gain. It usually takes about 3 - 6 month to start creating a good Unit tests and code. If you project will last more that a year, I recommend studding TDD and investing time in proper development environment.
I've created a web test solution (docker + cucumber); it's very basic and simple, so easy to understand and modify / improve. It lies in the web directory;
my solution: https://github.com/gyulaweber/hosting_tests