We have designed an application, actually a framework, keeping few use-cases/scenarios in-mind. Now we want to verify our design with few other use cases?
Are there any tools - probably process tools - to verify if the use cases out there fits into our design?
I guess you can always write a testcase for each new scenario and take it from there.
If you are interested in tools, Fitnesse (http://fitnesse.org/) is a well known test acceptance framework which you can use to document behavior and map it to automated test cases. It will allow you to define tests using simple sentences, group them in hierarchies, and then put simple implementations (Junit, TestNG, etc.) behind each statement.
Related
I have a C# api. Now I also need a Java implementation of it. And to maintain both api then.
To ensure both implementations don't diverge with time, I'd like to share the same tests between them (not necessarily the unit tests, but at least the end-to-end tests).
So far, I can think of two different ways to do it:
Put both implementations behind a rest api, and test through this
api. The advantage is that I can have the exact same tests for both
implementations. The drawback is that it’s a bit heavy to put in
place.
Use Behavior Driven Development tests (Cucumber for Java, SpecFlow for C#), and use the same feature files
for the different implementations. The drawback is that I’ll have to
provide an implementation of the steps for each language, so there’s
the risk that the tests are actually different in subtle ways.
I would be grateful for any idea to handle it in a more satisfactory way.
IMHO the Facade for the both APIs is the better option. It'll allow you to reuse the same test suites above unit testing (Integration, System, E2E, UAT).
The drawback is that it’s a bit heavy to put in place.
I doubt that having double BDD frameworks will be a less effort. Your language bindings will have to be different (e.g. *.feature.cs and .feature.java files). The unit test framework that you'll have to put in place (NUnit, MSTest, JUnit, TestNG etc.) will require a separate CI server handling.
In the same time having only one implementation for all your tests will be a lot more feasible. It's not mandatory to use the same language for tests as the SUT's one. All the tests will be in sync for both the APIs requirements, regardless of the language.
There are a lot of testing methods out there i.e. blackbox, graybox, unit, functional, regression etc.
Obviously, a project cannot take on all testing methods. So I asked this question to gain an idea of what test methods to use and why should I use them. You can answer in the following format:
Test Method - what you use it on
e.g.
Unit Testing - I use it for ...(blah, blah)
Regression Testing - I use it for ...(blah, blah)
I was asked to engage into TDD and of course I had to research testing methods. But there is a whole plethora of them and I don't know what to use (because they all sound useful).
1. Unit Testing is used by developers to ensure unit code he wrote is correct. This is usually white box testing as well as some level of black box testing.
2. Regression Testing is a functional testing used by testers to ensure that new changes in system has not broken any of existing functionality
3. Functional testing is testing conducted on a complete, integrated system to evaluate the system's compliance with its specified requirements. Functionality testing falls within the scope of black box testing, and as such, should require no knowledge of the inner design of the code or logic
.
This Test-driven development and Feature Driven Development wiki articles will be of great help for you.
For TDD you need to follow following process:
Document feature (or use case) that
you need to implement or enhance
in your application that
currently does not exists.
Write set of functional test
cases that can ensure above
feature (from step 1) works. You may need to
write multiple test cases for
above feature to test all different
possible work flows.
Write code to implement above feature (from step 1).
Test this code using test cases you
had written earlier (in step 2). The actual
testing can be manual but I would recommend to create automated tests
if possible.
If all test cases pass, you are good to
go. If not, you need to update code (go back to step 3)
so as to make the test case pass.
TDD is to ensure that functional test cases which were written before you coded should work and does not matter how code was implemented.
There is no "right" or "wrong" in testing. Testing is an art and what you should choose and how well it works out for you depends a lot from project to project and your experience.
But as a professional Test Expert my suggestion is that you have a healthy mix of automated and manual testing.
(Examples below are in PHP but you can easily find the correct examples for what ever langauge/framework you are using)
AUTOMATED TESTING
Unit Testing
Use PHPUnit to test your classes, functions and interaction between them.
http://phpunit.sourceforge.net/
Automated Functional Testing
If it's possible you should automate a lot of the functional testing. Some frame works have functional testing built into them. Otherwise you have to use a tool for it. If you are developing web sites/applications you might want to look at Selenium.
http://www.webinade.com/web-development/functional-testing-in-php-using-selenium-ide
Continuous Integration
Use CI to make sure all your automated tests run every time someone in your team makes a commit to the project.
http://martinfowler.com/articles/continuousIntegration.html
MANUAL TESTING
As much as I love automated testing it is, IMHO, not a substitute for manual testing. The main reason being that an automated can only do what it is told and only verify what it has been informed to view as pass/fail. A human can use it's intelligence to find faults and raise questions that appear while testing something else.
Exploratory Testing
ET is a very low cost and effective way to find defects in a project. It take advantage of the intelligence of a human being and a teaches the testers/developers more about the project than any other testing technique i know of. Doing an ET session aimed at every feature deployed in the test environment is not only an effective way to find problems fast, but also a good way to learn and fun!
http://www.satisfice.com/articles/et-article.pdf
This answer is (almost) identical to one that I gave to another question. Check out that question since it had some other good answers that might help you.
How can we decide which testing method can be used?
I usually do the following things:
Page consistency in case of multi-page web sties.
Testing the database connections.
Testing the functionalities that can be affected by the change I just made.
I test functions with sample input to make sure they work fine (especially those that are algorithm-like).
In some cases I implement features very simply hard-coding most of the settings then implement the settings later, testing after implementing every setting.
Most of these apply to applications, too.
Well before going to the answer i would like to clear testing concept about multiple methods.
There are six main testing types which cover all most all testing methods.
Black Box Testing
White Box Testing
Grey Box Testing
Functional Testing
Integration Testing
Usability Testing
Almost all Testing methods lies under these types, you can also use some testing method in multiple types like you can use Smoke testing in black box or white box approach on the basis of resources available to test.
So for testing a web site completely you need to use at least following testing methods on the basis of resources available to test. These are at least methods which should be used to test a web site, but there may be some more imp methods on the basic of nature of website.
Requirement Testing
Smock Testing
System Testing
Integration Testing
Regression Testing
Security Testing
Performance & Load Testing
Deployment Testing
You should at least use all of above (8) testing methods to test a web site no matter what testing type you are focusing. You can automate you test in some areas and you can do this manually it all depends upon the resources availability.
There is specifically no hard and fast rule to follow any testing type or any method. As you know "Testing Is An ART" so art don't have rules or boundaries. Its totally up to you What you use to test and how.......
Hope you got the answer of question.
Selenium is very good for testing websites.
The answer depends on the Web framework used (if any). Django for example has built-in testing functions.
For PHP (or functional web testing), SimpleTest is pretty good and well... simple. It support Unit Testing (PHP only) and Web Testing. Tests can run in the IDE (Eclipse), or in the browser (meaning on your server).
The other answers posted so far focus on unit/functional/performance/etc. testing, and they are all reasonable.
However, one the key questions you should ask is, "how effective is my testing?".
This is often answered with test coverage tools, that determine which parts of your application actually get exercised by some set of tests. The ideal test coverage tool lets you test your application by any method you can imagine (including all the standard answers above) and will then report what part and what percentage of your code was exercised. Most importantly, it will tell you what code you did not exercise. You can then inspect that code and decide if more testing is warranted, or if you don't care. If the untested code has to do with "disk full error handling" and you belive that 1TB disks are common, you might decide to ignore that. If the untested code is the input validation logic leading to SQL queries, you might decide that you must test that logic to ensure that no SQL injection attacks can occur.
What test coverage tools let you do it to make a rational decision that you have tested adequately, using data about what parts of your code has been exercised. So regardless of how you test, best practices indicates you should also do test coverage analysis.
Test coverage tools can be obtained from a variety of sources. SD provides a family of test coverage tools that handle C, C++, Java, C#, PHP and COBOL, all of which are used to support web site testing in various ways.
I am writing an application that uses 3rd party libraries to instantiate and make some operations on virtualmachines.
At first I was writing integration tests to every functionality of the application. But them I found that these tests were not really helping since my environment had to be at a determined state, which turned the tests more and more difficult to write. And I decided to make only the unit and acceptance tests.
So, my question ... is/can there be method or a clue to notice when the integration tests are not to be used?? (or I am wrong and on all cases they should be written)
When you don't plan on actually hooking your application up to anything "real"; no real containers, databases, resources or actual services. That's what an integration test is supposed to verify; that everything works properly together.
Integration tests are good to test a full system that has well-defined inputs and outputs that are unlikely to change. If your expected input/outputs change often then maintaining the test may become a maintenance challenge, or, worse, you may choose against improving an interface because of the amount of work that may be required to upgrade the integration tests.
The easy and short rule is: Test in integration test what breaks due to integration and test the rest in unit tests in isolation.
You can even hate integration tests. Writing a unit test for a function that takes only one integer parameter is hard enough. All possible combinations of state (internal and external(time, external systems)) and input can make integration testing practically impossible (for a decent application.)
Can anybody suggest a good tool to test GWT UI. What is the opinion about using Selenium & Webdriver ?
Any pointers and suggestions are welcome.
Thanks,
SD
Google themselves have an excellent article describing when and how to apply different testing strategies to GWT apps. The basic idea is to use unit-testing whenever you are testing business logic, but to use Selenium for specific end-to-end tests where you want to simulate user interaction.
The most important thing is limit how much GWT-specific testing you need to do since GWTTestCase and Selenium are so slow. The MVP pattern referenced in the Testing Methodologies helps enormously with this. Always ask yourself "Can I make this view any dumber?" Push logic for dynamically changing the UI into presenters that manipulate views so that you can test the logic in JUnit.
Then I'd suggest doing most of the UI testing in Selenium/Webdriver. Those tests should be fairly simple though: "Does X show up when I click Y?" kinds of things. GWTTestCase ends up being the most useful for me for code that needs to run under Java and GWT.
Or, as an alternative to the MVP pattern, you may use the gwt-test-utils framework to write simple tests in java, which runs faster then GWTTestCase tests :-)
You can do bog standard unit testing for your GWT components, if you want to do some layout checking things which can't be done through unit tests (can't give you examples since I don't know GWT nearly at all) then you may use Selenium, however Selenium does have some issues in specific cases, namely selecting elements and very rich content which may prove to be hard to handle since GWT is all about rich content.
I read the Wikipedia article on scenario testing, but I am sad to say it is very short. I am left wondering: are scenario tests a collection of sequential unit tests? Or, perhaps, like a single multi-step unit test? Do many frameworks support scenario tests, or are they covered by unit testing?
If they have nothing to do with automation, what are they?
I don't think there's any fixed relationship between the number and distribution of tests and scenario tests.
I think the most common code-representation of a scenario is a specific set of business data required to support a specific story (scenario). This is often provided in the form of database data, fake stub data or a combination of both.
The idea is that this dataset has known and well-defined characteristics that will provide well defined results all across a given business process.
For a web application I could have a single web-test (or several for variations) that click through the full scenario. In other cases the scenario is used at a lower level, possibly testing a part of the scenario in a functional test or a unit test. In this case I normally never group the tests by scenario, but choose the functional grouping of tests I normally use for unit/functional tests. Quite often there's a method within "Subsystem1Test" that is called "testScenario1" or maybe "testScenarioInsufficientCredit". I prefer to give my scenarios names.
In addition to korsenvoid's response, in my experience scenario based testing will often be automated as it will be included in regression testing. Regression testing is regularly automated as to do it manually does not scale well with regular releases.
In commercial software, good examples of scenerio tests are tutorials included with the user documentation. These obviously must work in each release or be removed from the docs, and hence must be tested.
While you can carry out scenario testing using sequenced unit tests, my guess is that it is more common to use GUI based automation tools. For example, I use TestComplete in this role with a scripting framework to good effect. Scenario tests are typically carried out from a user/client perspective which can be difficult to accurately replicate at a unit level.
IMHO, scenario testing is a testing activity, as opposed to development activity ; hence it's about testing a product, not unit(s) of that product. The test scenario are end-to-end scenarios, using the natural interfaces of the product. If the product has programmatic interfaces, then you could use an unit test framework, or Fitnesse.