So I'm trying to figure out a Software Testing technique that would decrease the amount of test combinations inputs of a Radio Button dialog. The dialog has three sets of radio buttons. The scenario is for an application on a Flight.
1st Set - {England, Scotland, Wales, Northern Ireland}
2nd Set - {Aisle, Window Seat}
3rd Set - {Coach, Business, Economy}
Using Cartesian Method the total number of tests for possible combinations inputs is 4X2X3 = 24 Tests
I want to reduce this number of combinations with a valid functional testing technique. So my question is what type of testing will this be?
Suggestions will be very helpful. Thanks
I did some reading and I managed to find the most suitable functional testing technique for this is Pairwise combinatorial testing.
This testing is the most suitable technique for combinations as it drastically decreases the number of test case count therefore, uses a much smaller test suite. Apart from decreasing test count, it cannot compromise on the test coverage.
The test suite can cover all combinations; therefore, it is not exhaustive yet very effective in finding bugs.
For more on pairwise testing you can read in this link:
pairwise testing!
Related
I am working on a project that attempts to use simulated annealing to organise various items into designated sections while maintaining some stability requirements.
For this problem the neighbourhood configuration should remove two or more of these items and switch their places (there are exactly as many items as there are places for them), but I am unsure of how calculated or random this switch should be.
When testing if the stability requirements are fulfilled, I am able to tell vaguely how items should be repositioned in order to improve the stability, but should the neighbourhood configuration take this into account? Or will that make the algorithm too "greedy"?
Any help/direction is much appreciated!
I have some experience with simulated annealing problems, but I don't know what you mean by 'neighbourhood configuration'. My scenario used the following rules: 1) if making the simulation lower energy, always accept the exchange 2) if making the simulation higher energy, calculate a probability of exchange using the Boltzmann factor and randomise a yes/no for acceptance. "Energy" is abstracted as counting all the non-same-type nearest neighbour interactions and subtracting all the same-type interactions, so that segregating into regions of similar neighbours is energetically favourable.
I am looking for a Test suite optimization using a Genetic Algorithm(GA). There are multiple IEEE research papers suggesting this, but I need help on implementing the same (using python).
These are my GA steps:
Representation: How to represent test case features in GA?
Initialize and evaluate (Fitness Function): How to create a Fitness function - formula?
Parent selection, Crossover and Mutation: Running though the GA
Output: Inferring from the output
I've done a lot of research on this, but I could not get on to the phase where I can implement.
For test case optimization (selecting the best test case) I use GA python packages.
Any help is much appreciated.
that's what I've been looking for as well.
Representation is a fist issue, but we can give ids for each step in the test case.
The fitness function could be a grade of how much that step covers, and the time spent to execute it.
Selection can be random, or we can specify pre/post step to use in a test case
(Each step can have a pre/post step).
Then, we can generate lots of test cases containing steps.
And get the best ones in each generation based on that FF.
Still working on some coding. We should be in contact.
What are the different metrics we use for assuring the quality of test suites written based on only requirements and specifications (black box)?
Simply put, given a set of requirements and a test suite on those requirements, what are different metrics to quantify the quality of specification/requirement based testing (test suite)?
I read through the following articles regarding specification based testing and metrics to define them. These topics are too abstract to digest.
http://link.springer.com/chapter/10.1007%2F978-3-642-21768-5_13#page-1
http://www.worldscientific.com/doi/abs/10.1142/S0218539301000530
Can you please explain in simple terms?
Thanks!
The simplest way to evaluate specification-based testing is to trace each specification to a test (whether manual or automated), count which specifications are tested and which are not, and calculate percent coverage.
The confusion related to the articles you linked to is due to confusion between "specification" used to refer to a human-written, structured but relatively informal document, and "specification" meaning a formal computer-readable specification from which tests can be automatically derived.
It's also possible to measure code coverage during specification-based testing. However, it's very difficult to improve coverage without looking inside the black box. Also, specification-based tests are slow, even when automated, so it's painful to achieve code coverage using only specification-based tests. A better approach is to combine black-box specification-based tests and white-box unit tests and consider overall coverage.
If someone has a webpage, the usual way of testing the web site for user interaction bugs is to create each test case by hand and use selenium.
Is there a tool to create these testcases automatically? So if I have a webpage that gets altered, new test cases get created automatically?
You can look at a paid product. That type of technology is not being developed as open source and will probably cost a bit. Some of the major test tools get closer to this, but full auto I have not heard of.
If this was the case the role of QA Engineer and especially Automation Engineer would not be as important and the jobs would spike downwards pretty quickly. I would imagine that if such a tool was out there that it would be breaking news to the entire industry and be world wide.
If you go down the artificial intelligence path this is possible in theory and concept, however, usually artificial intelligence development efforts costs more than the app being developed that needs the testing, so...that's not going to happen.
The best to do at this point is separate out as much of the maintenance into a single section from the rest so you limit the maintenance headache when modyfying and keep a core that stays the same. I usually focus on control manipulation as generic and then workflow and specific maps and data change. That will allow it to function against any website...but you still have to write/update the tests and maintain the maps.
I think Growing Test Cases Automatically is more of what your asking. To be more specific I'll try to introduce basics and if you're interested take a closer look at Evolutionary Testing
Usually there is a standard set of constraints we meet like changing functionality of the system under test (SUT), limited timeframe, lack of appropriate test tools and the list goes on… Yet there is another type of challenge which arises as technological solutions progress further – increase of system complexity.
While the typical constraints are solvable through different technical and management approaches, in the case of system complexity we are facing the limit of our capability of defining a straight-forward analytical method for assessing and validating system behavior. Complex system consist of multiple, often heterogeneous components which when working together amplify each other’s statistical and behavioral deviations, resulting in a system which acts in ways that were not part of its initial design. To make matter worse, complex systems increase sensitivity to their environment as well with the help of the same mechanism.
Options for testing complex systems
How can we test a system which behaves differently each time we run a test scenario? How can we reproduce a problem which costs days and millions to recover from, but happens only from time to time under conditions which are known just approximately?
One possible solution which I want to focus on is to embrace our lack of knowledge and work with whatever we have by using evolutionary testing. In this context the evolutionary testing can be viewed as a variant of black-box testing, because we are working with feeding input into and evaluating output from a SUT without focusing on its internal structure. The fine line here is that we are organizing this process of automatic test case generation and execution on a massive scale as an iterative optimization process which mimics the natural evolution.
Evolutionary testing
Elements:
• Population – set of test case executions, participating into the optimization process
• Generation – the part of the Population, involved into given iteration
• Individual – single test case execution and its results, an element from the Population
• Genome – unified definition of all test cases, model describing the Population
• Genotype – a single test case instance, a model describing an Individual, instance of the Genome
• Recombination – transformation of one or more Genotypes into a new Genotype
• Mutation – random change in a Genotype
• Fitness Function – formalized criterion, expressing the suitability of the Individual against the goal of the optimization
How we create these elements?
• Definition of the experiment goal (selection criteria) – sets the direction of the optimization process and is related to the behavior of the SUT. Involves certain characteristics of SUT state or environment during the performed test case experiments. Examples:
o “SUT should complete the test case execution with an error code”
o “The test case should drive the SUT through the largest number of branches in SUT’s logical structure”
o “Ambient temperature in the room where SUT is situated should not exceed 40 ºC during test case execution”
o “CPU utilization on the system, where SUT runs should exceed 80% during test case execution”
Any measurable parameters of SUT and its environment could be used in a goal statement. Knowledge of the relation between the test input and the goal itself is not obligatory. This gives a possibility to cover goals which are derived directly from requirements, rather than based on some late requirement derivative like business, architectural or technical model.
• Definition of the relevant inputs and outputs of the tested system – identification of SUT inputs and outputs, as well as environment parameters, relevant to the experiment goal.
• Formal definition of the experiment genome – encoding the summarized set of test cases into a parameterized model (usually a data structure), expressing relevant SUT input data, environment parameters and action sequences. This definition also needs to comply with the two major operations applied over genome instances – recombination and mutation. The mechanism for those two operations can be predefined for the type of data or action present in the genome or have custom definitions
• Formal definition of the selection criteria (fitness function) – an evaluation mechanism which takes SUT output or environment parameters resulting from a test case execution (Individual) and calculates a number (Fitness), signifying how close is this particular Individual to the experiment goal.
How the process works?
We use the Genome to create a Generation of random Genotypes (test case instances).
We execute the test cases (Genotypes) generating results (Individuals)
We evaluate each execution result (Individual) against our goal using the Fitness Function
We select only those Individuals from given Generation which have Fitness above a given threshold (the top 10 %, above the average, etc.)
We use the selected individuals to produce a new, full Generation set by applying Recombination and Mutation
We repeat the process, returning on step 2
The iteration process usually stops by setting a condition with regard to the evaluated Fitness of a Generation. For example:
• If the top Fitness hasn’t changed with more than 0.1% since the last Iteration
• If the difference between the top and the bottom Fitness in a Generation is less than 0.3%
then probably it is time to stop.
Upsides and downsides
Upsides:
• We can work with limited knowledge for the SUT and goal-oriented test definitions
• We use a test case model (Genome) which allows us to mass-produce a large number of test cases (Genotypes) with little effort
• We can “seed” test cases (Genotypes) in the first iteration instead of generating them at random in order to speed up the optimization process.
• We could run test cases in parallel in order to speed up the process
• We could find multiple solutions which meet our test goal
• If the optimization process in convergent we have a guarantee that each following Generation is a better approximate solution of our test goal. This means that even if we need to stop before we have reached optimal Fitness we will still have better test cases than the one we started with.
• We can achieve replay of very complex, hard to reproduce test scenarios which mimic real life and which are far beyond the reach of any other automated or manual testing technique.
Downsides:
• The process of defining the necessary elements for evolutionary test implementation is non-trivial and requires specific knowledge.
• Implementing such automation approach is time- and resource-consuming and should be employed only when it is justifiable.
• The convergence of the optimization process depends on the smoothness of the Fitness Function. If its definition results in a zones of discontinuity or small/no gradient then we can expect slow or no convergence
Update:
I also recommend you to look at Genetic algorithms and this article about Test data generation can give you approaches and guidelines.
I happen to develop ecFeed - an open-source tool that may assist in test design. It's in pre-release phase and we are going to add better integration with Selenium, but you may have a look at the current snapshot: https://github.com/testify-no/ecFeed/wiki . The next version should arrive in October and will have major improvements in usability. Anyway, I am looking forward for constructive criticism.
In the Microsoft development world there is Visual Studio's Coded UI Test framework. This will record your actions in a web browser and generate test cases to replicate that use case. It won't update test cases with any changes to code though, you would need to update them manually or re-generate.
What is Orthogonal testing technique?
What is Combinatorial testing technique?
What is the difference between them?
I went through wikipedia and other articles and books but still i am unable to understand them.
Orthogonal testing- Orthogonal array testing is a black box testing technique.
It is used when number of inputs to the application under test is small but too complex for an exhaustive testing. It is very effective in finding errors.
All Pair Testing- It is type of testing Technique to test all the pairs using combinatorial method.
1- Pairwise testing is more efficient and effective than orthogonal arrays for software testing.
2- Orthogonal testing is more effective for manufacturing, and agriculture, and advertising.
3- Pairwise testing strategy better than an orthogonal array strategy.
4- Both Type of testing Technique reduce the number of Test cases.
5- All Pair testing almost requires fewer test cases than orthogonal testing, sometimes both have same number of test cases.
Both type of testing Strategy have almost same features but at the same time the use(which testing Strategy to choose) depends on the requirement.
Please refer this pdf for more details-
http://www.51testing.com/ddimg/uploadsoft/20090113/OATSEN.pdf
Tutorial for Pairwise Testing with examples-
http://www.tutorialspoint.com/software_testing_dictionary/all_pairs_testing.htm
Refer the below link .I think it is useful to you.
http://www.softwaretestinghelp.com/combinational-test-technique/
Orthogonal Array can be of any strength.
when its combination of any 2 Variables its called all pairs
it can be combination of any 3, 4, 5...n variables based on the business need. All pairs is just one of the subclasses of orthogonal testing. Hope this helps