What is pairwise testing? Is this method preferred over equivalence classes and boundary value testing? Does pairwise testing require the use of automated test case generation software?
pairwise testing is a combinatorial method of software testing that, for each pair of input parameters to a system (typically, a software algorithm), tests all possible discrete combinations of those parameters.
Here is a more detailed example: http://www.tutorialspoint.com/software_testing_dictionary/pairwise_testing.htm
The rest of your question is similar to this Equivalence Class Testing vs. Boundary Value Testing
Equivalence Class Testing
EC Testing is when you have a number of test items (e.g. values) that you want to test but because of cost (time/money) you do not have time to test them all. Therefore you group the test item into class where all items in each class are suppose to behave exactly the same. The theory is that you only need to test one of each item to make sure the system works.
Example 1
Children under 2 ride the buss for free. Young people pay $10, Adults $15 and Senior Citizen pay $5.
Classes:
Price:0 -> Age:0-1
Price:10 -> Age:2-14
Price:15 -> Age:15-64
Price:5 -> Age:65-infinity
Example 2 (more than one parameter)
Cellphones K80, J64 and J54 run Java 5. K90 and J99 run Java 6. But there are two possible browsers FireFox and Opera, J models run FF and K models run O.
Classes:
Browser:FF, Java:5 -> Phones:J64,J54
Browser:FF, Java:6 -> Phones:J99
Browser:O, Java:5 -> Phones:K80
Browser:O, Java:6 -> Phones:K90
Dangers Of Equivalence Class Testing
There is a danger of using EC Testing that is rarely mentioned in the testing books but is very important to remember.
Just because two items/values are suppose to be in the same class and behave the same, does not mean they DO behave the same.
That means that just because you test one value in the class that ALL values in the class behave the same. Real world example of mine is with cell phones that all had a certain Java Platform. They were suppose to all work the same but they didn't in reality. So testing just one value in a class is good, but not good enough. EC Testing is a good tool, but it's not fool proof and be careful with it. If test cases are cheap and fast (like automation), test more, or why not test them all!
Boundary Value Testing
BV Testing is when you decide to test the values on the edge of each Class you have identified. The theory is that most defects is around the edges of a class.
Example
Classes:
Price:0 -> Age:0-1 ( Boundary values 0, 1)
Price:10 -> Age:2-14 ( Boundary values 2, 14)
Price:15 -> Age:15-64 ( Boundary values 15, 64)
Price:5 -> Age:65-infinity ( Boundary values 65)
Pairwise testing is the method to test the application for all possible inputs combination.
In market lot of Pairwise Testing Tools are available use below link as -
EX >> system has ten configuration parameters, and every configuration parameter has ten different values. To test that system behaves correctly with all the different configurations(around 10 billion combination) is not possible.
with the help of tool like (Allpairs, PICT etc) find a small set of test cases to satisfy that coverage standard.
The proposal of pairwise testing for above example is that it is enough to test for all the possible value combinations for any two of the ten variables.
All pairs or pairwise testing is simpler then you think. If you have 10 inputs and each input has 5 equivalence classes. You calculate the number of pair inputs which is the inputs - 1 summed. so 9+8+7+6+5+4+3+2+1 = 45. Then you take the equivalence classes^2 so 5^2 and you get 25. Then you multiply 25*45 = 1125 test cases.
I hope this helps.
Related
I'm interested in knowing the possibilities of this. I'm working on a project that validates the skills of a software engineer, currently we validate skills based on code reviews by credentialed developers.
I know the answer if far more completed that the question, I couldn't imagine how complex the program would have to be able to analyse complex code but I am starting with basic programming interview questions.
For example, the classic FizzBuzz question:
Write a program that prints the numbers from 1 to 20. But for multiples of three print “Fizz” instead of the number and for the multiples of five print “Buzz”. For numbers which are multiples of both three and five print “FizzBuzz”.
and below is the solution in python:
for num in range(1,21):
string = ""
if num % 3 == 0:
string = string + "Fizz"
if num % 5 == 0:
string = string + "Buzz"
if num % 5 != 0 and num % 3 != 0:
string = string + str(num)
print(string)
Question is, can we programatically analyse the validity of this solution?
I would like to know if anyone has attempted this, and if there are current implementations I can take a look at. Also if anyone has used z3, and if it is something I can use to solve this problem.
As Vilx- mentioned, correctness of programs (including whether or not they terminate) is in general known to be undecidable. However, tools such as Z3 show that relevant concrete cases can still be reasoned about, despite the general undecidability of the problem.
Static analysers typically look for "simple" problems (e.g. null dereferences, out-of-bounds accesses, numerical overflows), but are comparably fast and require little user guidance (think of guidance in the spirit of adding type annotations to your code).
A non-exhaustive (and biased) list of keywords to search for: "static analysers", "abstract interpretation"; "facebook infer", "airbus absint", "juliasoft".
Verifiers attempt to prove much richer properties, in particular functional correctness, e.g. "does this sort-implementation really sort my array (and not do anything else, e.g. deallocate some global memory or update an element reachable from the array)?" or "does that crypto-implementation really implement the crypto protocol it promises to implement?". This is a much harder task and tools from that line of research are typically rather slow, require expert users with a background in formal verification and significant user guidance.
A non-exhaustive (and biased) list of keywords to search for: "verification", "hoare logic", "separation logic"; "eth viper", "microsoft dafny", "kuleuven verifast", "microsoft f*".
Other formal methods exist, e.g. refinement (or correct-by-construction), but with even less tool support and, as far as I know, industry acceptance.
Let's put it this way: it's been mathematically proven that you CANNOT determine if a program will ever terminate. So if you want a mathematically perfect answer of if the target program is correct, you're doomed.
That said, you can still do unit tests and "linting" which will give you plenty of intetesting insights.
But for simple pieces of code like the FizzBuzz, I think that eyeballing by an experienced dev will probably bring the best results.
What are all the factors one should consider in order to compare 3 formal verification tools?
Eg: Jaspergold, Onespin, Incisive.
From my little research, Jaspergold comes on top. But i want to do it myself on a project.
I have noted down some points such as
1.Supported languages(vhdl, sv, verilog, sva, psl,etc)
2.GUI
3.Capability(how much big design can they handle)
4.Number of Evaluation cycles
5.Performance(How fast they find proof or counter example)
With what other features can i extend this list?
Thanks!
If you write a program, it is usually possible to drive it such that all paths are covered. Hence, 100% coverage is easy to obtain (ignoring unfeasible code paths which modern compilers catch anyways).
However, 100% code coverage should imply that all variable definition-use coverage is also achieved, because variables are defined within the program and used within it. If all code is covered, all DU pairs should also be covered.
Why then, is it said that path coverage is easier to obtain, but data flow coverage is not usually possible to achieve 100% ? I do not understand why not? What can be an example of that?
It's easier to achieve 100% code coverage than all of the possible inputs because the set of all possible inputs can be extremely large or practically unlimited. It would take too much time to test them all.
Let's look at a simple example function:
double invert(double x) {
return 1.0/x;
}
A unit test would could look like this:
double y = invert(5);
double expected = 1.0/5.0;
EXPECT_EQ( expected, y );
This test achieves 100% code coverage. However, it's only 1 in 1.8446744e+19 possible inputs (assuming a double is 64 bits wide).
The idea behind All-pairs Testing is that it's not practical to test every possible input, so we have to identify the ranges that would cover all cases.
With my invert() function, there are at least two sets that matter: {non-zero values} and {zero}.
We need to add another test, which covers the same code path, but has a different outcome:
EXPECT_THROWS( invert(0.0) );
Furthermore, since the test writer has to design the different possible sets of parameters to achieve full data input coverage to a test, it could be impossible to know what the correct sets are.
Consider this function:
double multiply(double x, double y);
My instinct would be to write tests for small numbers and another for big numbers, to test overflow.
However, the developer may have written it poorly, in this way:
double multiply(double x, double y) {
if(x==0) return 0;
return 1.0 / ( (1.0/x) * (1.0/y) );
}
If our tests didn't use 0 for y, then we'd miss a bug. Knowledge of how the algorithms are designed is very important in understanding the proper inputs for a unit test, and that's why the programmers who write the code need to be involved in unit testing.
i am currenty studying BDD, but i have a different doubt, can you tell me if the following is right or not:
1) Feature = it means "the problem" isn't it?
2) Scenario = the way (the beahaviour) to resolve the feature
I find very difficult to find the "given when then" sentences.
In this problem for example:
As a student
I would like / i want to calculate the rectangle perimeters if i have 2 number
Or the circle area if i have one
So i don't make mistake with the computation
I wrote down the scenario, is that correct?
Given 1 number
Or 2 number
When i have 1 positive number
Or 2 positive number
Then calculate the area
Or the Perimeters
About the terminology:
1) feature is not a "problem". It would rather be a solution.
In software programming, a feature is a something that your program does to solve a problem.
A feature could be the ability to compute the area of a rectangle.
2) a scenario is a description of the usage of your feature. Like an example.
Like a test case, but usually in a more human-readable form.
3) a story (in Agile terminology, in which BDD stands) is a way to describe the a need/problem.
Your problem ("as a student...") is presented as a story.
This story will lead to a new feature in your soft.
This new feature will be tested by scenarios.
About your scenarios.
Yours are not correct.
There is no way to know that if you have 1 nb you should compute an area.
You should have several scenarios, like
Given I send the number 2
When I launch the computation
Then I get the result 12,56
Given I send the number 2 and 3
When I launch the computation
Then I get the result 10
Given I send the number -4
When I launch the computation
Then I get the result error
Given I send the number 1 3 7
When I launch the computation
Then I get the result error
I understand how equivalence testing works.
How is it the same or different from boundary value testing?
Equivalence Class Testing
EC Testing is when you have a number of test items (e.g. values) that you want to test but because of cost (time/money) you do not have time to test them all. Therefore you group the test item into class where all items in each class are suppose to behave exactly the same. The theory is that you only need to test one of each item to make sure the system works.
Example 1
Children under 2 ride the bus for free. Young people pay $10, Adults $15 and Senior Citizen pay $5.
Classes:
Price:0 -> Age:0-1
Price:10 -> Age:2-14
Price:15 -> Age:15-64
Price:5 -> Age:65-infinity
Example 2 (more than one parameter)
Cellphones K80, J64 and J54 run Java 5. K90 and J99 run Java 6. But there are two possible browsers FireFox and Opera, J models run FF and K models run O.
Classes:
Browser:FF, Java:5 -> Phones:J64,J54
Browser:FF, Java:6 -> Phones:J99
Browser:O, Java:5 -> Phones:K80
Browser:O, Java:6 -> Phones:K90
Dangers Of Equivalence Class Testing
There is a danger of using EC Testing that is rarely mentioned in the testing books but is very important to remember.
Just because two items/values are suppose to be in the same class and behave the same, does not mean they DO behave the same.
That means that just because you test one value in the class that ALL values in the class behave the same. Real world example of mine is with cell phones that all had a certain Java Platform. They were suppose to all work the same but they didn't in reality. So testing just one value in a class is good, but not good enough. EC Testing is a good tool, but it's not fool proof and be careful with it. If test cases are cheap and fast (like automation), test more, or why not test them all!
Boundary Value Testing
BV Testing is when you decide to test the values on the edge of each Class you have identified. The theory is that most defects is around the edges of a class.
Example
Classes:
Price:0 -> Age:0-1 ( Boundary values 0, 1)
Price:10 -> Age:2-14 ( Boundary values 2, 14)
Price:15 -> Age:15-64 ( Boundary values 15, 64)
Price:5 -> Age:65-infinity ( Boundary values 65)
Critique of Boundary Value Testing
1) I, and other test professionals I have taken courses from, are not convinced that most defects are hidden around the edges of each class. And I have never seen any studies that proves this to be the case.
2) The fact that you need to use BV Testing proves that EC Testing is flawed since you test more than one value of each class.
3) It's easy to use when using values like integers. But what is a boundary value of class of phones models or browsers versions?
Hidden Boundary Value Testing
The boundary values of a class is often based on the specification of how the system should work. This is all good and well but most systems contain boundaries that are not explained in any spec and you will need to look for yourself. E.g. 'How many characters can I put into the test field before the system fails and breaks.','How big can the data file become before it so slow to read it gets annoying'.
Real world examples
- Pasting one million characters into a text area in FireFox 3.5 on win 7 crashes it
- ReCaptcha has a limit of 16003 characters, does your system handle the 413 that it passes back to it if somebody puts 16004+ characters in field. Or does it break
Summary
EC Testing and BV Testing are great tools and you should use them but they are not perfect and don't expect to find all defects using them. Use your know-how about the system and your intelligence and intuition to try more items and looks for other ways it could fail. And look for the hidden boundaries!
Boundary value analysis simply means to select values near the boundaries of the classes. So you are still dividing the input domain according to the classes then instead of selecting values from the middle of the class use values from the boundaries.
For example, if the input condition is a range from 20 to 70 then you have three classes of input
less than 20
between 20 and 70
more than 70
then for boundary value analysis select input = 19, 20, 21, 69, 70, 71. This type of analysis picks up errors on the boundaries.
The Equivalence testing needs to be supplemented with the Boundary value testing.
For example for equivalent testing of a function that takes values between 1 and 12
(say months of a year) the partitions would be:
values less than 1 (0,-1,-2), invalid partition
values between 1-12, valid partition
values greater than 12,invalid partition
For equivalence testing it is enough to pick one value as test input from each of these partition classes. That would mean tests with value of -2,6, and 15 would be considered enough to test behavior of the function. But these values doesn't catch Off-by-one error which can occur quite often.
With the boundary value testing the test inputs would be : -1,0,1,11,12,13 (at the boundaries), which would catch off-by-one errors.
I see both these testing methods to be a complement of each other.
Boundary value analysis is part or subset of equivalence partitioning. In boundary values analysis, instead of some random value, only values in the boundary are selected.
Boundary value analysis overcome the drawback of the Equivalance class partitioning. If a fix length is g9iven for eg. Mobile number (10 digit.)
The lower boundary in this case is - digit - 1 (ie. 10 - 1 =9)
The upper boundary in this case is - digit +1 (ie 10 + 1 =11)
Now we can perform test for the 9 and 11
Dynamic Testing Types –
Specification Based testing
A. Equivalance Partitioning
A1. Boundary value analysis
A2. Decision Tables
A3. Use case Testing
A4. State Transition testing
Structural Based testing
A. Test Coverage
B. Code coverage
C. Statement coverage
D. Decision coverage
Experience Based Testing
A. Error testing
B. Exploratory Testing
Equivalance Partitioning – It is a technique where tester divide the test conditions into groups and sets. System should handle them equivalently hence called equivalence classes. To test one condition from each partition will work to assume all condition will work in that partition.
EX >> Check addition of the single digit values. i.e. values between 0 and 9.
Values less than -9, i.e. -10,-11, …. (Invalid partition)
Values less than 0, i.e. -1,-2, …. till -9 (Valid Partition)
Values between 0-9 i.e. 0,1 …. till 9 (Valid Partition)
Values greater than 9, i.e. 10,11 (Invalid partition)
Testing the addition for any two values of each partition is enough.
Boundary value testing – It is based on testing at the boundaries between partition.
EX >> Consider below combination.
Addition of -9 and -10,
Addition of -10 and -11
Addition of 0 and -1
Addition of 0 and 1
Addition of 9 and 10
Addition of 11 and 10