Recently, I'm quite excited when reading about the idea of property based testing.
But I have 1 question that I still cannot find the answer anywhere:
How can property based testing ensures that it will test the corner cases every time?
To be more specific, let say I'm using ScalaCheck to test my division function:
def divide(a: Int, b: Int): Int
As the test cases will be generated randomly, how can I be sure that ScalaCheck will check the case where b = 0 every time?
Special cases are often properties of their own. If you try to write a single case that covers b = any integer, then all your properties will end up in one big and complicated test. But you can split the parameter space into multiple chunks e.g.:
b = positive integer
b = negative integer
b = 0
And check them separately.
Some PBT tools will always inject corner cases before generating random ones.
For example, jqwik (for Java) would try 0, 1, -1, Integer.MIN…VALUE and Integer.MAX…VALUE before any random values.
Related
Just wondering if anyone knows why Perl6's log function returns a Num type and not a Rat type.
say (e*e).log.WHAT;
> (Num)
say (2/3).WHAT;
> (Rat)
In mathematics Log is a Continuous function therefore it has mathematically-real values. Num type describes mathematically-real numbers in Perl 6. Rat type describes mathematically-rational numbers.
It's because no one has done the work to make it do anything else yet. It's a situation that the language could handle (not that it's special to Perl 6) but also a situation that you might not want it to handle.
There's no object that represents the natural base e and maintains it as such until it can't any longer (just as Rats don't turn into Nums unless they have to). That's possible and would also allow us to decide how to treat it. Maybe we want a Rat, or FatRat, or even a certain number of decimal places in a Num. But it doesn't do that.
It's not that e is special though. It doesn't work with powers of 10 either:
> 100.log10
2
> 100.log10.^name
Num
The code behind .log10 could check that the operand is a power of 10 and decide to return an Int in that case. But it would have to check every number for that and most numbers aren't a power of 10. Checking all of those would slow down the process. It's easier to make it a little "incorrect".
But you can use .narrow to get a more constrained type possibly:
> 100.log10.narrow.^name
Int
This is different from asking for a particular type and maybe getting a different number:
> (10/3).Int
3
> (10/3).narrow.^name
Rat
And for fun:
> i*i
-1+0i
> (i*i).^name
Complex
> (i*i).narrow.^name
Int
Perl6 is not a computer algebra system, so it treats e*e like any other Num - and once you've got a floating point number, only explicit operations such as rounding should change the type to something like Int or Rat: The computer cannot know if the return value 2e0 of (e*e).log actually represents 2, or some 2+ε.
I'm taking a course in Visual Basic 2010 and I'm trying to get a grasp on this new term called a flag. I kind of understand that it has something to do with a boolean condition. I don't quite understand what a flag is. I see references to it using the term flag. I understand it has something to do when a boolean, a condition triggers a flag. But what is the flag. How do you identify it? Can somebody give me an example.
In general, "Flag" is just another term for a true/false condition.
It may have more specific meanings in more specific contexts. For instance, a CPU may keep "arithmetic flags", each one indicating a true/false condition resulting from the previous arithmetic operation. For instance, if the previous operation was an "ADD", then the flags would indicate whether the result of the add was zero, less than zero, or greater than zero.
I believe the term comes from flags used to signal a go/no go condition, like, a railroad flagman indicating whether or not it is safe for the train to proceed.
You hear this quite a bit with BOOL being a 'Flag' since there are only 2 outcomes either TRUE or FALSE. Using BOOL in your decision making processes is an easy way to 'flag' a certain outcome if the condition is met.
An example could be:
if ($x == TRUE) {
// DO THIS
{
else {
//Flag not tripped, DO THIS
}
You can use this with bitwise operations. It can be used to pack 32 booleans into one integer. Here's a sample:
Dim flags As Integer
Const ADMINISTRATOR = 1
Const USER = 2
Const BLUE = 4
Const RED = 8
flags = ADMINISTRATOR or BLUE
If flags and ADMINISTRATOR then
' Do something since the person is an admin
End If
The ors add flags and ands check if the flag is set.
Now we can check up to 32 booleans for this one variable. Great for storing in a database. You can use bigger datatypes, like a long to store more.
I am working on fairly large Mathematica projects and the problem arises that I have to intermittently check numerical results but want to easily revert to having all my constructs in analytical form.
The code is fairly fluid I don't want to use scoping constructs everywhere as they add work overhead. Is there an easy way for identifying and clearing all assignments that are numerical?
EDIT: I really do know that scoping is the way to do this correctly ;-). However, for my workflow I am really just looking for a dirty trick to nix all numerical assignments after the fact instead of having the foresight to put down a Block.
If your assignments are on the top level, you can use something like this:
a = 1;
b = c;
d = 3;
e = d + b;
Cases[DownValues[In],
HoldPattern[lhs_ = rhs_?NumericQ] |
HoldPattern[(lhs_ = rhs_?NumericQ;)] :> Unset[lhs],
3]
This will work if you have a sufficient history length $HistoryLength (defaults to infinity). Note however that, in the above example, e was assigned 3+c, and 3 here was not undone. So, the problem is really ambiguous in formulation, because some numbers could make it into definitions. One way to avoid this is to use SetDelayed for assignments, rather than Set.
Another alternative would be to analyze the names in say Global' context (if that is the context where your symbols live), and then say OwnValues and DownValues of the symbols, in a fashion similar to the above, and remove definitions with purely numerical r.h.s.
But IMO neither of these approaches are robust. I'd still use scoping constructs and try to isolate numerics. One possibility is to wrap you final code in Block, and assign numerical values inside this Block. This seems a much cleaner approach. The work overhead is minimal - you just have to remember which symbols you want to assign the values to. Block will automatically ensure that outside it, the symbols will have no definitions.
EDIT
Yet another possibility is to use local rules. For example, one could define rule[a] = a->1; rule[d]=d->3 instead of the assignments above. You could then apply these rules, extracting them as say
DownValues[rule][[All, 2]], whenever you want to test with some numerical arguments.
Building on Andrew Moylan's solution, one can construct a Block like function that would takes rules:
SetAttributes[BlockRules, HoldRest]
BlockRules[rules_, expr_] :=
Block ## Append[Apply[Set, Hold#rules, {2}], Unevaluated[expr]]
You can then save your numeric rules in a variable, and use BlockRules[ savedrules, code ], or even define a function that would apply a fixed set of rules, kind of like so:
In[76]:= NumericCheck =
Function[body, BlockRules[{a -> 3, b -> 2`}, body], HoldAll];
In[78]:= a + b // NumericCheck
Out[78]= 5.
EDIT In response to Timo's comment, it might be possible to use NotebookEvaluate (new in 8) to achieve the requested effect.
SetAttributes[BlockRules, HoldRest]
BlockRules[rules_, expr_] :=
Block ## Append[Apply[Set, Hold#rules, {2}], Unevaluated[expr]]
nb = CreateDocument[{ExpressionCell[
Defer[Plot[Sin[a x], {x, 0, 2 Pi}]], "Input"],
ExpressionCell[Defer[Integrate[Sin[a x^2], {x, 0, 2 Pi}]],
"Input"]}];
BlockRules[{a -> 4}, NotebookEvaluate[nb, InsertResults -> "True"];]
As the result of this evaluation you get a notebook with your commands evaluated when a was locally set to 4. In order to take it further, you would have to take the notebook
with your code, open a new notebook, evaluate Notebooks[] to identify the notebook of interest and then do :
BlockRules[variablerules,
NotebookEvaluate[NotebookPut[NotebookGet[nbobj]],
InsertResults -> "True"]]
I hope you can make this idea work.
Is it possible to use rand() or any other pseudo-random generator to pick out random numbers, but have it be more likely that it will pick certain numbers that the user feeds it? In other words, is there a way, with rand() or something else, to pick a pseudo random number, but be able to adjust the odds of getting certain outcomes, and how do you do that if it is possible.
BTW, I'm just asking how to change the numbers that rand() outputs, not how to get the user input.
Well, your question is a bit vague... but if you wanted to pick a number from 0-100 but with a bias for (say) 43 and 27, you could pick a number in the range [0, 102] and map 101 to 43 and 102 to 27. It will really depend on how much bias you want to put in, what your range is etc.
You want a mapping function between uniform density of rand() and the probability density that you desire. The mapping function can be done lots of different ways.
You can certainly use any random number generator to skew the results. Example in C#, since I don't know objective-c syntax. I assume that rand() return a number tween 0 and 1, 0 inclusive and 1 exclusive. It should be quite easy to understand the idear and convert the code to any other language.
/// <summary>
/// Dice roll with a double chance of rolling a 6.
/// </summary>
int SkewedDiceRoll()
{
// Set diceRool to a value from 1 to 7.
int diceRool = Math.Floor(7 * rand()) + 1;
// Treat a value of 7 as a 6.
if (diceRoll == 7)
{
diceRoll = 6;
}
return diceRoll;
}
This is not too difficult..
simply create an array of all possible numbers, then pad the array with extra numbers of which you want to result more often.
ie:
array('1',1','1','1','2','3','4','4');
Obviously when you query that array, it will call "1" the most, followed by "4"
In other words, is there a way, with rand() or something else, to pick a pseudo random number, but be able to adjust the odds of getting certain outcomes, and how do you do that if it is possible.
For simplicity sake, let's use the drand48() which returns "values uniformly distributed over the interval [0.0,1.0)".
To make the values close to one more likely to appear, apply skew function log2():
log2( drand48() + 1.0 ); // +1 since log2() in is [0.0, 1.0) for values in [1.0, 2.0)
To make the values close to zero more likely to appear, use the e.g. exp():
(exp(drand48()) - 1.0) * (1/(M_E-1.0)); // exp(0)=1, exp(1)=e
Generally you need to crate a function which would map the uniformly distributed values from the random function into values which are distributed differently, non-uniformly.
You can use the follwing trick
This example has a 50 percent chance of producing one of your 'favourite' numbers
int[] highlyProbable = new int[]{...};
public int biasedRand() {
double rand = rand();
if (rand < 0.5) {
return highlyProbable[(int)(highlyProbable.length * rand())];
} else {
return (int)YOUR_RANGE * rand();
}
}
In addition to what Kevin suggested, you could have your regular group of numbers (the wide range) chopped into a number of smaller ranges, and have the RNG pick from the ranges you find favorable. You could access these ranges in a particular order, or, you can access them in some random order (but I can assume this wouldn't be what you want.) Since you're using manually specified ranges to be accessed within the wide range of elements, you're likely to see the numbers you want pop up more than others. Of course, this is just how I'd approach it, and it may not seem all that rational.
Good luck.
By definition the output of a random number generator is random, which means that each number is equally likely to occur next (1/10 chance) and you should not be able to affect the outcome.
Of-course, a pseudo-random generator creates an output that will always follow the same pattern for a given input seed. So if you know the seed, then you may have some idea of the output sequence. You can, of-course, use the modulus operator to play around with the set of numbers being output from the generator (eg. %5 + 2 to generate numbers from 2 to 7).
I understand how equivalence testing works.
How is it the same or different from boundary value testing?
Equivalence Class Testing
EC Testing is when you have a number of test items (e.g. values) that you want to test but because of cost (time/money) you do not have time to test them all. Therefore you group the test item into class where all items in each class are suppose to behave exactly the same. The theory is that you only need to test one of each item to make sure the system works.
Example 1
Children under 2 ride the bus for free. Young people pay $10, Adults $15 and Senior Citizen pay $5.
Classes:
Price:0 -> Age:0-1
Price:10 -> Age:2-14
Price:15 -> Age:15-64
Price:5 -> Age:65-infinity
Example 2 (more than one parameter)
Cellphones K80, J64 and J54 run Java 5. K90 and J99 run Java 6. But there are two possible browsers FireFox and Opera, J models run FF and K models run O.
Classes:
Browser:FF, Java:5 -> Phones:J64,J54
Browser:FF, Java:6 -> Phones:J99
Browser:O, Java:5 -> Phones:K80
Browser:O, Java:6 -> Phones:K90
Dangers Of Equivalence Class Testing
There is a danger of using EC Testing that is rarely mentioned in the testing books but is very important to remember.
Just because two items/values are suppose to be in the same class and behave the same, does not mean they DO behave the same.
That means that just because you test one value in the class that ALL values in the class behave the same. Real world example of mine is with cell phones that all had a certain Java Platform. They were suppose to all work the same but they didn't in reality. So testing just one value in a class is good, but not good enough. EC Testing is a good tool, but it's not fool proof and be careful with it. If test cases are cheap and fast (like automation), test more, or why not test them all!
Boundary Value Testing
BV Testing is when you decide to test the values on the edge of each Class you have identified. The theory is that most defects is around the edges of a class.
Example
Classes:
Price:0 -> Age:0-1 ( Boundary values 0, 1)
Price:10 -> Age:2-14 ( Boundary values 2, 14)
Price:15 -> Age:15-64 ( Boundary values 15, 64)
Price:5 -> Age:65-infinity ( Boundary values 65)
Critique of Boundary Value Testing
1) I, and other test professionals I have taken courses from, are not convinced that most defects are hidden around the edges of each class. And I have never seen any studies that proves this to be the case.
2) The fact that you need to use BV Testing proves that EC Testing is flawed since you test more than one value of each class.
3) It's easy to use when using values like integers. But what is a boundary value of class of phones models or browsers versions?
Hidden Boundary Value Testing
The boundary values of a class is often based on the specification of how the system should work. This is all good and well but most systems contain boundaries that are not explained in any spec and you will need to look for yourself. E.g. 'How many characters can I put into the test field before the system fails and breaks.','How big can the data file become before it so slow to read it gets annoying'.
Real world examples
- Pasting one million characters into a text area in FireFox 3.5 on win 7 crashes it
- ReCaptcha has a limit of 16003 characters, does your system handle the 413 that it passes back to it if somebody puts 16004+ characters in field. Or does it break
Summary
EC Testing and BV Testing are great tools and you should use them but they are not perfect and don't expect to find all defects using them. Use your know-how about the system and your intelligence and intuition to try more items and looks for other ways it could fail. And look for the hidden boundaries!
Boundary value analysis simply means to select values near the boundaries of the classes. So you are still dividing the input domain according to the classes then instead of selecting values from the middle of the class use values from the boundaries.
For example, if the input condition is a range from 20 to 70 then you have three classes of input
less than 20
between 20 and 70
more than 70
then for boundary value analysis select input = 19, 20, 21, 69, 70, 71. This type of analysis picks up errors on the boundaries.
The Equivalence testing needs to be supplemented with the Boundary value testing.
For example for equivalent testing of a function that takes values between 1 and 12
(say months of a year) the partitions would be:
values less than 1 (0,-1,-2), invalid partition
values between 1-12, valid partition
values greater than 12,invalid partition
For equivalence testing it is enough to pick one value as test input from each of these partition classes. That would mean tests with value of -2,6, and 15 would be considered enough to test behavior of the function. But these values doesn't catch Off-by-one error which can occur quite often.
With the boundary value testing the test inputs would be : -1,0,1,11,12,13 (at the boundaries), which would catch off-by-one errors.
I see both these testing methods to be a complement of each other.
Boundary value analysis is part or subset of equivalence partitioning. In boundary values analysis, instead of some random value, only values in the boundary are selected.
Boundary value analysis overcome the drawback of the Equivalance class partitioning. If a fix length is g9iven for eg. Mobile number (10 digit.)
The lower boundary in this case is - digit - 1 (ie. 10 - 1 =9)
The upper boundary in this case is - digit +1 (ie 10 + 1 =11)
Now we can perform test for the 9 and 11
Dynamic Testing Types –
Specification Based testing
A. Equivalance Partitioning
A1. Boundary value analysis
A2. Decision Tables
A3. Use case Testing
A4. State Transition testing
Structural Based testing
A. Test Coverage
B. Code coverage
C. Statement coverage
D. Decision coverage
Experience Based Testing
A. Error testing
B. Exploratory Testing
Equivalance Partitioning – It is a technique where tester divide the test conditions into groups and sets. System should handle them equivalently hence called equivalence classes. To test one condition from each partition will work to assume all condition will work in that partition.
EX >> Check addition of the single digit values. i.e. values between 0 and 9.
Values less than -9, i.e. -10,-11, …. (Invalid partition)
Values less than 0, i.e. -1,-2, …. till -9 (Valid Partition)
Values between 0-9 i.e. 0,1 …. till 9 (Valid Partition)
Values greater than 9, i.e. 10,11 (Invalid partition)
Testing the addition for any two values of each partition is enough.
Boundary value testing – It is based on testing at the boundaries between partition.
EX >> Consider below combination.
Addition of -9 and -10,
Addition of -10 and -11
Addition of 0 and -1
Addition of 0 and 1
Addition of 9 and 10
Addition of 11 and 10