I'd like to reuse the same code of a test case for several handcrafted combinations of input data and expected outcomes, but without copypasting the code for each set. Frameworks in other languages support it in different ways, for example in Groovy/Spock:
def "maximum of two numbers"(int a, int b, int c) {
expect:
Math.max(a, b) == c
where:
a | b | c
1 | 3 | 3
7 | 4 | 4
0 | 0 | 0
}
What's the preferred way to do this in ExUnit?
Maybe ExUnit is not the best framework to this?
I guess you could do something very simple like:
test "Math.max/2" do
data = [
{1, 3, 3},
{7, 4, 4},
]
for {a, b, c} <- data do
assert Math.max(b, c) == a
end
end
Pattern matching allows you to be pretty explicit when doing these thing in my opinion. And you can keep the simplicity of just variables and assertions, all inside ExUnit.
You can also wrap test definitions with an enumerable. Dave Thomas's Blog provides an extended example.
data = [
{1, 3, 3},
{7, 4, 4},
]
for {a,b,c} <- data do
#a a
#b b
#c c
test "func "<>#a<>", "<>#b<>" equals"<>#c do
assert fun(#a,#b) == #c
end
end
For a relatively simple test like this, I'm not sure this approach is any better, but for more complex tests this can provide better reporting and you can dynamically create tests at test time by using a function as in
the blog entry.
Related
Is it possible to continue the execution of test step even if one of the assert/match fails?
Ex:
Scenario: Testing
* def detail = {"a":{"data":[{"message":["push","dash"]},{"message":["data","Test"]}]}}
* match detail contains {"a":{"data":[{"message":["push","dash"]}]}}
* print detail
Here match will fail but execution stop at that point.
is there a way to do a soft assertion so that next step gets executed?
EDIT in 2021 - a PR introducing a continueOnStepFailure flag was contributed by Joel Pramos here and is available in Karate 1.0 onwards. You can find more details here: https://stackoverflow.com/a/66733353/143475
If you use a Scenario Outline each "row" is executed even if one fails.
Scenario Outline: Testing
* def detail = { a: 1, b: 2, c: 3 }
* match detail contains <expected>
Examples:
| expected |
| { a: 1 } |
| { b: 2 } |
| { c: 3 } |
Note that the concept of "soft assertions" is controversial and some consider it a bad practice:
a) https://softwareengineering.stackexchange.com/q/7823
b) https://martinfowler.com/articles/nonDeterminism.html
For those looking for a way to show all mis-matches between 2 JSON objects, see this: https://stackoverflow.com/a/61349887/143475
And finally, since some people want to do "conditional" match logic in JS, see this answer also: https://stackoverflow.com/a/50350442/143475
How do to use operators so that i always get the previous and the current value? If possible i want to avoid creating state outside the pipe.
- time ->
1 2 3 4
| | | |
Operations
| | |
(1,2) (2,3) (3,4)
Note that every value besides the first and the last one have to appear twice, so a simple buffer won't do.
I thought about combining skip with merge and buffer but merge does not seem to guarantee ordering.
val s = PublishSubject.create<Int>()
s.mergeWith(s.skip(1)).buffer(2).subscribe{i -> print(i)}
s.onNext(1)
s.onNext(2)
s.onNext(3)
s.onNext(4)
output:
[1, 2][2, 3][3, 4]
val o = Observable.just(1,2,3,4)
o.mergeWith(o.skip(1)).buffer(2).subscribe{i -> print(i)}
output:
[1, 2][3, 4][2, 3][4]
(the sole 4 is fine, and expected)
Looks like you still can use buffer:
Observable.just(1, 2, 3, 4)
.buffer(2, 1)
.subscribe { println(it) }
// prints
// [1, 2]
// [2, 3]
// [3, 4]
// [4]
Can PICT (=Pairwise Independent Combinatorial Testing) handle/model independent parameters.
For example in following input a and b are independent, so they should not be combined.
Input in PICT:
a: 1, 2, 3, 4
b: 5, 6, 7, 8
//some line which models the independence: a independent of b
Output, that I would expect:
a b
1 5
2 6
3 7
4 8
This example, with only 2 parameters, of course normally would not make much sense, but it's illustrative.
The same could be applied to 3 parameters (a,b,c), where a is independent of b, but not c.
The main goal of declaring parameters as independent would be the reduce the number of tests.
I read the paper/user guide to PICT, but I didn't found any useful information.
I will answer my question by myself:
The solution is to define submodels and set the default order from 2 (=pairwise) to 1 (= no combination).
For example parameter a = {a_1, a_2, a_3} should be independent of
b = {b_1, b_2, b_3} and
c = {c_1, ..., c_4}.
Therefor I would expect 12 tests ((b x c) + a).
Resulting in the following input file:
a: 1,2,3
b: 1,2,3
c: 1,2,3,4
{b,c}#2
{b,c}#2 defines a submodel, consisting of b and c, which uses pairwise combination.
And running pict with the option: "/o:1".
In PICT you can define conditions and achieve your goal to not combine them with the rest. You can add additional option as N/A and have something like this;
If [A] = '1' Then [B] = 'N/A';
This is one possible option to handle this case.
I am trying to verify two different outputs in the context of a single Spock method that runs multiple test cases of the form when-then-where. For this reason I use two assertions at the then block, as can be seen in the following example:
import spock.lang.*
#Unroll
class ExampleSpec extends Specification {
def "Authentication test with empty credentials"() {
when:
def reportedErrorMessage, reportedErrorCode
(reportedErrorMessage, reportedErrorCode) = userAuthentication(name, password)
then:
reportedErrorMessage == expectedErrorMessage
reportedErrorCode == expectedErrorCode
where:
name | password || expectedErrorMessage | expectedErrorCode
' ' | null || 'Empty credentials!' | 10003
' ' | ' ' || 'Empty credentials!' | 10003
}
}
The code is an example where the design requirement is that if name and password are ' ' or null, then I should always expect exactly the same expectedErrorMessage = 'Empty credentials!' and expectedErrorCode = 10003. If for some reason (presumably because of bugs in the source code) I get expectedErrorMessage = Empty! (or anything else other than 'Empty credentials!') and expectedErrorCode = 10001 (or anything else other than 1003), this would not satisfy the above requirement.
The problem is that if both assertions fail in the same test, I get a failing message only for the first assertion (here for reportedErrorMessage). Is it possible to get informed for all failed assertions in the same test?
Here is a piece of code that demonstrates the same problem without other external code dependencies. I understand that in this particular case it is not a good practice to bundle two very different tests together, but I think it still demonstrates the problem.
import spock.lang.*
#Unroll
class ExampleSpec extends Specification {
def "minimum of #a and #b is #c and maximum of #a and #b is #d"() {
expect:
Math.min(a, b) == c
Math.max(a, b) == d
where:
a | b || c | d
3 | 7 || 3 | 7
5 | 4 || 5 | 4 // <--- both c and d fail here
9 | 9 || 9 | 9
}
}
Based on the latest comment by OP, it looks like a solution different from my previous answer would be helpful. I'm leaving the previous answer in-place, as I feel it still provides useful information related to the question (specifically separating positive and negative tests).
Given that you want to see all failures, and not just have it fail at the first assert that fails, I would suggest concatenating everything together into a boolean AND operation. Not using the && shortcut operator, because it will only run until the first check that does not satisfy the entire operation. I would suggest using the &, so that all checks are made, regardless of any previously failing checks.
Given the max and min example above, I would change the expect block to this:
Math.min(a, b) == c & Math.max(a, b) == d
When the failure occurs, it gives you the following information:
Math.min(a, b) == c & Math.max(a, b) == d
| | | | | | | | | | |
4 5 4 | 5 false 5 5 4 | 4
false false
This shows you every portion of the failing assert. By contrast, if you used the &&, it would only show you the first failure, which would look like this:
Math.min(a, b) == c && Math.max(a, b) == d
| | | | | |
4 5 4 | 5 false
false
This could obviously get messy pretty fast if you have more than two checks on a single line - but that is a tradeoff you can make between all failing information on one line, versus having to re-run the test after fixing each individual component.
Hope this helps!
I think there are two different things at play here.
Having a failing assert in your code will throw an error, which will cease execution of the code. This is why you can't have two failing assertions in a single test. Any line of code in an expect or then block in Spock has an implicit assert before it.
You are mixing positive and negative unit tests in the same test. I ran into this before myself, and I read/watched something about this and Spock (I believe from the creator, Peter Niederwieser), and learned that these should be separated into different tests. Unfortunately I couldn't find that reference. So basically, you'll need one test for failing use cases, and one test for passing/successful use cases.
Given that information, here is your second example code, with the tests separated out, with the failing scenario in the second test.
#Unroll
class ExampleSpec extends Specification {
def "minimum of #a and #b is #c and maximum of #a and #b is #d - successes"() {
expect:
Math.min(a, b) == c
Math.max(a, b) == d
where:
a | b || c | d
3 | 7 || 3 | 7
9 | 9 || 9 | 9
}
def "minimum of #a and #b is #c and maximum of #a and #b is #d - failures"() {
expect:
Math.min(a, b) != c
Math.max(a, b) != d
where:
a | b || c | d
5 | 4 || 5 | 4
}
}
As far as your comment about the MongoDB test case - I'm not sure what the intent is there, but I'm guessing they are making several assertions that are all passing, rather than validating that something is failing.
I am new to R and am trying to find a better solution for accomplishing this fairly simple task efficiently.
I have a data.frame M with 100,000 lines (and many columns, out of which 2 columns are relevant to this problem, I'll call it M1, M2). I have another data.frame where column V1 with about 10,000 elements is essential to this task. My task is this:
For each of the element in V1, find where does it occur in M2 and pull out the corresponding M1. I am able to do this using for-loop and it is terribly slow! I am used to Matlab and Perl and this is taking for EVER in R! Surely there's a better way. I would appreciate any valuable suggestions in accomplishing this task...
for (x in c(1:length(V$V1)) {
start[x] = M$M1[M$M2 == V$V1[x]]
}
There is only 1 element that will match, and so I can use the logical statement to directly get the element in start vector. How can I vectorize this?
Thank you!
Here is another solution using the same example by #aix.
M[match(V$V1, M$M2),]
To benchmark performance, we can use the R package rbenchmark.
library(rbenchmark)
f_ramnath = function() M[match(V$V1, M$M2),]
f_aix = function() merge(V, M, by.x='V1', by.y='M2', sort=F)
f_chase = function() M[M$M2 %in% V$V1,] # modified to return full data frame
benchmark(f_ramnath(), f_aix(), f_chase(), replications = 10000)
test replications elapsed relative
2 f_aix() 10000 12.907 7.068456
3 f_chase() 10000 2.010 1.100767
1 f_ramnath() 10000 1.826 1.000000
Another option is to use the %in% operator:
> set.seed(1)
> M <- data.frame(M1 = sample(1:20, 15, FALSE), M2 = sample(1:20, 15, FALSE))
> V <- data.frame(V1 = sample(1:20, 10, FALSE))
> M$M1[M$M2 %in% V$V1]
[1] 6 8 11 9 19 1 3 5
Sounds like you're looking for merge:
> M <- data.frame(M1=c(1,2,3,4,10,3,15), M2=c(15,6,7,8,-1,12,5))
> V <- data.frame(V1=c(-1,12,5,7))
> merge(V, M, by.x='V1', by.y='M2', sort=F)
V1 M1
1 -1 10
2 12 3
3 5 15
4 7 3
If V$V1 might contain values not present in M$M2, you may want to specify all.x=T. This will fill in the missing values with NAs instead of omitting them from the result.