When I have a mocked method whose calling expectations fail, the test is still passing. At first glance I thought there would be some kind of flag to control this behavior but I could not find one.
In case more context is required, this is happening to me with any kind of expectation, but as an example I have the following one, using a custom Matcher.
// Expectation:
Foo expectedFoo;
Foo foo;
EXPECT_CALL(foo, GenerateEvent(1, FooEq(&expectedFoo)));
//Matcher:
MATCHER_P(FooEq, expected, "")
{
return false;
}
if foo.GenerateEvent(1, new Foo) is called, the expectaction fails as expected, but the test still passes as shown below (had to hack the names due to privacy issues, hope I didn't mess the final result).
[ RUN ] Test.FooTest
unknown file: error:
Unexpected mock function call - returning directly.
Function call: GenerateEvent(1, 0D386FDC)
Google Mock tried the following 1 expectation, but it didn't match:
Foo.cpp(307): EXPECT_CALL(foo, GenerateEvent(1, FooEq(&expectedFoo)))...
Expected arg #1: foo eq 012FF238
Actual: 0D386FDC
Expected: to be called once
Actual: never called - unsatisfied and active
[ OK ] Test.FooTest (15 ms)
StrictMock< Foo > foo;
Should cause the test to fail by throwing an exception when the unexpected call happens. See strict-mock
Related
In my code i catch the error that might occure and give them a reporter.log(); which contain the problem that happened since i dont want my code to crash after one probleme found the probleme with this is that i always get Passed as a result even though my Report Log contains a failed step.
What i want is in case that reporter.log(); gonna be written i want to change the status of the methode into failed.
The report that i write :
String FormatMessageError = "<font color='red'></br><img src='"+ Bad +"'/>-";
Reporter.log(FormatMessageError +" Wasnt abel to connect</font>");
What i tried :
I used this :
Reporter.getCurrentTestResult().setStatus(ITestResult.FAILURE);
but the status stay as passed After that i tried this :
SoftAssert SAAjoutPanier = new SoftAssert();
SAAjoutPanier.assertEquals(ValidationOfPurchase,false,FormatMessageError +" Wasnt abel to connect</font>");
with : SAAjoutPanier.assertAll(); in the end of my #test as you can imagine it didn't work
Then i tried this :
Throwable throwable = new Throwable("One or more tests in this group has failed");
throwable.setStackTrace(new StackTraceElement[0]);
Reporter.getCurrentTestResult().setThrowable(throwable);
But it gives just a message but the status it self is still passed.
Can some one help me with this probleme ty !
MSTest has the same limitation. The solutuon I used, just throw a real error at the end.
throw new Exception("some test steps failed");
If that works, keep a list somewhere with the real errors and inserts in the error you throw.
When I separately run the runAsyncWithMock test, it waits for 3 seconds until the mock's execution is finalised, rather than get terminated like the other 2 tests.
I was not able to figure out why.
It is interesting that:
When multiple Runnables are executed by CompletableFuture.runAsync in a row in the runAsyncWithMock test, only the first one waits, the others not.
When having multiple duplicated runAsyncWithMock tests, each and every of them runs for 3s when the whole specification is executed.
When using Class instance rather than a Mock, the test is finalised immediately.
Any idea what I got wrong?
My configuration:
macOS Mojave 10.14.6
Spock 1.3-groovy-2.4
Groovy 2.4.15
JDK 1.8.0_201
The repo containing the whole Gradle project for reproduction:
https://github.com/lobodpav/CompletableFutureMisbehavingTestInSpock
The problematic test's code:
#Stepwise
class SpockCompletableFutureTest extends Specification {
def runnable = Stub(Runnable) {
run() >> {
println "${Date.newInstance()} BEGIN1 in thread ${Thread.currentThread()}"
sleep(3000)
println "${Date.newInstance()} END1 in thread ${Thread.currentThread()}"
}
}
def "runAsyncWithMock"() {
when:
CompletableFuture.runAsync(runnable)
then:
true
}
def "runAsyncWithMockAndClosure"() {
when:
CompletableFuture.runAsync({ runnable.run() })
then:
true
}
def "runAsyncWithClass"() {
when:
CompletableFuture.runAsync(new Runnable() {
void run() {
println "${Date.newInstance()} BEGIN2 in thread ${Thread.currentThread()}"
sleep(3000)
println "${Date.newInstance()} END2 in thread ${Thread.currentThread()}"
}
})
then:
true
}
}
This is caused by the synchronized methods in https://github.com/spockframework/spock/blob/master/spock-core/src/main/java/org/spockframework/mock/runtime/MockController.java when a mock is executed it delegates through the handle method. The Specification also uses the synchronized methods, in this case probably leaveScope, and is thus blocked by the sleeping Stub method.
Since this is a thread interleaving problem I guess that additional closure in runAsyncWithMockAndClosure moves the execution of the stub method behind the leaveScope and thus changes the ordering/blocking.
Oh, just now after writing my last comment I saw a difference:
You use #Stepwise (I didn't when I tried at first), an annotation I almost never use because it creates dependencies between feature methods (bad, bad testing practice). While I cannot say why this has the effect described by you only when running the first method, I can tell you that removing the annotation fixes it.
P.S.: With #Stepwise you cannot even execute the second or third method separately because the runner will always run the preceding one(s) first, because - well, the specification is said to be executed step-wise. ;-)
Update: I could briefly reproduce the problem with #Stepwise, but after recompilation now it does not happen anymore, neither with or without that annotation.
I have to create a per-customer backpressure, and implemented this with returning a Mono.justOrEmpty(authentication).delayElement(Duration.ofMillis(calculatedDelay));
This seems to be working fine in my unit-tests, BUT if I have a calculatedDelay of 0, I can't test this. This snippet returns a java.lang.AssertionError: unexpected end during a no-event expectation:
#Test
public void testSomeDelay_8CPUs_Load7() throws IOException {
when(cut.getLoad()).thenReturn("7");
when(cut.getCpuCount()).thenReturn(8L);
when(authentication.getName()).thenReturn("delayeduser");
Duration duration = StepVerifier
.withVirtualTime(() -> cut.get(Optional.of(authentication)))
.expectSubscription() // swallow subscribe event
.expectNoEvent(Duration.ofMillis(0)) // here is the culprit
.expectNextCount(1)
.verifyComplete();
}
I don't know how to check for the cases, where I'm expected no delay at all. This is BTW the same, when I return a Mono.justOrEmpty(authentication) (without any delayed subscription). I can't seem to check, that I have created the correct non-delayed flow.
Yeah this is a corner case that is hard to cover. Especially when you know that foo.delayElement(0) is actually simply returning foo unmodified.
What you could do is test the delay differently, by appending an elapsed() operator:
Duration duration = StepVerifier
.withVirtualTime(() -> cut.get(Optional.of(authentication))
.elapsed() //this will get correct timing of 0 with virtual time
.map(Tuple2::getT1) //we only care about the timing, T2 being the value
)
.expectNext(calculateDelay)
.verifyComplete();
Consider i have a function like newConvert. I am sopposed to recieve an error for newCovertor("IL") . To generate this erorr: I used failwith "Invalid Input"
The erorr is :
System.Exception: Invalid Input
> at FSI_0160.inputChecker(FSharpList`1 numberList) in C:\Users\Salman\Desktop\coursework6input\itt8060-master\coursework6input\BrokenRomanNumbers\Library1.fs:line 140
at FSI_0160.newCovertor(String romanNumber) in C:\Users\Salman\Desktop\coursework6input\itt8060-master\coursework6input\BrokenRomanNumbers\Library1.fs:line 147
at <StartupCode$FSI_0165>.$FSI_0165.main#() in C:\Users\Salman\Desktop\coursework6input\itt8060-master\coursework6input\BrokenRomanNumbers\newtest.fs:line 32
Stopped due to error
I used FsUnit and Nunit, they are loaded and installed and working rightly.
Then I made a tes for it using
[<TestFixture>]
type ``Given a Roman number15 ``()=
[<Test>]
member this.
``Whether the right convert for this number must be exist``()=
newCovertor("IL") |> should equal System.Exception
I cannot understand!! The function fails rightly, but the test does not accept it, so why??????
newCovertor does not produce a System.Execption - it throws it - so you never get to the should equal ... part
to catch the exception with FsUnit you have to wrap it as an action:
(fun () -> newCovertor("IL") |> ignore) |> should throw typeof<System.Exception>
also see the really good docs - there are quite a few ways to achieve this
The reason your version is not working is that NUnit will normaly mark a test as failed exactly when a exception is thrown inside the test-methods body - which you do.
If you wrap it in an action than the should throw can choose to execute this delayed action inside try ... match block to catch the expected exception and filter it out (or in this case throw an exception if there where none)
btw: just like in C#/VB.net you can also use the ExpectedExceptionAttribute from NUnit on the method level if you like - this way the test-framework will handle it for you.
I forked Codeception/Codeception and added a Command module check that throws an exception.
When testing, this exception causes the test to fail but I'm running a test to force the exception and failure, to make sure it catches it.
Here's the edit to src/Codeception/Command/GenerateSuite.php
if ($this->containsInvalidCharacters ($suite)) {
throw new \Exception("Suite name '{$suite}' contains invalid characters. ([A-Za-z0-9_]).");
}
When updating tests/cli/GenerateSuiteCept.php when I run the command $I->executeCommand('generate:suite invalid-suite'); it fails because I'm throwing the exception in src/Codeception/Command/GenerateSuite.php in the first place.
Here is the addition to tests/cli/GenerateSuiteCept.php:
$I->executeCommand('generate:suite invalid-dash-suite');
When I run it, it says it failed. However, I want it to pass because I'm deliberately adding the dash to verify it catches invalid characters. What is the best way to do that? I'm wary of a try/catch block in these tests. I'm not sure if that is the best way to do that.
After looking at the other Codeception tests, I found this to be a better way than throwing an exception:
if ($this->containsInvalidCharacters ($suite)) {
$output->writeln("<error>Suite name '{$suite}' contains invalid characters. ([A-Za-z0-9_]).</error>");
return;
}
So I modified the test to:
$I->expect ('suite is not created due to dashes');
$I->executeCommand('generate:suite invalid-dash-suite');
$I->seeInShellOutput('contains invalid characters');