I want the test to report all assertions and verifications. So both the mockk verification AND the the assertion library (in this case, KotlinTest) assertions should run and not shortcircuit.
In other words I don't want the test to stop ...
verify(exactly = 1) { mock.methodcall(any()) } // ... here
success shouldBe true // how can I check this line too
nor ...
success shouldBe true // ... here
verify(exactly = 1) { mock.methodcall(any()) } // how can I check this line too
How to do this? I am open to use just one tool if I can do both with it.
As per your comment, you said you are using KotlinTest.
In KotlinTest, I believe you can use assertSoftly for the behavior you want:
Normally, assertions like shouldBe throw an exception when they fail. But sometimes you want to perform multiple assertions in a test, and would like to see all of the assertions that failed. KotlinTest provides the assertSoftly function for this purpose.
assertSoftly {
foo shouldBe bar
foo should contain(baz)
}
If any assertions inside the block failed, the test will continue to run. All failures will be reported in a single exception at the end of the block.
And then, we can convert your test to use assertSoftly:
assertSoftly {
success shouldBe true
shouldNotThrowAny {
verify(exactly = 1) { mock.methodcall(any()) }
}
}
It's necessary to wrap verify in shouldNotThrowAny to make assertSoftly aware of it when it throws an exception
Related
Let us suppose we have a class member whose purpose is
to bring 2 objects (let's say object1 and object2) from two different places and then create the final
result merging these two object in another one, which is finally returned.
Suppose then the operation of retrieving object1 and object2 can be done concurrently,
so this leads to a typical use case of kotlin coroutines.
What has been described so far is shown in the following example:
fun parallelCall(): MergedObject {
return runBlocking(context = Dispatchers.Default) {
try {
val object1 : Deferred<Object1> = async {
bringObject1FromSomewhere()
}
val object2 : Deferred<Object2> = async {
bringObject2FromSomewhere()
}
creteFinalObject(object1.await(), object2.await())
} catch (ex: Exception) {
throw ex
}
}
}
The surrounding try block should intercept any kind of exception thrown while
object1 and object2 are retrieved, as well as in the createFinalObject method.
This latter simply merges together the awaited results from previous calls,
waiting for both of them to be accomplished.
Note that the awaiting of the deferred object1 and object2 happens almost at the same time,
since they are both awaited when passed as arguments to the createFinalObject method.
In this scenario I can perform a test using mockk as mocking library such that whenever bringObject1FromSomewhere()
throws an exception, then the creteFinalObject method is NEVER called. Namely, something like:
#Test
fun `GIVEN bringObject1FromSomewhere throws exception WHEN parallelCall executes THEN creteFinalObject is never executed`() {
every { bringObject1FromSomewhere() } throws NullPointerException()
every { bringObject2FromSomewhere() } returns sampleObject2
assertThrows<NullPointerException> { parallelCall() }
verify(atMost = 1) { bringObject1FromSomewhere() }
verify(atMost = 1) { bringObject2FromSomewhere() }
//should never be called since bringObject1FromSomewhere() throws nullPointer exception
verify(exactly = 0) { creteFinalObject(any(), any()) }
}
The problem is that the test above works almost always, but, there are some cases in which it randomly fails,
calling the createFinalObject method regardless of the mocked values.
Is this issue related to the slight difference in time in which the deferred object1 and object2
are awaited when creteFinalObject(object1.await(), object2.await()) is called?
Another thing which comes to my mind could be the way in which I am expecting argument in the last line of the test:
verify(exactly = 0) { creteFinalObject(any(), any()) }
does mockk could have any problem when any() is used?.
Further, can potentially be an issue the fact that the try { } block is not able to detect the exception
before the createFinalObject method is called? I would never doubt about this in a non-parallel environment but probably
the usage of runBlocking as coroutineScope changes the rule of the game?
Any hints will be helpful, thanks!
Kotlin version:1.6.0 Corutines version: 1.5.2 mockk version: 1.12.2
Are you sure it fails because it attempts to call the creteFinalObject function? Because when reading your code, I think that should be impossible (of course, never say never :D). The creteFinalObject function can only be called if both object1.await() and object2.await() return successfully.
I think something else is going on. Because you're doing 2 separate async tasks (getting object 1 and getting object 2), I suspect that the ordering of these 2 tasks would result in either a success or a failure.
Running your code locally, I notice that it sometimes fails at this line:
verify(atMost = 1) { bringObject2FromSomewhere() }
And I think there is your error. If bringObject1FromSomewhere() is called before bringObject2FromSomewhere(), the exception is thrown and the second function invocation never happens, causing the test to fail. The other way around (2 before 1) would make the test succeed. The Dispatchers.Default uses an internal work queue, where jobs that are cancelled before they are even started will never start at all. And the first task can fail fast enough for the second task to not being able to start at all.
I thought the fix would be to use verify(atLeast = 0, atMost = 1) { bringObject2FromSomewhere() } instead, but as I see on the MockK GitHub issues page, this is not supported (yet): https://github.com/mockk/mockk/issues/806
So even though you specify that bringObject2FromSomewhere() should be called at most 1 time, it still tries to verify it is also called at least 1 time, which is not the case.
You can verify this by adding a delay to the async call to get the first object:
val object1 : Deferred<Object1> = async {
delay(100)
bringObject1FromSomewhere()
}
This way, the test always succeeds, because bringObject2FromSomewhere() always has enough time to be called.
So how to fix this? Either hope MockK fixes the functionality to specify verify(atLeast = 0, atMost = 1) { ... }, or disable the verification on this call for now.
I want to run cleanup code after a certain block of code completes, regardless of exceptions. This is not a closeable resource and I cannot use try-with-resources (or Kotlin's use).
In Java, I could do the following:
try {
// ... Run some code
} catch(Exception ex) {
// ... Handle exception
} finally {
// ... Cleanup code
}
Is the following Kotlin code equivalent?
runCatching {
// ... Run some code
}.also {
// ... Cleanup code
}.onFailure {
// ... Handle exception
}
Edit: added boilerplate exception handling - my concern is with ensuring the cleanup code runs, and maintainability.
There is one important difference, where the code inside runCatching contains an early return. A finally block will be executed even after a return, whereas also has no such magic.
This code, when run, will print nothing:
fun test1()
runCatching {
return
}.also {
println("test1")
}
}
This code, when run, will print "test2":
fun test2() {
try {
return
} finally {
println("test2")
}
}
There is one big difference between both code samples. try...finally propagates exceptions while runCatching().also() catches/consumes them. To make it similar you would have to throw the result at the end:
runCatching {
// ... Run some code
}.also {
// ... Cleanup code
}.getOrThrow()
But still, it is not really 1:1 equivalent. It catches all exceptions just to rethrow them. For this reason, it is probably less performant than simple try...finally.
Also, I think this is less clear for the reader. try...finally is a standard way of dealing with exceptions. By using runCatching() just to immediately rethrow, you actually confuse people reading this code later.
Your question sounded a little like you believed Kotlin does not have try...finally and you need to search for alternatives. If this is the case, then of course Kotlin has try...finally and I think you should use it instead of runCatching().
As per Kotlin's doc for runCatching:
Calls the specified function block and returns its encapsulated result if invocation was successful, catching any Throwable exception that was thrown from the block function execution and encapsulating it as a failure.
Even if finally always runs after a try block and also always runs after a runCatching, they do not serve the same purpose.
finally doesn't receive any argument and cannot operate on the values of the try block, while also receives the Result of the runCatching block.
TLDR; .runCatching{}.also{} is a more advanced try{}finally{}
There is also a difference in what is the result of evaluating the expression.
Consider the following code:
fun main() {
val foo = try {
throw Exception("try")
} catch(e: Exception) {
"catch"
} finally {
"finally"
}
val bar = runCatching{
throw Exception("runCatching")
}.also{
"also"
}.onFailure {
"onFailure"
}
println(foo)
println(bar)
}
The output will be:
catch
Failure(java.lang.Exception: runCatching)
https://pl.kotl.in/a0aByS5l1
EDIT:
An interesting article that points out some differences as well:
https://medium.com/#mattia23r/a-take-on-functional-error-handling-in-kotlin-515b67b4212b
Now let’s give a second look at the implementation of runCatching in the gist above. What does it do? It catches everything.
In this case, it goes even further: it catches all Throwables. For those not knowing, Throwable is everything that can go after a throw keyword; it has two descendants: Exceptions and Errors. We haven’t mentioned Errors so far; Errors usually represent something wrong that happened at a lower level than your business logic, something that can’t usually be recovered with a simple catch.
The following test succeeds with Process finished with exit code 0. Note, this test does print the exception to the logs, but does not fail the test (which is the behavior I want).
#Test
fun why_does_this_test_pass() {
val job = launch(Unconfined) {
throw IllegalStateException("why does this exception not fail the test?")
}
// because of `Unconfined` dispatcher, exception is thrown before test function completes
}
As expected, this test fails with Process finished with exit code 255
#Test
fun as_expected_this_test_fails() {
throw IllegalStateException("this exception fails the test")
}
Why do these tests not behave the same way?
Compare your test with the following one that does not use any coroutines, but starts a new thread instead:
#Test
fun why_does_this_test_pass() {
val job = thread { // <-- NOTE: Changed here
throw IllegalStateException("why does this exception not fail the test?")
}
// NOTE: No need for runBlocking any more
job.join() // ensures exception is thrown before test function completes
}
What happens here? Just like the test with launch, this test passes if you run it, but the exception gets printed on the console.
So, using launch to start a new coroutine is very much like using thread to start a new thread. If it fails, the error gets handled by uncaught exception handler in thread and by CoroutineExceptionHandler (see it in the docs) by launch. Exceptions in launch are not swallowed, but are handled by the coroutine exception handler.
If you want exception to propagate to the test, you shall replace launch with async and replace join with await in your code. See also this question: What is the difference between launch/join and async/await in Kotlin coroutines
UPDATE: Kotlin coroutines had recently introduced the concept of "Structured Concurrency" to avoid this kind of exception loss. The code in this question does not compile anymore. To compile it, you'd have to either explicitly say GlobalScope.launch (as in "I confirm that it Ok to loose my exceptions, here is my signature") or wrap the test into runBlocking { ... }, in which case exception is not lost.
I was able to create an exception throwing CoroutineContext for tests.
val coroutineContext = Unconfined + CoroutineExceptionHandler { _, throwable ->
throw throwable
}
Though this would probably not be suitable for production. Maybe need to catch cancellation exceptions or something, I'm not sure
A custom test rule so far seems to be the best solution.
/**
* Coroutines can throw exceptions that can go unnoticed by the JUnit Test Runner which will pass
* a test that should have failed. This rule will ensure the test fails, provided that you use the
* [CoroutineContext] provided by [dispatcher].
*/
class CoroutineExceptionRule : TestWatcher(), TestRule {
private val exceptions = Collections.synchronizedList(mutableListOf<Throwable>())
val dispatcher: CoroutineContext
get() = Unconfined + CoroutineExceptionHandler { _, throwable ->
// I want to hook into test lifecycle and fail test immediately here
exceptions.add(throwable)
// this throw will not always fail the test. this does print the stacktrace at least
throw throwable
}
override fun starting(description: Description) {
// exceptions from a previous test execution should not fail this test
exceptions.clear()
}
override fun finished(description: Description) {
// instead of waiting for test to finish to fail it
exceptions.forEach { throw AssertionError(it) }
}
}
I'm hoping to improve it via this post though
UPDATE: just use runBlocking - like Roman suggests.
I'm currently learning Kotlin, and one way I'm doing it is by automatically converting Java code to Kotlin and studying the results. One piece of Java code I tried to convert is the following static block in a class that tries to ensure that assertions are enabled:
static {
boolean assertsEnabled = false;
assert assertsEnabled = true;
if (!assertsEnabled)
throw new AssertionError("Please enable assertions!");
}
This relies on the assertsEnabled = true expression as an argument to assert. In Java, assignments are expressions. In Kotlin, they're not, and so this can't be converted. Is there any other way to do it?
Unfortunately, Kotlin doesn't have the assert keyword with its special semantics. Instead it has this function:
inline fun assert(value: Boolean, lazyMessage: () -> Any)
You can see that the expression passed as the first argument is evaluated unconditionally. This means you can't achieve the same lightweight check as in Java; you have to trigger an actual assertion failure to make sure.
So you need a check as suggested by #Zoe:
try {
assert(false)
throw IllegalStateException("Please enable assertions!")
} catch (e: AssertionError) {
// Things are looking good; carry on
}
If you insist on throwing an AssertionError instead of IllegalStateException, you can use a boolean variable for that.
var assertionsAreDisabled = false
try {
assert(false)
assertionsAreDisabled = true
} catch (e: AssertionError) {
// Things are looking good; carry on
}
if (assertionsAreDisabled) {
throw AssertionError("Please enable assertions!")
}
As per title, I'm trying to run a test case in a loop. To be able to calculate the number of failed assertions, I'm expecting that if AssertJ is trying to assert the returned value from a method call, it should softly fail a single iteration and carry on. Otherwise, it defies the purpose of soft assertions. Here's a snippet illustrating this:
public static void main(String[] args) {
SoftAssertions softAssertions = new SoftAssertions();
softAssertions.assertThat(throwException(10)).isTrue();
softAssertions.assertThat(throwException(10)).isTrue();
softAssertions.assertThat(throwException(1)).isTrue();
softAssertions.assertAll();
}
private static boolean throwException(int stuff){
if(stuff == 1){
throw new RuntimeException();
}
return true;
}
The output:
Exception in thread "main" java.lang.RuntimeException
at eLCMUpdate.throwException(MyClass.java:101)
at eLCMUpdate.main(MyClass.java:95)
I'm missing something here. Am I doing something wrong?
The problem in the code softAssertions.assertThat(throwException(10)).isTrue(); is that if the exception is thrown then assertThat is not executed at all.
What you need is to lazy evaluate the code you are passing in assertThat, you can do this with AssertJ assertThatCode as below:
final SoftAssertions softAssertions = new SoftAssertions();
softAssertions.assertThatCode(() -> throwException(10)).doesNotThrowAnyException();
softAssertions.assertThatCode(() -> throwException(1)).isInstanceOf(RuntimeException.class);
softAssertions.assertAll();
According to my understanding soft assertions work on boolean values and not on exceptions.
Also: if you throw an exception before calling softAssertions.assertAll(), obviously this method will also never be executed. This is actually the cause of the behaviour you reported.
Just try to debug through your code and you will see that the softAssertions.assertAll() is never called.
Soft assertions will work properly if you change your code to:
#Test
void soft_assertions() {
SoftAssertions softAssertions = new SoftAssertions();
softAssertions.assertThat(checkCondition(10)).isTrue();
softAssertions.assertThat(checkCondition(10)).isTrue();
softAssertions.assertThat(checkCondition(1)).isTrue();
softAssertions.assertThat(checkCondition(2)).isTrue();
softAssertions.assertThat(checkCondition(20)).isTrue();
softAssertions.assertAll();
}
private static boolean checkCondition(int stuff){
if(stuff == 1 || stuff == 2){
return false;
}
return true;
}
This will output the result of multiple assertions and not stop on the evaluation of the first failed assertion.
Output:
org.assertj.core.api.SoftAssertionError:
The following 2 assertions failed:
1)
Expecting:
<false>
to be equal to:
<true>
but was not.
at JsonStewardshipCustomerConversionTest.soft_assertions(JsonStewardshipCustomerConversionTest.java:301)
2)
Expecting:
<false>
to be equal to:
<true>
but was not.
at JsonStewardshipCustomerConversionTest.soft_assertions(JsonStewardshipCustomerConversionTest.java:302)
Update
SoftAssertion does not seem to fit your purpose.
I suggest you use instead JUnit 5 assertAll. According to my tests it evaluates all conditions in an assertAll block and survives exceptions too. The problem here is you need JUnit 5 which is probably not largely adopted yet.
Here is an example with a failure on a boolean condition and also an exception. Both are reported in the console.
#Test
void soft_assertions() {
assertAll("Check condition",
() -> assertThat(checkCondition(9)).isTrue(),
() -> assertThat(checkCondition(10)).isTrue(),
() -> assertThat(checkCondition(11)).isTrue(),
() -> assertThat(checkCondition(2)).isTrue(), // Throws exception
() -> assertThat(checkCondition(3)).isFalse(), // fails
() -> assertThrows(IllegalArgumentException.class, () -> {
checkCondition(1);
})
);
}
private static boolean checkCondition(int stuff) {
if (stuff == 1 || stuff == 2) {
throw new IllegalArgumentException();
}
return true;
}
You will see this in the output:
org.opentest4j.MultipleFailuresError: Check condition (2 failures)
<no message> in java.lang.IllegalArgumentException
Expecting:
<true>
to be equal to:
<false>
but was not.