Exceptions thrown while soft asserting fail the subsequent tests - testing

Question

As per title, I'm trying to run a test case in a loop. To be able to calculate the number of failed assertions, I'm expecting that if AssertJ is trying to assert the returned value from a method call, it should softly fail a single iteration and carry on. Otherwise, it defies the purpose of soft assertions. Here's a snippet illustrating this:
public static void main(String[] args) {
SoftAssertions softAssertions = new SoftAssertions();
softAssertions.assertThat(throwException(10)).isTrue();
softAssertions.assertThat(throwException(10)).isTrue();
softAssertions.assertThat(throwException(1)).isTrue();
softAssertions.assertAll();
}
private static boolean throwException(int stuff){
if(stuff == 1){
throw new RuntimeException();
}
return true;
}
The output:
Exception in thread "main" java.lang.RuntimeException
at eLCMUpdate.throwException(MyClass.java:101)
at eLCMUpdate.main(MyClass.java:95)
I'm missing something here. Am I doing something wrong?

Answer 1

The problem in the code softAssertions.assertThat(throwException(10)).isTrue(); is that if the exception is thrown then assertThat is not executed at all.
What you need is to lazy evaluate the code you are passing in assertThat, you can do this with AssertJ assertThatCode as below:
final SoftAssertions softAssertions = new SoftAssertions();
softAssertions.assertThatCode(() -> throwException(10)).doesNotThrowAnyException();
softAssertions.assertThatCode(() -> throwException(1)).isInstanceOf(RuntimeException.class);
softAssertions.assertAll();

Answer 2

According to my understanding soft assertions work on boolean values and not on exceptions.
Also: if you throw an exception before calling softAssertions.assertAll(), obviously this method will also never be executed. This is actually the cause of the behaviour you reported.
Just try to debug through your code and you will see that the softAssertions.assertAll() is never called.
Soft assertions will work properly if you change your code to:
#Test
void soft_assertions() {
SoftAssertions softAssertions = new SoftAssertions();
softAssertions.assertThat(checkCondition(10)).isTrue();
softAssertions.assertThat(checkCondition(10)).isTrue();
softAssertions.assertThat(checkCondition(1)).isTrue();
softAssertions.assertThat(checkCondition(2)).isTrue();
softAssertions.assertThat(checkCondition(20)).isTrue();
softAssertions.assertAll();
}
private static boolean checkCondition(int stuff){
if(stuff == 1 || stuff == 2){
return false;
}
return true;
}
This will output the result of multiple assertions and not stop on the evaluation of the first failed assertion.
Output:
org.assertj.core.api.SoftAssertionError:
The following 2 assertions failed:
1)
Expecting:
<false>
to be equal to:
<true>
but was not.
at JsonStewardshipCustomerConversionTest.soft_assertions(JsonStewardshipCustomerConversionTest.java:301)
2)
Expecting:
<false>
to be equal to:
<true>
but was not.
at JsonStewardshipCustomerConversionTest.soft_assertions(JsonStewardshipCustomerConversionTest.java:302)
Update
SoftAssertion does not seem to fit your purpose.
I suggest you use instead JUnit 5 assertAll. According to my tests it evaluates all conditions in an assertAll block and survives exceptions too. The problem here is you need JUnit 5 which is probably not largely adopted yet.
Here is an example with a failure on a boolean condition and also an exception. Both are reported in the console.
#Test
void soft_assertions() {
assertAll("Check condition",
() -> assertThat(checkCondition(9)).isTrue(),
() -> assertThat(checkCondition(10)).isTrue(),
() -> assertThat(checkCondition(11)).isTrue(),
() -> assertThat(checkCondition(2)).isTrue(), // Throws exception
() -> assertThat(checkCondition(3)).isFalse(), // fails
() -> assertThrows(IllegalArgumentException.class, () -> {
checkCondition(1);
})
);
}
private static boolean checkCondition(int stuff) {
if (stuff == 1 || stuff == 2) {
throw new IllegalArgumentException();
}
return true;
}
You will see this in the output:
org.opentest4j.MultipleFailuresError: Check condition (2 failures)
<no message> in java.lang.IllegalArgumentException
Expecting:
<true>
to be equal to:
<false>
but was not.