Why is Surefire not executing my JUnit5 tests? - junit5

I'm trying to convert one Maven SpringBoot (2.3.12) app from JUnit4 to JUnit5. I've read many different posts on how to do this.
I'm able to execute my JUnit5 tests in Eclipse.
My problem is that I am unable to get Maven Surefire to execute my JUnit5 tests. I've tried various configuration variations. When it gets to the Surefire step, it ONLY executes my old JUnit4 tests and simply ignores any JUnit5 tests. I've verified that if one of those JUni4 tests is executed from this, and I convert that test to JUnit5, it will then ignore that test. At this point, I'm only focusing on the tests that were using the Mockito runner, as they are simpler to convert than the Spring runner tests (the vast majority of which likely could be changed to just use the mockito runner).
This is what I have in the surefire plugin config:
<plugin>
<artifactId>maven-surefire-plugin</artifactId>
<configuration>
<!-- <groups>UnitTest</groups> -->
<reuseForks>false</reuseForks>
<redirectTestOutputToFile>true</redirectTestOutputToFile>
<argLine>${surefireArgLine}</argLine>
<skipTests>${skip.unit.tests}</skipTests>
<excludes>
<exclude>**/component/*.java</exclude>
<exclude>**/contract/*.java</exclude>
<exclude>**/integration/*.java</exclude>
</excludes>
<dependencies>
<dependency>
<groupId>org.junit.jupiter</groupId>
<artifactId>junit-jupiter-engine</artifactId>
<version>5.8.2</version>
</dependency>
</dependencies>
</configuration>
</plugin>
I currently have "groups" commented out until I understand what it is doing without that.
This is an excerpt of one of the unit tests:
import org.junit.jupiter.api.Tag;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.extension.ExtendWith;
import org.mockito.InjectMocks;
import org.mockito.junit.jupiter.MockitoExtension;
#ExtendWith(MockitoExtension.class)
public class BDSHelperTest {
Before I converted this to use the JUnit5 classes, it would execute when I run "mvn package". Now that I've converted it, Surefire is not executing it.
This is what I see at the start of the surefire step:
[INFO] --- maven-surefire-plugin:3.0.0-M5:test (default-test) # NotesMs ---
[INFO]
[INFO] -------------------------------------------------------
[INFO] T E S T S
[INFO] -------------------------------------------------------
At this point, it shows the test runs for all the old JUnit4 tests, but it doesn't run any of the JUnit5 tests.

I am the Apache developer of Surefire. I have created a project which proves that it works with Surefire 3.0.0-M5, JUnit Jupiter 5.8.2 and Mockito Extension.
Please do not use JUnit4 and Vintage in this case.
It is not necessary to use a dependency inside of the plugin.
Use a dependency in the project POM.
Follow it on Github:
https://github.com/Tibor17/junit5-mockito-examples

Related

Junit5 test suites with #Suite annotation doesn't execute tests with mvn test command

We intended to upgrade a JUnit4 based project to JUnit5. I modified JUnit4 suites according to the instruction in JUnit5 official site: https://junit.org/junit5/docs/current/user-guide/#running-tests-junit-platform-runner-test-suite.
Tests were not executed when using mvn test in command line: mvn test -Dtest=SuiteXXTest, neither in JUnit4 nor JUnit5 suite.
Error msg: [ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:3.0.0-M5:test (default-test) on project XXX: No tests were executed!
Yet both JUnit4 and JUnit5 suites can run and execute the included tests in IDEA. Maven surefire plugin version is 3.0.0-M5. Single test classes(ATest and BTest) are based on JUnit4 progamming model.
Old JUnit4 suite is like this:
import org.junit.runner.RunWith;
import org.junit.runners.Suite;
import test.XX.*;
#RunWith(Suite.class)
#Suite.SuiteClasses({
ATest.class,
BTest.class
})
public class SuiteXXTest {
}
Modified to:
import org.junit.platform.suite.api.SelectClasses;
import org.junit.platform.suite.api.Suite;
import test.XX.*;
#Suite
#SelectClasses({
ATest.class,
BTest.class
})
public class SuiteXXTest {
}
related dependencies:
<dependency>
<groupId>org.junit.jupiter</groupId>
<artifactId>junit-jupiter</artifactId>
<version>5.8.2</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.junit.vintage</groupId>
<artifactId>junit-vintage-engine</artifactId>
<version>5.8.2</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.junit.platform</groupId>
<artifactId>junit-platform-suite</artifactId>
<version>1.8.2</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.12</version>
</dependency>
Curiously, if I use the depracated #RunWith(JUnitPlatform.class) annotation, the test suites can execute properly. But according to JUnit5 official user guide, using this runner means tests will be executed in a JUnit4 envrionment, and I don't understand what condition is regarded as JUnit4 environment(JUnit5 dependency already configured in pom).
I don't known if it's a bug or my incorrect use.
RunWith is a JUnit 4 construct and won’t work with JUnit 5. Use JUnit Platform suites instead as described in https://junit.org/junit5/docs/current/user-guide/#junit-platform-suite-engine
Found the solution here: https://github.com/junit-team/junit5/discussions/2753.
According to this answer I just changed my command from -Dtest=SuiteXXTest to -Dinclude=SuiteXXTest and it works fine. I think it has something to do with the surefire plugin. It seems surefire cannot pick up tests form suite annotated with #Suite when using -Dtest command.

Visibility of plain jar in tycho-surefire-plugin

The problem
I am faced with the following scenario:
The sources of an Eclipse test plugin (tycho packaging type eclipse-test-plugin) depend on a "plain" jar (read: non-OSGi jar). I managed to get the tests to compile and run in Eclipse, however when running Maven/Tycho from the command line, tycho-surefire-plugin fails to execute the tests because the jar is not visible at test time. This results in a java.lang.NoClassDefFoundError while attempting to load a class from the jar.
Looking at mvn -e -X output does not reveal anything significant.
My question is, how can I include the jar in the classpath of tycho-surefire-plugin when running Maven/Tycho from the command line?
Attempts
Here is everything I have tried so far:
Use <extraRequirements> as per the tycho-surefire-plugin documentation. This however fails because the jar's packaging type is jar, while <extraRequirements> expects one of the eclipse-xxx packaging types.
For a good measure I also tried
<configuration>
<dependencies>
<dependency>
<groupId>. . .</groupId>
<artifactId>. . .</artifactId>
<version>. . .</version>
<scope>system</scope>
<systemPath>path-to-the-jar</systemPath>
</dependency>
</dependencies>
</configuration>
but the packaging type is still considered to be jar.
Use <argLine> as per tycho-surefire-plugin documentation:
<groupId>org.eclipse.tycho</groupId>
<artifactId>tycho-surefire-plugin</artifactId>
<configuration>
<argLine>-cp path-to-the-jar</argLine>
</configuration>
however this appears to have no effect as the java.lang.NoClassDefFoundError persists.
Include the jar in the Eclipse test plugin. The jar is present in the MANIFEST.MF
Bundle-ClassPath: the.jar
in the build.properties
bin.includes = META-INF/,\
the.jar
and in the .classpath (although this doesn't matter for tycho-surefire-plugin).
<classpathentry exported="true" kind="lib" path="the.jar"/>
tycho-surefire-plugin once again reports java.lang.NoClassDefFoundError.
Create a dedicated Eclipse plugin to house the jar. This is for the most part equivalent to 3, where the Eclipse test plugin simply depends on this new dedicated Eclipse plugin. The java.lang.NoClassDefFoundError still rears its head.
Transition to Tycho 2.x.x since it supports the Directory location type. Unfortunately it is not possible to include the jar in the target definition because its packaging type is jar.

How do I generate surefire report for all tests and make this work with jenkins?

I have a large number of Junit tests running selenium on eclipse, what I would like to do is generate a Junit or surefire report for the tests after all of them are run. I know that with maven you can do this for individual tests by running mvn test on the console and generating a report with:
mvn surefire-report:report-only
However, this tests and produces a report for each individual test, is there a way to make this work for multiple tests? The reason behind this is because I run these tests headlessly on Jenkins, I know that jenkins email plugin can allow me to pass a html report post build which would give me an idea of test success and failure.
You can try using this command:
#mvn clean test
This will clean the existing reports and run all test cases in one go and generate surefire-reports for all.
Please let me know if this what you are looking for. :)
Firstly you need to have all the right plugins and dependencies in your pom.xml file.
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-site-plugin</artifactId>
<version>3.7.1</version>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>3.0.0-M4</version>
</plugin>
</plugins>
</build>
<!-- `mvn clean test site` to generate the junit html report-->
<reporting>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-report-plugin</artifactId>
<version>3.0.0-M4</version>
</plugin>
</plugins>
</reporting>
Following this add the maven command to your maven build as shown.
After you run this your IDE will run all the tests in that maven build and generate a surefire.html report which can be found in target/site/surefire-report.html
(You may need to update your maven project to see the folder come up for the first time by right clicking the project, then hover over maven and then click update maven project.)
I would advise doing this as it then works with jenkins when it runs your tests in headless mode which can then be attached to your email extension plugin.

run maven tests from classpath

In order to clean up something of a giant mess, I set out to put the code of my tests all in one ordinary java project (all in src/main/java), and then declare that as a <scope>test</scope> dependency in another project, and expect the tests to run.
No such luck. surefire wants to just run the tests that it can see in the sources.
I can see a sadly obvious solution here involving the build-helper-plugin and adding the tests into the test compilation environment as a source directory, but I was hoping to avoid it.
In case anyone is wondering, the reason for all this is that the POM configuration for use of the failsafe plugin to run some integration tests got so complex that I wanted to split out the compiling of the test classes from the running of the tests.
This is now possible with Maven Surefire v2.15. Simply add the following kind of configuration to the surefire plugin:
<build>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.15</version>
<configuration>
<dependenciesToScan>
<dependency>com.group.id:my-artifact</dependency>
<dependency>com.group.id:my-other-artifact</dependency>
</dependenciesToScan>
...
</configuration>
...
</plugin>
...
</build>
You should also declare the actual dependencies in the dependencies section:
<dependencies>
<dependency>
<groupId>com.group.id</groupId>
<artifactId>my-artifact</artifactId>
<type>test-jar</type>
<version>1.1</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>com.group.id</groupId>
<artifactId>my-other-artifact</artifactId>
<type>test-jar</type>
<version>1.1</version>
<scope>test</scope>
</dependency>
</dependencies>
No such luck. surefire wants to just run the tests that it can see in the sources.
This is currently not possible out of the box, surefire just looks at classes in target/test-classes:
Re: Surefire not picking up tests from test-jar
Re: maven-surefire-plugin: run unit tests from classes in a jar, not a directory
This is actually logged as SUREFIRE-569 - There should be a way to run unit tests from a dependency jar.
I can see a sadly obvious solution here involving the build-helper-plugin and adding the tests into the test compilation environment as a source directory, but I was hoping to avoid it.
The current workaround is to use dependency:unpack to unpack the jar into target/test-classes before the test phase.
Can't you do it the other way round?
I mean put the code the src/test/java, depend on your main module, and run the tests in your test module?

Maven Eclipse unit test configuration

I use Eclipse and for some of my unit tests, I need to set some JVM args for the test to work, -Djava.library.path in particular. I set it in my POM file as follows:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<configuration>
<argLine>-Djava.library.path=target/dll</argLine>
</configuration>
</plugin>
and it works correctly when I build with Maven. However, when I run in Eclipse (even with the Maven plug-in installed - I have everything else in my Eclipse environment working properly with maven), this JVM argument is not applied. Does anyone know why this is or how to fix this?
Running the test in Eclipse with the JUnit test runner does not invoke maven , even if you are using m2eclipse.
I suggest you either:
create a maven launch configuration for the test goal;
add -Djava.library.path=target/dll to your JVM arguments in the JUnit launch configuration.