I am new to cucumber-jvm parallel plugin and want to understand 2 things basically -
First - Which is correct combination of version i shall choose of cucumber-jvm parallel plugin, maven surefire and compiler plugin
Second - what is difference between cucumber-jvm implementation supporting parallel execution from V 4.0 and cucumber-jvm-parallel plugin
There are 2 phases people configure in cucumber-jvm-parallel plugin. Goal as validate or generateRunners. Can someone guide me whts the difference between these 2.
<id>generateRunners</id>
<phase>generate-test-sources</phase>
<goals>
<goal>generateRunners</goal>
</goals>
Is there any mandatory configuring missing in below jvm implementation
<plugin>
<groupId>com.github.temyers</groupId>
<artifactId>cucumber-jvm-parallel-plugin</artifactId>
<version>2.2.0</version>
<executions>
<execution>
<id>generateRunners</id>
<phase>generate-test-sources</phase>
<goals>
<goal>generateRunners</goal>
</goals>
<configuration>
<glue>com.jacksparrow.automation.steps_definitions.functional</glue>
<outputDirectory>${project.build.directory}/generated-test-sources</outputDirectory>
<featuresDirectory>src/test/resources/features/functional/</featuresDirectory>
<cucumberOutputDir>target/cucumber-parallel</cucumberOutputDir>
<format>json,html</format>
<tags>"~#ignored"</tags>
</configuration>
</execution>
</executions>
</plugin>
Please guide me as this is making me confused.
Maven Surefire has two levels of parallel execution for JUnit tests. It can run individual classes in parallel and it can provide a scheduler to test runners that correctly implement the ParentRunner. Normally this is used to run methods in parallel by the BlockJUnit4ClassRunner.
Until version v4 Cucumber did not correctly implement ParentRunner so feature files could not be executed in parallel using the scheduler.
As explained in the cucumber-jvm-parallel-plugin/README the plugin will create an individual runner class for each feature. This means that parallel execution depends on Surefires ability to run classes in parallel rather then its ability to provide a scheduler.
If you are using Cucumber v4 you should not use the parallel plugin. For all but a few limited use cases there are no benefits to doing so and a whole slew of downsides.
The cucumber-jvm/junit/README.md describes how to configure surefire parallel execution (the latest stable version is surefire:2.22.1 and should work).
<build>
<plugins>
<plugin>
<artifactId>maven-surefire-plugin</artifactId>
<configuration>
<parallel>both</parallel>
<threadCount>4</threadCount>
</configuration>
</plugin>
</plugins>
</build>
So overall picture is, if we are using cucumber v 4.0.0 or above then we shall not use cucumber-jvm parallel plugin. Because one of the main principle of Cucumber 4.0.0 implementation is to provide support to parallel execution and coming to Maven Surefire version, we can use latest one as well i.e. 3.0.0-M3.
Related
Has anybody here experienced updating POM of the functional automation project with Karate-Gatling to be able to re-use functional scripts for performance but when executed both of the tests are triggered?
mvn test "-Dkarate.env=dev" -Dkarate.options="--tags #FunctionalScript classpath:services/project/functional.feature" -Dtest=TestRunner
The above mvn is supposed to run only the functional automation script, but the performance scripts are also getting triggered right after that.
Thanks in advance!
This depends on how you set up the maven project. If you have the following snippet in your gatling plugin setup, remove it:
<executions>
<execution>
<phase>test</phase>
<goals>
<goal>test</goal>
</goals>
</execution>
</executions>
Explained in detail here: https://github.com/intuit/karate/tree/develop/karate-gatling#maven
I'm researching using Maven for my team, and one of my goals for the build is to have static analysis tools run for every build, and perhaps fail the build for certain severe errors. However, in maven it seems that these are treated as reports and run on the site lifecycle. This presents a couple of problems as we don't intend to use the site for most projects, and it requires an extra step (or at least additional phase) to get the static analysis.
Is there a way to run static analysis tools in maven-2 in the compile or package phase (or equivalent)?
This can be done by configuring the plugin (findbugs for example) in the <build> section and configuring the <phase> in the <executions>:
<executions>
<execution>
<id>findbugs-run</id>
<phase>process-classes</phase>
<goals>
<goal>check</goal>
</goals>
</execution>
</executions>
I've got a vary simple profile in my maven pom file to run some integration tests during the normal test phase. Note I do not want to run these tests during the normal integration-test phase because I do not want to build the war and deploy etc. The tests run fine as normal JUnit tests.
So here is my profile:
<profile>
<id>AdminSeltests</id>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.12</version>
<configuration>
<includes>
<include>**/*/TestSellerSignupWizard.java</include>
</includes>
</configuration>
<executions>
<execution>
<id>execution2</id>
<phase>test</phase>
</execution>
</executions>
</plugin>
</plugins>
</build>
</profile>
My test is called:
com.xxxxx.xxx.client.selenium.seller_signup.TestCustomerSignupWizard
However when I run the above profile with:
mvn test -P AdminSeltests
No tests are run. I have tried the following as the value(s):
<include>**/TestSellerSignupWizard.*</include>
<include>**/TestSeller*.*</include>
<include>**/TestSeller*.java</include>
<include>**/*/TestSeller*.java</include>
<include>
com.xxxxx.xxx.client.selenium.seller_signup.TestCustomerSignupWizard.java
</include>
None of these work.
Any ideas anyone?
Thanks
Adam
SOLVED:
I'm using the maven-surefire-plugin and this has a automatic includes section which includes your normal test stuff. So I made an exclude configuration to exclude the normal unit tests, and then an include section to include a patter of integration test that I want to run
Not sure why it works this way, but it does:
<configuration>
<excludes>
<exclude>**/Test*.java</exclude>
<exclude>**/*Test.java</exclude>
<exclude>**/*TestCase.java</exclude>
</excludes>
<includes>
<include>**/ITTestSellerSignupWizard.java</include>
</includes>
</configuration>
Thanks for your help everyone.
The actual piece of code you did paste obviously doesn't work because you're mentioning class TestSellerSignupWizard while having tests in TestCustomerSignupWizard. I assume however it's a typo which doesn't matter actually because one of the Surefire's default mask for inclusion is **/Test*.java that fits well with you in this case.
So it all looks like working solution, so I'm afraid a problem is you don't have this class in your test classpath. You mentioned this is somehow related to integration testing so probably this class is located in src/it/java and not src/test/java that is Maven's default for Surefire. If I'm right, you should move this class into src/test/java or use (as you tried) alternative Surefire execution but with testSourceDirectory parameter overrided (link).
using Maven surefire, I'm unable to fork parallel test execution. That is, each of my test cases hs to run in a serapate JVM, hence the forking. In addition, I want my test cases to run in parallel. the first part is working without problem: I'm able to run each test case in its own JVM. the second part, however is still a challene for me. I haven't managed to get the paralle execution of test cases working. Here is how my plugin declaration look like:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.5</version>
<configuration>
<parallel>methods</parallel>
<forkMode>always</forkMode>
<argLine>-Xms512m -Xmx512m</argLine>
</configuration>
</plugin>
I've tried both methods and classes but haven't see any parallelization.
My JUnit version is 4.7 as shown by the depency declaration:
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.7</version>
<scope>compile</scope>
</dependency>
Any help would be much appricated.
Gregoire.
I think that you are supposed to use the threadCount parameter when using the parallel mode:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.6</version>
<configuration>
<forkMode>always</forkMode>
<argLine>-Xms512m -Xmx512m</argLine>
<parallel>methods</parallel>
<threadCount>4</threadCount>
</configuration>
</plugin>
I had the same problem, because i was using surefire version 2.7, after upgrade to 2.12 it worked with the following configuration:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.12</version>
<configuration>
<parallel>classes</parallel>
<forkMode>perthread</forkMode>
<threadCount>4</threadCount>
</configuration>
</plugin>
It spawned 4 threads, each running it's own jvm.
Make sure you get a log message something like this
[INFO] Concurrency config is {perCoreThreadCount=false, threadCount=1, parallel=classes, configurableParallelComputerPresent=true}
Just before this heading:
-------------------------------------------------------
T E S T S
-------------------------------------------------------
This message indicates that the parallel surefire junit provider is active.
If this is not present surefire may be picking up a different version of junit than you think. Anything below 4.7 will not work. Run mvn dependency:tree to check which version(s) are present.
You should also upgrade to surefire 2.6 since a number of minor bugs related to parallel running have been fixed. For the same reason you should use the latest junit.
Surefire's parallel mode is extremely buggy. For example, see http://jira.codehaus.org/browse/SUREFIRE-747 and http://jira.codehaus.org/browse/SUREFIRE-730
I haven't managed to get a single test running in parallel to date (not to mention forking).
Try changing your forkMode from always to "never". It does not state this in their documentation, but you can not have fork plus parallel at this time (we found this after digging through the surefire code.)
Just so you know, you will probably run into tests that are not thread safe due to many test/supporting libraries (easymock, powermock, etc) invalidating the ability to parallel your tests.
Are you sure it's not working? You may not gain much speedup if your tests do not contain many test methods. With forkMode=always the best you can do is run all test methods within a class, in parallel.
Isn't parallel setting a TestNG only attribute? according to this:
http://maven.apache.org/plugins/maven-surefire-plugin/test-mojo.html#parallel
The surefire 2.16 fixed the parallel execution regarding the JUnit tests.
When running a Maven build on the CI server, I generate the site to publish the documentation and reports, and also deploy the artifact to the snapshot repository for use by other projects. To do this I run the following goals:
mvn clean site deploy
This means the unit tests are run twice, once for the site lifecycle and once for the deploy lifecycle. If I configure the site goal to be bound to the standard lifecycle the tests are still run twice, running the site goal always causes the tests to be run because of the #requiresDependencyResolution test annotation. This is fine if you're only creating the site, but in the context of a deploy it greatly increases the build time for no benefit.
I have a workaround that involves copying the SiteMojo (and the required parents) to a new plugin and removing the #requiresDependencyResolution test annotation from the copy.
This modified mojo will generate the reports without forcing the tests to be run again but I'd prefer a solution that doesn't involve any hacking of the site plugin. Is there a way to suppress the requiresDependencyResolution annotation?
I'm surprised this works - the #requiresDependencyResolution test tag doesn't actually trigger the tests being built - that should be one of the reports that you've included. Normally, I recommend running the site and the build in separate Maven executions in CI so you can get fast feedback on your build and publish the latest site only when that succeeds.
Another alternative is to run it as mvn clean deploy site, and choose the report-only mojo for surefire-report-maven-plugin (this is usually the report that is running the tests again). This will use the previous test results. Of course, another alternative is disabling that report altogether, since you likely get those results from another source such as your CI server anyway.
My current approach is to create a new plugin with copies of the relevant types from the maven-site-plugin. These types are identical to the standard versions except for changing the type name, the goal name and the removal of the #requiresDependencyResolution test annotation.
The copied types are:
org.apache.maven.plugins.site.AbstractSiteMojo
org.apache.maven.plugins.site.AbstractSiteRenderingMojo
The parent mojos are required so Maven can process the javadoc-based annotations (this shouldn't be required for Maven 2.2.0+).
org.apache.maven.plugins.site.SiteMojo
org.apache.maven.plugins.site.SiteJarMojo
These two are copied as SiteOnlyMojo and SiteJarOnlyMojo respectively, SiteJarOnlyMojo is changed to inherit from SiteOnlyMojo . Otherwise the only changes are to change the goal namess and remove the annotation.
So SiteOnlyMojo has:
* #goal site
* #requiresDependencyResolution test
changed to:
* #goal site-only
and SiteJarOnlyMojo has:
* #goal jar
* #phase package
changed to:
* #goal jar-only
* #phase package
These types are declared in a maven-plugin project with artifactId maven-site-only-plugin with a dependency declared on the proper site plugin.
To use this I define a profile (I don't want the reports running on every execution, only when -Psite is declared on the command line) and bind it to the prepare-package phase (prior to 2.1.0, you'd have to bind it to the package phase instead).
<profile>
<id>site</id>
<build>
<plugins>
<plugin>
<artifactId>maven-site-only-plugin</artifactId>
<version>1.0.0</version>
<executions>
<execution>
<phase>prepare-package</phase>
<goals>
<goal>jar-only</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
</profile>
<build>
<pluginManagement>
<plugins>
...
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-site-plugin</artifactId>
<version>3.2</version>
<executions>
<execution>
<goals>
<goal>site</goal>
<goal>deploy</goal>
</goals>
<phase>deploy</phase>
</execution>
</executions>
</plugin>
...
</plugins>
</pluginManagement>
</build>
This will automatically execute the site action when issuing a 'mvn deploy' command, as well as ensuring the testing suite is only executed once.