SonarQube - integrationTest.exec - sonarRunner (Gradle) or "sonar-runner" command - showing 0.0% covereage - testing

I'm successfully generating 2 .exec files by Jacoco within "build/jacoco" folder after running a Gradle based build and integration tests.
Gradle command:
"gradle clean build integrationTest"
Once done, it generates the following .exec files under build/jacoco folder.
test.exec
integrationTest.exec
Following is my sonar-project.properties file. When, I run "sonar-runner" from Linux prompt it completes but on SonarQube dashboard for this project, I see Unit test says some 34.5% but integration tests says 0.0%. Both .exec files have valid size. I also did "cat" on the .exec files and piped the output to "strings" command in Linux and saw that integrationTest.exec did hit the Tests functions - I have only 1 .java file.
When I run "gradle clean build integrationTest sonarRunner -Dxxx.xxx=yyy -Dyyy.xx=zzz" i.e. by passing all the sonar variable as mentioned in the sonar-project.properties file using -D option, it works but same result on SonarQube project's dashboard. Project's sonar dashboard has both widgets configured for Unit / Integration Tests and I'm including IT tests for showing Overall coverage. Overall coverage is showing 34.5% (which is Unit test % value). Sonar does see test.exec, integrationTest.exec and also auto generates overall-xxx.exec file as well during this operation.
NOTE: I'm no where - while starting tomcat on a separate putty / linux console -OR within Gradle build script, providing any value or setting JAVA Agent for Jacoco. I'm getting integrationTest.exec file and test.exec file already so not sure if JVM needs to be stopped once IT tests are complete running. I don't think I need those as i have valid file size for .exec files.
My ?:
- Why sonar is not getting IT coverage on the dashboard even though I'm setting / passing the following variable correctly:
sonar.jacoco.itReportPath=build/jacoco/integrationTest.exec
-bash-3.2$ cat sonar-project.properties
# Root project information
sonar.projectKey=com:company:product:ProjectA
sonar.projectName=ProjectA
sonar.projectVersion=1.0
# optional description
sonar.projectDescription=ProjectA Service
#Tells SonarQube that the code coverage tool by unit tests is JaCoCo
sonar.java.coveragePlugin=jacoco
#Tells SonarQube to reuse existing reports for unit tests execution and coverage reports
sonar.dynamicAnalysis=reuseReports
# Some properties that will be inherited by the modules
sonar.sources=src/java,test/java,src/java-test
# Sonar Unit Test Report path
sonar.jacoco.reportPath=build/jacoco/test.exec
# Sonar Integration Test Report Path
sonar.jacoco.itReportPath=build/jacoco/integrationTest.exec
sonar.junit.reportsPath=build/UT/results
# Sonar Binaries
sonar.binaries=build/classes/main

Narrowing down the cause: I think it's due to the .exec file for Integration test. To proove it: I passed UT exex file to both reportsPaths in Sonar variables i.e. the following and SonarQube picked both UT/IT test coverage. This prooves that if .exec file for IT tests is good (which I think it's But I need to double check) then Sonar will pick the .exec file and show a valid coverage % instead of 0.0%. Note: the following is just to proove if Sonar is picking the values or not. itReportPath variable should use the .exe file which is generated by Integration tests by Jacoco.
sonar.jacoco.reportPath=build/jacoco/test.exec
# Sonar Integration Test Report Path
#sonar.jacoco.itReportPath=build/jacoco/testintegrationTest.exec
sonar.jacoco.itReportPath=build/jacoco/test.exec
OK Found the issue. I was running integrationTest task in Gradle and was NOT attaching the jacocoagent.jar (as per Jacoco documentation) to the target JVM (Tomcat's instance) scope. Once I did that, I removed jacoco { ... } section from integrationTest task in Gradle (build.gradle or GRADLE_HOME/init.d/some.common.gradle file as this attach jacoco agent to the Java JVM in which Gradle is running). Now, once jacocoagent.jar was attached to Tomcat's JVM (as per the line below which I added in Tomcat's startup.sh script and added the variable to the command which starts Tomcat), then I ran Gradle (integrationTest) task for running IT tests.
PROJ_EXTRA_JVM_OPTS=-javaagent:tomcat/jacocoagent.jar=destfile=build/jacoco/IT/jacocoIT.exec,append=false
Then while Gradle was in progress, tests ran and I got a file (jacocoIT.exec at the given location) with some file size BUT this is not yet the final one. I had to stop the Tomcat session/JVM instance by running Tomcat's stop.sh script. Once Tomcat was stopped, I saw jacocoIT.exec file size increased significantly and this was the valid final jacocoIT.exec file (which I needed for sonarRunner Gradle task OR sonar-runner exectuable to pick and successfully push IT code coverage data to project's sonar dashboard). Once done, I got both UT + IT and it's combined code coverage.
sonar.jacoco.reportPath=build/jacoco/UT/jacocoUT.exec
sonar.jacoco.itReportPath=build/jacoco/IT/jacocoIT.exec

Related

Validate that NUnit's ITestEventListener within Jenkins Pipeline for a netcoreapp3.1 assembly is called

Locally, I have successfully implemented the interface ITestEventListener within a C# netcoreapp3.1 csproj. However, when the tests are an within a Jenkins Pipeline, things appear to not be working(?).
I am using version 3.12.0 of NUnit.Engine.
By locally I am referring to using 1) Visual Studio Version 16.9.2 to run the tests and 2) command line dotnet test -c devint --test-adapter-path:. --logger:nunit to run the tests. I am getting successful test runs.
Success is my [Extension]public class ReportTestListener : ITestEventListener {...} generates an html file. I am to see the html file is created locally whereas from the Jenkins Pipeline the html file is not generated.
Within the Jenkins Pipeline, I am using the command sh "dotnet test -c ${env.TARGET_ENV} --test-adapter-path:. --logger:nunit" where env.TARGET_ENV resolves to devint. I know tests successfully run within the Jenkins Pipeline since the NUnit test results file is generated/published.
What I am not sure of is how to test/validate that the ReportTestListener is being called within the Jenkins Pipeline. I know that testing frameworks such as NUnit uses refection to identify test classes and methods. I am presuming that also happens with my implementation of [Extension]public class ReportTestListener : ITestEventListener {...}. Ideas/Suggestions on how to validate that ITestEventListener's method void OnTestEvent(string report) is being called besides writing out to disk?
Changed file writing text path to use / instead of \.

Empty test suite error when running unit test case in intellij

I check out the source code of Cassandra, and I want to run a unit test case in debug mode to understand how it works
Below is my JUnit run configuration set up. The code can compile correctly using ant. And I tried both targets build and build-test.
IntelliJ can pick up the class in the run configuration, but when I run this profile, I got.
Process finished with exit code 1
Class not found: "org.apache.cassandra.tools.SSTableExportTest"
Empty test suite.
What part do I need to change the configuration so that IntelliJ can run this unit test cases?
Since you're trying to debug Cassandra. You can reference https://wiki.apache.org/cassandra/RunningCassandraInIDEA. They have already set up a ant target to generate required configurations.

jenkins error FATAL: Unable to find build script at C:\selenium tests for moveon\moveon4tests1\build.xml

I am using Jenkins and Ant to run my selenium tests. It works fine when I run them on my local machine.
Now I have a Jenkins server which is on a different location and I have to run my tests on that server.
when I tried to run the tests on that server Its is failing because it couldn't find the build.xml file. can you please let me know how can I change the home directory path?
Started by user :**********
Building in workspace /var/lib/jenkins/jobs/moveon4 java tests/workspace
FATAL: Unable to find build script at C:\selenium tests for move\movetests1\build.xml
Build step 'Invoke Ant' marked build as failure
Finished: FAILURE
Are you using "Invoke Ant" in your Jenkins job configuration for the build step that executes your Ant build ? If so, the build file location is hidden away in the Advanced section - it's an annoying quirk of Jenkins.

Configuration for "SeleniumHQ htmlSuite Run" in Jenkins to run Selenium HTML TestSuite

I want to run my Selenium HTML Test Suite through Jenkins (a continuous integration). The following shows, how the build is configured for the current project:
And here's the console output after commiting a new test for example:
ERROR: The suiteFile is not a file or an url ! Check your build configuration.
Build step 'SeleniumHQ htmlSuite Run' changed build result to FAILURE
Build step 'SeleniumHQ htmlSuite Run' marked build as failure
Publishing Selenium report...
Finished: FAILURE
In fact, I get these log issues even after committing both extensionless test files AND .html files.
SeleniumHQ Jenkins plugin supports only ONE suite file per build step. Try out Selunit to run Selenese suites in batch and across multiple browsers. This tutorial shows hot to setup the test execution in Jenkins/Hudson.
Your suiteFile is written with wildcard as: tests/selenium/*.html. I think it is wrong.
You need to provide the exact/absolute path to your suite without the wildcard as below:
tests/selenium/suite.html

Generation of Selenium reports using Hudson (now known as Jenkins) from JUnit XML format file

For Test Automation of web project we use Hudson, PHPUnit, and Selenium. The results of the build are stored in the JUnit XML format.
When I try to include generation of reports using the Hudson Publish JUnit test result report option, the build finishes with Failed status.
Below is my Hudson configuration to run the tests.
sudo -u apache phpunit - log-junit /var/lib/hudson/jobs/Work-stars-Tests/builds/${BUILD_ID}/ seleniumReports/seleniumTests.xml + path to test php files
Generation of reports is enabled through the Hudson configuration option «Publish JUnit test result report», where I specify the path to the folder with PHP tests.
The user we use to run Hudson has permission to write/read files in the folder with the reports. As for the path we've tried to specify both full and relative.
The error No test report files were found. Configuration error? is displayed in the console after the build.
How do we resolve this issue?
Found my own solution to this problem :)
In Hudson project configuration settings modified execute shell command to
> #!/bin/sh -x phpunit --log-junit ${WORKSPACE}/zf/tests/_tmp/reports/seleniumTests.xml
> ${WORKSPACE}/zf/tests/selenium/; sed
> -i '/<testsuite name=".*\/"/D;/^ <\/testsuite>$/D'
> ${WORKSPACE}/zf/tests/_tmp/reports/seleniumTests.xml
and set following path to Junit reports
**/zf/tests/_tmp/reports/*.xml
Problem is solved. Yahoo!