Read dotCover total coverage output from GitLab CI - gitlab-ci

I configured dotCover to be run from our GitLab CI server.
It correctly runs the tests, produces the required output and the CI is configured to store the coverage HTML output in the GitLab artifacts. This works flawlessly.
What I'm trying to do is to read the total coverage output from the dotCover.exe console runner and parse it in gitlab CI. I read the dotCover documentation but I did not find a method to output a line containing the coverage to console. Gitlab CI can only read coverage values from the sdout of the ci job, matching it with a custom regex.
This is my config.xml:
<?xml version="1.0" encoding="utf-8"?>
<AnalyseParams>
<TargetExecutable>C:\NUnit\nunit3-console.exe</TargetExecutable>
<TargetArguments>--agents=1 MyDll.Spec.dll MyDll2.Spec.dll</TargetArguments>
<Output>cover/AppCoverageReport.html</Output>
<ReportType>html</ReportType>
<Scope>
<ScopeEntry>MyApp\bin\x86\Release\net461\MyApp.*.dll</ScopeEntry>
<ScopeEntry>MyApp\bin\x86\Release\net461\*.exe</ScopeEntry>
</Scope>
<Filters>
<ExcludeFilters>
<FilterEntry>
<ModuleMask>*.Spec</ModuleMask>
</FilterEntry>
</ExcludeFilters>
</Filters>
</AnalyseParams>
and I run it with .gitlab-ci.yml:
C:\dotCover\dotCover.exe analyse config.xml /TargetWorkingDir=.
Is there a way to view this value in GitLab CI? Am I missing something obvious?
Thank you

I ended up reading the HTML output file of dotCover and parsing the output.
Luckily the total coverage is at an easily parse-able portion of the output file. The coverage regex is
'/= \[\["Total",\d+\.?\d+/'
This is my final .gitlab-ci.yml file (for Windows runner, dotCover is windows-only):
my_job:
# your job configuration
# ...
scripts:
# build the solution here, ...
- dotCover.exe analyse dotCover.xml /TargetWorkingDir=.
- type cover\AppCoverageReport.html
coverage: '/= \[\["Total",\d+\.?\d+/'
Not a very long-term solution but it works for now, at least until I update dotCover.

Related

Cobertura - Coverlet.runsettings.xml values ignored by dotnetcore CLI test command

I am trying to generate code coverage for one of my solutions in an Azure DevOps build pipeline. Whilst I have the results of my tests appearing upon build completion along with the Cobertura code coverage report, there are a lot of files and namespaces I would like to exclude from the code coverage report as they are not testable units of code (e.g. models or database migrations).
I understand that I can utilise the coverlet.runsettings.xml file to do namespace exclusions but this does not appear to be working.
My runsettings file is setup inside one of the test projects that is run and is setup as follows:
<?xml version="1.0" encoding="utf-8" ?>
<RunSettings>
<DataCollectionRunSettings>
<DataCollectors>
<DataCollector friendlyName="XPlat code coverage">
<Configuration>
<Exclude>[MyDatabase.Migrations.*]*,[*]MyDatabase.Migrations*</Exclude>
</Configuration>
</DataCollector>
</DataCollectors>
</DataCollectionRunSettings>
</RunSettings>
I'm trying to exclude all files that sit under the MyDatabase.Migrations namespace but the syntax I have used above doesn't seem to have any effect and I still see the files beneath this namespace shown in final code coverage report in Devops.
My pipeline is setup as follows:
steps:
- task: DotNetCoreCLI#2
displayName: 'Build'
inputs:
command: 'build'
projects: '$(solution)'
arguments: '--configuration $(buildConfiguration)'
- task: DotNetCoreCLI#2
displayName: 'Tests'
inputs:
command: 'test'
projects: |
**\MyDatabase.Tests
**\MySearch.Common.Tests
**\MySearchService.Tests
**\MyConfiguration.Tests
arguments: '--configuration $(buildConfiguration) --collect:"XPlat Code Coverage"'
publishTestResults: true
# ReportGenerator extension to combine code coverage outputs into one
- task: reportgenerator#5
inputs:
reports: '$(Agent.WorkFolder)/**/coverage.cobertura.xml'
targetdir: '$(Build.SourcesDirectory)/CoverageResults'
- task: PublishCodeCoverageResults#1
displayName: 'Publish code coverage report'
inputs:
codeCoverageTool: 'Cobertura'
summaryFileLocation: '$(Build.SourcesDirectory)/CoverageResults/Cobertura.xml'
reportDirectory: '$(Build.SourcesDirectory)/CoverageResults'
I am aware I can use the [ExcludeFromCodeCoverage] tag but with things that are auto-generated like migrations this could get pretty messy having to go in and modify them every time a new one is generated.
I would like the migrations folder completely omitted from even appearing in the generated report.
Any help with this issue would be greatly appreciated.
The VSTestIntegration documentation indicates you will need to include the custom runsettings file as an agrument:
This runsettings file can easily be provided using command line option as given : dotnet test --collect:"XPlat Code Coverage" --settings coverlet.runsettings
Try the following:
arguments: '--configuration $(buildConfiguration) --collect:"XPlat Code Coverage" --settings [path to tests project]\coverlet.runsettings.xml'

What Gitlab tool used for code coverage reports?

Instead of using JaCoCo, I was told, that there would be an internal Gitlab tool, where I can create test coverage reports?
I do not want to use JaCoCo.
I am not interessted in any vizualization plugin within Gitlab.
I would like to generate a xml/html file(s) with e.g. bar graphs, what can be emailed and opened externally.
I couldn't find anything in the Gitlab dashboard menu. The project is a Android App Kotlin project.
the question is what part of Coverage you want to see/have:
just a number within the MR - therefore GitLab parses the logoutput of the Jobs
coverage visualization within MR - therefore you need to provide a report.
Coverage in Overview
For the coverage in the Overview and just to get a percentage, you need to configure your job with an regex how it can be parsed like
job1:
# ....
coverage: '/Code coverage: \d+\.\d+/'
https://docs.gitlab.com/ee/ci/yaml/#coverage
Visualization
We are actually using JaCoCo, but to make the coverage visible and to have the information in Merge Requests you have to convert everything into Cobertura Reports.
There are different approaches to achieve this:
with a gradle-plugin like https://github.com/kageiit/gradle-jacobo-plugin
the configuration is pretty neat, and if you do have already a gradle build it is easy to integrate
with an own step within the CI Pipeline - see https://docs.gitlab.com/ee/user/project/merge_requests/test_coverage_visualization.html
test-jdk11:
stage: test
image: gradle:6.6.1-jdk11
script:
- 'gradle test jacocoTestReport' # jacoco must be configured to create an xml report
artifacts:
paths:
- build/jacoco/jacoco.xml
coverage-jdk11:
# Must be in a stage later than test-jdk11's stage.
# The `visualize` stage does not exist by default.
# Please define it first, or chose an existing stage like `deploy`.
stage: visualize
image: registry.gitlab.com/haynes/jacoco2cobertura:1.0.7
script:
# convert report from jacoco to cobertura, using relative project path
- python /opt/cover2cover.py build/jacoco/jacoco.xml $CI_PROJECT_DIR/src/main/java/ > build/cobertura.xml
needs: ["test-jdk11"]
artifacts:
reports:
cobertura: build/cobertura.xml
important to note is that you always will have to tell GitLab CI your path to the artifact for cobertura with
job:
#...
artifacts:
reports:
cobertura: build/cobertura.xml
Our approach is the following.
have to tell Gitlab where your coverage report is, for example we have this setup for a java unit test report "jacoco.xml":
Unit Test:
stage: pruebas
script:
- echo "Iniciar Pruebas"
- mvn $MAVEN_CLI_OPTS test
artifacts:
when: always
reports:
junit:
- target/surefire-reports/*Test.xml
- target/failsafe-reports/*Test.xml
cobertura: target/site/jacoco/jacoco.xml
Our summary in Gitlab :
Unit Test Detaills:
The key is your "jacoco.xml".

dotCover not generating coverage for all assemblies

I've set up dotCover to run using an .xml
<?xml version="1.0" encoding="utf-8"?>
<CoverageParams>
<TargetExecutable>
c:\dotcover\xunit\xunit.console.exe
</TargetExecutable>
<TargetArguments>
"INWK.Configuration.UnitTests.dll"
</TargetArguments>
<TargetWorkingDir>
..\bin\x64\Debug\
</TargetWorkingDir>
<TempDir>
<!-- Directory for auxiliary files. Set to the system temp by default. -->
</TempDir>
<Output>
dotCover-xunit.dcvr
</Output>
<InheritConsole>
<!-- [True|False] Lets the application being analyzed to inherit dotCover console. True by default. -->
</InheritConsole>
</CoverageParams>
You can see (Service, Shared, UnitTests assemblies correctly included in the test coverage report (Shared, Service and UnitTest assemblies)
However, when running the same on the build server *Service and *Shared are missing.
After replacing Service.dll and Shared.dll and their "pdb's" from local copy to build server and running dotCover on build server again it works correctly.
This leads me to believe that build server runner does something different than msbuild.exe from VS when running build locally.
I found very similar issue description here: https://stackoverflow.com/questions/25855131/dotcover-and-xunit-not-gathering-coverage-statistics-in-some-environments, but not sure how to remedy this in my build server configuration.
Trace log output (one drive)
https://1drv.ms/t/s!AtxuuqGHIqXwgTVqQJ_Y_-rGE8W9?e=HrZgj7
Found the solution:
in my dotcover config xml I had to add: -noshadow switch, like so:
<CoverageParams>
<TargetExecutable>
c:\dotcover\xunit\xunit.console.exe
</TargetExecutable>
<TargetArguments>
"INWK.OrderIndexing.UnitTests.dll" -noshadow
</TargetArguments>
<TargetWorkingDir>
..\bin\x64\Release\
</TargetWorkingDir>
...
Now all assemblies (except the ones I do want to filter) are showing up

SonarQube - integrationTest.exec - sonarRunner (Gradle) or "sonar-runner" command - showing 0.0% covereage

I'm successfully generating 2 .exec files by Jacoco within "build/jacoco" folder after running a Gradle based build and integration tests.
Gradle command:
"gradle clean build integrationTest"
Once done, it generates the following .exec files under build/jacoco folder.
test.exec
integrationTest.exec
Following is my sonar-project.properties file. When, I run "sonar-runner" from Linux prompt it completes but on SonarQube dashboard for this project, I see Unit test says some 34.5% but integration tests says 0.0%. Both .exec files have valid size. I also did "cat" on the .exec files and piped the output to "strings" command in Linux and saw that integrationTest.exec did hit the Tests functions - I have only 1 .java file.
When I run "gradle clean build integrationTest sonarRunner -Dxxx.xxx=yyy -Dyyy.xx=zzz" i.e. by passing all the sonar variable as mentioned in the sonar-project.properties file using -D option, it works but same result on SonarQube project's dashboard. Project's sonar dashboard has both widgets configured for Unit / Integration Tests and I'm including IT tests for showing Overall coverage. Overall coverage is showing 34.5% (which is Unit test % value). Sonar does see test.exec, integrationTest.exec and also auto generates overall-xxx.exec file as well during this operation.
NOTE: I'm no where - while starting tomcat on a separate putty / linux console -OR within Gradle build script, providing any value or setting JAVA Agent for Jacoco. I'm getting integrationTest.exec file and test.exec file already so not sure if JVM needs to be stopped once IT tests are complete running. I don't think I need those as i have valid file size for .exec files.
My ?:
- Why sonar is not getting IT coverage on the dashboard even though I'm setting / passing the following variable correctly:
sonar.jacoco.itReportPath=build/jacoco/integrationTest.exec
-bash-3.2$ cat sonar-project.properties
# Root project information
sonar.projectKey=com:company:product:ProjectA
sonar.projectName=ProjectA
sonar.projectVersion=1.0
# optional description
sonar.projectDescription=ProjectA Service
#Tells SonarQube that the code coverage tool by unit tests is JaCoCo
sonar.java.coveragePlugin=jacoco
#Tells SonarQube to reuse existing reports for unit tests execution and coverage reports
sonar.dynamicAnalysis=reuseReports
# Some properties that will be inherited by the modules
sonar.sources=src/java,test/java,src/java-test
# Sonar Unit Test Report path
sonar.jacoco.reportPath=build/jacoco/test.exec
# Sonar Integration Test Report Path
sonar.jacoco.itReportPath=build/jacoco/integrationTest.exec
sonar.junit.reportsPath=build/UT/results
# Sonar Binaries
sonar.binaries=build/classes/main
Narrowing down the cause: I think it's due to the .exec file for Integration test. To proove it: I passed UT exex file to both reportsPaths in Sonar variables i.e. the following and SonarQube picked both UT/IT test coverage. This prooves that if .exec file for IT tests is good (which I think it's But I need to double check) then Sonar will pick the .exec file and show a valid coverage % instead of 0.0%. Note: the following is just to proove if Sonar is picking the values or not. itReportPath variable should use the .exe file which is generated by Integration tests by Jacoco.
sonar.jacoco.reportPath=build/jacoco/test.exec
# Sonar Integration Test Report Path
#sonar.jacoco.itReportPath=build/jacoco/testintegrationTest.exec
sonar.jacoco.itReportPath=build/jacoco/test.exec
OK Found the issue. I was running integrationTest task in Gradle and was NOT attaching the jacocoagent.jar (as per Jacoco documentation) to the target JVM (Tomcat's instance) scope. Once I did that, I removed jacoco { ... } section from integrationTest task in Gradle (build.gradle or GRADLE_HOME/init.d/some.common.gradle file as this attach jacoco agent to the Java JVM in which Gradle is running). Now, once jacocoagent.jar was attached to Tomcat's JVM (as per the line below which I added in Tomcat's startup.sh script and added the variable to the command which starts Tomcat), then I ran Gradle (integrationTest) task for running IT tests.
PROJ_EXTRA_JVM_OPTS=-javaagent:tomcat/jacocoagent.jar=destfile=build/jacoco/IT/jacocoIT.exec,append=false
Then while Gradle was in progress, tests ran and I got a file (jacocoIT.exec at the given location) with some file size BUT this is not yet the final one. I had to stop the Tomcat session/JVM instance by running Tomcat's stop.sh script. Once Tomcat was stopped, I saw jacocoIT.exec file size increased significantly and this was the valid final jacocoIT.exec file (which I needed for sonarRunner Gradle task OR sonar-runner exectuable to pick and successfully push IT code coverage data to project's sonar dashboard). Once done, I got both UT + IT and it's combined code coverage.
sonar.jacoco.reportPath=build/jacoco/UT/jacocoUT.exec
sonar.jacoco.itReportPath=build/jacoco/IT/jacocoIT.exec

Configuration for "SeleniumHQ htmlSuite Run" in Jenkins to run Selenium HTML TestSuite

I want to run my Selenium HTML Test Suite through Jenkins (a continuous integration). The following shows, how the build is configured for the current project:
And here's the console output after commiting a new test for example:
ERROR: The suiteFile is not a file or an url ! Check your build configuration.
Build step 'SeleniumHQ htmlSuite Run' changed build result to FAILURE
Build step 'SeleniumHQ htmlSuite Run' marked build as failure
Publishing Selenium report...
Finished: FAILURE
In fact, I get these log issues even after committing both extensionless test files AND .html files.
SeleniumHQ Jenkins plugin supports only ONE suite file per build step. Try out Selunit to run Selenese suites in batch and across multiple browsers. This tutorial shows hot to setup the test execution in Jenkins/Hudson.
Your suiteFile is written with wildcard as: tests/selenium/*.html. I think it is wrong.
You need to provide the exact/absolute path to your suite without the wildcard as below:
tests/selenium/suite.html