I have a Ruby on Rails project that I want to display test coverage in SonarQube. The project is built with Jenkins which sends the results to SonarQube. Unit tests are run with rspec and code coverage is calculated with SimpleCov. SonarQube requires that code coverage report is generated in json format so I used simplecov-json in my rails_helper.rb:
if Rails.env.development? || Rails.env.test?
require 'simplecov'
require 'simplecov-json'
SimpleCov.formatter = SimpleCov::Formatter::JSONFormatter
SimpleCov.start
end
Code coverage report is generated successfully and saved to coverage folder like this (from Jenkins console output):
Coverage report generated for RSpec to /my-project/coverage/coverage.json. 2000 / 2000 LOC (100.0%) covered.
I configured my sonar-project.properties like this (to read the coverage report):
sonar.sources=.
sonar.exclusions=**/*_test.go,**/vendor/**,**/coverage/**
sonar.tests=.
sonar.test.inclusions=**/*_spec.rb
sonar.test.exclusions=**/vendor/**
sonar.ruby.coverage.reportPaths=coverage/coverage.json
sonar.ruby.coverage.framework=RSpec
Jenkins builds the project, runs the tests, generates the coverage report, sends the results to SonarQube. But SonarQube always displays 0% coverage for some reason.
Running sonar-scanner -X (debug mode) shows this error (Jenkins console log):
15:52:30.458 ERROR: Cannot read coverage report file, expecting standard SimpleCov resultset JSON format: 'coverage/coverage.json'
java.lang.ClassCastException: java.lang.String cannot be cast to org.sonarsource.analyzer.commons.internal.json.simple.JSONObject
at org.sonarsource.ruby.plugin.SimpleCovSensor.mergeFileCoverages(SimpleCovSensor.java:112)
at org.sonarsource.ruby.plugin.SimpleCovSensor.execute(SimpleCovSensor.java:71)
at org.sonar.scanner.sensor.SensorWrapper.analyse(SensorWrapper.java:45)
It looks like SonarQube is having trouble reading the json coverage report - it’s not formatted in the way it expects.
Sonar's SimpleCov support expects the .resultset.json file, not the output of the JSON formatter. You'll need to point the configuration to the result set instead:
sonar.ruby.coverage.reportPaths=coverage/.resultset.json
Related
I'm using TestCafe Studio to create my tests and executing the tests written in .testcafe format using the testcafe docker container. Further I'm using 'Drone' as CI environment.
Below is the command that I use to execute my tests
`- /opt/testcafe/docker/testcafe-docker.sh -c 3 chromium -q --skip-js-errors --ass`ertion-timeout 60000 --selector-timeout 60000 CommonScenarios/*.testcafe
When a test failure occur I will not get enough information about the test failure. For example below is an error log printed.
1) AssertionError: expected false to be truthy
Browser: Chrome 91.0.4472.124 / Linux 0.0
7
Is there any way that I can get enough details about which step is actually failing when the tests are executed in .testcafe format?
(When I ran .js format of the test it gives which line is failing)
This looks like a bug in TestCafe Framework. I opened an issue in the GitHub repository: https://github.com/DevExpress/testcafe/issues/6424. Subscribe it to be notified of updates.
As a simple solution, you can convert your *.testcafe fixture file to *.js. Also, there is a more complex workaround - it allows you to determine which step is failing:
Open the *.testcafe file in VS Code or some other editor. You will see that it looks like a JSON file.
Find an object with the "name" property whose value corresponds to your test name: "name": "Your-failed-test-name"
Look at the "commands" array. Find an object with the "callsite" property whose value is equal to the number that you can see in the error console output. This object specifies the failed step.
Note that the format of this file is intended for internal use, and it is not recommend to modify it manually.
In our Yii project we're using Jenkins CI and Codeception for different types of tests. The problem is, that codeception report is empty, which causes whole Jenkins build failure.
All tests are running without errors. Jenkins execute shell for codeception:
php codecept.phar run --xml --html
Console output error line which causes failure:
[xUnit] [ERROR] - The result file '/var/lib/jenkins/jobs/project/workspace/code/protected/tests/_output/report.xml' for the metric 'PHPUnit' is empty. The result file has been skipped.
I understand simple logic, if report is empty -> build failed. But why is report empty? Is that a bug or can I do something about this?
The problem was, that in one of our tearDowns was the following line:
Yii::app()->end();
which makes Yii-Application die. For some reasons this caused that codeception has not generated the report.
Just started using codeception 2.0.9 and have installed it with phantomjs and got some sample tests running.
Codeception gives a concise html report at the end that summarises passed tests in red and green, which is nice.
In the reporting directory there is also a screenshot of failed tests, e.g. ExampleCest.CheckLabel.fail.png ExampleCest.CheckLabel.fail.html
Question, how can these be (automatically) linked into report.html so that one can see the failed screenshots from the report in the browser?
Yes, there is a enhancement (#3602) to include screenshot in the report, but it's not release in the current version (2.2).
You can check this feature status at https://github.com/Codeception/Codeception/pull/3602.
I'm successfully generating 2 .exec files by Jacoco within "build/jacoco" folder after running a Gradle based build and integration tests.
Gradle command:
"gradle clean build integrationTest"
Once done, it generates the following .exec files under build/jacoco folder.
test.exec
integrationTest.exec
Following is my sonar-project.properties file. When, I run "sonar-runner" from Linux prompt it completes but on SonarQube dashboard for this project, I see Unit test says some 34.5% but integration tests says 0.0%. Both .exec files have valid size. I also did "cat" on the .exec files and piped the output to "strings" command in Linux and saw that integrationTest.exec did hit the Tests functions - I have only 1 .java file.
When I run "gradle clean build integrationTest sonarRunner -Dxxx.xxx=yyy -Dyyy.xx=zzz" i.e. by passing all the sonar variable as mentioned in the sonar-project.properties file using -D option, it works but same result on SonarQube project's dashboard. Project's sonar dashboard has both widgets configured for Unit / Integration Tests and I'm including IT tests for showing Overall coverage. Overall coverage is showing 34.5% (which is Unit test % value). Sonar does see test.exec, integrationTest.exec and also auto generates overall-xxx.exec file as well during this operation.
NOTE: I'm no where - while starting tomcat on a separate putty / linux console -OR within Gradle build script, providing any value or setting JAVA Agent for Jacoco. I'm getting integrationTest.exec file and test.exec file already so not sure if JVM needs to be stopped once IT tests are complete running. I don't think I need those as i have valid file size for .exec files.
My ?:
- Why sonar is not getting IT coverage on the dashboard even though I'm setting / passing the following variable correctly:
sonar.jacoco.itReportPath=build/jacoco/integrationTest.exec
-bash-3.2$ cat sonar-project.properties
# Root project information
sonar.projectKey=com:company:product:ProjectA
sonar.projectName=ProjectA
sonar.projectVersion=1.0
# optional description
sonar.projectDescription=ProjectA Service
#Tells SonarQube that the code coverage tool by unit tests is JaCoCo
sonar.java.coveragePlugin=jacoco
#Tells SonarQube to reuse existing reports for unit tests execution and coverage reports
sonar.dynamicAnalysis=reuseReports
# Some properties that will be inherited by the modules
sonar.sources=src/java,test/java,src/java-test
# Sonar Unit Test Report path
sonar.jacoco.reportPath=build/jacoco/test.exec
# Sonar Integration Test Report Path
sonar.jacoco.itReportPath=build/jacoco/integrationTest.exec
sonar.junit.reportsPath=build/UT/results
# Sonar Binaries
sonar.binaries=build/classes/main
Narrowing down the cause: I think it's due to the .exec file for Integration test. To proove it: I passed UT exex file to both reportsPaths in Sonar variables i.e. the following and SonarQube picked both UT/IT test coverage. This prooves that if .exec file for IT tests is good (which I think it's But I need to double check) then Sonar will pick the .exec file and show a valid coverage % instead of 0.0%. Note: the following is just to proove if Sonar is picking the values or not. itReportPath variable should use the .exe file which is generated by Integration tests by Jacoco.
sonar.jacoco.reportPath=build/jacoco/test.exec
# Sonar Integration Test Report Path
#sonar.jacoco.itReportPath=build/jacoco/testintegrationTest.exec
sonar.jacoco.itReportPath=build/jacoco/test.exec
OK Found the issue. I was running integrationTest task in Gradle and was NOT attaching the jacocoagent.jar (as per Jacoco documentation) to the target JVM (Tomcat's instance) scope. Once I did that, I removed jacoco { ... } section from integrationTest task in Gradle (build.gradle or GRADLE_HOME/init.d/some.common.gradle file as this attach jacoco agent to the Java JVM in which Gradle is running). Now, once jacocoagent.jar was attached to Tomcat's JVM (as per the line below which I added in Tomcat's startup.sh script and added the variable to the command which starts Tomcat), then I ran Gradle (integrationTest) task for running IT tests.
PROJ_EXTRA_JVM_OPTS=-javaagent:tomcat/jacocoagent.jar=destfile=build/jacoco/IT/jacocoIT.exec,append=false
Then while Gradle was in progress, tests ran and I got a file (jacocoIT.exec at the given location) with some file size BUT this is not yet the final one. I had to stop the Tomcat session/JVM instance by running Tomcat's stop.sh script. Once Tomcat was stopped, I saw jacocoIT.exec file size increased significantly and this was the valid final jacocoIT.exec file (which I needed for sonarRunner Gradle task OR sonar-runner exectuable to pick and successfully push IT code coverage data to project's sonar dashboard). Once done, I got both UT + IT and it's combined code coverage.
sonar.jacoco.reportPath=build/jacoco/UT/jacocoUT.exec
sonar.jacoco.itReportPath=build/jacoco/IT/jacocoIT.exec
I want to run my Selenium HTML Test Suite through Jenkins (a continuous integration). The following shows, how the build is configured for the current project:
And here's the console output after commiting a new test for example:
ERROR: The suiteFile is not a file or an url ! Check your build configuration.
Build step 'SeleniumHQ htmlSuite Run' changed build result to FAILURE
Build step 'SeleniumHQ htmlSuite Run' marked build as failure
Publishing Selenium report...
Finished: FAILURE
In fact, I get these log issues even after committing both extensionless test files AND .html files.
SeleniumHQ Jenkins plugin supports only ONE suite file per build step. Try out Selunit to run Selenese suites in batch and across multiple browsers. This tutorial shows hot to setup the test execution in Jenkins/Hudson.
Your suiteFile is written with wildcard as: tests/selenium/*.html. I think it is wrong.
You need to provide the exact/absolute path to your suite without the wildcard as below:
tests/selenium/suite.html