I'm cleaning up an Android project that has several hundred Espresso tests of which several are flaky. Getting 100% instrumented test success on a regular basis isn't realistic in this context.
My Gradle task for Jacoco code coverage works perfectly (!) when all the Espresso tests do pass (achievable by skipping the flaky ones). When there are test failures, however, the Jacoco report shows 0% coverage even though the Gradle test report has the correct data. Looks like the coverage.ec file generated is 0 bytes.
The connected<flavor>DebugAndroidTest task has been configured with ignoreFailures = true, so the build no longer halts when there were failing tests.
Is there any way to get the test task to correctly generate the coverage.ec file in spite of failures?
Related
I have manually instrumented my code using:
istanbul instrument src --o temp --es-modules --config=.istanbul.yml.
This is my .istanbul.yml:
instrumentation:
excludes: ['*.spec.js']
extensions: ['.js','.jsx']
Once it is instrumented I am running e2e tests using Selenium inside IntelliJ, using the run with coverage button.
The tests pass but at the end it only gives me coverage information of the *.e2e.js files and not the actual *.jsx file that the e2e test is running.
Any ideas?
The JavaScript is executed in the browser, not by the test-runnner. So only the code that is used by the test-runner is included in the coverage. You need to instrument the front-end code and send it to the browser and collect the coverage from the browser.
Here is how it could work with istanbul and Selenium:
Instrument your front-end code with the istanbul
instrument command. (As far as I know, istanbul instrument writes out
instrumented code to disk, whereas istanbul cover does everything in
memory.)
Instead of sending the original JS code to the browser, send
the instrumented JS code. The really nice thing here, with Istanbul,
you don’t have to manually modify your source code at all to make this
all work. Istanbul does almost all of the work for us in the browser,
automatically.
Run your Selenium-based tests, and for each individual
driver in your tests, run a hook that will send the coverage results
from the browser to the backend test process.
Once you get the
coverage data in the test process, you can do whatever you want with
it. In this case, we will HTTP POST the data to a server which can
interpret and display the coverage results.
And that’s it!
Read the full article : https://medium.com/#the1mills/front-end-javascript-test-coverage-with-istanbul-selenium-4b2be44e3e98
The article goes over all the details how to set it up.
When I run unit tests with the coverage, I can see that lines are covered by unit tests. But when I commit them Sonarqube shows that lines are uncovered. How can i configure sonarqube to measure unit test written with using powermockito?
First of all read the docs of SonarJava - actually everything you need to know is in there :D
short outline:
you need to generate a report for the coverage with eg. JaCoCo or Cobertura
you need to provide a property with a path to those reports eg. for JaCoCo sonar.jacoco.reportPaths=<path>
you run the analysis and sonar will use those reports
I check out the source code of Cassandra, and I want to run a unit test case in debug mode to understand how it works
Below is my JUnit run configuration set up. The code can compile correctly using ant. And I tried both targets build and build-test.
IntelliJ can pick up the class in the run configuration, but when I run this profile, I got.
Process finished with exit code 1
Class not found: "org.apache.cassandra.tools.SSTableExportTest"
Empty test suite.
What part do I need to change the configuration so that IntelliJ can run this unit test cases?
Since you're trying to debug Cassandra. You can reference https://wiki.apache.org/cassandra/RunningCassandraInIDEA. They have already set up a ant target to generate required configurations.
I'm not sure whether the issue is with NUnit or SpecFlow but whenever I add the 'Run Functional Tests' task to my build, I get the following error when running a build in TFS2017...
2017-05-06T00:11:00.4676774Z ##[warning]DistributedTests: Test discovery started.
2017-05-06T00:11:00.4676774Z ##[warning]DistributedTests: Test Run Discovery Aborted . Test run id : 5
2017-05-06T00:11:00.4676774Z ##[warning]DistributedTests: UnExpected error occured during test execution. Try again.
2017-05-06T00:11:00.4676774Z ##[warning]DistributedTests: Error : NUnit Adapter 3.7.0.0: Test discovery complete
2017-05-06T00:11:00.4676774Z
2017-05-06T00:11:00.4676774Z ##[warning]DistributedTests: Test run aborted. Test run id: 5
2017-05-06T00:11:00.4676774Z ##[error]System.Exception: The test run was aborted, failing the task.
2017-05-06T00:11:00.5175379Z ##[error]PowerShell script completed with 1 errors.
Does anyone know what the problem is?
Many thanks,
Run Functional Tests task are used for below scenarios.
Typical scenarios include:
Tests that require additional installations on the test machines, such as different browsers for Selenium tests
Coded UI tests
Tests that require a specific operating system configuration
To execute a large number of unit tests more quickly by using multiple test machines
If you are using specflow you need to make sure that you implement using MS Test.
Regardless of the fact which unit test provider you use for SpecFlow, because all the major unit test frameworks provide the necessary adapter for it (this is the same adapter that you need to run the tests from the Visual Studio Test Explorer Window)
So you need to use Visual Studio Test task in this scenario(Also for handling standard unit tests). For details of the settings please refer this blog: SpecFlow Tips--Run only specific scenarios in TFS/VSTS build
I'm successfully generating 2 .exec files by Jacoco within "build/jacoco" folder after running a Gradle based build and integration tests.
Gradle command:
"gradle clean build integrationTest"
Once done, it generates the following .exec files under build/jacoco folder.
test.exec
integrationTest.exec
Following is my sonar-project.properties file. When, I run "sonar-runner" from Linux prompt it completes but on SonarQube dashboard for this project, I see Unit test says some 34.5% but integration tests says 0.0%. Both .exec files have valid size. I also did "cat" on the .exec files and piped the output to "strings" command in Linux and saw that integrationTest.exec did hit the Tests functions - I have only 1 .java file.
When I run "gradle clean build integrationTest sonarRunner -Dxxx.xxx=yyy -Dyyy.xx=zzz" i.e. by passing all the sonar variable as mentioned in the sonar-project.properties file using -D option, it works but same result on SonarQube project's dashboard. Project's sonar dashboard has both widgets configured for Unit / Integration Tests and I'm including IT tests for showing Overall coverage. Overall coverage is showing 34.5% (which is Unit test % value). Sonar does see test.exec, integrationTest.exec and also auto generates overall-xxx.exec file as well during this operation.
NOTE: I'm no where - while starting tomcat on a separate putty / linux console -OR within Gradle build script, providing any value or setting JAVA Agent for Jacoco. I'm getting integrationTest.exec file and test.exec file already so not sure if JVM needs to be stopped once IT tests are complete running. I don't think I need those as i have valid file size for .exec files.
My ?:
- Why sonar is not getting IT coverage on the dashboard even though I'm setting / passing the following variable correctly:
sonar.jacoco.itReportPath=build/jacoco/integrationTest.exec
-bash-3.2$ cat sonar-project.properties
# Root project information
sonar.projectKey=com:company:product:ProjectA
sonar.projectName=ProjectA
sonar.projectVersion=1.0
# optional description
sonar.projectDescription=ProjectA Service
#Tells SonarQube that the code coverage tool by unit tests is JaCoCo
sonar.java.coveragePlugin=jacoco
#Tells SonarQube to reuse existing reports for unit tests execution and coverage reports
sonar.dynamicAnalysis=reuseReports
# Some properties that will be inherited by the modules
sonar.sources=src/java,test/java,src/java-test
# Sonar Unit Test Report path
sonar.jacoco.reportPath=build/jacoco/test.exec
# Sonar Integration Test Report Path
sonar.jacoco.itReportPath=build/jacoco/integrationTest.exec
sonar.junit.reportsPath=build/UT/results
# Sonar Binaries
sonar.binaries=build/classes/main
Narrowing down the cause: I think it's due to the .exec file for Integration test. To proove it: I passed UT exex file to both reportsPaths in Sonar variables i.e. the following and SonarQube picked both UT/IT test coverage. This prooves that if .exec file for IT tests is good (which I think it's But I need to double check) then Sonar will pick the .exec file and show a valid coverage % instead of 0.0%. Note: the following is just to proove if Sonar is picking the values or not. itReportPath variable should use the .exe file which is generated by Integration tests by Jacoco.
sonar.jacoco.reportPath=build/jacoco/test.exec
# Sonar Integration Test Report Path
#sonar.jacoco.itReportPath=build/jacoco/testintegrationTest.exec
sonar.jacoco.itReportPath=build/jacoco/test.exec
OK Found the issue. I was running integrationTest task in Gradle and was NOT attaching the jacocoagent.jar (as per Jacoco documentation) to the target JVM (Tomcat's instance) scope. Once I did that, I removed jacoco { ... } section from integrationTest task in Gradle (build.gradle or GRADLE_HOME/init.d/some.common.gradle file as this attach jacoco agent to the Java JVM in which Gradle is running). Now, once jacocoagent.jar was attached to Tomcat's JVM (as per the line below which I added in Tomcat's startup.sh script and added the variable to the command which starts Tomcat), then I ran Gradle (integrationTest) task for running IT tests.
PROJ_EXTRA_JVM_OPTS=-javaagent:tomcat/jacocoagent.jar=destfile=build/jacoco/IT/jacocoIT.exec,append=false
Then while Gradle was in progress, tests ran and I got a file (jacocoIT.exec at the given location) with some file size BUT this is not yet the final one. I had to stop the Tomcat session/JVM instance by running Tomcat's stop.sh script. Once Tomcat was stopped, I saw jacocoIT.exec file size increased significantly and this was the valid final jacocoIT.exec file (which I needed for sonarRunner Gradle task OR sonar-runner exectuable to pick and successfully push IT code coverage data to project's sonar dashboard). Once done, I got both UT + IT and it's combined code coverage.
sonar.jacoco.reportPath=build/jacoco/UT/jacocoUT.exec
sonar.jacoco.itReportPath=build/jacoco/IT/jacocoIT.exec