I am trying to get MSTest working with CruiseControl.Net. I currently have it working prefectly for unit tests, but I would like to see my code coverage. When I run MSTest and export it to an XML file (using /resultsfile:"C:\output\results.xml") it seems that no Coverage information is exported. However when I run MS test and open the result TRX file, it contains all of the code coverage information. What am I missing?
Did you include the MsTestSummary.xsl in your web dashboard configuration?
EDIT: Code Coverage report is not supported, you'll have to modify the xsl file yourself (and preferably attach your changes to a bug report so that other users can benefit).
Related
We have QA and DEV environment in our automation repo. We are using karate as our framework. We have TestParallel class and integrated allure report.
How could we run all tests in QA first then in DEV back to back using TestParallel Class and see the results in the same report?
Thanks for such a great tool btw.
We are going to try and make this easier in the next version.
For now, you have to aggregate the reports yourself. Can you try this and let us know how it goes.
use the Runner class 2 times to run your tests with different settings and karate.env set for QA and then DEV
the important part is using a different value for the workingDir, e.g. target/reports/qa and then target/reports/dev - else the second run will overwrite the first
now when generating the HTML report, you can provide target/reports as the source folder. this should work for the Maven Cucumber Reports, for Allure, please figure this out on your own
if the above approach does not work well enough for your needs, please figure out a way to manually aggregate the Results object you get from each instance of the Runner, this should not be too complicated as Java code
I'm new to DevOps work and I've been tasked to setup a build for one of our .net Standard Library projects. I used the build template for this, which works great! But I want to add a code coverage into build output (there's a test project in the solution).
In the "VsTest - testAssemblies" task, I've checked the "Enable Code Coverage" option, as shown below:
I added a Publish code coverage task and it's asking for something called a "Summary File" - see below:
I'm not sure how to set this option? Has anyone done this?
I seem to get test results published in the build results without config, see below:
And I'd like the same for the Code Coverage tab in the build results, which is currently blank:
Also (may be a completely separate question), I was wondering - can I add a "Quality Gate" to say if Coverage is below 80% then fail the build?
Thanks for any advice in advance - it's a case of knowing what I want, but not sure how to achieve it!
Hurrah!!!
There is a solution for it. :)
You have to install a an Add-on to your organization in VSTS (Azure DevOps).
The name of the addon is Build Quality Checks
Here is the link:
https://marketplace.visualstudio.com/items?itemName=mspremier.BuildQualityChecks
It works perfectly. And this is what you want
By default, the VsTest task will publish the Code Coverage result to Build Summary page directly. So we don't need Publish Code Coverage result task to publish the coverage result. Just make sure you have selected "Code Coverage enable" section in VsTest task.
>> Also (may be a completely separate question), I was wondering - can I add a "Quality Gate" to say if Coverage is below 80% then fail the build?
In current VSTS, we could not set "Quality Gate" to set the build failed if it doesn't match the configured % value. There has other communities also have this requirement and have submit a user voice. Please feel free to add your comments or vote it from below link:
https://visualstudio.uservoice.com/forums/330519-visual-studio-team-services/suggestions/3817520-fail-build-on-insufficient-code-coverage
I understand that nunit console 3 can write to TestResult.xml after running the tests, where the TestResult.xml is located inside the directory specified by --work parameter.
But from what I can tell, TestResult.xml contains too many (irrelevant) details, that I don't need to fix my unit tests errors. All I need is just the failed or ignored test cases, just like what is displayed in command line prompt when I am running nunit console in it.
How to configure the parameters for nunit console 3 so that it only gives me failed or ignored test cases?
Sorry, I'm a few months late (unfortunately I came across this problem only now). But as far as I searched, the only possibility is to use XSL transformation. There are some applications that can convert the XML report file, but... I tested a few, but unfortunately the output was not that was I need.
Therefore I created a simple NUnit3summary application that can transform the standard XML output file to text file. I was surprised that nobody until now made something like this (or at least did not publish it). It were only two hours of work (first working version) and a few more to finish it to stare ready for publishing.
It is only a simple application that was aimed for my needs. You can use filtration for now only with another application, e.g.:
NUnit3summary.exe TestResult.xml | grep Failed >FailedTests.txt
You can see a practical application here (this is also the project where the application was needed, because of too many errors in unit tests).
I have a multi module Gradle script with 3 modules: common, services and web.
If I run gradle check all the test suite is performed correctly, but I don't have a single test report instead I found a test report for each subproject. How can I merge the test results?
If useful this is the script I'm using https://github.com/CarloMicieli/trenako/blob/master/build.gradle
Thanks
Carlo
The Gradle source contains a Task class used to accomplish this, which unfortunately isn't available in the public API. Basically all it does is copy all of the test reports from various projects into one directory and then run the standard test report task to get an aggregate. Check out the source code at https://github.com/gradle/gradle/blob/master/buildSrc/src/main/groovy/org/gradle/build/TestReportAggregator.groovy
Is it possible to set number of test runs variable on TFS 2010 build so that is displayed in the build summary? We are running NUnit tests and can get the number of tests run. At the moment we see "No Test Results" in build summary.
and congratulations on your promotion from shoe-salesman to TFS technician... :)
You can add the NUnit test run results to the TFS build report in one of two ways:
1. You can write a simple XSLT to transform the nunit report to a TRX file (which TFS can read).
2. You can use NUnit for Team Build.
Either way, the NUnit results will be added to the report, including the count of successful, failed, inconclusive and (IIRC) total count of tests.
Hope this helps,
Assaf.