I'm new to DevOps work and I've been tasked to setup a build for one of our .net Standard Library projects. I used the build template for this, which works great! But I want to add a code coverage into build output (there's a test project in the solution).
In the "VsTest - testAssemblies" task, I've checked the "Enable Code Coverage" option, as shown below:
I added a Publish code coverage task and it's asking for something called a "Summary File" - see below:
I'm not sure how to set this option? Has anyone done this?
I seem to get test results published in the build results without config, see below:
And I'd like the same for the Code Coverage tab in the build results, which is currently blank:
Also (may be a completely separate question), I was wondering - can I add a "Quality Gate" to say if Coverage is below 80% then fail the build?
Thanks for any advice in advance - it's a case of knowing what I want, but not sure how to achieve it!
Hurrah!!!
There is a solution for it. :)
You have to install a an Add-on to your organization in VSTS (Azure DevOps).
The name of the addon is Build Quality Checks
Here is the link:
https://marketplace.visualstudio.com/items?itemName=mspremier.BuildQualityChecks
It works perfectly. And this is what you want
By default, the VsTest task will publish the Code Coverage result to Build Summary page directly. So we don't need Publish Code Coverage result task to publish the coverage result. Just make sure you have selected "Code Coverage enable" section in VsTest task.
>> Also (may be a completely separate question), I was wondering - can I add a "Quality Gate" to say if Coverage is below 80% then fail the build?
In current VSTS, we could not set "Quality Gate" to set the build failed if it doesn't match the configured % value. There has other communities also have this requirement and have submit a user voice. Please feel free to add your comments or vote it from below link:
https://visualstudio.uservoice.com/forums/330519-visual-studio-team-services/suggestions/3817520-fail-build-on-insufficient-code-coverage
Related
Is it possible to generate auto API test report using Datadog? :)
I'm creating a project in Java using Rest Assured and Hamcrest. I have the project on gitlab and I would like the tests to be automatically run. Based on them, I would like to receive a report :).
At first I through I could use a allure but my supervisor asked me if I could do it in Datadog.
I tried to find background material on this subject, but failed :(
I will be grateful for every link with supporting materials :)
What do you use for build tool maven or gradle?
I think this should work for you:
https://docs.datadoghq.com/continuous_integration/tests/java
We have QA and DEV environment in our automation repo. We are using karate as our framework. We have TestParallel class and integrated allure report.
How could we run all tests in QA first then in DEV back to back using TestParallel Class and see the results in the same report?
Thanks for such a great tool btw.
We are going to try and make this easier in the next version.
For now, you have to aggregate the reports yourself. Can you try this and let us know how it goes.
use the Runner class 2 times to run your tests with different settings and karate.env set for QA and then DEV
the important part is using a different value for the workingDir, e.g. target/reports/qa and then target/reports/dev - else the second run will overwrite the first
now when generating the HTML report, you can provide target/reports as the source folder. this should work for the Maven Cucumber Reports, for Allure, please figure this out on your own
if the above approach does not work well enough for your needs, please figure out a way to manually aggregate the Results object you get from each instance of the Runner, this should not be too complicated as Java code
I've started using Jenkins in order to compile and test my builds periodically.
The job is fairly simple. It compiles the build and then executes the tests.
The test results are stored in a file. There's also a distinct line saying how many tests have passed. (Something like X tests passed out of Y).
I'd like to know what's the simplest way/plug-in to display these results at the end of the build.
I'd like a visual display, since I know Jenkins is very nice in displaying graphs over time/job.
I appreciate your help, and forgive me if this question already exists on Stackoverflow. I haven't found anything close enough for me.
Wouldn't it be better if you can publish your test results in a format that is more understandable by Jenkins? If so, see this link on how to generate a simple test result. Once you have done that, the visual display that Jenkins offers comes to you at free of cost.
You can install the Groovy Postbuild Plugin and use it to parse your distinct line and display it right next to your build result :
def matcher = manager.getLogMatcher("(.*) tests passed out of (.*)\$")
if(matcher != null && matcher.matches()) {
passedTests = matcher.group(1)
totalTests = matcher.group(2)
manager.addShortText("${passedTests} / ${totalTests}")
}
You could also enhance it with a badge or a customized color depending of the success ratio.
Hope it helps
I have a process for running automated functional tests which is external to Microsoft Team Foundation Server (TFS) 2010. Test cases are tracked as Test Case work items within TFS, however. After running these tests, how can I publish the results to TFS using the TFS API? Can someone point me to sample code that demonstrates this?
Please note that I expressly want to avoid a solution that requires transformation of my test results into the .trx file format. Searches have turned up dead links, or solutions that rely on this method.
It sounds like the following blogs may almost be what you're after
http://blogs.msdn.com/b/jpricket/archive/2010/02/23/creating-fake-builds-in-tfs-build-2010.aspx
http://msmvps.com/blogs/vstsblog/archive/2011/04/26/creating-fake-builds-in-tfs-build-2010-using-the-command-line.aspx
It doesn't actually have code for adding test results, however it does say this:
"In order to associate test results and the like, you have to create build project nodes with the fake build."
You should be able to create a Microsoft.TeamFoundation.Build.Client.TestSummary with a summary of your test results.
There's a couple of internal classes that look interesting, specifically Microsoft.TeamFoundation.Build.Controls.TestRunDetails, which could potentially be useful if you don't mind using some reflection.
However what I would recommend, is using the API to look at the nodes in a standard TFS build to see how they are built up.
We have a particular file, say X.zip that is only modified by 1 or 2 people. Hence we don't want the build to trigger on every check-in, as the other files are mostly untouched.
I need to check for a condition prior to building, whether the checked-in item is "X.zip" or not.. if yes, then trigger a build, else don't. We use only CI builds.
Any idea on how to trigger the build only when this particular file is checked-in? Any other approaches would be greatly appreciated as i am a newbie in TFS...
Tara.
I don't know of any OOTB feature which can do this, what you would need to do is write your own custom MSBuild task which is executed prior to the build running (pre-build action).
The task will then need to use the TFS API to check the current check in for the file you want and if it's not found you'll have to set the task to failed.
This isn't really ideal as it'll indicate to Team Build a build failure, which, depending on whether you're using check in policies, may be unhelpful. It'd also be harder to at-a-glance work out which builds failed because of the task and which failed because of a real problem.
You can change the build to occur less frequently rather than every check in, which will reduce load on your build server.
Otherwise you may want to dig into Cruise Control .NET, it may support better conditional builds.
If you could move X.zip into it's own folder, then you could set up a CI build with a workspace that only looked at the folder containing X.zip.
You would then need to add an explicit call to tf get to download the rest of the code as Team Build only downloads what the workspace is looking at.
But this might be simpler than the custom task approach?