I've started using Jenkins in order to compile and test my builds periodically.
The job is fairly simple. It compiles the build and then executes the tests.
The test results are stored in a file. There's also a distinct line saying how many tests have passed. (Something like X tests passed out of Y).
I'd like to know what's the simplest way/plug-in to display these results at the end of the build.
I'd like a visual display, since I know Jenkins is very nice in displaying graphs over time/job.
I appreciate your help, and forgive me if this question already exists on Stackoverflow. I haven't found anything close enough for me.
Wouldn't it be better if you can publish your test results in a format that is more understandable by Jenkins? If so, see this link on how to generate a simple test result. Once you have done that, the visual display that Jenkins offers comes to you at free of cost.
You can install the Groovy Postbuild Plugin and use it to parse your distinct line and display it right next to your build result :
def matcher = manager.getLogMatcher("(.*) tests passed out of (.*)\$")
if(matcher != null && matcher.matches()) {
passedTests = matcher.group(1)
totalTests = matcher.group(2)
manager.addShortText("${passedTests} / ${totalTests}")
}
You could also enhance it with a badge or a customized color depending of the success ratio.
Hope it helps
Related
We have QA and DEV environment in our automation repo. We are using karate as our framework. We have TestParallel class and integrated allure report.
How could we run all tests in QA first then in DEV back to back using TestParallel Class and see the results in the same report?
Thanks for such a great tool btw.
We are going to try and make this easier in the next version.
For now, you have to aggregate the reports yourself. Can you try this and let us know how it goes.
use the Runner class 2 times to run your tests with different settings and karate.env set for QA and then DEV
the important part is using a different value for the workingDir, e.g. target/reports/qa and then target/reports/dev - else the second run will overwrite the first
now when generating the HTML report, you can provide target/reports as the source folder. this should work for the Maven Cucumber Reports, for Allure, please figure this out on your own
if the above approach does not work well enough for your needs, please figure out a way to manually aggregate the Results object you get from each instance of the Runner, this should not be too complicated as Java code
I'm new to DevOps work and I've been tasked to setup a build for one of our .net Standard Library projects. I used the build template for this, which works great! But I want to add a code coverage into build output (there's a test project in the solution).
In the "VsTest - testAssemblies" task, I've checked the "Enable Code Coverage" option, as shown below:
I added a Publish code coverage task and it's asking for something called a "Summary File" - see below:
I'm not sure how to set this option? Has anyone done this?
I seem to get test results published in the build results without config, see below:
And I'd like the same for the Code Coverage tab in the build results, which is currently blank:
Also (may be a completely separate question), I was wondering - can I add a "Quality Gate" to say if Coverage is below 80% then fail the build?
Thanks for any advice in advance - it's a case of knowing what I want, but not sure how to achieve it!
Hurrah!!!
There is a solution for it. :)
You have to install a an Add-on to your organization in VSTS (Azure DevOps).
The name of the addon is Build Quality Checks
Here is the link:
https://marketplace.visualstudio.com/items?itemName=mspremier.BuildQualityChecks
It works perfectly. And this is what you want
By default, the VsTest task will publish the Code Coverage result to Build Summary page directly. So we don't need Publish Code Coverage result task to publish the coverage result. Just make sure you have selected "Code Coverage enable" section in VsTest task.
>> Also (may be a completely separate question), I was wondering - can I add a "Quality Gate" to say if Coverage is below 80% then fail the build?
In current VSTS, we could not set "Quality Gate" to set the build failed if it doesn't match the configured % value. There has other communities also have this requirement and have submit a user voice. Please feel free to add your comments or vote it from below link:
https://visualstudio.uservoice.com/forums/330519-visual-studio-team-services/suggestions/3817520-fail-build-on-insufficient-code-coverage
I'm trying to get acquainted with test automation using Microsoft TFS Api.
I've created the program which runs my test set - it uses code similar to one described here, e.g.:
var testRun = _testPoint.Plan.CreateTestRun(false);
testRun.DateStarted = DateTime.Now;
// ...
testRun.Save();
I believe this forces them to start as soon as any of agents can run them, instead of being delayed to certain time. Am I wrong? Anyway, it works all right.
But I was told by my lead that the task should be started each time the new input files are copied to certain folder (on the network I think, or perhaps in TFS).
So I'm searching for a way which allow to trigger tests on some condition - but currently without any luck. Probably I miss proper keywords.
I only found something vaguely related here but it seems they say it is not possible in a proper way.
So are there any facilities in TFS / MTM, any ways or approaches to achieve my goal? Thanks in advance for any hints / links.
You would need to write a system service (or other) that uses the file system watcher. Then when the file changes you can run your code above.
There is no built in feature in TFS to watch a folder for changes.
Currently our Jenkins server only displays a history/graph for the overall number of passed/skipped/failed tests - I'm assuming that's the behavior out of the box.
If you select a single test, you'll get information for how long the test was failing (assuming it did fail).
However, we'd like to see is a history for that single test across the different builds to identify whether the test has been failing in the past (and when) even though it just passed. If you find a build where it failed, you could click on it, and investigate what might have caused the failure; if it passes again, you could check whether something actually fixed the test, or whether it was failing randomly all along.
Is this something that can be done somehow through the config, or do we need an additional plugin for this? If yes, which one?
Not sure if this makes much difference, but we're using Java (Maven) & TestNG (Surefire).
Both the TestNG plugin and the JUnit plugin will actually display history of the test results.
You just need to pick a given result and then:
For JUnit click on "History" on the left side, and
For TestNG click you will see the history in the graph above the result. You can just click on the bars in the bars to see the older results, and also if you click closer to the edge, the scope of the test results will adjust
The Test Results Analyzer plugin does the job for me. There appears to be other suitable plugins out there as well.
https://wiki.jenkins-ci.org/display/JENKINS/Test+Results+Analyzer+Plugin
Does the Static Code Analysis plugin help?
So my typical workflow is
I write a data driven test using TestNG in IntelliJ.
I supply hundreds of data items
Run the test and one or two of them fail
I see the list of passed/failed tests in the "Run" pane.
I would like the ability to just right click that "instance" of the test and run that test alone (with breakpoints). Currently IntelliJ does not seem to have that feature. I would have to right click the test and when I run, it runs the whole set of tests with hundreds of data points.
Is this possible?
TestNG supports this at the testng.xml level, where you can specify which indices of your data provider should be used. It's called "invocation-numbers" and you can see what it looks like by running a test with a data provider, failing some of its invocation numbers and looking at the testng-failed.xml that gets generated.
Back to your question: your IDE needs to support this feature in order to make it available in the UI, so I suggest you ask on the IDEA forums
The feature has been added as of Intellij 142.1217: https://youtrack.jetbrains.com/issue/IDEA-57906