I have many builds with failed tests.
I learn one test and want to find last build, where this test was succesfully.
How I can find this build or how I can get test results history?
If as a post-build step you are publishing your test results, you should be able to go to the job page and see the test results graph. If you are looking for the last build where all tests passed, you can click on "enlarge" under the graph. You will then see a high level view of all the builds from which you should be able to see the last build where all tests passed.
If you are looking for the last time a particular test passed, you can click on the latest build in the test results graph, drill down to the particular test you are interested in, and then click the previous build button until you find a build where that test passed.
If you aren't publishing your test results, you may be able to write some groovy code to scan through your build results, but beyond that I don't know of a way to find what your looking for without publishing test results in your post-build.
Related
I have been searching for a while now and am surprised that I can't find any solutions out there for test result storage with grouping and searching capabilities.
I'd like a service or self hosted solution that supports:
storing test results in xunit/junit organized by keyword. In other words, I want to keep all my "test process A" test results together and all my "test process B" results together. I want to store failure traces and overall pass/fail at a minimum
get last run results for keyword: get the last "auth" test results with failure details
get run history results by keyword in some format
search of some sort on test results
I happen to be have:
Cypress tests
typescript/mocha tests without cypress
custom test framework tests that will need custom reporters
but I am fine with any test results solution that supports a generic input like xunit.
I am definitely open to suggestions that use any other storage system that can accomplish this even if it isn't strictly a test results tool.
How do we create a snapshot versions of the test results of a test set in HP ALM - Test LAB
I need this versioning to keep track of the past execution details.
Any suggestions on how to achieve this?
I do not believe you need to copy/archive/version anything to get the results you need. Consider this...
First Testing Effort
Follow these steps for the first time you are executing tests.
Create your test set
Add the tests that will be part of your effort.
All tests will be in the "No Run" status with no run history
Execute your tests (multiple times as necessary if they do not pass initially) until your effort is complete.
Second Testing Effort
The next time you need to run the tests, use the same test set.
Reset the test set. Right-click the test set in Test Lab and select 'Reset Test Set'. This step will reset the 'Status of all your previous test runs to 'No Run'.
Execute your tests (multiple times as necessary if they do not pass initially) until your effort is complete.
Comparison
At this time, each test has maintained a full run history. All the runs from the First Effort are still available along with those from the Second Effort. Open the Test Instance details of one of the tests in your test set, and then select the 'Runs' option. There you will be able to see each and every time that test instance was executed, no matter which testing effort it was.
Alternative
If this doesn't meet your needs, an alternative would be to leave the test set from your first testing effort alone when you are done. When your second effort beings, copy/paste your test set and perform all your new runs from that copy. When you copy/paste, it does not copy run history, so the new test set will be a blank slate. There is a COM-based API for doing all of this through code if necessary.
Currently our Jenkins server only displays a history/graph for the overall number of passed/skipped/failed tests - I'm assuming that's the behavior out of the box.
If you select a single test, you'll get information for how long the test was failing (assuming it did fail).
However, we'd like to see is a history for that single test across the different builds to identify whether the test has been failing in the past (and when) even though it just passed. If you find a build where it failed, you could click on it, and investigate what might have caused the failure; if it passes again, you could check whether something actually fixed the test, or whether it was failing randomly all along.
Is this something that can be done somehow through the config, or do we need an additional plugin for this? If yes, which one?
Not sure if this makes much difference, but we're using Java (Maven) & TestNG (Surefire).
Both the TestNG plugin and the JUnit plugin will actually display history of the test results.
You just need to pick a given result and then:
For JUnit click on "History" on the left side, and
For TestNG click you will see the history in the graph above the result. You can just click on the bars in the bars to see the older results, and also if you click closer to the edge, the scope of the test results will adjust
The Test Results Analyzer plugin does the job for me. There appears to be other suitable plugins out there as well.
https://wiki.jenkins-ci.org/display/JENKINS/Test+Results+Analyzer+Plugin
Does the Static Code Analysis plugin help?
So my typical workflow is
I write a data driven test using TestNG in IntelliJ.
I supply hundreds of data items
Run the test and one or two of them fail
I see the list of passed/failed tests in the "Run" pane.
I would like the ability to just right click that "instance" of the test and run that test alone (with breakpoints). Currently IntelliJ does not seem to have that feature. I would have to right click the test and when I run, it runs the whole set of tests with hundreds of data points.
Is this possible?
TestNG supports this at the testng.xml level, where you can specify which indices of your data provider should be used. It's called "invocation-numbers" and you can see what it looks like by running a test with a data provider, failing some of its invocation numbers and looking at the testng-failed.xml that gets generated.
Back to your question: your IDE needs to support this feature in order to make it available in the UI, so I suggest you ask on the IDEA forums
The feature has been added as of Intellij 142.1217: https://youtrack.jetbrains.com/issue/IDEA-57906
Sometimes the hudson build step fails and does not execute all my tests, which really screws up the test trend graph because it shows a 50% drop in the number of tests and then goes up again. Is there a way to exclude the failed builds? I tried to delete the whole failed build but that didn't help.
If you delete the "offending" build and then run another build I think you'll find the trend graph is rebuilt without the blip.
That said, I have to side with #DarkDust on this one. If your CI environment is prone to wobble then these stats can be useful for diagnostics.