I have been searching for a while now and am surprised that I can't find any solutions out there for test result storage with grouping and searching capabilities.
I'd like a service or self hosted solution that supports:
storing test results in xunit/junit organized by keyword. In other words, I want to keep all my "test process A" test results together and all my "test process B" results together. I want to store failure traces and overall pass/fail at a minimum
get last run results for keyword: get the last "auth" test results with failure details
get run history results by keyword in some format
search of some sort on test results
I happen to be have:
Cypress tests
typescript/mocha tests without cypress
custom test framework tests that will need custom reporters
but I am fine with any test results solution that supports a generic input like xunit.
I am definitely open to suggestions that use any other storage system that can accomplish this even if it isn't strictly a test results tool.
Related
I have manual test plan in Azure DevOps with tree of suites that correspond to different functions in my app. Let's say it looks like this:
Now, I need to have one place where I can review tests results from whole test plan ran for particular build. Like acceptance tests.
There's no way to run multiple suites in one run, I guess. Didn't find such possibility, though. Tests ran suite by suite produce multiple testruns, which is understandable.
What I want to achieve is one link to all test results for specific build which I can provide further to PM.
I have many builds with failed tests.
I learn one test and want to find last build, where this test was succesfully.
How I can find this build or how I can get test results history?
If as a post-build step you are publishing your test results, you should be able to go to the job page and see the test results graph. If you are looking for the last build where all tests passed, you can click on "enlarge" under the graph. You will then see a high level view of all the builds from which you should be able to see the last build where all tests passed.
If you are looking for the last time a particular test passed, you can click on the latest build in the test results graph, drill down to the particular test you are interested in, and then click the previous build button until you find a build where that test passed.
If you aren't publishing your test results, you may be able to write some groovy code to scan through your build results, but beyond that I don't know of a way to find what your looking for without publishing test results in your post-build.
I have a build job which takes a parameter (say which branch to build) that, when it completes triggers a testing job (actually several jobs) which does some stuff like download a bunch of test data and checks that the new version is works with the test data.
My problem is that I can't seem to figure out a way to show the test results in a sensible way. If I just use one testing job then the test results for "stable" and "dodgy-future-branch" get mixed up which isn't what I want and if I create a separate testing job for each branch that the build job understands it quickly becomes unmanageable because of combinatorial explosion (say 6 branches and 6 different types of testing mean I need 36 testing jobs and then when I want to make a change, say to save more builds, then I need to update all 36 by hand)
I've been looking at Job Generator Plugin and ez-templates in the hope that I might be able to create and manage just the templates for the testing jobs and have the actual jobs be created / updated on the fly. I can't shake the feeling that this is so hard because my basic model is wrong. Is it just that the separation of the building and testing jobs like this is not recommended or is there some other method to allow the filtering of test results for a job based on build parameters that I haven't found yet?
I would define a set of simple use cases:
Check in on development branch triggers build
Successful build triggers UpdateBuildPage
Successful build of development triggers IntegrationTest
Successful IntegrationTest triggers LoadTest
Successful IntegrationTest triggers UpdateTestPage
Successful LoadTest triggers UpdateTestPage
etc.
So especially I wouldn't look into all jenkins job results for overviews, but create a web page or something like that.
I wouldn't expect the full matrix of build/tests, and the combinations that are used will become clear from the use cases.
By many I mean hundreds/thousands. I need to test features that many users will need to see/hear. Obviously these users have different permission levels and some are in different programs. Can a test case be written to pull userids and passwords from the db to test in this way efficiently? Or is this something that is best manually tested by spot checking different log ons?
Call the DB before you run the test to get your users/passwords/whatever else.
Are you using NUnit? If so, you could use the Nunit ValueSourceAttribute to get the data into your test and use a variable for the credentials during your login step.
Currently our Jenkins server only displays a history/graph for the overall number of passed/skipped/failed tests - I'm assuming that's the behavior out of the box.
If you select a single test, you'll get information for how long the test was failing (assuming it did fail).
However, we'd like to see is a history for that single test across the different builds to identify whether the test has been failing in the past (and when) even though it just passed. If you find a build where it failed, you could click on it, and investigate what might have caused the failure; if it passes again, you could check whether something actually fixed the test, or whether it was failing randomly all along.
Is this something that can be done somehow through the config, or do we need an additional plugin for this? If yes, which one?
Not sure if this makes much difference, but we're using Java (Maven) & TestNG (Surefire).
Both the TestNG plugin and the JUnit plugin will actually display history of the test results.
You just need to pick a given result and then:
For JUnit click on "History" on the left side, and
For TestNG click you will see the history in the graph above the result. You can just click on the bars in the bars to see the older results, and also if you click closer to the edge, the scope of the test results will adjust
The Test Results Analyzer plugin does the job for me. There appears to be other suitable plugins out there as well.
https://wiki.jenkins-ci.org/display/JENKINS/Test+Results+Analyzer+Plugin
Does the Static Code Analysis plugin help?