Is there a way to track test case exection & progress in Jira? I currently use the Test Management Workflow & provides me various phases like TestCaseReview, Pass, Fail, Invalid, ...
But is there a way to assocaite the test status to a specific run.
For instance, I would like to execute 100 tests with a specfic filter for Release 1.0
Execute the same set of tests for Release 2.0 as well & track the status.
This was fairly simple in Zephyr (provided us options to create test cycles and add tests). But sadly, the plugin is not available for OnDemand version of Jira.
So, I am looking to accomplish the same functionality with a vanilla version of Jira
Unfortunately I don't believe that what you want to do is possible in 'vanilla' JIRA.
The only possible way to do this in JIRA would be to create a 'swimlane' in JIRA for your release and then create individual JIRA tickets for each test (or given the large number of tests you have, small groups of tests) and then migrate the tickets through appropriate status values either as vertical swimlanes (or as sub-status) within a swim lane.
Sorry I can't suggest anything easier to manage.
Related
I am trying to formalize the development workflow and here's the first draft. Welcome suggestions on the process and any tweaks for optimization. I am pretty new when it comes to setting up the processes and would be great to have feedback on it. P.S: We are working on an AWS Serverless application.
Create an issue link in JIRA - is tested by. The link 'is tested by' has no relevance apart from correcting displaying the relation while viewing the story.
Create a new issue type in JIRA - Testcase. This issue type should have some custom fields to fully describe the test case.
For every user story, there will be a set of test cases that are linked to the user story using the Jira linking function. The test cases will be defined by the QA.
The integration/e2e test cases will be written in the same branch as the developer. E2E test cases will be written in a separate branch as it's a separate repository (Open for discussion).
The Test case issue type should also be associated with a workflow that moves from states New => Under Testing => Success/Failure
Additionally, we could consider adding capability in the CI system to automatically move the Test case to Success when the test case passes in the CI. (This should be possible by using JIRA API ). This is completely optional and we will most probably be doing it manually.
When all the Test cases related to a user story to success, The user story can then be moved to Done.
A few points to note:
We will also be using https://marketplace.atlassian.com/apps/1222843/aio-tests-test-management-for-jira for test management and linking.
The QA should be working on the feature branch from day 1 for adding the test cases. Working in the same branch will enable the QA and developer to be always in Sync. This should ensure that the developer is not blocked waiting for the test cases to be completed for the branch to be merged into development.
The feature branch will be reviewed when the pull request is created by the developer. This is to ensure that the review is not pending until the test cases have been developed/passed. This should help with quick feedback.
The focus here is on the "feature-oriented QA" process to ensure the develop branch is always release-ready and that only well-tested code is merged into the develop branch.
A couple of suggestions:
For your final status consider using Closed rather than Success/Failure. Success/Failures are outcomes rather than states. You may have other outcomes like cancelled or duplicate. You can use the Resolved field for the outcomes. You could also create a custom field for Success/Failure and decouple it from both the outcome and status. You ideally do not want your issue jumping back in forth in your workflow. If Failure is a status then you set yourself up for a lot of back and forth
You may also want to consider a status after New Test Creation for the writing of the test case and a status after that such as Ready for Testing. This would allow you to more specifically where the work is in the flow and also capture the amount of time that is spent writing tests, how long test cases wait, and how much time is spent actually executing tests and defect remediation
Consider adding a verification rule to your Story workflow that prevents a story from being closed until all the linked test cases are closed
AIO Tests for Jira, unlike other test management systems, does not clutter Jira, by creating tests as issues. So you need not create an issue type at all.
With it's zero setup time, you can simply start creating tests against stories. It has a workflow from Draft to Published (essentially equaling Ready for Testing).
The AIO Tests jira panel shows the cases associated with the stories and their last execution status to get a glimpse of the testing progress of the story as shown below. It allows everyone from the Product to the Developer to get a glimpse of the testing status.
You can also create testing tasks and get a glimpse of the entire execution cycle in the AIO Tests panel.
It also has a Jenkins plugin + REST APIs to make it part of your CI/CD process.
Background->So i am using Zephyr for Test case management.I have a manual test suit with around 100 test cases .All test cases i have to execute on around 10 mobile devices.
I dont want to write same test cases for 10 devices as it will become huge(100X10),rather i was exploring if in any case i can use same zephyr test case for all 10 devices and get reports separate for 10 devices.The problem is execution will be done by multiple testers and keeping just 1 test case will also be a problem as all testers can not kep on executing the same test case . Any solution to overcome this situation?
With the caveat that my group is using the Zephyr for Jira plugin, the solution my group is experimenting with track release sign off across our platform matrix is:
Create test cases
Create a cycle to track release sign off
Create a folder under the cycle to represent one platform/device (think test run)
Add test cases to folder (think test results for this platform)
Clone the folder and it's contents, rename to next platform
Repeat the cloning until entire matrix covered
Testers go into each folder to run tests and log results
It looks like the reporting is broken out by folder in the Cycle Summary page in Jira. Which means it's possible to track progress by platform/device.
For the past day, the 'New Test Plan' button has been missing from our VSTS UI inside the Test-Test Plans area. I also can't find anywhere else in VSTS giving the option to create a new test plan. (I'm sure there used to be a few places, but they were obvious, and now missing)
Could it be anything to do with star symbol showing in the UI after ' Test Plans* ', like some changes need saving somewhere before new test plans can be added? In which case, what changes should I be looking for to save?
Generally Test plan creation is limited to users subscribed to Test Manager extension or VS/MSDN subscribers.
Basic license users can only run tests. You need a valid Visual Studio
subscription (Enterprise, Test Professional or MSDN Platforms) or Test
Manager license ($52 monthly):
https://marketplace.visualstudio.com/items?itemName=ms.vss-testmanager-web
to create test plans.
See this thread : Can't add a new test plan
So please check if you have the license to create the test plan, and for an paid extension you must assign that extension to users who need access, so they can start using that extension's capabilities. Please see Assign paid extension access to users
Whatever, I can reproduce the issue on myside, seems it's an issue with the preview feature.
As a workaround you can create the Test Plan with MTM (Microsoft Test Manager) or REST API (Test Plans - Create). Both work for me, but the test plan will not display immediately after creating it, you may need to wait for several minutes to sync it.
Just wondering if there is a way to integrate TFS with TestRail to replace(get rid of entirely) the Test Hub within TFS to use TestRail to record Test Plans?
My concern with removing the Test Hub, would be if Test Rail can still reference IDs in Bug and Stories within TFS and vice versa?
You currently cannot remove the standard Hubs in Visual Studio Team Services / TFS or replace them with something else. You can enable an extension that either adds its own functionality under said hub as a separate tab or adds another Test Rail hub to the top level (if there is any) or write your own. Extensions currently cannot leave their sandbox to overwrite standard functionality.
There is nothing preventing anything from keeping track of work item numbers anywhere, so the second part of your question, whether it would have broken any form of integrations, that's unlikely.
If you are on TFS, you could try creating a custom process template that doesn't have the Test Case, Shared Step, Test Suite and Test Plan work item types, this will likely at least completely cripple the existing functionality. In the on-prem version you can also customize the files on disk, I've never tried, but it's likely that you could probably hack the test hub away. That would be a totally unsupported scenario through.
I have a build job which takes a parameter (say which branch to build) that, when it completes triggers a testing job (actually several jobs) which does some stuff like download a bunch of test data and checks that the new version is works with the test data.
My problem is that I can't seem to figure out a way to show the test results in a sensible way. If I just use one testing job then the test results for "stable" and "dodgy-future-branch" get mixed up which isn't what I want and if I create a separate testing job for each branch that the build job understands it quickly becomes unmanageable because of combinatorial explosion (say 6 branches and 6 different types of testing mean I need 36 testing jobs and then when I want to make a change, say to save more builds, then I need to update all 36 by hand)
I've been looking at Job Generator Plugin and ez-templates in the hope that I might be able to create and manage just the templates for the testing jobs and have the actual jobs be created / updated on the fly. I can't shake the feeling that this is so hard because my basic model is wrong. Is it just that the separation of the building and testing jobs like this is not recommended or is there some other method to allow the filtering of test results for a job based on build parameters that I haven't found yet?
I would define a set of simple use cases:
Check in on development branch triggers build
Successful build triggers UpdateBuildPage
Successful build of development triggers IntegrationTest
Successful IntegrationTest triggers LoadTest
Successful IntegrationTest triggers UpdateTestPage
Successful LoadTest triggers UpdateTestPage
etc.
So especially I wouldn't look into all jenkins job results for overviews, but create a web page or something like that.
I wouldn't expect the full matrix of build/tests, and the combinations that are used will become clear from the use cases.