For the past day, the 'New Test Plan' button has been missing from our VSTS UI inside the Test-Test Plans area. I also can't find anywhere else in VSTS giving the option to create a new test plan. (I'm sure there used to be a few places, but they were obvious, and now missing)
Could it be anything to do with star symbol showing in the UI after ' Test Plans* ', like some changes need saving somewhere before new test plans can be added? In which case, what changes should I be looking for to save?
Generally Test plan creation is limited to users subscribed to Test Manager extension or VS/MSDN subscribers.
Basic license users can only run tests. You need a valid Visual Studio
subscription (Enterprise, Test Professional or MSDN Platforms) or Test
Manager license ($52 monthly):
https://marketplace.visualstudio.com/items?itemName=ms.vss-testmanager-web
to create test plans.
See this thread : Can't add a new test plan
So please check if you have the license to create the test plan, and for an paid extension you must assign that extension to users who need access, so they can start using that extension's capabilities. Please see Assign paid extension access to users
Whatever, I can reproduce the issue on myside, seems it's an issue with the preview feature.
As a workaround you can create the Test Plan with MTM (Microsoft Test Manager) or REST API (Test Plans - Create). Both work for me, but the test plan will not display immediately after creating it, you may need to wait for several minutes to sync it.
Related
I am trying to formalize the development workflow and here's the first draft. Welcome suggestions on the process and any tweaks for optimization. I am pretty new when it comes to setting up the processes and would be great to have feedback on it. P.S: We are working on an AWS Serverless application.
Create an issue link in JIRA - is tested by. The link 'is tested by' has no relevance apart from correcting displaying the relation while viewing the story.
Create a new issue type in JIRA - Testcase. This issue type should have some custom fields to fully describe the test case.
For every user story, there will be a set of test cases that are linked to the user story using the Jira linking function. The test cases will be defined by the QA.
The integration/e2e test cases will be written in the same branch as the developer. E2E test cases will be written in a separate branch as it's a separate repository (Open for discussion).
The Test case issue type should also be associated with a workflow that moves from states New => Under Testing => Success/Failure
Additionally, we could consider adding capability in the CI system to automatically move the Test case to Success when the test case passes in the CI. (This should be possible by using JIRA API ). This is completely optional and we will most probably be doing it manually.
When all the Test cases related to a user story to success, The user story can then be moved to Done.
A few points to note:
We will also be using https://marketplace.atlassian.com/apps/1222843/aio-tests-test-management-for-jira for test management and linking.
The QA should be working on the feature branch from day 1 for adding the test cases. Working in the same branch will enable the QA and developer to be always in Sync. This should ensure that the developer is not blocked waiting for the test cases to be completed for the branch to be merged into development.
The feature branch will be reviewed when the pull request is created by the developer. This is to ensure that the review is not pending until the test cases have been developed/passed. This should help with quick feedback.
The focus here is on the "feature-oriented QA" process to ensure the develop branch is always release-ready and that only well-tested code is merged into the develop branch.
A couple of suggestions:
For your final status consider using Closed rather than Success/Failure. Success/Failures are outcomes rather than states. You may have other outcomes like cancelled or duplicate. You can use the Resolved field for the outcomes. You could also create a custom field for Success/Failure and decouple it from both the outcome and status. You ideally do not want your issue jumping back in forth in your workflow. If Failure is a status then you set yourself up for a lot of back and forth
You may also want to consider a status after New Test Creation for the writing of the test case and a status after that such as Ready for Testing. This would allow you to more specifically where the work is in the flow and also capture the amount of time that is spent writing tests, how long test cases wait, and how much time is spent actually executing tests and defect remediation
Consider adding a verification rule to your Story workflow that prevents a story from being closed until all the linked test cases are closed
AIO Tests for Jira, unlike other test management systems, does not clutter Jira, by creating tests as issues. So you need not create an issue type at all.
With it's zero setup time, you can simply start creating tests against stories. It has a workflow from Draft to Published (essentially equaling Ready for Testing).
The AIO Tests jira panel shows the cases associated with the stories and their last execution status to get a glimpse of the testing progress of the story as shown below. It allows everyone from the Product to the Developer to get a glimpse of the testing status.
You can also create testing tasks and get a glimpse of the entire execution cycle in the AIO Tests panel.
It also has a Jenkins plugin + REST APIs to make it part of your CI/CD process.
I'm noticing the asterisk next to Test Plans in my view on Visual Studio Team Services. It shows next to the the text in the same manner it shows for items without saved changes. I want to remove the asterisk.
I made sure existing Test Plans were Saved and Closed. Even closed and reopened VSTS. I made sure to all switches were off from the Preview feature menu as suggested by the similar question below.
"What are the asterisks on some Team Services menu items?"
Asterisk next to Test Plans in Team Services.
That’s because we use new Test Plans hub to manage VSTS test plans now.
Introducing the new hub for Test Plans! Not only can you add, edit,
and delete plans from this page but also find and favorite plans
across your team and the project. We pioneered this type of experience
in Dashboards, Queries, and Plans. This is another step to bring this
consistency across the product.
More details, you can refer the release note for Manage test plans using the new Test Plans hub.
Just wondering if there is a way to integrate TFS with TestRail to replace(get rid of entirely) the Test Hub within TFS to use TestRail to record Test Plans?
My concern with removing the Test Hub, would be if Test Rail can still reference IDs in Bug and Stories within TFS and vice versa?
You currently cannot remove the standard Hubs in Visual Studio Team Services / TFS or replace them with something else. You can enable an extension that either adds its own functionality under said hub as a separate tab or adds another Test Rail hub to the top level (if there is any) or write your own. Extensions currently cannot leave their sandbox to overwrite standard functionality.
There is nothing preventing anything from keeping track of work item numbers anywhere, so the second part of your question, whether it would have broken any form of integrations, that's unlikely.
If you are on TFS, you could try creating a custom process template that doesn't have the Test Case, Shared Step, Test Suite and Test Plan work item types, this will likely at least completely cripple the existing functionality. In the on-prem version you can also customize the files on disk, I've never tried, but it's likely that you could probably hack the test hub away. That would be a totally unsupported scenario through.
Is there a way to track test case exection & progress in Jira? I currently use the Test Management Workflow & provides me various phases like TestCaseReview, Pass, Fail, Invalid, ...
But is there a way to assocaite the test status to a specific run.
For instance, I would like to execute 100 tests with a specfic filter for Release 1.0
Execute the same set of tests for Release 2.0 as well & track the status.
This was fairly simple in Zephyr (provided us options to create test cycles and add tests). But sadly, the plugin is not available for OnDemand version of Jira.
So, I am looking to accomplish the same functionality with a vanilla version of Jira
Unfortunately I don't believe that what you want to do is possible in 'vanilla' JIRA.
The only possible way to do this in JIRA would be to create a 'swimlane' in JIRA for your release and then create individual JIRA tickets for each test (or given the large number of tests you have, small groups of tests) and then migrate the tickets through appropriate status values either as vertical swimlanes (or as sub-status) within a swim lane.
Sorry I can't suggest anything easier to manage.
I'm looking for a some sort of management/reporting tool that collects the results of tests, stores them for reporting, then lets users generate reports based on those tests.
We have numerous test running tools that run on a variety of platforms, but all output test results in the JUnit format. The test are not specific to hardware or platform, but rather generic. What we would like to do is have an automated (or manual) test run be able to submit it to a central location along with additional information, like platform, OS, hardware configuration and maybe user defined data. The management/reporting tool would store this data.
Then, a manager would be able to go to the tool and request (or more likely, access a dashboard that developers have setup) an update on the current status. This could be a list of test results that were run in a particular configuration, or a hardware status, or just the results of specific test(s).
Any suggestions?
We built a test management tool Enterprise Tester (www.enterprisetester.com) that allows users to pull automated test results in nUnit, nUnit, XSLT, Selenium etc and report off the results.
In addition to pulling the results and reporting you are able to trace these tests back to requirements that have been created giving you the ability to see test coverage and the status of this coverage on dashboards. If you are using JIRA (or google) or anything that uses open social gadgets you can pass these gadgets to other tools also.
Feel free to contact me directly if you would like to talk further about it
Regards
Bryce
You can also try an open source project called Allure Framework. It was created specially for showing test execution results in a nice form. A set of popular test frameworks such as JUnit, PHPUnit, TestNG, py.test, RSpec, Scalatest are already supported. Other ones such as NUnit will be supported soon.
Check out Hudson. it's very useful and configurable