I am using the TFS Web Portal to create test suites for automated testing and from Visual Studio test project, I can associate any [Test Method] to any TFS Test case. The tests can then be run from TFS.
As you can see below, my test suite has some setup tests then some actual tests. I have a Test Method that sets a folder where my test suite will then crop it's screenshots, I then Launch the application I am going to test and login. I then do a test, set the screenshot prefix to the requirement number 1581 (as that's the requirement tested) and I repeat this for requirement 2062 then 2061.
The problem is that these tests run in the wrong order.
I know about Ordered Tests and I know I could simply put all of this in an ordered test and run it. In fact I used ordered tests, myself and 1 other tester in the team has a Visual Studio installation and the skills to use ordered tests. The requirement from the rest of the team (manual testers and managers) is that I need to split my tests up (as shown below).
As I created the test suite below, I thought it all worked. I can run the first 5 tests. Inspired by this I added the last 4 tests in my 9 test suite. It all started to fail though!
Now when I run all 9 tests, the 4th test (that had worked fine) fails as it is somehow run in the wrong order by TFS. I know this because my tests all create 1 or more screenshots. These screenshots work but are being created in the wrong order.
I know a lot of people will tell me to use Ordered Tests and they will tell me that each test should be self contained. The reason I am deviating from that is that I have 4 thousand tests that verify 3 thousand requirements. If I want to test that a member record can have attributes X, Y and Z then that is 3 requirements. I want 1 setup test (launch the application, login, create a basic record). Then I want 1 test that X works, 1 test that Y works and 1 test that Z works. I have 3 requirements so I need at least 3 tests. If each test repeats all of the setup (Launch the app, login and create a basic record) then my test suite will take forever to run and will quickly become too slow.
I also want these to be split up because manual testers with no programming skills and no visual studio experience will be expected to look at my test suites, find the failed tests and re-test manually. If my tests suites all have 1 test in TFS, they will not know what went wrong. If I have 9 tests in a test suite and 1 fails, they will know what to manually re-test.
Finally, I have seen similar question relating to Microsoft Test Manager. This is not a duplicate question, MTM is not supported anymore so I'm using the Web Portal (as pictured). I have also seen this question relating to Visual Studio 2010 etc. I am using Visual Studio 2019 so it's not a duplicate, VS has changed a lot.
Related
We have some generic test cases in Azure DevOps that are included in multiple test suites. When you run a test case for a web app, the test runner window displays the test case ID and name but not the test suite name. Our client is finding this is leading to some confusion as to what is actually being tested. is there a way to display the test suite name as well as the test case name? I've just discovered that you can use parameters in a test case so I'm about to investigate that, but I think that it may apply only to steps and not the title.
For the record, I have decided to use configurations for this. It's more of a workaround than a solution, because it means creating lots of configurations and assigning them to test suites and also that configurations can't be used for other purposes.
I have manual test plan in Azure DevOps with tree of suites that correspond to different functions in my app. Let's say it looks like this:
Now, I need to have one place where I can review tests results from whole test plan ran for particular build. Like acceptance tests.
There's no way to run multiple suites in one run, I guess. Didn't find such possibility, though. Tests ran suite by suite produce multiple testruns, which is understandable.
What I want to achieve is one link to all test results for specific build which I can provide further to PM.
We are using Scrum and in last Sprint (Sprint2) I manually run some test cases with success. We just started new Sprint3 and I first created new test suite for Sprint3. Then I noticed there is also possibility to copy test plan and I tried to copy Test Suite 2 for Test Suite 3. After that I somehow had two test suites for Sprint3 and I deleted another of those.
Then I tried to check what was tested in Sprint2 but now those successfull test cases are not in Test Suite 2 anymore. Do you know if there is a way to return to previous state or somehow restore the tests that I already run.
Our testing is manual and Test Cases are linked to Product Backlog Items.
Unfortunately, according to this docs:
We only support permanent deletion of test artifacts such as test
plans, test suites, test cases, shared steps and shared parameters.
Deleted test artifacts won't appear in the recycle bin and cannot be
restored.
Thus the deleted test artifacts will be permanently deleted.
I am trying to create a unit test to run on two machines in Microsoft Test Manager 2010. In this test I want some client and server side test code to run simultaneously; the client side test being dependent on server side test working successfully.
When putting together a Test Suite in Test Manager, I want to be able to set both tests to have the same order value (so they run at the same time) but the validation prevents this; setting the order as shown below:
Is there any way I can achieve the simultaneous test execution I am after?
Sorry for the late answer... I've missed the notification about your answers to my question :-( Sorry for that!
In case you are still looking for solution, here my suggestion.
I suppose you have a test environment consisting of two machines (for server and client).
If so, you will not be able to run tests on both of them, or better to say you will not have enough control over running tests. Check How to Run automated tests on multiple computers at the same time
Actually I posted a related question to "Visual Studio Development Forum", you could check the answers I got here: Is it possible to run test on several virtual machines, which belong to the same environment, using build-deploy-test workflow
That all means you will end up creating two environments each consisting of one machine (one for server and one for client).
But then you will not be able to reference both environment in your build definition it you can only select one environment in DefaultLabTemplate.
That leads to the solution I can suggest:
Create two lab environments
Create three build definitions
the first one will only build your test code
the second one will deploy last successful build from the first one and start tests on the server environment
the third one will deploy last successful build from the first one and start tests on the client environment.
Run the first build definition automatically at night
Trigger the latter two simultaneously later.
It's not really nice, I know...
You will have to synchronize the build definition building the test code with the two build definitions running the tests.
I was thinking about setting up similar tests some months ago and it was the best solution I came up with...
Another option I have not tried yet could be:
Use a single test environment consisting of two machines and use different roles for them (server and client respectively).
In MTM create two Test Settings (one for the server role and one for the client role).
Create a bat file starting tests using tcm.exe tool (see How to: Run Automated Tests from the Command Line Using Tcm for more details).
You will need two tcm.exe calls, one for each Test Settings you have created.
Since a tcm.exe call just queues a test run an returns (more or less) immediately this bath file will start tests (more or less) simultaneously.
Create a build definition using DefaultLabTemplate.
This definition will:
build test code
deploy them to both machines in your environment
run your bath script as the last deployment step
(you will have to make sure this script is located on the build machine or deploy it there or make it accessible from the build machine)
As I've said, I have not tried it yet.
The disadvantage of this approach will be that you will not see the test part in the build log since the tests will not be started by means provided by DefaultLabTemplate. So the build will not fail when tests fail.
But you will still be able to see test outcomes in MTM and will have test results for each machine.
But depending on what is more important to you (having rest results or having build definition that fails if tests fail or having both) it could be a solution for you.
Yes, you can with modified TestSettings file.
http://blogs.msdn.com/b/vstsqualitytools/archive/2009/12/01/executing-unit-tests-in-parallel-on-a-multi-cpu-core-machine.aspx
Is there a way to display the test results on the build results page returned by Build.aspx page? Because right now you only get to see the total number of tests and the number of failed and passed tests as in the sample:
Result details for Any CPU/Debug 0 errors, 90 warnings, 12 tests total, 4 tests passed, 8 tests failed
Errors and Warnings: 0 errors, 90 warnings
Test Results: 1 test runs completed, 12 tests total, 4 passed, 8 failed
Test Run Run By Total Passed Failed
TFSBUILD#TFB 2008-01-15 15:23:42_Any CPU_Debug PROJECTA\TFSBUILD 12 4 8
Furthermore in the Visual Studio 2008 GUI the Test Run can be opened because it is a link point to the .trx file in the TestResults subfolder of the \Build\ folder but the Build.aspx page does not show this as a link. Better would be maybe if you would get to see the list of tests and for each whether it passed or not.
I am saying this because we use the Team Foundation Build Notification Tool from the TFS 2008 Power Tools and right now you see that a build failed or partially succeeded but you cannot see the test list.
It's been one of my constant frustrations with VSTS that, out of the box, test results are not visible to anyone who does not have the VS Test SKU. See this question on how to at least get .trx files into HTML.
As for the build summary itself, last I heard there is no way to customize this, nor are there plans to allow doing so in the future. One could do the transformation of the .trx test results file, and post the HTML results to a known location, though.