Is there a way to display the test results on the build results page returned by Build.aspx page? Because right now you only get to see the total number of tests and the number of failed and passed tests as in the sample:
Result details for Any CPU/Debug 0 errors, 90 warnings, 12 tests total, 4 tests passed, 8 tests failed
Errors and Warnings: 0 errors, 90 warnings
Test Results: 1 test runs completed, 12 tests total, 4 passed, 8 failed
Test Run Run By Total Passed Failed
TFSBUILD#TFB 2008-01-15 15:23:42_Any CPU_Debug PROJECTA\TFSBUILD 12 4 8
Furthermore in the Visual Studio 2008 GUI the Test Run can be opened because it is a link point to the .trx file in the TestResults subfolder of the \Build\ folder but the Build.aspx page does not show this as a link. Better would be maybe if you would get to see the list of tests and for each whether it passed or not.
I am saying this because we use the Team Foundation Build Notification Tool from the TFS 2008 Power Tools and right now you see that a build failed or partially succeeded but you cannot see the test list.
It's been one of my constant frustrations with VSTS that, out of the box, test results are not visible to anyone who does not have the VS Test SKU. See this question on how to at least get .trx files into HTML.
As for the build summary itself, last I heard there is no way to customize this, nor are there plans to allow doing so in the future. One could do the transformation of the .trx test results file, and post the HTML results to a known location, though.
Related
I am using the TFS Web Portal to create test suites for automated testing and from Visual Studio test project, I can associate any [Test Method] to any TFS Test case. The tests can then be run from TFS.
As you can see below, my test suite has some setup tests then some actual tests. I have a Test Method that sets a folder where my test suite will then crop it's screenshots, I then Launch the application I am going to test and login. I then do a test, set the screenshot prefix to the requirement number 1581 (as that's the requirement tested) and I repeat this for requirement 2062 then 2061.
The problem is that these tests run in the wrong order.
I know about Ordered Tests and I know I could simply put all of this in an ordered test and run it. In fact I used ordered tests, myself and 1 other tester in the team has a Visual Studio installation and the skills to use ordered tests. The requirement from the rest of the team (manual testers and managers) is that I need to split my tests up (as shown below).
As I created the test suite below, I thought it all worked. I can run the first 5 tests. Inspired by this I added the last 4 tests in my 9 test suite. It all started to fail though!
Now when I run all 9 tests, the 4th test (that had worked fine) fails as it is somehow run in the wrong order by TFS. I know this because my tests all create 1 or more screenshots. These screenshots work but are being created in the wrong order.
I know a lot of people will tell me to use Ordered Tests and they will tell me that each test should be self contained. The reason I am deviating from that is that I have 4 thousand tests that verify 3 thousand requirements. If I want to test that a member record can have attributes X, Y and Z then that is 3 requirements. I want 1 setup test (launch the application, login, create a basic record). Then I want 1 test that X works, 1 test that Y works and 1 test that Z works. I have 3 requirements so I need at least 3 tests. If each test repeats all of the setup (Launch the app, login and create a basic record) then my test suite will take forever to run and will quickly become too slow.
I also want these to be split up because manual testers with no programming skills and no visual studio experience will be expected to look at my test suites, find the failed tests and re-test manually. If my tests suites all have 1 test in TFS, they will not know what went wrong. If I have 9 tests in a test suite and 1 fails, they will know what to manually re-test.
Finally, I have seen similar question relating to Microsoft Test Manager. This is not a duplicate question, MTM is not supported anymore so I'm using the Web Portal (as pictured). I have also seen this question relating to Visual Studio 2010 etc. I am using Visual Studio 2019 so it's not a duplicate, VS has changed a lot.
I understand that nunit console 3 can write to TestResult.xml after running the tests, where the TestResult.xml is located inside the directory specified by --work parameter.
But from what I can tell, TestResult.xml contains too many (irrelevant) details, that I don't need to fix my unit tests errors. All I need is just the failed or ignored test cases, just like what is displayed in command line prompt when I am running nunit console in it.
How to configure the parameters for nunit console 3 so that it only gives me failed or ignored test cases?
Sorry, I'm a few months late (unfortunately I came across this problem only now). But as far as I searched, the only possibility is to use XSL transformation. There are some applications that can convert the XML report file, but... I tested a few, but unfortunately the output was not that was I need.
Therefore I created a simple NUnit3summary application that can transform the standard XML output file to text file. I was surprised that nobody until now made something like this (or at least did not publish it). It were only two hours of work (first working version) and a few more to finish it to stare ready for publishing.
It is only a simple application that was aimed for my needs. You can use filtration for now only with another application, e.g.:
NUnit3summary.exe TestResult.xml | grep Failed >FailedTests.txt
You can see a practical application here (this is also the project where the application was needed, because of too many errors in unit tests).
I am working with an TFS 2017 environment with test agent 2015. Before this we had an TFS 2013 environment with test agent 2013 and MTM (this worked fine).
On the moment we have the following problem:
We run a set with around 40 tests, all of them have multiple iterations. if the first iteration fails we see this in tfs, the test status is set to failed this is perfect. However if the first iteration succeeds and the second fails the test case is set to passed in TFS. But if the second iteration fails we want the whole test to be set to failed. The way it is now it looks like almost all our tests pass however sometimes a lot of later iterations fail what means that we get false reporting.
When I open the .TRX file belonging to one machine I can see what iterations failed and which one succeeded.
so the problem in a nutshell:
if the first iteration of a test passes and the second one fails the whole test is set to Passed in stead of failed what gives us false reporting.
I have absolutely no idea what we are doing wrong. But now it gives is false information about our runs.
Is there anyone here that has experienced the same problem?
Any help would be really appreciated as I have not been able to find any information about this subject on google.
I have posted this on the Microsoft forum. They have answered that they can reproduce it what means it's probably an issue in tfs/testagent. More information can be found here:
https://social.msdn.microsoft.com/Forums/vstudio/en-US/4a384376-feae-46a9-a3da-e4445bc905d8/tfs-automated-tests-with-multiple-iterations-show-as-passed-even-when-the-second-iteration-fails?forum=tfsgeneral
I have many builds with failed tests.
I learn one test and want to find last build, where this test was succesfully.
How I can find this build or how I can get test results history?
If as a post-build step you are publishing your test results, you should be able to go to the job page and see the test results graph. If you are looking for the last build where all tests passed, you can click on "enlarge" under the graph. You will then see a high level view of all the builds from which you should be able to see the last build where all tests passed.
If you are looking for the last time a particular test passed, you can click on the latest build in the test results graph, drill down to the particular test you are interested in, and then click the previous build button until you find a build where that test passed.
If you aren't publishing your test results, you may be able to write some groovy code to scan through your build results, but beyond that I don't know of a way to find what your looking for without publishing test results in your post-build.
Is it possible to set number of test runs variable on TFS 2010 build so that is displayed in the build summary? We are running NUnit tests and can get the number of tests run. At the moment we see "No Test Results" in build summary.
and congratulations on your promotion from shoe-salesman to TFS technician... :)
You can add the NUnit test run results to the TFS build report in one of two ways:
1. You can write a simple XSLT to transform the nunit report to a TRX file (which TFS can read).
2. You can use NUnit for Team Build.
Either way, the NUnit results will be added to the report, including the count of successful, failed, inconclusive and (IIRC) total count of tests.
Hope this helps,
Assaf.