I am running our selenium based UnitTests on the WebUI thrue a different mechanism than the resular build steps in TFS 2015 (Update 1). Doing so generates a .trx file that I am trying to upload back into TFS so that the results are available in the build overview.
The build step running the UnitTests can fail the build if any of the tests fail, but then I would not have any test results visible in TFS. I can skip the failure by checking 'continue on error' in the step.
Next I can include a step 'Publish Test Results' that will pickup the generated trx file and results will be visible in the build.
But know the build doesn't fail!
Any idea's how to fail the build and have the testresults available?
I've checked:
vstest.console.exe (cannot publish testresults from a file)
MSTest (Enterprise/Premium license needed, we have MSDN Professional).
There has to be a better way. In my opinion, the build step 'Publish Test Results' should be able to check the file and fail the build?
I use "Command Line" task to call "ParallelTestRunner.exe" to run the test during the build.
With this, if the test is failed without "Continue on error" option checked, the build will stop at this step, "Publish Test Results" step will not run which looks the same as the behavior you get. And if the test is failed with "Continue on error" option checked, the test step is marked as "Passed" and "Publish Test Results" step will run to publish the test result. But my final build status is not "Success", it is "Build Partially succeeded".
However, if you want to fail the test step and still run the "Publish Test Results" step, you can uncheck the "Continue on error" option for test step and check "Always run" option for "Publish Test Results" step. This will make sure the "Publish Test Results" step always run even though previous steps are failed.
By the way, "Publish Test Results" step only publish the test result, it does not check the result. The build is passed/failed base on the execution status of all the build steps in the build definition.
Related
I have a TFS Release definition that runs several tests as part of the deployment:
When I check the "Rerun Failed Tests" Option in my "Visual Studio Test" step in:
My unit tests no longer run as part of my release. The release fails with the following error in the "Visual Studio Test" step:
2019-02-22T16:21:33.0983821Z ##[error]Error opening response file
'C:\TFS' 2019-02-22T16:21:33.1413767Z ##[error]The test source file
"C:\TFS Agent\ _work\r6\a\Agent_work_temp\ea14a091-36bd-11e9-99f5-51895cf3942e.txt" provided was not found.
I figured since the error started with the following statement:
Error opening response file 'C:\TFS'
That there is some sort of bug with the TFS test runner where the call to the task to execute the tests is not putting the path in quotes. I changed all my TFS build agents to using the following "type" of paths for all builds(I removed all spaces from the folder the TFS agents lived in):
C:\TFSAgent\ _work
And it seems to work now...
In the build definition in TFS 2015, I've got a Command Line step that runs the following command:
xunit.console.exe \PathToTests\Tests.dll -xml \PathToResultsFolder\Results.xml
During the build, I can see the tests are being discovered and executing and everything's looking good.
But if I don't check "Continue on error" in the Command Line step, after the tests run and the result XML file has been saved, the step fails with the following error:
Task CmdLine failed. This caused the job to fail. Look at the logs for the task for more details.
But there's actually no error or anything I can see. The tests have run and the XML file has saved properly and is able to published to TFS. And I don't see an error like this if I run the command from the build machine.
Any ideas?
I want to run a codedUI test case using MTM 2012
Here is what I already have so far:
I created codedUI test cases to test my window application. CodeedUI Test run passed successfully.
I linked the test case to associate Automation.
When trying to run an automated test from MTM, I am receiving the following warning message:
"Cannot create automated test run. A build associated with the active test plan is required to submit an automated test run"
When I open MTM>Testing Center>Properties Build Configurations I can see my build definition field is empty.
In MTM>Testing Center>Properties Build Configurations > "Build in use:" is set to None, I have not available build.
I need to select a build from TFS in order to run the CUIT automated test. How can I create a fake build in visual studio 2012 to run my automated test in MTM envioremrnt.
Thanks!
You need to create a build definition for your automation project and have TFS build that. Then you can associate the builds that definition creates to your test plan.
Is there any way to activate a test related log at our TFS build server?
If a tests fails, I can't find any information about it (besides the name of the test).
I checked:
The drop folder. There is the normal log, but this log ends after the compilation, so before the tests are checked
The compilation folder at the build server (C:\Builds\8...). There is a folder "Test Results" next to the "bin" and the "src" folder, but the "Test Results" folder is always empty.
The build definition. Couldn't find any option to activate a log regarding tests.
When i check the build in the web view, all I see is:
x/y test(s) passed, z failed, 0 inconclusive
No Code Coverage Results
Other Errors and Warnings
z+1 error(s), 0 warning(s)
Failed z1
Failed z2
...
Failed zn
Test Run Failed.
How can I find further information, why the tests are failing?
We are using:
TFS 2012 + a TFS 2012 build server
Rolling builds
Visual Studio 2013 + Resharper at our local machines
NUnit + MOQ
Rolling builds (no more than 30 minutes) build definition
We got rid of the problem after we upgraded to TFS 2015.
Now I just have to check the web interface, if a build is broken because of a unit test I click at the specific build to open the detail view of it. At the right side there is an area named Test result, where I can click at a link to get to the Test Run page. Under Attachments (1) I got a file of the type Visual Studio Test Results File, which shows me exactly what the problem was.
I am creating a TeamCity build configuration with a few steps. One of those steps is running tests using MSTest. Currently, my tests are failing (by design so I can test the build process), but the build steps after the step for running the tests happen, even though I can see the test process exits with code 1 (ie, something other than code 0) in the build log. It does mark the build as failed, but I'd prefer it if the steps in the build stopped once failing tests were detected. Is that possible and I'm just missing something in the configuration?
When you create build step there are select Execute step for execution policy. Where you should select Only if all previous steps were successful.
Here is it:
But in your case there are bug in the MsTest and NUnit build steps. Here is workaround. And here is related issue.