I'm trying to run automated tests from test plans in Azure devops.
I've added a task in my automated test pipeline to generate artifacts.
I've created a release of the test pipeline with the following tasks :
The problem is that when I run my test from my test plan, I have the following error saying that there is no test assemblies :
2022-09-22T15:27:08.6879981Z ##[error]ExecutionAndPublish.Run : Microsoft.VisualStudio.TestService.VstestAdapter.TestsNotFoundException: No test assemblies found on the test machine matching the source filter criteria or no tests discovered matching test filter criteria. Verify that test assemblies are present on the machine and test filter criteria is correct.
2022-09-22T15:27:08.6882575Z at Microsoft.VisualStudio.TestService.VstestAdapter.Execution.Run(ExecutionStateContext stateModelContext, CancellationToken cancellationToken)
2022-09-22T15:27:08.6883354Z at Microsoft.VisualStudio.TestService.VstestAdapter.ExecutionAndPublish.Run(ExecutionStateContext stateModelContext, CancellationToken cancellationToken)
2022-09-22T15:27:08.6885647Z ##[error]The slice of type 'Execution' is 'Aborted' because of the error : Microsoft.VisualStudio.TestService.VstestAdapter.TestsNotFoundException: No test assemblies found on the test machine matching the source filter criteria or no tests discovered matching test filter criteria. Verify that test assemblies are present on the machine and test filter criteria is correct.
Most Microsoft test Execution Engines including MSTest have a default pattern they use to discovery test Assemblies which is
**\$(BuildConfiguration)\*.test*.dll;-:**\obj\**
Since you are not showing the Files copied I'll have to assume that you might not have any dlls that match these patterns
To ovverride this behaviour you'll need to ensure you are using the version 2.x of the Test Execution Task and Update the Test Files Glob Patterns
Related
I have been searching for a while now and am surprised that I can't find any solutions out there for test result storage with grouping and searching capabilities.
I'd like a service or self hosted solution that supports:
storing test results in xunit/junit organized by keyword. In other words, I want to keep all my "test process A" test results together and all my "test process B" results together. I want to store failure traces and overall pass/fail at a minimum
get last run results for keyword: get the last "auth" test results with failure details
get run history results by keyword in some format
search of some sort on test results
I happen to be have:
Cypress tests
typescript/mocha tests without cypress
custom test framework tests that will need custom reporters
but I am fine with any test results solution that supports a generic input like xunit.
I am definitely open to suggestions that use any other storage system that can accomplish this even if it isn't strictly a test results tool.
I would like to merge an existing MTP (Master Test Plan) defined as a Test Suite with subfolders and TestCases, into another MTP (same structured), but located in a different project in TFS2017.
The idea is to have both project sharing the same MTP, so any change can be visible in both projects.
By now, I've just seen the existing possibilities about cloning/copying test cases between different test suites, but all of them should be part of the same project.
I guess there must be an easy way of doing that merge instead of manually, but I can't find the way to do it.
We can use an Existing Test Case (by reference) cross projects, but cannot share the same Test Plan cross projects.
If you wish to have a discrete copy of a test case, test cases, or test suites across projects (to a test plan that resides in a different project), you can perform a “clone” across projects via the command line (tcm.exe).
You could do this in MTM, could also use Tcm.exe command to copy test suites.
Please reference below articles :
Copying and cloning test suites and test cases.
Microsoft Test Manager – Working Across Projects: Adding an Existing
Test Case vs.Cloning/Copying
Actually, there has been a related uservoice for your requirement: Export test plan tree between projects and collections in MTM
We have enabled both Test Plan and Test Suite clone capabilities
within the product. See
https://msdn.microsoft.com/en-us/library/hh543843.aspx
This allows you to clone test plans/test suites across projects within
a collection. For moving artifacts (including test artifacts) across
collections, you can use the “TFS Integration Platform toolkit”.
Marking this item as “Completed”.
UPDATE:
To copy test cases, or test suites across projects, you can use “clone” via the command line (tcm.exe).
Below sample for your reference:
Run cmd
cd C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE
tcm suites /clone /collection:http://server:8080/tfs/CollectionLC /teamproject:0418Scrum /suiteid:271 /destinationteamproject:TFVC-Scrum /destinationsuiteid:274 /overridefield:"System.IterationPath"="TFVC-Scrum\Sprint 2" /overridefield:"System.AreaPath"=TFVC-Scrum
I am trying to create a unit test to run on two machines in Microsoft Test Manager 2010. In this test I want some client and server side test code to run simultaneously; the client side test being dependent on server side test working successfully.
When putting together a Test Suite in Test Manager, I want to be able to set both tests to have the same order value (so they run at the same time) but the validation prevents this; setting the order as shown below:
Is there any way I can achieve the simultaneous test execution I am after?
Sorry for the late answer... I've missed the notification about your answers to my question :-( Sorry for that!
In case you are still looking for solution, here my suggestion.
I suppose you have a test environment consisting of two machines (for server and client).
If so, you will not be able to run tests on both of them, or better to say you will not have enough control over running tests. Check How to Run automated tests on multiple computers at the same time
Actually I posted a related question to "Visual Studio Development Forum", you could check the answers I got here: Is it possible to run test on several virtual machines, which belong to the same environment, using build-deploy-test workflow
That all means you will end up creating two environments each consisting of one machine (one for server and one for client).
But then you will not be able to reference both environment in your build definition it you can only select one environment in DefaultLabTemplate.
That leads to the solution I can suggest:
Create two lab environments
Create three build definitions
the first one will only build your test code
the second one will deploy last successful build from the first one and start tests on the server environment
the third one will deploy last successful build from the first one and start tests on the client environment.
Run the first build definition automatically at night
Trigger the latter two simultaneously later.
It's not really nice, I know...
You will have to synchronize the build definition building the test code with the two build definitions running the tests.
I was thinking about setting up similar tests some months ago and it was the best solution I came up with...
Another option I have not tried yet could be:
Use a single test environment consisting of two machines and use different roles for them (server and client respectively).
In MTM create two Test Settings (one for the server role and one for the client role).
Create a bat file starting tests using tcm.exe tool (see How to: Run Automated Tests from the Command Line Using Tcm for more details).
You will need two tcm.exe calls, one for each Test Settings you have created.
Since a tcm.exe call just queues a test run an returns (more or less) immediately this bath file will start tests (more or less) simultaneously.
Create a build definition using DefaultLabTemplate.
This definition will:
build test code
deploy them to both machines in your environment
run your bath script as the last deployment step
(you will have to make sure this script is located on the build machine or deploy it there or make it accessible from the build machine)
As I've said, I have not tried it yet.
The disadvantage of this approach will be that you will not see the test part in the build log since the tests will not be started by means provided by DefaultLabTemplate. So the build will not fail when tests fail.
But you will still be able to see test outcomes in MTM and will have test results for each machine.
But depending on what is more important to you (having rest results or having build definition that fails if tests fail or having both) it could be a solution for you.
Yes, you can with modified TestSettings file.
http://blogs.msdn.com/b/vstsqualitytools/archive/2009/12/01/executing-unit-tests-in-parallel-on-a-multi-cpu-core-machine.aspx
In TFS 2012, we have several build definitions - CIs, Deployments and nightly.
Our CI builds run all of the (n)unit tests from our solution, however, we need to get it to ignore certain tests.
This is because we have some long running integration tests, and these only need to be run nightly.
Things I've tried:
Using the TestCategoryAttribute (from MSTest) and setting the Test Case Filter property try and exclude 'Integration'.
Using the CategoryAttribute (from NUnit) and setting the Test Case Filter property try and exclude 'Integration'.
A combination of the above.
The tests that need to be ignored are all in separate assemblies with the word IntegrationTests or Integration.Tests in the name.
Thanks,
Kieron
I've been using a combination of the MSTest TestCategory attribute on my unit tests, and the Test category filter setting for my build process definition in TFS 2012.
According to the Microsoft MSDN article found here you can specify which categories to use by setting the Test category filter to
TestCategory=CategoryName
According to your original post, you'd need to use the following filter:
TestCategory!=Integration
and decorate your tests with this attribute:
[TestCategory("Integration")]
Do this on all of your unit tests that you want ignored during your build. The test lists have been deprecated in Visual Studio, and it took a while to convert everything to the categories, but in the end it's worth it.
Hope that helps!
I have a multi module Gradle script with 3 modules: common, services and web.
If I run gradle check all the test suite is performed correctly, but I don't have a single test report instead I found a test report for each subproject. How can I merge the test results?
If useful this is the script I'm using https://github.com/CarloMicieli/trenako/blob/master/build.gradle
Thanks
Carlo
The Gradle source contains a Task class used to accomplish this, which unfortunately isn't available in the public API. Basically all it does is copy all of the test reports from various projects into one directory and then run the standard test report task to get an aggregate. Check out the source code at https://github.com/gradle/gradle/blob/master/buildSrc/src/main/groovy/org/gradle/build/TestReportAggregator.groovy