Environment is
1. Visual Studio 2017
2. Visual Studio Team server
In My requirement is, If i update "TEST SCRIPTS" code in Visual Studio Team server its will build automatically every release. But i need to to RUN the updated "TEST SCRIPTS" in Visual Studio Team server.
There are many tasks to run test, such as Visual Studio Test, Maven Test, Run Functional Test. So you can create a unit test/coded UI test project in VS 2017 and run the test through Visual Studio Test task.
Related article: Get started with continuous testing.
I recommend you run the test during the build.
Regarding run the test once the Test Script code be changed, you can setup a CI build (Triggers tab). If you want to do test in release, you need to publish artifact (test project build result), and configure release with Continuous Deployment mode (Triggers)
Related
I am a newbie in the DevOps world, and I would like to set up a CI server.
I have a Continuous Integration pipeline setup (with GO-CD pipelines) on my laptop. I am trying to move it to the server.
On the server, I do not have Visual Studio Tools so the existing MSBuild fails. Is there any way I can run the same pipeline on the server.
I have 2 projects in my MSBuild file.
Asp.Net website
MS Test Project
Regards,
Kritul
Build ASP.NET code on the Build Server without installing VS
Build ASP.NET 4.5 without Visual Studio on Build Server
Using MS Tests on Build Server without installing VS
Can I use mstest.exe without installing Visual Studio?
Our team is about to start a project using visual studio.net. We haven't decided about solution manager yet.
One of most important parts of our project is Testing. I have worked with TFS before and I know it's so powerful in testing (Link). I don't have any idea how Teamcity deals with different type of testing. May you please let me know about your experience and point of view?
Thanks
This question (CruiseControl [.Net] vs TeamCity for continuous integration? ) has a lot of good answers about TeamCity. It does not only talking about test and do the compare to TFS, but it might shed some light on TeamCity for you.
TeamCity directly supports the following testing frameworks:
JUnit and TestNG for the following runners:
Ant (when tests are run by the junit and testng tasks directly within the script, TeamCity reports tests on the fly)
Maven2 (when tests are run by Maven Surefire plugin or Maven Failsafe plugin, tests reporting occurs after each module test run
finish)
IntelliJ IDEA project (when run with appropriate IDEA run configurations)
NUnit for the following runners:
The NAnt (nunit2 task)
The MSBuild (NUnit community or NUnitTeamCity tasks)
Microsoft Visual Studio Solution runners (2003, 2005, 2008, 2010, 2012, 2013, and since Teamcity 9.1. Visual Studio 2015)
Any runner provided TeamCity Addin for NUnit is installed
MSTest 2005, 2008, 2010, 2012, 2013 and since Teamcity 9.1. MSTest 2015 (On-the-fly reporting is not available due to MSTest
limitations)
VSTest 2012, 2013, 2015
MSpec
You can even custom Testing Frameworks if there is no TeamCity support yet for your testing framework, more details please refer the official documentation from Teamcity: Testing Frameworks
For test in TFS. TFS itself won't shipped with Test Automation tools. TFS use Microsoft Test Manager, a separate Team Foundation client, lets you manage and execute test cases and create and manage physical or virtual environments. It installs with select Visual Studio editions.More info please see my answer in this Test Automation tools shipped with Visual Studio 2015 Enterprise?
As for which one is better, it's hard to say.It still depends on the actual situation of your team. There are many factors affecting this such as: staff's knowledge structure, financial budget, test environment, training of develop tools and so on.
After upgrading both Team Foundation Server and Visual Studio from 2013 update 4 to 2015, i created a build in the new build system.
The build includes only two steps: One Visual Studio Build step and one Visual Studio Test step. There's only one single simple solution with a class library project and a test project.
The Build step runs fine. The Test step does nothing. It just says
Starting test execution, please wait...
but the test are not running.
When i edit the VS Test step in the build definition and set the VSTest version to "Visual Studio 2013" instead of "Visual Studio 2015", everything works fine and it outputs as expected:
Starting test execution, please wait...
Passed <nameoftest1>
Passed <nameoftest2>
Passed <nameoftest3>
Passed <nameoftest4>
etc
I found out that during that build step, the following line gets executed:
C:\Program Files (x86)\Microsoft Visual Studio 12.0\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe "C:\pathToTestDll\TestProject.dll
So i tried to run this from commandline on my own pc, and found out that when i use vstest.console.exe from the Visual Studio 12.0 folder, everything works fine, but when i use vstest.console from the Visual Studio 14.0 folder, the tests are not running.
I also found out that when i run the command prompt as administrator, it runs all the tests with vstest.console from the Visual Studio 14.0 folder.
Has anybody also had this issue, or is there a way to run the whole build (or just the test step) with elevated rights, so i can run my tests with the VSTest Version set to Visual Studio 2015?
I have a mstest project that uses Microsoft fake assembly. The test methods are executing fine in my local Visual studio ultimate. I never check in the fake dll in to repository (SVN), assuming that it will be created in the build server (Jenkins). Later I understood that the build server is using MSBuild.exe to build the solution and since MSBuild.exe doesn't know how to create the fake assembly, the build seems failing. Installing the Visual studio ultimate in the build server is the only one option or is there any other ways to execute the unit test (like check in the fake dll in to repository).
You need Visual Studio Ultimate or Visual Studio Premium (if you are using a version later than Visual Studio 2012 Update 2) to be able to generate fakes. Also please make sure that your build script is using vstest.console.exe to execute tests. Fakes is not supported by MSTest.exe
is it somehow possible to run tests with it?
I am always getting message:
(CoreTestConfiguration target) ->
MSBUILD : warning : Visual Studio Team System for Software Testers or Visual Studio Team System for Software Developers is required to run tests as part of a Team Build.
Well it is possible, we have such configuration (VS2010 project and TFS2008 where we store it). Probably when you set up the build on build server you have chosen to run the tests with this build - so please install proper Visual Studio version (Visual Studio Team System for Software Testers or Visual Studio Team System for Software Developers) on the Build Machine. Or setup the build without test option.