Our team is about to start a project using visual studio.net. We haven't decided about solution manager yet.
One of most important parts of our project is Testing. I have worked with TFS before and I know it's so powerful in testing (Link). I don't have any idea how Teamcity deals with different type of testing. May you please let me know about your experience and point of view?
Thanks
This question (CruiseControl [.Net] vs TeamCity for continuous integration? ) has a lot of good answers about TeamCity. It does not only talking about test and do the compare to TFS, but it might shed some light on TeamCity for you.
TeamCity directly supports the following testing frameworks:
JUnit and TestNG for the following runners:
Ant (when tests are run by the junit and testng tasks directly within the script, TeamCity reports tests on the fly)
Maven2 (when tests are run by Maven Surefire plugin or Maven Failsafe plugin, tests reporting occurs after each module test run
finish)
IntelliJ IDEA project (when run with appropriate IDEA run configurations)
NUnit for the following runners:
The NAnt (nunit2 task)
The MSBuild (NUnit community or NUnitTeamCity tasks)
Microsoft Visual Studio Solution runners (2003, 2005, 2008, 2010, 2012, 2013, and since Teamcity 9.1. Visual Studio 2015)
Any runner provided TeamCity Addin for NUnit is installed
MSTest 2005, 2008, 2010, 2012, 2013 and since Teamcity 9.1. MSTest 2015 (On-the-fly reporting is not available due to MSTest
limitations)
VSTest 2012, 2013, 2015
MSpec
You can even custom Testing Frameworks if there is no TeamCity support yet for your testing framework, more details please refer the official documentation from Teamcity: Testing Frameworks
For test in TFS. TFS itself won't shipped with Test Automation tools. TFS use Microsoft Test Manager, a separate Team Foundation client, lets you manage and execute test cases and create and manage physical or virtual environments. It installs with select Visual Studio editions.More info please see my answer in this Test Automation tools shipped with Visual Studio 2015 Enterprise?
As for which one is better, it's hard to say.It still depends on the actual situation of your team. There are many factors affecting this such as: staff's knowledge structure, financial budget, test environment, training of develop tools and so on.
Related
Environment is
1. Visual Studio 2017
2. Visual Studio Team server
In My requirement is, If i update "TEST SCRIPTS" code in Visual Studio Team server its will build automatically every release. But i need to to RUN the updated "TEST SCRIPTS" in Visual Studio Team server.
There are many tasks to run test, such as Visual Studio Test, Maven Test, Run Functional Test. So you can create a unit test/coded UI test project in VS 2017 and run the test through Visual Studio Test task.
Related article: Get started with continuous testing.
I recommend you run the test during the build.
Regarding run the test once the Test Script code be changed, you can setup a CI build (Triggers tab). If you want to do test in release, you need to publish artifact (test project build result), and configure release with Continuous Deployment mode (Triggers)
We use TFS 2015 vNext as our build system. However we use dotCover from JetBrains as our code coverage tool. The reason we don't use the built-in TFS code coverage tool is that this requires an Enterprise licence and we only have Professional licences (and upgrading is not an option).
Is it possible to display code coverage from a different code coverage tool to be displayed on the build results page in TFS 2015 (as in the screen shot below)?
Possible duplicate with this question Display OpenCover results in TFS 2015 build Summary .Your requirement could be achieved after TFS2015 update2.
If you want to display your custom result in the build result summary,
seems you will have to write your own extensions. Help link:
Overview of extensions for Visual Studio Team Services
Base for this were a lot of examples provided by MS on github:
Visual Studio Team Services Tasks
Visual Studio Team Services Sample Extensions
To get a first feeling what places on your TFS Web Portal can be
extended/customized you can download and install this extension
Contributions Guide from the Visual Studio Marketplace.
My team has several Selenium Unit Tests that we run locally. Selenium opens a browser and runs through our application. This works fine locally. Now, management wants to run the tests, on demand, via TFS. Should be easy, right? WRONG!
I am having trouble setting up our TFS 2015 server to run these Selenium UI tests.
Everything I read tells me to run the "Test Agent for Visual Studio 2015" tool. Problem is, the tool does not exist on my machine and I cannot understand why. I have tried many things, like installing the MS Visual Studio "Agents", with no luck.
When I try to open the freshly installed "Agents" I get a message saying: "Test Agent for Visual Studio 2015 has no configuration tool".
So the VS.NET "Agents" panel never opens. Seems bizarre to me that I installed something that needs further configuration, with no way to perform that configuration.
From further reading, I think I need to provision a group machine in the TFS web administration page, but am totally unsure if that's required for my situation, as I cannot even get past the basics here.
I see other people have had this problem, with very little response or help from Microsoft. I am a little baffled at why this seems so very difficult to do on a TFS server? It runs locally just fine, like a breeze. But TFS? It's like a giant puzzle.
Once this is working, I need to configure the test agent to run in "interactive mode" so it can run the browser but I cannot even begin to figure that out yet. Where do you set it to "run interactive" because I dont see any of those options. Am I missing TFS installation components? Do I have the wrong Visual Studio? Do I need the Ultimate edition of Visual Studio to be able to perform UI tests with the browser? We have the Professional edition.
Here is someone with the same problem:
https://connect.microsoft.com/VisualStudio/feedback/details/1712725/test-agent-for-visual-studio-2015-has-no-configuration-tool
Does anyone have any ideas or instructions on how to setup the "Test Agent" I need to run my team's existing Selenium UI tests in TFS2015? Seems very difficult when it should not be.
In the link you provided, Allen has explained that "There is no configuration UI with the test agent anymore. This is because we have simplified the existing remote testing scenario by doing the install and configuration for you when running via the build pipeline."
TFS 2015 Update2 now have a "Run Functional Test" task that you can use to run tests (included Coded UI Tests) against machine groups. So, first you have to upgrade your TFS 2015 to TFS 2015 Update2 if you haven't.
The tasks you need for test scenarios using Visual Studio 2015 and Visual Studio Team Services (VSTS) or Team Foundation Server (TFS) 2015:
Create environments from physical or virtual machines that you've already set up.
Set up your build to run your app and tests in the environments that you created.
After your build finishes, review your test results to start resolving problems that you found.
So, your build process template should look like the screenshot below. You need to specify every task to meet the requirement of your project. All TFS tasks can be found at this website, you can get more information for each task from it. Coded UI or Selenium tests that are running on full fidelity browsers would need Interactive Process checked.:
I haven't been able to find anything about this directly online. I am about to make an automated Chocolatey script for build agent configuration.
Is MsTest part of Microsoft Build Tools?
I figured out that it is not the case. Visual Studio 2015 is needed, or you can choose to install something Microsoft released separately, called Microsoft Test Agents 2015.
is it somehow possible to run tests with it?
I am always getting message:
(CoreTestConfiguration target) ->
MSBUILD : warning : Visual Studio Team System for Software Testers or Visual Studio Team System for Software Developers is required to run tests as part of a Team Build.
Well it is possible, we have such configuration (VS2010 project and TFS2008 where we store it). Probably when you set up the build on build server you have chosen to run the tests with this build - so please install proper Visual Studio version (Visual Studio Team System for Software Testers or Visual Studio Team System for Software Developers) on the Build Machine. Or setup the build without test option.