I am currently working with Microsoft Visual Studio 2012 and the new Fakes framework. And after adding my unit tests to the MSBuild and continuous integration process, the Fakes unit tests happen to raise a ShimNotSupportedException.
I tried different stuff and read into that issue and found out that, this only happens when I run my tests from the Resharper feature, or directly in the code with MSTest. These exceptions DO NOT occur when running from the Test menu bar, which made me think about checking the setting files of this features.
After further research I found some articles where it says that it might has something to do with the code coverage setting specified in the Resharper or MSTest settings and if you switch that off, it solves the issue. ShimNotSupportedException in MS VisualStudio 2012 also relates to that problem.
I also found some input on the MSDN about that issue, which confirmed me that it has to do something with the code coverage: http://social.msdn.microsoft.com/Forums/en-US/vstest/thread/17fcfdc6-1cda-4692-a242-656b48195327/ and http://social.msdn.microsoft.com/Forums/en-US/vstest/thread/bfa792b0-b3fc-4a51-b49d-f7aaf1f2f4b8.
Nevertheless, since I have to implement code coverage into the continuous integration process somehow soon, and skipping that part is not an option, I am now asking here for any advice regarding this issue, and if there is any kind of solution to that problem!
Thanks for all advice!
Fakes is not supported by MsTest: http://msdn.microsoft.com/en-us/library/ms253138.aspx.
Here's how you can run the Visual Studio Testrunner from Team City (which supports Fakes): http://blog.degree.no/2012/09/unit-testing-visual-studio-2012-fakes-in-team-city/
You can also use the Visual Studio Testrunner if you use TFS.
Related
I recently switched all of my unit tests over from xunit to mstest, and on my local dev machine everything works great, but when I try to run my tests using a CI build in VSTS, it doesn't recognize any tests in my test dll, stating:
"Warning: No test is available in
c:\vsts_work\4\s\test\UnitTests\bin\Release\net452\UnitTests.dll".
I believe I have all of the important project.json bits in place:
"testRunner": "mstest",
"dependencies": {
"dotnet-test-mstest": "1.0.1-preview",
"MSTest.TestFramework": "1.0.1-preview"
}
Of note, I am using an on-premise build agent, but I've proven I can log into that build agent using the service account under which the agent runs, and I can open visual studio and run the tests that way.
Also, while one could contrive it by looking at the above path, it is worth explicitly noting that I'm using the full framework.
Hoping someone out there has already conquered this one.
I figured out how to use the "Visual Studio Test" action, thanks to this stackoverflow post
The only thing that answer lacks is how to make it work in VSTS (the answer provides the command line). To get it to work in VSTS, I put the following string in the "Test Assembly" parameter
**\test\**\project.json
And then put the rest of the command line arguments under "Advanced Execution Options" in the "Other console options" box.
/UseVsixExtensions:true /logger:trx
Hope this helps the next guy who is trying to stick with the visual studio tooling.
To run a .NET Core test during the build, you can add a "Command Line" task to run "dotnet test" command and set the "Working Folder" to the path where the "project.json" file placed:
I had a similar problem, getting the "Warning: No test is available" issue when I was looking right at the tests in Visual Studio.
It turns out the tests were trying to run in x86 when the application is 64 bit.
I recommend going to the Test menu at the top of Visual Studio, and open
Test>Test Settings>Default Processor Architecture>
and make sure the tests are on the right architecture, if you have a similar problem and the other answers aren't helping.
Any way to replace visual studio's usage of msbuild and still get error reporting in the IDE? I think I might want to compile in VS with NAnt or Rake or PowerShell, or something that is just as smart but uses a scripting language instead of XML. My build is fairly straight forward -- find all the .cs and compile them and put the dll in a Debug/ or Release/ etc.
I'd like to add a number of other steps that just seem easier from the command line:
Like hit the server to prime certain caches.
Compile with Closure
Minify CSS
Generate some metrics after build (in a certain config)
Run NUnit (or custom) testing framework and send an email
etc....
However, I'd still like to get the "error on line" feed back inside of the IDE. Is this a possibility?
You could write some extension for Visual Studio (Macro, AddIn, Package, etc.) that can augment Visual Studio to let other build tools run. The simplest form is probably just using "Tools\External Tools.." and add your Build Tool there. As long as you format your "build messages" as described here, the should show up the output window as if they were generated by MSBuild.
However, I don't think it is possible (nor desirable, BTW) to completely replace MSBuild in Visual Studio.
Visual Studio not just executes MSBuild.exe and parses it's output. The integration is very tight and a lot of GUI aspects rely directly on the content of the MSBuild file.
For example, when you change some project properties in the Visual Studio UI, the changes are (eventually) written to the MSBuild file of the project.
Also, there are some performance improvements, for example Visual Studio "replaces", so to say, the call of csc.exe (the C# compiler) from an MSBuild file by using an in-process compiler, which safes some compile time, because less external processes need to be launched (also described in the above mentioned document)
While all this, from a architectural point of view, is quit likely wrapped and encapsulated by some "interfaces" inside Visual Studio, I haven't yet found a way to have those "interfaces" implement something else that, for example, uses NAnt behind.
Anyway, I think even if technically possible, it would not be technically feasible.
For rake, check out Rake Runner extension. You can run the rake tasks from the solution explorer and check the errors and other output in the Output pane. I have little experience developing vs packages so if anyone want to help, the project is open sourced here.
I am trying to convert a build system setup with TeamCity and Nant scripts to use TFS2010 (We bought the license and might just as well make use of it) After some work I get the web project to build and deploy to the web-server. We have a domain, API, test and web project in our solution.
How do I configure TFS to run the unit tests that we have written so far? I did configure the build to look for ***.UnitTest.dll in(VS2010) Edit build definition>Process>Automated Tests
Now the build fails with a message that says:"Could not load file or assembly 'nunit.framework, Version=2.5.3.9345" Am I correct when I say that TFS is trying to run NUnit on the build server? I did install NUnit-2.5.3.9345 on that TFS2010 build server and still nothing?
Thank you
Jack
The build facility in TFS uses MSTest as test runner, with which it's tightly integrated.
If you want to run your unit tests with NUnit as part of your build, take a look at the NUnit for Team Build project on CodePlex.
The project started out for TFS 2008, however support for TFS 2010 has been added in version 2.0. Note that this feature is still in early stages of development, so your mileage may vary.
I'm late to the game, because I've had to deal with this issue recently. I found this article helpful for me in this. It didn't work right off the bat, but I found if I added it into my buildscript via the controls in a similar manner/pattern, it would work.
My only problem now has been getting it to actually error (right now it warns) even when flagging them to cause the build to error
Link: http://blog.gfader.com/2011/06/running-nunit-tests-in-tfs-2010.html
During .NET v1 days, I have tried without much success to convince colleagues to develop test-driven and automated-build work habits using the additional tools of NUnit and NAnt. When .NET Framework 2.0 and Visual Studio 2005 Team Suite came into picture, I was able to "force" my team into writing tests and provide themselves with visual testing right inside Visual Studio. I was further able to tweak project files with extra MSBuild tasks to carry out more build automation.
Of course this does not mean Microsoft has delivered perfect systems, but I believe they have taken these in the correct steps forward. With all these features baked right into the framework and products and becoming "native" it got abit easier to motion developers into better development practices.
Having long forgotten the open-source options (which I do miss), I am wondering what value proposition do the current incarnations of NUnit and NAnt hold? What case can one argue at this stage to convince a team not to use MSBuild or MSTest?
Clarification:
My company is a pure Microsoft SI. Visual Studio Team Suite editions, Database Professional edition, TFS, and the like are available for our use. We do not use Visual Studio Professional edition or lesser.
MSBuild is a pretty good build tool. I've used it in combination with NAnt and Cruisecontrol.Net a couple of times. NAnt together with the NAnt contrib extensions seems a bit more flexible in handling OSS tools like NUnit, NCover etc while the fact that MSBuild can build VS solutions is a big plus for it. I found the easiest way to create build scripts is to use both, NAnt to call the different parts of the build (build, fxcop, test, coverage etc) and MSBuild to do the actual building.
I haven't got much good to say about MSTest. It's slow and cumbersome to install on for example a build server. It's not very flexible and has all kinds of 'extras' that seem more geared to integration testing than unit testing. I found light weight XUnit.Net, MBUnit or NUnit to be far better suited for unit testing. Speed is important here, you want to run the tests often for quick feedback on the effects of your changes in code. Portability is important too. You want to have the tests run everywhere without a lot of setup and hacking like you need for MSTest on a machine that doesn't have team system installed. Although Unit testing isn't new there's still a lot of development going on there. Best practices change a lot and so do the tools. I don't want to be tied to a tool that only gets upgraded every few years with visual studio. Every one of the three OSS .net unit testing tools is set up to be extensible. MSTest isn't (as far as i've seen).
In my view NUnit is the de facto standard for unit tests in .NET. It's free and you can easily write tests in any edition of Visual Studio. If MS had made MS Test available in VS 2005 Pro (as they have in VS 2008 Pro), it might have taken over by now - but I think it's too late.
I view ReSharper as basically essential for Visual Studio, and that includes a great test runner which works very well with NUnit. If I'm developing an open source project, why would I want to require VS 2008 Pro or VS 2005 Team Suite?
MSBuild vs NAnt is slightly different, as MSBuild is bundled with either the framework or the SDK (I can't remember which now) - but I think NAnt is a more pleasant environment to work in. MSBuild is clearly better for doing the raw "build the solution" bit - but you can invoke it from NAnt.
Can anyone explain what advantages there are to using a tool like MSBuild (or NAnt) to build a collection of projects versus running DevEnv.exe from the command-line?
A colleague I had worked with in the past had explained that (at least with older versions of Visual Studio) using DevEnv.exe was much slower than the other techniques, but I haven't read any evidence of that or if that is now a moot point now that starting with 2005, Visual Studio uses MSBuild under the hood.
I know one advantage of using MSBuild allows you to build your projects without requiring Visual Studio to be installed on the build machines, but I wasn't sure if there were others.
One reason is because there's much more to building a product than just compiling it. Tasks such as creating installs, updating version numbers, creating escrows, distributing the final packages, etc. can be much easier because of what these tools (and their extensions) provide.
While you could do all this with regular scripts, using NAnt or MSBuild give you a solid framework for doing all this. There's a lot of community support for both, including additional tasks that can be downloaded (such as the MSBuild Community Tasks Project). Plus, there's support for them in numerous third party and open source products.
If you're just interested in compiling (and not the entire build process), you may find one time saving benefit of MSBuild is the support for building with multiple processors.
The obvious answer from my team is that not everbody has visual studio installed, in particular we do not install Visual Studio onto our build/CI servers.
The prime reason for using an external build tool like NAnt or MsBuild is the ability to automate your build process and thus provide continous feedback on the status of your system. Also they can be used for loads of things besides a "pure" build and that's where you really start to get value from them, it's an extremly valuable thing to be able to build and test your application with a single command.
You can also start adding stuff like collection of metrics, packinging of release binaries and all sorts of nifty stuff like that.
As far as C# goes, devenv.exe 2005 runs the compiler in-proc, which may cause out of memory exceptions for sizable solutions. Msbuild resorts to launching csc.exe process for each project. Projects that don't compile with devenv /build work fine with msbuild. Hope you like this reason.
We are experimenting with switching from DevEnv to a tool (Visual Build Pro) that uses MsBuild under the hood and we got a "Reference required to assembly 'System.Drawing..." error for a project which doesn't need it and which builds fine in Visual Studio.
We have a large system consisting of C#, managed C++, and plain old unmanaged C++ assemblies/dlls. There is C++ code that depends on managed C++ code that depends of C# code that depends of managed C++ code that depends on plain old C++ code (whew!). When we were setting up our automated build environment a few years ago we discovered that MSBuild.exe didn't properly handle all of the dependencies that we have.
Working with Microsoft we were able to solve some of the issues but not all of them. If my memory serves me, we never could get the C# assemblies that depended on managed C++ dlls to build. So we ended up making a custom build script that called devenv.exe from the command line and it worked just fine.
Of course, that was with VS2005, it might be fixed now, but the script is still working so we haven't revisited the issue.