Unable to get Visual Studio Online to execute my nUnit tests - msbuild

I have a solution with a number of nUnit tests that do not appear to run on Visual Studio Online servers. Here's what I've done so far:
Created a separate folder (outside of my branch structure) that has nUnit test adapter DLLs
Build controller is configured with a path to custom assemblies to point to nUnit folder that has adapter DLLs
Build's test Source is configured as:
" - Run tests in test sources matching *test*.dll;*test*.appx,Target platform: 'X64'"
Build controller reports during the build
Run optional script after MSBuild
Run optional script before Test
Runner Run VS Test Runner
Run Continuous Deployment
No test results afterwards.
No test results
No code coverage results
What am I doing wrong? Do nUnit tests require special attributes to run properly on VSO? Am I missing any other configuration settings?
I've followed this link when configuring: http://www.visualstudio.com/get-started/hosted-build-controller-vs#supplemental_binaries
Edit:
Test settings expanded as requested
Appreciate any help!

Well, I never setup a TFS build before, let alone an online one, until now (TeamCity fan), so I copied every single property from the screenshots and the only way I could get it to pseudo-"pass" (fully green, with no No test found. Make sure that installed test discoverers & executors warnings under Other Errors and Warnings) is when it didn't find any *test*.dll assemblies to load to begin with, rather than no [Test] methods to execute. Did you check your MSBuild log to make sure your test project assemblies are copied and match the pattern?
_CopyFilesMarkedCopyLocal:
Copying file from "C:\a\src\Alertera-Scheduler\packages\NUnit.2.6.3\lib\nunit.framework.dll" to "C:\a\bin\nunit.framework.dll".
Copying file from "C:\a\src\Alertera-Scheduler\packages\NUnit.2.6.3\lib\nunit.framework.xml" to "C:\a\bin\nunit.framework.xml".
CopyFilesToOutputDirectory:
Copying file from "obj\Debug\Alertera-Scheduler.Tests.dll" to "C:\a\bin\Alertera-Scheduler.Tests.dll".
Alertera-Scheduler.Tests -> C:\a\bin\Alertera-Scheduler.Tests.dll
Copying file from "obj\Debug\Alertera-Scheduler.Tests.pdb" to "C:\a\bin\Alertera-Scheduler.Tests.pdb".
Done Building Project "C:\a\src\Alertera-Scheduler\Alertera-Scheduler.Tests\Alertera-Scheduler.Tests.csproj" (default targets).
Done Building Project "C:\a\src\Alertera-Scheduler\Alertera-Scheduler.sln" (default targets).
Could you expand your Test source settings as well?
If you click Open Drop Folder (Build details in VS) > Diagnostics (tab in VSO) what does it say under Run VS Test Runner?
Run VS Test Runner 00:00:00
There were no matches for the search pattern C:\a\bin\*test2*.dll
There were no matches for the search pattern C:\a\bin\*test*.appx

Related

VSTS Build and Selenium

I am attempting to use Selenium to do some UI testing for a .net web app i am working on.
Every build fails with ether no Tests to run in the new configuration or #[warning]DistributedTests: Error : Test run directory 'C:\temp\' does not contain any assemblies that match the specified criteria: '***test*.dll'. Verify that the test run directory contains the target assemblies and the source filter is correct. in the deprecated version.
the test runs correctly when run from Visual Studio
Based on the error message, the test location is wrong, you need to specify the correct test location that contains the test assembly files (the path need to be accessible from test machine you specified).
Check Test Drop Location value if you are using Run Functional Tests task.

VSTS Test Assemblies - No test assemblies found

I'm attempting to run some selenium tests in my solution that are in a project called SA.SEPA.Web.UI.Selenium as part of a build definition in VSTS, but it is failing to run the Visual Studio Test task with the error...
No test assemblies found matching the pattern: **\*Selenium*.dll.
Search folder is set to - $(System.DefaultWorkingDirectory)
Preceding tasks are a nuget restore and build solution
Can anyone help?
You are specifying $(System.DefaultWorkingDirectory) as the search folder. By default (unless overridden), this points to the Source folder on the Agent.
If you are instructing MSBuild (using the OutDir parameter) to output your assemblies in a specific location then you should use that location.
Edit: If this is an On-premises Agent, it should also be running in interactive mode (not service) if you want to execute any UI tests
I ran into this same error but it was a different cause, so I'm adding my solution here.
I was getting the "no test assemblies found" error when trying to run Selenium tests with VSTS as part of a build.
My problem turned out to be that the test assemblies were not checked into change control (git, through VS2017), and therefore were not part of the build. The folders that contain the test assemblies were ignored during my original check-in by default. Once I added those folders to my repository, the build could find the tests and run them.

TFS2013 MsTest, invalid switch

I've recently moved to TFS213, did an install on a new machine.
After building one of my projects the result showed an error on MsTest. It turned out that it had invalid switches (as in Error when running unit tests from MSBuild)
Invalid switch "/publish".
Invalid switch "/publishbuild".
Invalid switch "/teamproject".
Invalid switch "/platform".
Invalid switch "/flavor".
I have found various posts online about this and have done the following :
I have installed vs2013 professional on the build-machine, in various posts it's says to install ultimate but I don't have a license for ultimate.
The problem still persists, the MsTest.exe is started from the correct location (it's only installed in C:\Program Files (x86)\Microsoft Visual Studio 12.0\Common7\IDE, and the build-process does find the MsTest executable.
for now I've disabled testing in my build-definition but that's not the 'ideal' situation ;-).
Should I just remove the invalid 'switches' or is there some other solution
TIA,
Jurjen.
I had the same issue and solved it by modifying the properties of the "Run MSTest for Test Assemblies" step in the build.
Summary of fix: I changed the Flavor, Platform and Publish properties of the "Run MSTest for Test Assemblies" step to "" (empty string).
Details of fix: I did this in a custom template but I think you could also make the change in the default template.
Disabled tests (as you did) to make sure my build would work as a baseline.
Followed instructions here to create and use to a custom template:
http://msdn.microsoft.com/en-us/library/dd647551.aspx
Tested that the Custom Template worked by adding a "Write Build Message" step to the build (per advice in article above).
Searched for (drilled down) to the "Run MSTest for Test Assemblies" step in the build. This took a long time because it is deep. I am sure there is a faster way to find it but I am not sure how.
Set Flavor, Platform and Publish properties to "" (empty string). (See photo).
Committed template changes to TFS.
Re-ran the build and it worked.
When all tests pass I don't see test results on the build results screen.
When a test fails it causes the build to fail. I don't see the test failure in the build summary but it does show up when I click "View Log".
This isn't ideal, but at least I have the tests running and preventing a build from passing when a test fails.

MSBuild and MSTest in non-TFS Build tool with MSBuild Outdir

I've been charged with coming up with a workflow in a non-TFS build tool that will build a solution, run unit test projects in the solution, then package the final output of the build for deployment. I originally had constructed a workflow that used MSBuild with the Outdir parameter to get the publishable application for later storage. When I added unit test projects to the solution they work fine in Visual Studio (2010) but when I run my solution through MSBuild the MSTest step can't find the tests because the VSMDI file points to the BIN folders in the test projects, but the Outdir parameter is applied a the Solution level so the test DLLs are constructed in the Outdir directory instead of the BIN folder. If I remove Outdir from MSBuild the tests run, but I don't get a publishable application. This is a problem for web applications where we don't want to publish the code files and other artifacts that Visual Studio Publish would exclude.
Any ideas how to get MSTest to run the tests correctly based on the VSMDI file when MSBuild is run with Outdir? I thought about scripting an XML transform to change the paths in VSMDI to the Publish folder, but that seems like a hack.
Any suggestions are greatly appreciated.
You could abandon the vsmdi file all together and pass the dlls in a list to the commandline tool?
Otherwise id plum for the transform. The transform is not as bad hack as you think because at least you know the devs are running the same tests as your build and you'd have this problem as it occurs when you try to build, test and package the release version.

How do you debug a unit test in Xcode 3?

I followed Apple's instructions to set up Unit Testing in my project. I followed the directions for making them dependent, so the tests run with every build of my main project. This works, and when my tests pass, the application runs; when they don't, I get build errors on the lines of the unit tests that failed.
I would like, however, to be able to step through my application code when the tests are failing, but can't get Xcode (3.2.5) configured properly. The project is a Mac project, not iOS.
I tried the instructions here and here, but execution never stopped at the breakpoints I set, neither in the the unit test code or in my application code. After following the first set of instructions, the breakpoints I set turned yellow with blue outlines, and I don't know what that meant, either. What do I need to do to step through my tests?
Update
I found another page attempting to solve this problem (here) by adding arguments and environment variables to my main executable, but again, those didn't cause execution to stop at my breakpoints. I also noticed that my test's (copious) log statements aren't showing up in my Debugger Console, either.
I also found out that the yellow breakpoints mean the code there can't be found at runtime. These are in my test case class, so that definitely seems to explain why those aren't firing.
I usually have no problem debugging my OCTest tests with Xcode 3.2, including halting at breakpoints.
The basic idea is to tell gdb to launch otest with your bundle as argument. You will do that by adding /Developer/Tools/otest as a custom executable to your Xcode project, then your OCTest bundle name as the sole argument (choose "Edit Active Executable otest" in the Project menu, and in the second tab add a line with Foo.octest in the top box to debug the Foo test).
Now if you hit the debug button it will start debugging your test bundle right now, and will stop at the declared breakpoints (if you hit build and debug it might not start if the test does not pass). Also note that you will maybe have to set an environment variable to YES to disable garbage collection (in the bottom box of the same "Arguments" tab), otest will tell you exactly which if you need.
If you did everything above and are still unable do step into your test code, this might either be because it is compiled with debug symbol generation turned off - check your debug build settings, but most likely because the test code is not recompiled at all. I say most likely, since your logs are not showing in the console, NSLog should write in the Xcode console. Clean your build and bin folders manually, sometimes when you change a path or a name you end up loading obsolete code. You might also want to check that the file has not jumped out of the test target (is the failing test in the same not-logging file ?).
I have a variant of what Hiedi Utley posted at http://hiediutley.wordpress.com/2011/03/08/xcode3-debugging-ios-unit-tests/. What I did not like about it is the duplication of unit test bundle targets, one that contained a Run Script phase to execute the unit tests after building, and another that did not run the unit tests. I had noticed in the Get Info pane for the Run Script phase the toggle "Run script only when installing" and thought that that would be a way to toggle between running the unit tests in normal mode and running them in the debugger.
As per Hiedi's instructions, create a new Executable item, say LogicTestsGDB. Configure it like so:
General Tab:
Path: Developer/usr/bin/otest (No leading /)
Path Type: Relative to Current SDK
Set the working directory to: Build Products directory
Arguments Tab:
Arguments: Your UnitTest Bundle (eg. LogicTests.octest)
Variables to be set in the environment
DYLD_LIBRARY_PATH ... : ${BUILD_PRODUCTS_DIR}:${DYLD_LIBRARY_PATH}
DYLD_FRAMEWORK_PATH . : ${SDKROOT}/Developer/Library/Frameworks
DYLD_ROOT_PATH ...... : ${SDKROOT}
IPHONE_SIMULATOR_ROOT : ${SDKROOT}
OBJC_DISABLE_GC : YES
DYLD_NEW_LOCAL_SHARED_REGIONS : YES
DYLD_NO_FIX_PREBINDING : YES
CFFIXED_USER_HOME : ${HOME}/Library/Application Support/iPhone Simulator/
Done. Now to debug your unit tests,
Expose the bundle's build phases and double-click on the Run Script phase.
Check the Run script only when installing option
Set the Active Target to be the unit test bundle (eg. LogicTests.octest).
Set the Active Executable to be the new executable created (eg. LogicTestsGDB)
Click Build and Debug
To run normally, executing unit tests as part of the build phase of the application:
Expose the bundle's build phases and double-click on the Run Script phase.
Uncheck the Run script only when installing option
Set the Active Target to be the application being built
Set the Active Executable to be the application being built
To automate the steps above, I created a simple AppleScript script that toggles between the two states:
property kApplicationName : "MyApp" -- name of the normal application to build
property kUnitTestName : "LogicTests" -- name of the bundle target to debug
property kUnitTestRunner : "LogicTestGDB" -- name of the executable to use when debugging the unit test bundle
tell application "Xcode"
tell the active project document
set theTarget to first target whose name is kUnitTestName
set thePhase to first run script phase of theTarget
if name of active target is kApplicationName then
set active target to theTarget
set theExecutable to first executable whose name is kUnitTestRunner
set active executable to theExecutable
set run only when installing of thePhase to true
else
set theTarget to first target whose name is kApplicationName
set active target to theTarget
set theExecutable to first executable whose name is kApplicationName
set active executable to theExecutable
set run only when installing of thePhase to false
end if
return "Targeting " & (name of active executable)
end tell
end tell
I also tried the solutions in the links you provided with, but none worked for me, too. But then I found this one: http://hiediutley.wordpress.com/2011/03/08/xcode3-debugging-ios-unit-tests/ -- it works like a charm :) Fresh stuff, too.
develop your unit tests so they wrap a program which resides in a project which has a target, then just open the dependent project and work in it. this would likely be a project with a library and an executable which calls through the library (2 targets). then you can just open the project and debug, while your unit tests in the dependency would then call through this library. this is a more modular approach.
good luck
The only thing that ended up working for me was upgrading to Xcode 4. It integrates beautifully. There was a little bit of pain in moving over to it, but now that it's over with, the integration is great. I'm fully able to step through my tests and application code.