I have a Selenium test project which is checked into a VSTS repo and I am trying to create a scheduled build definition in VSTS to run a subset of the test functions contained in the built project assembly "SeleniumTest.dll".
As part of the build definition I have a VS Test task which looks like the following:
In an attempt to exclude all the test classes I do not want run, I have included the following runsettings file.
<?xml version="1.0" encoding="utf-8"?>
<!-- File name extension must be .runsettings -->
<RunSettings>
<DataCollectionRunSettings>
<DataCollectors>
<DataCollector friendlyName="Code Coverage" uri="datacollector://Microsoft/CodeCoverage/2.0" assemblyQualifiedName="Microsoft.VisualStudio.Coverage.DynamicCoverageDataCollector, Microsoft.VisualStudio.TraceCollector, Version=11.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a">
<Configuration>
<CodeCoverage>
<Functions>
<Exclude>
<!-- Exclude methods in the SeleniumTests.Tests\ZoneTest: -->
<Function>SeleniumTests.Tests\.LoginTest\..*</Function>
<Function>SeleniumTests.Tests\.ProfileTest\..*</Function>
</Exclude>
</Functions>
</CodeCoverage>
</Configuration>
</DataCollector>
</DataCollectors>
</DataCollectionRunSettings>
</RunSettings>
I'm not sure if a runsettings file is the thing I need to achieve this. If not what apporach should be used?
The Visual Studio task include Test Filter criteria setting that can filter tests.
Allowed operators:
• = implies an exact match
• != imples an exact not match
• ~ implies a contains lookup
More information: TestCase filter
BTW, the runsettings file you provided is used for CodeCoverage instead of filter test to run.
Related
We are trying to get xUnit tests to run properly using TFS 2015 RC. We're trying to access the test runner stored in our solutions NuGet packages folder which lives in our Build Agent's drop folder.
We are trying to reference the test runner using this path in the "Path to Custom Test Adapters" $(Build.SourcesDirectory)\WebIZ\packages\xunit.runners.1.9.2\tools.The build is able to resolve the path okay (it was failing before with an invalid path message), but now it cannot find the test runner
2015-06-09T20:05:40.4008595Z Executing the powershell script: D:\TFS\Build_vNext\AgentSchlitz\agent\tasks\VSTest\1.0.8\VSTest.ps1
2015-06-09T20:05:41.9947507Z Warning: The path 'D:\TFS\Build_vNext\AgentSchlitz\1c692895\WebIZ\WebIZ\packages\xunit.runners.1.9.2\tools\' specified in the 'TestAdapterPath' does not contain any test adapters, provide a valid path and try again.
extensions. Example: vstest.console.exe myTests.dll /UseVsixExtensions:true
What should the proper configuration be for the "Path to Custom Test Adapters"?
Should the proper configuration be:
$(Build.SourcesDirectory)\WebIZ\packages\xunit.runners.1.9.2\tools*
$(Build.SourcesDirectory)\WebIZ\packages\xunit.runners.1.9.2\tools*
I think you need to use XUnit 2.0 with the Nuget based runners.
There are full instructions in this blog post by Esteban Garcia that goes though the process step by step.
There's too much information and images for me to copy any one bit into this answer as the question is quite broad.
Are you sure that it isn't that you're simply pointing at the tools\ folder instead of the bin\ folder?
e.g.
Path to Custom Test Adapters:
"$(Build.SourcesDirectory)\WebIZ\packages\xunit.runners.1.9.2\bin"
Also, if you're copying that path from a blog post or other source that uses non-standard double quotes, you'll get the path is invalid error. Just retype the double quotes directly in the input box or whatever to make sure you have "raw" double quotes.
In our XUnit configuration, our "path to custom test adapters" is to the solution directory, NOT the xunit custom adapters, and it finds it fine. We actually submitted this pull request: https://github.com/Microsoft/vso-agent-tasks/pull/222 to get the path to custom test adapters treated like a route, and not like a string. Making the same change on your server might help. We have no other custom settings:
I had tried all combinations with the same issue, what fixed this for me was adding the entry to the packages.config for the test project...
<?xml version="1.0" encoding="utf-8"?>
<packages>
<package id="NUnit" version="3.5.0" targetFramework="net45" />
<package id="NUnit3TestAdapter" version="3.5.0" targetFramework="net45" />
</packages>
Even though I had referenced the Nunit3Adapter in the project and solution, the last entry there in the config needed to be manually added.
In several projects’ App.config file, there is a line
<add key="url" value="http://www.example.com/"/>
Upon each build, I want to have a task to verify that the "url" key does not have the text "http://localhost". Is there a way to do this?
I'm assuming you have a team, and some of your team members inadvertently checkin those configs, changing that value to localhost.
If this is the case, why not have transform files for each environment, where your debug configuration can set the key to localhost, and your production/test/stage/qa/whatever configuration can set it to example.com or something else.
You might not have been aware that msbuild can transform your config files. Essentially you have your main config file, and then a config file containing just the things changed, for each environment. Upon doing a build, msbuild will modify the main one with whatever the changes are in the other "transform" files.
App.Config Transformation for projects which are not Web Projects in Visual Studio 2010?
Your transform file would look like:
<?xml version="1.0"?>
<!-- For more information on using web.config transformation visit http://go.microsoft.com/fwlink/?LinkId=125889 -->
<configuration xmlns:xdt="http://schemas.microsoft.com/XML-Document-Transform">
<appSettings>
<add key="url" value="http://www.example.com/" xdt:Locator="Match(key)" xdt:Transform="SetAttributes"/>
</appSettings>
</configuration>
The microsoft link is to http://msdn.microsoft.com/en-us/library/dd465326(VS.100).aspx
They can easily be used on web.configs, as well as app.configs, with a little tweaking to your project file.
An alternative solution would be to integrate unit tests as part of your build, and have the test verify the key in your web.config.
Go to your builds:
Right click on your build and click on edit build definition:
Choose process:
Now we can set fail on build here:
I'm trying to set up a C# project that'll run xUnit tests when I build, so I can use them in continuous integration. I have a regular project, a class library test project using xUnit, and my test runner project. From everything I've read, it appears that I should be able to get this working by doing this in the test runner project:
<?xml version="1.0" encoding="utf-8"?>
<Project ToolsVersion="4.0" DefaultTargets="Test"
xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
[auto-created project stuff]
<UsingTask AssemblyFile="xunit.runner.msbuild.dll"
TaskName="Xunit.Runner.MSBuild.xunit" />
<Target Name="Test">
<xunit Assembly="$(MSBuildProjectDirectory)\..\OnePageOneDb.Tests\bin\Debug\OnePageOneDb.Tests.dll" />
</Target>
</Project>
When I build my solution after a change (usually editing the .csproj file), I get this:
The "Xunit.Runner.MSBuild.xunit" task
could not be loaded from the assembly
C:\Users[myusername]\Code\OnePageOneDb\OnePageOneDb.TestRunner\xunit.runner.msbuild.dll.
Could not load file or assembly
'file:///C:\Users[myusername]\Code\OnePageOneDb\OnePageOneDb.TestRunner\xunit.runner.msbuild.dll'
or one of its dependencies. The system
cannot find the file specified.
Confirm that the
declaration is correct, that the
assembly and all its dependencies are
available, and that the task contains
a public class that implements
Microsoft.Build.Framework.ITask.
Even if I add xunit.runner.msbuild.dll and xunit.runner.utility.dll to the project in the location it refers to, I get this message. But if I build again with no changes, I consistently get this:
The "xunit" task was not found. Check
the following: 1.) The name of the
task in the project file is the same
as the name of the task class. 2.) The
task class is "public" and implements
the Microsoft.Build.Framework.ITask
interface. 3.) The task is correctly
declared with in the
project file, or in the *.tasks files
located in the
"C:\Windows\Microsoft.NET\Framework\v4.0.30319"
directory.
But I've checked all these things:
The task class in xunit.runner.msbuild.dll is Xunit.Runner.MSBuild.xunit (and xunit is lowercase in the class name).
The task class inherits from Task, which implements ITask.
So maybe there's a problem in UsingTask, but I don't know what it is.
(I also thought the problem might be that xunit.runner.msbuild.dll is targeted at .NET 2.0, and I'm using VS 2010, but I recreated the test runner project in .NET 2.0 and the problem persisted.)
Can anyone help?
You need to specify correct path to xunit.runner.msbuild.dll.
First of all, you can just set the full path and test that xunit just works as you want.
But for real environment you should specify relative path to the dll.
<UsingTask AssemblyFile="$(MSBuildProjectDirectory)\..\..\lib\xunit\xunit.runner.msbuild.dll"
TaskName="Xunit.Runner.MSBuild.xunit" />
MSBuildProjectDirectory is a reserved property and contains "the absolute path of the directory where the project file is located".
EDIT:
Try to use target by full name Xunit.Runner.MSBuild.xunit
<Target Name="Test">
<Xunit.Runner.MSBuild.xunit Assembly="$(MSBuildProjectDirectory)\..\OnePageOneDb.Tests\bin\Debug\OnePageOneDb.Tests.dll" />
</Target>
I get exactly the same error message if I have Pex and Moles installed. Everything works fine after uninstalling them.
By default building in "release" configuration triggers running xunit tests.
If you are trying to disable running xunit tests in tfsbuild pass the following build parameter.
This can be useful in the new cross platform builds where running unit tests is a separate step.
/p:RunXunitTests=false
Is it somehow possible to pass values to NUnit tests via the command line?
My tests use a certain URL. I have different instances of my code at different URLs and would like to specify the URL via the command line. File App.config is not an option, because I want to run the tests for different URLs via a batch file.
Use an environment variable to pass the information.
Use set from the command-line or <setenv> from NAnt. Then read the value using Environment.GetEnvironmentVariable().
NUnit 3 now allows passing parameters. Here is the usage
nunit3-console [inputfiles] --params:Key=Value
From the documentation
--params|p=PARAMETER
A test PARAMETER specified in the form NAME=VALUE for consumption by tests. Multiple parameters may be
specified, separated by semicolons or by repeating the --params option
multiple times. Case-sensitive.
Here's how you can access the parameter through code:
var value= TestContext.Parameters.Get("Key", "DefaultValue");
There seems to be no solution at the moment. The best option is to use NUnit project files, modify settings there and pass the solution file to the runner.
I had a similar issue. The answer of Achim put me on the right track, and for other readers:
Create a file, like example.nunit, like this:
<NUnitProject>
<Settings activeconfig="local"/>
<Config name="local" configfile="App.config">
<assembly path="bin\Debug\example.dll"/>
</Config>
<Config name="dev" configfile="App.Dev.config">
<assembly path="bin\Debug\\example.dll"/>
</Config>
<Config name="test" configfile="App.Test.config">
<assembly path="bin\Debug\\example.dll"/>
</Config>
</NUnitProject>
All the file / paths (of the configuration and assembly files) are relative to the location of the NUnit file. Also the App.config, App.Dev.config, etc. file are just .NET configuration files.
Next when you want to run it for a certain configuration you execute it like this:
nunit3-console.exe example.nunit /config:test
More information about the format of the NUnit file is in NUnit Project XML Format.
More information about command-line arguments is in
http://www.nunit.org/index.php?p=consoleCommandLine&r=2.2.5
One of my Eclipse plug-ins (OSGi bundles) is supposed to contain a directory (Database Elements) of .sql files. My build.properties shows:
bin.includes = META-INF/,\
.,\
Database Elements/
(...which looks right to me.)
When I build and run from within my interactive Eclipse IDE, everything works fine: calls to Bundle.getEntry(String) and Bundle.findEntries(String, String, bool) return valid URL objects; my tests are happy; my code is happy.
When I build via headless ant script (using PDE Build), those same calls end up returning null. My tests break; my code breaks. I find that Database Elements is quietly but simply missing from my plug-in's JAR package. (META-INF and the built classes still make it in there fine.) I scoured the build log (even eventually invoking ant -verbose on the relevant portion of the build script) but saw no mention of anything helpful.
What gives?
It appears there was a bug (though I was unable to search up a Bugzilla citation) in the PDE Build ant-script generation process as of 3.2 that produced an ant build.xml script fragment like this from the bin.includes:
<copy todir="${destination.temp.folder}/my_plugin" failonerror="true" overwrite="false">
<fileset dir="${basedir}" includes="META-INF/,Database Elements/" />
</copy>
The relevant Ant documentation says that includes contains a "comma- or space-separated list of patterns". Thus (since my directory name contains a space and was copied literally into the includes attribute value) I think the copy task was trying to include a file named Database and a directory named Elements/. Neither existed, so they were quietly ignored. I suspect the same problem would have bitten if I had a comma in my directory name, but I did not test this.
Since I use Eclipse 3.5 interactively, I decided to finally decouple my headless build's Eclipse instance from my target platform (which remains at 3.2 for the moment) and to update my headless PDE Build to 3.5 (by attempting to produce a minimal PDE Build configuration from my interactive instance's plug-ins). Now, the generated build.xml contains this instead:
<copy todir="${destination.temp.folder}/my_plugin" failonerror="true" overwrite="true">
<fileset dir="${basedir}">
<include name="META-INF/"/>
<include name="Database Elements/"/>
</fileset>
</copy>
The relevant Ant documentation this time indicates that the only special characters in an individual include are * and ?. Indeed, the bug seems to have been fixed sometime between 3.2 and 3.5: my 3.5-based headless PDE Build now produces a plugin that contains Database Elements; my tests are happy; my code is happy; I'm happy.