Pass parameters via the command line to NUnit - testing

Is it somehow possible to pass values to NUnit tests via the command line?
My tests use a certain URL. I have different instances of my code at different URLs and would like to specify the URL via the command line. File App.config is not an option, because I want to run the tests for different URLs via a batch file.

Use an environment variable to pass the information.
Use set from the command-line or <setenv> from NAnt. Then read the value using Environment.GetEnvironmentVariable().

NUnit 3 now allows passing parameters. Here is the usage
nunit3-console [inputfiles] --params:Key=Value
From the documentation
--params|p=PARAMETER
A test PARAMETER specified in the form NAME=VALUE for consumption by tests. Multiple parameters may be
specified, separated by semicolons or by repeating the --params option
multiple times. Case-sensitive.
Here's how you can access the parameter through code:
var value= TestContext.Parameters.Get("Key", "DefaultValue");

There seems to be no solution at the moment. The best option is to use NUnit project files, modify settings there and pass the solution file to the runner.

I had a similar issue. The answer of Achim put me on the right track, and for other readers:
Create a file, like example.nunit, like this:
<NUnitProject>
<Settings activeconfig="local"/>
<Config name="local" configfile="App.config">
<assembly path="bin\Debug\example.dll"/>
</Config>
<Config name="dev" configfile="App.Dev.config">
<assembly path="bin\Debug\\example.dll"/>
</Config>
<Config name="test" configfile="App.Test.config">
<assembly path="bin\Debug\\example.dll"/>
</Config>
</NUnitProject>
All the file / paths (of the configuration and assembly files) are relative to the location of the NUnit file. Also the App.config, App.Dev.config, etc. file are just .NET configuration files.
Next when you want to run it for a certain configuration you execute it like this:
nunit3-console.exe example.nunit /config:test
More information about the format of the NUnit file is in NUnit Project XML Format.
More information about command-line arguments is in
http://www.nunit.org/index.php?p=consoleCommandLine&r=2.2.5

Related

Exclude Selenium test methods in C# test project using VSTS

I have a Selenium test project which is checked into a VSTS repo and I am trying to create a scheduled build definition in VSTS to run a subset of the test functions contained in the built project assembly "SeleniumTest.dll".
As part of the build definition I have a VS Test task which looks like the following:
In an attempt to exclude all the test classes I do not want run, I have included the following runsettings file.
<?xml version="1.0" encoding="utf-8"?>
<!-- File name extension must be .runsettings -->
<RunSettings>
<DataCollectionRunSettings>
<DataCollectors>
<DataCollector friendlyName="Code Coverage" uri="datacollector://Microsoft/CodeCoverage/2.0" assemblyQualifiedName="Microsoft.VisualStudio.Coverage.DynamicCoverageDataCollector, Microsoft.VisualStudio.TraceCollector, Version=11.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a">
<Configuration>
<CodeCoverage>
<Functions>
<Exclude>
<!-- Exclude methods in the SeleniumTests.Tests\ZoneTest: -->
<Function>SeleniumTests.Tests\.LoginTest\..*</Function>
<Function>SeleniumTests.Tests\.ProfileTest\..*</Function>
</Exclude>
</Functions>
</CodeCoverage>
</Configuration>
</DataCollector>
</DataCollectors>
</DataCollectionRunSettings>
</RunSettings>
I'm not sure if a runsettings file is the thing I need to achieve this. If not what apporach should be used?
The Visual Studio task include Test Filter criteria setting that can filter tests.
Allowed operators:
• = implies an exact match
• != imples an exact not match
• ~ implies a contains lookup
More information: TestCase filter
BTW, the runsettings file you provided is used for CodeCoverage instead of filter test to run.

How to ignore the connection string parameter in SetParameter.xml

I am using msDeploy (3.0) to deploy my MVC applicaiton. I have a build pipeline that generates build artifacts for msdeploy and my deploy pipeline applies appropriate web.config transforms on the Web.config before deploying it to the production instances. One of the transforms includes changes to connection string. However, looking at the deployed instances, it seems like my web.config transforms are being overriden by the parameters in setParameters.xml in my build artifacts.
Ideal behavior would be that I would be to avoid adding any Connection string to the SetParameter.xml so that all my connection string overrides will be controlled by my deployment pipeline only. How do I achieve that?
Below is a sample of SetParameters.xml file
<parameters>
<setParameter name="IIS Web Application Name" value="Default/Foo"/>
<setParameter name="Foo-Web.config Connection String" value="Server=Foo,1433;Database=Bar;Integrated Security=SSPI;MultiSubnetFailover=True;App=Something;Connection Timeout=25"/>
</parameters>
Ideally it would look something like
<parameters>
<setParameter name="IIS Web Application Name" value="Default/Foo"/>
</parameters>
I have already tried passing a parameters.xml file to the msbuild step that does not contain the connectsion string parameter but that did not work
After bang my head against the wall for several hours, I finally figured out the solution. MsBuild takes in a parameter - p:AutoParameterizationWebConfigConnectionStrings=false that prevents the connection strings from being parameterized. Unfortunately, there is little or no documentation on this parameter.
You can also set this on a per project basis adding
<AutoParameterizationWebConfigConnectionStrings>false</AutoParameterizationWebConfigConnectionStrings>
to the PropertyGroup of your build configuration.

Use $(SolutionName) in the MsBuild commandline parameters

In order to emulate the "PerProject" option in TFS 2013's XAML build in the new Build 2015 task based builds, I'd like to be able to pass the SolutionName to the msbuild commandline arguments without having to manually set it every time.
I'd like to do something like:
/p:OutputPath=$(Build.BinariesDirectory)\$(SolutionName)\
Where I'd like MsBuild to infer the $(SolutionName) parameter. But when passing this on the commandline, the new task runner will substitute the $(Build.BinariesDirectory) with the correct target path and leaves $(SolutionName) alone. Unfortunately MsBuild subsequently also leaves the property alone:
Copying file from "obj\Debug\TFSBuild.exe" to "bin\debug\$(SolutionName)\TFSBuild.exe".
TFSBuild -> b\$(SolutionName)\TFSBuild.exe
Copying file from "obj\Debug\TFSBuild.pdb" to "b\$(SolutionName)\TFSBuild.pdb".
I can't remember a way to pass a property to the commandline and have it do late-expansion... Any tips?
For those looking to emulate SingleFolder or AsConfigured, those are easy:
SingleFolder -> /p:OutputPath="$(Build.BinariesDirectory)"
Asconfigured -> don't pass OutputPath
PerProject -> /p:OutputPath="$(Build.BinariesDirectory)\HARDCODESOLUTIONNAME"
As I feared there doesn't seem to be a simple way to override a property from the commandline and "inject" the value of another property into it during the evaluation stage.
There are a few ways to get around it, but they're not ideal and certainly not universal for each language supported by MsBuild. A pity.
I've debugged the MsBuild targets files and found a solution to reproduce the old behaviour from the 2005/2008 era. Not entirely per solution, but it does redirect projects into a subfolder.
/p:GenerateProjectSpecificOutputFolder=true /p:OutDirWasSpecified=true
/p:OutputPath=$(Build.BinariesDirectory)
Normally, $(SolutionName) is defined when executing solution-level MSBuild pipelines, such as running dotnet restore in the root solution directory.
To make $(SolutionName) available for project-level MSBuild pipelines, add a Directory.Build.props file in the root of your solution with the following contents:
<Project>
<PropertyGroup>
<SolutionName Condition="'$(SolutionName)' == ''">
$([System.IO.Path]::GetFileNameWithoutExtension($([System.IO.Directory]::GetFiles("$(MSBuildThisFileDirectory)", "*.sln")[0])))
</SolutionName>
</PropertyGroup>
</Project>
Now $(SolutionName) will be defined even when executing project-level MSBuild pipelines.
This answer works best when there is exactly one solution file in the root of the solution directory. You'll need to massage the above a bit for other project structures.
Of course, you can also be lazy and specify the solution name directly, but this opens up the possibility of refactoring issues (need to remember to update this file if the solution name changes).
<Project>
<PropertyGroup>
<SolutionName Condition="'$(SolutionName)' == ''">
MySolutionName
</SolutionName>
</PropertyGroup>
</Project>
One solution is to mimic such 'late evaluation' yourself by altering OutputPath withing the projectfile. To do without manually changing each single project file you can use the CustomBeforeMicrosoftCSharpTargets extension point. Which is an fancy way of saying it is just a property which when found and pointing to an existing file, will lead that file to be imported somewhere before all the actual build logic. Here's the idea: create a file like paths.targets somewhere - either include it in source control or you can generate it on the fly as part of the build process. Contents:
<?xml version="1.0" encoding="utf-8"?>
<Project ToolsVersion="12.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
<PropertyGroup>
<OutputPath Condition="'$(OutputPathBaseDir)'!=''">$(OutputPathBaseDir)\$(SolutionName)</OutputPath>
</PropertyGroup>
</Project>
So this just overrides OutputPath to some base dir + solutionname. Then if you build the solution like
msbuild my.sln /p:CustomBeforeMicrosoftCSharpTargets=paths.targets;
OutputPathBaseDir=$(Build.BinariesDirectory)
each project will import the paths.targets file and set output property to valueOfBinariesDirectory\my which I think is exactly what you are after.
You are right that TFS vNext build can't recognize $(SolutionName) in OutputPath, as $(SolutionName) doesn't list in the Predefined variables.
As an alternative, we may name the build definition with the solution name, then specify the MSBuild argument to: /p:OutputPath="$(Build.BinariesDirectory)\$(Build.DefinitionName)"in this way, we can get the output under the solution name.

TFS 2015 RC Build.Preview Execute xUnit Tests

We are trying to get xUnit tests to run properly using TFS 2015 RC. We're trying to access the test runner stored in our solutions NuGet packages folder which lives in our Build Agent's drop folder.
We are trying to reference the test runner using this path in the "Path to Custom Test Adapters" $(Build.SourcesDirectory)\WebIZ\packages\xunit.runners.1.9.2\tools.The build is able to resolve the path okay (it was failing before with an invalid path message), but now it cannot find the test runner
2015-06-09T20:05:40.4008595Z Executing the powershell script: D:\TFS\Build_vNext\AgentSchlitz\agent\tasks\VSTest\1.0.8\VSTest.ps1
2015-06-09T20:05:41.9947507Z Warning: The path 'D:\TFS\Build_vNext\AgentSchlitz\1c692895\WebIZ\WebIZ\packages\xunit.runners.1.9.2\tools\' specified in the 'TestAdapterPath' does not contain any test adapters, provide a valid path and try again.
extensions. Example: vstest.console.exe myTests.dll /UseVsixExtensions:true
What should the proper configuration be for the "Path to Custom Test Adapters"?
Should the proper configuration be:
$(Build.SourcesDirectory)\WebIZ\packages\xunit.runners.1.9.2\tools*
$(Build.SourcesDirectory)\WebIZ\packages\xunit.runners.1.9.2\tools*
I think you need to use XUnit 2.0 with the Nuget based runners.
There are full instructions in this blog post by Esteban Garcia that goes though the process step by step.
There's too much information and images for me to copy any one bit into this answer as the question is quite broad.
Are you sure that it isn't that you're simply pointing at the tools\ folder instead of the bin\ folder?
e.g.
Path to Custom Test Adapters:
"$(Build.SourcesDirectory)\WebIZ\packages\xunit.runners.1.9.2\bin"
Also, if you're copying that path from a blog post or other source that uses non-standard double quotes, you'll get the path is invalid error. Just retype the double quotes directly in the input box or whatever to make sure you have "raw" double quotes.
In our XUnit configuration, our "path to custom test adapters" is to the solution directory, NOT the xunit custom adapters, and it finds it fine. We actually submitted this pull request: https://github.com/Microsoft/vso-agent-tasks/pull/222 to get the path to custom test adapters treated like a route, and not like a string. Making the same change on your server might help. We have no other custom settings:
I had tried all combinations with the same issue, what fixed this for me was adding the entry to the packages.config for the test project...
<?xml version="1.0" encoding="utf-8"?>
<packages>
<package id="NUnit" version="3.5.0" targetFramework="net45" />
<package id="NUnit3TestAdapter" version="3.5.0" targetFramework="net45" />
</packages>
Even though I had referenced the Nunit3Adapter in the project and solution, the last entry there in the config needed to be manually added.

Mixing web.config transforms and Parameters.xml, or something to that effect

I've been using web.config transforms for a while now, for deployment on a few of our projects. What I'm now trying to achieve, is to have Web Deploy's 'Import Package' screen to prompt to check & update several of the variables in , adjusted for each environment.
I know I can use Parameters.xml to introduce these editable variables, but I haven't yet found how to have the defaults updated for different environment targets.
Consider the following neat, but non-overlapping example of wanting to have the user edit the 'specialServer' AppSetting, and have it present a different default when compiled for the NewEnv target:
Sample entry in Parameters.xml:
<parameter name="Special server" description="" tags="" defaultValue="server1-dev.domain">
<parameterEntry kind="XmlFile" scope="\\web.config$" match="/configuration/appSettings/add[#key='specialServer']/#value" />
</parameter>
Sample transform for Web.NewEnv.config, setting a different value for
<appSettings>
<add key="specialServer"
value="other-server.domain2"
xdt:Transform="SetAttributes" xdt:Locator="Match(key)"/>
</appSettings>
Sample of the prompt in Web Deploy:
Any suggestions as to how to update the default value for different build targets?
Thanks.
You would have to generate and new parameters definition file and embed it in your WebDeploy package for each environment.
This would give you different deploy packages per environment and would allow you to specify different default values for those parameters. Obviously doing so undermines the point if parameter transforms and you essentially end up baking in your config, but it's the only way to achieve what you want.
I don't recommend the approach but it may fit your needs.
We use a batch script to call msdeploy. It allows for a parm to specify the Parameters.xml file. Then with multiple Parameters.xml files (one per environment), you can just call msdeploy like:
"C:\Program Files\IIS\Microsoft Web Deploy V3\msdeploy.exe" -source:package='D:\mysite.zip' -dest:auto,computerName="testcomp1",includeAcls="False" -verb:sync -disableLink:AppPoolExtension -disableLink:ContentExtension -disableLink:CertificateExtension -setParamFile:"D:\mysite.test.SetParameters.xml"