In order to automate unit tests on TeamCity I had to create a test list in my vsmdi configuration file indicating that every test is part of a list I called CompleteCoverage. I dislike this a lot because in order to auto-run new tests I'll have to remember to include them on this list.
Is there some way to run every test in the solution using TeamCity and MSBuild (other than explicitly referencing the path to the output test assembly)?
Should I just drop MSTest and go for NUnit?
I'm using NUnit instead of MSTest, but this should work for you, too:
I've named all my test assemblies to include .NUnit in their name, e.g. Basic.NUnit.dll. In the build step performing the tests, I've declared **/*.NUnit.dll as the assemblies to run. To make sure that they are run in the right location, I've added **/obj/**/*.NUnit.dll to the exclude list. Together with test categories to in- or exclude, I've got perfect control over which tests to run on a purely declarative level without to name the individual tests.
Related
My colleagues and I have user specific settings in csproj.user files. They are not checked into the repository. I would like for the build server to use its own set of csproj.user files, overriding certain properties, leaving the "base" project configuration at a decent developer default. But from the looks of it there is no such option in the msbuild command-line for doing that.
Is there really no way, other than copy csproj.user-files to where it'll be picked up by subsequent msbuild invocations?
While writing I realize I'm too much of a prude about these things and should just copy as a step prior build. Still posting in case someone knows a better way, for instance a way that does not modify the source tree.
Passing properties to the MSBuild command line overrides properties in the solution, including dependent projects. Here omitting debug information in build server, otherwise generated for release build to improve profiling:
msbuild MySolution.sln /p:DebugType=none ...
This does not work should I want different properties for different projects. Building projects individually should work nicely though.
Finally, passing arguments on command line can get messy, so to get a more "settings file"-like experience one may instead use #file arguments and MSBuild response files.
I have tried many option but any of them not work as i need. I have created some feature file as i required time by time. so i have a disorder feature file.
I want to run the feature file as i wanted. like.
fil1.feature
fil2.feature
fil3.feature
fil4.feature
so i want to run in this sequence : file3.feature->fil4.feature->fil1.feature.
I have tried #tag, #feature in junit test runner option but its maintain the sequence its run only 3,4 but can't run 1.
So can you tell me how to run feature file randomly???
Cucumber picks up the feature files in alphabetic order from the folder given in the features parameter of the CucumberOptions. So one options would be to rename your feature files alphabetically in the order you want.
After the feature files in the initial folder are read, then the sub folders in the location are picked up alphabetically and the feature files inside them are read. So you can place the file you want to be used later into a sub-folder.
Saying all this, it is not a very good idea to have any dependency between tests which requires a sequence to be maintained.
I got the same stop. Have you solved this problem?
I did some test, use --tags #XXX to controll the sequence is useless, it's just action on different scenarios in a feature file.
so far, I have to do it like this cucumber features/c.feature features/a.feature features/b.feature I think this is a little bit better than rename feature files.
But you may say that: "If there is a lot of feature files......" In my project(ROR), I assigned the statement to ./config/cucumber.yml
In cucumber.yml I define a profile, test_dev: features/c.feature features/a.feature features/b.feature after that, I just to use cucumber -p test_dev and the feature files execute in order.
If you have a butter way, please share with us.
I have a CMake file with two targets (say, target1 and target2) defined via "add_custom_target" at the top level of my source directory. I have some external projects in lower level directory. In the extneral project, a "TEST_COMMAND ${Test_Command_Variable}" is defined as part of an ExternalProject_Add().
I would like to change that vaiarble ${Test_Command_Variable} depending on whether I am using custom target1 or target2. Currently, ${Test_Command_Variable} is defined at the top level CMakeLists.txt before either custom targets are defined. I simply want to vary that variable depending on whether target1 or target2 is called. Is there any way to redefine that variable? Maybe do a conditional if statement dependent on whether target1 or target 2 is chosen (this seems like a trivial thing, but I can't find the way to access the "name" of the custom target!).
To clarify: I have two collections of tests. I want to type "make target1" and it runs my first collection of tests. I want to type "make target2" and it tests my second collection of tests. The problem is I also have an external project, where some of the tests are there. The external projects have a TEST_COMMAND(test_command_variable) that does not differentiate between target1 tests and target2 tests. I would like to be able to change that variable depending on whether I run "make target1" or "make target2".
Well, I got it working finally. I did something completely different to meet my objective. I abandoned the goal of trying to toggle variables depending on the target. To be clear, here was my objective:
"make target1" runs only tests in Group 1.
"make target2" runs only tests in Group 2.
I configured each of the tests as per as per this question I asked earlier. My problem was that I had external projects I was trying to pass that variable down so it would toggle Group1 or Group 2 tests depending on the configuration defined at the top level. I knew passing ctest -C group1_configuration flag (or group2_configuration flag for group2 tests)would work, but I couldn't seem to pass down to my external projects. Fortunately, my solution was "simple".
I wrote a shell script that would execute the correct ctest ... -C group1_configuration (or ctest ... -C group2_configuration) and called it test_script1.sh and test_script2.sh. These scripts would move through the build directory and call the aforementioned ctest command in each of the external projects build directories. Then, in my custom target's I defined in the CMakeLists.txt I have it call the shell script and each custom target executes the right shell script, which move through the build directories calling the correct ctest command with the correct configuration flag in each project and external project.
I'm trying to prevent certain (MSTest) unit tests from being run on our build server. I'd really like to just add a TestCategory, and then specify:
/category:"!RequiresLoginCredentials"
But I'm not sure how to indicate that in the msbuild project file.
The relevant section of the build file currently has:
<ItemGroup>
<!-- TEST ARGUMENTS
If the RunTest property is set to true then the following test arguments will be used to run
tests. Tests can be run by specifying one or more test lists and/or one or more test containers.
To run tests using test lists, add MetaDataFile items and associated TestLists here. Paths can
be server paths or local paths, but server paths relative to the location of this file are highly
recommended:
<MetaDataFile Include="$(BuildProjectFolderPath)/HelloWorld/HelloWorld.vsmdi">
<TestList>BVT1;BVT2</TestList>
</MetaDataFile>
To run tests using test containers, add TestContainer items here:
<TestContainer Include="$(OutDir)\HelloWorldTests.dll" />
<TestContainer Include="$(SolutionRoot)\TestProject\WebTest1.webtest" />
<TestContainer Include="$(SolutionRoot)\TestProject\LoadTest1.loadtest" />
Use %2a instead of * and %3f instead of ? to prevent expansion before test assemblies are built
-->
<TestContainer Include="$(OutDir)\UnitTests.dll" />
</ItemGroup>
I'm guessing this is a simple addition, but I know very little about msbuild.
Thanks!
I did a quick search for the answer and I think there are two possible solutions:
From what you described, looks like you're trying to run the tests using the TestToolTask task of MSBuild. Unfortunately, I don't think you can pass MSTest arguments directly to this task. To accomplish what you want, you need to specify the tests you want to run in a test list, and pass the test list to this task. You need to use the MetadataFile property as shown in the example in your post.
You can invoke MSTest.exe directly using the Exec task of MSBuild. This way you have the freedom to pass in the arguments you want.
I'm starting to create a MSBuild scripts for my products, and I've encounter a dilema.
The code is divided into around 25 projects, some wll require obfuscation, some will require strong-name signing; others will require linking into a single file.
All these projects should result in 3 products, with 3 setups.
The question at hand is as follow: How do I divide the MSBuild scripts to make most sense?
Do I create a script for each product? do I create a script for each project? Do I have one script for building, another for obfuscation and so on?
I think this is good idea to have script per product.
To minimaze dublication create reusable "sub-scripts" and import them to main script (this could be done with Import directive).
<Import Project="..\Steps\Step1.proj" />
Script per product sounds like the way to go. You may want to consider having any number of shared or base scripts too, to import common build steps. Like Mike Chaliy already mentioned you can then use Import in your product's build script:
<Import Project="..\Shared\Base.proj" />
Another thing you might also want to take advantage of is target and property overriding. It's akin to overriding virtual methods in a .Net class. See the documentation and the MSBuild Team Blog for more details. I know I've taken advantage of this quite often by setting defaults in the included scripts then overriding them as necessary in the product build script in order to customize build behaviour. For instance I often have generated files that are required before the build so I hook those targets into the BuildDependsOn property group. This way my generated files are generated whenever I do an F5 from the IDE, call the build target from the command line or otherwise build the project or solution. Obviously if you have any build steps that run long or only need to be run in special circumstances (like building installers), you'll want to take care about exactly what gets hooked in.