MSBuild Batching on Three Independent Variables - msbuild

I have been writing a build system based on MSBuild, and am to the end of the project where I need to essentially run the one msbuild file 88 times by batching over three variables:
Configuration = Debug; Beta; Release; Evaluation
Platform = x86; x64
Language = CN; CS; DE; EN; ES; FR; IT; JP; KO; PL; TW
I want to build:
"Debug x86 CN", "Debug x86 CS", ... "Debug x86 TW"
"Debug x64 CN", ...
I can, of course, define 88 of these:
<ItemGroup>
<ToBuild Include="Debug_x86_CN">
<Configuration>Debug</Configuration>
<Platform>x86</Platform>
<Language>EN</Language>
</ToBuild>
<ItemGroup>
And then batch based on metadata. But what a drag! Can I create the 88 permutations in code, so adding a language is as easy as adding three characters to an ItemGroup:
<ItemGroup>
<AllConfigurations Include="Beta; Release; Evaluation;"/>
<AllPlatforms Include="x86; x64" />
<AllLanguages Include="CN; CS; DE; EN; ES; FR; IT; JP; KO; PL; TW" />
</ItemGroup>

Thanks to Anders Ljusberg for posting an answer to this years ago. The solution is to use the CreateItem task to combine the individual ItemGroups into one ItemGroup. The cross product of each item needs to be done one at a time into a new ItemGroup (in this case _Config_X_Language and _Config_X_Language_X_Platform) to prevent empty metadata from leaking in (if you try to reuse _Config_X_Language, you'll get items with empty Platform, n addition to the platforms in $(Platform).
<ItemGroup>
<Configuration Include="Beta; Release; Evaluation;"/>
<Platform Include="x86; x64" />
<Language Include="CN; CS; DE; EN; ES; FR; IT; JP; KO; PL; TW" />
</ItemGroup>
<!-- Create an ItemGroup that is the cross product of Configuration and Language: -->
<CreateItem Include="#(Configuration)" AdditionalMetadata="Language=%(Language.Identity);" >
<Output ItemName="_Config_X_Language" TaskParameter="Include"/>
</CreateItem>
<!-- Create another ItemGroup that is the cross product of _Configuration_X_Language and Platform: -->
<CreateItem Include="#(_Config_X_Language)" AdditionalMetadata="Platform=%(Platform.Identity);" >
<Output ItemName="_Config_X_Language_X_Platform" TaskParameter="Include"/>
</CreateItem>
<!-- Task batch over each unique set of metadata on AllBuilds -->
<MSBuild Projects="myproject.msbuild"
Properties="Configuration=%(_Config_X_Language_X_Platform.Identity);
Platform=%(_Config_X_Language_X_Platform.Platform);
Language=%(_Config_X_Language_X_Platform.Language);"
Targets="MyTarget"
BuildInParallel="true" />

It is a good idea to combine the individual ItemGroups into one ItemGroup.
However, when using this method MSBuild builds projects ignoring attribute BuildInParallel (all projects building consistently).
So you need something to supplement script:
<ItemGroup>
<Configuration Include="Beta; Release; Evaluation;"/>
<Platform Include="x86; x64" />
<Language Include="CN; CS; DE; EN; ES; FR; IT; JP; KO; PL; TW" />
</ItemGroup>
<!-- Create an ItemGroup that is the cross product of Configuration and Language: -->
<CreateItem Include="#(Configuration)" AdditionalMetadata="Language=%(Language.Identity);" >
<Output ItemName="_Config_X_Language" TaskParameter="Include"/>
</CreateItem>
<!-- Create another ItemGroup that is the cross product of _Configuration_X_Language and Platform: -->
<CreateItem Include="#(_Config_X_Language)" AdditionalMetadata="Platform=%(Platform.Identity);" >
<Output ItemName="_Config_X_Language_X_Platform" TaskParameter="Include"/>
</CreateItem>
<!--Creating groups for projects to build-->
<!--In the attribute AdditionalMetadata sign '=' is changed to '%3D', and the sign ';'' changed to '%3B'-->
<CreateItem Include="myproject.msbuild" AdditionalMetadata="Properties=
Configuration%3D%(_Config_X_Language_X_Platform.Identity)%3B
Platform%3D%(_Config_X_Language_X_Platform.Platform)%3B
Language%3D%(_Config_X_Language_X_Platform.Language)" >
<Output ItemName="ProjectToBuild" TaskParameter="Include"/>
</CreateItem>
<!--Task batch build all projects parallel -->
<MSBuild Projects="#(ProjectToBuild)"
Targets="MyTarget"
BuildInParallel="true"
StopOnFirstFailure="true" />

Related

Test if an MSBuild property constains a substring

I have a property in an MSBuild project which is a semicolon-separated-list of string values. How can I test if the list constains a particular value?
In the example listing below, I want the target DeployToServer only to be executed if the property $(DCC_Define) constains 'WebDeploy'.
<Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
<PropertyGroup>
<DCC_Define>WebDeploy;DEBUG</DCC_Define>
</PropertyGroup>
<Target Name="DeployToServer" Condition="$(DCC_Define) constains 'WebDeploy'">
<Message Text="Do something." />
</Target>
</Project>
I've used a bit of pseudo logic in the #Condition attribute to indicate what I mean. I am using a .NET framework version of 2.0.50727.3655; and MSBuild version of 3.4.30729.1 .
How can I achieve this? I don't have the luxury of being able to upgrade to MSBuild 4.
Well, since you can't use property function you have to get creative.
<Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
<PropertyGroup>
<DCC_Define>WebDeploy;DEBUG;WebDeploy</DCC_Define>
</PropertyGroup>
<Target Name="DeployToServer">
<CreateItem Include="$(DCC_Define)">
<Output TaskParameter="Include" ItemName="DCC_Define" />
</CreateItem>
<!-- Not required since MSBuild doesn't execute targets twice -->
<!-- <CreateProperty Value="True" Condition="%(DCC_Define.Identity) == WebDeploy">
<Output TaskParameter="Value" PropertyName="WebDeploy" />
</CreateProperty> -->
<CallTarget Targets="_DeployToServer" Condition="%(DCC_Define.Identity) == WebDeploy" />
</Target>
<Target Name="_DeployToServer">
<Message Text="Do something." />
</Target>
</Project>

msbuild output remove assemblies

I have some msbuild code that looks something like this:
<Target Name="Build">
<MSBuild
Projects="#(UnitTestProject)"
Properties="$(BuildProperties)">
<Output TaskParameter="TargetOutputs" ItemName="TestAssembly" />
</MSBuild>
</Target>
<Target Name="Test" DependsOnTargets="Build">
<ItemGroup>
<TestAssembly Remove="*.Example.dll" />
</ItemGroup>
<xunit Assemblies="#(TestAssembly)" />
</Target>
So I am building all of my unit test projects and caputuring the built dll's using the Output task on the TargetOutputs parameter. The problem is that one of the projects is calling a task that outputs some dll's that I don't want to actually run xunit against.
What's weird though is that the Remove="*.Example.dll" appears to not have any affect at all and xunit is trying to test the assembly anyway.
Why is Remove not working?
Actually I think I figured it out. It appears that the problem resides in the way the relative path is resolved in ItemGroups in the Target vs. outside of a Target. I need to be a little more explicit with my path and then it works. Basically I did this to get it to work:
<Target Name="Build">
<MSBuild
Projects="#(UnitTestProject)"
Properties="$(BuildProperties)">
<Output TaskParameter="TargetOutputs" ItemName="UnitTestOutput" />
</MSBuild>
<ItemGroup>
<TestAssembly Include="#(UnitTestOutput)" Exclude="$(RootTestPath)\**\*.Example.dll" />
</Target>
<Target Name="Test" DependsOnTargets="Build">
<xunit Assemblies="#(TestAssembly)" />
</Target>

Pass Output items to separate target with MSBuild

I am creating a buildscript, where I'm outputting the TargetOutputs of an MSBuild, then wanting to call FXCop in a separate target, and using those outputs in the TargetAssemblies.
<Target Name="Build">
<MSBuild Projects="#(Projects)"
Properties="Platform=$(Platform);Configuration=$(Configuration);"
Targets="Build"
ContinueOnError="false">
<Output TaskParameter="TargetOutputs" ItemName="TargetDLLs"/>
</MSBuild>
<CallTarget Targets="FxCopReport" />
</Target>
<Target Name="FxCopyReport">
<Message Text="FXCop assemblies to test: #(TargetDLLs)" />
<FxCop
ToolPath="$(FXCopToolPath)"
RuleLibraries="#(FxCopRuleAssemblies)"
AnalysisReportFileName="FXCopReport.html"
TargetAssemblies="#(TargetDLLs)"
OutputXslFileName="$(FXCopToolPath)\Xml\FxCopReport.xsl"
ApplyOutXsl="True"
FailOnError="False" />
</Target>
When I run this, in the FxCopyReport target, the Message of TargetDLLs in empty, whereas if I put this in the Build target, it populates.
How can I pass/reference this value?
There is a blog post by Sayed Ibrahim Hashimi (co-author of Inside MSBuild book), describing the issue you ran into, dating back in 2005. Essentially CallTarget task is behaving weird. I'm not sure if it is a bug or designed behavior, but the behavior is still the same in MSBuild 4.0.
As a workaround, use normal MSBuild mechanism of setting order of execution of targets in MSBuild, using attributes DependsOnTargets, BeforeTargets or AfterTargets.
I was able to figure this one out.
Essentially, after the MSBuild step, I created an ItemGroup, which I then referenced in the calling Target.
<Target Name="Build">
<Message Text="Building Solution Projects: %(Projects.FullPath)" />
<MSBuild Projects="#(Projects)"
Properties="Platform=$(Platform);Configuration=$(Configuration);"
Targets="Build"
ContinueOnError="false">
<Output TaskParameter="TargetOutputs" ItemName="TargetDllOutputs"/>
</MSBuild>
<ItemGroup>
<TestAssemblies Include="#(TargetDllOutputs)" />
</ItemGroup>
</Target>
<Target Name="FXCopReport">
<Message Text="FXCop assemblies to test: #(TestAssemblies)" />
<FxCop
ToolPath="$(FXCopToolPath)"
RuleLibraries="#(FxCopRuleAssemblies)"
AnalysisReportFileName="$(BuildPath)\$(FxCopReportFile)"
TargetAssemblies="#(TestAssemblies)"
OutputXslFileName="$(FXCopToolPath)\Xml\FxCopReport.xsl"
Rules="$(FxCopExcludeRules)"
ApplyOutXsl="True"
FailOnError="True" />
<Message Text="##teamcity[importData id='FxCop' file='$(BuildPath)\$(FxCopReportFile)']" Condition="'$(TEAMCITY_BUILD_PROPERTIES_FILE)' != ''" />
</Target>

MSBuild Build Sequence

Looking at this article from MS, I have a question about the SolutionToBuild section.
<ItemGroup>
<SolutionToBuild Include="$(SolutionRoot)\path\MySolution.sln />
<SolutionToBuild Include="$(SolutionRoot)\Scribble\scribble.sln" />
<SolutionToBuild Include="$(SolutionRoot)\HelloWorld\HelloWorld.sln" />
<SolutionToBuild Include="$(SolutionRoot)\TestProject1\TestProject1.sln" />
</ItemGroup>
It says that the sequence of the build is determined by the order above. So, for example, MySolution would be built before scribble.
However, is this safe if scribble is dependant on MySolution? For example, MySolution outputs one or more dlls that are used by scribble. If MySolution and scribble are changed simultaneously, will the build wait for MySolution to be completely compiled before moving to the next project?
You can try to use additional metadata for item SolutionToBuild. Some work with recursion and voilĂ !
<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
<ItemGroup>
<SolutionToBuild Include="$(SolutionRoot)\Scribble\levelone.sln">
<DependsOnSolutions>$(SolutionRoot)\Scribble\leveltwo.sln</DependsOnSolutions>
</SolutionToBuild>
<SolutionToBuild Include="$(SolutionRoot)\Scribble\leveltwo.sln">
<DependsOnSolutions>$(SolutionRoot)\Scribble\levelthree.sln;$(SolutionRoot)\TestProject1\TestProject1.sln</DependsOnSolutions>
</SolutionToBuild>
<SolutionToBuild Include="$(SolutionRoot)\Scribble\levelthree.sln" />
<SolutionToBuild Include="$(SolutionRoot)\TestProject1\TestProject1.sln" />
</ItemGroup>
<Target Name="Build">
<MSBuild Projects="$(MSBuildProjectFile)"
Targets="BuildSolution"
Properties="SolutionFullPath=%(SolutionToBuild.Identity)"/>
</Target>
<Target Name="BuildSolution">
<CreateItem Condition="'%(SolutionToBuild.Identity)'=='$(SolutionFullPath)'"
Include="%(SolutionToBuild.DependsOnSolutions)">
<Output TaskParameter="Include"
ItemName="DependentSolutions" />
</CreateItem>
<Message Text="Building solution $(SolutionFullPath)..." />
<Message Text="Solution $(SolutionFullPath) depends on %(DependentSolutions.Identity)..."
Condition="'#(DependentSolutions)'!=''"/>
<Message Text="Building dependent solutions..."
Condition="'#(DependentSolutions)'!=''"/>
<MSBuild Projects="$(MSBuildProjectFile)"
Targets="BuildSolution"
Properties="SolutionFullPath=%(DependentSolutions.Identity)"
Condition="'#(DependentSolutions)'!=''"/>
<!-- <MSBuild Projects="$(SolutionFullPath)" /> -->
<Message Text="Building solution $(SolutionFullPath)... OK" />
</Target>
</Project>
How do you manage solution dependency? Aren't you referencing projects instead? I'm also puzzled about the 'simultaneous' changes on some of your solutions. Please clarify the nature of these changes.
So far, the answers to your questions are:
No. They may be compiled one after the other, but does it qualify for dependency?
Yes. If the sequence is mandatory, the builder will 'wait' until each solution is built (either with success or error) before moving to the next.

How to invoke the same msbuild target twice with different parameters from within msbuild project file itself

I have the following piece of msbuild code:
<PropertyGroup>
<DirA>C:\DirA\</DirA>
<DirB>C:\DirB\</DirB>
</PropertyGroup>
<Target Name="CopyToDirA"
Condition="Exists('$(DirA)') AND '#(FilesToCopy)' != ''"
Inputs="#(FilesToCopy)"
Outputs="#(FilesToCopy -> '$(DirA)%(Filename)%(Extension)')">
<Copy SourceFiles="#(FilesToCopy)" DestinationFolder="$(DirA)" />
</Target>
<Target Name="CopyToDirB"
Condition="Exists('$(DirB)') AND '#(FilesToCopy)' != ''"
Inputs="#(FilesToCopy)"
Outputs="#(FilesToCopy -> '$(DirB)%(Filename)%(Extension)')">
<Copy SourceFiles="#(FilesToCopy)" DestinationFolder="$(DirB)" />
</Target>
<Target Name="CopyFiles" DependsOnTargets="CopyToDirA;CopyToDirB"/>
So invoking the target CopyFiles copies the relevant files to $(DirA) and $(DirB), provided they are not already there and up-to-date.
But the targets CopyToDirA and CopyToDirB look identical except one copies to $(DirA) and the other - to $(DirB). Is it possible to unify them into one target first invoked with $(DirA) and then with $(DirB)?
Thanks.
You should be able to generate an ItemGroup containing the Dirs and then % on that.
<ItemGroup>
<Dirs Include="C:\DirA\;C:\DirB\">
</ItemGroup>
<Target Name="CopyFiles"
Condition="Exists('%(Dirs)') AND '#(FilesToCopy)' != ''"
Inputs="#(FilesToCopy)"
Outputs="#(FilesToCopy -> '%(Dirs)%(Filename)%(Extension)')">
<Copy SourceFiles="#(FilesToCopy)" DestinationFolder="%(Dirs)" />
</Target>
Or you can do 2 explicit calls:
<Target Name="CopyFiles">
<MsBuild Projects="$(MSBuildProjectFullPath)" Targets="CopyASetOfFiles" Properties="FilesToCopy=#(FilesToCopy);DestDir=$(DirA)" />
<MsBuild Projects="$(MSBuildProjectFullPath)" Targets="CopyASetOfFiles" Properties="FilesToCopy=#(FilesToCopy);DestDir=$(DirB)" />
</Target>
<Target Name="CopyASetOfFiles"
Condition="Exists('$(DestDir)') AND '#(FilesToCopy)' != ''"
Inputs="#(FilesToCopy)"
Outputs="#(FilesToCopy -> '$(DestDir)%(Filename)%(Extension)')">
<Copy SourceFiles="#(FilesToCopy)" DestinationFolder="$(DestDir)" />
</Target>
I haven't tested either syntax, but am relatively more confident of the second.
(The answer, if there is one, is in my Sayed Hashimi book on my desk - you'll have to wait until the first of:
Get the book
I get bored
Sayed finds this post and comes up with a brilliant tested answer)
As someone already mentiond the answer is batching.
Here are some links:
MSBuild Batching Part 1
MSBuild Batching Part 2
MSBuild Batching Part 3
MSBuild RE: Enforcing the Build Agent in a Team Build
Yes, what you want is called batching in MSBuild. The
;%(Dirs.Identity)
Defined in the Outputs will cause this task to be executed for each item in the Dirs ItemGroup.
<?xml version="1.0" encoding="utf-8"?>
<Project DefaultTargets="CopyFiles"
xmlns="http://schemas.microsoft.com/developer/msbuild/2003"
ToolsVersion="3.5">
<ItemGroup>
<Dirs Include="C:\DirA" />
<Dirs Include="C:\DirB" />
</ItemGroup>
<Target Name="CopyFiles"
Inputs="#(FilesToCopy);#(Dirs)"
Outputs="#(FilesToCopy -> '%(Dirs.Identity)%(Filename)%(Extension)');%(Dirs.Identity)" >
<Message Text="%(Dirs.Identity)" />
</Target>
</Project>
Outputs:
Build started 8/19/2009 10:11:57 PM.
Project "D:\temp\test.proj" on node 0 (default targets).
C:\DirA
CopyFiles:
C:\DirB
Done Building Project "D:\temp\test.proj" (default targets).
Change the Message task to Copy task with the following condition and you are done:
Condition="Exists('%(Dirs.Identity)') AND '#(FilesToCopy)' != ''"