MSBuild - Comparing ItemGroups metadata - msbuild

I am attempting to write a build script for our source tree. This tree consists of a (large) number of solutions with assembly references between them. I have created an ItemGroup containing all the solutions and am batching over this ItemGroup to build the solutions.
I also need to copy some project outputs to an "exes output" folder, each in their own folder. I've attached some metadata to the solution item pointing to the projects that I want to grab the output from. As I can have potentially more than one project to output from each solution, I am doing this by giving the metadata the value that is passed to an ItemGroup's Include to create an ItemGroup separately. This works happily and I am able to batch over this dynamically created ItemGroup.
The final step I want to do, which is causing me a headache, is that for some of those output projects, I want to specify a special folder name. Now, I can do this by altering the metadata inside the target that is doing the work, like this:
<?xml version="1.0" encoding="utf-8"?>
<Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003"
DefaultTargets="Build">
<!-- Define all the solutions to build, and any projects we want to handle the output of -->
<ItemGroup>
<SolutionsToBuild Include="$(MSBuildProjectDirectory)\Solution1\Solution1.sln">
<ProjectsToOutput>
$(MSBuildProjectDirectory)\Solution1\Project1A\Project1A.csproj;
$(MSBuildProjectDirectory)\Solution1\Project1B\Project1B.csproj;
</ProjectsToOutput>
</SolutionsToBuild>
<SolutionsToBuild Include="$(MSBuildProjectDirectory)\Solution2\Solution2.sln">
<ProjectsToOutput>
$(MSBuildProjectDirectory)\Solution2\Project2A\Project2A.csproj;
$(MSBuildProjectDirectory)\Solution2\Project2B\Project2B.csproj;
</ProjectsToOutput>
</SolutionsToBuild>
</ItemGroup>
<Target Name="Build">
<CallTarget Targets="DoBuild" />
</Target>
<Target Name="DoBuild" Outputs="%(SolutionsToBuild.Identity)">
<Message Text="Building project: %(SolutionsToBuild.FullPath)" />
<!-- e.g. <MSBuild Projects="%(SolutionsToBuild.FullPath)" /> -->
<PropertyGroup>
<ProjectsToOutputIncludeMask>%(SolutionsToBuild.ProjectsToOutput)</ProjectsToOutputIncludeMask>
</PropertyGroup>
<ItemGroup>
<OutputProjects Include="$(ProjectsToOutputIncludeMask)" />
<!-- Now create the OutputTo metadata -->
<!-- Default to the same name as the project file -->
<OutputProjects>
<OutputTo>%(OutputProjects.FileName)</OutputTo>
</OutputProjects>
<!-- Now override specific projects OutputTo metadata -->
<OutputProjects Condition="'%(OutputProjects.FileName)' == 'Project1A'">
<OutputTo>ArbitraryFolder1</OutputTo>
</OutputProjects>
<OutputProjects Condition="'%(OutputProjects.FileName)' == 'Project2B'">
<OutputTo>ArbitraryFolder2</OutputTo>
</OutputProjects>
</ItemGroup>
<Message Text=" Outputting project: %(OutputProjects.FullPath) -> %(OutputTo)" />
</Target>
</Project>
However, this build script needs to be maintained by all developers on the team, not all of whom are familiar with MSBuild. Therefore, I'd like to define another ItemGroup at the top of the script that defines any special names. They can then ignore all the targets and tasks and just maintain the ItemGroups, like this:
<ItemGroup>
<SolutionsToBuild Include="$(MSBuildProjectDirectory)\Solution1\Solution1.sln">
<ProjectsToOutput>
$(MSBuildProjectDirectory)\Solution1\Project1A\Project1A.csproj;
$(MSBuildProjectDirectory)\Solution1\Project1B\Project1B.csproj;
</ProjectsToOutput>
</SolutionsToBuild>
<SolutionsToBuild Include="$(MSBuildProjectDirectory)\Solution2\Solution2.sln">
<ProjectsToOutput>
$(MSBuildProjectDirectory)\Solution2\Project2A\Project2A.csproj;
$(MSBuildProjectDirectory)\Solution2\Project2B\Project2B.csproj;
</ProjectsToOutput>
</SolutionsToBuild>
<OutputNames Include="Project1A">
<OutputFolder>ArbitraryFolder1</OutputFolder>
</OutputNames>
<OutputNames Include="Project2B">
<OutputFolder>ArbitraryFolder2</OutputFolder>
</OutputNames>
</ItemGroup>
However, any way I've tried to get the DoBuild target to update the metadata falls on its face. I thought I could do this:
<!-- Now override specific projects OutputTo metadata -->
<OutputProjects Condition="'%(OutputProjects.FileName)' == '%(OutputNames.Identity)'">
<OutputTo>%(OutputNames.OutputFolder)</OutputTo>
</OutputProjects>
But that code batches over The OutputProjects item group and then over the OutputNames item group, so the condition is never true (one of the arguments to the comparison is always empty).
I'm unfortunately, at this stage, unable to change either the solution structure or the output folder structure requirements. Is there any trick of MSBuild that I'm missing that could help me here? I'm not averse to including a custom task to do the job, but would prefer a straight MSBuild solution if possible.
If it makes a difference, I am using MSBuild v4.

Ah. Stumbled on an answer whilst playing around with this.
Firstly, I was investigating this post about getting the intersection of two item groups. I therefore changed my OutputNames item group to have the same Identity as the OutputProjects ItemGroup:
<OutputNames Include="$(MSBuildProjectDirectory)\Solution1\Project1A\Project1A.csproj">
<OutputFolder>ArbitraryFolder1</OutputFolder>
</OutputNames>
<OutputNames Include="$(MSBuildProjectDirectory)\Solution2\Project2B\Project2B.csproj">
<OutputFolder>ArbitraryFolder2</OutputFolder>
</OutputNames>
This let me batch on %(Identity) and get the intersection like this:
<Message Condition="'%(Identity)' != '' and
'#(OutputProjects)' != '' and
'#(OutputNames)' != ''"
Text="Found a match for %(Identity)" />
However, when also referring to the OutputFolder metadata in the same Task, that became part of the batching as well resulting in the following never printing to the output:
<Message Condition="'%(Identity)' != '' and
'#(OutputProjects)' != '' and
'#(OutputNames)' != ''"
Text="Found a match for %(Identity): %(OutputNames.OutputFolder)" />
But, by using a transform over the property instead of direct access, it isn't treated as part of the batching:
<Message Condition="'%(Identity)' != '' and
'#(OutputProjects)' != '' and
'#(OutputNames)' != ''"
Text="Found a match for %(Identity): #(OutputNames->'%(OutputFolder)')" />
Therefore, I can do the following to update my metadata:
<!-- Now override specific projects OutputTo metadata -->
<OutputProjects Condition="'%(Identity)' != '' AND
'#(OutputProjects)' != '' AND
'#(OutputNames)' != ''">
<OutputTo>#(OutputNames->'%(OutputFolder)')</OutputTo>
</OutputProjects>

Related

MSBuild: Compare ItemGroups or access by index

For a C++ project, I want to autogenerate a defs.h file with project definitions, such as the date, git commit, ... to automate the versioning process of my application.
Therefore I am trying to create a MSBuild Target that will extract the latest git tag, git commit, and the current date and save it to a temporary gitinfo.txt file.
Another build target will depend on that file and generate a .h file.
In order to avoid unnecessary recompiles of my project, the .h file and for that reason the gitinfo.txt file shall only be rewritten, if any of the information has changes.
So my idea is the following:
Calculate git and date info
If available, read in the existing gitinfo.txt
Compare the calculated values to those in the txt file
If anything has changed, rewrite the gitinfo.txt
I've mastered steps 1. and 2., however I am not sure how to process the values after reading them.
<!-- The purpose of this target is to update gitinfo.txt if git information (commit...) has changed -->
<Target
Name="GetHeaderInfos"
BeforeTargets="ClCompile"
Outputs="$(IntDir)\gitinfo.txt"
>
<!-- Get information about the state of this repo-->
<GitDescribe>
<Output TaskParameter="Tag" PropertyName="NewGitTag" />
<Output TaskParameter="CommitHash" PropertyName="NewGitCommitHash" />
<Output TaskParameter="CommitCount" PropertyName="NewGitCommitCount" />
</GitDescribe>
<!-- Get the current date -->
<Time Format="dd.MM.yyyy">
<Output TaskParameter="FormattedTime" PropertyName="NewBuildDate" />
</Time>
<ReadLinesFromFile File="$(IntDir)\gitinfo.txt" Condition="Exists('$(IntDir)\gitinfo.txt')">
<Output TaskParameter="Lines" ItemName="Version" />
</ReadLinesFromFile>
<!-- Comparison here! HOW TO DO IT PROPERLY -->
<PropertyGroup>
<TagChanged> <!-- `$(NewGitTag)` == `$(Version)[0]` --> </TagChanged>
<!-- Other comparisons -->
</PropertyGroup>
</Target>
And this could be the content of gitinfo.txt
v4.1.4
04fe34ab
1
31.07.2016
I am not quite sure how to compare the values now. I need to compare $(NewGitTag) to the first value in the $(Version) version variable, and so on.
I haven't found an example, that actually accesses the variables after reading them from a file. The official documentation provides no help, nor have I found anything on stackoverflow or the likes.
I only know that the $(Version) variable holds a list, and I can batch process it. How can I compare its content to the defined variables $(NewGitTag), $(NewGitCommitHash), $(NewGitCommitCount) and $(NewBuildDate)?
Suppose we start with this data:
<ItemGroup>
<Version Include="v4.1.4;04fe34ab;1;31.07.2016"/>
</ItemGroup>
<PropertyGroup>
<GitTag>v4.1.4</GitTag>
<GitSHA>04fe34ab</GitSHA>
<Count>1</Count>
<Date>31.07.2016</Date>
</PropertyGroup>
Then here are at least 3 ways to achieve comparision (apart from the one mentioned in the comment) and there are probably other ways as well (I'll post them if I can come up with something else):
Just compare the items
I'm not sure why you want to compare everything seperately when this works just as well: compare the whole ItemGroup at once.
<Target Name="Compare1">
<PropertyGroup>
<VersionChanged>True</VersionChanged>
<VersionChanged Condition="'#(Version)' == '$(GitTag);$(GitSHA);$(Count);$(Date)'">False</VersionChanged>
</PropertyGroup>
<Message Text="VersionChanged = $(VersionChanged)" />
</Target>
Batch and check if there's one difference
Each item of Version is compared with e.g. GitTag via batching. The result will be False;False;False;False if there's a difference, else it will be True;False;False;False. Count the distinct elements and if it's 2 it means we got the latter so GitTag did not change. Note this obviousle only works if each of your source items can never have the same value as one of the other items.
<Target Name="Compare2">
<PropertyGroup>
<TagChanged>True</TagChanged>
<TagChanged Condition="'#(Version->Contains($(GitTag))->Distinct()->Count())' == '2'">False</TagChanged>
</PropertyGroup>
<Message Text="TagChanged = $(TagChanged)" />
</Target>
you can then compare the other items as well and combine the result.
Use an inline task to access items by index
This comes closest to what's in your question, but it does need a bit of inline code.
<UsingTask TaskName="IndexItemGroup" TaskFactory="CodeTaskFactory" AssemblyFile="$(MSBuildToolsPath)\Microsoft.Build.Tasks.v4.0.dll" >
<ParameterGroup>
<Items Required="true" ParameterType="Microsoft.Build.Framework.ITaskItem[]"/>
<Index Required="true" ParameterType="System.Int32"/>
<Item Output="true" ParameterType="Microsoft.Build.Framework.ITaskItem"/>
</ParameterGroup>
<Task>
<Code Type="Fragment" Language="cs">
<![CDATA[Item = Items[ Index ];]]>
</Code>
</Task>
</UsingTask>
<Target Name="Compare3">
<IndexItemGroup Items="#(Version)" Index="1">
<Output PropertyName="OldGitSHA" TaskParameter="Item"/>
</IndexItemGroup>
<PropertyGroup>
<SHAChanged>True</SHAChanged>
<SHAChanged Condition="'$(GitSHA)' == '$(OldGitSHA)'">False</SHAChanged>
</PropertyGroup>
<Message Text="OldGitSHA = $(OldGitSHA), changed = $(SHAChanged)" />
</Target>

Using the Zip task in MSBuild

I have been attempting to use the zip task of msbuild in a project I am working on at the moment.
My project file looks something like this:
<PropertyGroup> <MSBuildCommunityTasksPath>$(SolutionDir)\.build</MSBuildCommunityTasksPath> </PropertyGroup>
<Import Project="$(MSBuildCommunityTasksPath)\MSBuild.Community.Tasks.Targets" />
<ItemGroup>
<FileToZip include="C:\FilePath"></FilesToZip>
<FileToZip include="C:\FilePath"></FilesToZip>
</ItemGroup>
<Target Name="BeforeBuild">
<PropertyGroup>
<ReleasePath>\releasepath</ReleasePath>
<Zip Files="#(FilesToZip)" WorkingDirectory="$(ReleasePath)" ZipFileName="HTMLeditor.html" ZipLevel="9" />
</Target>
However, the zip file updates but does not contain the files specified in the item group FilesToZip. I cannot figure out why they aren't being recognised! I have double checked file paths and they are correct. Any ideas?
I think you want to do something like this:
<ItemGroup>
<FileToZip include="C:\FilePath;C:\FilePath"/>
</ItemGroup>
As I mentioned in my comment, simply creating a variable (FileToZip) and repeating it twice with different values does not give you an array that contains both of the values. You end up with only the last value (and not an array at all). Your include attribute is a selector which is used to build the array and it can contain multiple values, wildcards and other patterns which are used to build out that array for you.
Here's a link to MSDN that gives you more information on how to use the Include attribute: http://msdn.microsoft.com/en-us/library/ms171454.aspx
I ditched the ItemGroup in the end, and went with another way of doing it.
<Target Name="Zip">
<CreateItem Include="FilesToInclude" >
<Output ItemName="ZipFiles" TaskParameter="Include"/>
<Zip ZipFileName="ZipFile.zip" WorkingDirectory="FolderToWriteZipTo" Files="#(ZipFiles)" />
</Target>
This method seemed to be easier and wasn't adding files to the root of the file.
Thanks for the help though guys.

Pass list item to Properties when calling reusable msbuild target

I'm trying to create a reusable Target in msbuild, following the basic model outlined in How to invoke the same msbuild target twice?
I'm stuck trying to pass a property that I want interpreted as a list. I haven't found an example online that deals with this situation. As I understand it, the problem is that Properties is already treated as a list item, so it doesn't like having a list item passed in as well. Is there a way to get msbuild to pack and unpack the list correctly here?
Msbuild is complaining with:
error MSB4012: The expression "FilesToZip=#(Scripts)" cannot be used in this context. Item lists cannot be concatenated with other strings where an item list is expected. Use a semicolon to separate multiple item lists.
Here's an example caller:
<Target Name="BuildMigrationZip">
<MSBuild Projects="BuildZip.msbuild"
Targets="BuildZip"
Properties="FilesToZip=#(Scripts);OutputZipFile=$(MigrationPackageFilePath);OutputFolder=$(MigrationPackagePath);Flatten=true"/>
<Message Text="Created database migration zip: $(MigrationPackageFilePath)" Importance="high"/>
</Target>
And the base target:
<Target Name="BuildZip">
<MakeDir Directories="$(OutputFolder)"/>
<Zip Files="#(FilesToZip)"
ZipFileName="$(OutputZipFile)"
Flatten="$(Flatten)"
ParallelCompression="false" />
</Target>
I'm basically at the point of just going back to cut and paste for these, although I want to package up a number of zips here.
UPDATE: The same issue applies to setting Inputs on the reusable target. My question up to this point addresses the raw functionality, but it would be nice to keep dependencies working. So for example:
<Target Name="BuildZip"
Inputs="#(FilesToZip)"
Outputs="$(OutputZipFile)">
<MakeDir Directories="$(OutputFolder)"/>
<Zip Files="#(FilesToZip)"
ZipFileName="$(OutputZipFile)"
Flatten="$(Flatten)"
ParallelCompression="false" />
</Target>
They key is to pass the list around as a property. So when your Scripts list is defined as
<ItemGroup>
<Scripts Include="A"/>
<Scripts Include="B"/>
<Scripts Include="C"/>
</ItemGroup>
then you first convert it into a property (which just makes semicolon seperated items, but msbuild knows how to pass this via the Properties of the MSBuild target) then pass it to the target:
<Target Name="BuildMigrationZip">
<PropertyGroup>
<ScriptsProperty>#(Scripts)</ScriptsProperty>
</PropertyGroup>
<MSBuild Projects="$(MSBuildThisFile)" Targets="BuildZip"
Properties="FilesToZip=$(ScriptsProperty)" />
</Target>
(note I'm using $(MSBuildThisFile) here: you don't necessarily need to create seperate build files for every single target, in fact for small targets like yours it's much more convenient to put it in the same file)
Then in your destination target you turn the property into a list again:
<Target Name="BuildZip">
<ItemGroup>
<FilesToZipList Include="$(FilesToZip)"/>
</ItemGroup>
<Message Text="BuildZip: #(FilesToZipList)" />
</Target>
Output:
BuildZip: A;B;C
Update
When working with Inputs, you cannot pass #(FilesToZip) since that expands to nothing because is not a list: it's a property - which happens to be a number of semicolon-seperated strings. And as such, it is usable for Inputs you just have to expand it as the property it is i.e. $(FilesToZip):
<Target Name="BuildZip"
Inputs="$(FilesToZip)"
Outputs="$(OutputZipFile)">
...
</Target>
Output of second run:
BuildZip:
Skipping target "BuildZip" because all output files are up-to-date with respect to the input files.

How do I dump a list of imported property sheets from MSBuild

I am constructing a hierarchy of property sheets (many that are conditionally included according to Platform and Configuration) and I am attempting to write a set of targets that can help diagnose any errors that may sneak in.
What I would like is a list of property sheets that have been included.
Example:
<ImportGroup Condition="$(Configuration.Contains('Debug'))">
<Import Project="ps.cpp.config.debug.props"/>
</ImportGroup>
<ImportGroup Condition="$(Configuration.Contains('Release'))">
<Import Project="ps.cpp.config.release.props"/>
</ImportGroup>
<ImportGroup Condition="'$(Platform)' == 'x64'">
<Import Project="ps.cpp.plat.x64.props"/>
</ImportGroup>
<ImportGroup Condition="'$(Platform)' == 'Win32'">
<Import Project="ps.cpp.plat.win32.props"/>
</ImportGroup>
And a target like this:
<Target Name="DumpPropertySheets">
<!-- This doesn't work! -->
<!-- <Message Text="%(Import.Project)"/> -->
</Target>
Which should result in console output like this when built with msbuild test.vcxproj /t:DumpPropertySheets /p:Platform=x64 /p:Configuration:Debug
DumpPropertySheets:
ps.cpp.config.debug.props
ps.cpp.plat.x64.props
There is not an obvious way to do what you are trying to do. Imports are pre-processed to aggregate all of the content into a single file, they are not a datatype like item arrays or properties that can be referenced later on.
The syntax %(Import.Project) doesn't work because that syntax is valid only for item arrays, and you are trying to use it on the Import keyword, which is not a populated item array.
Also note that your use of the ImportGroup elements surrounding the imports is optional (and probably a bit verbose). The following two constructs in an MSBuild file are equivalent...
<ImportGroup Condition="$(Configuration.Contains('Debug'))">
<Import Project="ps.cpp.config.debug.props"/>
</ImportGroup>
...and (line-wrapped for clarity)...
<Import
Condition="$(Configuration.Contains('Debug'))"
Project="ps.cpp.config.debug.props"
/>
If you are trying to diagnose property sheet import errors, don't forget about the /pp command-line switch, which will dump the complete preprocessed file. You also could (for your own files at least) give each import a unique entry into an item array, e.g.
<Import
Condition="$(Configuration.Contains('Debug'))"
Project="ps.cpp.config.debug.props"
/>
then inside ps.cpp.config.debug.props,
<ItemGroup>
<_Import Include="$(MSBuildThisFile)" />
</ItemGroup>
then later in your build you could get what you appear to be looking for, to some degree, with,
<Target Name="DumpPropertySheets">
<!-- This does work! -->
<Message Text="%(_Import.Identity)" />
</Target>

Including files with directory specified separately in MSBuild

This seems like it should be simple but I can't work it out from the reference and my google-fu is apparently weak.
I just want to specify the file names and base folder separately in the build file...
<?xml version="1.0" encoding="utf-8"?>
<Project DefaultTargets="Build" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
<PropertyGroup>
<TestFilesWithFolder>
B:\Root\Test1.*;
B:\Root\Test2.*
</TestFilesWithFolder>
<TestFiles>Test1.*;Test2.*</TestFiles>
<TestFileRoot>B:\Root</TestFileRoot>
</PropertyGroup>
<Target Name="Build">
<ItemGroup>
<TestFilesGroupWithFolder Include="$(TestFilesWithFolder)" />
<TestFilesGroup Include="$(TestFileRoot)\$(TestFiles)" />
</ItemGroup>
<Warning Text="Source files with folder: #(TestFilesGroupWithFolder)" />
<Warning Text="Source files: #(TestFilesGroup)" />
</Target>
</Project>
When I run this, the first warning shows both files as expected, but the second warning only shows the first file (since the straight string concat put the folder name on the first but not second).
How would I get the ItemGroup "TestFilesGroup" to include both the files given the "TestFiles" and "TestFileRoot" properties?
It is possible to convert a semicolon delimited list of things into an item, which would make this possible, except that the items in your property contain wildcards, so if you want to have MSBuild treat them as items in a list, at the moment MSBuild first sees it the path must be valid. There may be a way to do that but I can't think of one. In other words...
<ItemGroup>
<TestFiles Include="$(TestFiles)" />
</ItemGroup>
...only works if $(TestFiles) contains a delimited list of either things with no wildcards, or qualified paths that actually exist.
Further, MSBuild can't compose a path with a wildcard inside the Include attribute and evaluate it at the same time, so you need a trick to first compose the full path separately, then feed it into the Include attribute. The following will work, but it requires changing your delimited property into a set of items. It batches a dependent target on this item list, with each batched target execution calculating a meta value for one item, which is stored off in a new meta value. When the original target executes, it is able to use that meta value in a subsequent Include.
<PropertyGroup>
<TestFilesWithFolder>
D:\Code\Test1.*;
D:\Code\Test2.*
</TestFilesWithFolder>
<TestFileRoot>D:\Code</TestFileRoot>
</PropertyGroup>
<ItemGroup>
<TestFilePattern Include="TestFilePattern">
<Pattern>Test1.*</Pattern>
</TestFilePattern>
<TestFilePattern Include="TestFilePattern">
<Pattern>Test2.*</Pattern>
</TestFilePattern>
</ItemGroup>
<Target Name="Compose" Outputs="%(TestFilePattern.Pattern)">
<ItemGroup>
<TestFilePattern Include="TestFilePattern">
<ComposedPath>#(TestFilePattern->'$(TestFileRoot)\%(Pattern)')</ComposedPath>
</TestFilePattern>
</ItemGroup>
</Target>
<Target Name="Build" DependsOnTargets="Compose">
<ItemGroup>
<TestFilesGroupWithFolder Include="$(TestFilesWithFolder)" />
</ItemGroup>
<Warning Text="Source files with folder: #(TestFilesGroupWithFolder)" />
<ItemGroup>
<ComposedTestFiles Include="%(TestFilePattern.ComposedPath)" />
</ItemGroup>
<Warning Text="Source files: #(ComposedTestFiles)" />
</Target>
Which produces the following output:
(Build target) ->
d:\Code\My.proj(80,5): warning : Source files with folder:
D:\Code\Test1.txt;D:\Code\Test2.txt
d:\Code\My.proj(84,5): warning : Source files:
D:\Code\Test1.txt;D:\Code\Test2.txt