MsBuild Target with multiple Outputs - msbuild

Is there a way to overcome Targets one-to-one mapping when you have multiple outputs? Seems like that should be possible, but I cannot find out how, given I'm pretty new to MsBuild I'm probably missing something.
The following piece of msbuild script is from microsoft's documentation. What should I change when I have multiple backup folders? So a list #(BackupFolders) and I would like to keep the incremental behaviour of the build?
<Target Name="Backup" Inputs="#(Compile)"
Outputs="#(Compile->'$(BackupFolder)%(Identity).bak')">
<Copy SourceFiles="#(Compile)" DestinationFiles=
"#(Compile->'$(BackupFolder)%(Identity).bak')" />
</Target>

First of all, the Inputs and Outputs attributes on the Target node are for incremental builds. They need to have the same amount of entries in order for msbuild to understand which items should be filtered when building. MSBuild checks if the output is already there and up-to-date, and if so, the matching input item is filtered from the input list. If you don't care for incremental building, you can skip this mechanism altogether. If inputs and outputs don't match (or are not present), msbuild will always execute the target with all items, because it can't decide which items lead to which output.
Second, what these attributes expect is a list of items. This doesn't have to be one list, it could be an arbitrary list. So it's perfectly fair to extend your example like this:
<Target Name="Backup" Inputs="#(Compile);#(Compile2)"
Outputs="#(Compile->'$(BackupFolder)%(Identity).bak');#(Compile2->'$(BackupFolder)%(Identity).bak')">
<Copy SourceFiles="#(Compile)" DestinationFiles=
"#(Compile->'$(BackupFolder)%(Identity).bak')" />
<Copy SourceFiles="#(Compile2)" DestinationFiles=
"#(Compile2->'$(BackupFolder)%(Identity).bak')" />
</Target>
But you want to copy the same items to different backup folders, right? So something like this should do:
<Target Name="Backup">
<Copy SourceFiles="#(Compile)" DestinationFiles=
"#(Compile->'$(BackupFolder)%(Identity).bak')" />
<Copy SourceFiles="#(Compile)" DestinationFiles=
"#(Compile->'$(BackupFolder2)%(Identity).bak')" />
</Target>
With two backup folders, an item may actually already be up-to-date in one folder, but missing in the other. You could define one as the "main" backup folder, and tell MSBuild to use this as reference for incremental builds.
Edit: For incremental builds to two locations, probably the easiest solution is to combine two targets, both building incrementally:
<Target Name="Backup" DependsOnTargets="_Backup1;_Backup2">
</Target>
<Target Name="_Backup1" Inputs="#(Compile)"
Outputs="#(Compile->'$(BackupFolder)%(Identity).bak')">
<Copy SourceFiles="#(Compile)" DestinationFiles=
"#(Compile->'$(BackupFolder)%(Identity).bak')" />
</Target>
<Target Name="_Backup2" Inputs="#(Compile)"
Outputs="#(Compile->'$(BackupFolder2)%(Identity).bak')">
<Copy SourceFiles="#(Compile)" DestinationFiles=
"#(Compile->'$(BackupFolder2)%(Identity).bak')" />
</Target>

Related

Pass list item to Properties when calling reusable msbuild target

I'm trying to create a reusable Target in msbuild, following the basic model outlined in How to invoke the same msbuild target twice?
I'm stuck trying to pass a property that I want interpreted as a list. I haven't found an example online that deals with this situation. As I understand it, the problem is that Properties is already treated as a list item, so it doesn't like having a list item passed in as well. Is there a way to get msbuild to pack and unpack the list correctly here?
Msbuild is complaining with:
error MSB4012: The expression "FilesToZip=#(Scripts)" cannot be used in this context. Item lists cannot be concatenated with other strings where an item list is expected. Use a semicolon to separate multiple item lists.
Here's an example caller:
<Target Name="BuildMigrationZip">
<MSBuild Projects="BuildZip.msbuild"
Targets="BuildZip"
Properties="FilesToZip=#(Scripts);OutputZipFile=$(MigrationPackageFilePath);OutputFolder=$(MigrationPackagePath);Flatten=true"/>
<Message Text="Created database migration zip: $(MigrationPackageFilePath)" Importance="high"/>
</Target>
And the base target:
<Target Name="BuildZip">
<MakeDir Directories="$(OutputFolder)"/>
<Zip Files="#(FilesToZip)"
ZipFileName="$(OutputZipFile)"
Flatten="$(Flatten)"
ParallelCompression="false" />
</Target>
I'm basically at the point of just going back to cut and paste for these, although I want to package up a number of zips here.
UPDATE: The same issue applies to setting Inputs on the reusable target. My question up to this point addresses the raw functionality, but it would be nice to keep dependencies working. So for example:
<Target Name="BuildZip"
Inputs="#(FilesToZip)"
Outputs="$(OutputZipFile)">
<MakeDir Directories="$(OutputFolder)"/>
<Zip Files="#(FilesToZip)"
ZipFileName="$(OutputZipFile)"
Flatten="$(Flatten)"
ParallelCompression="false" />
</Target>
They key is to pass the list around as a property. So when your Scripts list is defined as
<ItemGroup>
<Scripts Include="A"/>
<Scripts Include="B"/>
<Scripts Include="C"/>
</ItemGroup>
then you first convert it into a property (which just makes semicolon seperated items, but msbuild knows how to pass this via the Properties of the MSBuild target) then pass it to the target:
<Target Name="BuildMigrationZip">
<PropertyGroup>
<ScriptsProperty>#(Scripts)</ScriptsProperty>
</PropertyGroup>
<MSBuild Projects="$(MSBuildThisFile)" Targets="BuildZip"
Properties="FilesToZip=$(ScriptsProperty)" />
</Target>
(note I'm using $(MSBuildThisFile) here: you don't necessarily need to create seperate build files for every single target, in fact for small targets like yours it's much more convenient to put it in the same file)
Then in your destination target you turn the property into a list again:
<Target Name="BuildZip">
<ItemGroup>
<FilesToZipList Include="$(FilesToZip)"/>
</ItemGroup>
<Message Text="BuildZip: #(FilesToZipList)" />
</Target>
Output:
BuildZip: A;B;C
Update
When working with Inputs, you cannot pass #(FilesToZip) since that expands to nothing because is not a list: it's a property - which happens to be a number of semicolon-seperated strings. And as such, it is usable for Inputs you just have to expand it as the property it is i.e. $(FilesToZip):
<Target Name="BuildZip"
Inputs="$(FilesToZip)"
Outputs="$(OutputZipFile)">
...
</Target>
Output of second run:
BuildZip:
Skipping target "BuildZip" because all output files are up-to-date with respect to the input files.

MSBUILD operating on repeated Items in an ItemGroup

I have an ItemGroup that contains some Items that are duplicates of one another. I would like to execute a task on a modified version of all of the Items in the ItemGroup. But, so far, I have not been able to find a way that does not remove the duplicated Items.
Does such exist?
<ItemGroup>
<MyMessage Include="foo" />
<MyMessage Include="bar" />
<MyMessage Include="baz" />
<MyMessage Include="foo" />
<MyMessage Include="baz" />
<MyMessage Include="baz" />
</ItemGroup>
<Target Name="DemoBug">
<Message Importance="High" Text="FIRST VERSION USING ItemGroup ..."/>
<Message Importance="High" Text="#(MyMessage)"/>
<Message Importance="High" Text="SECOND VERSION USING Batching and Metadata ..."/>
<Message Importance="High" Text="someprefix;%(MyMessage.Identity);somesuffix"/>
</Target>
In my example above, the FIRST VERSION output has all the Items in it, but has only executed Message task once on the concatenated values of the Items. So this isn't what I want, at all.
5> FIRST VERSION USING ItemGroup ...
5> foo;bar;baz;foo;baz;baz
The SECOND VERSION is executing Message task several times, like I want, but it is stripping the duplicates, which I don't want it to do.
5> SECOND VERSION USING Batching and Metadata ...
5> someprefix;foo;somesuffix
5> someprefix;bar;somesuffix
5> someprefix;baz;somesuffix
Is what I want to do fundamentally impossible (in which case it's maybe time to switch to PowerShell for my task), or is there some reasonable way to do it?
(In my real task, the Items came from ReadLinesFromFile and, after various processing, will eventually end up in WriteLinesToFile. It works fine, except when duplicated lines are encountered)
Msbuild task batching is not looping. It does what it is supposed to do, grouping the items in batches that have the same metadata values. In your example it's grouping by %(Identity).
Would item transformation work in your case? Something like
<Message Importance="High" Text="#(MyMessage->'someprefix;%(Identity);somesuffix')"/>

Popping single items off list using MSBuild

I am trying to use an item in a list several times. The list is created from a names of directories using MSBuild extensions:
<MSBuild.ExtensionPack.FileSystem.FindUnder TaskAction="FindDirectories" Recursive="false" Path="path\to\stuff\">
<Output ItemName="AllFoundDirectories" TaskParameter="FoundItems"/>
</MSBuild.ExtensionPack.FileSystem.FindUnder>
I wish to use each item within this list in different instances i.e.:
<Message Text="##teamcity[testStarted name='%(AllFoundDirectories.FileName)']"/>
<!-- do some stuff using %(AllFoundDirectories.FileName) here -->
<Message Text="##teamcity[testFinished name='%(AllFoundDirectories.FileName)' duration='test_duration_in_milliseconds']"/>
The problem I have is that everything in the list is being called the first time MSBuild see's the variable %(AllFoundDirectories.FileName). Is there a way to get a single variable out of the list at a time and use it in various places?
Use target batching on a dependent target.
<Target Name="MakeDirectoryItems">
<MSBuild.ExtensionPack.FileSystem.FindUnder
TaskAction="FindDirectories"
Recursive="false" Path="path\to\stuff\">
<Output
ItemName="AllFoundDirectories"
TaskParameter="FoundItems"
/>
</MSBuild.ExtensionPack.FileSystem.FindUnder>
</Target>
<Target Name="UseDirectoryItemsInBatch"
Outputs="%(AllFoundDirectories.Identity)">
<!--
while inside this target, the value of both #(AllFoundDirectories)
and %(AllFoundDirectories.Meta) will be just the single item
in each batch.
-->
</Target>
<Target Name="Driver"
DependsOnTargets="MakeDirectoryItems;UseDirectoryItemsInBatch">
</Target>

Is it possible to refer to metadata of the target from within the target implementation in MSBuild?

My msbuild targets file contains the following section:
<ItemGroup>
<Targets Include="T1">
<Project>A\B.sln"</Project>
<DependsOnTargets>The targets T1 depends on</DependsOnTargets>
</Targets>
<Targets Include="T2">
<Project>C\D.csproj"</Project>
<DependsOnTargets>The targets T2 depends on</DependsOnTargets>
</Targets>
...
</ItemGroup>
<Target Name="T1" DependsOnTargets="The targets T1 depends on">
<MSBuild Projects="A\B.sln" Properties="Configuration=$(Configuration)" />
</Target>
<Target Name="T2" DependsOnTargets="The targets T2 depends on">
<MSBuild Projects="C\D.csproj" Properties="Configuration=$(Configuration)" />
</Target>
As you can see, A\B.sln appears twice:
As Project metadata of T1 in the ItemGroup section.
In the Target statement itself passed to the MSBuild task.
I am wondering whether I can remove the second instance and replace it with the reference to the Project metadata of the target, which name is given to the Target task?
Exactly the same question is asked for the (Targets.DependsOnTargets) metadata. It is mentioned twice much like the %(Targets.Project) metadata.
Thanks.
EDIT:
I should probably describe the constraints, which must be satisfied by the solution:
I want to be able to build individual projects with ease. Today I can simply execute msbuild file.proj /t:T1 to build the T1 target and I wish to keep this ability.
I wish to emphasize, that some projects depend on others, so the DependsOnTargets attribute is really necessary for them.
Target names must be fixed values, so what you have here wouldn't work.
Also I would recommend not using Batching Expressions inside of the DependsOnTargets expression as well. This could lead to strange behavior if you do not fully understand what is happening.
In your case you may be able to just create a "driver" target which uses those items to perform the build. The only difficult part would be the DependsOnTargets that you are trying to perform. I'm not sure about the details on what you are trying to do with that so cannot make any suggestions but as for the other take a look at creating a target similar to.
<Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
<ItemGroup>
<Targets Include="T1">
<Project>A\B.sln"</Project>
<DependsOnTargets>The targets T1 depends on</DependsOnTargets>
</Targets> <Targets Include="T2">
<Project>C\D.csproj"</Project>
<DependsOnTargets>The targets T2 depends on</DependsOnTargets>
</Targets> ...
</ItemGroup>
<Target Name="Build">
<!--
This will be executed once for every unique value of Project in the
Target item group
-->
<MSBuild Projects="%(Targets.Project)"
Properties="Configuration=$(Configuration)"
</Target>
</Project>

MSBuild: asterisks and strange ItemGroup Exclude behaviour

I have a script that attempts to construct an ItemGroup out of all files in a certain directory while excluding files with certain names (regardless of extension).
The list of files to be excluded initially contains file extensions, and I am using Community Tasks' RegexReplace to replace the extensions with an asterisk. I then use this list in the item's Exclude attribute. For some reason the files do not get excluded properly, even though the list appears to be correct.
To try and find the cause I created a test script (below) which has two tasks: first one initialises two properties with the list of file patterns in two different ways. The second task prints both properties and the files resulting from using both these properties in the Exclude attribute.
The properties' values appear to be identical, however the resulting groups are different. How is this possible?
<Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003"
DefaultTargets="Init;Test" ToolsVersion="3.5">
<Import Project="$(MSBuildExtensionsPath)\MSBuildCommunityTasks\MSBuild.Community.Tasks.Targets"/>
<Target Name="Init">
<ItemGroup>
<OriginalFilenames Include="TestDir\SampleProj.exe"/>
<OriginalFilenames Include="TestDir\SampleLib1.dll"/>
</ItemGroup>
<RegexReplace Input="#(OriginalFilenames)" Expression="\.\w+$" Replacement=".*">
<Output TaskParameter="Output" ItemName="PatternedFilenames"/>
</RegexReplace>
<PropertyGroup>
<ExcludeFilesA>TestDir\SampleProj.*;TestDir\SampleLib1.*</ExcludeFilesA>
<ExcludeFilesB>#(PatternedFilenames)</ExcludeFilesB>
</PropertyGroup>
</Target>
<Target Name="Test">
<Message Text='ExcludeFilesA: $(ExcludeFilesA)' />
<Message Text='ExcludeFilesB: $(ExcludeFilesB)' />
<ItemGroup>
<AllFiles Include="TestDir\**"/>
<RemainingFilesA Include="TestDir\**" Exclude="$(ExcludeFilesA)"/>
<RemainingFilesB Include="TestDir\**" Exclude="$(ExcludeFilesB)"/>
</ItemGroup>
<Message Text="
**AllFiles**
#(AllFiles, '
')" />
<Message Text="
**PatternedFilenames**
#(PatternedFilenames, '
')" />
<Message Text="
**RemainingFilesA**
#(RemainingFilesA, '
')" />
<Message Text="
**RemainingFilesB**
#(RemainingFilesB, '
')" />
</Target>
</Project>
Output (reformatted somewhat for clarity):
ExcludeFilesA: TestDir\SampleProj.*;TestDir\SampleLib1.*
ExcludeFilesB: TestDir\SampleProj.*;TestDir\SampleLib1.*
AllFiles:
TestDir\SampleLib1.dll
TestDir\SampleLib1.pdb
TestDir\SampleLib2.dll
TestDir\SampleLib2.pdb
TestDir\SampleProj.exe
TestDir\SampleProj.pdb
PatternedFilenames:
TestDir\SampleProj.*
TestDir\SampleLib1.*
RemainingFilesA:
TestDir\SampleLib2.dll
TestDir\SampleLib2.pdb
RemainingFilesB:
TestDir\SampleLib1.dll
TestDir\SampleLib1.pdb
TestDir\SampleLib2.dll
TestDir\SampleLib2.pdb
TestDir\SampleProj.exe
TestDir\SampleProj.pdb
Observe that both ExcludeFilesA and ExcludeFilesB look identical, but the resulting groups RemainingFilesA and RemainingFilesB differ.
Ultimately I want to obtain the list RemainingFilesA using the pattern generated the same way ExcludeFilesB is generated. Can you suggest a way, or do I have to completely rethink my approach?
The true cause of this was revealed accidentally when a custom task threw an exception.
The actual value of ExcludeFilesA is TestDir\SampleProj.*;TestDir\SampleLib1.* like one might expect. However the actual value of ExcludeFilesB is TestDir\SampleProj.%2a;TestDir\SampleLib1.%2a.
Presumably Message unescapes the string before using it, but Include and Exclude do not. That would explain why the strings look the same but behave differently.
Incidentally, the execution order doesn't seem to have anything to do with this, and I'm pretty sure (following extensive experimentation) that everything gets executed and evaluated exactly in the order in which it appears in this script.
ItemGroups need to be evaluated before targets execution, and the PatternedFilenames ItemGroup is being created on the fly within its target container.
You could workaround this using the CreateItem task, which will ensure the PatternedFilenames scope throughout the execution:
<RegexReplace Input="#(OriginalFilenames)" Expression="\.\w+$" Replacement=".*">
<Output TaskParameter="Output" ItemName="PatternedFilenames_tmp"/>
</RegexReplace>
<CreateItem Include="#(PatternedFilenames_tmp)">
<Output TaskParameter="Include" ItemName="PatternedFilenames"/>
</CreateItem>