Assume I have a ItemGroup with metadata that takes some times to build (10 seconds):
<Target Name="BuildItemGroup">
<ItemGroup>
<File Include="5">
<Value>5a</Value>
</File>
<File Include="4">
<Value>4a</Value>
</File>
...
</ItemGroup>
<Message Text="Wait 10 seconds..." Importance="High" />
</Target>
And I am going to use the same ItemGroup few times in a recursive MSBuild task:
<Target Name="Recursive" DependsOnTargets="BuildItemGroup" Condition="$(Value) > 0" >
<PropertyGroup>
<Value>$([MSBuild]::Subtract($(Value), 1))</Value>
</PropertyGroup>
<MSBuild Projects="$(MSBuildProjectFile)" Targets="Recursive" Properties="Value=$(Value)" />
</Target>
<PropertyGroup>
<Value>5</Value>
</PropertyGroup>
<Target Name="Default" DependsOnTargets="Recursive" />
It will takes extra 40 seconds finish the Recursive task for 4 loops. Is there a way to cache the ItemGroup in such manner for loop usage?
No, calling MSBuild task will create new context which will re-evaluate all the properties and items.
The only way you can use to pass data between parent\child contexts is properties (and environment variables which become global properties).
Depending on how big this list is - you can try to pass it as a property (loosing metadata) and re-create itemgroup in your child msbuild context.
Or you can switch to inline tasks using C# and d whatever you want with full C# power - e.g. save data to disk and read it back from the file.
UPDATE: Finally I've done such inline task. Actually it's a pair of tasks one to save and another one - to load persisted items from disk. A bit ugly code - I have to create a serializable copy for Item and metadata classes. But it's good enough as a PoC. It's not guaranteed that it will safely work in multi-threaded environment, especially in your case of 200+ projects.
See the code in this repository
Related
Is there a way to overcome Targets one-to-one mapping when you have multiple outputs? Seems like that should be possible, but I cannot find out how, given I'm pretty new to MsBuild I'm probably missing something.
The following piece of msbuild script is from microsoft's documentation. What should I change when I have multiple backup folders? So a list #(BackupFolders) and I would like to keep the incremental behaviour of the build?
<Target Name="Backup" Inputs="#(Compile)"
Outputs="#(Compile->'$(BackupFolder)%(Identity).bak')">
<Copy SourceFiles="#(Compile)" DestinationFiles=
"#(Compile->'$(BackupFolder)%(Identity).bak')" />
</Target>
First of all, the Inputs and Outputs attributes on the Target node are for incremental builds. They need to have the same amount of entries in order for msbuild to understand which items should be filtered when building. MSBuild checks if the output is already there and up-to-date, and if so, the matching input item is filtered from the input list. If you don't care for incremental building, you can skip this mechanism altogether. If inputs and outputs don't match (or are not present), msbuild will always execute the target with all items, because it can't decide which items lead to which output.
Second, what these attributes expect is a list of items. This doesn't have to be one list, it could be an arbitrary list. So it's perfectly fair to extend your example like this:
<Target Name="Backup" Inputs="#(Compile);#(Compile2)"
Outputs="#(Compile->'$(BackupFolder)%(Identity).bak');#(Compile2->'$(BackupFolder)%(Identity).bak')">
<Copy SourceFiles="#(Compile)" DestinationFiles=
"#(Compile->'$(BackupFolder)%(Identity).bak')" />
<Copy SourceFiles="#(Compile2)" DestinationFiles=
"#(Compile2->'$(BackupFolder)%(Identity).bak')" />
</Target>
But you want to copy the same items to different backup folders, right? So something like this should do:
<Target Name="Backup">
<Copy SourceFiles="#(Compile)" DestinationFiles=
"#(Compile->'$(BackupFolder)%(Identity).bak')" />
<Copy SourceFiles="#(Compile)" DestinationFiles=
"#(Compile->'$(BackupFolder2)%(Identity).bak')" />
</Target>
With two backup folders, an item may actually already be up-to-date in one folder, but missing in the other. You could define one as the "main" backup folder, and tell MSBuild to use this as reference for incremental builds.
Edit: For incremental builds to two locations, probably the easiest solution is to combine two targets, both building incrementally:
<Target Name="Backup" DependsOnTargets="_Backup1;_Backup2">
</Target>
<Target Name="_Backup1" Inputs="#(Compile)"
Outputs="#(Compile->'$(BackupFolder)%(Identity).bak')">
<Copy SourceFiles="#(Compile)" DestinationFiles=
"#(Compile->'$(BackupFolder)%(Identity).bak')" />
</Target>
<Target Name="_Backup2" Inputs="#(Compile)"
Outputs="#(Compile->'$(BackupFolder2)%(Identity).bak')">
<Copy SourceFiles="#(Compile)" DestinationFiles=
"#(Compile->'$(BackupFolder2)%(Identity).bak')" />
</Target>
I have an ItemGroup declared as follows:
<ItemGroup>
<MyCustomProjectType Include="..\path_to_my_project">
<Name>MyProjectName</Name>
</MyCustomProjectType>
</ItemGroup>
This is a custom project type that I want to perform some specific manipulations on.
Later I have a Target (example only but it communicates what I am after):
<Target Name="MyTarget">
<ItemGroup>
<CustomProjectReferenceFiles
KeepMetadata="Name"
Include="#(MyCustomProjectType->'%(Identity)\%(Name)\**\*')"
Exclude="**\*.x;**\*.y"
/>
</ItemGroup>
<Message Text="#(CustomProjectReferenceFiles)" />
</Target>
So I have a Target based ItemGroup where I am attempting, using a transform, to create a new Include. This does run, but it appears the Include is literally set to:
..\path_to_my_project\MyProjectName\**\*
AKA that glob/wildcards are not expanded.
I'm pretty new to MSBuild so maybe I am missing something in my search of the documentation. One solution I thought of here would be just just create a new Custom Task that handles pulling out the files I need and setting that Output on an intermediate Target.
I also found this SO question:
https://stackoverflow.com/a/3398872/1060314
Which brings up the point about CreateItem being deprecated which leaves me with not knowing what the alternatives are.
The easiest way is to use an intermediate property so that the actual text is used and not the escaped transformed items:
<PropertyGroup>
<_CustomProjectReferenceFileIncludes>#(MyCustomProjectType->'%(Identity)\%(Name)\**\*')</_CustomProjectReferenceFileIncludes>
</PropertyGroup>
<ItemGroup>
<CustomProjectReferenceFiles
KeepMetadata="Name"
Include="$(_CustomProjectReferenceFileIncludes)"
Exclude="**\*.x;**\*.y"
/>
</ItemGroup>
For a C++ project, I want to autogenerate a defs.h file with project definitions, such as the date, git commit, ... to automate the versioning process of my application.
Therefore I am trying to create a MSBuild Target that will extract the latest git tag, git commit, and the current date and save it to a temporary gitinfo.txt file.
Another build target will depend on that file and generate a .h file.
In order to avoid unnecessary recompiles of my project, the .h file and for that reason the gitinfo.txt file shall only be rewritten, if any of the information has changes.
So my idea is the following:
Calculate git and date info
If available, read in the existing gitinfo.txt
Compare the calculated values to those in the txt file
If anything has changed, rewrite the gitinfo.txt
I've mastered steps 1. and 2., however I am not sure how to process the values after reading them.
<!-- The purpose of this target is to update gitinfo.txt if git information (commit...) has changed -->
<Target
Name="GetHeaderInfos"
BeforeTargets="ClCompile"
Outputs="$(IntDir)\gitinfo.txt"
>
<!-- Get information about the state of this repo-->
<GitDescribe>
<Output TaskParameter="Tag" PropertyName="NewGitTag" />
<Output TaskParameter="CommitHash" PropertyName="NewGitCommitHash" />
<Output TaskParameter="CommitCount" PropertyName="NewGitCommitCount" />
</GitDescribe>
<!-- Get the current date -->
<Time Format="dd.MM.yyyy">
<Output TaskParameter="FormattedTime" PropertyName="NewBuildDate" />
</Time>
<ReadLinesFromFile File="$(IntDir)\gitinfo.txt" Condition="Exists('$(IntDir)\gitinfo.txt')">
<Output TaskParameter="Lines" ItemName="Version" />
</ReadLinesFromFile>
<!-- Comparison here! HOW TO DO IT PROPERLY -->
<PropertyGroup>
<TagChanged> <!-- `$(NewGitTag)` == `$(Version)[0]` --> </TagChanged>
<!-- Other comparisons -->
</PropertyGroup>
</Target>
And this could be the content of gitinfo.txt
v4.1.4
04fe34ab
1
31.07.2016
I am not quite sure how to compare the values now. I need to compare $(NewGitTag) to the first value in the $(Version) version variable, and so on.
I haven't found an example, that actually accesses the variables after reading them from a file. The official documentation provides no help, nor have I found anything on stackoverflow or the likes.
I only know that the $(Version) variable holds a list, and I can batch process it. How can I compare its content to the defined variables $(NewGitTag), $(NewGitCommitHash), $(NewGitCommitCount) and $(NewBuildDate)?
Suppose we start with this data:
<ItemGroup>
<Version Include="v4.1.4;04fe34ab;1;31.07.2016"/>
</ItemGroup>
<PropertyGroup>
<GitTag>v4.1.4</GitTag>
<GitSHA>04fe34ab</GitSHA>
<Count>1</Count>
<Date>31.07.2016</Date>
</PropertyGroup>
Then here are at least 3 ways to achieve comparision (apart from the one mentioned in the comment) and there are probably other ways as well (I'll post them if I can come up with something else):
Just compare the items
I'm not sure why you want to compare everything seperately when this works just as well: compare the whole ItemGroup at once.
<Target Name="Compare1">
<PropertyGroup>
<VersionChanged>True</VersionChanged>
<VersionChanged Condition="'#(Version)' == '$(GitTag);$(GitSHA);$(Count);$(Date)'">False</VersionChanged>
</PropertyGroup>
<Message Text="VersionChanged = $(VersionChanged)" />
</Target>
Batch and check if there's one difference
Each item of Version is compared with e.g. GitTag via batching. The result will be False;False;False;False if there's a difference, else it will be True;False;False;False. Count the distinct elements and if it's 2 it means we got the latter so GitTag did not change. Note this obviousle only works if each of your source items can never have the same value as one of the other items.
<Target Name="Compare2">
<PropertyGroup>
<TagChanged>True</TagChanged>
<TagChanged Condition="'#(Version->Contains($(GitTag))->Distinct()->Count())' == '2'">False</TagChanged>
</PropertyGroup>
<Message Text="TagChanged = $(TagChanged)" />
</Target>
you can then compare the other items as well and combine the result.
Use an inline task to access items by index
This comes closest to what's in your question, but it does need a bit of inline code.
<UsingTask TaskName="IndexItemGroup" TaskFactory="CodeTaskFactory" AssemblyFile="$(MSBuildToolsPath)\Microsoft.Build.Tasks.v4.0.dll" >
<ParameterGroup>
<Items Required="true" ParameterType="Microsoft.Build.Framework.ITaskItem[]"/>
<Index Required="true" ParameterType="System.Int32"/>
<Item Output="true" ParameterType="Microsoft.Build.Framework.ITaskItem"/>
</ParameterGroup>
<Task>
<Code Type="Fragment" Language="cs">
<![CDATA[Item = Items[ Index ];]]>
</Code>
</Task>
</UsingTask>
<Target Name="Compare3">
<IndexItemGroup Items="#(Version)" Index="1">
<Output PropertyName="OldGitSHA" TaskParameter="Item"/>
</IndexItemGroup>
<PropertyGroup>
<SHAChanged>True</SHAChanged>
<SHAChanged Condition="'$(GitSHA)' == '$(OldGitSHA)'">False</SHAChanged>
</PropertyGroup>
<Message Text="OldGitSHA = $(OldGitSHA), changed = $(SHAChanged)" />
</Target>
I'm trying to create a reusable Target in msbuild, following the basic model outlined in How to invoke the same msbuild target twice?
I'm stuck trying to pass a property that I want interpreted as a list. I haven't found an example online that deals with this situation. As I understand it, the problem is that Properties is already treated as a list item, so it doesn't like having a list item passed in as well. Is there a way to get msbuild to pack and unpack the list correctly here?
Msbuild is complaining with:
error MSB4012: The expression "FilesToZip=#(Scripts)" cannot be used in this context. Item lists cannot be concatenated with other strings where an item list is expected. Use a semicolon to separate multiple item lists.
Here's an example caller:
<Target Name="BuildMigrationZip">
<MSBuild Projects="BuildZip.msbuild"
Targets="BuildZip"
Properties="FilesToZip=#(Scripts);OutputZipFile=$(MigrationPackageFilePath);OutputFolder=$(MigrationPackagePath);Flatten=true"/>
<Message Text="Created database migration zip: $(MigrationPackageFilePath)" Importance="high"/>
</Target>
And the base target:
<Target Name="BuildZip">
<MakeDir Directories="$(OutputFolder)"/>
<Zip Files="#(FilesToZip)"
ZipFileName="$(OutputZipFile)"
Flatten="$(Flatten)"
ParallelCompression="false" />
</Target>
I'm basically at the point of just going back to cut and paste for these, although I want to package up a number of zips here.
UPDATE: The same issue applies to setting Inputs on the reusable target. My question up to this point addresses the raw functionality, but it would be nice to keep dependencies working. So for example:
<Target Name="BuildZip"
Inputs="#(FilesToZip)"
Outputs="$(OutputZipFile)">
<MakeDir Directories="$(OutputFolder)"/>
<Zip Files="#(FilesToZip)"
ZipFileName="$(OutputZipFile)"
Flatten="$(Flatten)"
ParallelCompression="false" />
</Target>
They key is to pass the list around as a property. So when your Scripts list is defined as
<ItemGroup>
<Scripts Include="A"/>
<Scripts Include="B"/>
<Scripts Include="C"/>
</ItemGroup>
then you first convert it into a property (which just makes semicolon seperated items, but msbuild knows how to pass this via the Properties of the MSBuild target) then pass it to the target:
<Target Name="BuildMigrationZip">
<PropertyGroup>
<ScriptsProperty>#(Scripts)</ScriptsProperty>
</PropertyGroup>
<MSBuild Projects="$(MSBuildThisFile)" Targets="BuildZip"
Properties="FilesToZip=$(ScriptsProperty)" />
</Target>
(note I'm using $(MSBuildThisFile) here: you don't necessarily need to create seperate build files for every single target, in fact for small targets like yours it's much more convenient to put it in the same file)
Then in your destination target you turn the property into a list again:
<Target Name="BuildZip">
<ItemGroup>
<FilesToZipList Include="$(FilesToZip)"/>
</ItemGroup>
<Message Text="BuildZip: #(FilesToZipList)" />
</Target>
Output:
BuildZip: A;B;C
Update
When working with Inputs, you cannot pass #(FilesToZip) since that expands to nothing because is not a list: it's a property - which happens to be a number of semicolon-seperated strings. And as such, it is usable for Inputs you just have to expand it as the property it is i.e. $(FilesToZip):
<Target Name="BuildZip"
Inputs="$(FilesToZip)"
Outputs="$(OutputZipFile)">
...
</Target>
Output of second run:
BuildZip:
Skipping target "BuildZip" because all output files are up-to-date with respect to the input files.
I have a script that attempts to construct an ItemGroup out of all files in a certain directory while excluding files with certain names (regardless of extension).
The list of files to be excluded initially contains file extensions, and I am using Community Tasks' RegexReplace to replace the extensions with an asterisk. I then use this list in the item's Exclude attribute. For some reason the files do not get excluded properly, even though the list appears to be correct.
To try and find the cause I created a test script (below) which has two tasks: first one initialises two properties with the list of file patterns in two different ways. The second task prints both properties and the files resulting from using both these properties in the Exclude attribute.
The properties' values appear to be identical, however the resulting groups are different. How is this possible?
<Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003"
DefaultTargets="Init;Test" ToolsVersion="3.5">
<Import Project="$(MSBuildExtensionsPath)\MSBuildCommunityTasks\MSBuild.Community.Tasks.Targets"/>
<Target Name="Init">
<ItemGroup>
<OriginalFilenames Include="TestDir\SampleProj.exe"/>
<OriginalFilenames Include="TestDir\SampleLib1.dll"/>
</ItemGroup>
<RegexReplace Input="#(OriginalFilenames)" Expression="\.\w+$" Replacement=".*">
<Output TaskParameter="Output" ItemName="PatternedFilenames"/>
</RegexReplace>
<PropertyGroup>
<ExcludeFilesA>TestDir\SampleProj.*;TestDir\SampleLib1.*</ExcludeFilesA>
<ExcludeFilesB>#(PatternedFilenames)</ExcludeFilesB>
</PropertyGroup>
</Target>
<Target Name="Test">
<Message Text='ExcludeFilesA: $(ExcludeFilesA)' />
<Message Text='ExcludeFilesB: $(ExcludeFilesB)' />
<ItemGroup>
<AllFiles Include="TestDir\**"/>
<RemainingFilesA Include="TestDir\**" Exclude="$(ExcludeFilesA)"/>
<RemainingFilesB Include="TestDir\**" Exclude="$(ExcludeFilesB)"/>
</ItemGroup>
<Message Text="
**AllFiles**
#(AllFiles, '
')" />
<Message Text="
**PatternedFilenames**
#(PatternedFilenames, '
')" />
<Message Text="
**RemainingFilesA**
#(RemainingFilesA, '
')" />
<Message Text="
**RemainingFilesB**
#(RemainingFilesB, '
')" />
</Target>
</Project>
Output (reformatted somewhat for clarity):
ExcludeFilesA: TestDir\SampleProj.*;TestDir\SampleLib1.*
ExcludeFilesB: TestDir\SampleProj.*;TestDir\SampleLib1.*
AllFiles:
TestDir\SampleLib1.dll
TestDir\SampleLib1.pdb
TestDir\SampleLib2.dll
TestDir\SampleLib2.pdb
TestDir\SampleProj.exe
TestDir\SampleProj.pdb
PatternedFilenames:
TestDir\SampleProj.*
TestDir\SampleLib1.*
RemainingFilesA:
TestDir\SampleLib2.dll
TestDir\SampleLib2.pdb
RemainingFilesB:
TestDir\SampleLib1.dll
TestDir\SampleLib1.pdb
TestDir\SampleLib2.dll
TestDir\SampleLib2.pdb
TestDir\SampleProj.exe
TestDir\SampleProj.pdb
Observe that both ExcludeFilesA and ExcludeFilesB look identical, but the resulting groups RemainingFilesA and RemainingFilesB differ.
Ultimately I want to obtain the list RemainingFilesA using the pattern generated the same way ExcludeFilesB is generated. Can you suggest a way, or do I have to completely rethink my approach?
The true cause of this was revealed accidentally when a custom task threw an exception.
The actual value of ExcludeFilesA is TestDir\SampleProj.*;TestDir\SampleLib1.* like one might expect. However the actual value of ExcludeFilesB is TestDir\SampleProj.%2a;TestDir\SampleLib1.%2a.
Presumably Message unescapes the string before using it, but Include and Exclude do not. That would explain why the strings look the same but behave differently.
Incidentally, the execution order doesn't seem to have anything to do with this, and I'm pretty sure (following extensive experimentation) that everything gets executed and evaluated exactly in the order in which it appears in this script.
ItemGroups need to be evaluated before targets execution, and the PatternedFilenames ItemGroup is being created on the fly within its target container.
You could workaround this using the CreateItem task, which will ensure the PatternedFilenames scope throughout the execution:
<RegexReplace Input="#(OriginalFilenames)" Expression="\.\w+$" Replacement=".*">
<Output TaskParameter="Output" ItemName="PatternedFilenames_tmp"/>
</RegexReplace>
<CreateItem Include="#(PatternedFilenames_tmp)">
<Output TaskParameter="Include" ItemName="PatternedFilenames"/>
</CreateItem>