Filter an existing itemgroup so it only includes files that match some condition - msbuild

How do you filter an existing ItemGroup based on a specific condition, such as file extension or the item's metadata?
For this example, I'll use the file extension. I'm trying to filter the 'None' ItemGroup defined by VS so that my target can operate on all files of a given extension.
For example, the following may be defined:
<ItemGroup>
<None Include="..\file1.ext" />
<None Include="..\file2.ext" />
<None Include="..\file.ext2" />
<None Include="..\file.ext3" />
<None Include="..\file.ext4" />
</ItemGroup>
I want to filter the 'None' ItemGroup above so it only includes the ext extension. Note that I do not want to specify all the extensions to exclude, as they'll vary per project and I'm trying to make my target reusable without modification.
I've tried adding a Condition within a target:
<Target Name="Test">
<ItemGroup>
<Filtered
Include="#(None)"
Condition="'%(Extension)' == 'ext'"
/>
</ItemGroup>
<Message Text="None: '%(None.Identity)'"/>
<Message Text="Filtered: '%(Filtered.Identity)'"/>
</Target>
But sadly, it doesn't work. I get the following for output:
Test:
None: '..\file1.ext'
None: '..\file2.ext'
None: '..\file.ext2'
None: '..\file.ext3'
None: '..\file.ext4'
Filtered: ''

<ItemGroup>
<Filtered Include="#(None)" Condition="'%(Extension)' == '.ext'" />
</ItemGroup>

For advanced filtering I suggest using the RegexMatch from MSBuild Community Tasks.
In this Example we will Filter for Versionnumbers
<RegexMatch Input="#(Items)" Expression="\d+\.\d+\.\d+.\d+">
<Output ItemName ="ItemsContainingVersion" TaskParameter="Output" />
</RegexMatch>
Install MSBuild Community Tasks via Nuget: PM> Install-Package MSBuildTasks or download it here
Then Import it in your MSBuild Script:
<PropertyGroup>
<MSBuildCommunityTasksPath>..\.build\</MSBuildCommunityTasksPath>
</PropertyGroup>
<Import Project="$(MSBuildCommunityTasksPath)MsBuild.Community.Tasks.Targets" />

Related

MSBuild well-known item metadata and NuGet pack

I am trying to add custom files to a specific NuGet package (basically I need all output files included in the NuGet package since it serves as a tool for Chocolatey).
After some searching, I found this potential fix:
<PropertyGroup>
<TargetsForTfmSpecificBuildOutput>$(TargetsForTfmSpecificBuildOutput);GetToolsPackageFiles</TargetsForTfmSpecificBuildOutput>
</PropertyGroup>
<Target Name="GetToolsPackageFiles">
<ItemGroup>
<BuildOutputInPackage Include="$(OutputPath)\**\*.dll" />
<BuildOutputInPackage Include="$(OutputPath)\**\*.exe" />
</ItemGroup>
</Target>
Unfortunately, this won't work correctly for subdirectories, so I tried this:
<PropertyGroup>
<TargetsForTfmSpecificBuildOutput>$(TargetsForTfmSpecificBuildOutput);GetToolsPackageFiles</TargetsForTfmSpecificBuildOutput>
</PropertyGroup>
<Target Name="GetToolsPackageFiles">
<ItemGroup>
<BuildOutputInPackage Include="$(OutputPath)\**\*.dll">
<TargetPath>$([MSBuild]::MakeRelative('$(OutputPath)', %(FullPath)))</TargetPath>
</BuildOutputInPackage>
<BuildOutputInPackage Include="$(OutputPath)\**\*.exe">
<TargetPath>$([MSBuild]::MakeRelative('$(OutputPath)', %(FullPath)))</TargetPath>
</BuildOutputInPackage>
</ItemGroup>
</Target>
According to the docs, I should be able to use %(FullPath), but I am getting this error:
error MSB4184: The expression "[MSBuild]::MakeRelative(C:\Sour
ce\RepositoryCleaner\output\Release\RepositoryCleaner\netcoreapp3.1\, '')" cannot be evaluated. Parameter "path" cannot have zero length.
[C:\Source\RepositoryCleaner\src\RepositoryCleaner\RepositoryCleaner.csproj]
Any idea why the well-known items don't seem to work in this scenario?
Got a fix by specifying the item group outside the target and then using that instead.
<PropertyGroup>
<TargetsForTfmSpecificBuildOutput>$(TargetsForTfmSpecificBuildOutput);GetToolsPackageFiles</TargetsForTfmSpecificBuildOutput>
</PropertyGroup>
<ItemGroup>
<ToolDllFiles Include="$(OutputPath)\**\*.dll" />
<ToolExeFiles Include="$(OutputPath)\**\*.exe" />
</ItemGroup>
<Target Name="GetToolsPackageFiles">
<ItemGroup>
<BuildOutputInPackage Include="#(ToolDllFiles)">
<TargetPath>$([MSBuild]::MakeRelative('$(OutputPath)', %(ToolDllFiles.FullPath)))</TargetPath>
</BuildOutputInPackage>
<BuildOutputInPackage Include="#(ToolExeFiles)">
<TargetPath>$([MSBuild]::MakeRelative('$(OutputPath)', %(ToolExeFiles.FullPath)))</TargetPath>
</BuildOutputInPackage>
</ItemGroup>
</Target>

Deploying from MSBuild without overwriting specific files

So here's what I want to do:
I want a build script that will xcopy deploy build outputs for a legacy winform app to a given directory. I want to specify a list of files to not overwrite (some config files).
I would rather have the list of files to not overwrite be passed as a parameter than hard code them.
This seems to be really unexpectedly hard. Here's what I have so far:
<!-- A property that is passed a semicolon delimited list of file names -->
<PropertyGroup>
<ProtectedFiles/>
</PropertyGroup>
<--! An ItemGroup to pick up the files>
<ItemGroup>
<FilesToDelete Include=$(DeploymentTargetFolder)\*.* Exclude="#(ProtectedFiles->'$(DeployTargetFolder)\%(identity)')"
<ItemGroup/>
<--! the delete isn't working, so I will stop just with that to keep the code brief -->
<Delete Files="#(FilesToDelete)"/>
The delete just ignores the exclude files and deletes everything
Is there a better way to do this? It doesn't seem too crazy -- I just want to
Delete all files from the target directory, except for the config files
Copy all of the files from the build outputs to the target directory, without overwriting the config files.
The first problem with your particular markup appears to confuse MsBuild $(properties) with MsBuild %(items) and MsBuild #(itemgroups).
ProtectedFiles is a property:
<!-- A property that is passed a semicolon delimited list of file names -->
<PropertyGroup>
<ProtectedFiles/>
</PropertyGroup>
But it's being treated as an Item and wouldn't have any %item.metadata:
<--! An ItemGroup to pick up the files>
<ItemGroup>
<FilesToDelete Include=$(DeploymentTargetFolder)\*.* Exclude="#(ProtectedFiles->'$(DeployTargetFolder)\%(identity)')"
<ItemGroup/>
Save the following markup locally as "foo.xml", then call "msbuild.exe foo.xml" and observe the output:
<Project ToolsVersion="4.0" DefaultTargets="foo" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
<PropertyGroup>
<FilesProp>FileA.txt;FileB.txt</FilesProp>
</PropertyGroup>
<ItemGroup>
<ProtectedFiles Include="FileA.txt" />
<ProtectedFiles Include="FileA.txt" />
</ItemGroup>
<Target Name="foo">
<Message Importance="high" Text="ProtectedFiles ItemGroup: #(ProtectedFiles)" />
<Message Importance="high" Text="ProtectedFiles ItemGroup transform: #(ProtectedFiles->'%(Identity)')" />
<Message Importance="high" Text="FilesProp Property: $(FilesProp)" />
<Message Importance="high" Text="FilesProp Property: #(FilesProp->'%(FilesProp.Identity)')" />
</Target>
</Project>
Will yield the following output:
foo:
ProtectedFiles ItemGroup: FileA.txt;FileA.txt
ProtectedFiles ItemGroup transform: FileA.txt;FileA.txt
FilesProp Property: FileA.txt;FileB.txt
FilesProp Property:
If you're unable to change the design and need to convert a Property comprising a semi-colon delimited list of file paths, use the MsBuild <CreateItem /> task.
Add this markup to foo.xml occurring after the Foo target, then invoke msbuild again, but using the "bar" target (e.g. msbuild.exe foo.xml /t:bar)
<Target Name="bar">
<CreateItem Include="$(FilesProp)">
<Output TaskParameter="Include" ItemName="TheFiles"/>
</CreateItem>
<Message Text="TheFiles ItemGroup: #(TheFiles)" Importance="high" />
<Message Text="Output each item: %(TheFiles.Identity)" Importance="high" />
</Target>
Will yield the following output:
bar:
TheFiles ItemGroup: FileA.txt;FileB.txt
Output each item: FileA.txt
Output each item: FileB.txt
Next you should rethink some of your assumptions. I don't believe the file extension should be the determining factor when deciding which files to update, rather you should rely on MsBuild's ability to build tasks incrementally allowing it to perform a task only if the inputs are newer than the outputs. You can do this by using an MsBuild <Copy /> task configured to skip unchanged files.
Add this markup to the above Xml file, then modify the $(SourceFolder) and $(TargetFolder) to point to a source folder you'd like to copy recursively, and a destination folder to place the files. Build using "msbuild.exe foo.xml /t:Deployment" and observe the output.
<Target Name="Deployment">
<PropertyGroup>
<SourceFolder>c:\sourcefolder\</SourceFolder>
<TargetFolder>c:\destinationfolder\</TargetFolder>
</PropertyGroup>
<CreateItem Include="$(SourceFolder)\**\*.*">
<Output TaskParameter="Include" ItemName="FilesToCopy" />
</CreateItem>
<Copy SourceFiles="#(FilesToCopy)" DestinationFolder="$(TargetFolder)%(RecursiveDir)" SkipUnchangedFiles="true" />
</Target>
Without modifying any of the source files, run the command again and note that no files were copied.
Modify a file in the source folder, then run the command again. Notice that only the updated files were copied?
I hope this gets you on the right track.
There seems to be an already existing post, similar to this. Please check this Trying to exclude certain extensions doing a recursive copy (MSBuild)

MSBuild reference assembly not being included in build

I can build my project with the following command...
csc /reference:lib\Newtonsoft.Json.dll SomeSourceFile.cs
... but when I use this command...
msbuild MyProject.csproj
... with the following .csproj file my .dll reference isn't included. Any thoughts?
<PropertyGroup>
<AssemblyName>MyAssemblyName</AssemblyName>
<OutputPath>bin\</OutputPath>
</PropertyGroup>
<ItemGroup>
<Compile Include="SomeSourceFile.cs" />
</ItemGroup>
<ItemGroup>
<Reference Include="Newtonsoft.Json">
<HintPath>lib\Newtonsoft.Json.dll</HintPath>
</Reference>
</ItemGroup>
<Target Name="Build">
<MakeDir Directories="$(OutputPath)" Condition="!Exists('$(OutputPath)')" />
<Csc Sources="#(Compile)" OutputAssembly="$(OutputPath)$(AssemblyName).exe" />
</Target>
You didn't get your Reference group hooked up to the Csc task. Also the references the way you specified could not be used directly inside the task. Tasks that ship with MSBuild include ResolveAssemblyReference, that is able to transform short assembly name and search hints into file paths. You can see how it is used inside c:\Windows\Microsoft.NET\Framework64\v4.0.30319\Microsoft.Common.targets.
Without ResolveAssemblyReference, the simplest thing that you can do is to write it like this:
<PropertyGroup>
<AssemblyName>MyAssemblyName</AssemblyName>
<OutputPath>bin\</OutputPath>
</PropertyGroup>
<ItemGroup>
<Compile Include="SomeSourceFile.cs" />
</ItemGroup>
<ItemGroup>
<Reference Include="lib\Newtonsoft.Json.dll" />
</ItemGroup>
<Target Name="Build">
<MakeDir Directories="$(OutputPath)" Condition="!Exists('$(OutputPath)')" />
<Csc Sources="#(Compile)" References="#(Reference)" OutputAssembly="$(OutputPath)$(AssemblyName).exe" />
</Target>
Notice that Reference item specifies direct path to the referenced assembly.
What you've done is to overload the default Build target typically imported via Microsoft.CSharp.targets. In the default Build target, it takes the item array #(Compile), in which your .cs source files are resident, and also the #(Reference) array, among other things, and composites the proper call to the C# compiler. You've done no such thing in your own minimal Build target, which effectivly ignores the declaration of #(Reference) and only supplies #(Compile) to the Csc task.
Try adding the References="#(References)" attribute to the Csc task.

Add Custom Metadata to Already-defined ItemGroup from Another ItemGroup

I have the following:
<ItemGroup>
<Files Include="C:\Versioning\**\file.version" />
<ItemGroup>
<ReadLinesFromFile File="%(Files.Identity)">
<Output TaskParameter="Lines" ItemName="_Version"/>
</ReadLinesFromFile>
where each file.version file contains simply one line which is - you guessed it - a version of the form Major.Minor.Build.Revision.
I want to be able to associate each item in the Files ItemGroup with its _Version by adding the latter as metadata, so that I can do something like:
<Message Text="%(Files.Identity): %(Files.Version)" />
and have MSBuild print out a nice list of file-version associations.
Is this possible?
This can be achieved by using target batching to add your Version member to the metadata. This involves moving your ReadLinesFromFile operation to its own target, using the #(Files) ItemGroup as an input.
This causes the target to be executed for each item in your ItemGroup, allowing you to read the contents (i.e. version number) from each individual file and subsequently update that item to add the Version metadata:
<Project DefaultTargets="OutputFilesAndVersions" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
<ItemGroup>
<Files Include="C:\Versioning\**\file.version" />
</ItemGroup>
<Target Name="OutputFilesAndVersions" DependsOnTargets="RetrieveVersions">
<Message Text="#(Files->'%(Identity): %(Version)')" />
</Target>
<Target Name="RetrieveVersions" Inputs="#(Files)" Outputs="%(Files.Identity)">
<ReadLinesFromFile File="%(Files.Identity)">
<Output TaskParameter="Lines" PropertyName="_Version"/>
</ReadLinesFromFile>
<PropertyGroup>
<MyFileName>%(Files.Identity)</MyFileName>
</PropertyGroup>
<ItemGroup>
<Files Condition="'%(Files.Identity)'=='$(MyFileName)'">
<Version>$(_Version)</Version>
</Files>
</ItemGroup>
</Target>
</Project>

Additional paths in msbuild script

How to specify additional assembly reference paths for the MSBuild tasks?
I have following script so far, but can't figure out how to specify additional search paths.
<ItemGroup>
<ProjectsToBuild Include="..\Main\Main.sln" />
</ItemGroup>
<!-- The follwing paths should be added to reference search paths for the build tasks -->
<ItemGroup>
<MyAddRefPath Include="$(MSBuildProjectDirectory)\..\..\Build\Lib1" />
<MyAddRefPath Include="$(MSBuildProjectDirectory)\..\..\Build\Lib2" />
</ItemGroup>
<MSBuild
Projects="#(ProjectsToBuild)"
Properties="Configuration=Debug;OutputPath=$(BuildOutputPath)">
</MSBuild>
UPDATE:
Please show one complete working script which invokes original project, such as an SLN with multiple additional reference paths.
No suggestions on how to improve the project structure please.
I know how to build a good structure, but now it's the task of building an existing piece of crap.
I have finaly figured out how to do it:
<Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
<ItemGroup>
<ProjectsToBuild Include="ConsoleApplication1\ConsoleApplication1.csproj" />
</ItemGroup>
<ItemGroup>
<AdditionalReferencePaths Include="..\Build\ClassLibrary1" />
<AdditionalReferencePaths Include="..\Build\ClassLibrary2" />
</ItemGroup>
<PropertyGroup>
<BuildOutputPath>..\Build\ConsoleApplication1</BuildOutputPath>
</PropertyGroup>
<Target Name="MainBuild">
<PropertyGroup>
<AdditionalReferencePathsProp>#(AdditionalReferencePaths)</AdditionalReferencePathsProp>
</PropertyGroup>
<MSBuild
Projects="ConsoleApplication1\ConsoleApplication1.csproj"
Properties="ReferencePath=$(AdditionalReferencePathsProp);OutputPath=$(BuildOutputPath)"
>
</MSBuild>
</Target>
The property you want to modify is AssemblySearchPaths. See the ResolveAssemblyReference task more information.
<Target Name="AddToSearchPaths">
<CreateProperty Value="x:\path\to\assemblies;$(AssemblySearchPaths)">
<Output PropertyName="AssemblySearchPaths" TaskParameter="Value" />
</CreateProperty>
</Target>
Making use of item groups, as in your example, it would look like:
<Target Name="AddToSearchPaths">
<CreateProperty Value="#(MyAddRefPath);$(AssemblySearchPaths)">
<Output PropertyName="AssemblySearchPaths" TaskParameter="Value" />
</CreateProperty>
</Target>
Looking in %WINDIR%\Microsoft.NET\Framework\v2.0.50727\Microsoft.Common.targets, you can see that the ResolveAssemblyReference Task is executed as part of the ResolveAssemblyReferences target. Thus, you want the newly added target to modify the AssemblySearchPaths property before ResolveAssemblyReferences is executed.
You've stated that you want to be able to modify the assembly search paths without modifying the project files directly. In order to accomplish that requirement you need to set an environment variable that will override the AssemblySearchPaths. With this technique you will need to provide every assembly reference path used by all the projects in the solutions. (Modifying the projects or copies of the projects would be easier. See final comments.)
One technique is to create a batch file that runs your script at sets the environment variable:
set AssemblySearchPaths="C:\Tacos;C:\Burritos;C:\Chalupas"
msbuild whatever.msbuild
Another way is to define a PropertyGroup in your custom msbuild file (otherwise known as the "hook" needed to make this work):
<?xml version="1.0" encoding="utf-8"?>
<Project ToolsVersion="4.0" DefaultTargets="Build" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
<ItemGroup>
<ProjectsToBuild Include="..\Main\Main.sln" />
</ItemGroup>
<PropertyGroup>
<AssemblySearchPaths>$(MSBuildProjectDirectory)\..\..\Build\Lib1;$(MSBuildProjectDirectory)\..\..\Build\Lib2</AssemblySearchPaths>
</PropertyGroup>
<Target Name="Build">
<MSBuild Projects="#(ProjectsToBuild)" Properties="AssemblySearchPaths=$(AssemblySearchPaths);Configuration=Debug;OutputPath=$(OutputPath)" />
</Target>
</Project>
Now if it were me, and for whatever unexplained reason I couldn't modify the project files to include the updated references that I am going to build with, I would make copies of the project files, load them into the IDE, and correct the references in my copies. Synching the projects becomes a simple diff/merge operation which is automatic with modern tools like mercurial (heck I'm sure clearcase could manage it too).
...and remember that you don't need to use a target for this, you can use project-scoped properties or items, as...
<ItemGroup>
<MyAddRefPath Include="$(MSBuildProjectDirectory)\..\..\Build\Lib1" />
<MyAddRefPath Include="$(MSBuildProjectDirectory)\..\..\Build\Lib2" />
</ItemGroup>
<PropertyGroup>
<MyAddRefPath>$(MSBuildProjectDirectory)\..\..\Build\Lib3</MyAddRefPath>
<!-- add in the property path -->
<AssemblySearchPaths>$(MyAddRefPath);$(AssemblySearchPaths)</AssemblySearchPaths>
<!-- add in the item paths -->
<AssemblySearchPaths>#(MyAddRefPath);$(AssemblySearchPaths)</AssemblySearchPaths>
</PropertyGroup>
...and if you do need to do this in a target to pick up paths from a dynamically populated item group, use inline properties, not the CreateProperty task (if you are not stuck in v2.0)
<Target Name="AddToSearchPaths">
<PropertyGroup>
<!-- add in the item paths -->
<AssemblySearchPaths>#(MyDynamicAddRefPath);$(AssemblySearchPaths)</AssemblySearchPaths>
</PropertyGroup>
</Target>