csproj include (recursive) folders based on a list (batching) - msbuild

I want to add some files recursively to a single project that are located in folders inside my solution root.
Right now, I found a working solution to add those files by specifying each folder manually:
<ItemGroup>
<Content Include="..\Folder1\**\*">
<Link>Folder1\%(RecursiveDir)\%(Filename)%(Extension)</Link>
</Content>
<Content Include="..\Folder2\**\*">
<Link>Folder2\%(RecursiveDir)\%(Filename)%(Extension)</Link>
</Content>
<Content Include="..\Folder3\**\*">
<Link>Folder3\%(RecursiveDir)\%(Filename)%(Extension)</Link>
</Content>
</ItemGroup>
As those can be a handfull folders and they can be different depending on my solution, I want to have an easy list to define the folder names.
Something like this:
<PropertyGroup>
<DefinedResourceFolders>Folder1;Folder2;Folder3</DefinedResourceFolders>
</PropertyGroup>
This would also allow me to input this as a property directly when calling msbuild to extend the files that are going to the build output, maybe.
I tried to look into the MSBuild Batching documentation and several other online sources, but I could not figure it out. Most batching examples I found were working with Targets, not Includes into the solution items.
Is this possible? How would I define the <ItemGroup> for the content then?
P.S.: I don't care about wildcard issues with .csproj files when new files are added, etc. This will either be a single "Resources" project only containing those displayed files, or I am using my external .props file that I am importing into each .csproj file anyway.

Suppose you want to include all files in folders root/Folder1, root/Folder2 and root/Folder3 recursively.
This is how the builds (both VS and CLI) do what you want. However, VS will not show the files as part of the project.
<Project Sdk="Microsoft.NET.Sdk" InitialTargets="__AddBatchedContent;$(InitialTargets)">
<!-- the usual properties -->
<!-- You could of course define the folder array directly in an Item,
but this is what you wanted :-) -->
<PropertyGroup>
<Lookups>Folder1;Folder2;Folder3</Lookups>
</PropertyGroup>
<ItemGroup>
<LookupDir Include="$(Lookups)" />
</ItemGroup>
<Target Name="__AddBatchedContent">
<ItemGroup>
<!-- save the original source folder of the globbed file name in custom
metadata "Folder" so we can use it later as its output base folder -->
<__BatchedFiles Include="..\%(LookupDir.Identity)\**\*" Folder="%(LookupDir.Identity)" />
<Content Include="#(__BatchedFiles)" Link="%(Folder)\%(RecursiveDir)\%(Filename)%(Extension)" CopyToOutputDirectory="Always" />
</ItemGroup>
</Target>
</Project>
Note that if you want to put this in a .targets file to use this technique in multiple projects, you should replace the ..\ with $(MSBuildThisFileDirectory)..\ so that the path is relative to the (known) location of the .targets file instead of the (unknown) location of the importing project.
Thanks to #Ilya Kozhevnikov for the basis of this answer.
You are not the only one who would like this to be simpler :-): https://github.com/dotnet/msbuild/issues/3274

Related

NuGet Package Packing - Is it possible to copy files to a custom directory?

I'm trying to package a few files into a NuGet package, but the issue is that all of the files are sent to the "content" folder within the NuGet package by default when packaged. Normally this is okay, but for the JSON files I have in "ABCJsons" I'd like them to be sent to "content/NewFolderName".
In my example below, the first block is my AbcToolTester, which has all of its project files files being successfully sent to the content directory in the NuGet package. The second block, is where I attempted to copy all the json files with ABCLibrary (ABCLibrary has subfolders where the actual Jsons are located) to the destination folder "ABCJsons". I thought this would do the trick, but unfortunately the ABCJson files just get sent to the content folder along with all the other files.
<ItemGroup>
<Content Include="..\AbcToolTester\bin\Debug\netcoreapp3.1\**" Exclude="..\AbcToolTester\bin\Debug\netcoreapp3.1\*.pdb">
<IncludeInPackage>true</IncludeInPackage>
<CopyToOutputDirectory>Always</CopyToOutputDirectory>
</Content>
<ItemGroup>
<Target Name="CopyABCLibrary" AfterTargets="AfterBuild">
<ItemGroup>
<ABCJsonsInclude="..\..\tests\ABCLibrary\**\*.*"/>
</ItemGroup>
<Copy SourceFiles="#(ABCJsons)" DestinationFolder="$(TargetDir)\ABCJsons" SkipUnchangedFiles="true" />
</Target>
It's hard to tell if the NuGet package you are creating is actually dependent on AbcToolTester project because there are easier ways to package that. That's another question though.
For your actual issue, you can simplify the copying process while also telling it where to pack the files. Replace your CopyABCLibrary target with this:
<ItemGroup>
<Content Include="..\..\tests\ABCLibrary\**\*.*">
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
<Pack>true</Pack>
<PackagePath>ABCJsons\%(RecursiveDir)</PackagePath>
<!-- This line hides the items from showing in the solution explorer -->
<Visible>false</Visible>
</Content>
</ItemGroup>
This will put all those files into the root of the nuget package into the ABCJsons folder and preserve the directory structure. Change the path accordingly to put it somewhere else.

Build script to copy files from various source folder to various destination folder

The basic requirement is to copy the various files and folders from different solution/project directories to the single build_output folder(/subfolders).
Currently, I am doing this operation using the Robocopy commands. The only issue is my script is too long just using multiple Robocopy commands.
<Copy SourceFiles="$(Docs)\Manual.pdf" DestinationFolder="$(BuildPath)\Help"/>
<RoboCopy Source="$(Web1)" Destination="$(BuildPath)" Files="*.aspx" Options="/E"/>
<RoboCopy Source="$(Web1)\Images" Destination="$(BuildPath)\Images" Files="*.jpg;*.png" Options="/E"/>
<RoboCopy Source="$(Web2)\Images" Destination="$(BuildPath)\Images" Files="*.jpg;*.png" Options="/E"/>
<!--- 100s of such RoboCopy & Copy commands (note that in last two commands i need to copy from different sources to same destination -->
How this job is implemented in real enterprise applications, so that
the build script is concise and clear.
Is my thinking below is the way to approach the solution. If yes, can anybody provide me sample steps using MSBuild or CommandScript easily. (free to use any MSBuild extensions)
Define the mapping of the all source folders, file types (can be xyz.png/.png/.*) and the destination path.
Copy the files (Robocopy) using the above mentioned mappings using a single target or task)
Is there any other better way to do this problem?
Insights/Solution ???
I do exactly this sort of thing to stage build output for harvesting by the installer build. I have a custom targets file for consistent processing and have some msbuild property files with the item groups describing that needs to be done.
<ItemGroup Label="AcmeComponent1Payload">
<FileToHarvest Include="$(SourceRoot)AcmeProjects\ServerManager\$(Configuration)\**\*;
$(SourceRoot)Library\SQLServerCompact\**\*;
$(SourceRoot)Utility Projects\PropertyDataValidator\PropertyDataValidator\bin\$(Configuration)\PropertyDataValidator.*"
Exclude="$(SourceRoot)Server Manager Projects\AcmeServerManager\$(Configuration)\IntegrationTests.*;
$(SourceRoot)Server Manager Projects\AcmeServerManager\$(Configuration)\**\Microsoft.Practices.*.xml;
$(SourceRoot)Server Manager Projects\AcmeServerManager\$(Configuration)\obj\**\*;
$(SourceRoot)Server Manager Projects\AcmeServerManager\$(Configuration)\**\Microsoft.VisualStudio.*;
$(SourceRoot)Server Manager Projects\AcmeServerManager\$(Configuration)\**\Microsoft.Web.*;
$(SourceRoot)Utility Projects\PropertyDataValidator\PropertyDataValidator\bin\$(Configuration)\PropertyDataValidator.xml">
<Group>AcmeServerManager</Group>
<SubDir>Utilities\</SubDir>
</FileToHarvest>
</ItemGroup>
The custom targets file has the functionality to process it.
<Target Name="CopyFiles">
<Copy Condition="#(FileToHarvest)!=''"
SourceFiles="#(FileToHarvest)"
DestinationFiles="#(FileToHarvest->'$(OutputPath)\%(Group)\%(SubDir)%(RecursiveDir)%(Filename)%(Extension)')"
OverwriteReadOnlyFiles="true"
SkipUnchangedFiles="true" />
</Target>
You can make the properties file as simple or as complicated as you like. I use multiple ones and import them into the project file using wildcards.
Thanks #daughey, I got my first part working, where I need to copy from different sources to the same destination.
<!--Declare an ItemGroup that points to source Locations-->
<ItemGroup>
<ItemToCopy Include="$(Web1)\Audit"/>
<ItemToCopy Include="$(Utilities)\Service"/>
<ItemToCopy Include="$(Web1)\NET"/>
</ItemGroup>
<!--Declare an ItemGroup that points to destination Locations-->
<ItemGroup>
<DestLocations Include="$(BuildPath)" />
</ItemGroup>
<Target Name="CopyFiles">
<!-- Run the copy command to copy the item to your dest locations-->
<!-- The % sign says to use Batching. So Copy will be run for each unique source ItemToCopy(s) in the DestLocation.-->
<RemoveDir Directories="$(BuildPath)"/>
<Message Importance="high" Text="Deploy folder is $(BuildPath)"/>
<RoboCopy Source="%(ItemToCopy.FullPath)" Destination="$(BuildPath)" Files="*.dll"/>
</Target>
After struggling with Item Metadata for Task Batching, the second part is also working great.
Scenario: Copy files from list of source directories to output directory on their respective sub-folders
$(Web1)\Audit*.dll => $(BuildPath)\Audit*.dll
$(Utilities)\Service*.jpg => $(BuildPath)\Utilities\Service*.jpg
Solution
<!--Modify an ItemGroup with metadata-->
<ItemGroup>
<ItemToCopy Include="$(Web1)\Audit">
<ToPath>$(BuildPath)\Audit</ToPath>
<FileType>*.dll</FileType>
</ItemToCopy>
<ItemToCopy Include="$(Utilities)\Service">
<ToPath>$(BuildPath)\Utilities\Service</ToPath>
<FileType>*.jpg;*.bmp</FileType>
</ItemToCopy>
</ItemGroup>
<Target Name="CopyBatch">
<RoboCopy Source="%(ItemToCopy.Identity)" Destination="%(ItemToCopy.ToPath)" Files="%(ItemToCopy.Filetype)"/>
</Target>

How to change build Action for files using MSBuild

We are using NHibernate in our project and need all .hbm.xml files to be embedded resources.
To automate this boring task we added this target definition to our project file:
<Target Name="BeforeBuild">
<ItemGroup>
<EmbeddedResource Include="**\*.hbm.xml" />
</ItemGroup>
</Target>
It works only if all hbm.xml files have build action None. But in our project some files are manually set to Embedded Resource and there is no chance to change all of them right now.
So we get build error "The item was specified more than once in the "Resources" parameter and both items had the same value for the "LogicalName" metadata. Duplicate items are not supported by the "Resources" parameter unless they have different values for the "LogicalName" metadata."
Is it possible to write a target that changes build action only for .hbm.xml files with build action None (or NOT Embedded Resource)?
Will it work to first remove the ones already set to Embedded resource, before adding them all in?
<ItemGroup>
<EmbeddedResource Remove="**\*.hbm.xml" />
<EmbeddedResource Include="**\*.hbm.xml" />
</ItemGroup>

Can I use both wildcard and Link element inside the Compile element?

In the .csrpoj file, If I have
<Compile Include="c:\path\File1.cs">
<Link>Dir1\File1.cs</Link>
</Compile>
Then Visual Studio shows that file as a shortcut under Dir1 folder in the Solution Explorer.
If I have
<Compile Include="c:\path\*.cs"></Compile>
Then all .cs files show up as shortcuts in Solution Explorer at top level:
Is there a way to include all files in some folder and make then show up under a sub-folder? Omitting the filename in Link element does not work:
<Compile Include="c:\path\*.cs">
<Link>Dir1\</Link>
</Compile>
The files still show up at top level.
How do I include all files in a folder and still use the Link element? The reason I need this is, I need to include files from multiple folders and some of them have the same name. Two files at top level cannot have the same name.
Any other way to achieve this?
Others have suggested the using the Link attribute with placeholders, which indeed works. However, Microsoft has implemented a new attribute (which isn't in any of my code completion suggestions), named LinkBase, shown below.
<ItemGroup>
<Compile Include="..\SomeDirectory\*.cs" LinkBase="SomeDirectoryOfYourChoosing" />
</ItemGroup>
Sources:
https://github.com/Microsoft/msbuild/issues/2795
https://github.com/dotnet/sdk/pull/1246
Link Additional Files in Visual Studio
<Content Include="..\..\SomeDirectory\**\*.xml">
<Link>SomeLinkDirectoryOfYourChoosing\%(Filename)%(Extension)</Link>
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
</Content>
For the sake of others, here's the answer plus the comment from Dean and Miserable Variable, which I found useful:
I have two projects, and I need to include *.xsd from one in the other, without copying the files or having to update the referencing csproj file every time a new XSD is added to the first.
The solution was to add the following to the csproj file
<Content Include="..\BusinessLayer\Schemas\*.xsd">
<Link>Contract\Schemas\xxx.xsd</Link>
</Content>
Note xxx.xsd, you have to give a dummy filename in the Link element. It just gets replaced.
Also, you can include all sub folders with:
<Content Include="..\BusinessLayer\Schemas\**\*.xsd">
<Link>Contract\Schemas\ThisTextDoesntMatter</Link>
</Content>
And files of all type (useful for pulling in CSS/JS/Style folders from 3rd parties) with:
<Content Include="..\PresentationLayer\CustomerStyle\**\*.*">
<Link>CustomerStyle\placeHolder</Link>
</Content>
To include subfolders:
<ItemGroup>
<Compile Include="..\SomeExternalFolder\**\*.cs" LinkBase="YourProjectFolder" />
</ItemGroup>

MSBuild CopyTask: Copying the same file to multiple locations

Is there a way to get the CopyTask to copy the same file to multiple locations?
eg. I've generated an AssemblyInfo.cs file and want to copy it across to all my projects before building.
Check out the RoboCopy build task which is part of the Community Build Tasks library which you can find here. RoboCopy can copy one source file to multiple destinations.
On a side note: why don't you use one AssemblyInfo file on solution level and link to that in your projects if you need the same information in every project? Check out my accepted answer on this question: Automatic assembly version number management in VS2008
Right, well maybe I should attempt to do the things I want to do before asking for help :)
<ItemGroup>
<AssemblyInfoSource
Include="AssemblyInfo.cs;AssemblyInfo.cs" />
<AssemblyInfoDestination
Include="$(Destination1)\AssemblyInfo.cs;$(Destination2)\AssemblyInfo.cs" />
</ItemGroup>
<Copy SourceFiles="#(AssemblyInfoSource)" DestinationFiles="#(AssemblyInfoDestination)" />
I had a need to copy the contents of a directory to multiple locations, this is what I came up with that works. So I am posting it here ins case anyone else is in similar need and comes across this question like I did.
<!-- Create a list of the objects in PublishURL so it will copy to multiple directories -->
<ItemGroup>
<PublishUrls Include="$(PublishUrl)"/>
</ItemGroup>
<PropertyGroup>
<Files>$(OutputPath)\**\*</Files>
</PropertyGroup>
<!-- CopyNewFiles will copy all the files in $(OutputPath) to all the directories in the
in $(PublishUrl). $(PublishUrl) can be a single directory, or a list of directories
separated by a semicolon -->
<Target Name ="CopyNewFiles">
<!-- Get list of all files in the output directory; Cross product this with all
the output directories. -->
<CreateItem Include ="$(Files)"
AdditionalMetadata="RootDirectory=%(PublishUrls.FullPath)">
<Output ItemName ="OutputFiles" TaskParameter ="Include"/>
</CreateItem>
<Message Text="'#(OutputFiles)' -> '%(RootDirectory)\%(RecursiveDir)'"/>
<Copy SourceFiles="#(OutputFiles)"
DestinationFolder ="%(RootDirectory)\%(RecursiveDir)"/>
</Target>
If you want to copy AssemblyInfo.cs to Folders A and B you would set the property Files="AssemblyInfo.cs" and PublishUrls="A;B"
What makes this work is the extra metadata in the CreateItem task AdditionalMetadata="RootDirectory=%(PublishUrls.FullPath)" so for each files found in File it creates 1 entry for each item found in PublishUrls. In your case of a single file the equivelent in writing out the xml would be:
<ItemGroup>
<OutputFiles Include="AssemblyInfo.cs">
<RootDirectory>A</RootDirectory>
</OutputFiles>
<OutputFiles Include="AssemblyInfo.cs">
<RootDirectory>B</RootDirectory>
</OutputFiles>
</ItemGroup>
Now if you copied the contents of a folder that had files 1.txt and 2.txt copied to A and B the equivalent xml would be:
<ItemGroup>
<OutputFiles Include="1.txt">
<RootDirectory>A</RootDirectory>
</OutputFiles>
<OutputFiles Include="2.txt">
<RootDirectory>A</RootDirectory>
</OutputFiles>
<OutputFiles Include="1.txt">
<RootDirectory>B</RootDirectory>
</OutputFiles>
<OutputFiles Include="2.txt">
<RootDirectory>B</RootDirectory>
</OutputFiles>
</ItemGroup>