I have a target that copies source files to a staging directory, this happens for multiple projects. In a later project, those files are passed into a type of compiler.
Currently that staging directory is not cleaned, so when a source file is deleted, a stale copy lives on in the staging directory.
What's the best way to clean that staging directory of stale files without negatively impacting my incremental build times? (ie. without fully deleting the staging directory each time)
Ended up using this bit of script to clean up files which had been removed from the source, but still exist in the destination:
<Target Name="DeleteStaleFiles">
<!-- Remove stale files (files which have been deleted at the source but still exist in the destination). -->
<ItemGroup>
<SourceFiles Include="#(SourceFiles->'$(Destination)\%(RecursiveDir)%(Filename)%(Extension)')" />
<DestinationFiles Include="$(Destination)**\*.*" />
<StaleFiles Include="#(DestinationFiles)" Exclude="#(SourceFiles)" />
</ItemGroup>
<Message Text="Deleting Stale Files: #(StaleFiles, '%0A')" Condition="'#(StaleFiles->Count())' > 0"/>
<Delete Files="#(StaleFiles)"/>
</Target>
Related
I'm using MSBuild SDK style projects with VS 2019. I'm trying to run a custom file generation tool which depends on the output of the build of the current project. The files should be treated as if it was regular content for which CopyToOutputDirectory is set. In dependent projects I expect the files to be part of the output directory as well. The solution I now have works, but not from clean builds, which is obviously not acceptable.
I currently have this in the project file:
<Target Name="Generation" AfterTargets="AfterBuild">
<Exec Command="GeneratedFiles" />
<ItemGroup>
<Content Include="$(TargetDir)\GeneratedFiles.*.xml">
<TargetPath>GeneratedFiles\%(Filename)%(Extension)</TargetPath>
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
</Content>
</ItemGroup>
</Target>
This works, but only for non-clean builds.
The reason the Generation target doesn't work is because the logic that performs the copies, based on the presence and value of the CopyToOutputDirectory metadata, runs as part of the 'CoreBuild'. The Generation target has AfterTargets="AfterBuild". It runs after the build (and the copying) has already been completed.
The Generation target doesn't work for the first build and it doesn't work for subsequent incremental builds either.
The question description says that the files are copied for an incremental build. While it may be true that the files are copied, they can't be getting copied because of the Generation target. Without seeing the complete project file, I assume there is another place in the project where the CopyToOutputDirectory metadata is being set for the files.
To have the Generation target run after compilation and before files are copied, the following can be used:
<Target Name="Generation" AfterTargets="AfterCompile" BeforeTargets="GetCopyFilesToOutputDirectoryItems">
The GetCopyFilesToOutputDirectoryItems is publicly documented and it runs before another publicly documented target named CopyFilesToOutputDirectory. The CopyFilesToOutputDirectory is defined with a DependsOnTargets attribute which runs a set of targets that perform the actually copying. This means that the file copies are completed before the CopyFilesToOutputDirectory target is completed.
To ensure the correct order, BeforeTargets="GetCopyFilesToOutputDirectoryItems" is used and not BeforeTargets="CopyFilesToOutputDirectory".
If the scenario were different and <Exec Command="GeneratedFiles" /> didn't depend on the compilation step, i.e. the GeneratedFiles command didn't use the assembly being created by the project, then the Generation target could occur even earlier, e.g.
<Target Name="Generation" BeforeTargets="BeforeBuild">
Update - Execute a Target if ProjectReference has a specific Project
This update is a response to discussion in the comments.
Let's say we have a FooApp project and a BarLib project and FooApp depends on BarLib and needs to copy arbitrary files from the BarLib project directory.
FooApp has a ProjectReference to BarLib, e.g.
<ItemGroup>
<ProjectReference Include="..\BarLib\BarLib.csproj" />
</ItemGroup>
ProjectReference is an ItemGroup and we can check if it includes BarLib.
<Target Name="CopyFilesFromBar" BeforeTargets="BeforeBuild">
<PropertyGroup>
<!-- The name of the project to look for -->
<BarProjectName>BarLib</BarProjectName>
<!-- Use batching to check the ItemGroup -->
<!-- Save the project's directory because we will need it later -->
<BarProjectDirectory Condition="'%(ProjectReference.Filename)' == '$(BarProjectName)'">%(ProjectReference.Directory)</BarProjectDirectory>
<!-- Set up a boolean that indicates if the project was found or not -->
<HasProjectRefToBar>false</HasProjectRefToBar>
<HasProjectRefToBar Condition="$(FooProjectDirectory) != ''">true</HasProjectRefToBar>
</PropertyGroup>
<!-- Copy if the project was found in ProjectReference -->
<Copy SourceFiles="$(TargetDir)$(BarProjectDirectory)\bin\GeneratedFiles\*.*" DestinationFolder="$(OutputPath)GeneratedFiles" Condition="$(HasProjectRefToBar)" />
</Target>
This target could be defined once in a Directory.Build.targets file and shared across a solution.
If the generated files (in BarLib in the example scenario) don't change based on Configuration and Platform, consider using an output path location that doesn't change as in the example - 'bin\GeneratedFiles'. This makes it much easier for consuming projects. Otherwise keep all the projects in sync with regards to using the same Configuration and Platform values and the same $(OutputPath).
I have a nuget with a .targets file that tells the consuming project to copy all files within a "Dependencies" folder to the output directory.
<ItemGroup>
<Files Include="$(MSBuildThisFileDirectory)/../contentFiles/Dependencies/*.*" />
</ItemGroup>
<Target Name="CopyDependencies" AfterTargets="Build">
<Copy SourceFiles="#(Files)"
DestinationFolder="$(TargetDir)" />
</Target>
This nuget is consumed by two projects: Project A and Project B. For this question, let's say we have a System.Runtime.InteropServices.RuntimeInformation.dll that is one of the dependencies within this nuget. The output directory of Project A does not already have System.Runtime.InteropServices.RuntimeInformation.dll, so it gets copied to the output directory when the project is built. Project B however already contains System.Runtime.InteropServices.RuntimeInformation.dll in the output directory. This causes a runtime issue at startup since the targets file is trying to overwrite the existing DLL of the same name with the System.Runtime.InteropServices.RuntimeInformation.dll file from within the nuget (which is a dependency of other files within the output directory).
So, how can I adjust my .targets file to only copy in files that do not already exist within the output directory based on name, and not size or date modified?
There are several ways but probably the most succinct change to your example code would be the following:
<ItemGroup>
<Files Include="$(MSBuildThisFileDirectory)/../contentFiles/Dependencies/*.*" />
</ItemGroup>
<Target Name="CopyDependencies" AfterTargets="Build">
<Copy SourceFiles="#(Files)" DestinationFolder="$(TargetDir)" Condition="!Exists('$(TargetDir)/%(Filename)%(Extension)')" />
</Target>
The change is adding a Condition on the Copy that is using the metadata of the #(Files) collection to test that the file does not exist in $(TargetDir).
Because of the use of metadata, the Copy is a task batch. Essentially the #(Files) collection is divided into batches by %(Filename)%(Extension) and there is a separate Copy task invoked for each batch.
If there is a large number of files in the Dependencies folder, the following variant may provide better performance.
<ItemGroup>
<Files Include="$(MSBuildThisFileDirectory)/../contentFiles/Dependencies/*.*" />
</ItemGroup>
<Target Name="CopyDependencies" AfterTargets="Build">
<ItemGroup>
<FilesToCopy Include="#(Files)" Condition="!Exists('$(TargetDir)/%(Filename)%(Extension)')" />
</ItemGroup>
<Copy SourceFiles="#(FilesToCopy)" DestinationFolder="$(TargetDir)" />
</Target>
The task batching is moved to the definition of a new ItemGroup collection and the Copy task is invoked once for the set of files. The potential performance improvement is that the implementation of the Copy task tries to parallelize copies, which it can't do when invoked per file.
I'm trying to package a few files into a NuGet package, but the issue is that all of the files are sent to the "content" folder within the NuGet package by default when packaged. Normally this is okay, but for the JSON files I have in "ABCJsons" I'd like them to be sent to "content/NewFolderName".
In my example below, the first block is my AbcToolTester, which has all of its project files files being successfully sent to the content directory in the NuGet package. The second block, is where I attempted to copy all the json files with ABCLibrary (ABCLibrary has subfolders where the actual Jsons are located) to the destination folder "ABCJsons". I thought this would do the trick, but unfortunately the ABCJson files just get sent to the content folder along with all the other files.
<ItemGroup>
<Content Include="..\AbcToolTester\bin\Debug\netcoreapp3.1\**" Exclude="..\AbcToolTester\bin\Debug\netcoreapp3.1\*.pdb">
<IncludeInPackage>true</IncludeInPackage>
<CopyToOutputDirectory>Always</CopyToOutputDirectory>
</Content>
<ItemGroup>
<Target Name="CopyABCLibrary" AfterTargets="AfterBuild">
<ItemGroup>
<ABCJsonsInclude="..\..\tests\ABCLibrary\**\*.*"/>
</ItemGroup>
<Copy SourceFiles="#(ABCJsons)" DestinationFolder="$(TargetDir)\ABCJsons" SkipUnchangedFiles="true" />
</Target>
It's hard to tell if the NuGet package you are creating is actually dependent on AbcToolTester project because there are easier ways to package that. That's another question though.
For your actual issue, you can simplify the copying process while also telling it where to pack the files. Replace your CopyABCLibrary target with this:
<ItemGroup>
<Content Include="..\..\tests\ABCLibrary\**\*.*">
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
<Pack>true</Pack>
<PackagePath>ABCJsons\%(RecursiveDir)</PackagePath>
<!-- This line hides the items from showing in the solution explorer -->
<Visible>false</Visible>
</Content>
</ItemGroup>
This will put all those files into the root of the nuget package into the ABCJsons folder and preserve the directory structure. Change the path accordingly to put it somewhere else.
The basic requirement is to copy the various files and folders from different solution/project directories to the single build_output folder(/subfolders).
Currently, I am doing this operation using the Robocopy commands. The only issue is my script is too long just using multiple Robocopy commands.
<Copy SourceFiles="$(Docs)\Manual.pdf" DestinationFolder="$(BuildPath)\Help"/>
<RoboCopy Source="$(Web1)" Destination="$(BuildPath)" Files="*.aspx" Options="/E"/>
<RoboCopy Source="$(Web1)\Images" Destination="$(BuildPath)\Images" Files="*.jpg;*.png" Options="/E"/>
<RoboCopy Source="$(Web2)\Images" Destination="$(BuildPath)\Images" Files="*.jpg;*.png" Options="/E"/>
<!--- 100s of such RoboCopy & Copy commands (note that in last two commands i need to copy from different sources to same destination -->
How this job is implemented in real enterprise applications, so that
the build script is concise and clear.
Is my thinking below is the way to approach the solution. If yes, can anybody provide me sample steps using MSBuild or CommandScript easily. (free to use any MSBuild extensions)
Define the mapping of the all source folders, file types (can be xyz.png/.png/.*) and the destination path.
Copy the files (Robocopy) using the above mentioned mappings using a single target or task)
Is there any other better way to do this problem?
Insights/Solution ???
I do exactly this sort of thing to stage build output for harvesting by the installer build. I have a custom targets file for consistent processing and have some msbuild property files with the item groups describing that needs to be done.
<ItemGroup Label="AcmeComponent1Payload">
<FileToHarvest Include="$(SourceRoot)AcmeProjects\ServerManager\$(Configuration)\**\*;
$(SourceRoot)Library\SQLServerCompact\**\*;
$(SourceRoot)Utility Projects\PropertyDataValidator\PropertyDataValidator\bin\$(Configuration)\PropertyDataValidator.*"
Exclude="$(SourceRoot)Server Manager Projects\AcmeServerManager\$(Configuration)\IntegrationTests.*;
$(SourceRoot)Server Manager Projects\AcmeServerManager\$(Configuration)\**\Microsoft.Practices.*.xml;
$(SourceRoot)Server Manager Projects\AcmeServerManager\$(Configuration)\obj\**\*;
$(SourceRoot)Server Manager Projects\AcmeServerManager\$(Configuration)\**\Microsoft.VisualStudio.*;
$(SourceRoot)Server Manager Projects\AcmeServerManager\$(Configuration)\**\Microsoft.Web.*;
$(SourceRoot)Utility Projects\PropertyDataValidator\PropertyDataValidator\bin\$(Configuration)\PropertyDataValidator.xml">
<Group>AcmeServerManager</Group>
<SubDir>Utilities\</SubDir>
</FileToHarvest>
</ItemGroup>
The custom targets file has the functionality to process it.
<Target Name="CopyFiles">
<Copy Condition="#(FileToHarvest)!=''"
SourceFiles="#(FileToHarvest)"
DestinationFiles="#(FileToHarvest->'$(OutputPath)\%(Group)\%(SubDir)%(RecursiveDir)%(Filename)%(Extension)')"
OverwriteReadOnlyFiles="true"
SkipUnchangedFiles="true" />
</Target>
You can make the properties file as simple or as complicated as you like. I use multiple ones and import them into the project file using wildcards.
Thanks #daughey, I got my first part working, where I need to copy from different sources to the same destination.
<!--Declare an ItemGroup that points to source Locations-->
<ItemGroup>
<ItemToCopy Include="$(Web1)\Audit"/>
<ItemToCopy Include="$(Utilities)\Service"/>
<ItemToCopy Include="$(Web1)\NET"/>
</ItemGroup>
<!--Declare an ItemGroup that points to destination Locations-->
<ItemGroup>
<DestLocations Include="$(BuildPath)" />
</ItemGroup>
<Target Name="CopyFiles">
<!-- Run the copy command to copy the item to your dest locations-->
<!-- The % sign says to use Batching. So Copy will be run for each unique source ItemToCopy(s) in the DestLocation.-->
<RemoveDir Directories="$(BuildPath)"/>
<Message Importance="high" Text="Deploy folder is $(BuildPath)"/>
<RoboCopy Source="%(ItemToCopy.FullPath)" Destination="$(BuildPath)" Files="*.dll"/>
</Target>
After struggling with Item Metadata for Task Batching, the second part is also working great.
Scenario: Copy files from list of source directories to output directory on their respective sub-folders
$(Web1)\Audit*.dll => $(BuildPath)\Audit*.dll
$(Utilities)\Service*.jpg => $(BuildPath)\Utilities\Service*.jpg
Solution
<!--Modify an ItemGroup with metadata-->
<ItemGroup>
<ItemToCopy Include="$(Web1)\Audit">
<ToPath>$(BuildPath)\Audit</ToPath>
<FileType>*.dll</FileType>
</ItemToCopy>
<ItemToCopy Include="$(Utilities)\Service">
<ToPath>$(BuildPath)\Utilities\Service</ToPath>
<FileType>*.jpg;*.bmp</FileType>
</ItemToCopy>
</ItemGroup>
<Target Name="CopyBatch">
<RoboCopy Source="%(ItemToCopy.Identity)" Destination="%(ItemToCopy.ToPath)" Files="%(ItemToCopy.Filetype)"/>
</Target>
There are some files residing in other directories that, I would like to copy to project folder automatically before build and publishing.
After some research and experimentation, I have come up with the following .csproj file.
<Project Sdk="Microsoft.NET.Sdk.Web">
<PropertyGroup>
<TargetFramework>netcoreapp2.1</TargetFramework>
<RuntimeFrameworkVersion>2.1.4</RuntimeFrameworkVersion>
<TieredCompilation>true</TieredCompilation>
<PreserveCompilationContext>true</PreserveCompilationContext>
</PropertyGroup>
<ItemGroup>
<APIDefinition Include="D:\SomePlace\*.API.*.yaml" />
</ItemGroup>
<Target Name="CopyFiles" BeforeTargets="Compile;Build;Publish">
<Copy SourceFiles="#(APIDefinition)" DestinationFolder="wwwroot" />
<Copy SourceFiles="D:\SomePlaceElse\BaseAPISettings.json" DestinationFolder="$(MSBuildProjectDirectory)" />
</Target>
<ItemGroup>
<Compile Remove="wwwroot\**\*;node_modules;bower_components" />
<None Update="**.user;**.vspscc">
<CopyToPublishDirectory>Never</CopyToPublishDirectory>
</None>
</ItemGroup>
<ItemGroup>
<PackageReference Include="Microsoft.AspNetCore.App" Version="2.1.4" />
</ItemGroup>
</Project>
Here I have defined CopyFiles target, which should be run before the targets I've placed there. This target uses Copy task to copy YAML format API definition files and base API settings to the project directory.
This works well during build, publish etc. Also, if I delete the local file in the IDE, it instantly recopies it from the source.
Sometimes I make changes to these files between debugging sessions. Then, when I start debugging from Visual Studio, since the project files aren't changed, obviously the already built project is run.
Since the project is not built, my copy tasks are not triggered, and I end up with stale files during debuging.
Is there anything I can do to have my Copy tasks triggered, when I do "Start Debugging F5" in the IDE, regardless of the project build state ?
P.S. : I'm using Visual Studio 2017 15.8.5 and targeting .NET Core 2.1.4 runtime, if it makes any difference.
To integrate fully into the up-to-date check of the project system inside Visual Studio, I susggest the following changes:
Make the items' source and target paths known before
Register them to the up-to-date check system. (Also needs a hack to make sure the project source code is recompiled so that the output will have a newer time stamp)
Make the MSBuild target itself incremental. This also helps for command-line builds when the files don't have to be copied.
The complete changes look like this:
<ItemGroup>
<CustomCopyFile Include="..\TestFiles\*.API.*.yaml"
TargetPath="wwwroot\%(Filename)%(Extension)" />
<CustomCopyFile Include="..\TestFiles\BaseAPISettings.json"
TargetPath="%(Filename)%(Extension)" />
<UpToDateCheckInput Include="#(CustomCopyFile)" />
<UpToDateCheckBuild Include="#(CustomCopyFile->'%(TargetPath)')"
Original="#(CustomCopyFile)" />
<CustomAdditionalCompileInputs Include="#(CustomCopyFile->'%(TargetPath)')" />
</ItemGroup>
<Target Name="CopyFiles"
BeforeTargets="BeforeBuild;BeforePublish"
Inputs="#(CustomCopyFile)"
Outputs="#(CustomCopyFile->'%(TargetPath)')">
<Copy SourceFiles="#(CustomCopyFile)"
DestinationFiles="#(CustomCopyFile->'%(TargetPath)')" />
</Target>
CustomCopyFile now collects all the source files and we put the expected destination file name into the TargetPath metadata.
UpToDateCheckInput items tell Visual Studio to rebuild the project if one of these items change.
UpToDateCheckBuild items instruct Visual Studio to only check these items against special source items. This is redundant for this example project but may be helpful if the target path wasn't inside the project directory but some intermediate output (obj..) folder and no re-evaluation would see these new files. It would also be helpful if the files were also modified as part of processing (e.g. replacing variables inside the files).
CustomAdditionalCompileInputs is a hack here since the items are copied to the project folder and are considered to be "inputs to the output" automatically.. So we force the project to recompile if our source files change. If we don't do so, it would never consider the project up-to-date after a change to the source yaml files since they would be newer than the compiled app.dll file.