Determining outputs of a ProjectReference in MSBuild without triggering redundant rebuilds - msbuild

As part of a solution containing many projects, I have a project that references (via a <ProjectReference> three other projects in the solution, plus some others). In the AfterBuild, I need to copy the outputs of 3 specific dependent projects to another location.
Via various SO answers, etc. the way I settled on to accomplish that was:
<MSBuild
Projects="#(ProjectReference)"
Targets="Build"
BuildInParallel="true"
Condition="'%(Name)'=='ProjectA' OR '%(Name)'=='ProjectB' OR '%(Name)'=='ProjectC'">
<Output TaskParameter="TargetOutputs" ItemName="DependentAssemblies" />
</MSBuild>
<Copy SourceFiles="#(DependentAssemblies)" DestinationFolder="XX" SkipUnchangedFiles="true" />
However, I ran into problems with this. The <MSBuild step's IncrementalClean task ends up deleting a number of the outputs of ProjectC. When running this under VS2008, a build.force file being deposited in the obj/Debug folder of ProjectC which then triggers ProjectC getting rebuilt if I do a Build on the entire solution if the project containing this AfterBuild target, whereas if one excludes this project from the build, it [correctly] doesn't trigger a rebuild of ProjectC (and critically a rebuild of all dependents of ProjectC). This may be VS-specific trickery in this case which would not occur in the context of a TeamBuild or other commandline MSBuild invocation (but the most common usage will be via VS so I need to resolve this either way)
The dependent projects (and the rest of the solution in general) have all been created interactively with VS, and hence the ProjectRefences contain relative paths etc. I've seen mention of this being likely to causing issues - but without a full explanation of why, or when it'll be fixed or how to work around it. In other words, I'm not really interested in e.g. converting the ProjectReference paths to absolute paths by hand-editing the .csproj.
While it's entirely possible I'm doing something stupid and someone will immediately point out what it is (which would be great), be assured I've spent lots of time poring over /v:diag outputs etc. (although I havent tried to build a repro from the ground up - this is in the context of a relatively complex overall build)

As noted in my comment, calling GetTargetPath on the referenced project only returns the Primary output assembly of that project. To get all the referenced copy-local assemblies of the referenced project it's a bit messier.
Add the following to each project that you are referencing that you want to get the CopyLocals of:
<Target
Name="ComputeCopyLocalAssemblies"
DependsOnTargets="ResolveProjectReferences;ResolveAssemblyReferences"
Returns="#(ReferenceCopyLocalPaths)" />
My particular situation is that I needed to recreate the Pipeline folder structure for System.AddIn in the bin folder of my top level Host project. This is kinda messy and I was not happy with the MSDN suggested solutions of mucking with the OutputPath - as that breaks on our build server and prevents creating the folder structure in a different project (eg a SystemTest)
So along with adding the above target (using a .targets import), I added the following to a .targets file imported by each "host" that needs the pipeline folder created:
<Target
Name="ComputePipelineAssemblies"
BeforeTargets="_CopyFilesMarkedCopyLocal"
Outputs="%(ProjectReference.Identity)">
<ItemGroup>
<_PrimaryAssembly Remove="#(_PrimaryAssembly)" />
<_DependentAssemblies Remove="#(_DependentAssemblies)" />
</ItemGroup>
<!--The Primary Output of the Pipeline project-->
<MSBuild Projects="%(ProjectReference.Identity)"
Targets="GetTargetPath"
Properties="Configuration=$(Configuration)"
Condition=" '%(ProjectReference.PipelineFolder)' != '' ">
<Output TaskParameter="TargetOutputs"
ItemName="_PrimaryAssembly" />
</MSBuild>
<!--Output of any Referenced Projects-->
<MSBuild Projects="%(ProjectReference.Identity)"
Targets="ComputeCopyLocalAssemblies"
Properties="Configuration=$(Configuration)"
Condition=" '%(ProjectReference.PipelineFolder)' != '' ">
<Output TaskParameter="TargetOutputs"
ItemName="_DependentAssemblies" />
</MSBuild>
<ItemGroup>
<ReferenceCopyLocalPaths Include="#(_PrimaryAssembly)"
Condition=" '%(ProjectReference.PipelineFolder)' != '' ">
<DestinationSubDirectory>%(ProjectReference.PipelineFolder)</DestinationSubDirectory>
</ReferenceCopyLocalPaths>
<ReferenceCopyLocalPaths Include="#(_DependentAssemblies)"
Condition=" '%(ProjectReference.PipelineFolder)' != '' ">
<DestinationSubDirectory>%(ProjectReference.PipelineFolder)</DestinationSubDirectory>
</ReferenceCopyLocalPaths>
</ItemGroup>
</Target>
I also needed to add the required PipelineFolder meta data to the actual project references. For example:
<ProjectReference Include="..\Dogs.Pipeline.AddInSideAdapter\Dogs.Pipeline.AddInSideAdapter.csproj">
<Project>{FFCD0BFC-5A7B-4E13-9E1B-8D01E86975EA}</Project>
<Name>Dogs.Pipeline.AddInSideAdapter</Name>
<Private>False</Private>
<PipelineFolder>Pipeline\AddInSideAdapter\</PipelineFolder>
</ProjectReference>

Your original solution should work simply by changing
Targets="Build"
to
Targets="GetTargetPath"
The GetTargetPath target simply returns the TargetPath property and doesn't require building.

You may protect your files in ProjectC if you call a target like this first:
<Target Name="ProtectFiles">
<ReadLinesFromFile File="obj\ProjectC.csproj.FileListAbsolute.txt">
<Output TaskParameter="Lines" ItemName="_FileList"/>
</ReadLinesFromFile>
<CreateItem Include="#(_DllFileList)" Exclude="File1.sample; File2.sample">
<Output TaskParameter="Include" ItemName="_FileListWitoutProtectedFiles"/>
</CreateItem>
<WriteLinesToFile
File="obj\ProjectC.csproj.FileListAbsolute.txt"
Lines="#(_FileListWitoutProtectedFiles)"
Overwrite="true"/>
</Target>

My current workaround is based on this SO question, i.e, I have:
<ItemGroup>
<DependentAssemblies Include="
..\ProjectA\bin\$(Configuration)\ProjectA.dll;
..\ProjectB\bin\$(Configuration)\ProjectB.dll;
..\ProjectC\bin\$(Configuration)\ProjectC.dll">
</DependentAssemblies>
</ItemGroup>
This however will break under TeamBuild (where all the outputs end up in one directory), and also if the names of any of the outputs of the dependent projects change.
EDIT: Also looking for any comments on whether there's a cleaner answer for how to make the hardcoding slightly cleaner than:
<PropertyGroup>
<_TeamBuildingToSingleOutDir Condition="'$(TeamBuildOutDir)'!='' AND '$(CustomizableOutDir)'!='true'">true</_TeamBuildingToSingleOutDir>
</PropertyGroup>
and:
<ItemGroup>
<DependentAssemblies
Condition="'$(_TeamBuildingToSingleOutDir)'!='true'"
Include="
..\ProjectA\bin\$(Configuration)\ProjectA.dll;
..\ProjectB\bin\$(Configuration)\ProjectB.dll;
..\ProjectC\bin\$(Configuration)\ProjectC.dll">
</DependentAssemblies>
<DependentAssemblies
Condition="'$(_TeamBuildingToSingleOutDir)'=='true'"
Include="
$(OutDir)\ProjectA.dll;
$(OutDir)\ProjectB.dll;
$(OutDir)\ProjectC.dll">
</DependentAssemblies>
</ItemGroup>

Related

.Net Core MS Build Package Creation Lifecycle

MSBuild quite powerful and important tool, but sometimes configuration of it is the same hard as making a spaceship or flying to Mars. Answers on next questions can help makes it a little bit more usefull and developer friendly:
- How the MSBuild create a Package(zip)?
- Does it use some temporary folders or direct from builds output ?
- Why can the output folder content differ from a package content
- What events can be used to add some custom logic?
The answers to this question could help me with the next long story short:
I'm working with a .Net Core solution, that has the dependency on some other solutions DLLs files( that are not referenced directly, but required to be in the solution folder). On post-build event CL there is the option added as an entry point "\TaskEP" that is a starting point of the next pipeline :
<PropertyGroup>
<PipelineCopyAllFilesToOneFolderForMsdeployDependsOn>
TaskEP;
Task2;
Task3;
$(PipelineCopyAllFilesToOneFolderForMsdeployDependsOn);
</PipelineCopyAllFilesToOneFolderForMsdeployDependsOn>
<RunPostBuildEvent>Always</RunPostBuildEvent>
Tasks are mostly done for add extra DLLs to package and in general looks like :
<Target Name="TaskEP" Condition="$(SomeConditions) == 'True'">
<Message Importance="high" Text="TaskEP: Copying some files..." />
<ItemGroup>
<ItemsName Include="$(MSBuildProjectDirectory)\..\..somepath..\Extra.dll;....dll" />
<FilesForPackagingFromProject Include="%(ItemsName.Identity)">
<DestinationRelativePath>bin\%(RecursiveDir)%(Filename)%(Extension)</DestinationRelativePath>
</FilesForPackagingFromProject>
</ItemGroup>
</Target>
Basically, for me, this instruction to MSBuild to add "Extra.dll" to output package.
Pubxml is quite usual :
<PropertyGroup>
<WebPublishMethod>Package</WebPublishMethod>
<LastUsedBuildConfiguration>Release</LastUsedBuildConfiguration>
<LastUsedPlatform>Any CPU</LastUsedPlatform>
<SiteUrlToLaunchAfterPublish />
<LaunchSiteAfterPublish>True</LaunchSiteAfterPublish>
<_PackagePathShortened Condition="'$(_PackagePathShortened)' == ''">SuperWebSite</_PackagePathShortened>
<ExcludeApp_Data>False</ExcludeApp_Data>
<PackageLocation>..\..\..\..\BIN\WEB\Release\Publish\Super.WebApplication.zip</PackageLocation>
<PackageAsSingleFile>true</PackageAsSingleFile>
<TargetFramework>netcoreapp2.1</TargetFramework> ....
And one instruction that replaces the long long path in the package to Short and defined one:
<Target Name="AddReplaceRuleForAppPath" BeforeTargets="BeforePublish">
<Message Text="Adding replace rules for application path '$(PublishIntermediateOutputPath)' replace with '$(_PackagePathShortened)'" Importance="high" />
<EscapeTextForRegularExpressions Text="$(PublishIntermediateOutputPath)">
<Output PropertyName="_PackagePathRegex" TaskParameter="Result" />
</EscapeTextForRegularExpressions>
<!-- Add a replace rule for VSMSDeploy resp. MSdeploy to update the path -->
<ItemGroup>
<MsDeployReplaceRules Include="replaceFullPath">
<Match>$(_PackagePathRegex)</Match>
<Replace>$(_PackagePathShortened)</Replace>
</MsDeployReplaceRules>
</ItemGroup>
I also duplicated the same "Copy" tasks here in .pubxml file , play with different events "AfterBuild", "BeforePublish" but seems that all of the just ignoring. I can see the "Extra" DLLs files in the build output directory and all Info Messages on publishing, but not in the final "Super.WebApplication.zip" file !
The solution I found to copy extra files in .Net Core is taken from this documentation
<ItemGroup>
<DotnetPublishFiles Include="$(MSBuildProjectDirectory)\..\..somepath..\Extra.dll;....dll" >
<DestinationRelativePath>bin\x86\%(RecursiveDir)%(Filename)%(Extension)</DestinationRelativePath>
</DotnetPublishFiles>
</ItemGroup>

how to add files to a projects output files from an MSBuild Task

Given an MSBuild Task that runs in AfterTargets="AfterCompile" and produces some files how do you get those files to be included in the current projects output so that the files will be copied to the bin directory of any projects referencing that project?
I have no guarantees that this is the right solution but it seems to work:
<Target Name="MyTarget" AfterTargets="AfterCompile">
<PropertyGroup>
<MyInput>D:\1.txt</MyInput>
<MyOutput>$([System.IO.Path]::GetFullPath('$(MSBuildThisFileDirectory)$(OutDir)\1.txt'))</MyOutput>
</PropertyGroup>
<Copy SourceFiles="$(MyInput)" DestinationFolder="$(OutDir)" SkipUnchangedFiles="true" />
<ItemGroup>
<AllItemsFullPathWithTargetPath Include="$(MyOutput)">
<TargetPath>1.txt</TargetPath>
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
</AllItemsFullPathWithTargetPath>
</ItemGroup>
</Target>
The relevant logic is here:
http://sourceroslyn.io/#MSBuildTarget=GetCopyToOutputDirectoryItems
http://sourceroslyn.io/#MSBuildItem=AllItemsFullPathWithTargetPath
Basically we rely on the fact that to determine the list of files to copy from dependent projects MSBuild calls the GetCopyToOutputDirectoryItems target of the dependent projects and uses its output (which is AllItemsFullPathWithTargetPath).
By adding ourselves to AllItemsFullPathWithTargetPath at the last minute we get picked up when a dependent project calls us.
Thank you, Kirill. That was an excellent answer and it helped me when trying to copy ETW manifest files from a different project's output. Below is the final output.
Since I've simply expanded upon Kirill's answer, I do not expect this answer to be accepted. I post this here in the hope it helps someone else.
<Target Name="IncludeEtwFilesInOutput"
BeforeTargets="GetCopyToOutputDirectoryItems">
<PropertyGroup>
<EtwManifestFile>$([System.IO.Path]::GetFullPath('$(MSBuildThisFileDirectory)$(OutDir)\My.etwManifest.man'))</EtwManifestFile>
<EtwResourceFile>$([System.IO.Path]::GetFullPath('$(MSBuildThisFileDirectory)$(OutDir)\My.etwManifest.dll'))</EtwResourceFile>
</PropertyGroup>
<ItemGroup>
<AllItemsFullPathWithTargetPath Include="$(EtwManifestFile)">
<TargetPath>My.etwManifest.man</TargetPath>
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
</AllItemsFullPathWithTargetPath>
<AllItemsFullPathWithTargetPath Include="$(EtwResourceFile)">
<TargetPath>My.etwManifest.dll</TargetPath>
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
</AllItemsFullPathWithTargetPath>
</ItemGroup>
</Target>

Issue with using MSBuild to build and copy all outputs to a common folder

We are trying to write a msbuild script that will build the solution and copy over all the compiled binaries and dependencies over to a specific output folder. While the build script that we have does build and copy over the binaries to a common folder, but we are not getting the dependencies copied.
This probably has to do with the way we have used the msbuild task to build the solution and we are accepting the targetoutputs of the task into an itemgroup and iterating over the item group to copy all the compiled dlls and exes over to a common folder. But this is not including the dependency dlls which gets placed into the individual bin folder of each project.
<Project ToolsVersion="4.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
<PropertyGroup>
<ParentSolutionFile />
</PropertyGroup>
<ItemGroup>
<Assemblies Include="*.dll, *.exe" />
</ItemGroup>
<Target Name="BuildAll">
<CombinePath BasePath="$(MSBuildProjectDirectory)" Paths="Source\Solutions\xxx.sln">
<Output TaskParameter="CombinedPaths" PropertyName="ParentSolutionFile" />
</CombinePath>
<Message Text="$(ParentSolutionFile)" />
<MSBuild Projects="$(ParentSolutionFile)">
<Output TaskParameter="TargetOutputs" ItemName="Assemblies" />
</MSBuild>
<Message Text="%(Assemblies.Identity)" />
<Copy SourceFiles="%(Assemblies.Identity)" DestinationFolder="$(MSBuildProjectDirectory)\Binary" OverwriteReadOnlyFiles="True" SkipUnchangedFiles="True" />
</Target>
What will be the preferred way to copy over all the binaries along with the necessary dependencies to a common output folder?
Does not overriding OutputPath do the trick alone?
<MSBuild Projects="$(ParentSolutionFile)" Properties="OutputPath=$(MSBuildProjectDirectory)\Binary">
<Output TaskParameter="TargetOutputs" ItemName="Assemblies" />
</MSBuild>
And leave out the copy task alltogether?
The build process will place the final result in the directory represented by OutputPath - at least if you are building c# projects. For C/C++ the internal structure and variable names are completely different.
Thus, in theory, you could pass the OutputPath in the MsBuild-task that builds the solution.
<MsBuild Projects="$(ParentSolutionFile)"
Properties="OutputPath=$(MSBuildProjectDirectory)\Binary"/>
However, the csproj-files will overwrite that value unconditionally with the following code:
<PropertyGroup Condition=" '$(Configuration)|$(Platform)' == 'Debug|AnyCPU' ">
<DebugSymbols>true</DebugSymbols>
<OutputPath>bin\Debug\</OutputPath>
I have solved this by injecting my own build system in each and every csproj-file.
<Import Project="$(MSBuildToolsPath)\Microsoft.CSharp.targets" />
<Import Project="..\..\..\build\buildsystem.targets" />
The path is relative to the csproj-file. An absolute path is fine too, or a variable. The trick is to make it work on all dev machines as well as the build agents.
Now, in buildsystem.targets, simply redefine OutputPath as much as you like. Again, the trick is to ensure you get the same - or at least a well defined - location regardless of who builds it (dev, build agent) and regardless how the build was initiated (VS, command line).
A simple way of handling the differences is to import conditionally.
<Import Project="..\..\..\build\buildsystem.targets"
Condition="'$(BuildingInsideVisualStudio)'!='true'"/>
That will give you no changes if initiating the build from VS and whatever changes you code for if you build from command line.
--Jesper

Incremental Build of Nuget Packages

I want to execute an msbuild project which uses batching to determine that one or more csproj projects have been freshly-built, and therefore require fresh nuget packaging. The script I've made so far seems like a reasonable start, but it the incremental-build mechanism isn't working. The MainBuild target executes every time, no matter what.
Here is what I have:
<?xml version="1.0" encoding="utf-8"?>
<Project ToolsVersion="4.0" DefaultTargets="MainBuild" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
<PropertyGroup>
<Configuration Condition=" '$(Configuration)'=='' ">Debug</Configuration>
<Content>content\plugins\</Content>
</PropertyGroup>
<ItemGroup>
<Nuspec Include="$(MSBuildProjectDirectory)\plugins\*\*.nuspec" />
</ItemGroup>
<Target Name="MainBuild"
Inputs="%(Nuspec.RootDir)%(Nuspec.Directory)bin\$(Configuration)\*.dll"
Outputs="%(Nuspec.RootDir)%(Nuspec.Directory)%(FileName).pkg" >
<ItemGroup>
<Inputs Include="%(Nuspec.RootDir)%(Nuspec.Directory)bin\$(Configuration)\*.dll" />
<Outputs Include="%(Nuspec.RootDir)%(Nuspec.Directory)%(FileName).pkg" />
</ItemGroup>
<Message Text="INPUTS: %(Inputs.FullPath)" />
<Message Text="OUTPUTS: #(Outputs->'%(FullPath)')" />
<Copy SourceFiles="#(Inputs)" DestinationFiles="#(Outputs->'%(FullPath)')" />
</Target>
</Project>
The Copy task is just a debugging placeholder for calling-out to nuget and creating a new package.
The idea is that if any files in the bin\Debug directory are newer than the corresponding .nuspec file (found two folders above bin\Debug), then the MainBuild target should execute.
Any ideas?
p.s. The Inputs and Outputs attributes of the Target presumably each create an item. I think it strange that the items created can't be referenced inside the target. In the above example, I had to make a target-interna dynamic ItemGroup to re-create the items, just so that I could access them. Is there a way around that?
I read this in the MSBuild Batching documentation
If a task inside of a target uses batching, MSBuild needs to determine
if the inputs and outputs for each batch of items is up-to-date.
Otherwise, the target is executed every time it is hit.
Which may be the cuprit. Try changing your copy target to use batching instead of an ite transform (I don't think using item metadata in an item group satisfies the above requirement).
<Target Name="MainBuild"
Inputs="%(Nuspec.RootDir)%(Nuspec.Directory)bin\$(Configuration)\*.dll"
Outputs="%(Nuspec.RootDir)%(Nuspec.Directory)%(FileName).pkg" >
<ItemGroup>
<Inputs Include="%(Nuspec.RootDir)%(Nuspec.Directory)bin\$(Configuration)\*.dll" />
<Outputs Include="%(Nuspec.RootDir)%(Nuspec.Directory)%(FileName).pkg" />
</ItemGroup>
<Message Text="INPUTS: %(Inputs.FullPath)" />
<Message Text="OUTPUTS: #(Outputs->'%(FullPath)')" />
<Copy SourceFiles="#(Inputs)" DestinationFiles="%(Outputs.FullPath)" />
</Target>
It looks like the number of inputs may be different than the number of outputs (I suspect there is more than one .dll files in the output directory for each project), which will also cause the target to execute.

Output all files from a solution in an MSBuild task

In MSBuild, I would like to call a task that extracts all the files in all the project in a specific solution and hold these files in a property that can be passed around to other tasks (for processing etc.)
I was thinking something along the lines of:
<ParseSolutionFile SolutionFile="$(TheSolutionFile)">
<Output TaskParameter="FilesFound" ItemName="AllFilesInSolution"/>
</ParseSolutionFile>
<Message Text="Found $(AllFilesInSolution)" />
which would output the list of all files in the projects in the solution and I could use the AllFilesInSolution property as input to other analysis tasks. Is this an already existing task or do I need to build it myself? If I need to build it myself, should the task output an array of strings or of ITaskItems or something else?
I don't know about tasks, but there are already properties that hold all items. Just look in your typical project file and you'll see which collection they're being added to.
Note the properties Content, Compile, Folder... any time you add a file to a project, it gets put in one of the main collections like this:
<ItemGroup>
<Content Include="Default.aspx" />
<Content Include="Web.config" />
</ItemGroup>
<ItemGroup>
<Compile Include="Default.aspx.cs">
<SubType>ASPXCodeBehind</SubType>
<DependentUpon>Default.aspx</DependentUpon>
</Compile>
<Compile Include="Default.aspx.designer.cs">
<DependentUpon>Default.aspx</DependentUpon>
</Compile>
</ItemGroup>
<ItemGroup>
<Folder Include="App_Data\" />
</ItemGroup>
Then you can do stuff like this to put the values from existing properties into your properties (the Condition attribute acts as a filter):
<CreateItem Include="#(Content)" Condition="'%(Extension)' == '.aspx'">
<Output TaskParameter="Include" ItemName="ViewsContent" />
</CreateItem>
Or you can do it manually (the Include attribute uses the existing property OutputPath, but it indicates a path that inclues all files):
<CreateItem Include="$(OutputPath)\**\*">
<Output TaskParameter="Include" ItemName="OutputFiles" />
</CreateItem>
There are more details in the MSDN MSBuild documentation that I read when I was mucking with custom build tasks and stuff that was very helpful. Go read up on the CreateItem task and you'll be able to make more sense out of what I posted here. It's really easy to pick up on.
I use the following for solutions with SSRS projects (which dont build under TFS w/o vs installed on the build box). Basically we require that the RDLs be bundled into a build output so we can mark a build for release.
<Target Name="CopyArtifactstoDropLocation">
<CreateItem Include="$(SolutionRoot)\**\*.*">
<Output TaskParameter="Include" ItemName="YourFilesToCopy" />
</CreateItem>
<Copy
SourceFiles="#(YourFilesToCopy)"
DestinationFiles="#(YourFilesToCopy->'$(DropLocation)\$(BuildNumber)\Release\%(RecursiveDir)%(Filename)%(Extension)')" />
</Target>
Just replace the usage of the Copy Task with whatever you need to do with your bundle. Granted this is going to get everything in your solution root, but if your using TFS then you should only have buildable artifacts.