.Net Core MS Build Package Creation Lifecycle - msbuild

MSBuild quite powerful and important tool, but sometimes configuration of it is the same hard as making a spaceship or flying to Mars. Answers on next questions can help makes it a little bit more usefull and developer friendly:
- How the MSBuild create a Package(zip)?
- Does it use some temporary folders or direct from builds output ?
- Why can the output folder content differ from a package content
- What events can be used to add some custom logic?
The answers to this question could help me with the next long story short:
I'm working with a .Net Core solution, that has the dependency on some other solutions DLLs files( that are not referenced directly, but required to be in the solution folder). On post-build event CL there is the option added as an entry point "\TaskEP" that is a starting point of the next pipeline :
<PropertyGroup>
<PipelineCopyAllFilesToOneFolderForMsdeployDependsOn>
TaskEP;
Task2;
Task3;
$(PipelineCopyAllFilesToOneFolderForMsdeployDependsOn);
</PipelineCopyAllFilesToOneFolderForMsdeployDependsOn>
<RunPostBuildEvent>Always</RunPostBuildEvent>
Tasks are mostly done for add extra DLLs to package and in general looks like :
<Target Name="TaskEP" Condition="$(SomeConditions) == 'True'">
<Message Importance="high" Text="TaskEP: Copying some files..." />
<ItemGroup>
<ItemsName Include="$(MSBuildProjectDirectory)\..\..somepath..\Extra.dll;....dll" />
<FilesForPackagingFromProject Include="%(ItemsName.Identity)">
<DestinationRelativePath>bin\%(RecursiveDir)%(Filename)%(Extension)</DestinationRelativePath>
</FilesForPackagingFromProject>
</ItemGroup>
</Target>
Basically, for me, this instruction to MSBuild to add "Extra.dll" to output package.
Pubxml is quite usual :
<PropertyGroup>
<WebPublishMethod>Package</WebPublishMethod>
<LastUsedBuildConfiguration>Release</LastUsedBuildConfiguration>
<LastUsedPlatform>Any CPU</LastUsedPlatform>
<SiteUrlToLaunchAfterPublish />
<LaunchSiteAfterPublish>True</LaunchSiteAfterPublish>
<_PackagePathShortened Condition="'$(_PackagePathShortened)' == ''">SuperWebSite</_PackagePathShortened>
<ExcludeApp_Data>False</ExcludeApp_Data>
<PackageLocation>..\..\..\..\BIN\WEB\Release\Publish\Super.WebApplication.zip</PackageLocation>
<PackageAsSingleFile>true</PackageAsSingleFile>
<TargetFramework>netcoreapp2.1</TargetFramework> ....
And one instruction that replaces the long long path in the package to Short and defined one:
<Target Name="AddReplaceRuleForAppPath" BeforeTargets="BeforePublish">
<Message Text="Adding replace rules for application path '$(PublishIntermediateOutputPath)' replace with '$(_PackagePathShortened)'" Importance="high" />
<EscapeTextForRegularExpressions Text="$(PublishIntermediateOutputPath)">
<Output PropertyName="_PackagePathRegex" TaskParameter="Result" />
</EscapeTextForRegularExpressions>
<!-- Add a replace rule for VSMSDeploy resp. MSdeploy to update the path -->
<ItemGroup>
<MsDeployReplaceRules Include="replaceFullPath">
<Match>$(_PackagePathRegex)</Match>
<Replace>$(_PackagePathShortened)</Replace>
</MsDeployReplaceRules>
</ItemGroup>
I also duplicated the same "Copy" tasks here in .pubxml file , play with different events "AfterBuild", "BeforePublish" but seems that all of the just ignoring. I can see the "Extra" DLLs files in the build output directory and all Info Messages on publishing, but not in the final "Super.WebApplication.zip" file !

The solution I found to copy extra files in .Net Core is taken from this documentation
<ItemGroup>
<DotnetPublishFiles Include="$(MSBuildProjectDirectory)\..\..somepath..\Extra.dll;....dll" >
<DestinationRelativePath>bin\x86\%(RecursiveDir)%(Filename)%(Extension)</DestinationRelativePath>
</DotnetPublishFiles>
</ItemGroup>

Related

how to add files to a projects output files from an MSBuild Task

Given an MSBuild Task that runs in AfterTargets="AfterCompile" and produces some files how do you get those files to be included in the current projects output so that the files will be copied to the bin directory of any projects referencing that project?
I have no guarantees that this is the right solution but it seems to work:
<Target Name="MyTarget" AfterTargets="AfterCompile">
<PropertyGroup>
<MyInput>D:\1.txt</MyInput>
<MyOutput>$([System.IO.Path]::GetFullPath('$(MSBuildThisFileDirectory)$(OutDir)\1.txt'))</MyOutput>
</PropertyGroup>
<Copy SourceFiles="$(MyInput)" DestinationFolder="$(OutDir)" SkipUnchangedFiles="true" />
<ItemGroup>
<AllItemsFullPathWithTargetPath Include="$(MyOutput)">
<TargetPath>1.txt</TargetPath>
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
</AllItemsFullPathWithTargetPath>
</ItemGroup>
</Target>
The relevant logic is here:
http://sourceroslyn.io/#MSBuildTarget=GetCopyToOutputDirectoryItems
http://sourceroslyn.io/#MSBuildItem=AllItemsFullPathWithTargetPath
Basically we rely on the fact that to determine the list of files to copy from dependent projects MSBuild calls the GetCopyToOutputDirectoryItems target of the dependent projects and uses its output (which is AllItemsFullPathWithTargetPath).
By adding ourselves to AllItemsFullPathWithTargetPath at the last minute we get picked up when a dependent project calls us.
Thank you, Kirill. That was an excellent answer and it helped me when trying to copy ETW manifest files from a different project's output. Below is the final output.
Since I've simply expanded upon Kirill's answer, I do not expect this answer to be accepted. I post this here in the hope it helps someone else.
<Target Name="IncludeEtwFilesInOutput"
BeforeTargets="GetCopyToOutputDirectoryItems">
<PropertyGroup>
<EtwManifestFile>$([System.IO.Path]::GetFullPath('$(MSBuildThisFileDirectory)$(OutDir)\My.etwManifest.man'))</EtwManifestFile>
<EtwResourceFile>$([System.IO.Path]::GetFullPath('$(MSBuildThisFileDirectory)$(OutDir)\My.etwManifest.dll'))</EtwResourceFile>
</PropertyGroup>
<ItemGroup>
<AllItemsFullPathWithTargetPath Include="$(EtwManifestFile)">
<TargetPath>My.etwManifest.man</TargetPath>
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
</AllItemsFullPathWithTargetPath>
<AllItemsFullPathWithTargetPath Include="$(EtwResourceFile)">
<TargetPath>My.etwManifest.dll</TargetPath>
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
</AllItemsFullPathWithTargetPath>
</ItemGroup>
</Target>

How to generate files during build using msbuild

Does anyone know how to modify a csproj file in a way to generate code files during build without actually referencing the files?
A process like :
create file,
dynamically reference temporary file during build
compiled assembly has additional members, depending on the files created during build
The purpose of this is to create a way of generating code files using roslyn instead of using t4 templates, which are very awkward to use once you're trying to do something depending on attributes.
Hence i am planning on providing a way to use a special csharp file (for full syntax support) to generate files programatically based on the contents of that special file.
I've spent a couple of weeks looking into resources on the internet (with the topic msbuild), but until now it seems i didn't use the right keywords.
This one has been the most insightful one to me yet:
https://www.simple-talk.com/dotnet/.net-tools/extending-msbuild/
My guess is, that the correct build target for my purpose should be "BeforeCompile" in order to somehow populate the build process with custom code files.
Does anyone have experience with my issue, or is aware of any particular resources which deal with the task?
Solution i got it working with:
<UsingTask TaskName="DynamicCodeGenerator.DynamicFileGeneratorTask" AssemblyFile="..\DynamicCodeGenerator\bin\Debug\DynamicCodeGenerator.dll" />
<Target Name="DynamicCodeGeneratorTarget" BeforeTargets="BeforeBuild;BeforeRebuild">
<DynamicFileGeneratorTask>
<Output ItemName="Generated" TaskParameter="GeneratedFilePaths" />
</DynamicFileGeneratorTask>
<ItemGroup>
<Compile Include="#(Generated)" />
<FileWrites Include="#(Generated)" />
<!-- For clean to work properly -->
</ItemGroup>
</Target>
Unfortunately i did not get it to work with a propertygroup override as suggested
Update: This link is interesting too: https://github.com/firstfloorsoftware/xcc/blob/master/FirstFloor.Xcc/Targets/Xcc.targets
Generating the code file can be achieved by msbuild task or msbuild inline task. It is up to you to generate the proper code. One thing that you must care of is creating output item parameter in order to append it to the #(Compile) item. You can use $(IntDir) location to locate your newly generated file, and add them to the #(FileWrites) item group in order for Clean target work properly.
When you finish writing your task, you must use it in your project like this:
<UsingTask TaskName="TaskTypeFullName" AssemblyFile="YourAssembly.dll"/>
<PropertyGroup>
<!-- Here you need to experiment with [Build/Compile/SomeOther]DependsOn property -->
<BuildDependsOn>
MyCodeGenerator;
$(BuildDependsOn)
</BuildDependsOn>
</PropertyGroup>
<Target Name="MyCodeGenerator">
<YourTaskName>
<Output ItemName="Generated" TaskParameter="GeneratedFiles" />
</YourTaskName>
<ItemGroup>
<Compile Include="#(Generated)" />
<FileWrites Include="#(Generated)" /> <!-- For clean to work properly -->
</ItemGroup>
</Target>
I wanted to use a bash script to generate code for a project using dotnet core on Linux. Here is what worked for me. And thanks #stukselbax, I built this off of your answer.
<Target Name="GenerateProtocolBuffers" BeforeTargets="BeforeBuild;BeforeRebuild">
<Exec Command="./generatecode.sh" Outputs="proto/*.cs">
<Output ItemName="Generated" TaskParameter="Outputs" />
</Exec>
<ItemGroup>
<Compile Include="#(Generated)" />
<FileWrites Include="#(Generated)" />
</ItemGroup>
</Target>
Note that the script I'm using to generate the code is called generatecode.sh. Replace this with your own.

Issue with using MSBuild to build and copy all outputs to a common folder

We are trying to write a msbuild script that will build the solution and copy over all the compiled binaries and dependencies over to a specific output folder. While the build script that we have does build and copy over the binaries to a common folder, but we are not getting the dependencies copied.
This probably has to do with the way we have used the msbuild task to build the solution and we are accepting the targetoutputs of the task into an itemgroup and iterating over the item group to copy all the compiled dlls and exes over to a common folder. But this is not including the dependency dlls which gets placed into the individual bin folder of each project.
<Project ToolsVersion="4.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
<PropertyGroup>
<ParentSolutionFile />
</PropertyGroup>
<ItemGroup>
<Assemblies Include="*.dll, *.exe" />
</ItemGroup>
<Target Name="BuildAll">
<CombinePath BasePath="$(MSBuildProjectDirectory)" Paths="Source\Solutions\xxx.sln">
<Output TaskParameter="CombinedPaths" PropertyName="ParentSolutionFile" />
</CombinePath>
<Message Text="$(ParentSolutionFile)" />
<MSBuild Projects="$(ParentSolutionFile)">
<Output TaskParameter="TargetOutputs" ItemName="Assemblies" />
</MSBuild>
<Message Text="%(Assemblies.Identity)" />
<Copy SourceFiles="%(Assemblies.Identity)" DestinationFolder="$(MSBuildProjectDirectory)\Binary" OverwriteReadOnlyFiles="True" SkipUnchangedFiles="True" />
</Target>
What will be the preferred way to copy over all the binaries along with the necessary dependencies to a common output folder?
Does not overriding OutputPath do the trick alone?
<MSBuild Projects="$(ParentSolutionFile)" Properties="OutputPath=$(MSBuildProjectDirectory)\Binary">
<Output TaskParameter="TargetOutputs" ItemName="Assemblies" />
</MSBuild>
And leave out the copy task alltogether?
The build process will place the final result in the directory represented by OutputPath - at least if you are building c# projects. For C/C++ the internal structure and variable names are completely different.
Thus, in theory, you could pass the OutputPath in the MsBuild-task that builds the solution.
<MsBuild Projects="$(ParentSolutionFile)"
Properties="OutputPath=$(MSBuildProjectDirectory)\Binary"/>
However, the csproj-files will overwrite that value unconditionally with the following code:
<PropertyGroup Condition=" '$(Configuration)|$(Platform)' == 'Debug|AnyCPU' ">
<DebugSymbols>true</DebugSymbols>
<OutputPath>bin\Debug\</OutputPath>
I have solved this by injecting my own build system in each and every csproj-file.
<Import Project="$(MSBuildToolsPath)\Microsoft.CSharp.targets" />
<Import Project="..\..\..\build\buildsystem.targets" />
The path is relative to the csproj-file. An absolute path is fine too, or a variable. The trick is to make it work on all dev machines as well as the build agents.
Now, in buildsystem.targets, simply redefine OutputPath as much as you like. Again, the trick is to ensure you get the same - or at least a well defined - location regardless of who builds it (dev, build agent) and regardless how the build was initiated (VS, command line).
A simple way of handling the differences is to import conditionally.
<Import Project="..\..\..\build\buildsystem.targets"
Condition="'$(BuildingInsideVisualStudio)'!='true'"/>
That will give you no changes if initiating the build from VS and whatever changes you code for if you build from command line.
--Jesper

Determining outputs of a ProjectReference in MSBuild without triggering redundant rebuilds

As part of a solution containing many projects, I have a project that references (via a <ProjectReference> three other projects in the solution, plus some others). In the AfterBuild, I need to copy the outputs of 3 specific dependent projects to another location.
Via various SO answers, etc. the way I settled on to accomplish that was:
<MSBuild
Projects="#(ProjectReference)"
Targets="Build"
BuildInParallel="true"
Condition="'%(Name)'=='ProjectA' OR '%(Name)'=='ProjectB' OR '%(Name)'=='ProjectC'">
<Output TaskParameter="TargetOutputs" ItemName="DependentAssemblies" />
</MSBuild>
<Copy SourceFiles="#(DependentAssemblies)" DestinationFolder="XX" SkipUnchangedFiles="true" />
However, I ran into problems with this. The <MSBuild step's IncrementalClean task ends up deleting a number of the outputs of ProjectC. When running this under VS2008, a build.force file being deposited in the obj/Debug folder of ProjectC which then triggers ProjectC getting rebuilt if I do a Build on the entire solution if the project containing this AfterBuild target, whereas if one excludes this project from the build, it [correctly] doesn't trigger a rebuild of ProjectC (and critically a rebuild of all dependents of ProjectC). This may be VS-specific trickery in this case which would not occur in the context of a TeamBuild or other commandline MSBuild invocation (but the most common usage will be via VS so I need to resolve this either way)
The dependent projects (and the rest of the solution in general) have all been created interactively with VS, and hence the ProjectRefences contain relative paths etc. I've seen mention of this being likely to causing issues - but without a full explanation of why, or when it'll be fixed or how to work around it. In other words, I'm not really interested in e.g. converting the ProjectReference paths to absolute paths by hand-editing the .csproj.
While it's entirely possible I'm doing something stupid and someone will immediately point out what it is (which would be great), be assured I've spent lots of time poring over /v:diag outputs etc. (although I havent tried to build a repro from the ground up - this is in the context of a relatively complex overall build)
As noted in my comment, calling GetTargetPath on the referenced project only returns the Primary output assembly of that project. To get all the referenced copy-local assemblies of the referenced project it's a bit messier.
Add the following to each project that you are referencing that you want to get the CopyLocals of:
<Target
Name="ComputeCopyLocalAssemblies"
DependsOnTargets="ResolveProjectReferences;ResolveAssemblyReferences"
Returns="#(ReferenceCopyLocalPaths)" />
My particular situation is that I needed to recreate the Pipeline folder structure for System.AddIn in the bin folder of my top level Host project. This is kinda messy and I was not happy with the MSDN suggested solutions of mucking with the OutputPath - as that breaks on our build server and prevents creating the folder structure in a different project (eg a SystemTest)
So along with adding the above target (using a .targets import), I added the following to a .targets file imported by each "host" that needs the pipeline folder created:
<Target
Name="ComputePipelineAssemblies"
BeforeTargets="_CopyFilesMarkedCopyLocal"
Outputs="%(ProjectReference.Identity)">
<ItemGroup>
<_PrimaryAssembly Remove="#(_PrimaryAssembly)" />
<_DependentAssemblies Remove="#(_DependentAssemblies)" />
</ItemGroup>
<!--The Primary Output of the Pipeline project-->
<MSBuild Projects="%(ProjectReference.Identity)"
Targets="GetTargetPath"
Properties="Configuration=$(Configuration)"
Condition=" '%(ProjectReference.PipelineFolder)' != '' ">
<Output TaskParameter="TargetOutputs"
ItemName="_PrimaryAssembly" />
</MSBuild>
<!--Output of any Referenced Projects-->
<MSBuild Projects="%(ProjectReference.Identity)"
Targets="ComputeCopyLocalAssemblies"
Properties="Configuration=$(Configuration)"
Condition=" '%(ProjectReference.PipelineFolder)' != '' ">
<Output TaskParameter="TargetOutputs"
ItemName="_DependentAssemblies" />
</MSBuild>
<ItemGroup>
<ReferenceCopyLocalPaths Include="#(_PrimaryAssembly)"
Condition=" '%(ProjectReference.PipelineFolder)' != '' ">
<DestinationSubDirectory>%(ProjectReference.PipelineFolder)</DestinationSubDirectory>
</ReferenceCopyLocalPaths>
<ReferenceCopyLocalPaths Include="#(_DependentAssemblies)"
Condition=" '%(ProjectReference.PipelineFolder)' != '' ">
<DestinationSubDirectory>%(ProjectReference.PipelineFolder)</DestinationSubDirectory>
</ReferenceCopyLocalPaths>
</ItemGroup>
</Target>
I also needed to add the required PipelineFolder meta data to the actual project references. For example:
<ProjectReference Include="..\Dogs.Pipeline.AddInSideAdapter\Dogs.Pipeline.AddInSideAdapter.csproj">
<Project>{FFCD0BFC-5A7B-4E13-9E1B-8D01E86975EA}</Project>
<Name>Dogs.Pipeline.AddInSideAdapter</Name>
<Private>False</Private>
<PipelineFolder>Pipeline\AddInSideAdapter\</PipelineFolder>
</ProjectReference>
Your original solution should work simply by changing
Targets="Build"
to
Targets="GetTargetPath"
The GetTargetPath target simply returns the TargetPath property and doesn't require building.
You may protect your files in ProjectC if you call a target like this first:
<Target Name="ProtectFiles">
<ReadLinesFromFile File="obj\ProjectC.csproj.FileListAbsolute.txt">
<Output TaskParameter="Lines" ItemName="_FileList"/>
</ReadLinesFromFile>
<CreateItem Include="#(_DllFileList)" Exclude="File1.sample; File2.sample">
<Output TaskParameter="Include" ItemName="_FileListWitoutProtectedFiles"/>
</CreateItem>
<WriteLinesToFile
File="obj\ProjectC.csproj.FileListAbsolute.txt"
Lines="#(_FileListWitoutProtectedFiles)"
Overwrite="true"/>
</Target>
My current workaround is based on this SO question, i.e, I have:
<ItemGroup>
<DependentAssemblies Include="
..\ProjectA\bin\$(Configuration)\ProjectA.dll;
..\ProjectB\bin\$(Configuration)\ProjectB.dll;
..\ProjectC\bin\$(Configuration)\ProjectC.dll">
</DependentAssemblies>
</ItemGroup>
This however will break under TeamBuild (where all the outputs end up in one directory), and also if the names of any of the outputs of the dependent projects change.
EDIT: Also looking for any comments on whether there's a cleaner answer for how to make the hardcoding slightly cleaner than:
<PropertyGroup>
<_TeamBuildingToSingleOutDir Condition="'$(TeamBuildOutDir)'!='' AND '$(CustomizableOutDir)'!='true'">true</_TeamBuildingToSingleOutDir>
</PropertyGroup>
and:
<ItemGroup>
<DependentAssemblies
Condition="'$(_TeamBuildingToSingleOutDir)'!='true'"
Include="
..\ProjectA\bin\$(Configuration)\ProjectA.dll;
..\ProjectB\bin\$(Configuration)\ProjectB.dll;
..\ProjectC\bin\$(Configuration)\ProjectC.dll">
</DependentAssemblies>
<DependentAssemblies
Condition="'$(_TeamBuildingToSingleOutDir)'=='true'"
Include="
$(OutDir)\ProjectA.dll;
$(OutDir)\ProjectB.dll;
$(OutDir)\ProjectC.dll">
</DependentAssemblies>
</ItemGroup>

MSBuild and _PublishedWebsites

After MSbuild has built my solution (with an asp.net website), and the webdeployment project has built and put the website in the directory _PublishedWebsites:
c:\mybuilds\buildName\Daily_20090519.3\Release_PublishedWebsites\MyWebsite.
How do I copy this to the fixed directory where IIS points to for the test website?
I have found loads of code snippets, but I cannot seem to find one that will take into account the fact that this directory name changes.
This is pretty easy. You can edit the project and insert something similar to the following.
<PropertyGroup>
<OutputDest>$(MSBuildProjectDirectory)\..\OutputCopy\</OutputDest>
</PropertyGroup>
<Target Name="AfterBuild">
<!-- Create an item with all the output files -->
<ItemGroup>
<_OutputFiles Include="$(OutputPath)**\*" Exclude="$(OutputPath)obj\**\*" />
</ItemGroup>
<!-- You probably don't want to include the files in the obj folder so exclude them. -->
<Message Text="OutputDest : $(OutputDest)" />
<Copy SourceFiles="#(_OutputFiles)"
DestinationFiles="#(_OutputFiles->'$(OutputDest)%(RecursiveDir)%(Filename)%(Extension)')"/>
</Target>
Is this what you are looking for?
My Book: Inside the Microsoft Build Engine : Using MSBuild and Team Foundation Build
I'm using different technique.
<PropertyGroup>
<BinariesRoot>c:\BinariesForIis\</BinariesRoot>
</PropertyGroup>
The c:\BinariesForIis\ will be used for direct output compiled binaries (before copy to ...\Daily_20090519.3\Release_ ...).