Parallel MSBUILD - critical sections? - msbuild

Question

My organization has some huge builds that run on build servers, building lots and and lots of MSBUILD projects that are linked with ProjectReferences. We need to be able to build projects and configurations in parallel with msbuild /m.
My problem is that I have a project that is referenced from a large number of other projects, but the project itself is not reentrant. If more that two or more nodes try to build that project in parallel, it will fail.
How can I wrap this one Project, or it's Target, inside a critical section?
What I really need to do is something like this:
<Target>
<EnterCriticalSection ID=$(ProjectGuid) />
<Exec something />
<LeaveCriticalSection ID=$(ProjectGuid) />
</Target>
The idea being that if multiple MSBUILD nodes tried to build this project in parallel, only one of the nodes could do the execution, and the rest of the nodes would have to wait (or go do something else).
I suppose I could write custom MSBUILD tasks to do this, but isn't there some way of doing this built into the MSBUILD system?
=== EDIT 4/5/13. To clarify, the project is building a 3rd-party library with the build script provided for it. Completely rewriting their build script to make it reentrant -- by insuring that each build used a different set of folders for intermediate files, etc. -- is possible in theory, but not a practical solution. For one thing, all that work would would have to be redone on each new release of that library.
=== EDIT 4/6/13. On further reflection, I don't think it's even theoretically possible to insure a project is reentrant. Let me explain:
Suppose project XYZ is set up to use different temporary directories depending on platform and configuration in the usual way:
XYZ.proj:
<PropertyGroup>
<MyWorkingDir>tmp.$(Platform).$(Configuration)</MyWorkingDir>
</PropertyGroup>
Now suppose some other project GraphicsWindow references project XYZ, either through ProjectReferences or MSBuild tasks. And suppose the GraphicsWindow project can be built to use various graphics APIs. I.e., there's an OpenGL version, a DirectX 9 version, a DirectX 10, version...
So somewhere there's a .proj or .targets file containing a target to build all four versions:
<Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
<ItemGroup>
<ProjectToBuild Include="GraphicsWindow.proj">
<Properties>GraphicsApi=OpenGL</Properties>
</ProjectToBuild>
<ProjectToBuild Include="GraphicsWindow.proj">
<Properties>GraphicsApi=D3D9</Properties>
</ProjectToBuild>
<ProjectToBuild Include="GraphicsWindow.proj">
<Properties>GraphicsApi=D3D10</Properties>
</ProjectToBuild>
<ProjectToBuild Include="GraphicsWindow.proj">
<Properties>GraphicsApi=D3D11</Properties>
</ProjectToBuild>
</ItemGroup>
<Target Name="All">
<MSBuild Projects="#(ProjectToBuild)" BuildInParallel="true" />
</Target>
</Project>
or the equivalent using batching.
Now MSBuild will build the XYZ project 4 times with the same Platform|Configuration combination and the same working directory.
This will work fine as long as you build without the /m option and MSBuild runs a single thread. Depending on how the XYZ project is written, the 2nd, 3rd, and 4th builds might do nothing because the outputs are up-to-date, or it might do some redundant work, but the final result will be correct and the build will succeed.
But as soon as you start using parallel MSBuild, this build is broken! There is now a race condition where multiple threads can enter the XYZ project's target(s) concurrently, and start building using the same working directories, which will fail.

Answer 1

Irrespective of how you execute MSBuild, with multi-proc option /m or without, it is guaranteed to execute a project once for every configuration requested by the build. Here is a quote from MSDN:
When the Microsoft Build Engine encounters a project-to-project (P2P) reference while it is using parallel builds to build a project, it builds the reference only one time. If two projects have the same P2P reference, the reference is not rebuilt for each project. Instead, the build engine returns the same P2P reference to both projects that depend on it. Future requests in the session for the same target are provided the same P2P reference.
If you see the same project built more than once, it means it was referenced in two (or more) different configurations. By configuration here I mean a set of parameters that was passed to the project, e.g. project platform (x86, x64, AnyCPU, etc.,) flavor(debug/retail), localization language, any other parameters you might use.
More often than not, this is problem with mixture of project platforms. For example, you have project A built for x64, project B built for AnyCPU, and both A and B reference C. Now C has to be built twice -- for x64 and AnyCPU. If C correctly handles both platforms by cleanly separating outputs into separate directories, there is no problem. However if C treats x64 and AnyCPU as the same, it will fail randomly in multi-proc build.
Start by inspecting your solution configuration dialog. Make sure all projects have consistent set of platform/configuration parameters. If you have a need to build same project in different configuraions, make sure it puts output into separate locations.