I need to synchronize access to a shared resource between projects built in MsBuild. In the past, when projects ran sequentially and in the same process, this was not a problem. But now when MsBuild can run project builds in several threads and several processes, I just can't find a way to achieve this.
Basically, what I want is that when my projects start to build, I can do some operation on the resource. And when they all have built to the end, I can do another operation.
If thought first that I could implement a custom task and have every project modify the build process, injecting the task before the build and after the build. But since they can run in parallel and even across processes, I seem to be out of option.
When trying to figure this out, the closest I came was when I found the BuildSubmission class in the Microsoft.Build.Execution namespace (http://msdn.microsoft.com/en-us/library/microsoft.build.execution.buildsubmission.aspx). But I can't find anything on how I'm suppose to interact with these classes from my custom tasks.
Any type of input or pointers in some direction would be highly appreciated!
/Per
You can implement your resource access in a custom MSBuild task and protect it by a named mutex (these can be used to synchronize separate processes). That is, the implementation will first acquire the mutex, check whether the resource needs to be touched or not, and only then will it perform its work on the resource and finally release the mutex. You can force this task to run either before or after the build using AfterTargets or BeforeTargets.
This way, if two projects that use this task reach the pre-build or post-build stage concurrently, only one of them will perform work on the resource while the other one waits. When the first project releases the mutex, the second one will acquire it, realize the job has already been done and immediately release the mutex, without touching the resource.
You should consider whether you want to use the same mutex or two different ones for the pre-build and post-build resource access (can they run concurrently?).
Related
What are msbuild tasks and what situations should you be using them or not using them? What are the alternatives? What advantages or disadvantages do they offer?
A task implements the Task class and as such can be called from msbuild code. Usually this is done by e.g. writing some C# code implementing the class and building it into a dll which is passed to the UsingTask element to make it available. There's also a shorter way to do this: using Inline Tasks. This allows writing the code directlry in the msbuild file.
Sctrictly speaking there is no alternative since a Task has the definition given above and there is only one such thing with exactly those properties in msbuild. There is also a Target though which is used to call Tasks (and has a bunch of other functionality like expressing dependencies to other targets, defining it's in/outputs, ...). So it's an alternative considering there is some overlap, and I assume this is what you're asking about: you can create functionality either by calling multiple Tasks consecutively in a Target (or having targets depend on other targets etc), or by writing your own Task which performs all or some of those actions.
By example: suppose you want to list a directory and copy all .c files to another directory then zip the directory. Either you write a Target in which you list the files (using an ItemGroup), then call Copy and Zip tasks. Or you write a custom task which uses C# calls like Directory.GetFiles/File.Copy/ZipFile.CreateDirectory and have the target call just your custom task.
Advantages of custom tasks: they can contain arbitrary code so you can basically do anything you can imagine. Disadvantage: needs to be built, maintained and shipped with the msbuild code using them (either as a dll or else as source code in which case they need to be built on the fly before they can be used).
Advantages of targets with existing (built-in) tasks: most common functionality found in build systems is readily available in tried and tested code with ample documentation and/or SO questions as additional resources, no reinventing of wheels, others know that code already as well, no maintainance of custom code. Disadvantage: not every single piece of functionality is available, number of composed tasks to achieve functionality might be too high or impractical.
When to use tasks is basically answered by the two paragraphs above. I can't possibly tell you how much custom tasks you'll be writing in practice as I don't know your usecases. Looking at all msbuild code I have myself (for dealing with a mix of C/C++/C#/Python projects) I'd say it's about 95% built-in tasks and 5% custom tasks. Of those 5% most is from tasks written by others ike MSBuild Community Tasks and MSBuild Extension Pack.
My goal is to create build definitions within Visual Studio Team Services for both test and production environments. I need to update 2 variables in my code which determine which database and which blob storage the environment uses. Up till now, I've juggled this value in a Resource variable, and pulled that value in code from My.Resources.DB for a library, and Microsoft.Azure.CloudConfigurationManager.GetSetting("DatabaseConnectionString") for an Azure worker role. However, changing 4 variables every time I do a release is getting tiring.
I see a lot of posts that get close to what I want, but they're geared towards C#. For reasons beyond my influence, this project is written in VB.NET. It seems I have 2 options. First, I could call the MSBuild process with a couple of defined properties, passing them to the .metaproj build file, but I don't know how to get them to be used in VB code. That's preferable, but, at this point, I'm starting to doubt that this is possible.
I've been able to set some pre-processor constants, to be recognized in #If-#Else directives.
#If DEBUG = True Then
BarStaticItemVersion.Caption = String.Format("Version: {0}", "1.18.0.xxx")
#Else
BarStaticItemVersion.Caption = String.Format("Version: {0}", "1.18.0.133")
#End If
msbuild CalbertNG.sln.metaproj /t:Rebuild /p:DefineConstants="DEBUG=False"
This seems to work, though I need to Rebuild to change the value of that constant. Should I have to? Should Build be enough? Is this normal, or an indication that I don't have something set quite right?
I've seen other posts that talk about pre-processing the source files with some other builder, like Ant, but that seems like overkill. It feels like I'm close here. But I want to zoom out and ask, from a clean sheet of paper, if you're given 2 variables which need to change per environment, you're using VB.NET, and you want to incorporate those variable values in an automated VS Team Services build process upon code check-in, what's the best way to do it? (I want to define the variables in the VSTS panel, but this just passes them to my builder, so I have to know how to parse the call to MSBuild to make these useful.)
I can control picking between 2 static strings, now, via compiler directives, but I'd really like to reference the Build.BuildNumber that comes out of the MSBuild process to display to the user, and, if I can do that, I can just feed the variables for database and blob container via the same mechanism, and skip the pre-processor.
You've already found the way you can pass data from the MsBuild Arguments directly into the code. An alternative is to use the Condition Attribute in your project files to make certain property groups optional, it allows you to even include specific files conditionally. You can control conditions by passing in /p:ConditionalProperty=value on the MsBuild command. This at least ensures people use a set of values that make sense together.
The problem is that when MsBuild is running in Incremental mode it is likely to not process your changes (as you've noticed), the reason for this, is that the input files remain unchanged since the last build and are all older than the last generated output files.
To by-pass this behavior you'd normally create a separate solution configuration and override the output location for all projects to be unique for that configuration. Combined with setting the Compiler constants for that specific configuration you're ensured that when building that Configuration/Platform combination, incremental builds work as intended.
I do want to echo some of the comments from JerryM and Daniel Mann. Some items are better stored in else where or updated before you actually start the compile phase.
Possible solutions:
Store your configuration data in config files and use Configuration Transformation to generate the right config file base don the selected solution configuration. The process is explained on MSDN. To enable configuration transformation on all project types, you can use SlowCheetah.
Store your ocnfiguration data in the config files and use MsDeploy and specify a Parameters.xml file that matches the deploy package. It will perform the transformation on deploy time and will actually allow your solution to contain a standard config file you use at runtime, plus a publish profile which will post-process your configuration. You can use a SetParameters.xml file to override the variables at deploy time.
Create an installer project (such as through Wix) and merge the final configuration at install time (similar to the MsDeploy). You could even provide a UI which prompts for specific values (and can supply default values).
Use a CI server, like the new TFS/VSTS 2015 task based build engine and combine it with a task that can search&replace tokens, like the Replace Tokens task, Tokenization Task, Colin's ALM Corner Build and Release Tasks. And a whole bunch that specifically deal with versioning. Handling these things in the CI server also allows you to do a quick build locally at all times and do these relatively expensive steps on the build server (patching source code breaks incremental build in MsBuild, because there are always newer input files.
When talking specifically about versioning, there are a number of ways to set the AssemblyVersion and AssemblyFileVersion just before compile time, usually it involves overriding the AssemblyInfo.cs file before compilation. Your code could then use reflection to read the value at runtime. You can use the AssemblyInformationalversion to specify something like you do in the example above which contains .xxx or other text. It also ensures that the version displayed always reflects the information obtained when reading the file properties through Windows Explorer.
As far as I understand, Roslyn have introduced the concept of Workspaces. One implementation of Workspaces is the MsBuildWorkspace.
My question is, can I from within a custom build task access a Roslyn Workspace representing the project being built?
I suspect that this is the purpose of MsBuildWorkspace. If so, can I access this workspace from the Execute method in my custom task (derived from Microsoft.Build.Utilities.Task)?
In case you are wondering why, I need to traverse other aspects of the project being built.
It is not enough for me to have access to the specific input file of the task to generate the output.
I suspect that this is the purpose of MsBuildWorkspace
No.
The point of MsBuildWorkspace is to parse an MSBuild project or solution into a Roslyn workspace.
MSBuild itself (which is what actually runs your MSBuild task) does not use Roslyn at all (it just invokes the C# compilation task, which is implemented using Roslyn), so there is no existing MsBuildWorkspace that you could fetch.
You could create your own MsBuildWorkspace from the project file.
I'm trying to improve build times using CruiseControl.NET and MSBUILD, and one of the commandline switches, maxcpucount can be used to allow the build occur in parallel. Our solution has 60+ projects so any improvement would be helpful. However, whenever I up the maxcpucount above one, we have frequent build failures due to:
"The process cannot access the file xxxx because it is being used by
another process. msbuild"
It appears that the additional parallel build threads/processes are locking each other.
I think I found a solution. It appears that if I add the /nodeReuse:false switch I don't get the file locks. It seems like the nodeReuse functionality is keeping msbuild processes around and those are hanging on to file locks for subsequent builds.
http://msdn.microsoft.com/en-us/library/ms164311.aspx
Are you building from a solution file? If so, make sure that you are using direct project-to-project references and not using the Solution's project-dependency feature. If you happen to be using a bit of both, there can be issues. See this article.
Better yet, if at all possible, ditch the solution file and create your own MSBuild file to drive your build.
Your assembly is probably being used by another assembly thats being built. Make sure each assembly gets built before it's needed by other assemblies
Are there any interesting custom loggers for MSBuild already written? I am especially interested in loggers that would record system load (CPU/Memory/IO) data during the build and relate that to actions performed by the build.
Checkout the MSBuild Extension pack. This contains a lot of extra custom tasks and loggers.
I don't know about your specific needs for the logger. Remember that the purpose of the logger is only to report on different steps of the build. It only accepts a message and an importance property. You would need to add system load collection by yourself. But again, I don't think this kind of code would belong in a custom logger.