I have run into a problem with msbuild. I have a three tiers of project files. The lowest create libraries, some of which are language neutral, and some of which are language specific. The next tier builds a project that combines the language neutral libraries with the libraries for a specific language into a product. The final tier builds the product multiple times for the different languages.
The problem that I am having is that the language neutral library is being built multiple times, and when I invoke msbuild with the /m flag I get sharing violations during the build, since multiple threads end up building the language neutral project. I have been told that if I use the RemoveProperties parameter on the MSBuild task so that all of the properties match, msbuild will only build the project once. I have tried to do this, but have not been able to get the project to only build once. I have looked at a diagnostic log and created a custom logger to try and figure out which properties to add to the RemoveProperties parameter.
My question is how does msbuild decide that two invocations of the same project file are the same, and should only be built once. Based on the info passed to the OnProjectStarted event of my custom logger, all of the Global properties are the same. Is there something else that I need to make match?
Related
Long ago, when Kotlin version 1.3.20 was released (https://blog.jetbrains.com/kotlin/2019/01/kotlin-1-3-20-released/), the ability to build in parallel using Gradle Workers was added. Simply adding the kotlin.parallel.tasks.in.project = true setting does not give any gain in build speed. As far as I understand, this parameter can be useful only if I have several folders with classes independent of each other within the same project. I saw the use of this setting when assembling the gradle itself, but did not see anywhere that separate source sets were created for each folder.
Could you provide examples of how to correctly describe the build process in build.gradle.kts so that mentioned option is really used and gives an increase in build speed when there are several processor cores.
As of yet, there's no simple way to parallelize compilation of a single source set containing Kotlin code (like just the main sources), as the compiler has to analyze all of the sources together and resolve cross-references within the source set.
By default, without any additional options, Gradle runs compilation of Kotlin sources in parallel only in different subprojects. The option kotlin.parallel.tasks.in.project also allows Gradle to run parallel compilation tasks in one project, but that only works for different source sets (that don't depend on each other!), or different targets.
For example, in multiplatform projects, if you have several targets, kotlin.parallel.tasks.in.project allows Gradle to build the compilation outputs (JVM/Android classes, *.js, Kotlin/Native *.klibs and binaries) in parallel. In Android projects, if you build multiple product variants, this option also allows parallel Kotlin compilation for those variants.
In simpler project layouts, where you only have main and test source sets and a single target, there's no way to improve Kotlin compilation speed by using multiple processors, unless you split one project into several projects.
What are msbuild tasks and what situations should you be using them or not using them? What are the alternatives? What advantages or disadvantages do they offer?
A task implements the Task class and as such can be called from msbuild code. Usually this is done by e.g. writing some C# code implementing the class and building it into a dll which is passed to the UsingTask element to make it available. There's also a shorter way to do this: using Inline Tasks. This allows writing the code directlry in the msbuild file.
Sctrictly speaking there is no alternative since a Task has the definition given above and there is only one such thing with exactly those properties in msbuild. There is also a Target though which is used to call Tasks (and has a bunch of other functionality like expressing dependencies to other targets, defining it's in/outputs, ...). So it's an alternative considering there is some overlap, and I assume this is what you're asking about: you can create functionality either by calling multiple Tasks consecutively in a Target (or having targets depend on other targets etc), or by writing your own Task which performs all or some of those actions.
By example: suppose you want to list a directory and copy all .c files to another directory then zip the directory. Either you write a Target in which you list the files (using an ItemGroup), then call Copy and Zip tasks. Or you write a custom task which uses C# calls like Directory.GetFiles/File.Copy/ZipFile.CreateDirectory and have the target call just your custom task.
Advantages of custom tasks: they can contain arbitrary code so you can basically do anything you can imagine. Disadvantage: needs to be built, maintained and shipped with the msbuild code using them (either as a dll or else as source code in which case they need to be built on the fly before they can be used).
Advantages of targets with existing (built-in) tasks: most common functionality found in build systems is readily available in tried and tested code with ample documentation and/or SO questions as additional resources, no reinventing of wheels, others know that code already as well, no maintainance of custom code. Disadvantage: not every single piece of functionality is available, number of composed tasks to achieve functionality might be too high or impractical.
When to use tasks is basically answered by the two paragraphs above. I can't possibly tell you how much custom tasks you'll be writing in practice as I don't know your usecases. Looking at all msbuild code I have myself (for dealing with a mix of C/C++/C#/Python projects) I'd say it's about 95% built-in tasks and 5% custom tasks. Of those 5% most is from tasks written by others ike MSBuild Community Tasks and MSBuild Extension Pack.
My goal is to create build definitions within Visual Studio Team Services for both test and production environments. I need to update 2 variables in my code which determine which database and which blob storage the environment uses. Up till now, I've juggled this value in a Resource variable, and pulled that value in code from My.Resources.DB for a library, and Microsoft.Azure.CloudConfigurationManager.GetSetting("DatabaseConnectionString") for an Azure worker role. However, changing 4 variables every time I do a release is getting tiring.
I see a lot of posts that get close to what I want, but they're geared towards C#. For reasons beyond my influence, this project is written in VB.NET. It seems I have 2 options. First, I could call the MSBuild process with a couple of defined properties, passing them to the .metaproj build file, but I don't know how to get them to be used in VB code. That's preferable, but, at this point, I'm starting to doubt that this is possible.
I've been able to set some pre-processor constants, to be recognized in #If-#Else directives.
#If DEBUG = True Then
BarStaticItemVersion.Caption = String.Format("Version: {0}", "1.18.0.xxx")
#Else
BarStaticItemVersion.Caption = String.Format("Version: {0}", "1.18.0.133")
#End If
msbuild CalbertNG.sln.metaproj /t:Rebuild /p:DefineConstants="DEBUG=False"
This seems to work, though I need to Rebuild to change the value of that constant. Should I have to? Should Build be enough? Is this normal, or an indication that I don't have something set quite right?
I've seen other posts that talk about pre-processing the source files with some other builder, like Ant, but that seems like overkill. It feels like I'm close here. But I want to zoom out and ask, from a clean sheet of paper, if you're given 2 variables which need to change per environment, you're using VB.NET, and you want to incorporate those variable values in an automated VS Team Services build process upon code check-in, what's the best way to do it? (I want to define the variables in the VSTS panel, but this just passes them to my builder, so I have to know how to parse the call to MSBuild to make these useful.)
I can control picking between 2 static strings, now, via compiler directives, but I'd really like to reference the Build.BuildNumber that comes out of the MSBuild process to display to the user, and, if I can do that, I can just feed the variables for database and blob container via the same mechanism, and skip the pre-processor.
You've already found the way you can pass data from the MsBuild Arguments directly into the code. An alternative is to use the Condition Attribute in your project files to make certain property groups optional, it allows you to even include specific files conditionally. You can control conditions by passing in /p:ConditionalProperty=value on the MsBuild command. This at least ensures people use a set of values that make sense together.
The problem is that when MsBuild is running in Incremental mode it is likely to not process your changes (as you've noticed), the reason for this, is that the input files remain unchanged since the last build and are all older than the last generated output files.
To by-pass this behavior you'd normally create a separate solution configuration and override the output location for all projects to be unique for that configuration. Combined with setting the Compiler constants for that specific configuration you're ensured that when building that Configuration/Platform combination, incremental builds work as intended.
I do want to echo some of the comments from JerryM and Daniel Mann. Some items are better stored in else where or updated before you actually start the compile phase.
Possible solutions:
Store your configuration data in config files and use Configuration Transformation to generate the right config file base don the selected solution configuration. The process is explained on MSDN. To enable configuration transformation on all project types, you can use SlowCheetah.
Store your ocnfiguration data in the config files and use MsDeploy and specify a Parameters.xml file that matches the deploy package. It will perform the transformation on deploy time and will actually allow your solution to contain a standard config file you use at runtime, plus a publish profile which will post-process your configuration. You can use a SetParameters.xml file to override the variables at deploy time.
Create an installer project (such as through Wix) and merge the final configuration at install time (similar to the MsDeploy). You could even provide a UI which prompts for specific values (and can supply default values).
Use a CI server, like the new TFS/VSTS 2015 task based build engine and combine it with a task that can search&replace tokens, like the Replace Tokens task, Tokenization Task, Colin's ALM Corner Build and Release Tasks. And a whole bunch that specifically deal with versioning. Handling these things in the CI server also allows you to do a quick build locally at all times and do these relatively expensive steps on the build server (patching source code breaks incremental build in MsBuild, because there are always newer input files.
When talking specifically about versioning, there are a number of ways to set the AssemblyVersion and AssemblyFileVersion just before compile time, usually it involves overriding the AssemblyInfo.cs file before compilation. Your code could then use reflection to read the value at runtime. You can use the AssemblyInformationalversion to specify something like you do in the example above which contains .xxx or other text. It also ensures that the version displayed always reflects the information obtained when reading the file properties through Windows Explorer.
In a project where some targets are to be build and run on the build platform and other targets are to be build for a cross platform; what options do we have, when using cmake?
Currently I use CMAKE_BUILD_TYPE to define tool chain, build type and platform (for example -D CMAKE_BUILD_TYPE=arm_debug). In one place in the build, I switch tools (compilers, linke etc.), command line flags, libraries etc. according to the value of CMAKE_BUILD_TYPE. For every build type, I create a build directory.
This approach has it's drawbacks: multiple build directories and no easy way to depend one target from one build type on a target in an other build type (some kind of precompiler needed on the build platform by the build for the cross platform for example).
As currently every build targets has a single tool chain to be used I would love to associate a target with a target platform / tools set. This implies that some libraries have to be build for more than one target platform with different tool sets.
The 'one build type and platform per CMake run' limitation is fundamental and I would strongly advise against trying to work around it.
The proper solution here seems to me to split the build into several stages. In particular, for the scenario where a target from one build type depends on a target from another build type, you should not try to have those two targets in the same CMake project. Proper modularization is key here. Effective use of CMake's include command can help to avoid code duplication in the build scripts.
The big drawback of this approach is that the build process becomes more complex, as you now have several interdependent CMake projects that need to be built in a certain order with specific configurations. Although you already seem to be way beyond the point where you can build your whole system with a single command anyway. CMake can help manage this complexity with tools like ExternalProject, that allows you to build a CMake project from within another. Depending on your particular setup, a non-CMake layer written in your favorite scripting language might also be a viable alternative for ensuring that the different subprojects get built in the correct order.
The sad truth is though that complex build setups are hard to manage. CMake does a great job at providing a number of tools for tackling this complexity but it cannot magically make the problem easier. Most of the limitations that CMake imposes on its user are there for a reason, namely that things would be even harder if you tried to work without them.
This question is about the project command and, by extension, what the concept of a project means in cmake. I genuinely don't understand what a project is, and how it differs from a target (which I do understand, I think).
I had a look at the cmake documentation for the project command, and it says that the project command does this:
Set a name, version, and enable languages for the entire project.
It should go without saying that using the word project to define project is less than helpful.
Nowhere on the page does it seem to explain what a project actually is (it goes through some of the things the command does, but doesn't say whether that list is exclusive or not). The cmake.org examples take us through a basic build setup, and while it uses the project keyword it also doesn't explain what it does or means, at least not as far as I can tell.
What is a project? And what does the project command do?
A project logically groups a number of targets (that is, libraries, executables and custom build steps) into a self-contained collection that can be built on its own.
In practice that means, if you have a project command in a CMakeLists.txt, you should be able to run CMake from that file and the generator should produce something that is buildable. In most codebases, you will only have a single project per build.
Note however that you may nest multiple projects. A top-level project may include a subdirectory which is in turn another self-contained project. In this case, the project command introduces additional scoping for certain values. For example, the PROJECT_BINARY_DIR variable will always point to the root binary directory of the current project. Compare this with CMAKE_BINARY_DIR, which always points to the binary directory of the top-level project. Also note that certain generators may generate additional files for projects. For example, the Visual Studio generators will create a .sln solution file for each subproject.
Use sub-projects if your codebase is very complex and you need users to be able to build certain components in isolation. This gives you a very powerful mechanism for structuring the build system. Due to the increased coding and maintenance overhead required to make the several sub-projects truly self-contained, I would advise to only go down that road if you have a real use case for it. Splitting the codebase into different targets should always be the preferred mechanism for structuring the build, while sub-projects should be reserved for those rare cases where you really need to make a subset of targets self-contained.