I have an MSBuild target which transforms a file in place, that is the input and output file is the same file. How do I get Visual Studio to handle the integration build properly in this case? Must I necessarily move the file?
A short answer: never transform file in place during the build. Any transformation should necessarily create a new file in some other location, or with a different name.
Long answer. MSBuild tracks if file is up-to-date based on time stamps. This is the same mechanism used by make and most of other build systems. This allows your incremental builds perform a partial rebuild of only necessary build steps. Note, that in order to make incremental builds work, you have to declare all input and output files in target's Inputs and Outputs attributes. If you have the same file in Inputs as well as in Outputs, it will be considered up-to-date and target would not be executed, which breaks your build.
If you don't care about incremental build, you can actually make it work, by simply creating a target that has no Inputs and Outputs. In this case target will always be executed. That comes with a price that you are doing full build every time.
Related
Lets make obscure question simple...
We have a solution which consists of many projects and some of them have set Custom build events using 3rd party stuff for some dark magic compilation and looks similar like this:
<CustomBuild Include="..\folder\somestuff.xyz">
<FileType>Document</FileType>
<Command Condition="'$(Configuration)|$(Platform)'=='Release|x64'">C:\Tcl\bin\tclsh.exe $(APP_PATH)\modules\APP\bin\generator.tcl -o %(RelativeDir)%(Filename) %(RelativeDir)%(Filename).xyz</Command>
<Message Condition="'$(Configuration)|$(Platform)'=='Release|x64'">APPGEN %(RelativeDir)%(Filename)</Message>
<Outputs Condition="'$(Configuration)|$(Platform)'=='Release|x64'">%(RelativeDir)%(Filename).cpp;%(RelativeDir)%(Filename).h;%(RelativeDir)%(Outputs)</Outputs>
</CustomBuild>
This was working properly until we switch form VS2015 to VS2019 as now during the compilation it reports that:
Project is not up-to-date: build output 'd:\projects\program\app\src\plugins\shared\' is missing. This would be more or less ok, but it forces the compiler to recompile also the dependencies of this project and this start to be really annoying as you need to rebuild several projects everytime even when no changes were done.
I found out that the problem originates from this line of Custom build:
<Outputs Condition="'$(Configuration)|$(Platform)'=='Release|x64'">%(RelativeDir)%(Filename).cpp;%(RelativeDir)%(Filename).h;%(RelativeDir)%(Outputs)</Outputs>
More precisely from this part: %(RelativeDir)%(Outputs) as the check for .cpp and .h file in the same tag do not generate any issues. So I removed this check for directory. When this chunk of code is removed the project compiles properly and do not re-compile all day long.
So why the Custom build's Output check is now working properly just with files and directories are generating this kind of issue?
And yes, the examined dir exists and it refers to the existing correct path.
The real problem is that your real project is always rebuild due to the metadata Outputs.
The special point is that you should make sure the validity and legitimacy of the value of Outputs.
The problem is under %(RelativeDir) of %(RelativeDir)%(Outputs). When you add it, the outputs has an illegal folder structure rather than a file which makes the outputs always find the missing illegal folder structure so that causes the project always rebuild.
Let me describe it in detail,
when msbuild reads outputs proeperty, when it reads till %(RelativeDir)%(Filename).cpp;%(RelativeDir)%(Filename).h;%(RelativeDir), the value of Outputs is this:
..\folder\somestuff.cpp;..\folder\somestuff.h;..\folder\
Then, it reads %(Outputs)(reads itself), which is more like copy the above value twice:
..\folder\somestuff.cpp;..\folder\somestuff.h;..\folder\..\folder\somestuff.cpp;..\folder\somestuff.h;..\folder\
You will find the last part ..\folder\ is not a file and it is a folder structure which is illegal for the outputs.
That is the reason.
And it is more like your problem that the folder structure d:\projects\program\app\src\plugins\shared\ is missing.
Suggestion
So you should not add outputs again.
<Outputs Condition="'$(Configuration)|$(Platform)'=='Release|x64'">%(RelativeDir)%(Filename).cpp;%(RelativeDir)%(Filename).h;</Outputs>
My goal is to create build definitions within Visual Studio Team Services for both test and production environments. I need to update 2 variables in my code which determine which database and which blob storage the environment uses. Up till now, I've juggled this value in a Resource variable, and pulled that value in code from My.Resources.DB for a library, and Microsoft.Azure.CloudConfigurationManager.GetSetting("DatabaseConnectionString") for an Azure worker role. However, changing 4 variables every time I do a release is getting tiring.
I see a lot of posts that get close to what I want, but they're geared towards C#. For reasons beyond my influence, this project is written in VB.NET. It seems I have 2 options. First, I could call the MSBuild process with a couple of defined properties, passing them to the .metaproj build file, but I don't know how to get them to be used in VB code. That's preferable, but, at this point, I'm starting to doubt that this is possible.
I've been able to set some pre-processor constants, to be recognized in #If-#Else directives.
#If DEBUG = True Then
BarStaticItemVersion.Caption = String.Format("Version: {0}", "1.18.0.xxx")
#Else
BarStaticItemVersion.Caption = String.Format("Version: {0}", "1.18.0.133")
#End If
msbuild CalbertNG.sln.metaproj /t:Rebuild /p:DefineConstants="DEBUG=False"
This seems to work, though I need to Rebuild to change the value of that constant. Should I have to? Should Build be enough? Is this normal, or an indication that I don't have something set quite right?
I've seen other posts that talk about pre-processing the source files with some other builder, like Ant, but that seems like overkill. It feels like I'm close here. But I want to zoom out and ask, from a clean sheet of paper, if you're given 2 variables which need to change per environment, you're using VB.NET, and you want to incorporate those variable values in an automated VS Team Services build process upon code check-in, what's the best way to do it? (I want to define the variables in the VSTS panel, but this just passes them to my builder, so I have to know how to parse the call to MSBuild to make these useful.)
I can control picking between 2 static strings, now, via compiler directives, but I'd really like to reference the Build.BuildNumber that comes out of the MSBuild process to display to the user, and, if I can do that, I can just feed the variables for database and blob container via the same mechanism, and skip the pre-processor.
You've already found the way you can pass data from the MsBuild Arguments directly into the code. An alternative is to use the Condition Attribute in your project files to make certain property groups optional, it allows you to even include specific files conditionally. You can control conditions by passing in /p:ConditionalProperty=value on the MsBuild command. This at least ensures people use a set of values that make sense together.
The problem is that when MsBuild is running in Incremental mode it is likely to not process your changes (as you've noticed), the reason for this, is that the input files remain unchanged since the last build and are all older than the last generated output files.
To by-pass this behavior you'd normally create a separate solution configuration and override the output location for all projects to be unique for that configuration. Combined with setting the Compiler constants for that specific configuration you're ensured that when building that Configuration/Platform combination, incremental builds work as intended.
I do want to echo some of the comments from JerryM and Daniel Mann. Some items are better stored in else where or updated before you actually start the compile phase.
Possible solutions:
Store your configuration data in config files and use Configuration Transformation to generate the right config file base don the selected solution configuration. The process is explained on MSDN. To enable configuration transformation on all project types, you can use SlowCheetah.
Store your ocnfiguration data in the config files and use MsDeploy and specify a Parameters.xml file that matches the deploy package. It will perform the transformation on deploy time and will actually allow your solution to contain a standard config file you use at runtime, plus a publish profile which will post-process your configuration. You can use a SetParameters.xml file to override the variables at deploy time.
Create an installer project (such as through Wix) and merge the final configuration at install time (similar to the MsDeploy). You could even provide a UI which prompts for specific values (and can supply default values).
Use a CI server, like the new TFS/VSTS 2015 task based build engine and combine it with a task that can search&replace tokens, like the Replace Tokens task, Tokenization Task, Colin's ALM Corner Build and Release Tasks. And a whole bunch that specifically deal with versioning. Handling these things in the CI server also allows you to do a quick build locally at all times and do these relatively expensive steps on the build server (patching source code breaks incremental build in MsBuild, because there are always newer input files.
When talking specifically about versioning, there are a number of ways to set the AssemblyVersion and AssemblyFileVersion just before compile time, usually it involves overriding the AssemblyInfo.cs file before compilation. Your code could then use reflection to read the value at runtime. You can use the AssemblyInformationalversion to specify something like you do in the example above which contains .xxx or other text. It also ensures that the version displayed always reflects the information obtained when reading the file properties through Windows Explorer.
I created some msbuild scripts that build my projects using msbuild task. I pass as parameters the list of projects to build (not the solution file).
In VS2010, building the solution the second time without modifying any files will finish almost at the same time since it detect that nothing was changed.
Executing the msbuild scripts the second/third/forth time will take the same time.
Any ideas how to make the msbuild to behave the same as vs2010?
Thanks,
Eran.
Many MSBuild tasks shipped by Microsoft, uses IncrementallBuild or Filetracker in order to track which files were used to produce outputs of given file. Knowing relation between inputs and outputs MSBuild can create dependency graph.
By comparing timestamp of output file and all input files can eliminate execution on many tasks. If timestamps of all input files for given output file are "lower/before" than output file, it means none input file were modified, MSBuild can skip execution of this task.
I case of VC++ project this information is logged into *.tlog files (stored in IntermediateDirectory $(IntDir) of project).
By execution of target Rebuild you can skip using IncrementalBuild feature it because rebeuild means clean and build (so tlog files are deleted).
Is it possible to take a list of projects and parallel build them only if they aren't up to date?
<Target Name="DependenciesLevel7" DependsOnTargets="DependenciesLevel6">
<Message Text="5 items to build" />
<MSBuild Projects="C:\Projects\ApplicationManager.csproj;C:\Projects\Metrics.csproj" Properties="$(CustomAllProperties)" BuildInParallel="true">
<Output TaskParameter="TargetOutputs" ItemName="built_DependenciesLevel7" />
</MSBuild>
This is an example of the format i'm building in, I was hoping to be able to parallel build only items that aren't up to date here? Perhaps internal msbuild task calls are automatically parallel? If so how would I set this up so that it's incremental based on the previous build task? (Target DependenciesLevel6) I believed for incremental building you have to use Inputs/Outputs in your target.
Question summary:
Is the list of projects passed to an msbuild task automatically incremental (building is skipped if the build is already up to date)?
If not, is it possible to do incremental on a list of projects while parallelizing?
Can you do between target incremental where each target is a parallel build?
what you are describing "parallel incremental building" is already built in (kind of) but what you really need is parallel partially building.
Let me first explain the concepts and then I'll circle back how it works in your scenario specifically. There are two things you need to know about; incremental building and partial building. I just wrote a blog post discussing this at http://sedodream.com/2010/09/23/MSBuildYouveHeardOfIncrementalBuildingButHaveYouHeardOfPartialBuilding.aspx but I'll paste the relevant parts here.
Incremental building is the concept that you should only build what is out of date. To support this MSBuild has the attributes, inputs and outputs on the Target element. With these attributes you can specify the files that go into a target (via inputs attribute), and the files that you are expecting to come out of a target (via outputs attribute). Once you do this MSBuild will compare the timestamp of the inputs to the outputs and if all outputs are up-to-date (i.e. the inputs are older) then the target will be skipped. Take a look at the very simple project file below.
<Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
<ItemGroup>
<Files Include="src\01.txt;src\02.txt;src\03.txt;src\04.txt;src\05.txt;"/>
</ItemGroup>
<PropertyGroup>
<Dest>dest\</Dest>
</PropertyGroup>
<Target Name="CopyFiles"
Inputs="#(Files)"
Outputs="#(Files->'$(Dest)%(Filename)%(Extension)')">
<Message Text="CopyFiles" />
<Copy SourceFiles="#(Files)"
DestinationFiles="#(Files->'$(Dest)%(Filename)%(Extension)')"/>
</Target>
<Target Name="DeleteTwoFiles">
<Message Text="DeleteTwoFiles" />
<Delete Files="$(dest)01.txt;$(dest)02.txt"/>
</Target>
</Project>
In this project file we have two targets; CopyFiles and DeleteTwoFiles. Ignore DeleteTwoFiles for now. Also take a note that the directory where I’m executing this build has a folder, src, with the files listed in the Files item. On the CopyFiles target I have specified the inputs and outputs. The inputs is just #(Files), this are the files that the target is acting upon. The outputs contains the expression #(Files->'$(Dest)%(Filename)%(Extension)'). Which is the same expression from the Copy statement. If the Dest folder is empty and I execute the CopyFiles target the result is shown below.
So just as expected the files were copied over, so its all good. Now what happens if I execute it again? The output is shown below
So as you can see the target was skipped, the message statement “CopyFiles” was not executed nor was the copy as a result. So this, in a nutshell, is incremental building.
Now, with the dest folder containing all files, what do you think would happen I execute the command msbuild.exe PartialBuilding01.proj /t:DeleteTwoFiles;CopyFiles? This command will first delete two files from the output directory and then call the CopyFiles target again. Let’s see the result below.
When the CopyFiles target was executed you see that statement “Building target ‘CopyFiles’ partially, …”. When the time came to execute the target MSBuild examined the inputs and outputs, it determined that the files 01.txt & 02.txt were out of date (because they didn’t exist in the target) but 03.txt, 04.txt and 05.txt were up to date. So MSBuild feed the CopyFiles target a value for the Files item that only contained the 01.txt and 02.txt and let it do its thing.
Now this relates to your problem in many ways some not as direct as you might hope. Firstly MSBuild will incrementally build your project, so if your project is up to date then it will not be built again. The thing is though that in order for MSBuild to determine that your project is up to date it has to load the project run the default target (usually Build) and then the targets themselves will figure out that there is no work to do. This stuff itself takes time. So if you have a huge number of projects, or a huge number of files inside of a project then you can take matters into your own hands. What you need is a way to determine if your projects are up to date or not and correctly express that inside of your inputs and outputs attributes. Once you do this you should be able to skip building the projects which are up to date.
The core of the problem is how do you craft the inputs/outputs to be correct. If you can think of a way to do that then you will get what you want. How you craft this will depend on your scenario but I could see something like this:
After each project build drop a file to a known location that is specific to that project
Before you build a project scan its directory, find the newest file and then update the timestamp of the project file to be that value
Then you can place the project files as the Inputs values and the marker files as the Outputs
Then call your target
In this case you assume that all dependencies are fully contained in files under the directory of the project (which may not be true). I'm not saying this is the ideal solution, but a solution.
==============================================
Edit: Update based on questoins below.
You will want to put the projects into an item (though not required) like ProjectFiles and then use #(ProjectFiles) for inputs. For outputs that is what I was saying is the hard part. You have to figure out a way to know (or indicate to you via your own process) that the projects are up to date. There is nothing built in for this.
Concern fo incremental build vs. clean build. In a perfect world incremental & clean builds are the same. But sometimes that is not the case. For many projects it is. If you start adding a bunch of targets to your build process and you set them up to do incremental build, but you do not implement that properly then you may MSBuild may skip targets when they were indeed out of date. A good example of this would be when you create a target with Inputs set to a list of files and then the Outputs set to a list of created files. If you do not extend the clean process to delete those created files, then the next time you Rebuild (assuming you didn't change the files) the target will be skipped when it should have been cleaned on the previous Rebuild.
You're probably looking for IncrediBuild.
I'm not sure about the incremental part but I read that it checks inputs/ouputs.
I make parallel builds with MsBuild with a configuration similar to yours, don't forget to run msbuild with the /m option.
You can try on a simple project and check the logs to see if it's incremental or not.
First, I have a base assumption from watching Visual Studio compile things with its default .*proj files that, if you build the same solution twice in a row, it detects that nothing has changed and seems to fly through the solution build. Does this mean it knows that nothing was changed in a project and doesn't have to make a new DLL output?
If that's the case, I have a question. Say I have a solution with multiple class libraries, and an MSBuild task in each project that automatically increments the build's version by modifying AssemblyInfo.cs. Thing is (if my previous assumption is correct) it does this every time and triggers a new rebuild of each class library. Is there a target or property in MSBuild that can tell if the project needs recompilation, and skip my versioning step if so?
I ask because let's say I update project A, but not project B in a solution. If I run a build on the solution, I want it to update the version on project A, but since project B hasn't changed, I want to leave it alone.
Found something: http://msdn.microsoft.com/en-us/library/ms171483.aspx
MSBuild can compare the timestamps of
the input files with the timestamps of
the output files and determine whether
to skip, build, or partially rebuild a
target. In the following example, if
any file in the #(CSFile) item
collection is newer than the hello.exe
file, MSBuild will run the target;
otherwise it will be skipped:
<Csc
Sources="#(CSFile)"
OutputAssembly="hello.exe"/> </Target>
...that worked. But then got me thinking, what if someone pulls the code down from source control without the assemblies (which is how we do it)? Since it has no output to compare against, it'll do a compile and increment the version anyway. I think the complexities might lead me to abandon this approach.
It doesn't really matter if you increment on a developers box - what's important is that your daily/CI build is only incremented when needed. So, what I've done in the past is have some small XML file contain the next build number, and have an MSBuild task take this xml file and create a file called Version.cs (containing the versioning attributes you'd usually find in AssemblyInfo.cs).
Version.cs is never checked into your soure control - it's generated by the build.
Developers will sync the current XML file, build their binaries, and get the current version number. The continous integration build may also do the same thing. But a daily/official build will check out the XML file, increment the version information, and then check it in. From that moment on the version number has officially changed.
There are variations on this theme, but the general idea works.