I created some msbuild scripts that build my projects using msbuild task. I pass as parameters the list of projects to build (not the solution file).
In VS2010, building the solution the second time without modifying any files will finish almost at the same time since it detect that nothing was changed.
Executing the msbuild scripts the second/third/forth time will take the same time.
Any ideas how to make the msbuild to behave the same as vs2010?
Thanks,
Eran.
Many MSBuild tasks shipped by Microsoft, uses IncrementallBuild or Filetracker in order to track which files were used to produce outputs of given file. Knowing relation between inputs and outputs MSBuild can create dependency graph.
By comparing timestamp of output file and all input files can eliminate execution on many tasks. If timestamps of all input files for given output file are "lower/before" than output file, it means none input file were modified, MSBuild can skip execution of this task.
I case of VC++ project this information is logged into *.tlog files (stored in IntermediateDirectory $(IntDir) of project).
By execution of target Rebuild you can skip using IncrementalBuild feature it because rebeuild means clean and build (so tlog files are deleted).
Related
I have an MSBuild target which transforms a file in place, that is the input and output file is the same file. How do I get Visual Studio to handle the integration build properly in this case? Must I necessarily move the file?
A short answer: never transform file in place during the build. Any transformation should necessarily create a new file in some other location, or with a different name.
Long answer. MSBuild tracks if file is up-to-date based on time stamps. This is the same mechanism used by make and most of other build systems. This allows your incremental builds perform a partial rebuild of only necessary build steps. Note, that in order to make incremental builds work, you have to declare all input and output files in target's Inputs and Outputs attributes. If you have the same file in Inputs as well as in Outputs, it will be considered up-to-date and target would not be executed, which breaks your build.
If you don't care about incremental build, you can actually make it work, by simply creating a target that has no Inputs and Outputs. In this case target will always be executed. That comes with a price that you are doing full build every time.
I just joined a team that has no CI process in place (not even an overnight build) and some sketchy development practices. There's desire to change that, so I've now been tasked with creating an overnight build. I've followed along with this series of articles to: create a master solution that contains all our projects (some web apps, a web service, some Windows services, and couple off tools that compile to command line executables); created an MSBuild script to automatically build, package, and deploy our products; and created a .cmd file to do it all in one click. Here's a task that I'm trying to accomplish now as part of all this:
The team currently has a practice of keeping the web.config and app.config files outside of source control, and to put into source control files called web.template.config and app.template.config. The intention is that the developer will copy the .template.config file to .config in order to get all of the standard configuration values, and then be able to edit the values in the .config file to whatever he needs for local development/testing. For obvious reasons, I would like to automate the process of renaming the .template.config file to .config. What would be the best way to do this?
Is it possible to do this in the build script itself, without having to stipulate within the script every individual file that needs to be renamed (which would require maintenance to the script any time a new project is added to the solution)? Or might I have to write some batch file that I simply run from the script?
Furthermore, is there a better development solution that I can suggest that will make this entire process unnecessary?
After a lot of reading about Item Groups, Targets, and the Copy task, I've figured out how to do what I need.
<ItemGroup>
<FilesToCopy Include="..\**\app.template.config">
<NewFilename>app.config</NewFilename>
</FilesToCopy>
<FilesToCopy Include="..\**\web.template.config">
<NewFilename>web.config</NewFilename>
</FilesToCopy>
<FilesToCopy Include"..\Hibernate\hibernate.cfg.template.xml">
<NewFilename>hibernate.cfg.xml</NewFilename>
</FilesToCopy>
</ItemGroup>
<Target Name="CopyFiles"
Inputs="#(FilesToCopy)"
Outputs="#(FilesToCopy->'%(RootDir)%(Directory)%(NewFilename)')">
<Message Text="Copying *.template.config files to *.config"/>
<Copy SourceFiles="#(FilesToCopy)"
DestinationFiles="#(FilesToCopy->'%(RootDir)%(Directory)%(NewFilename)')"/>
I create an item group that contains the files that I want to copy. The ** operator tells it to recurse through the entire directory tree to find every file with the specified name. I then add a piece of metadata to each of those files called "NewFilename". This is what I will be renaming each file to.
This snippet adds every file in the directory structure named app.template.config and specifies that I will be naming the new file app.config:
<FilesToCopy Include="..\**\app.template.config">
<NewFilename>app.config</NewFilename>
</FilesToCopy>
I then create a target to copy all of the files. This target was initially very simple, only calling the Copy task in order to always copy and overwrite the files. I pass the FilesToCopy item group as the source of the copy operation. I use transforms in order to specify the output filenames, as well as my NewFilename metadata and the well-known item metadata.
The following snippet will e.g. transform the file c:\Project\Subdir\app.template.config to c:\Project\Subdir\app.config and copy the former to the latter:
<Target Name="CopyFiles">
<Copy SourceFiles="#(FilesToCopy)"
DestinationFiles="#(FilesToCopy->'%(RootDir)%(Directory)%(NewFileName)')"/>
</Target>
But then I noticed that a developer might not appreciate having his customized web.config file being over-written every time the script is run. However, the developer probably should get his local file over-written if the repository's web.template.config has been modified, and now has new values in it that the code needs. I tried doing this a number of different ways--setting the Copy attribute "SkipUnchangedFiles" to true, using the "Exist()" function--to no avail.
The solution to this was building incrementally. This ensures that files will only be over-written if the app.template.config is newer. I pass the names of the files as the target input, and I specify the new file names as the target output:
<Target Name="CopyFiles"
Input="#(FilesToCopy)"
Output="#(FilesToCopy->'%(RootDir)%(Directory)%(NewFileName)')">
...
</Target>
This has the target check to see if the current output is up-to-date with respect to the input. If it isn't, i.e. the particular .template.config file has more recent changes than its corresponding .config file, then it will copy the web.template.config over the existing web.config. Otherwise, it will leave the developer's web.config file alone and unmodified. If none of the specified files needs to be copied, then the target is skipped altogether. Immediately after a clean repository clone, every file will be copied.
The above turned out be a satisfying solution, as I've only started using MSBuild and I'm surprised by its powerful capabilities. The only thing I don't like about it is that I had to repeat the exact same transform in two places. I hate duplicating any kind of code, but I couldn't figure out how to avoid this. If anyone has a tip, it'd be greatly appreciated. Also, while I think the development practice that necessitates this totally sucks, this does help in mitigating that suck factor.
Short answer:
Yes, you can (and should) automate this. You should be able to use MSBuild Move task to rename files.
Long answer:
It is great that there is a desire to change from a manual process to an automatic one. There are usually very few real reasons not to automate. Your build script will act as living documentation of how build and deployment actually works. In my humble opinion, a good build script is worth a lot more than static documentation (although I am not saying you should not have documentation - they are not mutually exclusive after all). Let's address your questions individually.
What would be the best way to do this?
I don't have a full understanding of what configuration you are storing in those files, but I suspect a lot of that configuration can be shared across the development team.
I would suggest raising the following questions:
Which of the settings are developer-specific?
Is there any way to standardise local developer machines so that settings could be shared?
Is it possible to do this in the build script itself, without having to stipulate within the script every individual file that needs to be renamed?
Yes, have a look at MSBuild Move task. You should be able to use it to rename files.
...which would require maintenance to the script any time a new project is added to the solution?
This is inevitable - your build scripts must evolve together with your solution. Accept this as a fact and include in your estimates time to make changes to your build scripts.
Furthermore, is there a better development solution that I can suggest that will make this entire process unnecessary?
I am not aware of all the requirements, so it is hard to recommend something very specific. I can say suggest this:
Create a shared build script for your solution
Automate manual tasks as much as possible (within reason)
If you are struggling to automate something - it could be an indicator of an area that needs to be rethought/redesigned
Make sure your team mates understand how the build works and are able to make changes to it themselves - don't "own" the build and become a bottleneck
Bear in mind that going from no build script to full automation is not an overnight process. Be patient and first focus on automating areas that are causing the most pain.
If I have misinterpreted any of your questions, please let me know and I will update the answer.
As the title says, I want to output 2 .msi files from one project (one is per-user, and the other is per-machine). I have seen in this thread that it can't be done in some conventional way, but perhaps there is a way to do so as some kind of hack in post-build.
I only need to rerun compilation and linking after the original build with slightly changed command line (actually, I need only different Product.wxs file). However, my light and candle command lines are huge, and I would risk making my project hard to maintain if I would hard-code them.
So, in conclusion, I need to know if there is a way to write a command line that would behave the same as Votive does when creating its build command line (getting all the files in project, linking them, passing project dependencies...), only in post build.
P.S.: I also had an idea of getting the whole command line from Votive, and only changing the Product file, that would also help, so if someone has suggestion on how to do it...
Create multiple configurations of your solution / project and set a preprocessor variable to some value for one of the configurations. In your wix source, conditionally include whatever else it is that needs included based on your preprocessor variable.
Is it possible to take a list of projects and parallel build them only if they aren't up to date?
<Target Name="DependenciesLevel7" DependsOnTargets="DependenciesLevel6">
<Message Text="5 items to build" />
<MSBuild Projects="C:\Projects\ApplicationManager.csproj;C:\Projects\Metrics.csproj" Properties="$(CustomAllProperties)" BuildInParallel="true">
<Output TaskParameter="TargetOutputs" ItemName="built_DependenciesLevel7" />
</MSBuild>
This is an example of the format i'm building in, I was hoping to be able to parallel build only items that aren't up to date here? Perhaps internal msbuild task calls are automatically parallel? If so how would I set this up so that it's incremental based on the previous build task? (Target DependenciesLevel6) I believed for incremental building you have to use Inputs/Outputs in your target.
Question summary:
Is the list of projects passed to an msbuild task automatically incremental (building is skipped if the build is already up to date)?
If not, is it possible to do incremental on a list of projects while parallelizing?
Can you do between target incremental where each target is a parallel build?
what you are describing "parallel incremental building" is already built in (kind of) but what you really need is parallel partially building.
Let me first explain the concepts and then I'll circle back how it works in your scenario specifically. There are two things you need to know about; incremental building and partial building. I just wrote a blog post discussing this at http://sedodream.com/2010/09/23/MSBuildYouveHeardOfIncrementalBuildingButHaveYouHeardOfPartialBuilding.aspx but I'll paste the relevant parts here.
Incremental building is the concept that you should only build what is out of date. To support this MSBuild has the attributes, inputs and outputs on the Target element. With these attributes you can specify the files that go into a target (via inputs attribute), and the files that you are expecting to come out of a target (via outputs attribute). Once you do this MSBuild will compare the timestamp of the inputs to the outputs and if all outputs are up-to-date (i.e. the inputs are older) then the target will be skipped. Take a look at the very simple project file below.
<Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
<ItemGroup>
<Files Include="src\01.txt;src\02.txt;src\03.txt;src\04.txt;src\05.txt;"/>
</ItemGroup>
<PropertyGroup>
<Dest>dest\</Dest>
</PropertyGroup>
<Target Name="CopyFiles"
Inputs="#(Files)"
Outputs="#(Files->'$(Dest)%(Filename)%(Extension)')">
<Message Text="CopyFiles" />
<Copy SourceFiles="#(Files)"
DestinationFiles="#(Files->'$(Dest)%(Filename)%(Extension)')"/>
</Target>
<Target Name="DeleteTwoFiles">
<Message Text="DeleteTwoFiles" />
<Delete Files="$(dest)01.txt;$(dest)02.txt"/>
</Target>
</Project>
In this project file we have two targets; CopyFiles and DeleteTwoFiles. Ignore DeleteTwoFiles for now. Also take a note that the directory where I’m executing this build has a folder, src, with the files listed in the Files item. On the CopyFiles target I have specified the inputs and outputs. The inputs is just #(Files), this are the files that the target is acting upon. The outputs contains the expression #(Files->'$(Dest)%(Filename)%(Extension)'). Which is the same expression from the Copy statement. If the Dest folder is empty and I execute the CopyFiles target the result is shown below.
So just as expected the files were copied over, so its all good. Now what happens if I execute it again? The output is shown below
So as you can see the target was skipped, the message statement “CopyFiles” was not executed nor was the copy as a result. So this, in a nutshell, is incremental building.
Now, with the dest folder containing all files, what do you think would happen I execute the command msbuild.exe PartialBuilding01.proj /t:DeleteTwoFiles;CopyFiles? This command will first delete two files from the output directory and then call the CopyFiles target again. Let’s see the result below.
When the CopyFiles target was executed you see that statement “Building target ‘CopyFiles’ partially, …”. When the time came to execute the target MSBuild examined the inputs and outputs, it determined that the files 01.txt & 02.txt were out of date (because they didn’t exist in the target) but 03.txt, 04.txt and 05.txt were up to date. So MSBuild feed the CopyFiles target a value for the Files item that only contained the 01.txt and 02.txt and let it do its thing.
Now this relates to your problem in many ways some not as direct as you might hope. Firstly MSBuild will incrementally build your project, so if your project is up to date then it will not be built again. The thing is though that in order for MSBuild to determine that your project is up to date it has to load the project run the default target (usually Build) and then the targets themselves will figure out that there is no work to do. This stuff itself takes time. So if you have a huge number of projects, or a huge number of files inside of a project then you can take matters into your own hands. What you need is a way to determine if your projects are up to date or not and correctly express that inside of your inputs and outputs attributes. Once you do this you should be able to skip building the projects which are up to date.
The core of the problem is how do you craft the inputs/outputs to be correct. If you can think of a way to do that then you will get what you want. How you craft this will depend on your scenario but I could see something like this:
After each project build drop a file to a known location that is specific to that project
Before you build a project scan its directory, find the newest file and then update the timestamp of the project file to be that value
Then you can place the project files as the Inputs values and the marker files as the Outputs
Then call your target
In this case you assume that all dependencies are fully contained in files under the directory of the project (which may not be true). I'm not saying this is the ideal solution, but a solution.
==============================================
Edit: Update based on questoins below.
You will want to put the projects into an item (though not required) like ProjectFiles and then use #(ProjectFiles) for inputs. For outputs that is what I was saying is the hard part. You have to figure out a way to know (or indicate to you via your own process) that the projects are up to date. There is nothing built in for this.
Concern fo incremental build vs. clean build. In a perfect world incremental & clean builds are the same. But sometimes that is not the case. For many projects it is. If you start adding a bunch of targets to your build process and you set them up to do incremental build, but you do not implement that properly then you may MSBuild may skip targets when they were indeed out of date. A good example of this would be when you create a target with Inputs set to a list of files and then the Outputs set to a list of created files. If you do not extend the clean process to delete those created files, then the next time you Rebuild (assuming you didn't change the files) the target will be skipped when it should have been cleaned on the previous Rebuild.
You're probably looking for IncrediBuild.
I'm not sure about the incremental part but I read that it checks inputs/ouputs.
I make parallel builds with MsBuild with a configuration similar to yours, don't forget to run msbuild with the /m option.
You can try on a simple project and check the logs to see if it's incremental or not.
First, I have a base assumption from watching Visual Studio compile things with its default .*proj files that, if you build the same solution twice in a row, it detects that nothing has changed and seems to fly through the solution build. Does this mean it knows that nothing was changed in a project and doesn't have to make a new DLL output?
If that's the case, I have a question. Say I have a solution with multiple class libraries, and an MSBuild task in each project that automatically increments the build's version by modifying AssemblyInfo.cs. Thing is (if my previous assumption is correct) it does this every time and triggers a new rebuild of each class library. Is there a target or property in MSBuild that can tell if the project needs recompilation, and skip my versioning step if so?
I ask because let's say I update project A, but not project B in a solution. If I run a build on the solution, I want it to update the version on project A, but since project B hasn't changed, I want to leave it alone.
Found something: http://msdn.microsoft.com/en-us/library/ms171483.aspx
MSBuild can compare the timestamps of
the input files with the timestamps of
the output files and determine whether
to skip, build, or partially rebuild a
target. In the following example, if
any file in the #(CSFile) item
collection is newer than the hello.exe
file, MSBuild will run the target;
otherwise it will be skipped:
<Csc
Sources="#(CSFile)"
OutputAssembly="hello.exe"/> </Target>
...that worked. But then got me thinking, what if someone pulls the code down from source control without the assemblies (which is how we do it)? Since it has no output to compare against, it'll do a compile and increment the version anyway. I think the complexities might lead me to abandon this approach.
It doesn't really matter if you increment on a developers box - what's important is that your daily/CI build is only incremented when needed. So, what I've done in the past is have some small XML file contain the next build number, and have an MSBuild task take this xml file and create a file called Version.cs (containing the versioning attributes you'd usually find in AssemblyInfo.cs).
Version.cs is never checked into your soure control - it's generated by the build.
Developers will sync the current XML file, build their binaries, and get the current version number. The continous integration build may also do the same thing. But a daily/official build will check out the XML file, increment the version information, and then check it in. From that moment on the version number has officially changed.
There are variations on this theme, but the general idea works.