I just joined a team that has no CI process in place (not even an overnight build) and some sketchy development practices. There's desire to change that, so I've now been tasked with creating an overnight build. I've followed along with this series of articles to: create a master solution that contains all our projects (some web apps, a web service, some Windows services, and couple off tools that compile to command line executables); created an MSBuild script to automatically build, package, and deploy our products; and created a .cmd file to do it all in one click. Here's a task that I'm trying to accomplish now as part of all this:
The team currently has a practice of keeping the web.config and app.config files outside of source control, and to put into source control files called web.template.config and app.template.config. The intention is that the developer will copy the .template.config file to .config in order to get all of the standard configuration values, and then be able to edit the values in the .config file to whatever he needs for local development/testing. For obvious reasons, I would like to automate the process of renaming the .template.config file to .config. What would be the best way to do this?
Is it possible to do this in the build script itself, without having to stipulate within the script every individual file that needs to be renamed (which would require maintenance to the script any time a new project is added to the solution)? Or might I have to write some batch file that I simply run from the script?
Furthermore, is there a better development solution that I can suggest that will make this entire process unnecessary?
After a lot of reading about Item Groups, Targets, and the Copy task, I've figured out how to do what I need.
<ItemGroup>
<FilesToCopy Include="..\**\app.template.config">
<NewFilename>app.config</NewFilename>
</FilesToCopy>
<FilesToCopy Include="..\**\web.template.config">
<NewFilename>web.config</NewFilename>
</FilesToCopy>
<FilesToCopy Include"..\Hibernate\hibernate.cfg.template.xml">
<NewFilename>hibernate.cfg.xml</NewFilename>
</FilesToCopy>
</ItemGroup>
<Target Name="CopyFiles"
Inputs="#(FilesToCopy)"
Outputs="#(FilesToCopy->'%(RootDir)%(Directory)%(NewFilename)')">
<Message Text="Copying *.template.config files to *.config"/>
<Copy SourceFiles="#(FilesToCopy)"
DestinationFiles="#(FilesToCopy->'%(RootDir)%(Directory)%(NewFilename)')"/>
I create an item group that contains the files that I want to copy. The ** operator tells it to recurse through the entire directory tree to find every file with the specified name. I then add a piece of metadata to each of those files called "NewFilename". This is what I will be renaming each file to.
This snippet adds every file in the directory structure named app.template.config and specifies that I will be naming the new file app.config:
<FilesToCopy Include="..\**\app.template.config">
<NewFilename>app.config</NewFilename>
</FilesToCopy>
I then create a target to copy all of the files. This target was initially very simple, only calling the Copy task in order to always copy and overwrite the files. I pass the FilesToCopy item group as the source of the copy operation. I use transforms in order to specify the output filenames, as well as my NewFilename metadata and the well-known item metadata.
The following snippet will e.g. transform the file c:\Project\Subdir\app.template.config to c:\Project\Subdir\app.config and copy the former to the latter:
<Target Name="CopyFiles">
<Copy SourceFiles="#(FilesToCopy)"
DestinationFiles="#(FilesToCopy->'%(RootDir)%(Directory)%(NewFileName)')"/>
</Target>
But then I noticed that a developer might not appreciate having his customized web.config file being over-written every time the script is run. However, the developer probably should get his local file over-written if the repository's web.template.config has been modified, and now has new values in it that the code needs. I tried doing this a number of different ways--setting the Copy attribute "SkipUnchangedFiles" to true, using the "Exist()" function--to no avail.
The solution to this was building incrementally. This ensures that files will only be over-written if the app.template.config is newer. I pass the names of the files as the target input, and I specify the new file names as the target output:
<Target Name="CopyFiles"
Input="#(FilesToCopy)"
Output="#(FilesToCopy->'%(RootDir)%(Directory)%(NewFileName)')">
...
</Target>
This has the target check to see if the current output is up-to-date with respect to the input. If it isn't, i.e. the particular .template.config file has more recent changes than its corresponding .config file, then it will copy the web.template.config over the existing web.config. Otherwise, it will leave the developer's web.config file alone and unmodified. If none of the specified files needs to be copied, then the target is skipped altogether. Immediately after a clean repository clone, every file will be copied.
The above turned out be a satisfying solution, as I've only started using MSBuild and I'm surprised by its powerful capabilities. The only thing I don't like about it is that I had to repeat the exact same transform in two places. I hate duplicating any kind of code, but I couldn't figure out how to avoid this. If anyone has a tip, it'd be greatly appreciated. Also, while I think the development practice that necessitates this totally sucks, this does help in mitigating that suck factor.
Short answer:
Yes, you can (and should) automate this. You should be able to use MSBuild Move task to rename files.
Long answer:
It is great that there is a desire to change from a manual process to an automatic one. There are usually very few real reasons not to automate. Your build script will act as living documentation of how build and deployment actually works. In my humble opinion, a good build script is worth a lot more than static documentation (although I am not saying you should not have documentation - they are not mutually exclusive after all). Let's address your questions individually.
What would be the best way to do this?
I don't have a full understanding of what configuration you are storing in those files, but I suspect a lot of that configuration can be shared across the development team.
I would suggest raising the following questions:
Which of the settings are developer-specific?
Is there any way to standardise local developer machines so that settings could be shared?
Is it possible to do this in the build script itself, without having to stipulate within the script every individual file that needs to be renamed?
Yes, have a look at MSBuild Move task. You should be able to use it to rename files.
...which would require maintenance to the script any time a new project is added to the solution?
This is inevitable - your build scripts must evolve together with your solution. Accept this as a fact and include in your estimates time to make changes to your build scripts.
Furthermore, is there a better development solution that I can suggest that will make this entire process unnecessary?
I am not aware of all the requirements, so it is hard to recommend something very specific. I can say suggest this:
Create a shared build script for your solution
Automate manual tasks as much as possible (within reason)
If you are struggling to automate something - it could be an indicator of an area that needs to be rethought/redesigned
Make sure your team mates understand how the build works and are able to make changes to it themselves - don't "own" the build and become a bottleneck
Bear in mind that going from no build script to full automation is not an overnight process. Be patient and first focus on automating areas that are causing the most pain.
If I have misinterpreted any of your questions, please let me know and I will update the answer.
Related
My colleagues and I have user specific settings in csproj.user files. They are not checked into the repository. I would like for the build server to use its own set of csproj.user files, overriding certain properties, leaving the "base" project configuration at a decent developer default. But from the looks of it there is no such option in the msbuild command-line for doing that.
Is there really no way, other than copy csproj.user-files to where it'll be picked up by subsequent msbuild invocations?
While writing I realize I'm too much of a prude about these things and should just copy as a step prior build. Still posting in case someone knows a better way, for instance a way that does not modify the source tree.
Passing properties to the MSBuild command line overrides properties in the solution, including dependent projects. Here omitting debug information in build server, otherwise generated for release build to improve profiling:
msbuild MySolution.sln /p:DebugType=none ...
This does not work should I want different properties for different projects. Building projects individually should work nicely though.
Finally, passing arguments on command line can get messy, so to get a more "settings file"-like experience one may instead use #file arguments and MSBuild response files.
My goal is to create build definitions within Visual Studio Team Services for both test and production environments. I need to update 2 variables in my code which determine which database and which blob storage the environment uses. Up till now, I've juggled this value in a Resource variable, and pulled that value in code from My.Resources.DB for a library, and Microsoft.Azure.CloudConfigurationManager.GetSetting("DatabaseConnectionString") for an Azure worker role. However, changing 4 variables every time I do a release is getting tiring.
I see a lot of posts that get close to what I want, but they're geared towards C#. For reasons beyond my influence, this project is written in VB.NET. It seems I have 2 options. First, I could call the MSBuild process with a couple of defined properties, passing them to the .metaproj build file, but I don't know how to get them to be used in VB code. That's preferable, but, at this point, I'm starting to doubt that this is possible.
I've been able to set some pre-processor constants, to be recognized in #If-#Else directives.
#If DEBUG = True Then
BarStaticItemVersion.Caption = String.Format("Version: {0}", "1.18.0.xxx")
#Else
BarStaticItemVersion.Caption = String.Format("Version: {0}", "1.18.0.133")
#End If
msbuild CalbertNG.sln.metaproj /t:Rebuild /p:DefineConstants="DEBUG=False"
This seems to work, though I need to Rebuild to change the value of that constant. Should I have to? Should Build be enough? Is this normal, or an indication that I don't have something set quite right?
I've seen other posts that talk about pre-processing the source files with some other builder, like Ant, but that seems like overkill. It feels like I'm close here. But I want to zoom out and ask, from a clean sheet of paper, if you're given 2 variables which need to change per environment, you're using VB.NET, and you want to incorporate those variable values in an automated VS Team Services build process upon code check-in, what's the best way to do it? (I want to define the variables in the VSTS panel, but this just passes them to my builder, so I have to know how to parse the call to MSBuild to make these useful.)
I can control picking between 2 static strings, now, via compiler directives, but I'd really like to reference the Build.BuildNumber that comes out of the MSBuild process to display to the user, and, if I can do that, I can just feed the variables for database and blob container via the same mechanism, and skip the pre-processor.
You've already found the way you can pass data from the MsBuild Arguments directly into the code. An alternative is to use the Condition Attribute in your project files to make certain property groups optional, it allows you to even include specific files conditionally. You can control conditions by passing in /p:ConditionalProperty=value on the MsBuild command. This at least ensures people use a set of values that make sense together.
The problem is that when MsBuild is running in Incremental mode it is likely to not process your changes (as you've noticed), the reason for this, is that the input files remain unchanged since the last build and are all older than the last generated output files.
To by-pass this behavior you'd normally create a separate solution configuration and override the output location for all projects to be unique for that configuration. Combined with setting the Compiler constants for that specific configuration you're ensured that when building that Configuration/Platform combination, incremental builds work as intended.
I do want to echo some of the comments from JerryM and Daniel Mann. Some items are better stored in else where or updated before you actually start the compile phase.
Possible solutions:
Store your configuration data in config files and use Configuration Transformation to generate the right config file base don the selected solution configuration. The process is explained on MSDN. To enable configuration transformation on all project types, you can use SlowCheetah.
Store your ocnfiguration data in the config files and use MsDeploy and specify a Parameters.xml file that matches the deploy package. It will perform the transformation on deploy time and will actually allow your solution to contain a standard config file you use at runtime, plus a publish profile which will post-process your configuration. You can use a SetParameters.xml file to override the variables at deploy time.
Create an installer project (such as through Wix) and merge the final configuration at install time (similar to the MsDeploy). You could even provide a UI which prompts for specific values (and can supply default values).
Use a CI server, like the new TFS/VSTS 2015 task based build engine and combine it with a task that can search&replace tokens, like the Replace Tokens task, Tokenization Task, Colin's ALM Corner Build and Release Tasks. And a whole bunch that specifically deal with versioning. Handling these things in the CI server also allows you to do a quick build locally at all times and do these relatively expensive steps on the build server (patching source code breaks incremental build in MsBuild, because there are always newer input files.
When talking specifically about versioning, there are a number of ways to set the AssemblyVersion and AssemblyFileVersion just before compile time, usually it involves overriding the AssemblyInfo.cs file before compilation. Your code could then use reflection to read the value at runtime. You can use the AssemblyInformationalversion to specify something like you do in the example above which contains .xxx or other text. It also ensures that the version displayed always reflects the information obtained when reading the file properties through Windows Explorer.
I've got a question about building a deployment script using SSDT.
Could anyone tell me if it's possible to build a deployment script using SQLPackage.exe where the source file is NOT a dacpac file, but uses the .sql files instead?
To give some background, I've created a project in Visual Studio 2012 for my database schema. This works great, and SSDT builds the folder structure without a problem (functions, stored procedures etc which contain all the .sql files).
Here's the problem - the database in question is from a legacy system, and is riddled with errors. Most of these errors we don't care about anymore and it's not practical or safe to fix them all, so for years we've basically ignored them. However it means we can't build the project and therefore can't generate the dacpac file. Now this doesn't prevent us from doing the schema compare and syncing the database with the file system (a local mercurial repository). However it does seemingly prevent us from building a deployment script.
What I'm looking for is a way of building the deployment script using SQLPackage.exe without having to generate the dacpac file. I need to use the .sql files in the file system instead. Visual Studio will produce a script of the differences without building the dacpac, so this makes me think it must be possible to do it using SQLPackage.exe using one of the parameters.
Here's an example of SQLPackage.exe which I'd like to adapt to use the .sql files instead of the dacpac:
sqlpackage.exe /Action:Script /SourceFile:"E:\SourceControl\Project\Database
\test_SSDTProject\bin\Debug\test_SSDTProject.dacpac" /TargetConnectionString:"Data
Source=local;Initial Catalog=TestDB;User ID=abc;Password=abc" /OutputPath:"C:
\temp\hbupdate.sql" /OverwriteFiles:true /p:IgnoreExtendedProperties=True
/p:IgnorePermissions=True /p:IgnoreRoleMembership=True /p:DropObjectsNotInSource=True
This works fine because it uses the dacpac file. However I need to point it at the folder structure where the .sql files are instead.
Any help would be great.
As has been suggested in comments, I think that biting the bullet and fixing the errors is the way ahead. You say
it's not practical or safe to fix them all,
but I think you should give this a bit more thought. I have recently been in a similar situation to you, and the key to emerging from it is to realise that the operational risk associated with dropping procedures and functions that will throw an exception as soon as they are called is zero.
Note that this does not apply if the reason these objects won't build is that they contain cross-database or cross-server references that are present in production but not in your project; this is a separate problem altogether, but also a solvable one.
Nor am I in favour of "exclude from build" as an alternative to "delete"; a while ago I saw a project where this technique had been deployed extensively; it makes it harder to see what does what from the source files and I am now of the opinion that "Build Action=None" is simply "commenting out the bits that don't work" for the Snapchat generation.
The key to all of this, of course, is source control. This addresses the residual risk that one day you might indeed want to implement a working version of one of your currently non-working procedures, using the non-working code as a starting point. It also obviates the need to keep stuff hanging around in the solution using Build Action=None, as one can simply summon an earlier revision of the code that contained the offending objects.
If my experience is any guide, 60 build errors is nothing; these could easily be caused by references to three or four objects that no longer exists, and can be consigned to the dustbin of source control with some enthusiastic use of the "Delete" key.
Do you have a copy of SQL Compare at your disposal? If not, it might be worth downloading the trial to see if it will work in your scenario.
Here are the available switches:
http://documentation.red-gate.com/display/SC10/Switches+used+in+the+command+line
At the very least you'll need to specify the following:
/scripts1:
/server2:
/database2:
/ScriptFile:
I have a C# project which is built in a few different configurations. Some of the source files should be always included, and some only in certain configurations. So far I've been doing this with #if ... #endif around the entire files, but I was hoping to create a small extension to do this a nicer way.
I've created an extension that adds an item to files' context menus, but I can't find any way to set the Condition attribute on the item node in the project file.
I've looked at the Properties collection of the EnvDTE.ProjectItem interface, but can't see anything useful there (except BuildAction... I'll come back to that).
Then I tried getting an IVsBuildPropertyStorage on the item and calling SetItemAttribute(). This does add information to the project file, but as a child element like this:
<ItemGroup>
<Compile Include="Program.cs">
<Condition>%27%24%28Configuration%29%27==%27Debug%27</Condition>
</Compile>
</ItemGroup>
when what I was trying to achieve was:
<ItemGroup>
<Compile Include="Program.cs" Condition="'$(Configuration)'=='Debug'" />
</ItemGroup>
There's also an IVsBuildPropertyStorage.SetPropertyValue() but that adds a similar child element to a PropertyGroup section near the top, not to the item node.
I've looked at 'Project Subtypes/Flavors', but that looks like it's just going to get me another IVsBuildPropertyStorage, which doesn't seem to be useful. They do look capable of a lot of complex things, but documentation on the subject appears to be minimal and vague.
I've seen some posts describing how to use the MSBuild assemblies to directly load and manipulate the project file, but I'm not sure when is safe to do that without confusing Visual Studio and potentially losing changes, since VS prompts to reload when it detects changes to the project file.
As a last idea, I thought about manipulating the BuildAction property between Compile and None, but that sounds like it could be a lot of work for my extension to maintain correctly, keeping it in sync with every time the user switches configurations in the IDE for example.
Is there anyone with any experience with this kind of thing that has any advice to offer me, or should I give up hope and stick with manually adding #if directives everywhere?
You may like to explore the MSBuild option you mentioned.
You don't actually have to load the MSBuild project from file, because Visual Studio gives you a way of accessing the MSBuild project directly, i.e.:
string projectPath = projectItem.ContainingProject.FullName;
MsBuildProject project = ProjectCollection.GlobalProjectCollection.GetLoadedProjects(projectPath);
var compileItems = project.GetItems("Compile");
From there you can locate your specific items and potentially add the condition attribute, though I haven't tried this step myself (if this doesn't work, you might have to try modifying the project elements under the project.Xml property instead).
You can then call project.Save(), which shouldn't trigger the "Reload project?" dialog because of the way the MsBuild project instance is linked to the Visual Studio project hierarchy.
However, you may like to force Visual Studio to reload the project anyway, because if you switch build configurations (e.g. between Debug and Release), the MSBuild engine may not re-evaluate your item conditions during build. The code to do this programmatically can be found here:
How do I programmatically refresh/reload a VS project after modifying the underlying file?
Unfortunately I never got the time to persue the original goal of creating an extension for doing this, however I did achieve what I needed using the suggestion by lex-li: using separate project files per configuration.
Since the project files can all reside in the same directory, it's easy to simply use the 'Include/Exclude from project' context menu item in the solution explorer to choose which files are included. There's also no need for file linking this way, which I'd tried before and found very time-consuming to manage.
Partial Methods are also worth looking at, if you have similar needs. They allow you to define the signature of a method in one place, but optionally implement it elsewhere. If you don't implement it, no call is generated by the compiler.
With respect to the original idea of the extension, I suspect the answer by Daniel Nolan was heading in the right direction, but unfortunately I didn't get to try it out.
As the title says, I want to output 2 .msi files from one project (one is per-user, and the other is per-machine). I have seen in this thread that it can't be done in some conventional way, but perhaps there is a way to do so as some kind of hack in post-build.
I only need to rerun compilation and linking after the original build with slightly changed command line (actually, I need only different Product.wxs file). However, my light and candle command lines are huge, and I would risk making my project hard to maintain if I would hard-code them.
So, in conclusion, I need to know if there is a way to write a command line that would behave the same as Votive does when creating its build command line (getting all the files in project, linking them, passing project dependencies...), only in post build.
P.S.: I also had an idea of getting the whole command line from Votive, and only changing the Product file, that would also help, so if someone has suggestion on how to do it...
Create multiple configurations of your solution / project and set a preprocessor variable to some value for one of the configurations. In your wix source, conditionally include whatever else it is that needs included based on your preprocessor variable.