TeamCity - Microsoft.Bcl.Build Dependency - msbuild

I've just committed some code changes to my repository and all of a sudden (after weeks of being fine). The TC build starts failing as it fails to download the NuGet package for Microsoft.Bcl.Build.1.0.6.
I ended up having to manually copy the contents of the packages directory to the TC build location which utterly defeats the point of NuGet.
What can I check to get to the root cause of this?
Everything about NuGet is enabled in the solution for getting packages.

I've blogged about this issue at http://sedodream.com/2012/12/24/SlowCheetahBuildServerSupportUpdated.aspx. To summarize NuGet package restore (prior to 2.7) is implemented as a part of the MSBuild build process. When MSBuild starts a build it will evaluate the project file and any Import declarations importing other files. This happens before any target gets executed.
Since NuGet pkg restore is a part of the build process the .targets files get restored at a point in time that it is too late for the Import statement to have any impact.
You can work around this by either checking in the .targets file as you stated, or by invoking pkg restore before the build process. I've created a NuGet package, PackageRestore, which can help with the latter approach.
To use PackageRestore just add the NuGet package to your project which will automatically create a file named packageRestore.proj in your projects directory. When configuring your build, you will need to build that item before your .sln/.csproj file.

OK This is a bit of a nasty issue.
If you are having this problem you need to do something rather ugly to your repository.
Make sure you are checking in the packages\repositories.config file.
Then if your build is failing with unresolved references to Microsoft.Bcl.Build you will need to also checkin the .targets file for this package. eg:
package\Microsoft.Bcl.Build.x.x.x\tools\Microsoft.Bcl.Build.targets
Hideous ...

This blog post is the most thorough one I've seen for explaining the workaround options:
http://blogs.msdn.com/b/dotnet/archive/2013/06/12/nuget-package-restore-issues.aspx
None are great IMO - this problem still needs a better solution.
But for the time being, the best recommended option is to check the Bcl.Build .targets file into source control - which means when the version of Bcl.Build is updated, you'll need to add the new .targets file, and remove the old one.

I think (but am not sure, so I created this SO question: What does the Microsoft.Bcl.Build NuGet package do?) that Microsoft.Bcl.Build is only needed for development, and is not needed on a build server. So, I have a Builder.targets file that only exists in the build environment, that is indirectly <import> ed into all of our projects, which includes this bit of MSBuild xml:
<!-- Skip Microsoft.Bcl.Build functionality when building only from Source. Presumably Microsoft.BclBuild is only needed for development. -->
<PropertyGroup>
<BclBuildImported>Ignore</BclBuildImported>
</PropertyGroup>
Since the block of MSBuild logic inserted into your project by the Bcl.Build nuget package is dependent on the BclBuildImported property being empty, this effectively sidesteps the problem in my build environment - the Microsoft.Bcl.Build steps are skipped, and it no longer breaks my CI builds.
Note that since it appears that this package manages binding redirects in your app.config, and ensures that transitive dependencies are included in your projects, it's important to leave in for development. But I'm not currently aware of a need for it in a build server environment.

Related

Create a nuget package with GeneratePackageOnBuild and version patching

What I've been strugling with for some time:
I have a "traditional" (i.e. non-SDK) csproj file. I want to create a nuget package. That's easy but devil hides in the details and I end up in the following catch-22
I can manually call nuget pack .\Foo.csproj, with or without a nuspec file, but then I get the dreaded NU5128 error. The chosen answer suggests to simply ignore the error. I would happily do that, but after publishing the package locally, I noticed that the dependencies are not included. Though there is one dependency to JSON.net that I added for the sake of testing. The same error appears in AppVeyor if I configure my yml to create the package.
I can specify all the packaging related properties in the .csproj file by setting GeneratePackageOnBuild to true. It works and takes the dependencies right but then I can't patch the version like I could do with the replacement tokens of nuspec. The version defaults to 1.0.0.0 even if I change it in the AssemblyInfo.cs to 1.2.3.4.
Notes
This answer suggests that nuspec and csproj configuration can coexist and they are merged. The question is from 2013 so I guess it's not referring to SDK projects. Unfortunately that's not the case for me.
I tried adding a <NuspecFile> in the csproj but then it seems that replacement tokens don't work and I get the error described here. Probably because this instructed nuget to run pack on the nuspec instead of running it on csproj and merge the nuspec. I can't know for sure though as setting the build verbosity to Detailed didn't help to understand what nuget pack tried to do.
Getting the AssemblyInformationalVersion in msbuild and use it straight to the <Version> property isn't as straightforward as one might think and it also defeats the purpose as $version$ is doing exactly that according to the documentation:
AssemblyInformationalVersion if present, otherwise AssemblyVersion
Is there a way to really "merge" .csproj and .nuspec with $version$ patching and get the dependencies in the nuspec right?
So far the best course of action was to convert the project to SDK format. It has several advantages. Less configuration, no need for .nuspec file, no need for AssemblyInfo.cs or VersionInfo.cs files, .nupkg dependencies are correct and AppVeyor can patch directly the .csproj file if it's an SDK project file.
Not a solution per se if one wants to keep the legacy format, but it gets you going with the CI/CD instead of having you fighting with tools.

Wix Nuget Packages

So here's the scenario: I have several wix projects that the exact same extra msbuild targets and tasks to automate some very tedious tasks: things like building a help file and authenticode signing the msi and output files using signtool.exe.
Having gotten tired of copying changes to these scripts into 5 different repositories every time I made a fix. So I got clever and I turned them into nuget packages with a .targets file in the build folder. A simple nuget restore turns a laundry list of packages into targets files.
The problem
Wix v3 doesn't like packages. It doesn't support package references so packages.config looks like the thing. Except that visual studio doesn't like packages.config. It will happily restore the packages (into the folder I specified with a nuget.config) but none of the .targets files are automatically imported.
Easy enough to fix: I just added an import statement into my wixproj file. Except that visual studio doesn't automatically re-evaluate the wixproj after running the restore operation. We can work with this, but it is really annoying to have to build, unload the project, reload the project, and rebuild any time the packages change.
So I tried to fix it
I crawled through the log output of a standard C# project build and discovered the Nuget.Targets file, which can be imported into a project to provide pack and restore targets. It also let me convert to packagereferences, which are somewhat more convenient and less messy than packages.config, but I still had to manually add the import statement for the nuget.g.targets and nuget.g.props files that were produced by the restore target.
And, since I had to manually import those files, I'm right back to the build, unload, reload, rebuild workflow.
Attempt number 2
I discovered through intense internet searching that msbuild recently added a /restore flag to their command line utility specifically to force a project re-evaluation after restoring packages. This works on the command line, but visual studio seems to be clueless. (Maybe because of something that wix v3 does behind the scenes in the Visual Studio template?) In any case, they mentioned in the issue notes that they are essentially using an msbuild task with new global properties to force a re-evaluation. So I attempted to add such a step to the pre-build event of my wixproj, but no sauce.
The question
How do I tell visual studio to re-evaluate this wixproj after running restore? Is there some sort of property or item group I need to add to the wixproj? Is there some kind of target I can insert into the build process?
Update 1:
Found a project property that looked promising: UseHostCompilerIfAvailable, but it didn't work. Still got the same problem.

Code builds fine in VS (2017) but not msbuild or TFS "assemblies manifest does not match the assembly reference

OK - here we go again. I posted an almost identical question here, but this one is a little different. I just generated a new version of a NuGet package and updated all of our csproj and config files to point to it and it builds locally fine, but when I kick off a TFS build it fails with the message:
##[error]ASPNETCOMPILER(0,0): Error ASPCONFIG: Could not load file or assembly 'PacsgearLib, Version=2.5.4.0, Culture=neutral,
PublicKeyToken=05b30ac9ab9dbb8e' or one of its dependencies. The
located assembly's manifest definition does not match the assembly
reference. (Exception from HRESULT: 0x80131040)
Pacsgearlib 2.5.4 is the NuGet package I just generated, but all of our other projects build fine (and they share a lot of the same code). The main difference is that this project has a website in it. MSBuild is version 15.5.180.51428.
Here's a picture of the actual error:
Any ideas here? I've tried NuGet package manager to uninstall the package and reinstall it for the website and that hasn't helped.
I've been at this for a few hours so I thought maybe someone else (or a second set of eyes) might help.
I also just saw this and will check it out first thing Monday morning.
I finally got this to work by taking out the assemblybinding section of the web.config (someone else suggested that in another related post) and I also found a few .csproj files that had not been properly updated and were still pointing at 2.5.3.
If the local builds fine, and the error exist with TFS build. Besides, all of our other projects build fine (and they share a lot of the same code).
Combine with both, the issue may related to the specific package PacsgearLib, Version=2.5.4.0 on build agent. You could try to delete the packages and clean build agent, then try again.
Another way is update your package version number(do not update the package content), use TFS trigger the build again. Check if TFS build server will pick up the latest version of package.
Also set system.debug=true to Enable Verbose Debug Mode with more detail log info for troubleshooting

Integrating NuGet into a large, existing code base, with a shared packages directory and a shared solutions directory

Q: Is it possible/feasible, to have a multiple solutions stored in a single 'Solutions' directory, and multiple NuGet packages stored in another single 'Packages' directory, and for everything to work nicely with different versions?
Further details...
For example: I have 2 projects. ProjectA requires Newstonsoft.Json.4.5.11, ProjectB requires Newstonsoft.Json.5.0.6.
For sake of example I have a solution file for both. I need all my solution files in the same directory (this is just the process that is followed, all the solutions in a directory are built in turn).
By default NuGet will create a packages directory alongside each solution file.
I have created a nuget.config file to allow me to store packages in a single directory, called 'SharedPackages', following this answer: Nu-Get & issue with project level dependences for projects referenced by multiple solutions
<settings>
<repositoryPath>..\SharedPackages</repositoryPath>
</settings>
This works great so far, so my structure is:
\Projects\ProjectA
\Projects\ProjectB
\Solutions
\SharedPackages
If I create ProjectB, it has Json.NET 4.5.11 by default. If I go to Manage NuGet Packages for Solution I have the option to update it to version 5.0.6. This is great as ProjectB needs the newer version. What is even better is now in my Shared Packages directory I have a directory for both versions of Json.NET side-by-side, so ProjectA can use the older version.
However, now I want to create ProjectC as a full MVC4 Web Application. For JQuery, you get version 1.8.2 currently when creating an ASP.NET MVC4 application in VS2012. I also get Knockout 2.2.0.
My process is, I delete the default packages directory, move the new solution to the Solutions directory alongside the existing nuget.config and edit the new solution file to update the relative path to the new .csproj file. Then when I build, NuGet Package Manager restores the extra packages I need (that weren't in use by ProjectA and ProjectB) to the Shared Packages directory. However... I get build errors, it cannot resolve some references including DotNetOpenAuth, WebGrease, System.Spatial... the references are pointing to the packages directory, not the SharedPackages directory...
As an aside: if I Enable Package Restore for solution, then it also tries to restore them to a packages folder within the Solutions directory by default, instead of restoring them to the SharedPackages directory.
Around about this point I realise that just creating the nuget.config file wasn't enough for ProjectA and ProjectB either, although they appeared to be working originally, the references in the .csproj. file are pointing to the bin folder beneath the project file, instead of my SharedPackages directory.
So I manually 'Find and Replace' ..\packages with ..\..\SharedPackages for all the references. I have to do this for ProjectA, ProjectB and ProjectC. Now everything builds and seems to work OK, new packages go into the right place.
Now, if I go back to ProjectA, and add the Knockout package, this is version 2.3.0. This installs happily alongside the other Knockout package in use by Project C which is version 2.2.0. Doing this also installs JQuery 2.0.3, alongside JQuery 1.8.2. So far so good.
Just as a sanity check, I create another Web Application - ProjectD, move the files around, update the references in the solution and the project. This time everything builds first time. I try and update WebGrease in ProjectD to see if it will retain the older version for ProjectC. This results in more issues, it installs it to the packages directory instead. WebGrease seems to have a separate config setting as well <WebGreaseLibPath>... it won't seem to restore...
I then go back to ProjectB and try 'Update All' - it looks like the files that already exist are updated in SharedPackages, with new version directories alongside the existing ones, but any new dependencies (e.g. now I have a reference to Owin.dll) get placed in the packages folder :( If I delete the packages folder, and the bin folder within ProjectB, then build the ProjectB solution, understandably I get build errors, the packages aren't automatically restored to the SharedPackages directory at any point.
Is it even possible to set NuGet to update packages in a common directory other than packages alongside the solution?
Would it be easier to just use the default packages folder, instead of SharedPackages, or would I still have problems?
This is turning into way too many questions. To try and keep it in scope, has anyone attempted a similar setup, what obstacles did they overcome and how did they manage it, or did they give up altogether? If you gave up, how did you end up using NuGet to manage packages in a massive code base?
I appreciate this is close to this question, which was well answered for that particular question, however the use case here is slightly different: NuGet and multiple solutions. It is also pretty much identical to this question: Setting up a common nuget packages folder for all solutions when some projects are included in multiple solutions, but I have decided to add this anyway as that question is more focused on the having different configurations for different solutions, whereas here I want all the packages in one place, I just want to implement it and see if it is possible. Also I think the troubleshooting and research time may be useful to someone.

MSBuild overwriting dependencies

Ok, so I've got a somewhat complicated problem with my build environment that I'm trying to deal with.
I have a solution file that contains multiple C# projects which is built by a NAnt script calling MSBuild - passing MSBuild the name of the solution file and a path to copy the binaries to. This is because I want my automated build environment (CruiseControl.Net) to create a folder named after the revision of each build - this way I can easily go back to previous binaries for any reason.
So idealy I have a folder layout like this
c:\build\nightly\rev1
c:\build\nightly\rev2
c:\build\nightly\rev3
...
c:\build\nightly\rev10
etc.
The problem that's arisen is I recently added the latest version of the Unity IoC container to my project, checking it directly out of MS's online SVN repository. What's happening is I have a Silverlight 3 project that references the Silverlight version of Unity but I also have other projects (namely my Unit testing project) that reference the standard (non-Silverlight) version of Unity.
So what happens is since MSBuild is dumping everything into one single folder the Silverlight version of the Unity assembly is overwriting the non-Silverlight version because they have the exact same assembly file name.
Then when CruistControl runs my unit tests they fail because they don't have the proper dependencies available anymore (they try to load the Silverlight specific Unity assembly which obviously doesn't work).
So what I want to do is:
keep my desired output directory
structure (folder\revision)
I don't want to have to manually edit
every single proj file I have as this
is error prone when adding new
projects to the solution
Idealy I would like MSBuild to put everything into a folder structure similar to this:
nightly\revision1\project1
nightly\revision1\project2
nightly\revision1\project3
...
nightly\revision2\project1
nightly\revision2\project2
nightly\revision2\project3
etc
I can't modify the Unity project to give it a different file name because it comes from another SVN repository I cannot commit changes to. I found a similar question posted here and the suggested solution was to use a "master" MSBuild file that used a custom task to extract all the project file names out of the solution then loop over each one building them. I tried that but it doesn't build them in the order of their dependencies, so it fails for my project.
Help?
Firstly I would always have the build server delete the old working copy and check out a fresh copy to avoid any problems with stale artifacts from the previous build.
Next I would have nant or msbuild build the solutions as before with the artifacts from each build going to their local working output folders.
After that I'd move the artifacts from their working paths to their output paths, this shouldn't require digging through the project files since you can just tell msbuild/nant to copy working\project1\bin\release\**\*.* to artifacts\project1\.
The script that does this should ideally be stored along with the source with the main file, e.g. build.nant or build.proj in top level of the trunk.
For third party libraries I would simple include the DLLs directory in your repository. Nothing worse than writing some code and having a third party dependency break your build because of changes on their end.
Simply document the versions of the libraries you are using, and if you must update them, you'll have a better sense of what breaks the build before you even check it in.
Also, doesn't CC.Net automatically handle the providing of releases based on revision? I'm using TeamCity and it keeps a copy of the artifacts of every build.
I highly recommend reading JP Boodhoo's Automating Builds with NAnt blog series. That's been my starting point and have made lots of changes for my own taste. I also highly recommend checking out the builds of many open sources projects for examples. I've learned a lot from the builds of the Castle/Nhibernate/Rhino-Tools stack.