Azure DevOps - conditional package restoration - msbuild

A solution that is built via Azure DevOps pipeline has some projects that use conditional package references such as:
<Choose>
<When Condition="'$(Configuration)'=='Debug'">
<ItemGroup>
<PackageReference Include="Mock.MyPackage" Version="1.0.0" />
</ItemGroup>
</When>
<Otherwise>
<ItemGroup>
<PackageReference Include="MyPackage" Version="1.2.0" />
</ItemGroup>
</Otherwise>
</Choose>
The package source is a private NuGet feed.
The dotnet restore task does not respect the conditional package selection. (it restores Mock.MyPackage)
Question:
how can I conditionally restore packages (based on a $Configuration) ?
Remarks:
I have also tried restoring during Visual Studio Build task by specifying an MsBuild argument: /t:restore.
In that case it fails with a message: Failed to retrieve information about XX from remote source. If this command can restore packages how can I specify authorization args for the private feed ?
There is an issue on Github: https://github.com/NuGet/Home/issues/5895 where such issue is mentioned at the end.

Azure DevOps - conditional package restoration
This issue should be related your specify requirement and the limitation for current Visual Studio Build task/dotnet restore.
Just as you test, if we use the restore task, we could not specify the configuration parameter with this task. Since there is no such option to receive the configuration parameter for the restore task. That is the reason why it always restores the default package Mock.MyPackage.
If you use the Visual Studio build task, we could not to specify authorization args for the private feed.
To resolve this issue, I use the Command line V1 task to invoke MSBuild to restore and build the project with following MSBuild argument:
-t:restore;build "CoreConditionRestore/CoreConditionRestore/CoreConditionRestore.csproj" -p:RestoreSources="<MyFeed>/v3/index.json" -p:RestoreConfigFile="<MyNugetConfigPath>\nuget.config" /p:Configuration=Debug
Note: we could save the authorization args for the private feed in the nuget.config file.
As test, it works fine on my side with Devops.
Hope this helps.

A much, much simpler answer is to add in NUGET_RESTORE_MSBUILD_ARGS with a value of /p:Configuration=$(BuildConfiguration) to your pipeline variables. This will actually just pass your build configuration to the NuGet restore task.
This is incredibly poorly documented and it took scouring github bug requests to find it
Environment variable documentation:
https://learn.microsoft.com/en-us/nuget/reference/cli-reference/cli-ref-environment-variables
Github issue I found this on:
https://github.com/NuGet/Home/issues/7575
Not sure why this isn't enabled by default, either.
As a note, you still need to do the (imo strange)
<Choose>
<When Condition="'$(Configuration)'=='Debug'">
<ItemGroup>
<PackageReference Include="Mock.MyPackage" Version="1.0.0" />
</ItemGroup>
</When>
</Choose>
in your project file rather than the more intuitive:
<PackageReference Include="Mock.MyPackage" Version="1.0.0" Condition="'$(Configuration)' == 'Debug'" />
...which does work with normal Reference directives eg:
<Reference Include="MOCK_DLL" Condition="'$(Configuration)' == 'Debug'">
<HintPath>..\References\MOCK_DLL.dll</HintPath>
</Reference>

I wanted minimal change in deployment configuration, so the following was acceptable.
An empty project is created and all packages which are restored conditionally are referenced there (without any conditions).
Additional argument to msbuild task is added: '/t:restore'
During build the following will happen:
'dotnet restore' will restore all the packages thanks to 1).
Since it does not know a selected configuration it may pick a wrong package (Mock.MyPackage instead of MyPackage).
That is where 2) comes, where msbuild task will restore packages from a local cache made by 'dotnet restore'.
Remarks:
As #(Leo Liu-MSFT) wrote, dotnet restore can authenticate but it does not know configuration, and msbuild knows configuration but it can not authenticate, so package restoration in AzDevOps from private feeds is tricky.

Related

How to kill VBCSCompiler.exe on Ubuntu agent?

I created an Azure Devops Build pipeline and i am trying to build my ASP.NET MVC and Angular hybrid site project on bitbucket (git).
The project first gets checked out, and nuget restores the necessary packages, and then the .NET builds. I used windows 2019 as azure pipeline agent for the build to succeed. however, Its taking about 7 minutes to complete, whilst running the tasks (besides .Net) on a ubuntu agent is much faster! takes around 2 mins instead!
Therefore, I'd like to use ubuntu, but im running into an issue with the MSBuild task...
"/home/vsts/work/1/s/Bobby.ProjectA/Bobby.ProjectA.csproj" (default target) (1) ->
(KillVBCSCompilerAndRetryCopy target) ->
/home/vsts/work/1/s/packages/Microsoft.CodeDom.Providers.DotNetCompilerPlatform.1.0.8/build/net45/Microsoft.CodeDom.Providers.DotNetCompilerPlatform.props(23,5):
error MSB4044: The "KillProcess" task was not given a value for the required parameter "ImagePath". [/home/vsts/work/1/s/Bobby.ProjectA/Bobby.ProjectA.csproj]
According to this post, VBCSCompiler.exe continues running from the Compiler Nuget package (nuget restore task?) so it locks the src folders and prevented future builds from running, e.g. causing error like this:
/home/vsts/work/1/s/packages/Microsoft.CodeDom.Providers.DotNetCompilerPlatform.1.0.8/build/net45/Microsoft.CodeDom.Providers.DotNetCompilerPlatform.props(17,5):
warning MSB3021: Unable to copy file "/home/vsts/work/1/s/packages/Microsoft.Net.Compilers.2.4.0/build/../tools/csc.exe" to "/bin/roslyn/csc.exe". Access to the path '/bin/roslyn' is denied. [/home/vsts/work/1/s/Bobby.ProjectA/Bobby.ProjectA.csproj]
So the solution would be to kill the VBCSCompiler.exe but since i cant actually access the hosted machine during the build, im not sure how to do that.
screenshot of my pipeline so far:
Am i facing a dead-end path here with this approach? The build runs fine on windows 2019 but it just takes too long, so thats why if i can make it run on ubuntu successfully that would be great!
You can have a try with below workarounds:
1,Set MSBUILD arguements /p:UseSharedCompilation=false.
You can add above arguement to the msbuild arguements field of the msbuild task. See here.
2,Upgrade Microsoft.CodeDom.Providers.DotNetCompilerPlatform nupkg to the latest and remove Microsoft.Net.Compilers nupkg from your project. See here for more information.
3, Try Specifing the TTL of Roslyn compiler server.
You can define a pipeline variable VBCSCOMPILER_TTL on the Variable tab to specify a shorter idle time for VBCSCompiler.exe
Or you can add <providerOption name="CompilerServerTimeToLive" value="[num of seconds]" /> under system.codedom/compilers/compiler in the config file. See here for more information.
4, Use CheckIfShouldKillVBCSCompiler target:
You can try add below to your csproj file:
<Target Name="CheckIfShouldKillVBCSCompiler">
<PropertyGroup>
<ShouldKillVBCSCompiler>true</ShouldKillVBCSCompiler>
</PropertyGroup>
</Target>
See here.
The build on Ubuntu 20 finally worked! I don't know why removing these lines resolved the VBCSCompiler issue, but by doing so, the msbuild completed successfully on Ubunutu 20 agent!!
Remove the following lines from the .csproj file:
<Import Project="..\packages\Microsoft.Net.Compilers.2.4.0\build\Microsoft.Net.Compilers.props" Condition="Exists('..\packages\Microsoft.Net.Compilers.2.4.0\build\Microsoft.Net.Compilers.props')" />
<Error Condition="!Exists('..\packages\Microsoft.Net.Compilers.2.4.0\build\Microsoft.Net.Compilers.props')" Text="$([System.String]::Format('$(ErrorText)', '..\packages\Microsoft.Net.Compilers.2.4.0\build\Microsoft.Net.Compilers.props'))" />
<Error Condition="!Exists('..\packages\Microsoft.CodeDom.Providers.DotNetCompilerPlatform.1.0.8\build\net45\Microsoft.CodeDom.Providers.DotNetCompilerPlatform.props')" Text="$([System.String]::Format('$(ErrorText)', '..\packages\Microsoft.CodeDom.Providers.DotNetCompilerPlatform.1.0.8\build\net45\Microsoft.CodeDom.Providers.DotNetCompilerPlatform.props'))" />
To give some context to the answer, this post here indicated that converting from MSBuild-Integrated Package Restore to Automatic Package Restore (nuget restore task) implied that the Microsoft.Net.Compilers <Import> and <Error Condition> snippets are no longer relevant/needed in the .csproj file.

How do you disable Roslyn Analyzers when using msbuild through the command line?

The Roslyn Analyzers are installed as nuget packages, which are dependencies of the FxCop Analyzers (also installed as nuget packages).
I have enabled full solution analysis as instructed here: How to Enable and disable full solution analysis for managed code.
I have a fairly large solution with most of the projects using the FxCop/Roslyn Analyzers and Visual Studio builds fine, usually in under a minute.
However, when running msbuild through the command line using:
"C:/Program Files (x86)/Microsoft Visual Studio/2017/Community/MSBuild/15.0/Bin/MSBuild.exe" "C:\Source\MySolution\MySmartClient.sln" /p:Configuration=Develop;Platform="Any CPU" /
t:Build
Building the solution takes anywhere from 4-15 minutes. The same is true on the build server which uses the same command.
I've tried /p:RunCodeAnalysis=False and that has no effect. I've also used process monitor to emulate the msbuild command that VS sends to msbuild with no change.
And, according to this doc: How to: Enable and disable automatic code analysis for managed code
The Enable Code Analysis on Build check box only affects static code analysis. It doesn't affect Roslyn code analyzers, which always execute at build if you installed them as a NuGet package.
These excessive build times are not practical. Is there any way to disable when using msbuild through the command line?
It's not really supported, but there is a workaround:
Create a Directory.Build.targets (msbuild >= v15.0), After.{SolutionName}.sln.targets (msbuild < 15.0) file in your solution root folder and add:
<Project>
<Target Name="DisableAnalyzers"
BeforeTargets="CoreCompile"
Condition="'$(UseRoslynAnalyzers)' == 'false'">
<!--
Disable analyzers via an MSBuild property settable on the command line.
-->
<ItemGroup>
<Analyzer Remove="#(Analyzer)" />
</ItemGroup>
</Target>
</Project>
You can pass in /p:UseRoslynAnalyzers=false now to remove all analyzers configured in the project.
See also:
https://github.com/dotnet/roslyn/issues/23591#issuecomment-507802134
https://learn.microsoft.com/en-us/visualstudio/msbuild/customize-your-build?view=vs-2019#directorybuildprops-and-directorybuildtargets
You can edit the condition to also trigger on RunCodeAnalysis=False or Never.
<Target Name="DisableAnalyzers"
BeforeTargets="CoreCompile"
Condition="
'$(UseRoslynAnalyzers)' == 'false'
or '$(RunCodeAnalysis)' == 'false'
or '$(RunCodeAnalysis)' == 'never'" >
To disable a specific analyzer, use this trick:
We just spent 2 hours figuring out how to disable an analyzer based on an MSBuild property, AMA.
https://twitter.com/Nick_Craver/status/1173996405276467202?s=09
The documentation has changed since the original answers. There is now this page documenting how to disable code analysis from analyzers:
There are 3 MSBuild properties you can use to control analyzer behavior (all default to true):
RunAnalyzersDuringBuild Controls whether analyzers run at build time.
RunAnalyzersDuringLiveAnalysis Controls whether analyzers analyze code live at design time.
RunAnalyzers Disables analyzers at both build and design time. This property takes precedence over RunAnalyzersDuringBuild and RunAnalyzersDuringLiveAnalysis.
Edit: it looks like there is an issue being tracked where these props don't work unless your project has Microsoft.CodeAnalysis.targets included. So your mileage may vary until this is fixed.
In case anyone else happens to find themselves here, I came across this issue on the dotnet/roslyn project on Github:
Feature: MSBuild switch for turning on/off analysis #23591
The preceding issue describes a work-around:
Substitute for old MSBuild properties? #1431
<PropertyGroup>
<RunCodeAnalysis Condition="'$(RunCodeAnalysis)' == ''">true</RunCodeAnalysis>
</PropertyGroup>
<ItemGroup>
<PackageReference Include="<whatever analyzers package you are depending on>" Condition="'$(RunCodeAnalysis)' == 'true'" />
</ItemGroup>
# You'll need to run a restore when changing this value
msbuild /p:RunCodeAnalysis=false
Although, I had a couple of differences though since I'm not using package references. This worked for me.
<ItemGroup>
<Analyzer Include="<whatever analyzers package you are depending on>" Condition="'$(RunCodeAnalysis)' == 'true'" />
</ItemGroup>
<!-- I added the condition to the EnsureNugetPackageBuildImports too. -->
<Target Name="EnsureNuGetPackageBuildImports" BeforeTargets="PrepareForBuild">
<PropertyGroup>
<ErrorText>This project references NuGet package(s) that are missing on this computer. Use NuGet Package Restore to download them. For more information, see http://go.microsoft.com/fwlink/?LinkID=322105. The missing file is {0}.</ErrorText>
</PropertyGroup>
<Error Condition="'$(RunCodeAnalysis)' == 'true' AND !Exists('<relative path to the prop of whatever analyzers you are depending on>')" Text="$([System.String]::Format('$(ErrorText)', '<relative path to the prop of whatever analyzers you are depending on>'))" />
</Target>

Is it possible to utilize OctoPack with the new PackageReference NuGet format?

We recently upgraded our assemblies to use the PackageReference format instead of the packages.config for our NuGet dependencies. One of the packages, OctoPack, stopped working after doing this. Is there any way to get OctoPack to work while still using the PackageReference format?
Yes, but you have to use Octo.exe pack (same process as for ASP.NET Core applications)
Yes. Recent Nuget versions will autogenerate imports for the MSBuild tasks in the obj/xxx.csproj.nuget.g.targets file.
If it doesn't...
When using packages.config, Nuget will add something like this to your csproj.
<Target Name="EnsureNuGetPackageBuildImports" BeforeTargets="PrepareForBuild">
<PropertyGroup>
<ErrorText>This project references NuGet package(s) that are missing on this computer. Enable NuGet Package Restore to download them. For more information, see http://go.microsoft.com/fwlink/?LinkID=322105. The missing file is {0}.</ErrorText>
</PropertyGroup>
<Error Condition="!Exists('..\packages\OctoPack.3.6.4\build\OctoPack.targets')" Text="$([System.String]::Format('$(ErrorText)', '..\packages\OctoPack.3.6.4\build\OctoPack.targets'))" />
and
<Import Project="..\packages\OctoPack.3.6.4\build\OctoPack.targets" Condition="Exists('..\packages\OctoPack.3.6.4\build\OctoPack.targets')" />
The EnsureNuGetPackageBuildImports target is probably already there, even before.
If you add this to your csproj (or keep it, if you converted from packages.config) it will sort of still work. The problem is that packages are not stored in "..\packages".
This is easily fixed by replacing e.g "..\packages\OctoPack.3.6.4" with "$(NuGetPackageRoot)\octopack\3.6.4". Of course you need the PackageReference to octopack. It will be enough to make a restore operation put the package in $(NuGetPackageRoot).
YMMV, but it works for me, both locally and in CI (TeamCity)

.net core (csproj) global.json 'projects' equivalent

With .net core (project.json) I used to switch between nuget packages and source code by adding the path to source code to the projects field in the global.json. After I did that it would add all the projects that it could find in that path that could replace the nuget packages I referenced.
I used this feature alot because I have my own nuget packages that I use, but I want to test the changes in my other project before I publish. But once I switched to Sdk 1.0.0/VS 2017/csproj .net core that feature seemed to disappear.
The alternative is just manually adding each project, switch the references manually (since they are broken up into project, nuget and sdk references), and then after switch it all back.
Any thoughts or advice would be great.
UPDATE:
Sounds like there is no equivalent in csproj (as expected), but there are msbuild workarounds for now (As of the initial VS 2017/.NET Core SDK 1.0.0 release)
Yes, I too had gotten used to this functionality and built my workflow around it. I am still looking for a solution but I'm currently playing with the idea of using conditional logic in the csproj files. Since it's now msbuild, you can do things like this:
<Choose>
<When Condition="Exists('..\..\..\MyProject')">
<ItemGroup>
<ProjectReference Include="..\..\..\MyProject\src\MyProject\MyProject.csproj" />
</ItemGroup>
</When>
<Otherwise>
<ItemGroup>
<PackageReference Include="MyProject" Version="1.0.0" />
</ItemGroup>
</Otherwise>
</Choose>
This replaces the hard reference to a package with a conditional that uses a project reference if it can find the source code (in this case the directory), and a package reference if can't.
So by default you would be referencing the package, but if you want to debug one of your projects, you check it out in the location that the conditional checks, and add the project to your solution.
This way you only need to change your solution file (by adding the project) when you want to include source code, instead of rewiring all your project references.
For others that are interested in attempting to emulate with Global.json did, I worked around this for now using a couple powershell scripts and a custom json file that mimics it. Check out my answer here:
https://stackoverflow.com/a/43795974/5504245

MSBuild: Build Custom Project Target When Solution Specified

I need to build at the solution level using MSBuild since I have multiple configurations. I have a custom target in my web project that publishes the output. I can specify the web project as the target and it builds fine. But I need to publish the web application. Is there a way to build a project's custom target when building at the solution level?
I found this link, but I don't understand the solution. If someone could break that down a little, it would help a lot.
This is what my current PowerShell command-line looks like:
& "$env:windir\Microsoft.NET\Framework64\v4.0.30319\MSBuild.exe" `
"..\..\Solution.sln" `
"/t:WebApplication:PublishToFileSystem" `
"/p:Configuration=$configuration;PublishDestination=$publishPath"
The :PublishToFileSystem causes it to fail, with this message:
error MSB4057: The target "PublishToFileSystem" does not exist in the project. [C:\Source\Solution.sln]
This is what my custom target in my .csproj file looks like:
<Target Name="PublishToFileSystem" DependsOnTargets="PipelinePreDeployCopyAllFilesToOneFolder">
<Error Condition="'$(PublishDestination)'==''" Text="The PublishDestination property must be set to the intended publishing destination." />
<MakeDir Condition="!Exists($(PublishDestination))" Directories="$(PublishDestination)" />
<ItemGroup>
<PublishFiles Include="$(_PackageTempDir)\**\*.*" />
</ItemGroup>
<Copy SourceFiles="#(PublishFiles)" DestinationFiles="#(PublishFiles->'$(PublishDestination)\%(RecursiveDir)%(Filename)%(Extension)')" SkipUnchangedFiles="True" />
</Target>
So, after some research, I found that creating a custom target is not the best way to publish a web application. Assuming you have just one web project in your solution, MS extended MSBuild to use publish profiles (the same ones used when you publish in Visual Studio). I was able to create a separate publish profile for each configuration (environment) and call the following command line:
& "$env:windir\Microsoft.NET\Framework64\v4.0.30319\MSBuild.exe" `
"..\..\Solution.sln" `
"/p:DeployOnBuild=true" `
"/p:PublishProfile=$configuration"
I just had to make sure my publish profiles had the same name as my $configuration.