I'm following the instructions on this link https://blogs.msdn.microsoft.com/azuredatalake/2017/10/24/continuous-integration-made-easy-with-msbuild-support-for-u-sql-preview/
It states that
After running MSBuild from command line or as a VSTS task, all scripts in the U-SQL project are built and output to a single file at "Build output path/script name/script name.usql". You can copy this composite U-SQL script to the release folder for further deployment.
Within my Visual Studio project (.usqlproj) I have multiple .usql scripts
CreateDatabase.usql
CreateTable.usql
CreateTvf.usql
when I do clean and msbuild and then check bin\debug folder, all I get is just CreateDatabase.usql and within that there is only CREATE DATBASE statement. As per the blog I would have thought all the 3 usql scripts would have merged into 1 composite usql script. The msbuild command I executed from command prompt on my machine
msbuild TheProject\TheProject.usqlproj /t:Clean /t:Rebuild /property:USQLSDKPath=C:\TheProject\src\packages\Microsoft.Azure.DataLake.USQL.SDK.1.3.1019-preview\build\runtime,USQLTargetType=Merge
I'm using Visual Studio 2017 15.4.3 and Azure Data Lake tools 2.3.0000.1
What am I doing wrong?
Due to some legacy reasons, to make sure all files in the usqlproj are built, the condition are defined in
USqlSDKBuild.targets as below
<ItemGroup>
<FileToBuild Condition="'$(JustOneFile)' == '' and
'$(Build_all_files_in_this_project)' != 'true'" Include="$(ActiveFile)" />
<FileToBuild Condition="'$(JustOneFile)' != '' " Include="$(JustOneFile)" />
<FileToBuild Condition="'$(Build_all_files_in_this_project)' == 'true'"
Include="Build_all_files_in_this_project" />
</ItemGroup>
For current preview release, please add "/property:Build_all_files_in_this_project=true,JustOneFile=''" to the command line to make sure all scripts are built. We will update this in later releases with a easier and simpler condition.
The command msbuild TheProject\TheProject.usqlproj /t:Clean /t:Rebuild /property:USQLSDKPath=C:\TheProject\src\packages\Microsoft.Azure.DataLake.USQL.SDK.1.3.1019-preview\build\runtime,USQLTargetType=Merge actually build the script you selected.
If you are selecting CreateDatabase.usql in VS, it will build the script CreateDatabase.usql in command line.
As below example, since it's selecting test.usql, when execute msbuild command, it will only build test.usql. As the way clicking Build button for the USQL project.
If you want to build all the scripts under the USQL project, you should select Build All Scripts button in VS.
Related
I'm trying to execute msbuild on Azure Devops. Because of that I cannot use the MSBuild task provided.
When I use a Command Line task the command is not recognised. On my local machine I load vcvarsall.bat before I use msbuild. But I've not been unable to work out how to obtain that path in Azure Devops. Doesn't appear to be a Develop Command Prompt task for Azue Devops either.
Any ideas on how I can use msbuild from a Command Line task or Batch Script task? Using their Hosted VS agent.
The best way to do this in a supported way is to use vswhere. The following bit of script will install vswhere (using chocolatey) and then query the installer registry where msbuild can be found. Replace -latest with a more specific version if you need that:
choco install vswhere
for /f "tokens=*" %%i in ('vswhere -latest -requires Microsoft.Component.MSBuild -find MSBuild\**\Bin\MSBuild.exe -nologo') do set msbuildpath="%%i"
echo "##vso[task.setvariable variable=msbuildpath]%msbuildpath%"
This will save the path to msbuild to the environment variable %msbuildpath% as well as the pipeline variable (for this stage) $(msbuildpath).
You can then either use a second run commandline task and pass in $(msbuildpath) or you can simply call MsBuild from the same piece of script mentioned above by calling:
%msbuildpath%
This will make sure your script will remain working, even if Microsoft upgrades their images and moves some things around (which does happen).
You can also get vswhere using wget or invoke-webrequest -outfile from the following location:
https://github.com/Microsoft/vswhere/releases/latest/download/vswhere.exe
Other samples for vswhere syntax can be found on the project wiki, including the syntax for PowerShell.
If you use Hosted Agent 2017 you can run the msbuild.exe from the Command Line task in this way:
Command Line version 1:
Command Line version 2:
Results:
If you are interested in seeing how the built-in Microsoft task resolves the path, all the Azure Devops tasks are provided open-source. These are the path functions you probably care to review.
Here is the solution I came up with using only built-in pipeline tasks which makes the MSBuild bin directory available on the path environment variable.
Create a PowerShell task to generate an MSBuild project to capture and output to a file the variables you are interested in (ex. MSBuildBinPath)
PowerShell script
"<Project DefaultTargets=`"DetectMsBuild`">
<ItemGroup>
<OutFile Include=`"`$(MsBuildDetectionFile)`" />
<OutFile Condition=`"'`$(OutFile)' == ''`" Include=`"msbuildInfo.json`" />
</ItemGroup>
<Target Name=`"DetectMsBuild`">
<PropertyGroup>
<MsBuildPaths>
[{
`"Name`": `"BinPath`",
`"Value`": `"`$(MSBuildBinPath.Replace('\', '\\'))`"
}]
</MsBuildPaths>
</PropertyGroup>
<WriteLinesToFile
File=`"#(Outfile)`"
Lines=`"`$(MsBuildPaths)`"
Overwrite=`"true`"
Encoding=`"UTF-8`" />
</Target>
</Project>" | Out-File -FilePath "msbuilddetect.proj" -Encoding utf8
Set the working directory and any variables accordingly.
PowerShell task settings screenshot:
Create an MSBuild task to run the project file generated by the previous task. Ensure the MSBuild version is set to the version you want to use.
MSBuild task settings screenshot:
Last, create another PowerShell task that will parse the outputted JSON file of the extracted variables and sets environment variables accordingly.
PowerShell script
Write-Host "Current path: $($env.Path)`n`n"
$msBuildVariables = Get-Content -Path msbuildInfo.json | ConvertFrom-Json
$Path = "$($msBuildVariables[0].Value);$($env:Path)"
Write-Host "##vso[task.setvariable variable=Path;]$Path"
PowerShell task settings screenshot:
Here is a screenshot of the task order in the build pipeline.
I have a TFS xaml Build template that runs the msbuild.sonarqube.runner start (and end) before and (after the) msbuild task in the xaml.
It all works well enough with a .sln file. As the file under build.
However, when I attempt to use this on a build that runs msbuild on a .csproj file the end process reports that it cannot find the postprocess exe.
"Execution failed. The specified executable does not exist: .sonarqube\bin\MSBuild.SonarQube.Internal.PostProcess.exe"
there no error indication given by the Start command.
is there something special I need to do to get this to work, or is this a limitation of the MSBuild runner?
Please help.
My whole team is keen to use the tool, but as it is only a fraction of the existing builds are being analyzed
thanks
Jeff Gedney
The begin and end commands of the SonarQube Scanner for MSBuild, as well as all msbuild commands MUST be launched from the same current working directory. Indeed, they will all need access to the .sonarqube folder that is created by the begin command.
Other than that, you can launch MSBuild on a *.csproj file instead of a *.sln if you prefer - that is supported by the SonarQube Scanner for MSBuild.
I've created the following ticket to improve the error message in case end is launched from the wrong folder: https://jira.sonarsource.com/browse/SONARMSBRU-160
Make sure you put yourself in the root folder of the project you want to analyze, then run the following commands:
a.MSBuild.SonarQube.Runner.exe begin /k:"sonarqube_project_key" /n:"sonarqube_project_name" /v:"sonarqube_project_version"
b.Build the project, for example: msbuild /t:Rebuild
c.MSBuild.SonarQube.Runner.exe end
Check:http://docs.sonarqube.org/display/PLUG/C%23+Plugin
I've created a custom MSBuild task for our database project. This tasks uses XLSX file to generate reference data insert scripts that get merged into the post deployment script.
I tested this with a test MSBuild proj and it works well.
Now when I integrate it into the real DBProj file, the output of the task is duplicated and I cant see the MSBuild output logging.
So, my questions are:
1) How can I see the full MSBuild logs in Visual Studio?
2) I'm not sure AfterBuild or BeforeBuild is running twice but maybe?
Thanks
You can debug your MSBuild scripts, set breakpoints, inspect values, etc... See here: http://blogs.msdn.com/b/visualstudio/archive/2010/07/06/debugging-msbuild-script-with-visual-studio.aspx
I'm using MsBuild to build and publish my project (along with CruiseControl.Net). I have set everything up and it works great but the problem is it's overwriting all my existing files in the deployed folder(and the folder contains user data, i could do an xcopy after the build/publish but the user data is a few gigs and that would be too much disk activity on each automated build).
So what I would like to do is use a "Replace only Existing files" option instead of removing everything from the folder. I get this option in Visual Studio 2010 when publishing, you can either "Replace Exiting Files" or "Delete All Files First", how to do this using the msbuild command parameters.
<msbuild>
<executable>C:\WINDOWS\Microsoft.NET\Framework\v4.0.30319\MSBuild.exe</executable>
<workingDirectory>C:\CCnet\dmisr-web_workingdir\MAKANI</workingDirectory>
<projectFile>MAKANI.sln</projectFile >
<buildArgs>/noconsolelogger /v:quiet /p:Configuration=Staging /p:DeployOnBuild=true /p:DeployTarget=PipelinePreDeployCopyAllFilesToOneFolder /p:_PackageTempDir="c:\dMisr\Web - Deployed" /p:AutoParameterizationWebConfigConnectionStrings=false</buildArgs>
<targets>ReBuild</targets >
<timeout>600</timeout >
</msbuild>
This answer may help
How do I keep MSDeploy from deleting extra folders in my project?
Looks like you may just need to supply an extra property to msbuild.
SkipExtraFilesOnServer=True
/p:SkipExtraFilesOnServer=true is still removing other files for me when using these build arguments
/p:DeployOnBuild=true;DeployTarget=PipelinePreDeployCopyAllFilesToOneFolder;_PackageTempDir=\\Network\Share\code /p:AutoParameterizationWebConfigConnectionStrings=false /p:SkipExtraFilesOnServer=true
I do not seem to have the ability to comment. Hence providing my comments in the answer section.
I have a build.proj, that is a MSBuild file and can be run locally.
All I need from TFS is
Get the sources from TFS Source Control.
Call "MSBuild.exe /t:Deploy".
Update the build status based on the result of MSBuild.
I have tried to make a template combining the DefaultTemplate.xaml and UpgradeTemplate.xaml.
But so far, no luck :-(
Can someone help me make this template?
If you select the upgrade template that comes out of the box when you create a new TFS project with 2010, you can supply your old TFS2008 proj (MSBuild) file without problems. Please read http://msdn.microsoft.com/en-us/library/dd647553.aspx for more details.
You should use DefaultTemplate. I had the same problem and I solved it this way.
You can do it using UpgradeTempate also, but using DefaultTemplate was easier for me.
On Process section follow these steps:
Select Default template
Add your project into Items To Build collection
Set MSBuild Arguments (Advanced section) to "/t:Deploy"
I have MSBuild project file for running builds locally. This script is used also for sever builds. I have three MSBuild projects in Items To Build collection. One for PreBuild step (some checks before build is executed), main build script used also for local build and the last script for additional post build tasks (deploy process). I'm setting additional MSBuild propertires like IncrementalBuild and ServerBuild properties in MSBuild Arguments.