MSBuild: asterisks and strange ItemGroup Exclude behaviour - msbuild

I have a script that attempts to construct an ItemGroup out of all files in a certain directory while excluding files with certain names (regardless of extension).
The list of files to be excluded initially contains file extensions, and I am using Community Tasks' RegexReplace to replace the extensions with an asterisk. I then use this list in the item's Exclude attribute. For some reason the files do not get excluded properly, even though the list appears to be correct.
To try and find the cause I created a test script (below) which has two tasks: first one initialises two properties with the list of file patterns in two different ways. The second task prints both properties and the files resulting from using both these properties in the Exclude attribute.
The properties' values appear to be identical, however the resulting groups are different. How is this possible?
<Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003"
DefaultTargets="Init;Test" ToolsVersion="3.5">
<Import Project="$(MSBuildExtensionsPath)\MSBuildCommunityTasks\MSBuild.Community.Tasks.Targets"/>
<Target Name="Init">
<ItemGroup>
<OriginalFilenames Include="TestDir\SampleProj.exe"/>
<OriginalFilenames Include="TestDir\SampleLib1.dll"/>
</ItemGroup>
<RegexReplace Input="#(OriginalFilenames)" Expression="\.\w+$" Replacement=".*">
<Output TaskParameter="Output" ItemName="PatternedFilenames"/>
</RegexReplace>
<PropertyGroup>
<ExcludeFilesA>TestDir\SampleProj.*;TestDir\SampleLib1.*</ExcludeFilesA>
<ExcludeFilesB>#(PatternedFilenames)</ExcludeFilesB>
</PropertyGroup>
</Target>
<Target Name="Test">
<Message Text='ExcludeFilesA: $(ExcludeFilesA)' />
<Message Text='ExcludeFilesB: $(ExcludeFilesB)' />
<ItemGroup>
<AllFiles Include="TestDir\**"/>
<RemainingFilesA Include="TestDir\**" Exclude="$(ExcludeFilesA)"/>
<RemainingFilesB Include="TestDir\**" Exclude="$(ExcludeFilesB)"/>
</ItemGroup>
<Message Text="
**AllFiles**
#(AllFiles, '
')" />
<Message Text="
**PatternedFilenames**
#(PatternedFilenames, '
')" />
<Message Text="
**RemainingFilesA**
#(RemainingFilesA, '
')" />
<Message Text="
**RemainingFilesB**
#(RemainingFilesB, '
')" />
</Target>
</Project>
Output (reformatted somewhat for clarity):
ExcludeFilesA: TestDir\SampleProj.*;TestDir\SampleLib1.*
ExcludeFilesB: TestDir\SampleProj.*;TestDir\SampleLib1.*
AllFiles:
TestDir\SampleLib1.dll
TestDir\SampleLib1.pdb
TestDir\SampleLib2.dll
TestDir\SampleLib2.pdb
TestDir\SampleProj.exe
TestDir\SampleProj.pdb
PatternedFilenames:
TestDir\SampleProj.*
TestDir\SampleLib1.*
RemainingFilesA:
TestDir\SampleLib2.dll
TestDir\SampleLib2.pdb
RemainingFilesB:
TestDir\SampleLib1.dll
TestDir\SampleLib1.pdb
TestDir\SampleLib2.dll
TestDir\SampleLib2.pdb
TestDir\SampleProj.exe
TestDir\SampleProj.pdb
Observe that both ExcludeFilesA and ExcludeFilesB look identical, but the resulting groups RemainingFilesA and RemainingFilesB differ.
Ultimately I want to obtain the list RemainingFilesA using the pattern generated the same way ExcludeFilesB is generated. Can you suggest a way, or do I have to completely rethink my approach?

The true cause of this was revealed accidentally when a custom task threw an exception.
The actual value of ExcludeFilesA is TestDir\SampleProj.*;TestDir\SampleLib1.* like one might expect. However the actual value of ExcludeFilesB is TestDir\SampleProj.%2a;TestDir\SampleLib1.%2a.
Presumably Message unescapes the string before using it, but Include and Exclude do not. That would explain why the strings look the same but behave differently.
Incidentally, the execution order doesn't seem to have anything to do with this, and I'm pretty sure (following extensive experimentation) that everything gets executed and evaluated exactly in the order in which it appears in this script.

ItemGroups need to be evaluated before targets execution, and the PatternedFilenames ItemGroup is being created on the fly within its target container.
You could workaround this using the CreateItem task, which will ensure the PatternedFilenames scope throughout the execution:
<RegexReplace Input="#(OriginalFilenames)" Expression="\.\w+$" Replacement=".*">
<Output TaskParameter="Output" ItemName="PatternedFilenames_tmp"/>
</RegexReplace>
<CreateItem Include="#(PatternedFilenames_tmp)">
<Output TaskParameter="Include" ItemName="PatternedFilenames"/>
</CreateItem>

Related

Use uglifyjs in Publish Profile

This is a snippet of my Publish Profile:
<Exec WorkingDirectory="$(_PackageTempDir)"
Command="uglifyjs ..\..\..\..\js\file1.js ..\..\..\..\js\file2.js --mangle --reserved "$" --compress > js\outfile.min.js" />
Certain files (say file1.js) is located outside my project and therefore is not copied to the _PackageTempDir. Here I have to ..\ up several levels to get there. I'm wondering if there is a good way to use an ItemGroup or full path that will allow me the same results.
The above code "works". It is just complicated and difficult to maintain. Looking for a better solution.
EDIT:
Based on Sayed's suggestions, I refined my profile to:
<ItemGroup>
<UglifyJSFiles Include="$(MSBuildThisFileDirectory)..\..\js\mcm\mcm.js" />
<UglifyJSFiles Include="$(_PackageTempDir)\js\main.js" />
</ItemGroup>
<Exec WorkingDirectory="$(_PackageTempDir)"
Command="uglifyjs #(UglifyJSFiles,' ') > js\app.min.js" />
But I am running into an issue because the paths contain spaces. How can I either quote the path strings or escape the spaces?
Here is an example showing a better approach
<PropertyGroup>
<JsFilesToUglifyRoot Condition=" '$(JsFilesToUglifyRoot)'=='' ">$(MSBuildThisFileDirectory)\..\..\..\..\js\</JsFilesToUglifyRoot>
</PropertyGroup>
<ItemGroup>
<JsFilesToUglify Include="$(JsFilesToUglifyRoot)**\*.js" />
</ItemGroup>
<Target Name="AfterBuild">
<Message Text="Files: [#(JsFilesToUglify,' ')]" Importance="high" />
<!-- If you need to quote the file paths use the transform below -->
<Message Text="Files: [#(JsFilesToUglify->'"%(FullPath)"',' ')]" Importance="high" />
</Target>
Here I define a new property JsFilesToUglify that is populated with the path you indicated above. Note the usage of the MSBuildThisFileDirectory reserved property. You should not rely on just ..\ as its value may be different in VS versus outside of VS. Also do not use the MSBuildProjectDirectory property, only MSBuildThisFileDirectory.
Then inside of the target I transform the list of files with #(JsFilesToUglify,' ') the ,' ' makes a space the separator between values like your command above.

Using the Zip task in MSBuild

I have been attempting to use the zip task of msbuild in a project I am working on at the moment.
My project file looks something like this:
<PropertyGroup> <MSBuildCommunityTasksPath>$(SolutionDir)\.build</MSBuildCommunityTasksPath> </PropertyGroup>
<Import Project="$(MSBuildCommunityTasksPath)\MSBuild.Community.Tasks.Targets" />
<ItemGroup>
<FileToZip include="C:\FilePath"></FilesToZip>
<FileToZip include="C:\FilePath"></FilesToZip>
</ItemGroup>
<Target Name="BeforeBuild">
<PropertyGroup>
<ReleasePath>\releasepath</ReleasePath>
<Zip Files="#(FilesToZip)" WorkingDirectory="$(ReleasePath)" ZipFileName="HTMLeditor.html" ZipLevel="9" />
</Target>
However, the zip file updates but does not contain the files specified in the item group FilesToZip. I cannot figure out why they aren't being recognised! I have double checked file paths and they are correct. Any ideas?
I think you want to do something like this:
<ItemGroup>
<FileToZip include="C:\FilePath;C:\FilePath"/>
</ItemGroup>
As I mentioned in my comment, simply creating a variable (FileToZip) and repeating it twice with different values does not give you an array that contains both of the values. You end up with only the last value (and not an array at all). Your include attribute is a selector which is used to build the array and it can contain multiple values, wildcards and other patterns which are used to build out that array for you.
Here's a link to MSDN that gives you more information on how to use the Include attribute: http://msdn.microsoft.com/en-us/library/ms171454.aspx
I ditched the ItemGroup in the end, and went with another way of doing it.
<Target Name="Zip">
<CreateItem Include="FilesToInclude" >
<Output ItemName="ZipFiles" TaskParameter="Include"/>
<Zip ZipFileName="ZipFile.zip" WorkingDirectory="FolderToWriteZipTo" Files="#(ZipFiles)" />
</Target>
This method seemed to be easier and wasn't adding files to the root of the file.
Thanks for the help though guys.

Pass list item to Properties when calling reusable msbuild target

I'm trying to create a reusable Target in msbuild, following the basic model outlined in How to invoke the same msbuild target twice?
I'm stuck trying to pass a property that I want interpreted as a list. I haven't found an example online that deals with this situation. As I understand it, the problem is that Properties is already treated as a list item, so it doesn't like having a list item passed in as well. Is there a way to get msbuild to pack and unpack the list correctly here?
Msbuild is complaining with:
error MSB4012: The expression "FilesToZip=#(Scripts)" cannot be used in this context. Item lists cannot be concatenated with other strings where an item list is expected. Use a semicolon to separate multiple item lists.
Here's an example caller:
<Target Name="BuildMigrationZip">
<MSBuild Projects="BuildZip.msbuild"
Targets="BuildZip"
Properties="FilesToZip=#(Scripts);OutputZipFile=$(MigrationPackageFilePath);OutputFolder=$(MigrationPackagePath);Flatten=true"/>
<Message Text="Created database migration zip: $(MigrationPackageFilePath)" Importance="high"/>
</Target>
And the base target:
<Target Name="BuildZip">
<MakeDir Directories="$(OutputFolder)"/>
<Zip Files="#(FilesToZip)"
ZipFileName="$(OutputZipFile)"
Flatten="$(Flatten)"
ParallelCompression="false" />
</Target>
I'm basically at the point of just going back to cut and paste for these, although I want to package up a number of zips here.
UPDATE: The same issue applies to setting Inputs on the reusable target. My question up to this point addresses the raw functionality, but it would be nice to keep dependencies working. So for example:
<Target Name="BuildZip"
Inputs="#(FilesToZip)"
Outputs="$(OutputZipFile)">
<MakeDir Directories="$(OutputFolder)"/>
<Zip Files="#(FilesToZip)"
ZipFileName="$(OutputZipFile)"
Flatten="$(Flatten)"
ParallelCompression="false" />
</Target>
They key is to pass the list around as a property. So when your Scripts list is defined as
<ItemGroup>
<Scripts Include="A"/>
<Scripts Include="B"/>
<Scripts Include="C"/>
</ItemGroup>
then you first convert it into a property (which just makes semicolon seperated items, but msbuild knows how to pass this via the Properties of the MSBuild target) then pass it to the target:
<Target Name="BuildMigrationZip">
<PropertyGroup>
<ScriptsProperty>#(Scripts)</ScriptsProperty>
</PropertyGroup>
<MSBuild Projects="$(MSBuildThisFile)" Targets="BuildZip"
Properties="FilesToZip=$(ScriptsProperty)" />
</Target>
(note I'm using $(MSBuildThisFile) here: you don't necessarily need to create seperate build files for every single target, in fact for small targets like yours it's much more convenient to put it in the same file)
Then in your destination target you turn the property into a list again:
<Target Name="BuildZip">
<ItemGroup>
<FilesToZipList Include="$(FilesToZip)"/>
</ItemGroup>
<Message Text="BuildZip: #(FilesToZipList)" />
</Target>
Output:
BuildZip: A;B;C
Update
When working with Inputs, you cannot pass #(FilesToZip) since that expands to nothing because is not a list: it's a property - which happens to be a number of semicolon-seperated strings. And as such, it is usable for Inputs you just have to expand it as the property it is i.e. $(FilesToZip):
<Target Name="BuildZip"
Inputs="$(FilesToZip)"
Outputs="$(OutputZipFile)">
...
</Target>
Output of second run:
BuildZip:
Skipping target "BuildZip" because all output files are up-to-date with respect to the input files.

Can't get MSBuild Community Task RegexReplace to work

I'm trying to copy a bunch of files whose names begin with the prefix DR__, but the copies must have that prefix removed. That is, DR__foo must be copied as foo. I'm trying this, which is based in the example provided in the documentation (the .chm):
<Target Name="CopyAuxiliaryFiles">
<MakeDir Directories="$(TargetDir)Parameters" Condition="!Exists('$(TargetDir)Parameters')" />
<ItemGroup>
<ContextVisionParameterFiles Include="$(SolutionDir)CVParameters\DR__*" />
</ItemGroup>
<Message Text="Files to copy and rename: #(ContextVisionParameterFiles)"/>
<RegexReplace Input="#(ContextVisionParametersFiles)" Expression="DR__" Replacement="">
<Output ItemName ="DestinationFullPath" TaskParameter="Output" />
</RegexReplace>
<Message Text="Renamed Files: #(DestinationFullPath)"/>
<Copy SourceFiles="#(ContextVisionParameterFiles)" DestinationFiles="#(DestinationFullPath)" />
</Target>
DestinationFullPath comes out empty (or that's what I see when I display it with Message). Thus, Copy fails because no DestinationFiles are specified. What's wrong here?
Edit: ContextVisionParameterFiles is not empty, it contains this:
D:\SVN.DRA.WorkingCopy\CVParameters\DR__big_bone.alut;D:\SVN.DRA.WorkingCopy\CVParameters\DR__big_medium.gop
They're actually 40 files, but I trimmed it for the sake of clarity
Got it! It seems to have been the combination of a stupid error and a seemingly compulsory parameter. As for the first one, there were two Targets called CopyAuxiliaryFiles. As for the second one, it seems the Count parameter is needed.
The final, working version:
<Target Name="CopyCvParameters">
<ItemGroup>
<CvParamFiles Include="$(SolutionDir)CVParameters\DR__*" />
</ItemGroup>
<Message Text="Input:
#(CvParamFiles, '
')"/>
<!-- Replaces first occurance of "foo." with empty string-->
<RegexReplace Input="#(CvParamFiles)" Expression="^.*DR__" Replacement="$(TargetDir)Parameters\" Count="1">
<Output ItemName ="RenamedCvParamFiles" TaskParameter="Output" />
</RegexReplace>
<Message Text="
Output RenamedCvParamFiles:
#(RenamedCvParamFiles, '
')" />
<Copy SourceFiles="#(CvParamFiles)" DestinationFiles="#(RenamedCvParamFiles)" SkipUnchangedFiles="True" />
</Target>
Notice that:
I renamed the Target to solve the name collision (Why doesn't Visual Studio detect this as an error?)
I pretty-printed the ItemGroups with the #(CvParamFiles, '
') syntax, which seems to replace ; with line breaks
My regex replaces the absolute path and the prefix
Count="1" is now passed to RegexReplace

Including files with directory specified separately in MSBuild

This seems like it should be simple but I can't work it out from the reference and my google-fu is apparently weak.
I just want to specify the file names and base folder separately in the build file...
<?xml version="1.0" encoding="utf-8"?>
<Project DefaultTargets="Build" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
<PropertyGroup>
<TestFilesWithFolder>
B:\Root\Test1.*;
B:\Root\Test2.*
</TestFilesWithFolder>
<TestFiles>Test1.*;Test2.*</TestFiles>
<TestFileRoot>B:\Root</TestFileRoot>
</PropertyGroup>
<Target Name="Build">
<ItemGroup>
<TestFilesGroupWithFolder Include="$(TestFilesWithFolder)" />
<TestFilesGroup Include="$(TestFileRoot)\$(TestFiles)" />
</ItemGroup>
<Warning Text="Source files with folder: #(TestFilesGroupWithFolder)" />
<Warning Text="Source files: #(TestFilesGroup)" />
</Target>
</Project>
When I run this, the first warning shows both files as expected, but the second warning only shows the first file (since the straight string concat put the folder name on the first but not second).
How would I get the ItemGroup "TestFilesGroup" to include both the files given the "TestFiles" and "TestFileRoot" properties?
It is possible to convert a semicolon delimited list of things into an item, which would make this possible, except that the items in your property contain wildcards, so if you want to have MSBuild treat them as items in a list, at the moment MSBuild first sees it the path must be valid. There may be a way to do that but I can't think of one. In other words...
<ItemGroup>
<TestFiles Include="$(TestFiles)" />
</ItemGroup>
...only works if $(TestFiles) contains a delimited list of either things with no wildcards, or qualified paths that actually exist.
Further, MSBuild can't compose a path with a wildcard inside the Include attribute and evaluate it at the same time, so you need a trick to first compose the full path separately, then feed it into the Include attribute. The following will work, but it requires changing your delimited property into a set of items. It batches a dependent target on this item list, with each batched target execution calculating a meta value for one item, which is stored off in a new meta value. When the original target executes, it is able to use that meta value in a subsequent Include.
<PropertyGroup>
<TestFilesWithFolder>
D:\Code\Test1.*;
D:\Code\Test2.*
</TestFilesWithFolder>
<TestFileRoot>D:\Code</TestFileRoot>
</PropertyGroup>
<ItemGroup>
<TestFilePattern Include="TestFilePattern">
<Pattern>Test1.*</Pattern>
</TestFilePattern>
<TestFilePattern Include="TestFilePattern">
<Pattern>Test2.*</Pattern>
</TestFilePattern>
</ItemGroup>
<Target Name="Compose" Outputs="%(TestFilePattern.Pattern)">
<ItemGroup>
<TestFilePattern Include="TestFilePattern">
<ComposedPath>#(TestFilePattern->'$(TestFileRoot)\%(Pattern)')</ComposedPath>
</TestFilePattern>
</ItemGroup>
</Target>
<Target Name="Build" DependsOnTargets="Compose">
<ItemGroup>
<TestFilesGroupWithFolder Include="$(TestFilesWithFolder)" />
</ItemGroup>
<Warning Text="Source files with folder: #(TestFilesGroupWithFolder)" />
<ItemGroup>
<ComposedTestFiles Include="%(TestFilePattern.ComposedPath)" />
</ItemGroup>
<Warning Text="Source files: #(ComposedTestFiles)" />
</Target>
Which produces the following output:
(Build target) ->
d:\Code\My.proj(80,5): warning : Source files with folder:
D:\Code\Test1.txt;D:\Code\Test2.txt
d:\Code\My.proj(84,5): warning : Source files:
D:\Code\Test1.txt;D:\Code\Test2.txt