msbuild refuses to copy unsigned dll if it is of lower version than demanded by one of the project dependencies - is it by design? - msbuild

Question

I have the following situation:
All the involved dlls are unsigned
All the projects in the solution depend on version 1.0.21221.1 of Shared.dll
Some NuGet dependencies of some projects in the solution depend on version 1.0.21237.1 of the same dll.
When the web application is built (let us name it Api) it is expected to copy Shared.dll from $(OutDir) to the $(OutDir)_PublishedWebsites\Api\bin folder. The Shared.dll found in $(OutDir) has the version 1.0.21221.1.
The Shared.dll is NOT copied and the web application fails to run.
Here is the evidence from the binary log:
Exhibit A - The conflict of versions:
Exhibit B - ResolveAssemblyReference instructs NOT to copy Shared.dll:
I understand that msbuild does not like the idea of conflicting versions, but NOT copying the dll produces a downright bug, because the application fails to start.
I understand one can resolve it by adding an assembly binding redirect. But I thought it was unnecessary for unsigned assemblies. Am I understanding wrong or am I missing something?
EDIT 1
Here are my answers to the questions posted in the comments:
(Unfortunately I was asked to obfuscate some keywords, I do not know why)
How exactly does the Api project reference Shared.dll?
As we can see in the exhibit B Shared.dll is a transitive dependency of Api. Indeed, Api depends on Xyz.BusinessApi like this:
<Reference Include="Xyz.BusinessAPI" />
Now that DLL depends on Shared.dll through the respective NuGet dependency, here is a snippet from the project.assets.json file of Xyz.BusinessAPI:
What other projects reference Shared.dll and how?
There are a lot of projects referencing it as a NuGet package at version 1.0.21221.1. The problem is that some projects also reference two other NuGet packages which in turn depend on the version 1.0.21237.1 of the same NuGet package. This is indicated in the RAR output - see the exhibit A.
I would like to emphasize - no project references Shared.dll as a raw dll, only either as NuGet package or indirectly through other NuGets or projects or project dlls. Project dll is a dll of a project from a previously built solution - we do not allow project references to other solutions, so if a project is built in a previous solution, then it would be referenced as DLL in subsequent solutions.
What is the mechanism used to copy from OutDir to _PublishedWebsites\api\bin?
This is the standard web application publishing target _CopyFilesMarkedCopyLocal from C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\MSBuild\Microsoft\VisualStudio\v16.0\WebApplications\Microsoft.WebApplication.targets:
<!-- copy any referenced assemblies to _PublishedWebsites\app\bin folder -->
<Copy SourceFiles="#(ReferenceCopyLocalPaths)"
DestinationFiles="#(ReferenceCopyLocalPaths->'$(WebProjectOutputDir)\bin\%(DestinationSubDirectory)%(Filename)%(Extension)')"
SkipUnchangedFiles="true"
Retries="$(CopyRetryCount)"
RetryDelayMilliseconds="$(CopyRetryDelayMilliseconds)"/>
How does Shared.dll end up in OutDir?
All of our code is built into the shared bin directory - we set OutDir to the same value for all the projects. Thus all the project binaries and their dependencies, including Shared.dll first end up there.
Are there any double-writes in the binlog?
Yes, but I do not think they are relevant:

Answer 1

The short answer is that your projects indirectly depend on two versions of Shared.dll: 1.0.21221.1 and 1.0.21237.1.
RAR (the MSBuild ResolveAssemblyReference task) inspected all the references of all .dlls and found these two versions. It reported a conflict:
Pay attention how it reported the found file path in square brackets for Shared.dll 21221 and reported [] (meaning no file of such version was found) for 21237.
It's useful to search for these using There was a conflict under($rar) or $warning under($rar).
Now, the OutDir only contains the Shared.dll 21221, so the Shared.dll with version 21237 couldn't be found anywhere.
The trick is to search for Shared.dll under($rar project(api.csproj)) You will find the relevant messages from RAR:
Considered "C:\Xyz\61\bin.link\Shared.dll",
but its name "Shared, Version=1.0.21221.1, Culture=neutral, PublicKeyToken=null"
didn't match the expected name "Shared, Version=1.0.21237.1, Culture=neutral, PublicKeyToken=null".
So, it saw a conflict, decided to unify on a later version (21237), but didn't find the file of that version. So the reference of 21221 was not CopyLocal because it didn't unify on that version, and the reference to 21237 was not resolved because a file of that version was not found.
To resolve this, I recommend adding an explicit reference to Shared.dll of the version 21237 (either via NuGet or via GeneratePathProperty metadata on the package reference: https://learn.microsoft.com/en-us/nuget/consume-packages/package-references-in-project-files#generatepathproperty). If you use GeneratePathProperty you can then reference the .dll directly using $(PkgFoo_Bar)\lib\net472\Shared.dll or similar. Also add a binding redirect from 21221 to 21237 to resolve the conflict. Once the correct version (21237) will be in your OutDir, it will get copied to output correctly.
Hope this clarifies.