Dynamically include precompiled (Razor) views in .NET Core 2.1 - asp.net-core

Question

Idea
I'm trying to build a modular .NET Core MVC application, were I can add 'modules' (extensions) just by copying the required libraries to a predefined folder. The application defines a set of dependencies (DI), error handling and basic layout; the modules provide the actual application or page logic.
We've build a proof of concept, using ExtCore - the concept works great, Controllers are added automatically and the 'plain' Views are recognized easily (the ones set as 'Embedded Resource').
Challenge
However, I don't like the fact that Views are set to 'Embedded Resource', as it will be a small performance hit as the Razor page is now build at runtime.
What we've tried
As .NET Core 2.1 now has the ability to create Reusable Razor UI, I've created a project with a reference to <Project Sdk="Microsoft.NET.Sdk.Razor"> and compiled it to a Razor library (*.Views.dll).
When I link the Razor View library as project dependency in (main) application, everything works fine. However, the goal still is to make the application fully plug-and-play and allow libraries to be included just by deploying them to a folder.
I've tried to achieve this extending the Razor View Engine using a PhysicalFileProvider, but it won't recognize the views in the library.
services.Configure<RazorViewEngineOptions>(opt =>
{
opt.FileProviders.Add(new PhysicalFileProvider("path/to/extension"));
});
The result always is an exception telling me that /Views/[controller]/[action].cshtml is not found.
Searching other answers I came across this project, which creates a ViewLocationExpander. I tried implementing various versions of this idea without luck - most likely because this is an older concept based on .NET Core 1.x which still uses a project dependency; although the Razor View engine now searches the namespace of the assembly.
services.Configure<RazorViewEngineOptions>(opt =>
{
opt.FileProviders.Add(new PhysicalFileProvider("path/to/extension");
opt.ViewLocationExpanders.Add(new DynamicLocationExpander());
});
Where DynamicLocationExpander looks like:
public class DynamicLocationExpander: IViewLocationExpander
{
public DynamicLocationExpander()
{
}
public void PopulateValues(ViewLocationExpanderContext context)
{
}
public IEnumerable<string> ExpandViewLocations(ViewLocationExpanderContext context, IEnumerable<string> viewLocations)
{
string assembly = ((ControllerActionDescriptor)context.ActionContext.ActionDescriptor)
.ControllerTypeInfo.AsType().Assembly.GetName().Name;
foreach (var viewLocation in viewLocations)
yield return $"{assembly}/{viewLocation}";
}
}
Question
How can we dynamically link a Razor View library, containing precompiled views, to the Razor Engine in a (parent) application?