Address already in use when restarting my web application - asp.net-core

Question

What I have:
ASP.NET Core 2.1 Web Application (self-hosted deployment, not using IIS).
When I pass --register-service or --unregister-service as command line arguments to my app, it (un)registers itself as a Windows Service and then exits.
MSI installer (WiX Toolset 3.11) which registers the web application as a Windows Service by using the above command line argument in a Custom Action.
When doing an upgrade from version 1 to version 2 of my application, the MSI installer successfully performs the following steps:
Unregister version 1 as Windows Service
Remove binaries of version 1
Copy binaries of version 2
Register and (attempt to) start version 2 as Windows Service
Problem:
The newly installed version 2 does not start properly because apparently the port is still blocked:
System.IO.IOException: Failed to bind to address https://[::]:5001: address already in use. ---> Microsoft.AspNetCore.Connections.AddressInUseException: Only one usage of each socket address (protocol/network address/port) is normally permitted ---> System.Net.Sockets.SocketException: Only one usage of each socket address (protocol/network address/port) is normally permitted
Interestingly when any of these two versions is installed, I can stop and start the Windows Service without any issues. The problem with the blocked port seems to occur specifically in the context of an MSI upgrade.
What I tried:
Once the MSI installer is finished, I can just restart the newly installed service manually and it runs fine. So it looks like it just takes some more time for the port to become available. So what I tried is some kind of a re-try mechanism, along these lines (simplified):
IWebHost host = null;
Start:
host?.Dispose();
host = CreateWebHost();
try
{
await host.RunAsync(ct);
}
catch (IOException ex) when (ex.InnerException is AddressInUseException)
{
await Task.Delay(TimeSpan.FromSeconds(5), ct);
goto Start;
}
However, then I get the following exception in some service where I try to use an injected IServiceScopeFactory when doing the first re-try:
System.ObjectDisposedException: Cannot access a disposed object.
Object name: 'IServiceProvider'.
So it looks like even I'm creating a completely new IWebHost for retrying, the DI container still seems to be in a bad state from the first try.
Questions:
What could be the reason for this blocked port, apparently only in the context of an MSI update? Could it be that the Windows Installer itself is not releasing the port? How to solve or workaround this?
With regards to the workaround I tried: Why do I get that ObjectDisposedException? Is there a better way to re-try while the port is still blocked?