Asp.Net Core 3 graceful shutdown throws OperationCanceledException - asp.net-core

Question

I'm trying to gracefully terminate a ASP.Net Core 3.1 service (which will run in Kubernetes). When Kubernetes stops a service, it will send a SIGTERM event to the application, at which point I want in-flight requests to complete (which may take several seconds) before terminating... I think I can catch this in a hostedservice, as below, and hence not stop immediately.
The following works, but with a timeout of 5 seconds or longer, I receive an OperationCanceledException. Could anyone shed any light on why I get an OperationCanceledException or how shed any light on an alternative way to delay a SIGTERM event, to allow a graceful shutdown?
public static int Main(string[] args)
{
var logger = NLogBuilder
.ConfigureNLog("nlog.config")
.GetCurrentClassLogger();
try
{
CreateHostBuilder(args)
.ConfigureServices((hostBuilderContext, services) => { services.AddHostedService<LifetimeEventsHostedService>(); })
.Build()
.Run();
return 0;
}
catch (Exception e)
{
logger.Fatal(e, "Stopping due to exception");
return -1;
}
finally
{
LogManager.Shutdown();
}
}
This is the hosted service...
internal class LifetimeEventsHostedService : IHostedService
{
private readonly Microsoft.Extensions.Logging.ILogger _logger;
private readonly IHostApplicationLifetime _appLifetime;
public LifetimeEventsHostedService(
ILogger<LifetimeEventsHostedService> logger,
IHostApplicationLifetime appLifetime)
{
_logger = logger;
_appLifetime = appLifetime;
}
public Task StartAsync(CancellationToken cancellationToken)
{
_appLifetime.ApplicationStarted.Register(OnStarted);
_appLifetime.ApplicationStopping.Register(OnStopping);
_appLifetime.ApplicationStopped.Register(OnStopped);
return Task.CompletedTask;
}
public Task StopAsync(CancellationToken cancellationToken)
{
return Task.CompletedTask;
}
private void OnStarted()
{
_logger.LogInformation("OnStarted has been called.");
// Perform post-startup activities here
}
private void OnStopping()
{
_logger.LogInformation("OnStopping has been called.");
// Perform on-stopping activities here
// This works, but a timeout of 5 seconds or more subsequently causes an OperationCanceledException
Thread.Sleep(5000);
}
private void OnStopped()
{
_logger.LogInformation("OnStopped has been called.");
// Perform post-stopped activities here
}
}

Answer 1

I'm open to alternative approaches to graceful shutdown with ASP.Net Core 3.1, as it stands, I'm using a hosted service.
Within the .Net Core app, I was setting ShutdownTimeout on the webhost, however, setting the ShutdownTimeout on the generic host, does allow me to gracefully wait a number of seconds (more than the default sigterm, which is 5 seconds) prior to shutdown. The hint from #PmanAce helped me work that out.
As such, the following codes allow me to gracefully terminate. One caveat, the Thread.Sleep in LifetimeEventsHostedService must be less than option.ShutdownTimeout.
public class Program
{
public static void Main(string[] args)
{
CreateHostBuilder(args).Build().Run();
}
public static IHostBuilder CreateHostBuilder(string[] args) =>
Host.CreateDefaultBuilder(args)
.ConfigureServices((hostBuilderContext, services) =>
{
services.AddHostedService<LifetimeEventsHostedService>();
services.Configure<HostOptions>(option =>
{
option.ShutdownTimeout = TimeSpan.FromSeconds(30);
});
})
.ConfigureWebHostDefaults(webBuilder =>
{
webBuilder.UseKestrel();
webBuilder.UseStartup<Startup>();
});
}
The following LifetimeEventsHostedService
public class LifetimeEventsHostedService : IHostedService
{
private readonly IHostApplicationLifetime _hostApplicationLifetime;
public LifetimeEventsHostedService(IHostApplicationLifetime hostApplicationLifetime)
{
_hostApplicationLifetime = hostApplicationLifetime;
}
public Task StartAsync(CancellationToken cancellationToken)
{
_hostApplicationLifetime.ApplicationStarted.Register(OnStarted);
_hostApplicationLifetime.ApplicationStopping.Register(OnStopping);
_hostApplicationLifetime.ApplicationStopped.Register(OnStopped);
return Task.CompletedTask;
}
public Task StopAsync(CancellationToken cancellationToken)
{
return Task.CompletedTask;
}
private void OnStopped()
{
Console.WriteLine("OnStopped");
}
private void OnStopping()
{
Console.WriteLine("OnStopping");
Console.WriteLine(DateTime.Now.ToLongTimeString());
Thread.Sleep(15000);
Console.WriteLine("Sleep finished");
Console.WriteLine(DateTime.Now.ToLongTimeString());
}
private void OnStarted()
{
Console.WriteLine("OnStarted");
}
}