How to restart Kestrel Hosted in Windows Service? - asp.net-core

I want to restart Kestrel (asp.net core 3.1) via an authorized http request.
Kestrel is contained in a Windows service configured to restart itself automatically upon failure.
As far as i know the simplest way is to return an exit code different from 0, windows will do all the rest.
In light of that, the code I wrote is actually simple:
public MaintenanceController(IHostApplicationLifetime hostLifetime)
{
this.HostLifetime = hostLifetime ?? throw new ArgumentNullException(nameof(hostLifetime));
}
[HttpPost]
[Route("service/restart")]
[Authorize("AdminOnly")]
public ActionResult RestartService()
{
Program.IsRestart = true; //see below
this.HostLifetime.StopApplication();
//this does not work!
if (HostLifetime is WindowsServiceLifetime lifetime)
{
lifetime.ExitCode = 1;
}
//neither this!
Environment.ExitCode = 1;
return Ok();
}
The only way to make windows restarts the service is just actually call
Environment.Exit(1);
without HostLifetime.StopApplication();
But the issue with Environment.Exit called alone is it causes a non graceful shutdown, something I want absolutely to avoid.
Another approach I tried, to force an exit code different from 0, which did not work, was to put in Startup.cs:
public void Configure(IApplicationBuilder app, IWebHostEnvironment env, IHostApplicationLifetime applicationLifetime)
{
[...]
applicationLifetime.ApplicationStopped.Register(() =>
{
//this can be set in the controller
if (Program.IsRestart)
{
Environment.Exit(1);
}
});
}
but, when called after ApplicationStopped, Environment.Exit(1) does apparently nothing, in fact even inside event viewer there is no trace of the services' shutdown with error, so windows does nothing.
UPDATE
Going back to classic, I changed the Main entry point to return an int and returned 1.
Still windows does not restart the service nor a failure entry is written to event viewer. It looks like is always stopped gracefully

Everything is working like it should. You may want to consider writing a wrapper batch file or watcher process.
Your watcher would wait for the process to exit, and if a flag file (e.g. .staydown) is present, it would exit gracefully. If the file doesn't exist, it would then restart the process.
In your RestartService method, use Environment.Exit(0) which would be a graceful shutdown, but since the .staydown file doesn't exist, your watcher would then restart the server. Your watcher would only stop running if the .staydown file exists and then the server is stopped, or the Windows Service itself is stopped.
When your app starts up, be sure to delete .staydown if present.
If you have node installed, you might be able to use a utility like forever in place of the watcher batch file.

You need to check the 'Enable actions for stops with errors' on the service, or run 'sc failureflag "My Service" 1'
More info here: https://blog.stephencleary.com/2020/06/servicebase-gotcha-recovery-actions.html
(this is a very strange behaviour on Windows part I think, honouring exit codes should be default!)

Related

Minimal Hosting Model: Exited from Program.Main with exit code = '0'

After migrating an ASP.NET Core 6 app from the legacy Program/Startup to use the new minimal hosting model, I am receiving a non-descript set of errors.
Error Messages
I have explicitly enabled UseDeveloperExceptionPage() to be safe, but all I am receiving in the browser is the generic:
HTTP Error 500.30 - ASP.NET Core app failed to start
In the Event Viewer, Event 1011 gets logged:
Application '/LM/W3SVC/2/ROOT' with physical root 'C:\Code' has exited from Program.Main with exit code = '0'. Please check the stderr logs for more information.
Followed by Event 1007:
Application '/LM/W3SVC/2/ROOT' with physical root 'C:\Code' failed to load coreclr.
Exception message:
CLR worker thread exited prematurely
No further information is available in the debug console when running either IIS Express or Kestrel. I am, however, able to see other startup processes from my Program being logged, so know they're loading correctly.
Code
This is a simplified version of my Program file, with custom components removed:
using Microsoft.AspNetCore.Mvc.Controllers;
using Microsoft.AspNetCore.Mvc.ViewComponents;
var builder = WebApplication.CreateBuilder(args);
builder.Services.AddControllersWithViews();
var app = builder.Build();
// Should be implicit in Debug mode, but explicitly including to be safe
app.UseDeveloperExceptionPage();
app.UseStaticFiles();
app.UseRouting();
app.MapControllers();
Previous Threads
As these are really general errors, there are quite a few previous threads available related to this issue—but none that seem applicable to this case. (Many have to do with the ASP.NET Core Hosting Bundle, which I am not using.)
Question
Are there common causes for this issue? Alternatively, are there other approaches for debugging this scenario?
The error messages aren't terribly intuitive, but this error essentially means that the application ran, and then immediately exited. A common cause of this is when you neglect to add the Run() command at the end of the Program:
app.Run();
Background
This is easy to miss if you're focused on migrating your ConfigureServices() and Configure() methods from your Startup class. While most of your previous configuration code likely lives in that class, the Run() method is one of a few pieces that needs to be migrated from your legacy Program class. You might recognize it from your legacy Main() method; e.g.,
public static void Main(string[] args) => CreateHostBuilder(args).Build().Run();
In the new ASP.NET Core 6 minimal hosting model, this gets translated to:
var builder = WebApplication.CreateBuilder(args);
// Configuration from Startup.ConfigureService()
var app = builder.Build();
// Configuration from Startup.Configure()
app.Run();
When it exits from Program.Main() with exit code 0, that's telling you that your Program ran correctly—and, thus, seeing logging for your configuration code—but then it stopped without ever actually launching your web application.
Simple fix once you know to look for it.

WCF start failure message box

I have a WCF project using C#, Visual Studio. When i want to start my project in debug mode i have this error: "Cannot start service from the command line or debugger. ...". I shouln't install my service using installutil program.
In my code i have a wrapper for starting program in debug mode to prevent this error:
#if DEBUG
MyService service = new MyService();
service.MyMethod();
#else
ServiceBase[] services;
services= new ServiceBase[]
{
new MyService()
};
ServiceBase.Run(services);
#endif
But error is still there. It appears before going inside to the method MyMethod() [F11]
In my opinion, please make sure that the account running the window service has the required permissions, preferably using localsystem, which could be configured in the ServcieProcessInstaller.
If the service needs to interact with the database, you can use a specific user to run the service so that the service can properly connect to the database.
Feel free to let me know if the problem still exists.

Topshelf Windows Service times out Error 7000 7009

I have a windows service programmed in vb.NET, using Topshelf as Service Host.
Once in a while the service doesn't start. On the event log, the SCM writes errors 7000 and 7009 (service did not respond in a timely fashion). I know this is a common issue, but I (think) I have tried everything with no result.
The service only relies in WMI, and has no time-consuming operations.
I read this question (Error 1053: the service did not respond to the start or control request in a timely fashion), but none of the answers worked for me.
I Tried:
Set topshelf's start timeout.
Request additional time in the first line of "OnStart" method.
Set a periodic timer wich request additional time to the SCM.
Remove TopShelf and make the service with the Visual Studio Service Template.
Move the initialization code and "OnStart" code to a new thread to return inmediately.
Build in RELEASE mode.
Set GeneratePublisherEvidence = false in the app.config file (per application).
Unchecked "Check for publisher’s certificate revocation" in the internet settings (per machine).
Deleted all Alternate Streams (in case some dll was marked as web and blocked).
Removed any "Debug code"
Increased Window's general service timeout to 120000ms.
Also:
The service doesn't try to communicate with the user's desktop in any way.
The UAC is disabled.
The Service Runs on LOCAL SYSTEM ACCOUNT.
I believe that the code of the service itself is not the problem because:
It has been on production for over two years.
Usually the service starts fine.
There is no exception logged in the Event Log.
The "On Error" options for the service dosn't get called (since the service doesn't actually fails, just doesn't respond to the SCM)
I've commented out almost everything on it, pursuing this error! ;-)
Any help is welcome since i'm completely out of ideas, and i've been strugling with this for over 15 days...
For me the 7009 error was produced by my NET core app because I was using this construct:
var builder = new ConfigurationBuilder()
.SetBasePath(Directory.GetCurrentDirectory())
.AddJsonFile("appsettings.json");
and appsettings.json file obviously couldn't be found in C:\WINDOWS\system32.. anyway, changing it to Path.Combine(AppContext.BaseDirectory, "appsettings.json") solved the issue.
More general help - for Topshelf you can add custom exception handling where I finally found some meaningfull error info, unlike event viewer:
HostFactory.Run(x => {
...
x.OnException(e =>
{
using (var fs = new StreamWriter(#"C:\log.txt"))
{
fs.WriteLine(e.ToString());
}
});
});
I've hit the 7000 and 7009 issue, which fails straight away (even though the error message says A timeout was reached (30000 milliseconds)) because of misconfiguration between TopShelf and what the service gets installed as.
The bottom line - what you pass in HostConfigurator.SetServiceName(name) needs to match exactly the SERVICE_NAME of the Windows service which gets installed.
If they don't match it'll fail straight away and you get the two event log messages.
I had this start happening to a service after Windows Creator's Edition update installed. Basically it made the whole computer slower, which is what I think triggered the problem. Even one of the Windows services had a timeout issue.
What I learned online is that the constructor for the service needs to be fast, but OnStart has more leeway with the SCM. My service had a C# wrapper and it included an InitializeComponent() that was called in the constructor. I moved that call to OnStart and the problem went away.

How to debug NServiceBus ServiceControl instance

I've installed the ServiceControl Management Utility and I'm trying to add an instance.
I would like to run the instance under a service account because we use SQLServer transport but pmthe installation page I get the error "Invalid password".
The account is hosting another windows service on the same machine.
I've tried other admin accounts and creating the instance through the UI and Powershell scripts.
I'm 200% sure the password is correct.
Is there anyway I can increase the logging to determine what is failing?
Strangely, I can change the service account under the initial install and it works.. I was eventually able to get the service running using an SQL login but I would prefered to use integrated security and not keep the username and password in the connection string.
A patch that addresses this bug has been released. See - https://github.com/Particular/ServiceControl/releases/tag/1.7.3. Thanks Kye for making us aware of the issue
This is code that does the validation.
public bool CheckPassword(string password)
{
if (Domain.Equals("NT AUTHORITY", StringComparison.OrdinalIgnoreCase))
{
return true;
}
var localAccount = Domain.Equals(Environment.MachineName, StringComparison.OrdinalIgnoreCase);
var context = localAccount ? new PrincipalContext(ContextType.Machine) : new PrincipalContext(ContextType.Domain);
return context.ValidateCredentials(QualifiedName, password);
}
So in a multi-domain environment it might run into trouble.
Raise a bug here and we will be able to give you a better response.

Task Persistence C#

I'm having a hard time trying to get my task to stay persistent and run indefinitely from a WCF service. I may be doing this the wrong way and am willing to take suggestions.
I have a task that starts to process any incoming requests that are dropped into a BlockingCollection. From what I understand, the GetConsumingEnumerable() method is supposed to allow me to persistently pull data as it arrives. It works with no problem by itself. I was able to process dozens of requests without a single error or flaw using a windows form to fill out the request and submit them. Once I was confident in this process I wired it up to my site via an asmx web service and used jQuery ajax calls to submit request.
The site submits request based on a url that is submitted, the Web Service downloads the html content from the url and looks for other urls within the content. It then proceeds to create a request for each url it finds and submits it to the BlockingCollection. Within the WCF service, if the application is Online (i.e. Task has started) - it pulls the request using the GetConsumingEnumerable via a Parallel.ForEach and Processes the request.
This works for the first few submissions, but then the task just stops unexpectedly. Of course, this is doing 10x more request than I could simulate in testing - but I expected it to just throttle. I believe the issue is in my method that starts the task:
public void Start()
{
Online = true;
Task.Factory.StartNew(() =>
{
tokenSource = new CancellationTokenSource();
CancellationToken token = tokenSource.Token;
ParallelOptions options = new ParallelOptions();
options.MaxDegreeOfParallelism = 20;
options.CancellationToken = token;
try
{
Parallel.ForEach(FixedWidthQueue.GetConsumingEnumerable(token), options, (request) =>
{
Process(request);
options.CancellationToken.ThrowIfCancellationRequested();
});
}
catch (OperationCanceledException e)
{
Console.WriteLine(e.Message);
return;
}
}, TaskCreationOptions.LongRunning);
}
I've thought about moving this into a WF4 Service and just wire it up in a Workflow and use Workflow Persistence, but am not willing to learn WF4 unless necessary. Please let me know if more information is needed.
The code you have shown is correct by itself.
However there are a few things that can go wrong:
If an exception occurs, your task stops (of course). Try adding a try-catch and log the exception.
If you start worker threads in a hosted environment (ASP.NET, WCF, SQL Server) the host can decide arbitrarily (without reason) to shut down any worker process. For example, if your ASP.NET site is inactive for some time the app is shut down. The hosts that I just mentioned are not made to have custom threads running. Probably, you will have more success using a dedicated application (.exe) or even a Windows Service.
It turns out the cause of this issue was with the WCF Binding Configuration. The task suddenly stopped becasue the WCF killed the connection due to a open timeout. The open timeout setting is the time that a request will wait for the service to open a connection before timing out. In certain situations, it reached the limit of 10 max connection and caused the incomming connections to get backed up waiting for a connection. I made sure that I closed all connections to the host after the transactions were complete - so I gave in to upping the max connections and the open timeout period. After this - it ran flawlessly.