Asp.net Core API with CORS on Service Fabric 100% CPU bottleneck - asp.net-core

Question

I have an ASP.net Core on .Net Framework 4.5.2 hosted on Service Fabric as STATELESS Service.
The API is a vanilla API, empty
[Route("Test")]
public class TestController : Controller
{
[HttpGet]
public IActionResult Get()
{
return Ok("Done");
}
}
This is my Startup Code
public class Startup
{
public Startup(IHostingEnvironment env)
{
var builder = new ConfigurationBuilder()
.SetBasePath(env.ContentRootPath)
.AddJsonFile("appsettings.json", optional: true, reloadOnChange: true)
.AddJsonFile($"appsettings.{env.EnvironmentName}.json", true)
.AddEnvironmentVariables();
Configuration = builder.Build();
}
public IConfigurationRoot Configuration { get; }
public void ConfigureServices(IServiceCollection services)
{
services.AddCors(options =>
{
options.AddPolicy("CorsPolicy",
builder => builder.AllowAnyOrigin()
.AllowAnyMethod()
.AllowAnyHeader()
.AllowCredentials());
});
services.AddResponseCompression();
services.AddMvc().AddJsonOptions(opts =>
{
// Force Camel Case to JSON
opts.SerializerSettings.ContractResolver = new CamelCasePropertyNamesContractResolver();
});
}
// This method gets called by the runtime. Use this method to configure the HTTP request pipeline.
public void Configure(IApplicationBuilder app, IHostingEnvironment env, ILoggerFactory loggerFactory)
{
app.UseCors("CorsPolicy");
app.UseResponseCompression();
app.UseMvc();
}
}
This is the OpenAsync method:
Task<string> ICommunicationListener.OpenAsync(CancellationToken cancellationToken)
{
var endpoint = FabricRuntime.GetActivationContext().GetEndpoint(_endpointName);
string serverUrl = $"{endpoint.Protocol}://{FabricRuntime.GetNodeContext().IPAddressOrFQDN}:{endpoint.Port}";
//.UseWebListener()
_webHost = new WebHostBuilder().UseKestrel()
.UseContentRoot(Directory.GetCurrentDirectory())
.UseStartup<Startup>()
.UseUrls(serverUrl)
.Build();
_webHost.Start();
return Task.FromResult(serverUrl);
}
Everything plain and simple, no customizations.
The CORS call works, everything is perfect.
I did a test using Visual Studio Team Services with 15K users load, and it all worked like a charm, with 14K RPS. I think the Load test from VS do not use the CORS middleware by the way.
Now the problem is that when I put this exactly empty API in production, receiving calls from around just 100 simultaneous users, the CPU jump to 100% in 3 minutes. The calls are answered until the CPU reach 100% and then start sending errors back.
Seems that with 15000 users and NO CORS it all WORKS, and with 100 users + CORS DOES NOT WORK, CPU goes to 100% and remain like that until I reboot the VM Scale set.
If I stop sending calls, the CPU of the 5 nodes remains stable at 99% without receiving any single call.
How is this possible?
I tried everything, the project is plain and simple, the VS load test works, it's only when I put this on real CORS calls from different sites and different IP addresses that this happens.
I did a performance Trace in the server before sending traffic to it, again with a load test from Visual studio using EXACTLY the CORS headers everything is blazing fast.
With Real world calls this is what I see in the profiler:
There is nothing in it but the CORS middleware and the usual Kestrel processes.
The Stateless service eat up 99% of the CPU, and keep it even if I STOP the Traffic.
This is another 30 seconds trace with no traffic coming but CPU at 90%
I don't know what else to do, there is something wrong with CORS, I'm sure of it, even if it works, somehow something goes wrong.
This is a CORS call, correctly served.
Is there a bug in the Asp.net Core CORS middleware?
UPDATE:
I tried many combinations to isolate the problem:
New cluster, same Asp.net Core vanilla service=> Problem still there
Same cluster new project same Asp.net Core vanilla service=> Problem still there
Same Cluster WebAPI OWIN service, same code => Problem VANISHED!
The problem occurs with Asp.Net Core on Service Fabric using CORS with more than 50 concurrent requests.
This is the CPU (0.85%) with a Asp.Net Stateless service using Visual Studio template, OWIN and CORS, with around 100 concurrent connections and the same empty web API above
At this point I need help from a Microsoft official source to address the problem.
I'm almost sure this is a Asp.net Core CORS bug, that happens when you host it on Service Fabric as a stateless service and send it some minimum traffic (not just a couple of refresh in the browser).