Programmatic configuration of NLog in ASP.NET Core application to filter unwanted chatter? - asp.net-core

I'm attempting to filter out all the unnecessary chatter from the Microsoft hosting assemblies, while still allowing Debug-level messages from our own code to pass through to our logging targets.
Here's code that configures NLog and initializes a web app. Unfortunately, the Microsoft.* and System.* namespace filtering does not functional at all. The result is that all messages Debug and higher are logged.
I do not want to use the old nlog.config process. I want to be able to build the LoggingConfiguration in code.
I really must be missing something easy here, but I don't see it!
public static void Main(string[] args)
{
// Read configuration from a variety of sources to compose our final IConfiguration
var config = ReadOurConfigFromLotsOfPlaces();
var nLogConfig = new LoggingConfiguration();
var consoleTarget = new ColoredConsoleTarget("console")
{
AutoFlush = true,
ErrorStream = true,
Layout = #"${date:format=yyyy-MM-dd HH\:mm\:ss.fff} ${level} [${logger}] - ${message}${onexception:${newline}}${exception:format=shortType,message,stackTrace:maxInnerExceptionLevel=5}"
};
nLogConfig.AddTarget(consoleTarget);
nLogConfig.LoggingRules.Add(new LoggingRule("Microsoft.*", LogLevel.Warn, consoleTarget) {Final = true});
nLogConfig.LoggingRules.Add(new LoggingRule("System.*", LogLevel.Warn, consoleTarget) {Final = true});
nLogConfig.LoggingRules.Add(new LoggingRule("*", LogLevel.Debug, consoleTarget));
var logger = NLogBuilder.ConfigureNLog(nLogConfig).GetCurrentClassLogger();
logger.Debug("NLog initialization complete");
try
{
logger.Debug("init main");
CreateHostBuilder(config).Build().Run();
}
catch (Exception exception)
{
logger.Error(exception, "Stopped program because of exception");
throw;
}
finally
{
// Ensure to flush and stop internal timers/threads before application-exit (Avoid segmentation fault on Linux)
LogManager.Shutdown();
}
}
public static IWebHostBuilder CreateHostBuilder(IConfiguration config)
{
// we don't call CreateDefaultBuilder() because we already have assembled the configuration we want to use
return new WebHostBuilder()
.UseConfiguration(config)
.UseKestrel(options => options.UseSystemd())
.UseStartup<Startup>()
.ConfigureLogging(logging =>
{
logging.ClearProviders();
logging.SetMinimumLevel(Microsoft.Extensions.Logging.LogLevel.Trace);
})
.UseNLog();
}
Any tips would be greatly appreciated.

Ugh. I understand my problem now: The 'LoggingRules' is looking for a positive match between logger name and the min/max level values, so my first two LoggingRule objects did not match messages coming from the Microsoft.* and System.* namespaces, so the rules did nothing with those messages.
In order to accomplish the filtering I want, this is the solution:
var nLogConfig = new LoggingConfiguration();
var consoleTarget = new ColoredConsoleTarget("console")
{
AutoFlush = true,
ErrorStream = true,
Layout = #"${date:format=yyyy-MM-dd HH\:mm\:ss.fff} ${level} [${logger}] - ${message}${onexception:${newline}}${exception:format=shortType,message,stackTrace:maxInnerExceptionLevel=5}"
};
nLogConfig.AddTarget(consoleTarget);
var nullTarget = new NullTarget("null");
nLogConfig.AddTarget(nullTarget);
// matches every Microsoft.* logger with a LogLevel less than LogLevel.Warn
nLogConfig.LoggingRules.Add(new LoggingRule("Microsoft.*", LogLevel.Trace, LogLevel.Info, nullTarget) {Final = true});
// matches every System.* logger with a LogLevel less than LogLevel.Warn
nLogConfig.LoggingRules.Add(new LoggingRule("System.*", LogLevel.Trace, LogLevel.Info, nullTarget) {Final = true});
// and everything else, LogLevel.Debug and higher
nLogConfig.LoggingRules.Add(new LoggingRule("*", LogLevel.Debug, consoleTarget));

Related

NServiceBus with RabbitMQ Simple event

I want to be able to used NServiceBus to add a message on a queue in RabbitMQ. I dont want to handle it yet so just want to see an item on the queue, my code is as follows, but I get this error when I run it?
I have been trying to look at the documentation but is seems overly confusing. I am familar with RabbitMq and using it as is or with the Rabbit client library, but NService bus seems to complicate and confuse the situation!
using Shared;
using System;
using System.Threading.Tasks;
namespace NServiceBus.RabbitMqTest
{
class Program
{
static async Task Main(string[] args)
{
var endpointConfiguration = new EndpointConfiguration("UserChanged");
var transport = endpointConfiguration.UseTransport<RabbitMQTransport>();
transport.UseConventionalRoutingTopology();
transport.ConnectionString("host=localhost;username=guest;password=guest");
//transport.Routing().RouteToEndpoint(typeof(MyCommand), "Samples.RabbitMQ.SimpleReceiver");
endpointConfiguration.EnableInstallers();
var endpointInstance = await Endpoint.Start(endpointConfiguration).ConfigureAwait(false);
await SendMessages(endpointInstance);
//await endpointInstance.Publish(new UserChanged { UserId = 76 });
await endpointInstance.Stop().ConfigureAwait(false);
}
static async Task SendMessages(IMessageSession messageSession)
{
Console.WriteLine("Press [e] to publish an event. Press [Esc] to exit.");
while (true)
{
var input = Console.ReadKey();
Console.WriteLine();
switch (input.Key)
{
//case ConsoleKey.C:
// await messageSession.Send(new MyCommand());
// break;
case ConsoleKey.E:
await messageSession.Publish(new UserChanged { UserId = 87 });
break;
case ConsoleKey.Escape:
return;
}
}
}
}
}
Your endpoint is publishing the message as well as receiving it. Since there's no handler defined to handle the UserChanged messages (events), NServiceBus recoverability kicks in. Your options are
Declare the endpoint as send-only to avoid handling the messages when there are no handlers defined
Define a handler for UserChanged

Serilog Elasticsearch

i have a microservice api where I try to log all the request which come in...so the elasticsaerch service and kibana are on a different server. I'm using the serilog.sinks.elasticsearch package to send data to the elasticsearch.
Both servers are not running with docker, they are just normal windows server.
My code looks like this to setup the logging...
public static Logger Create(IConfiguration configuration)
{
var elasticsearchSection = configuration.GetSection("Elasticsearch");
if (elasticsearchSection != null)
{
return CreateLoggerConfiguration(elasticsearchSection).CreateLogger();
}
return null;
}
private static LoggerConfiguration CreateLoggerConfiguration(IConfigurationSection section)
{
var loggerConfiguration = new LoggerConfiguration();
var url = section.GetValue<string>("Url");
var minimumLogLevel = section.GetValue<string>("MinimumLogLevel");
var minimumLogEventLevel = section.GetValue<string>("MinimumLogEventLevel");
SetLoggerConfigurationMinimumLogLevel(minimumLogLevel, loggerConfiguration);
loggerConfiguration.WriteTo.Elasticsearch(new ElasticsearchSinkOptions(GetLoggingUri(url))
{
MinimumLogEventLevel = ReturnLogEventLevel(minimumLogEventLevel),
AutoRegisterTemplate = true
});
loggerConfiguration.Enrich.FromLogContext();
return loggerConfiguration;
}
And in my startup,cs I'm using
services.AddLogging(loggingBuilder => loggingBuilder.AddSerilog(dispose: true));
in the ConfigureServices Method...
But apparently I cant create an Index inside my Kibana.
Any ideas why this isnt working?

Why am I getting this error from NLog during integration testing: LayoutRenderer cannot be found

I added some aspnet layout renderers to my NLog configuration, and everything works fine for my website, but when I try to run my integration tests, they fail with this error message:
NLog.NLogConfigurationException : Error when setting property 'Layout' on NLog.Targets.DatabaseParameterInfo
----> System.ArgumentException : LayoutRenderer cannot be found: 'aspnet-request-cookie'. Is NLog.Web not included?
My initial instinct was to follow their suggestion and add the NLog.Web.AspNetCore package to my integration Test project. That did not do anything.
Here's what my integration test setup looks like:
[OneTimeSetUp]
public static void BaseOneTimeSetUp()
{
var environment = Environment.GetEnvironmentVariable("ASPNETCORE_ENVIRONMENT") ?? "Development";
Assume.That(environment, Is.AnyOf("Development", "CI"), "Integration tests can only run in one of those environments");
var serviceCollection = new ServiceCollection();
serviceCollection.AddSingleton<IHostingEnvironment>(p =>
{
var mock = new Mock<IHostingEnvironment>();
mock.SetupGet(m => m.ContentRootPath).Returns(TestContext.CurrentContext.TestDirectory);
mock.SetupGet(m => m.EnvironmentName).Returns(environment);
return mock.Object;
});
Startup.AddConfiguration(serviceCollection); //throws NLogConfigurationException
}
And here's my implementation for the Startup AddConfiguration method:
public static void AddConfiguration(IServiceCollection services)
{
services.AddSingleton<IConfiguration>(s =>
{
var env = s.GetService<IHostingEnvironment>();
var builder = new ConfigurationBuilder()
.SetBasePath(env.ContentRootPath)
.AddJsonFile("appsettings.json", false, true)
.AddJsonFile($"appsettings.{env.EnvironmentName}.json", optional: true)
.AddEnvironmentVariables();
var config = builder.Build();
// This next line throws the NLogConfigurationException
LogManager.Configuration = new NLogLoggingConfiguration(config.GetSection("NLog"));
return config;
});
}
The integration test is not calling the UseNLog method for the asp.net WebHost, as shown in their example code:
WebHost.CreateDefaultBuilder(args)
.UseStartup<Startup>()
.ConfigureLogging(logging =>
{
logging.ClearProviders();
logging.SetMinimumLevel(Microsoft.Extensions.Logging.LogLevel.Trace);
})
.UseNLog(); // NLog: setup NLog for Dependency injection
If you don't want to set up a WebHost for your tests, you can make do with a Mock<IWebHostBuilder>.
All you need to do is add this to your test setup before configuring NLog:
var mockHostBuilder = new Mock<IWebHostBuilder>();
mockHostBuilder.Object.UseNLog();

How to control log level in ASP.Net core Integration tests

I am trying to limit the amount the log information being printed when running integration tests for asp.net core. Currently, everything down to debug level is being printed and it obscures useful information. I would really like to restrict it to warning and above.
I am running integration tests using the example at https://learn.microsoft.com/en-us/aspnet/core/test/integration-tests?view=aspnetcore-2.2#customize-webapplicationfactory
public class CustomWebApplicationFactory<TStartup>
: WebApplicationFactory<TStartup> where TStartup: class
{
protected override void ConfigureWebHost(IWebHostBuilder builder)
{
builder.ConfigureServices(services =>
{
// Create a new service provider.
var serviceProvider = new ServiceCollection()
.AddEntityFrameworkInMemoryDatabase()
.BuildServiceProvider();
// Add a database context (ApplicationDbContext) using an in-memory
// database for testing.
services.AddDbContext<ApplicationDbContext>(options =>
{
options.UseInMemoryDatabase("InMemoryDbForTesting");
options.UseInternalServiceProvider(serviceProvider);
});
// Build the service provider.
var sp = services.BuildServiceProvider();
// Create a scope to obtain a reference to the database
// context (ApplicationDbContext).
using (var scope = sp.CreateScope())
{
var scopedServices = scope.ServiceProvider;
var db = scopedServices.GetRequiredService<ApplicationDbContext>();
var logger = scopedServices
.GetRequiredService<ILogger<CustomWebApplicationFactory<TStartup>>>();
// Ensure the database is created.
db.Database.EnsureCreated();
try
{
// Seed the database with test data.
Utilities.InitializeDbForTests(db);
}
catch (Exception ex)
{
logger.LogError(ex, "An error occurred seeding the database. Error: {Message}", ex.Message);
}
}
});
}
}
Logs:
dbug: Microsoft.AspNetCore.Hosting.Internal.WebHost[4]
Hosting started
dbug: Microsoft.AspNetCore.Hosting.Internal.WebHost[0]
Loaded hosting startup assembly BackEnd.Api
info: Microsoft.AspNetCore.Hosting.Internal.WebHost[1]
Request starting HTTP/2.0 GET http://localhost/test
dbug: Microsoft.AspNetCore.HostFiltering.HostFilteringMiddleware[0]
Wildcard detected, all requests with hosts will be allowed.
...lots more dbug logs...
Things I have tried in the ConfigureWebHost:
builder.ConfigureLogging(o =>
{
o.SetMinimumLevel(LogLevel.Warning);
});
No effect. I could not come up with any other combination of levels and filters that had any effect.
builder.ConfigureLogging(o =>
{
o.ClearProviders();
});
Stops all logging, but that's not really what I want.
Try to use AddFilter like
builder.ConfigureLogging(o=> {
//o.SetMinimumLevel(LogLevel.Warning);
o.AddFilter(logLevel => logLevel >= LogLevel.Warning);
});
I resolved this for my case. In the appsettings.json for the asp.net core app I was testing, was the following config:
"Console": {
"LogLevel": {
"Default": "Debug"
}
}
It seems that this would override any of the filtering or level changes I made in the code for the integration tests. Changing this level to Warning resolved my issue.

NServiceBus Send-Only endpoints not generating heartbeats

I'm using NServiceBus.Core v6.4.3 and NServiceBus.Heartbeat v2.0.0
I have a console application running as a Scheduled Task, it extracts data and send commands to an endpoint for processing.
The console application is configured as a SendOnly endpoint.
My code is as follows:
Main
// Local NServiceBus Configuration
var endpointConfiguration = EndpointConfiguration();
// Global NServiceBus & Ninject configuration
var conventions = new NServiceBusConventions();
conventions.Customize(endpointConfiguration);
// Create and start endpoint
var endpointInstance = await Endpoint.Start(endpointConfiguration).ConfigureAwait(false);
EndpointConfiguration
private static EndpointConfiguration EndpointConfiguration()
{
var configuration = new EndpointConfiguration("EndpointName");
// To ensure OctopusDeploy doesn't cause ServicePulse to think multiple services have been deployed
// http://docs.particular.net/nservicebus/hosting/override-hostid
configuration.UniquelyIdentifyRunningInstance()
.UsingNames("EndpointName", Environment.MachineName);
configuration.SendOnly();
return configuration;
}
Conventions
public class NServiceBusConventions
{
public IKernel Kernel;
public void Customize(EndpointConfiguration configuration)
{
// Custom Logging Factory implementation
LogManager.UseFactory(new NServiceBusTraceLoggerFactory());
Kernel = NinjectCommon.Start();
configuration.UseContainer<NinjectBuilder>(b => b.ExistingKernel(Kernel));
configuration.UsePersistence<NHibernatePersistence>();
configuration.UseSerialization<JsonSerializer>();
configuration.UseTransport<MsmqTransport>();
var transport = configuration.UseTransport<MsmqTransport>();
// Enabled by default in MsmqTransport, but to ensure we have it
transport.Transactions(TransportTransactionMode.TransactionScope);
configuration.DefineCriticalErrorAction(NServiceBusOnCriticalError.OnCriticalError);
configuration.EnableInstallers();
configuration.Conventions()
.DefiningCommandsAs(t => t.Namespace != null && t.Namespace.Equals("Contracts.Commands"))
.DefiningEventsAs(t => t.Namespace != null && t.Namespace.Equals("Contracts.Interfaces.Events"));
configuration.AuditProcessedMessagesTo(ConfigurationManager.AppSettings["Messaging.NServiceBus.QueueNames.AuditQueue"]);
configuration.SendFailedMessagesTo(ConfigurationManager.AppSettings["Messaging.NServiceBus.QueueNames.ErrorQueue"]);
configuration.SendHeartbeatTo(ConfigurationManager.AppSettings["Messaging.NServiceBus.QueueNames.ServiceControlQueue"]);
var scanner = configuration.AssemblyScanner();
var excludeRegexs = new List<string>
{
#"DevExpress.*\.dll"
};
var baseDirectory = AppDomain.CurrentDomain.BaseDirectory;
foreach (var fileName in Directory.EnumerateFiles(baseDirectory, "*.dll").Select(Path.GetFileName))
{
foreach (var pattern in excludeRegexs)
{
if (Regex.IsMatch(fileName, pattern, RegexOptions.IgnoreCase))
{
scanner.ExcludeAssemblies(fileName);
break;
}
}
}
}
}
Removing the configuration.SendOnly(); line in EndpointConfiguration makes the endpoint appear in ServicePulse, but it doesn't appear otherwise.
I knew this was an issue in previous versions, but I thought this had been fixed in NServiceBus V5.
I don't have to configure the endpoint as Send-Only, but I was just for completeness.
The reason behind the missing heartbeats was:
I have a console application running as a Scheduled Task, it extracts
data and send commands to an endpoint for processing.
The process would start fresh each time and the time taken to extract data, process and send the commands was too short for NServiceBus to get the heartbeat messages sent.
Putting an await Task.Delay(10000) at the end of the application was enough to allow NServiceBus to complete its necessary bootstrapping and didn't impact our SLA.
Thanks to Sean Farmar for his help in diagnosing