i have a microservice api where I try to log all the request which come in...so the elasticsaerch service and kibana are on a different server. I'm using the serilog.sinks.elasticsearch package to send data to the elasticsearch.
Both servers are not running with docker, they are just normal windows server.
My code looks like this to setup the logging...
public static Logger Create(IConfiguration configuration)
{
var elasticsearchSection = configuration.GetSection("Elasticsearch");
if (elasticsearchSection != null)
{
return CreateLoggerConfiguration(elasticsearchSection).CreateLogger();
}
return null;
}
private static LoggerConfiguration CreateLoggerConfiguration(IConfigurationSection section)
{
var loggerConfiguration = new LoggerConfiguration();
var url = section.GetValue<string>("Url");
var minimumLogLevel = section.GetValue<string>("MinimumLogLevel");
var minimumLogEventLevel = section.GetValue<string>("MinimumLogEventLevel");
SetLoggerConfigurationMinimumLogLevel(minimumLogLevel, loggerConfiguration);
loggerConfiguration.WriteTo.Elasticsearch(new ElasticsearchSinkOptions(GetLoggingUri(url))
{
MinimumLogEventLevel = ReturnLogEventLevel(minimumLogEventLevel),
AutoRegisterTemplate = true
});
loggerConfiguration.Enrich.FromLogContext();
return loggerConfiguration;
}
And in my startup,cs I'm using
services.AddLogging(loggingBuilder => loggingBuilder.AddSerilog(dispose: true));
in the ConfigureServices Method...
But apparently I cant create an Index inside my Kibana.
Any ideas why this isnt working?
Related
Note: this is a follow-up of Reusing a Polly retrial policy for multiple Refit endpoints without explicitly managing the HttpClient
When making Refit work with Polly and an Azure AD-based authentication (On Behalf Of flow), I realized that acquiring an OBO token can be very slow (>400ms). The code for acquiring an OBO token based on the current logger in the user access token is shown below:
public async Task<string> GetAccessToken(CancellationToken token)
{
var adSettings = _azureAdOptions.Value;
string[] scopes = new string[] { "https://foo.test.com/access_as_user" };
string? httpAccessToken = _httpContextAccessor.HttpContext?.Request?.Headers[HeaderNames.Authorization]
.ToString()
?.Replace("Bearer ", "");
if (httpAccessToken == null)
throw new ArgumentNullException("Failed to generate access token (OBO flow)");
string cacheKey = "OboToken_" + httpAccessToken;
string oboToken = await _cache.GetOrAddAsync(cacheKey, async () =>
{
IConfidentialClientApplication cca = GetConfidentialClientApplication(adSettings);
var assertion = new UserAssertion(httpAccessToken);
var result = await cca.AcquireTokenOnBehalfOf(scopes, assertion).ExecuteAsync(token);
return result.AccessToken;
},
new MemoryCacheEntryOptions { AbsoluteExpirationRelativeToNow = TimeSpan.FromMinutes(TokenCacheExpirationInMinutes) });
return oboToken;
}
private IConfidentialClientApplication GetConfidentialClientApplication(AzureAdAuthOptions adSettings)
{
var certMetadata = _azureAdOptions.Value.ClientCertificates[0];
string certPath = certMetadata.CertificateDiskPath;
_logger.LogInformation($"GetAccessToken certificate path = {certPath}");
string certPassword = certMetadata.CertificatePassword;
var certificate = new X509Certificate2(certPath, certPassword);
_logger.LogInformation($"GetAccessToken certificate = {certificate}");
var cca = ConfidentialClientApplicationBuilder
.Create(adSettings.ClientId)
.WithTenantId(adSettings.TenantId)
.WithCertificate(certificate)
// .WithClientSecret(adSettings.ClientSecret)
.Build();
return cca;
}
This seems to work fine (not tested in a production environment though). however, I feel that I am reinventing the wheel here as I managing the OBO token caching myself.
Currently, this flow is used by Refit configuration:
private static IServiceCollection ConfigureResilience(this IServiceCollection services)
{
services
.AddRefitClient(typeof(IBarIntegration), (sp) =>
{
var accessTokenHelperService = sp.GetRequiredService<IAccessTokenHelperService>();
return new RefitSettings
{
AuthorizationHeaderValueGetter = () => accessTokenHelperService.GetAccessToken(default)
};
})
.ConfigureHttpClient((sp, client) =>
{
var BarSettings = sp.GetRequiredService<IOptions<BarApiSettings>>();
string baseUrl = BarSettings.Value.BaseUrl;
client.BaseAddress = new Uri(baseUrl);
})
.AddPolicyHandler(Policy<HttpResponseMessage>
.Handle<HttpRequestException>()
.OrResult(x => x.StatusCode is >= HttpStatusCode.InternalServerError or HttpStatusCode.RequestTimeout)
.WaitAndRetryAsync(Backoff.DecorrelatedJitterBackoffV2(TimeSpan.FromSeconds(1), RetryPolicyMaxCount)));
return services;
}
Are there any caveats with the current implementation? I am only interested in possible security, performance or "reinventing-the-wheel" issues.
I'm using Apache Ignite on Azure Kubernetes as a distributed cache. Also, I have a web API on Azure based on .NET6.
I connect to Ignite with IgniteClient class. I made it a singleton but the connection closes in 5 seconds after starting.
I've tried
ReconnectDisabled = false
AND
SocketTimeout = TimeSpan.FromMilliseconds(System.Threading.Timeout.Infinite)
but both of them didn't work. How can I keep alive the connection all the time as a singleton?
Here is my configuration code of the Ignite;
public CacheManager()
{
ConnectIgnite();
}
public void ConnectIgnite()
{
_ignite = Ignition.StartClient(GetIgniteConfiguration());
}
public IgniteClientConfiguration GetIgniteConfiguration()
{
var appSettingsJson = AppSettingsJson.GetAppSettings();
var igniteEndpoints = appSettingsJson["AppSettings:IgniteEndpoint"];
var igniteUser = appSettingsJson["AppSettings:IgniteUser"];
var ignitePassword = appSettingsJson["AppSettings:IgnitePassword"];
var nodeList = igniteEndpoints.Split(",");
var config = new IgniteClientConfiguration
{
Endpoints = nodeList,
UserName = igniteUser,
Password = ignitePassword,
EnablePartitionAwareness = true,
SocketTimeout = TimeSpan.FromMilliseconds(System.Threading.Timeout.Infinite)
};
return config;
}
I have written a simple console application to get all application clients from OpenIddict server. I tried all the possibilities and getting the syntax error. The code is below. I did not find any example in Github and found some outdated example (2017) is no longer relevant now. Please help
public static async Task<bool> Test()
{
var services = CreateServices();
var provider = services.BuildServiceProvider();
var scope = provider.CreateScope();
var context = scope.ServiceProvider.GetRequiredService<CustomDbContext>();
await context.Database.EnsureCreatedAsync();
var manager = scope.ServiceProvider.GetRequiredService<IOpenIddictApplicationManager>();
var result = await manager.FindByClientIdAsync("TestApp"); // It Works
IQueryable<OpenIddictEntityFrameworkCoreApplication> _applicationsQuery = Enumerable.Empty<OpenIddictEntityFrameworkCoreApplication>().AsQueryable();
_applicationsQuery.Where(apps => apps.ClientId != "");
var clients = manager.ListAsync<Func<OpenIddictEntityFrameworkCoreApplication>>(_applicationsQuery); //Compiler Error
return (result != null);
}
private static IServiceCollection CreateServices()
{
var services = new ServiceCollection();
services.AddDbContext<CustomDbContext>(opts =>
{
opts.UseSqlServer(
ConfigurationManager.ConnectionStrings["DefaultConnection"].ConnectionString,
b => b.MigrationsAssembly("Program"));
opts.UseOpenIddict();
});
services.AddOpenIddict() // Register the OpenIddict core components.
.AddCore(options =>
{
// Configure OpenIddict to use the Entity Framework Core stores and models.
// Note: call ReplaceDefaultEntities() to replace the default OpenIddict entities.
options.UseEntityFrameworkCore()
.UseDbContext<CustomDbContext>();
// Enable Quartz.NET integration.
options.UseQuartz();
});
return services;
}
ListAsync() returns an IAsyncEnumerable<T> collection, so you can use await foreach to iterate the collection:
await foreach (var application in manager.ListAsync())
{
Console.WriteLine(await manager.GetClientIdAsync(application));
}
You can also reference the System.Linq.Async package and use the async LINQ extensions. For instance, here's how you could retrieve all the client identifiers of all existing applications:
var identifiers = await manager.ListAsync()
.SelectAwait(application => manager.GetClientIdAsync(application))
.ToListAsync();
yml string concatenate does not work with .NET applications.I have tried by removing '$' sign, but it is still not working(Java application uses $ sign - Working fine with Java apps). It is working fine for a single value, but not with concatenation.
yml-01
cicd:
dbname: 172.10.10.110
port: 5432
yml-02
datasource:
url: jdbc:postgresql://${cicd:dbname}:${cicd:port}/sample-db
A solution for placeholder resolution in .NET Configuration (similar to that provided by spring) is available in Steeltoe.Common. We haven't added WebHostBuilder or IConfigurationBuilder extensions just yet, but if you add a recent reference to Steeltoe.Common from the Steeltoe Dev feed you should be able to do something like this:
public static IWebHostBuilder ResolveConfigurationPlaceholders(this IWebHostBuilder hostBuilder, LoggerFactory loggerFactory = null)
{
return hostBuilder.ConfigureAppConfiguration((builderContext, config) =>
{
config.AddInMemoryCollection(PropertyPlaceholderHelper.GetResolvedConfigurationPlaceholders(config.Build(), loggerFactory?.CreateLogger("Steeltoe.Configuration.PropertyPlaceholderHelper")));
});
}
The code above is used in the Steeltoe fork of eShopOnContainers
You should take a look at YamlDotNet.
Here's an example of how to solve your problem using that lib
using YamlDotNet.RepresentationModel;
using YamlDotNet.Core;
Then in your method
var dbname = "172.10.10.110";
var port = "5432";
string content;
using (var reader = new StreamReader("your yml file"))
{
content = reader.ReadToEnd();
}
var doc = new StringReader(content);
var yaml = new YamlStream();
yaml.Load(doc);
// Add the url where you use string interpolation to replace the values
var ymlFile = (YamlMappingNode)yaml.Documents[0].RootNode;
ymlFile.Children["datasource"] = new YamlMappingNode
{
{ "url", $"jdbc:postgresql://{dbname}:{port}/sample-db" }
};
yaml.Save(File.CreateText("C:\\yourNewFile.yml"), assignAnchors: false);
Here's a link to the NetCore package
I've solved this by writing an extension method to the IConfiguration interface.
public static string ReadFromConfigRepo(this IConfiguration configuration, string key)
{
var pattern = #"\{(.*?)\}";
var query = configuration[key];
if (query.Contains('{'))
{
var matches = Regex.Matches(query, pattern);
string value;
foreach (Match m in matches)
{
value = configuration[m.Value.Substring(1, m.Value.Length - 2)];
query = query.Replace(m.Value, value);
}
}
return query.Trim();
}
I am trying to adapt an existing WebApi/MVC4 app to use Breeze lookups.
Currently I retrieve my DTOs via
[HttpGet]
public IQueryable<ThingDto> GetThings()
{
var channelFactory = ThingServiceConfiguration.CreateChannelFactory();
_serviceFactory = () => new WcfProxy<IThingService>(channelFactory.CreateChannel());
var client = _serviceFactory();
IQueryable<ThingDto> result = client.Execute(p => p.GetThings()).OrderBy(x => x.Name).AsQueryable();
return result;
}
I'm not sure how I implement this method Metadata()
public string Metadata()
{
//normally something like this if using a EF DataContext
// return _someContextProvider.Context.Things;
}
How I setup the WCF config
public class ThingServiceConfiguration
{
const string AppSettingKey = "ThingServiceUrl";
public static ChannelFactory<IThingService> CreateChannelFactory()
{
// var serviceUrl = ConfigurationManager.AppSettings[AppSettingKey];
var serviceUrl = "http://localhost:86/ThingService.svc";
var binding = new BasicHttpBinding(BasicHttpSecurityMode.None)
{
MaxReceivedMessageSize = 200000000,
SendTimeout = TimeSpan.FromMinutes(2),
ReceiveTimeout = TimeSpan.FromMinutes(2)
};
var address = new EndpointAddress(serviceUrl);
return new ChannelFactory<IThingService>(binding, address);
}
}
Could this metadata be provided with the WCF call into the Metadata() property (by providing arguments through BasicHttpBinding ?
Many thanks!
You can return Breeze 'native' metadata simply by returning the metadata in json form. Something like this:
[HttpGet]
public String Metadata() {
var folder = Path.Combine(HttpRuntime.AppDomainAppPath, "App_Data");
// metadata.json is the name of a file containing your metadata - pick any file name you like.
var fileName = Path.Combine(folder, "metadata.json");
var jsonMetadata = File.ReadAllText(fileName);
return jsonMetadata;
}
where the syntax of the metadata file is described here: Breeze metadata format.