Substitute parts of a typed array in ASP.NET core appsettings.json from secrets/environment variables? - asp.net-core

We have an ASP.NET Core web app with this appsettings.json:
{
"Subscriptions": [
{
"Name": "Production",
"PublishSettings": "<PublishData>SECRET</PublishData>",
"Environments": [
{
"Name": "Prod",
"DeploymentServiceNames": [
"api1",
"api2",
"api3"
]
}
]
},
{
"Name": "Test",
"PublishSettings": "<PublishData>SECRET</PublishData>",
"Environments": [
{
"Name": "Test1",
"DeploymentServiceNames": [
"api1",
"api2"
]
},
{
"Name": "Test2",
"DeploymentServiceNames": [
"api1",
"api2"
]
}
]
}
]
}
The PublishSettings values are secret so I want these in my local user secrets file, and in environment variables for my deployments. But, because Subscriptions is an array I'm not sure how. I don't particularly want to swap in the entire Subscriptions section. Is there a way to swap in a single property for each item in such an array, perhaps by defining a key property on the strongly typed subscription model?

When you load configuration in .NET Core, under the hood it's represented as a set of key-value pairs (both key and value have string type) supplied by added configuration providers.
For example, appsettings.json will be represented by JsonConfigurationProvider as the following settings list:
{Subscriptions:0:Environments:0:DeploymentServiceNames:0, api1}
{Subscriptions:0:Environments:0:DeploymentServiceNames:1, api2}
{Subscriptions:0:Environments:0:DeploymentServiceNames:2, api3}
{Subscriptions:0:Environments:0:Name, Prod}
{Subscriptions:0:Name, Production}
{Subscriptions:0:PublishSettings, <PublishData>SECRET</PublishData>}
{Subscriptions:1:Environments:0:DeploymentServiceNames:0, api1}
{Subscriptions:1:Environments:0:DeploymentServiceNames:1, api2}
{Subscriptions:1:Environments:0:Name, Test1}
{Subscriptions:1:Environments:1:DeploymentServiceNames:0, api1}
{Subscriptions:1:Environments:1:DeploymentServiceNames:1, api2}
{Subscriptions:1:Environments:1:Name, Test2}
{Subscriptions:1:Name, Test}
{Subscriptions:1:PublishSettings, <PublishData>SECRET</PublishData>}
As you see JSON structure was flattened and keys are built by joining inner section names with a colon. Array element are added with appropriate index as a name.
If you add another configuration source, e.g. environment variables or another secrets json file, which will have settings with the same keys, it will overwrite the setting.
So if you want to add or overwrite PublishSettings, you could add either another JSON file as configuration source:
{
"Subscriptions": [
{
"PublishSettings": "<PublishData>SECRET</PublishData>"
},
{
"PublishSettings": "<PublishData>SECRET</PublishData>"
}
]
}
Or add it as environment variables with the following keys:
Subscriptions:0:PublishSettings
Subscriptions:1:PublishSettings
Such setting override (or addition) is transparent for .NET Core configuration binder. Settings POCO will contain value of PublishSettings from the last configuration source that provides such value.

Related

Grafana-LogQL: HowTo extract labels from key-value objects in json array

I am working with ASP.NET 5.0 json logger and logging scopes. I want to populate the scope key-values as labels.
The json produced is of the following format (excerpt):
{
"LogLevel": "Information",
"Scopes": [
{
"Message": "System.Collections.Generic.Dictionary\u00602[System.String,System.Object]",
"MsgId": "c08e834e8edb4287ab8abf0b5510bb53"
},
{
"Message": "System.Collections.Generic.Dictionary\u00602[System.String,System.Object]",
"EventId": "03ec8be0-9975-482e-95b9-2ba6185a4ed4",
"EventName": "someEvent",
"EntityKeyValue": "someNonTechId"
}
]
}
The only way I found was to do
| json MsgId="Scopes[0].MsgId", EventName="Scopes[1].EventName" etc. ...
Problem is that:
not all scopes are present at all times
so also the indices could change...
Is there any solution for that?
BTW we operate on a managed cluster, so custom plugins won't work...

Generate "Instances" definition programmatically to create EMR cluster in StepFunctions

I have a case where I want to dynamically create an EMR cluster based on a user-defined configuration and execute a sequence of steps on it using AWS Step Functions.
For this, I am planning to provide the instance configuration as an input to the step functions workflow.
Based on the StepFunctions-EMR Integration Documentation, the definition is the same as that of the RunJobFlow API.
However, when I try to generate the definition by serializing an instance of JobFlowInstancesConfig to JSON and pass it to the StateMachine as an input, it throws an error saying:
The field 'Instances.KeepJobFlowAliveWhenNoSteps' is required but was missing
Here is the JSON generated post serialization:
{
"instanceFleets": [
{
"instanceFleetType": "MAIN",
"targetOnDemandCapacity": 1,
"instanceTypeConfigs": [
{
"instanceType": "m5.xlarge"
}
]
},
{
"instanceFleetType": "CORE",
"targetOnDemandCapacity": 1,
"instanceTypeConfigs": [
{
"instanceType": "c5.2xlarge"
}
]
}
],
"keepJobFlowAliveWhenNoSteps": true
}
I am passing this in the input, and accessing it in my StepFunctions definition in the below Task (where I expect the above definition to be replacing $.jobFlowInstancesConfig):
...
"GetCluster": {
"Type": "Task",
"Resource": "arn:aws:states:::elasticmapreduce:createCluster.sync",
"Parameters": {
"Name.$": "$.clusterName",
"VisibleToAllUsers": true,
"ReleaseLabel": "emr-5.30.0",
"Applications": [
{
"Name": "Spark"
}
],
"ServiceRole": "EMR_DefaultRole",
"JobFlowRole": "EMR_EC2_DefaultRole",
"LogUri": "s3://my-aws-logs/elasticmapreduce/",
"Instances.$": "$.jobFlowInstancesConfig"
}
}
...
My suspicion is that this is failing because StepFunctions expects the field names to start with upper case.
Question: How do I programmatically generate the appropriate definition without having to play around with Strings for generating the JSON? Is there a straightforward way to serialize the above definition to one that will work with StepFunctions?

Validating correctness of $ref in json schema

The requirement is to validate given json schema that there are no dangling $ref pointing to the definitions within the file.
{
"$schema": "http://json-schema.org/draft-6/schema#",
"definitions": {
"date": {
"type": "string",
"pattern": "^(0?[1-9]|[12][0-9]|3[01])\\-(0?[1-9]|1[012])\\-\\d{4}$"
},
},
"properties": {
"my_date": {"$ref": "#/definitions/dat"}
}
}
Here, there is a typo in the reference (dat instead of date). I want to catch such instances rather than having a run time failure.
Library being used: https://github.com/java-json-tools/json-schema-validator
You could validate that the use of $ref resolves by digesting the JSON, recursivly extracting the value of $ref, splitting on slash, and checking the path exists.
This COULD get more complicated as you might have external references which target URLs.
I can't give you any code as I don't know JAVA. It doesn't seem like what you want is specifically available using that library.

splitting swagger definition across many files

Question: how can I split swagger definition across files? What are the possibilities in that area? The question details are described below:
example of what I want - in RAML
I do have experience in RAML and what I do is, for example:
/settings:
description: |
This resource defines application & components configuration
get:
is: [ includingCustomHeaders ]
description: |
Fetch entire configuration
responses:
200:
body:
example: !include samples/settings.json
schema: !include schemas/settings.json
The last two lines are important here - theones with !include <filepath> - in RAML I can split my entire contract into many files that just get included dynamically by the RAML parser (and RAML parser is used by all tools that base on RAML).
My benefit from this is that:
I get my contract more clear and easier to maintain, because schemas are not inline
but that's really important: I can reuse the schema files within other tools to do validation, mock generation, stubs, generate tests, etc. In other words, this way I can reuse schema files within both contract (RAML, this case) and other tools (non-RAML, non-swagger, just JSONschema-based ones).
back to Swagger
As far as I read, swagger supports $ref keyword which allows to load external files. But is that files fetched through HTTP/AJAX or can they just be local files?
And is that supported by the whole specification or is it just some tools that support it and some that don't?
What I found here is that the input for swagger has to be one file. And this is extremely inconvenient for big projects:
because of size
and because I can't reuse the schema if I want to use something non-swagger
Or, in other words, can I achieve the same with swagger, that I can with RAML - in terms of splitting files?
The specification allows for references in multiple locations but not everywhere. These references are resolved depending on where the specification is being hosted--and what you're trying to do.
For something like rendering a dynamic user interface, then yes you do need to eventually load the entire definition into "a single object" which may be composed from many files. If performing a code generation, the definitions may be loaded directly from the file system. But ultimately there are swagger parsers doing the resolution, which is much more fine grained and controllable in Swagger than other definition formats.
In your case, you would use a JSON pointer to the schema reference:
responses:
200:
description: the response
schema:
via local reference
$ref: '#/definitions/myModel'
via absolute reference:
$ref: 'http://path/to/your/resource'
via relative reference, which would be 'relative to where this doc is loaded'
$ref: 'resource.json#/myModel
via inline definition
type: object
properties:
id:
type: string
When I split OpenAPI V3 files using references, I try to avoid the sock drawer anti-pattern and instead use functional groupings for the YAML files.
I also make it so that each YAML file itself is a valid OpenAPI V3 spec.
I start out with the openapi.yaml file.
openapi: 3.0.3
info:
title: MyAPI
description: |
This is the public API for my stuff.
version: "3"
tags:
# NOTE: the name is needed as the info block uses `title` rather than name
- name: Authentication
$ref: 'authn.yaml#/info'
paths:
# NOTE: here are the references to the other OpenAPI files
# from the path. Note because OpenAPI requires paths to
# start with `/` and that is already used as a separator
# replace the `/` with `%2F` for the path reference.
'/authn/start':
$ref: 'authn.yaml#/paths/%2Fstart'
Then in the functional group:
openapi: 3.0.3
info:
title: Authentication
description: |
This is the authentication module.
version: "3"
paths:
# NOTE: don't include the `/authn` prefix here that top level grouping is
# in the `openapi.yaml` file.
'/start':
get:
responses:
"200":
description: OK
By doing this separation you can independently test each file or the whole API as a group.
There may be points where you repeat yourself, but by doing this you limit the chance of breaking changes to other API endpoints when using a "common" library.
However, you should still have a common definition library for some things such as:
errors
security
There is a limitation on this approach and that's the "Discriminators" (it may be a ReDoc issue though, but if you had types that have discriminators outside of the openapi.yaml ReDoc fails to render correctly.
See this answer for details on how to split your Swagger documentation across many files. This is done using JSON, but the same concept can apply to RAML.
EDIT: Adding content of link here
The basic structure of your Swagger JSON should look something like this:
{
"swagger": "2.0",
"info": {
"title": "",
"version": "version number here"
},
"basePath": "/",
"host": "host goes here",
"schemes": [
"http"
],
"produces": [
"application/json"
],
"paths": {},
"definitions": {}
}
The paths and definitions are where you need to insert the paths that your API supports and the model definitions describing your response objects. You can populate these objects dynamically. One way of doing this could be to have a separate file for each entity's paths and models.
Let's say one of the objects in your API is a "car".
Path:
{
"paths": {
"/cars": {
"get": {
"tags": [
"Car"
],
"summary": "Get all cars",
"description": "Returns all of the cars.",
"responses": {
"200": {
"description": "An array of cars",
"schema": {
"type": "array",
"items": {
"$ref": "#/definitions/car"
}
}
},
"404": {
"description": "error fetching cars",
"schema": {
"$ref": "#/definitions/error"
}
}
}
}
}
}
Model:
{
"car": {
"properties": {
"_id": {
"type": "string",
"description": "car unique identifier"
},
"make": {
"type": "string",
"description": "Make of the car"
},
"model":{
"type": "string",
"description": "Model of the car."
}
}
}
}
You could then put each of these in their own files. When you start your server, you could grab these two JSON objects, and append them to the appropriate object in your base swagger object (either paths or definitions) and serve that base object as your Swagger JSON object.
You could also further optimize this by only doing the appending once when the server is started (since the API documentation will not change while the server is running). Then, when when the "serve Swagger docs" endpoint is hit, you can just return the cached Swagger JSON object that you created when the server was started.
The "serve Swagger docs" endpoint can be intercepted by catching a request to /api-docs like below:
app.get('/api-docs', function(req, res) {
// return the created Swagger JSON object here
});
You can use $ref but not have good flexibility, I suggest you process YAML with an external tool like 'Yamlinc' that mix multiple files into one using '$include' tag.
read more: https://github.com/javanile/yamlinc

ARM - How can I get the access key from a storage account to use in AppSettings later in the template?

I'm creating an Azure Resource Manager template that instantiates multiple resources, including an Azure storage account and an Azure App Service with a Web App.
I'd like to be able to capture the primary access key (or the full connection string, either way is fine) from the newly-created storage account, and use that as a value for one of the AppSettings for the Web App.
Is that possible?
Use the listkeys helper function.
"appSettings": [
{
"name": "STORAGE_KEY",
"value": "[listKeys(resourceId('Microsoft.Storage/storageAccounts', parameters('storageAccountName')), providers('Microsoft.Storage', 'storageAccounts').apiVersions[0]).keys[0].value]"
}
]
This quickstart does something similar:
https://azure.microsoft.com/en-us/documentation/articles/cache-web-app-arm-with-redis-cache-provision/
The syntax has changed since the other answer was accepted. The error you will now hit is 'Template language expression property 'key1' doesn't exist, available properties are 'keys'
Keys are now represented as an array of keys, and the syntax is now:
"StorageAccount": "[Concat('DefaultEndpointsProtocol=https;AccountName=',variables('StorageAccountName'),';AccountKey=',listKeys(resourceId('Microsoft.Storage/storageAccounts', variables('StorageAccountName')), providers('Microsoft.Storage', 'storageAccounts').apiVersions[0]).keys[0].value)]",
See: http://samcogan.com/retrieve-azure-storage-key-in-arm-script/
I faced with this issue two times. First in the 2015 and last today in May of 2017.
I need to add connection strings to the WebApp - I want to add strings automatically from generated resources during deployment from the ARM template. It can help later to not add manually this values.
First time I used old version of the function listKeys (it looks like old version returns result not as object but as value):
"AzureWebJobsStorage": {
"type": "Custom",
"value": "[concat(variables('storageConnectionString'), listKeys(resourceId('Microsoft.Storage/storageAccounts', parameters('storageAccountName')), '2015-05-01-preview').key1)]"
},
Today last version of the working template is:
"resources": [
{
"apiVersion": "2015-08-01",
"type": "config",
"name": "connectionstrings",
"dependsOn": [
"[resourceId('Microsoft.Web/Sites/', parameters('webSiteName'))]"
],
"properties": {
"DefaultConnection": {
"value": "[concat('Data Source=tcp:', reference(resourceId('Microsoft.Sql/servers/', parameters('sqlserverName'))).fullyQualifiedDomainName, ',1433;Initial Catalog=', parameters('databaseName'), ';User Id=', parameters('administratorLogin'), '#', parameters('sqlserverName'), ';Password=', parameters('administratorLoginPassword'), ';')]",
"type": "SQLServer"
},
"AzureWebJobsStorage": {
"type": "Custom",
"value": "[concat(variables('storageConnectionString'), listKeys(resourceId('Microsoft.Storage/storageAccounts', parameters('storageName')), '2016-01-01').keys[0].value)]"
},
"AzureWebJobsDashboard": {
"type": "Custom",
"value": "[concat(variables('storageConnectionString'), listKeys(resourceId('Microsoft.Storage/storageAccounts', parameters('storageName')), '2016-01-01').keys[0].value)]"
}
}
},
Thanks.
below is example for adding storage account to ADLA
"storageAccounts": [
{
"name": "[parameters('DataLakeAnalyticsStorageAccountname')]",
"properties": {
"accessKey": "[listKeys(variables('storageAccountid'),'2015-05-01-preview').key1]"
}
}
],
in variable you can keep
"variables": {
"apiVersion": "[providers('Microsoft.Storage', 'storageAccounts').apiVersions[0]]",
"storageAccountid": "[concat(resourceGroup().id,'/providers/','Microsoft.Storage/storageAccounts/', parameters('DataLakeAnalyticsStorageAccountname'))]"
},