I have a .Net 6 web application that I am building in devops. I am trying to substitute settings in the appsettings.json, but somehow can't get it working.
What I have:
Azure Key vault set up, with a secret with the name:
Configuration--ConnectionStrings--ConnectionString
Pipelines have access to this Key Vault with a Variable Group
I have tried the transform task in both build and release pipeline
I have tried the
option in the IIS Web app deploy task for substitution
1 - All is fine
2 - All is fine
3 - I have tried this task in build and release with the following configuration:
steps:
- task: FileTransform#1
displayName: 'File Transform: '
inputs:
folderPath: '$(System.DefaultWorkingDirectory)/**/WebAppFront.zip'
fileType: json
targetFiles: '**/appsettings.json'
The logs look OK for transform:
2022-03-17T10:04:32.9753812Z ##[section]Starting: File Transform:
2022-03-17T10:04:33.0157518Z ==============================================================================
2022-03-17T10:04:33.0158091Z Task : File transform
2022-03-17T10:04:33.0158579Z Description : Replace tokens with variable values in XML or JSON configuration files
2022-03-17T10:04:33.0159048Z Version : 1.198.0
2022-03-17T10:04:33.0159390Z Author : Microsoft Corporation
2022-03-17T10:04:33.0159938Z Help : https://learn.microsoft.com/azure/devops/pipelines/tasks/utility/file-transform
2022-03-17T10:04:33.0160522Z ==============================================================================
2022-03-17T10:04:33.6732027Z [command]C:\azagent\A1\_work\_tasks\FileTransform_8ce97e91-56cc-4743-bfab-9a9315be5f27\1.198.0\node_modules\azure-pipelines-tasks-webdeployment-common\7zip\7zip\7z.exe x -oC:\azagent\A1\_work\_temp\temp_web_package_2021667764440822 C:\azagent\A1\_work\r1\a\_Vind\drop\WebAppFront.zip
2022-03-17T10:04:33.7472675Z
2022-03-17T10:04:33.7679746Z 7-Zip [64] 16.00 : Copyright (c) 1999-2016 Igor Pavlov : 2016-05-10
2022-03-17T10:04:33.7687417Z
2022-03-17T10:04:33.7689291Z Scanning the drive for archives:
2022-03-17T10:04:33.7693972Z 1 file, 21535247 bytes (21 MiB)
2022-03-17T10:04:33.7839085Z
2022-03-17T10:04:33.7855780Z Extracting archive: C:\azagent\A1\_work\r1\a\_Vind\drop\WebAppFront.zip
2022-03-17T10:04:36.6203882Z --
2022-03-17T10:04:36.6205010Z Path = C:\azagent\A1\_work\r1\a\_Vind\drop\WebAppFront.zip
2022-03-17T10:04:36.6206577Z Type = zip
2022-03-17T10:04:36.6238261Z Physical Size = 21535247
2022-03-17T10:04:36.6250740Z
2022-03-17T10:04:36.6282342Z Everything is Ok
2022-03-17T10:04:36.6282941Z
2022-03-17T10:04:36.6283536Z Folders: 24
2022-03-17T10:04:36.6284339Z Files: 112
2022-03-17T10:04:36.6284868Z Size: 58919697
2022-03-17T10:04:36.6288304Z Compressed: 21535247
2022-03-17T10:04:36.6338841Z Applying JSON variable substitution for **/appsettings.json
2022-03-17T10:04:36.7353081Z Applying JSON variable substitution for C:\azagent\A1\_work\_temp\temp_web_package_2021667764440822\Content\D_C\a\1\s\Vind\WebAppFront\obj\Release\net6.0\PubTmp\Out\appsettings.json
2022-03-17T10:04:36.7444592Z JSON variable substitution applied successfully.
2022-03-17T10:04:40.1757797Z ##[section]Finishing: File Transform:
And also for the rest of the log like getting keyvault (done before transform):
2022-03-17T10:04:32.0237340Z ##[section]Starting: Download secrets: my-key-vault
2022-03-17T10:04:32.0691326Z ==============================================================================
2022-03-17T10:04:32.0691706Z Task : Azure Key Vault
2022-03-17T10:04:32.0691934Z Description : Download Azure Key Vault secrets
2022-03-17T10:04:32.0692142Z Version : 2.200.0
2022-03-17T10:04:32.0692355Z Author : Microsoft Corporation
2022-03-17T10:04:32.0692657Z Help : https://learn.microsoft.com/azure/devops/pipelines/tasks/deploy/azure-key-vault
2022-03-17T10:04:32.0693010Z ==============================================================================
2022-03-17T10:04:32.7133186Z SubscriptionId: my-subscription-id.
2022-03-17T10:04:32.7145990Z Key vault name: my-key-vault.
2022-03-17T10:04:32.7152879Z Downloading secret value for: Configuration--ConnectionStrings--ConnectionString.
2022-03-17T10:04:32.9707096Z ##[section]Finishing: Download secrets: my-key-vault
The appsettings.json looks like this:
{
"Configuration": {
"ApplicationName": "Lorem ipsum",
"Logging": {
"LogLevel": {
"Default": "Information",
"Microsoft": "Warning",
"Microsoft.Hosting.Lifetime": "Information"
}
},
"ConnectionStrings": {
"ConnectionString": ""
}
}
}
But the connectionstring stays empty. As I understood from reading a lot of blogs and documentation this should be working. The double hyphen -- in the key name, should translate to a dot, so Json path expression is correct (I think): Configuration.ConnectionStrings.ConnectionString
The IIS Web app deploy task, with transform gives similar result. So an empty connectionstring. Even though it explicitly states it updated the json in the zippackage.
Logs for that task:
2022-03-17T10:04:43.3419710Z ##[section]Starting: IIS Web App Deploy
2022-03-17T10:04:43.3940659Z ==============================================================================
2022-03-17T10:04:43.3941018Z Task : IIS web app deploy
2022-03-17T10:04:43.3941276Z Description : Deploy a website or web application using Web Deploy
2022-03-17T10:04:43.3941526Z Version : 0.198.0
2022-03-17T10:04:43.3941721Z Author : Microsoft Corporation
2022-03-17T10:04:43.3942065Z Help : https://learn.microsoft.com/azure/devops/pipelines/tasks/deploy/iis-web-app-deployment-on-machine-group
2022-03-17T10:04:43.3943670Z ==============================================================================
2022-03-17T10:04:44.0803824Z [command]C:\azagent\A1\_work\_tasks\IISWebAppDeploymentOnMachineGroup_1b467810-6725-4b6d-accd-886174c09bba\0.198.0\node_modules\azure-pipelines-tasks-webdeployment-common\7zip\7zip\7z.exe x -oC:\azagent\A1\_work\_temp\temp_web_package_39200019901712446 C:\azagent\A1\_work\r1\a\_Vind\drop\WebAppFront.zip
2022-03-17T10:04:44.1320582Z
2022-03-17T10:04:44.1365755Z 7-Zip [64] 16.00 : Copyright (c) 1999-2016 Igor Pavlov : 2016-05-10
2022-03-17T10:04:44.1393221Z
2022-03-17T10:04:44.1427445Z Scanning the drive for archives:
2022-03-17T10:04:44.1436856Z 1 file, 23620753 bytes (23 MiB)
2022-03-17T10:04:44.1451518Z
2022-03-17T10:04:44.1587844Z Extracting archive: C:\azagent\A1\_work\r1\a\_Vind\drop\WebAppFront.zip
2022-03-17T10:04:47.1556142Z --
2022-03-17T10:04:47.1556846Z Path = C:\azagent\A1\_work\r1\a\_Vind\drop\WebAppFront.zip
2022-03-17T10:04:47.1558328Z Type = zip
2022-03-17T10:04:47.1558787Z Physical Size = 23620753
2022-03-17T10:04:47.1559027Z
2022-03-17T10:04:47.1566021Z Everything is Ok
2022-03-17T10:04:47.1566315Z
2022-03-17T10:04:47.1566617Z Folders: 35
2022-03-17T10:04:47.1567095Z Files: 112
2022-03-17T10:04:47.1567474Z Size: 58919916
2022-03-17T10:04:47.1567818Z Compressed: 23620753
2022-03-17T10:04:47.2311338Z Applying JSON variable substitution for **/appsettings.json
2022-03-17T10:04:47.3655684Z Applying JSON variable substitution for C:\azagent\A1\_work\_temp\temp_web_package_39200019901712446\Content\D_C\a\1\s\Vind\WebAppFront\obj\Release\net6.0\PubTmp\Out\appsettings.json
2022-03-17T10:04:47.3751003Z JSON variable substitution applied successfully.
2022-03-17T10:04:50.9446139Z [command]"C:\azagent\A1\_work\_tasks\IISWebAppDeploymentOnMachineGroup_1b467810-6725-4b6d-accd-886174c09bba\0.198.0\node_modules\azure-pipelines-tasks-webdeployment-common\MSDeploy3.6\MSDeploy3.6\msdeploy.exe" -verb:sync -source:package='C:\azagent\A1\_work\r1\a\temp_web_package_8577780759906015.zip' -dest:auto -setParam:name='IIS Web Application Name',value='www.mydomain.com' -enableRule:DoNotDeleteRule
2022-03-17T10:04:51.7358688Z Info: Updating file (www.mydoain.com\appsettings.json).
2022-03-17T10:04:51.7839580Z Total changes: 1 (0 added, 0 deleted, 1 updated, 0 parameters changed, 2281 bytes copied)
2022-03-17T10:04:51.8207636Z ##[section]Finishing: IIS Web App Deploy
What am I doing wrong? The only thing I noticed when I was echoing out the variable it got cut off on a ; in the connectionstring, but even when I tried some simple value it was not working.
I ended up solving it by creating variables in the pipelines and using the Key Vault secrets as the value.
Keyvault secret name: 'MyConnectionSecret'
You have to use the dot notation to specify a path. See my question for the json structure.
Name of var in Dev ops:
Configuration.ConnectionStrings.ConnectionString
Value of var in Dev ops:
$(MyConnectionSecret)
And of course in the Library you have to add the variable group based on the Keyvault
I managed to resolve this issue automatically without the need to manage all the variables by yourself (and keep the KeyVault).
See this answser :
https://stackoverflow.com/questions/61469253/azure-devops-pipelines-releases-access-azure-key-vault-secret-nested/73635255#73635255
Related
I am trying to deploy sparkpool (Microsoft.Synapse.workspaces/bigDataPools#2021-03-01)
with the libraryRequirements section via bicep.
https://learn.microsoft.com/en-us/azure/templates/microsoft.synapse/workspaces/bigdatapools?pivots=deployment-language-bicep
libraryRequirements: {
content: 'string'
filename: 'string'
}
My problem is - azurecli returns error that:
Synapse Spark pool must be created before libraries are installed.
So I get that because of some Azure limitation, I would need to have 2 steps and add the libs in the second one.
I am more familiar with terraform than azure, so i tried with dependsOn, which also exists in bicep
pseudocode:
resource sparkpool 'Microsoft.Synapse.workspaces/bigDataPools#2021-03-01'{
}
resource sparkpoolchild 'Microsoft.Synapse.workspaces/bigDataPools#2021-03-01'{
dependsOn: [
sparkpool
]
libraryRequirements: {
content: 'string'
filename: 'string'
}
}
I was hoping that I would be able to add this libs to existing sparkpool resource.
But no luck here, somehow this is not allowed by azure.
Any ideas how to tackle this issue?
I'm trying to define my serverless framework deployment bucket.
My serverless.yml looks like this:
provider:
name: aws
runtime: nodejs14.x
region: us-east-1
stage: dev
deploymentBucket:
name: ${self:environment.DEPLOYMENT_BUCKET}
environment:
${file(../evn.${opt:stage, 'dev'}.json)}
and the evn.dev.json file looks like this:
{
"DEPLOYMENT_BUCKET": "myBucketName"
}
(both of these files have non-relevant parts removed)
I'm getting a "cannot resolve variable at "provicer.deploymentBucket.name" error when trying to deploy.
How do I reference the DEPLOYMENT_BUCKET variable in the serverless.yml file?
EDIT: Other errors:
${environment}:DEPLOYMENT_BUCKET -> Could not locate deployment bucket. Error: The specified bucket is not valid
name: ${environment:DEPLOYMENT_BUCKET}1 -> Unrecognized configuration variable sources: "environment"
name: ${self:provider.environment:DEPLOYMENT_BUCKET}
and
name: ${self:environment:DEPLOYMENT_BUCKET}
-> Cannot resolve serverless.yml: Variables resolution errored with - Cannot resolve variable at "provider.deploymentBucket.name": Value not found at "self" source
I was able to solve the problem with this:
${file(../evn.${opt:stage, 'dev'}.json):DEPLOYMENT_BUCKET}
But 'reading' that file twice -- both here and in the 'environment' area seems to somewhat defeat the purpose of the environments area.
I am trying to configure my spec flow project with the Azure CI pipeline. When I try to create Specflow+LivingDoc with TestExecution.json, the pipeline is unable to find the path. Attaching my YAML, and specflow.json along with this. Can anybody help me with this??
YAML
- task: SpecFlowPlus#0
displayName: 'Upload SpecFlow Living Docs'
inputs:
projectFilePath: 'MyProjecct'
projectName: 'MyProjecct'
testExecutionJson: '**\TestExecution.json'
projectLanguage: 'en'
specflow.json
{
"livingDocGenerator": {
"enabled": true,
"filePath": "{CurrentDirectory}\\TestResults\\TestExecution.json"
}
}
Error
Error
##[error]Error: Command failed: dotnet D:\a_tasks\SpecFlowPlus_32f3fe66-8bfc-476e-8e2c-9b4b59432ffa\0.6.859\CLI\LivingDoc.CLI.dll feature-folder "D:\a\1\s\MyProjecct" --output-type JSON --test-execution-json "**/TestExecution.json" --output "D:\a\1\s\16707\FeatureData.json" --project-name "MyProjecct" --project-language "en"
Before the SpecFlowPlus task you can add a shell task (such as Bask) to execute the ls command to see if the TestExecution.json file has been generated or existing in the specified Feature folder under current directory.
ls -R
If the TestExecution.json file is not existing, you should go to check if there is any issue on the step that generates this file.
I got the same error when I tried to configure this a few days back.
Try giving the full path to your TestExecution.json, that should work. The pattern matching is not working in the file paths so provide full path to your json files as well as project/test assembly etc...
I tried to built github .yml file but I’m getting error like
|GitHub Actions/ Main Workflow
Invalid workflow file
The workflow is not valid. .github/workflows/build.yml (Line: 22, Col: 22): Unexpected symbol: '<hash_value>'. Located at position 9 within expression: secrets.<hash_value>|
CODE
on:
Trigger analysis when pushing in master or pull requests, and when creating
a pull request.
push:
branches:
- master
pull_request:
types: [opened, synchronize, reopened]
name: Main Workflow
jobs:
sonarcloud:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v2
with:
# Disabling shallow clone is recommended for improving relevancy of reporting
fetch-depth: 0
- name: SonarCloud Scan
uses: SonarSource/sonarcloud-github-action#v1.3
env:
GITHUB_TOKEN: {{ secrets.<hash_value>}} SONAR_TOKEN: {{ secrets.<hash_value>}}
AND
#Configure here general information about the environment, such as SonarQube server connection details for example
#No information about specific project should appear here
#----- Default SonarQube server
sonar.host.url=https://sonarcloud.io/
#----- Default source code encoding
#sonar.sourceEncoding=UTF-8
sonar.organization=blah blah
sonar.projectKey=blah blah
— optional properties —
defaults to project key
sonar.projectName=Toolsdemo
defaults to ‘not provided’
sonar.projectVersion=1.0
Path is relative to the sonar-project.properties file. Defaults to .
sonar.sources=https://github.com/abcd/xyz
Encoding of the source code. Default is default system encoding
sonar.sourceEncoding=UTF-8
Please help . I need to test my code very fast .
I am doing the DBT hello world tutorial found here, and I have created my first project on a windows machine. My profiles.yml file looks like this:
my-bigquery-db:
target: dev
outputs:
dev:
type: bigquery
method: service-account
project: abiding-operand-286102
dataset: xxxDBTtestO1
threads: 1
keyfile: C:\Users\xxx\.dbt\abiding-operand-286102-ed9e3c9c13cd.json
timeout_seconds: 300
when I execute dbt run I get:
Running with dbt=0.17.2 Encountered an error while reading profiles: ERROR Runtime Error
dbt encountered an error while trying to read your profiles.yml
file.
Profile my-bigquery-db in profiles.yml is empty
Defined profiles:
my-bigquery-db
target
outputs
dev
type
method
project
dataset
threads
keyfile
timeout_seconds
Any idea?
At first glance from both your code and the source walkthrough, this is just a yml config problem. YML is a markup language that is white-space sensitive. And by just looking at the example that you may have pulled from - it doesn't look appropriately white spaced to me.
I'm not sure if you can simply copy from the below but it might be worth a shot.
my-bigquery-db:
target: dev
outputs:
dev:
type: bigquery
method: service-account
project: abiding-operand-286102
dataset: xxxDBTtestO1
threads: 1
keyfile: C:\Users\xxx\.dbt\abiding-operand-286102-ed9e3c9c13cd.json
timeout_seconds: 300
Basically - your dbt profile.yml needs to be setup with the sections at certain levels (not unlike python indentation or any other white spacing scheme).