I am trying to deploy sparkpool (Microsoft.Synapse.workspaces/bigDataPools#2021-03-01)
with the libraryRequirements section via bicep.
https://learn.microsoft.com/en-us/azure/templates/microsoft.synapse/workspaces/bigdatapools?pivots=deployment-language-bicep
libraryRequirements: {
content: 'string'
filename: 'string'
}
My problem is - azurecli returns error that:
Synapse Spark pool must be created before libraries are installed.
So I get that because of some Azure limitation, I would need to have 2 steps and add the libs in the second one.
I am more familiar with terraform than azure, so i tried with dependsOn, which also exists in bicep
pseudocode:
resource sparkpool 'Microsoft.Synapse.workspaces/bigDataPools#2021-03-01'{
}
resource sparkpoolchild 'Microsoft.Synapse.workspaces/bigDataPools#2021-03-01'{
dependsOn: [
sparkpool
]
libraryRequirements: {
content: 'string'
filename: 'string'
}
}
I was hoping that I would be able to add this libs to existing sparkpool resource.
But no luck here, somehow this is not allowed by azure.
Any ideas how to tackle this issue?
Related
I'm trying to define my serverless framework deployment bucket.
My serverless.yml looks like this:
provider:
name: aws
runtime: nodejs14.x
region: us-east-1
stage: dev
deploymentBucket:
name: ${self:environment.DEPLOYMENT_BUCKET}
environment:
${file(../evn.${opt:stage, 'dev'}.json)}
and the evn.dev.json file looks like this:
{
"DEPLOYMENT_BUCKET": "myBucketName"
}
(both of these files have non-relevant parts removed)
I'm getting a "cannot resolve variable at "provicer.deploymentBucket.name" error when trying to deploy.
How do I reference the DEPLOYMENT_BUCKET variable in the serverless.yml file?
EDIT: Other errors:
${environment}:DEPLOYMENT_BUCKET -> Could not locate deployment bucket. Error: The specified bucket is not valid
name: ${environment:DEPLOYMENT_BUCKET}1 -> Unrecognized configuration variable sources: "environment"
name: ${self:provider.environment:DEPLOYMENT_BUCKET}
and
name: ${self:environment:DEPLOYMENT_BUCKET}
-> Cannot resolve serverless.yml: Variables resolution errored with - Cannot resolve variable at "provider.deploymentBucket.name": Value not found at "self" source
I was able to solve the problem with this:
${file(../evn.${opt:stage, 'dev'}.json):DEPLOYMENT_BUCKET}
But 'reading' that file twice -- both here and in the 'environment' area seems to somewhat defeat the purpose of the environments area.
I have a .Net 6 web application that I am building in devops. I am trying to substitute settings in the appsettings.json, but somehow can't get it working.
What I have:
Azure Key vault set up, with a secret with the name:
Configuration--ConnectionStrings--ConnectionString
Pipelines have access to this Key Vault with a Variable Group
I have tried the transform task in both build and release pipeline
I have tried the
option in the IIS Web app deploy task for substitution
1 - All is fine
2 - All is fine
3 - I have tried this task in build and release with the following configuration:
steps:
- task: FileTransform#1
displayName: 'File Transform: '
inputs:
folderPath: '$(System.DefaultWorkingDirectory)/**/WebAppFront.zip'
fileType: json
targetFiles: '**/appsettings.json'
The logs look OK for transform:
2022-03-17T10:04:32.9753812Z ##[section]Starting: File Transform:
2022-03-17T10:04:33.0157518Z ==============================================================================
2022-03-17T10:04:33.0158091Z Task : File transform
2022-03-17T10:04:33.0158579Z Description : Replace tokens with variable values in XML or JSON configuration files
2022-03-17T10:04:33.0159048Z Version : 1.198.0
2022-03-17T10:04:33.0159390Z Author : Microsoft Corporation
2022-03-17T10:04:33.0159938Z Help : https://learn.microsoft.com/azure/devops/pipelines/tasks/utility/file-transform
2022-03-17T10:04:33.0160522Z ==============================================================================
2022-03-17T10:04:33.6732027Z [command]C:\azagent\A1\_work\_tasks\FileTransform_8ce97e91-56cc-4743-bfab-9a9315be5f27\1.198.0\node_modules\azure-pipelines-tasks-webdeployment-common\7zip\7zip\7z.exe x -oC:\azagent\A1\_work\_temp\temp_web_package_2021667764440822 C:\azagent\A1\_work\r1\a\_Vind\drop\WebAppFront.zip
2022-03-17T10:04:33.7472675Z
2022-03-17T10:04:33.7679746Z 7-Zip [64] 16.00 : Copyright (c) 1999-2016 Igor Pavlov : 2016-05-10
2022-03-17T10:04:33.7687417Z
2022-03-17T10:04:33.7689291Z Scanning the drive for archives:
2022-03-17T10:04:33.7693972Z 1 file, 21535247 bytes (21 MiB)
2022-03-17T10:04:33.7839085Z
2022-03-17T10:04:33.7855780Z Extracting archive: C:\azagent\A1\_work\r1\a\_Vind\drop\WebAppFront.zip
2022-03-17T10:04:36.6203882Z --
2022-03-17T10:04:36.6205010Z Path = C:\azagent\A1\_work\r1\a\_Vind\drop\WebAppFront.zip
2022-03-17T10:04:36.6206577Z Type = zip
2022-03-17T10:04:36.6238261Z Physical Size = 21535247
2022-03-17T10:04:36.6250740Z
2022-03-17T10:04:36.6282342Z Everything is Ok
2022-03-17T10:04:36.6282941Z
2022-03-17T10:04:36.6283536Z Folders: 24
2022-03-17T10:04:36.6284339Z Files: 112
2022-03-17T10:04:36.6284868Z Size: 58919697
2022-03-17T10:04:36.6288304Z Compressed: 21535247
2022-03-17T10:04:36.6338841Z Applying JSON variable substitution for **/appsettings.json
2022-03-17T10:04:36.7353081Z Applying JSON variable substitution for C:\azagent\A1\_work\_temp\temp_web_package_2021667764440822\Content\D_C\a\1\s\Vind\WebAppFront\obj\Release\net6.0\PubTmp\Out\appsettings.json
2022-03-17T10:04:36.7444592Z JSON variable substitution applied successfully.
2022-03-17T10:04:40.1757797Z ##[section]Finishing: File Transform:
And also for the rest of the log like getting keyvault (done before transform):
2022-03-17T10:04:32.0237340Z ##[section]Starting: Download secrets: my-key-vault
2022-03-17T10:04:32.0691326Z ==============================================================================
2022-03-17T10:04:32.0691706Z Task : Azure Key Vault
2022-03-17T10:04:32.0691934Z Description : Download Azure Key Vault secrets
2022-03-17T10:04:32.0692142Z Version : 2.200.0
2022-03-17T10:04:32.0692355Z Author : Microsoft Corporation
2022-03-17T10:04:32.0692657Z Help : https://learn.microsoft.com/azure/devops/pipelines/tasks/deploy/azure-key-vault
2022-03-17T10:04:32.0693010Z ==============================================================================
2022-03-17T10:04:32.7133186Z SubscriptionId: my-subscription-id.
2022-03-17T10:04:32.7145990Z Key vault name: my-key-vault.
2022-03-17T10:04:32.7152879Z Downloading secret value for: Configuration--ConnectionStrings--ConnectionString.
2022-03-17T10:04:32.9707096Z ##[section]Finishing: Download secrets: my-key-vault
The appsettings.json looks like this:
{
"Configuration": {
"ApplicationName": "Lorem ipsum",
"Logging": {
"LogLevel": {
"Default": "Information",
"Microsoft": "Warning",
"Microsoft.Hosting.Lifetime": "Information"
}
},
"ConnectionStrings": {
"ConnectionString": ""
}
}
}
But the connectionstring stays empty. As I understood from reading a lot of blogs and documentation this should be working. The double hyphen -- in the key name, should translate to a dot, so Json path expression is correct (I think): Configuration.ConnectionStrings.ConnectionString
The IIS Web app deploy task, with transform gives similar result. So an empty connectionstring. Even though it explicitly states it updated the json in the zippackage.
Logs for that task:
2022-03-17T10:04:43.3419710Z ##[section]Starting: IIS Web App Deploy
2022-03-17T10:04:43.3940659Z ==============================================================================
2022-03-17T10:04:43.3941018Z Task : IIS web app deploy
2022-03-17T10:04:43.3941276Z Description : Deploy a website or web application using Web Deploy
2022-03-17T10:04:43.3941526Z Version : 0.198.0
2022-03-17T10:04:43.3941721Z Author : Microsoft Corporation
2022-03-17T10:04:43.3942065Z Help : https://learn.microsoft.com/azure/devops/pipelines/tasks/deploy/iis-web-app-deployment-on-machine-group
2022-03-17T10:04:43.3943670Z ==============================================================================
2022-03-17T10:04:44.0803824Z [command]C:\azagent\A1\_work\_tasks\IISWebAppDeploymentOnMachineGroup_1b467810-6725-4b6d-accd-886174c09bba\0.198.0\node_modules\azure-pipelines-tasks-webdeployment-common\7zip\7zip\7z.exe x -oC:\azagent\A1\_work\_temp\temp_web_package_39200019901712446 C:\azagent\A1\_work\r1\a\_Vind\drop\WebAppFront.zip
2022-03-17T10:04:44.1320582Z
2022-03-17T10:04:44.1365755Z 7-Zip [64] 16.00 : Copyright (c) 1999-2016 Igor Pavlov : 2016-05-10
2022-03-17T10:04:44.1393221Z
2022-03-17T10:04:44.1427445Z Scanning the drive for archives:
2022-03-17T10:04:44.1436856Z 1 file, 23620753 bytes (23 MiB)
2022-03-17T10:04:44.1451518Z
2022-03-17T10:04:44.1587844Z Extracting archive: C:\azagent\A1\_work\r1\a\_Vind\drop\WebAppFront.zip
2022-03-17T10:04:47.1556142Z --
2022-03-17T10:04:47.1556846Z Path = C:\azagent\A1\_work\r1\a\_Vind\drop\WebAppFront.zip
2022-03-17T10:04:47.1558328Z Type = zip
2022-03-17T10:04:47.1558787Z Physical Size = 23620753
2022-03-17T10:04:47.1559027Z
2022-03-17T10:04:47.1566021Z Everything is Ok
2022-03-17T10:04:47.1566315Z
2022-03-17T10:04:47.1566617Z Folders: 35
2022-03-17T10:04:47.1567095Z Files: 112
2022-03-17T10:04:47.1567474Z Size: 58919916
2022-03-17T10:04:47.1567818Z Compressed: 23620753
2022-03-17T10:04:47.2311338Z Applying JSON variable substitution for **/appsettings.json
2022-03-17T10:04:47.3655684Z Applying JSON variable substitution for C:\azagent\A1\_work\_temp\temp_web_package_39200019901712446\Content\D_C\a\1\s\Vind\WebAppFront\obj\Release\net6.0\PubTmp\Out\appsettings.json
2022-03-17T10:04:47.3751003Z JSON variable substitution applied successfully.
2022-03-17T10:04:50.9446139Z [command]"C:\azagent\A1\_work\_tasks\IISWebAppDeploymentOnMachineGroup_1b467810-6725-4b6d-accd-886174c09bba\0.198.0\node_modules\azure-pipelines-tasks-webdeployment-common\MSDeploy3.6\MSDeploy3.6\msdeploy.exe" -verb:sync -source:package='C:\azagent\A1\_work\r1\a\temp_web_package_8577780759906015.zip' -dest:auto -setParam:name='IIS Web Application Name',value='www.mydomain.com' -enableRule:DoNotDeleteRule
2022-03-17T10:04:51.7358688Z Info: Updating file (www.mydoain.com\appsettings.json).
2022-03-17T10:04:51.7839580Z Total changes: 1 (0 added, 0 deleted, 1 updated, 0 parameters changed, 2281 bytes copied)
2022-03-17T10:04:51.8207636Z ##[section]Finishing: IIS Web App Deploy
What am I doing wrong? The only thing I noticed when I was echoing out the variable it got cut off on a ; in the connectionstring, but even when I tried some simple value it was not working.
I ended up solving it by creating variables in the pipelines and using the Key Vault secrets as the value.
Keyvault secret name: 'MyConnectionSecret'
You have to use the dot notation to specify a path. See my question for the json structure.
Name of var in Dev ops:
Configuration.ConnectionStrings.ConnectionString
Value of var in Dev ops:
$(MyConnectionSecret)
And of course in the Library you have to add the variable group based on the Keyvault
I managed to resolve this issue automatically without the need to manage all the variables by yourself (and keep the KeyVault).
See this answser :
https://stackoverflow.com/questions/61469253/azure-devops-pipelines-releases-access-azure-key-vault-secret-nested/73635255#73635255
I'm trying to use a dependency block for the first time, but get aws s3 list object permission denied issues and have trouble debugging the issue.
The setup is as follows, using an s3 backend for storing terraform state:
A git repo containing the terraform modules:
archive
s3_inventory
Instantiations of the above:
prod/eu/archive/terragrunt.hcl:
terraform {
source = "git::ssh://git#my_server//archive?ref=v1.0.0"
}
include {
path = find_in_parent_folders()
}
dependency "s3-inventory" {
config_path = "../s3-inventory/"
}
prod/eu/s3_inventory/terragrunt.hcl:
terraform {
source = "git::ssh://git#my_server//s3_inventory?ref=v1.0.0"
}
include {
path = find_in_parent_folders()
}
Running terragrunt apply in prod/eu/archive works just fine when I remove the dependency block from the hcl file. It fails when I add the dependency block in.
Running terragrunt output -json in prod/eu/s3-inventory also works just fine.
With debugging flags on I still don't seem to get enough info as to why it's failing.
terragrunt apply --terragrunt-log-level debug --terragrunt-debug in prod/eu/archive results in something like this:
...<omitted>...
DEBU[0000] Detected module /Users/tim.kersten/prod/eu/s3-inventory/terragrunt.hcl is already init-ed. Retrieving outputs directly from working directory. prefix=[/Users/tim.kersten/prod/eu/s3-inventory]
DEBU[0000] Running command: terraform output -json prefix=[/Users/tim.kersten/prod/eu/s3-inventory]
Failed to load state: AccessDenied: Access Denied
status code: 403, request id: ABC123DEF456GHI, host id: WW91J3JlIHRlcnJpYmx5IG5vc2UgZm9yIHRyeWluZyB0byBsb29rIGF0IG15IGhvc3QK
ERRO[0003] exit status 1
Something is clearly different, but the debugging options I set on terragrunt don't seem to give me enough info to understand what's different.
Anyone understand what's going on here?
Edit:
terragrunt version: 0.28.6
So I'm pretty new to CloudFormation and also to Serverless framework. I've been trying to work through some exercises (such as an automatic thumbnail generator) and then create some simple projects that I can hopefully generalize for my own purposes.
Right now I'm attempting create a stack/function that creates two S3 buckets and has the Lambda Function take a CSV file form one, perform some simple transformations, and place it in the other receiving bucket.
In trying to build off the exercise I've done, I created a Yaml file with the following code:
provider:
name: aws
runtime: python3.8
region: us-east-1
profile: serverless-admin
timeout: 10
memorySize: 128
iamRoleStatements:
- Effect: "Allow"
Action:
- "s3:*"
Resource: "*"
custom:
assets:
targets:
- bucket1: csvbucket1-08-16-2020
pythonRequirements:
dockerizePip: true
- bucket2: csvbucket2-08-16-2020
pythonRequirements:
dockerizePip: true
functions:
protomodel-readcsv:
handler: handler.readindata
events:
s3:
- bucket: ${self:custom.bucket1}
event: s3:ObjectCreated:*
suffix: .csv
- bucket: ${self:custom.bucket2}
plugins:
- serverless-python-requirements
- serverless-s3-deploy
However, when i do a Serverless deploy from my command prompt, I get:
Serverless Warning --------------------------------------
A valid service attribute to satisfy the declaration 'self:custom.bucket1' could not be found.
Serverless Warning --------------------------------------
A valid service attribute to satisfy the declaration 'self:custom.bucket2' could not be found.
Serverless Error ---------------------------------------
Events for "protomodel-readcsv" must be an array, not an object
I've tried to make the events object in the protohandler-readcsv: by adding a - but I then get a bad indentation error that for some reason I cannot reconcile. But, more fundamentally, I'm not exactly sure why that item would need be an array anyway, and I wasn't clear about the warnings with the buckets either.
So sorry about a pretty newbie question about this, but running tutorials/examples online leaves a lot to try to figure out in trying to generalize/customize these examples.
custom:
assets:
targets:
- bucket1
I guess you need self:custom.assets.targets.bucket1, not sure if this nested assets will work.
Please check the example below is supposed to work.
service: MyService
custom:
deploymentBucket: s3_my_bucket
provider:
name: aws
deploymentBucket: ${self:custom.deploymentBucket}
stage: dev
When I create an "IoT device configuration" for deployment to multiple edge devices I can see the deployment "Configuration Details" which look ok.
However when i actually go to make a deployment it appears to set the schema of the $edgeAgent to 1,0 not 1.0 (note the comma)
eg:
"modulesContent": {
"$edgeAgent": {
"properties.desired": {
"modules": {},
"runtime": {
"settings": {
"minDockerVersion": "v1.25"
},
"type": "docker"
},
"schemaVersion": "1,0",
on the documentation it says that the schemaVersion: Has to be "1.0"
https://learn.microsoft.com/en-us/azure/iot-edge/module-edgeagent-edgehub
When I deploy the device configuration it seems to continually come back with an IoTEdge status of
412 -- The deployment configuration schema version is invalid
I cant seem to find where I can change the property to be the correct value
Any ideas?
Sorry for the inconvenience caused, this is now fixed. Please continue/try creating New IoT Edge deployment from Azure portal-->IoT Hub and leave a comment here if you need further help on this matter. Thanks for your patience.