I have created a PowerShell Script which creates a new version of User Managed Encryption Key in Azure Key Vault and update the Data Lake Store Encryption Key to the new version. Below is the script
$subscriptionID = 'XXXX'
$keyVaultName = 'XXXX'
$keyName = 'XXXX'
$adlsName = 'XXXX'
Login-AzureRmAccount
Select-AzureRmSubscription -SubscriptionId $subscriptionID
$azureKVKey = Add-AzureKeyVaultKey -VaultName $keyVaultName -Name $keyName -Destination Software
$adlsKVVersion = $azureKVKey.Version
Set-AzureRmDataLakeStoreAccount -Name $adlsName -KeyVersion $adlsKVVersion
(Get-AzureRmDataLakeStoreAccount -Name $adlsName).Properties.EncryptionConfig.KeyVaultMetaInfo.EncryptionKeyVersion
The script successfully creates a new version of the key but the ADLS encryption key version does not change.
I debugged it and I can also see that the the PS command sends a request for updating the encryption key version
The response of the PS script still shows the old version of the encryption key
Any hints as to what might be going wrong?
Thanks
Related
I've googled this for hours and found little code snippets in bash and powershell but the ones that don't error completely end up giving me a 403. Here's my script this far.
$resourceGroupName = "MyResourceGroup"
$storageAccountName = "MyStorageAcc"
$containerName = "mynewcontainer"
$storageAccKey = (Get-AzStorageAccountKey -ResourceGroupName $resourceGroupName -AccountName $storageAccountName)[0].value
$storagecontext = New-AzStorageContext -StorageAccountName $storageAccountName -StorageAccountKey $storageAccKey
New-AzureStorageContainer -Name $containerName -Permission Container -Context $storagecontext
I originally didn't get the storage account key, but after that gave me a 403 I started looking into potential reasons for the error and many suggested getting the storage account key. However this still doesn't work. What am I missing?
Update
I found this: https://github.com/Azure/azure-cli/issues/9396, which suggests it's an issue with the ip address you're connecting from. However, I've added my ip address and still get the same issue.
Update 2
I couldn't find a sufficient way to do this through a script so decided to do it as part of the initial creation using the ARM template.
Make sure your user account login the Azure powershell(if using cloud shell, the account is which you login the portal) has the correct RBAC role e.g. Storage Account Contributor at your storage account/group/subscription, if not, follow this to add it.
Besides, you mixed the old AzureRM command New-AzureStorageContainer with the new Az command Get-AzStorageAccountKey, New-AzStorageContext, it may cause you to use a wrong context, change the last line as below.
New-AzStorageContainer -Name $containerName -Permission Container -Context $storagecontext
I have a Gen2 Azure storage account and try to create a stored access policy on a container using Powershell. I am signed into the account and the relevant subscription is selected. I save the context in a variable for further use by the following statements.
Connect-AzAccount
Set-AzContext -Subscription "<subscriptionid>"
$context = New-AzStorageContext -StorageAccountName "MyStorageAccount" -UseConnectedAccount
Creating a stored access policy failed so I added one manually through the portal and tried to get a list of the policies:
Get-AzStorageContainerStoredAccessPolicy -Container "MyContainer" -Context $context
This resulted in an error Get-AzStorageContainerStoredAccessPolicy: The specified resource does not exist., the same error that the "New-AzStorageContainerStoredAccessPolicy" command yielded. Adding the name of the existing policy with the -Policy parameter did not change the outcome, neither did changing the access level on the container.
I think I can also rule out typos as Get-AzStorageContainer "MyContainer" -Context $context gives me the details of the container as expected.
I am unclear as to what resource it is that does not exists, as the container clearly exists and it also contains at least one stored access policy. Can the container stored access policy command not be used on a Gen2 storage account or am I missing something else?
This is because Get/New-AzStorageContainerStoredAccessPolicy isn't supported by Oauth. You can find all operations supported by Oauth https://learn.microsoft.com/en-us/azure/role-based-access-control/resource-provider-operations#microsoftstorage.
As a workaround, you can use connection string to do this:
Connect-AzAccount
Set-AzContext -Subscription "<subscriptionid>"
$context = New-AzStorageContext -ConnectionString "<Connection string>"
Get-AzStorageContainerStoredAccessPolicy -Container "<Container Name>" -Context $context
Same issue in Github: https://github.com/Azure/azure-powershell/issues/10391.
I need to add AD administrator to Azure SQL Managed Instances through PowerShell in order to automate deployments.
But it seems there's no way to do it with Azure PowerShell or the REST APIs.
So far I've been trying to set it up like a normal SQL Server.
$sql = Get-AzureRmResource -ResourceGroupName "RSGName" -Name "InstanceName"
-ResourceType "Microsoft.Sql/managedInstances" -ExpandProperties
$dbaId = Get-AzureRmADGroup -DisplayName "ADGroupName" | Select-Object Id
Set-AzureRmSqlServerActiveDirectoryAdministrator -DisplayName "ADGroupName"
-ResourceGroupName "RSGName" -ServerName "InstanceName" -ObjectId $dbaId.Id
But it is giving me errors saying the Server cannot be found on the resource group.
I am exploring VSTS Release Management and I wanted to backup my production database hosted on Azure SQL DB before I apply any migration scripts to it. I fail to find any particular task or preferred way of waiting till the Azure SQL DB is fully backed up so that I can proceed with deployment only after the database is correctly backed up.
I have looked at either using a PowerShell task or Azure SQL CMD task, but I am not sure how to make rest of the tasks wait for the backup to complete.
Would appreciate if anyone could point me in the right direction. Thanks.
You can backup Azure SQL database and check the status in a loop.
$exportRequest = New-AzureRmSqlDatabaseExport -ResourceGroupName $ResourceGroupName -ServerName $ServerName `
-DatabaseName $DatabaseName -StorageKeytype $StorageKeytype -StorageKey $StorageKey -StorageUri $BacpacUri `
-AdministratorLogin $creds.UserName -AdministratorLoginPassword $creds.Password
$importStatus = Get-AzureRmSqlDatabaseImportExportStatus -OperationStatusLink $importRequest.OperationStatusLink
[Console]::Write("Exporting")
while ($importStatus.Status -eq "InProgress")
{
$importStatus = Get-AzureRmSqlDatabaseImportExportStatus -OperationStatusLink $importRequest.OperationStatusLink
[Console]::Write(".")
Start-Sleep -s 10
}
[Console]::WriteLine("")
$importStatus
More information, you can refer to Export an Azure SQL database to a BACPAC file.
Another way is that you can backup Azure SQL database by call Microsoft.SqlServer.Dac.DacServices.ExportBacpac method with PowerShell.
param([string]$ConnectionString, [string]$DatabaseName,[string]$OutputFile,[string]$s)
Add-Type -Path "$s\AzureDatabaseSolution\SQLDatabase\lib\Microsoft.SqlServer.Dac.dll"
$now = $(Get-Date).ToString("HH:mm:ss")
$Services = new-object Microsoft.SqlServer.Dac.DacServices $ConnectionString
Write-Host "Starting at $now"
$Watch = New-Object System.Diagnostics.StopWatch
$Watch.Start()
$Services.ExportBacpac($OutputFile, $DatabaseName)
$Watch.Stop()
Write-Host "Backup completed in" $Watch.Elapsed.ToString()
Note: Using the assembly in this package: Microsoft.SqlServer.Dac 1.0.3 (I add it to the source control and map to build agent)
On the other hand, to add firewall rule, you can refer to this thread: Deploy Dacpac packages via power shell script to Azure SQL Server.
BTW, you can build the custom build/release step/task with these PowerShell scripts. Add a build task
Azure SQL Databases are continually backed up automatically. If you are trying to create a copy of the database or archive the database to a BACPAC file, you can do either.
https://learn.microsoft.com/en-us/azure/sql-database/sql-database-automated-backups
https://learn.microsoft.com/en-us/azure/sql-database/sql-database-copy
https://learn.microsoft.com/en-us/azure/sql-database/sql-database-export
I am trying to create a VSTS task, which should create an AD application.
Taken the DeployAzureResouceGroup as a sample, I have created to following script:
[CmdletBinding()]
param()
Trace-VstsEnteringInvocation $MyInvocation
Import-VstsLocStrings "$PSScriptRoot\Task.json"
$connectedServiceNameSelector = Get-VstsInput -Name "connectedServiceNameSelector" -Require
$connectedServiceName = Get-VstsInput -Name "connectedServiceName"
$connectedServiceNameClassic = Get-VstsInput -Name "connectedServiceNameClassic"
$domains = (Get-VstsInput -Name "domains").Split(";")
$appName = Get-VstsInput -Name "appName"
if($connectedServiceNameSelector -eq "ConnectedServiceNameClassic")
{
$connectedServiceName = $connectedServiceNameClassic
$action = $actionClassic
$resourceGroupName = $cloudService
}
Import-Module $PSScriptRoot\ps_modules\VstsAzureHelpers_
Initialize-Azure
# Import the loc strings.
Import-VstsLocStrings -LiteralPath $PSScriptRoot/Task.json
# Import all the dlls and modules which have cmdlets we need
Import-Module "$PSScriptRoot\DeploymentUtilities\Microsoft.TeamFoundation.DistributedTask.Task.Deployment.Internal.psm1"
Import-Module "$PSScriptRoot\DeploymentUtilities\Microsoft.TeamFoundation.DistributedTask.Task.Deployment.dll"
# Load all dependent files for execution
. "$PSScriptRoot\Utility.ps1"
try
{
Validate-AzurePowerShellVersion
$azureUtility = Get-AzureUtility "$connectedServiceName"
Write-Verbose "Loading $azureUtility"
. "$PSScriptRoot\$azureUtility"
Write-Output "test"
Write-Output "Creating a new Application in AAD (App URI -)" -Verbose
$azureAdApplication = New-AzureRmADApplication -DisplayName "test" -IdentifierUris "https://app.com" -HomePage "https://app.com"
$appId = $azureAdApplication.ApplicationId
Write-Output "Azure AAD Application creation completed successfully (Application Id: $appId)" -Verbose
Write-Verbose "Completing Azure Resource Group Deployment Task" -Verbose
}
catch
{
Write-TaskSpecificTelemetry "UNKNOWNDEP_Error"
throw
}
When I use a Service principal as Service Endpoint user, I got the error Resource me not found.
When I use my custom AD account, I got the error:Run Login-AzureRmAccount to login.
What am I doing wrong? How can I get this script working?
If you don't need Powershell scripting, go install Azure AD Application Management extension from https://marketplace.visualstudio.com/items?itemName=RalphJansen.Azure-AD-Application-Management
You can add new tasks from pipeline GUI for managing AD applications.
If you do need Powershell scripting, then things get tricky.
Get Powershell code from https://stackoverflow.com/a/51848069/1548275 as a base. The difference is, that if you're not running your code from an extension, you don't have Get-VstsInput nor Get-VstsEndpoint available to execute.
Also, you don't have AzureAD module cmdlets to run. You need to get the Nuget-package, unzip it to your own repo and have it as part of your scripts to be later Import-Module in a pipeline task.
Finally, you need an auth token for Graph API. As the extension code shows, you will need 3 variables:
$tenantId = (Get-AzureRmSubscription).TenantId
$clientId = (Get-AzureRmADServicePrincipal -DisplayName "Your Project Service Connection name from Azure AD App Registrations").ApplicationId.Guid
$clientSecret = 'hard-coded, reset SPN password'
As you can see, an extension would have access to all three, but regular script (to my knowledge) doesn't.
SPN password reset is covered in The Net. Briefly, it is something like this:
$clientId = (Get-AzureRmADServicePrincipal -DisplayName "Your Project Service Connection name from Azure AD App Registrations").Id.Guid
$password = ConvertTo-SecureString –asplaintext –force "oh, this is very secret!"
New-AzureRmADSpCredential -ObjectId $clientId -Password $password
Also: Update the plaintext password into Azure DevOps project settings, Service Connections for Pipeline to know about the update.