How to find hot or cool files in blob container - azure-storage

Is there some way or script to search in your blob container which files are hot or cool to change the to archive?
I have thousands of folders and files and to make this work manually is a nightmare

If you want to change the blob tier(hot or cool) to archive tier, there is a built-in feature named lifecycle management.
You can just set a rule for your storage account(the rule can be applied for container level or account level or subfolder level as per your need), then the blob service can automatically change the tier(hot and cool) to archive.
Here is an example for container level:
1.Nav to azure portal -> your storage account -> lifecycle management, then click "Add a rule":
In the Details panel -> Specify a "rule name", select "Rule scope"(here, select "Limit blobs with filter" for container level), "Blob type" and "Blob subtype":
3.In the "Base blobs", specify the settings as below:
4.In "Filter set", just type your container name for Prefix match:
5.Click "Add" button to save the rule. Note that the rule will be executed after 24 hours.

You could change access tier with Powershell.
#Initialize the following with your resource group, storage account, container, and blob names
$rgName = ""
$accountName = ""
$containerName = ""
#Select the storage account and get the context
$storageAccount = Get-AzStorageAccount -ResourceGroupName $rgName -Name $accountName
$ctx = $storageAccount.Context
#list the blobs in a container
$blobs = Get-AzStorageBlob -Container $containerName -Context $ctx
foreach($blob in $blobs)
{
#if tier not equal "Archive"
if($blob.AccessTier -ne "Archive"){
#Change the blob’s access tier to archive
$blob.ICloudBlob.SetStandardBlobTier("Archive")
}
}
Another method uses the BlobBatch.SetBlobAccessTier Method SDK in .Net.
// Get a connection string to our Azure Storage account.
string connectionString = "<connection_string>";
string containerName = "sample-container";
// Get a reference to a container named "sample-container" and then create it
BlobServiceClient service = new BlobServiceClient(connectionString);
BlobContainerClient container = service.GetBlobContainerClient(containerName);
container.Create();
// Create three blobs named "foo", "bar", and "baz"
BlobClient foo = container.GetBlobClient("foo");
BlobClient bar = container.GetBlobClient("bar");
BlobClient baz = container.GetBlobClient("baz");
foo.Upload(new MemoryStream(Encoding.UTF8.GetBytes("Foo!")));
bar.Upload(new MemoryStream(Encoding.UTF8.GetBytes("Bar!")));
baz.Upload(new MemoryStream(Encoding.UTF8.GetBytes("Baz!")));
// Set the access tier for all three blobs at once
BlobBatchClient batch = service.GetBlobBatchClient();
batch.SetBlobsAccessTier(new Uri[] { foo.Uri, bar.Uri, baz.Uri }, AccessTier.Archive);

Related

Terraform for_each on custom registry

We have a custom VPC registry built within our organization. The objective is to enable user of registry to be able to create VPC, create multiple public and private subnets for multiple availability zones within VPC.
I have a for_each variable set at
variable az_sub {list(object(
az = string,
public_cidr_block = list,
private_cidr_block =list
))}
Variable has a value of
[{az='us-east-1a",
public_cidr_block =[list of cidr],
private_cidr_block= [list of cidr]
},
az='us-east-1b",
public_cidr_block =[list of cidr],
private_cidr_block= [list of cidr} ]
When I set for_each on this within registry,
module "az"{
source="./modules/az"
vpc_id = module.vpc.vpc_id
for_each = toset(keys({for i,v in var.az_sub: i => v}))
availability_zone = var.az_sub[each.value]["az"]
public_cidr_block = var.az_sub[each.value]["public_cidr_block"]
private_cidr_block =var.az_sub[each.value]["private_cidr_block"]
}
I get unsupported attribute error on terraform/modules/modReg/output.tf in output public_subnet_ids
module.az is object with 2 attributes
output.tf has the outputs defined.
The az variables.tf has all three variables defined.
Also Note, if I replace for_each with below, do not get errors, but objects are successfully created on aws
availability_zone="us-east-1a"
public_cidr_block = ["10.97.224.0/22"]
private_cidr_block =["10.97.228.0/22"]
So doubt its issue with az module
Note, if I use only 1 az zone on the variable value for list(object) I the error changes to
module.az is object with 1 attribute "us-east-1"
output "private_subnet_ids" {
description = "List of private subnet IDs"
value = values(module.az).*.private_subnet_ids
}
For anyone having same issue, this resolved the problem. I had used list(object and not map(object. If map(object was used, then output.tf would need to have map values.
If you use for_each to send values to the module, the outputs would need to be defined as shown above.

How to generate SAS token using Access policy for a container of ADLS gen 2

How to generate SAS token using Access policy for a folder in container of ADLS gen 2.
exactly like below image but for ADLS gen 2 containers or folders. thank you in advance.
To generate SAS token using Access policy on ADLS containers need to create a Access Policy first . You can create Access Policy through Azure portal (Please Check with this link) or Storage Explorer.
Based on your attached
Screenshot you are using the Microsoft Storage Explorer so here are steps create access policy
1)Go to your container --> right click on container
2)Select the manage access policy
3)Click on the add. There you can provide the Access policy id and permissions you need to give on container like read ,write (click on check boxes).And click on save
4)Once access policy created. You can create the SAS based on that access policy .Right click on
The container select Get Share Access Signature. From the dropdown select the access policy and click
On the create
Generate SAS using terraform
terraform {
required_providers {
azurerm = {
source = "hashicorp/azurerm"
version = "~> 2.65" }
}
required_version = ">= 0.14.9"
}
provider "azurerm" {
features {}
}
resource "azurerm_resource_group" "rg" {
name = "terraformtest"
location = "West Europe"
}
resource "azurerm_storage_account" "storage" {
name = "storage name"
resource_group_name = azurerm_resource_group.rg.name
location = azurerm_resource_group.rg.location
account_tier = "Standard"
account_replication_type = "GRS"
allow_blob_public_access = true
}
resource "azurerm_storage_container" "container" {
name = "terraformcont"
storage_account_name = azurerm_storage_account.storage.name
container_access_type = "private"
}
data "azurerm_storage_account_blob_container_sas" "example" {
connection_string = azurerm_storage_account.storage.primary_connection_string
container_name = azurerm_storage_container.container.name
https_only = true
start = "Date"
expiry = "Date"permissions {
read = true
add = true
create = false
write = false
delete = true
list = true
}
}
output "sas_url_query_string" {
value = data.azurerm_storage_account_blob_container_sas.example.sas
sensitive = true
}
After running the above command you will get output inside terraform.tfstate
For more information check with this link

Terraform update access policies

I am facing a problem and workflow related to terraform to automate the creation of storage account, key vaults and access policies.
What I am trying to achieve is as follow:
I have a storage-account that runs with a for_each loop:
//==================================================
// Automation storage accounts
//==================================================
resource "azurerm_storage_account" "storage-foreach" {
for_each = var.storage-foreach
access_tier = "Hot"
account_kind = "StorageV2"
account_replication_type = "LRS"
account_tier = "Standard"
location = var.location
name = each.value
resource_group_name = azurerm_resource_group.tenant-testing-hamza.name
depends_on = [azurerm_key_vault_key.client-key]
identity {
type = "SystemAssigned"
}
lifecycle {
prevent_destroy = false
}
}
this storage account resource, loops through this variable to create the storage accounts
variable "storage-foreach" {
type = map(string)
default = { "storage1" = "storage1", "storage2" = "storage2", "storage3" = "storage3", "storage4" = "storage4"}
}
so far everything works smoothly. Than I wanted to add those storage accounts object id to my key vault access policy, as follow:
resource "azurerm_key_vault_access_policy" "storage" {
for_each = var.storage-foreach
key_vault_id = azurerm_key_vault.tenantsnbshared.id
tenant_id = "<tenant-id"
object_id = azurerm_storage_account.storage-foreach[each.key].identity.0.principal_id
key_permissions = ["get", "Create", "List", "Restore", "Recover", "Unwrapkey", "Wrapkey", "Purge", "Encrypt", "Decrypt", "Sign", "Verify"]
secret_permissions = ["get", "set", "list", "delete", "recover"]
}
so far everything works just fine while creating the resource, I have all the access policies in place. But, If I try to remove, for example the storage1 from my variable, the storage account get deleted and the access policies related to that specific storage, which is good.
And here the main issue I am facing. If I try to add again the same storage in the variable and run a terraform apply , what happen is that the 3 policies still existing they get removed and the access policy for the storage account get created. If I do one more time terraform apply the logic get inverted, it will delete the first storage account access policy and add the other 3.
I can't find a solution to just update my access policies accordingly to the element I have set in my variable.

Automatically create new connection strings for web applications Azure

I am attempting to automate a workflow in our Azure environment.
We have several web applications with connectionstrings to several databases. Each new customer recives a new database.
I've hit a snag in the script with our connectionstrings. I want the script to update all web applications and add a new connectionstring for the newly created customer db.
The problem is "Set-Azurermwebapp -Name -ResourceGroup -ConnectionStrings" takes a hashtable which replaces any previously configured data.
I would only like to append a new connectionstring, or get the previously configered cstrings and add them to an array, then replacing all data.
Example code;
$test= #{"Type"="Custom"; "Value" = "TestValue"}
$Connectionstring=#{"test"=$test }
Set-AzureRmWebApp
-Name "testapp"
-ResourceGroupName "testgrp"
-ConnectionStrings $Connectionstring"
Any ideas here?
$connStrings = #{
AzureWebJobsDashboard = #{
Type = "Custom";
Value = $AzureWebJobsDashboard 
};
AzureWebJobsStorage = #{
Type = "MySql";
Value = $connstring
}
};
Set-AzureRMWebApp -Name $webServiceName -ResourceGroupName $rgName -ConnectionStrings $connStrings
Cannot Delete All Azure Website Connection Strings
#Add new connection string
$newConnString = New-Object Microsoft.WindowsAzure.Commands.Utilities.Websites.Services.WebEntities.ConnStringInfo
$newConnString.Name = $ConnStringName
$newConnString.ConnectionString = $ConnStringValue
$newConnString.Type = $ConnStringType
$connStrings.Add($newConnString)
Set-AzureWebsite $WebAppName -ConnectionStrings $connStrings
You can download detail script from How to automatically create new connection strings for web applications Azure

Setting Sharepoint File Field Attributes

In out SP site, we have a library with files. These are files associated with a user. We now cstomized the user's profiles to accept a list of files. And now, to this list of files in the user's profile, we would like to add a reference to the file so that the user doesn't have to upload again.
Current Library:
/personal/my/User Files/[filename]
So, I was wondering how to do this? The data looks like this in the new User Files field (JSON):
{
[
{
"Id":"1",
"Title":"Test",
"Url":"\/personal\/my\/User+Files\/testfile.doc"
}
]
}
I have a csv file that I iterate over. The csv file contains the user name:filename pairs.
The Id value has to be gotten from the SP instance libarary at that location for that file.
Powershell code:
$upAttribute = "UserFiles"
$profile_info = Import-Csv profiles.csv
foreach ($row in $profile_info) {
$userId = $row.user # User ID
$fullAdAccount = $adAccount += $userId
#Check to see if user profile exists
if ($profileManager.UserExists($fullAdAccount))
{
$up = $profileManager.GetUserProfile($fullAdAccount)
$upAttributeValue += $row.filename # Filename
# CODE ??????
$up.Commit()
}
}
That is the all the data that I have.
Thanks for any and all help.
Eric
You will first need to add the custom property to the User Profile like so:
http://www.paulgrimley.com/2011/02/adding-custom-user-profile-property-to.html
Then this should help you out:
http://get-spscripts.com/2010/07/modify-single-value-user-profile.html
#Get user profile and change the value
$up = $profileManager.GetUserProfile($adAccount)
$up[$upAttribute].Value = $upAttributeValue
$up.Commit()