I have been struggling a lot to find Microsoft documentation on Azure Synapse Analytics to provision Managed Private Endpoints using powershell or CLI?
I can easily do it thru Azure Portal (basically thru Studio) but we want to do it as part of our Azure DevOps pipeline and provision the Managed Private endpoints (not the customer private endpoints) thru Powershell/CLI.
Any help?
Thank you
In the updated PowerShell module, there is now a powershell commandlet available: New-AzSynapseManagedPrivateEndpoint.
This should be the way to go. If you don't have PowerShell available, you can still use this CLI approach.
Additionally we have done the following to approve the Managed Endpoint connection. You need to have permissions on the resource that you are connecting to, if you want to be able to approve the Endpoint connection.
# list all private endpoints
endpointList = Get-AzPrivateEndpointConnection -PrivateLinkResourceId '/subscriptions/<xxx>/resourceGroups/<xxx>/providers/Microsoft.Storage/storageAccounts/<xxx>'
# filter the synapse endpoint
$synpapseEndpoint = $endpointList | Where-Object { $_.PrivateEndpoint.Id.EndsWith('<yourworkspacename>.<yourmanagedenpointname>') }
Approve-AzPrivateEndpointConnection -ResourceId $synpapseEndpoint.Id
Currently we don’t have native Synapse PowerShell and CLI commands for creating managed private endpoints. There is a workaround to create private endpoints through Azure CLI. The sample code is as below.
Setup
$workspaceName = "your-workspace-name"
$synapseAnalyticsResourceId = "https://dev.azuresynapse.net"
$managedVirtualNetworkName = "your-managed-virtual-network-name"
$newEndpointName = "your-new-endpoint-name"
To list private endpoints
az rest --resource $synapseAnalyticsResourceId --method GET --url "https://$($workspaceName).dev.azuresynapse.net/managedVirtualNetworks/$($managedVirtualNetworkName)/managedPrivateEndpoints?api-version=2019-06-01-preview"
To create private endpoints
Doc for this API
# GroupId in the body is specifically for a Data Lake endpoint (Storage with Hierarchical namespace enabled)
$body = "{""properties"":{""privateLinkResourceId"": ""<resourceid-to-your-resource-for-endpoint>"", ""groupId"": ""dfs""}}"
$body | Out-File body.json
az rest --resource $synapseAnalyticsResourceId --method PUT --headers "Content-Type=application/json" --url "https://$($workspaceName).dev.azuresynapse.net/managedVirtualNetworks/$($managedVirtualNetworkName)/$($newEndpointName)?api-version=2019-06-01-preview" --body `#body.json
Remove-Item body.json
Related
I am working on a databricks notebook running with ADLSV2 using service
priciple id but receive the following error after mounting my drive.
StatusCode=403
StatusDescription=This request is not authorized to perform this operation using this permission.
configs = {"dfs.adls.oauth2.access.token.provider.type":
"ClientCredential",
"dfs.adls.oauth2.client.id": "78jkj56-2ght-2345-3453-b497jhgj7587",
"dfs.adls.oauth2.credential": dbutils.secrets.get(scope =
"DBRScope", key = "AKVsecret"),
"dfs.adls.oauth2.refresh.url":
"https://login.microsoftonline.com/bdef8a20-aaac-4f80-b3a0-
d9a32f99fd33/oauth2/token"}
dbutils.fs.mount(source =
"adl://<accountname>.azuredatalakestore.net/tempfile",mount_point =
"/mnt/tempfile",extra_configs = configs)
%fs ls mnt/tempfile
The uri for your lake is a gen1 uri not gen2. Either way your service principal does not have permission to access the lake. As a test make it a resource owner, then remove it and work out what permissions are missing.
We just started using Apache airflow in our project for our data pipelines .While exploring the features came to know about configuring remote folder as log destination in airflow .For that we
Created a google cloud bucket.
From Airflow UI created a new GS connection
I am not able to understand all the fields .I just created a sample GS Bucket under my project from google console and gave that project ID to this Connection.Left key file path and scopes as blank.
Then edited airflow.cfg file as follows
remote_base_log_folder = gs://my_test_bucket/
remote_log_conn_id = test_gs
After this changes restarted the web server and scheduler .But still my Dags is not writing logs to the GS bucket .I am able to see the logs which is creating logs in base_log_folder .But nothing is created in my bucket .
Is there any extra configuration needed from my side to get it working
Note: Using Airflow 1.8 .(Same issue I faced with AmazonS3 also. )
Updated on 20/09/2017
Tried the GS method attaching screenshot
Still I am not getting logs in the bucket
Thanks
Anoop R
I advise you to use a DAG to connect airflow to GCP instead of UI.
First, create a service account on GCP and download the json key.
Then execute this DAG (you can modify the scope of your access):
from airflow import DAG
from datetime import datetime
from airflow.operators.python_operator import PythonOperator
def add_gcp_connection(ds, **kwargs):
"""Add a airflow connection for GCP"""
new_conn = Connection(
conn_id='gcp_connection_id',
conn_type='google_cloud_platform',
)
scopes = [
"https://www.googleapis.com/auth/pubsub",
"https://www.googleapis.com/auth/datastore",
"https://www.googleapis.com/auth/bigquery",
"https://www.googleapis.com/auth/devstorage.read_write",
"https://www.googleapis.com/auth/logging.write",
"https://www.googleapis.com/auth/cloud-platform",
]
conn_extra = {
"extra__google_cloud_platform__scope": ",".join(scopes),
"extra__google_cloud_platform__project": "<name_of_your_project>",
"extra__google_cloud_platform__key_path": '<path_to_your_json_key>'
}
conn_extra_json = json.dumps(conn_extra)
new_conn.set_extra(conn_extra_json)
session = settings.Session()
if not (session.query(Connection).filter(Connection.conn_id ==
new_conn.conn_id).first()):
session.add(new_conn)
session.commit()
else:
msg = '\n\tA connection with `conn_id`={conn_id} already exists\n'
msg = msg.format(conn_id=new_conn.conn_id)
print(msg)
dag = DAG('add_gcp_connection', start_date=datetime(2016,1,1), schedule_interval='#once')
# Task to add a connection
AddGCPCreds = PythonOperator(
dag=dag,
task_id='add_gcp_connection_python',
python_callable=add_gcp_connection,
provide_context=True)
Thanks to Yu Ishikawa for this code.
Yes, you need to provide additional information for both, S3 and GCP connection.
S3
Configuration is passed via extra field as JSON. You can provide only profile
{"profile": "xxx"}
or credentials
{"profile": "xxx", "aws_access_key_id": "xxx", "aws_secret_access_key": "xxx"}
or path to config file
{"profile": "xxx", "s3_config_file": "xxx", "s3_config_format": "xxx"}
In case of the first option, boto will try to detect your credentials.
Source code - airflow/hooks/S3_hook.py:107
GCP
You can either provide key_path and scope (see Service account credentials) or credentials will be extracted from your environment in this order:
Environment variable GOOGLE_APPLICATION_CREDENTIALS pointing to a file with stored credentials information.
Stored "well known" file associated with gcloud command line tool.
Google App Engine (production and testing)
Google Compute Engine production environment.
Source code - airflow/contrib/hooks/gcp_api_base_hook.py:68
The reason for logs not being written to your bucket could be related to service account rather than config on airflow itself. Make sure it has access to the mentioned bucket. I had same problems in the past.
Adding more generous permissions to the service account, e.g. even project wide Editor and then narrowing it down. You could also try using gs client with that key and see if you can write to the bucket.
For me personally this scope works fine for writing logs: "https://www.googleapis.com/auth/cloud-platform"
In the Azure Powershell version 0.8 and 0.9, there is command
Get-AzureResource -ResourceGroupName "RGName" -OutputObjectFormat New
And, It returns the resources in the mentioned Resource Group of Azure. It necessitates the azure mode to be ARM mode.
But, in the Azure PowerShell version 1.2 and above
Get-AzureRMResource -ResourceGroupName "RGName"
fails to provide the resources present in a Resource Group. It needs further parameters like "ResourceID" or "ResourceName" which makes it resource specific.
What I need is that, it should return all the resources in a resource group.
Is it a bug with the newer version or am I missing something!
Suggest
You can use Find-AzureRmResource:
Find-AzureRmResource -ResourceGroupNameContains "RGName"
The Get-AzureRMResource PowerShell Command is implemented from REST API. If you check the REST API of listing resources in a resource group. You will see something like this.
https://management.azure.com/subscriptions/<Subscription ID>/resourceGroups/<resource group name>/resources?api-version=2015-01-01
And, if you add -debug option to Get-AzureRMResource -ResourceId <the resource id>, you will find the REST API it's using.
https://management.azure.com<the resource id>?api-version=2015-01-01
Comparing this two REST API, you will see that the following PowerShell command will list the resources in a resource group.
Get-AzureRmResource -ResourceId "/subscriptions/<Subscription ID>/resourceGroups/<resource group name>/resources"
I know it's tricky, but it does work.
Try
Get-AzureRmResource | where {$_.ResourceGroupName -eq "RG"}
Get the resource group in an object
$groups = Get-AzureRmResourceGroup -Name $RG_Name
fetch all the resources of a resource group in a variable
$t=(Find-AzureRmResource -ResourceGroupNameEquals
$groups.ResourceGroupName).ResourceName
I want to export an XML file which is contains all the cloud services and the VMs in each cloud service.
I would only call Get-AzureVM once to optimize processing. Afterwards I would group and output with PowerShell commands which are faster than filtering with the Azure PowerShell commands.
And with the grouping you also get an answer to your question in your title: how many VMs are in the Cloud Service? This is simply to get from the Count property.
$allVMs = Get-AzureVM
$allVMs | Group-Object -Property ServiceName
<# Output
Count Name Group
----- ---- -----
2 pksttest1 {Microsoft.WindowsAzure.Commands.ServiceManagement.Model.PersistentVMRoleListContext, Microsoft.WindowsAzure.Commands.ServiceManage...
1 pksttest2 {Microsoft.WindowsAzure.Commands.ServiceManagement.Model.PersistentVMRoleListContext}
#>
$services = Get-AzureVM | Group-Object -Property ServiceName
foreach ($service in $services) {
# Output cloud service name
"Cloud Service '$($service.Name)'"
# Output VMs in that cloud service
foreach ($vm in $service.Group) {
"VM '$($vm.HostName)'"
}
}
<# Output
Cloud Service 'pksttest1'
VM 'host1'
VM 'host3'
Cloud Service 'pksttest2'
VM 'host2'
#>
In your sample you call the Azure Service Management API once to get all cloud services and then again for every cloud service to get the VMs. With Get-AzureVM you get already all necessary data.
I also published the sample snippet on GitHub Gist: https://gist.github.com/pkirch/1ec6f3c1ca057b8beefb
I am writing some code for automatic deployment of Azure websites (including the creation of the website in Azure). I'm using the Azure Management Libraries and Azure Resource Management Libraries available in Nuget. Most of it is in place, however I have not been able to find a way to enable the "Always On" property through any of the APIs I've seen. This property can be set through the azure management portal under the Configure tab for a website.
I've checked:
The properties reference on MSDN: http://msdn.microsoft.com/en-us/library/azure/dn236426.aspx
The powershell APIs (get-azureresource, get-azurewebsite, ...) to see if there's a reference to Always On (there's not)
The REST calls the management portal is sending, through Fiddler. Here there is a reference to Always On in a POST going to https://manage.windowsazure.com/Websites/UpdateConfig (which is not part of the management or resource management APIs as far as I know). The exact path in the JSON body that is sent is /siteConfig/AlwaysOn.
So, the question is, is it possible to enable/disable Always On through an "official" API?
Thanks!
I believe I found the solution!
Using the resource management API, I can set the AlwaysOn property through the siteConfig object. In powershell:
Set-AzureResource -ApiVersion 2014-04-01 -PropertyObject #{"siteConfig" = #{"AlwaysOn" = $false}} -Name mywebsite -ResourceGroupName myrg -ResourceType Microsoft.Web/sites
In the resource management API in .NET it would be similar to this.
The resulting REST call, to
https://management.azure.com/subscriptions/xxx/resourcegroups/yyy/providers/Microsoft.Web/sites/zzz?api-version=2014-04-01:
{
"location": "West Europe",
"properties": {
"siteConfig": {
"AlwaysOn": true
}
},
"tags": {}
}
Using updated ARM (Azure Resource Manager) Powershell, v1.0+
Get-AzureRmResource: https://msdn.microsoft.com/en-us/library/mt652503.aspx
Set-AzureRmResource: https://msdn.microsoft.com/en-us/library/mt652514.aspx
# Variables - substitute your own values here
$ResourceGroupName = 'My Azure RM Resource Group Name'
$WebAppName = 'My Azure RM WebApp Name'
$ClientAffinityEnabled = $false
# Property object for nested, not exposed directly properties
$WebAppPropertiesObject = #{"siteConfig" = #{"AlwaysOn" = $true}}
# Variables
$WebAppResourceType = 'microsoft.web/sites'
# Get the resource from Azure (consider adding sanity checks, e.g. is $webAppResource -eq $null)
$webAppResource = Get-AzureRmResource -ResourceType $WebAppResourceType -ResourceGroupName $ResourceGroupName -ResourceName $WebAppName
# Set a directly exposed property, in this case whether client affinity is enabled
$webAppResource.Properties.ClientAffinityEnabled = $ClientAffinityEnabled
# Pass the resource object into the cmdlet that saves the changes to Azure
$webAppResource | Set-AzureRmResource -PropertyObject $WebAppPropertiesObject -Force
For those using the .Net API, it's
var cfg = await websiteClient.Sites.GetSiteConfigAsync(site.ResourceGroup, site.Name, cancellationToken).ConfigureAwait(false);
if (!cfg.AlwaysOn.GetValueOrDefault())
{
cfg.AlwaysOn = true;
await websiteClient.Sites.UpdateSiteConfigAsync(site.ResourceGroup, site.Name, cfg, cancellationToken).ConfigureAwait(false);
}