How to add data gateway to SQL connector with ARM template - sql

I'm trying to connect to a database through a data gateway (SQL Server Connector) with ARM templates. But I'm not sure if I miss something because I'm getting connection error with the gateway.
This is what I have so far in my api connection:
{
"$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"connections_sql_name": {
"defaultValue": "sql",
"type": "String"
},
"connections_sql_displayName": {
"defaultValue": "displaynameDB",
"type": "String"
},
"server": {
"defaultValue": "SERV01",
"type": "String"
},
"database": {
"defaultValue": "DB01",
"type": "String"
},
"authType": {
"defaultValue": "windows",
"type": "String"
},
"username": {
"defaultValue": "USER01",
"type": "String"
},
"password": {
"defaultValue": "PASS123",
"type": "String"
}
},
"variables": {},
"resources": [
{
"type": "Microsoft.Web/connections",
"apiVersion": "2016-06-01",
"name": "[parameters('connections_sql_name')]",
"location": "northeurope",
"properties": {
"displayName": "[parameters('connections_sql_displayName')]",
"customParameterValues": {},
"parameterValues": {
"server": "[parameters('server')]",
"database": "[parameters('database')]",
"authType": "[parameters('authType')]",
"username": "[parameters('username')]",
"password": "[parameters('password')]"
},
"api": {
"id": "[concat('/subscriptions/{sub-id}/providers/Microsoft.Web/locations/northeurope/managedApis/', parameters('connections_sql_name'))]"
}
}
}
]
}
And this is a part of my logic app under inputs:
"gateway": {
"gatewaySettings": {
"connectionDetails": [
"[parameters('gatewayServer')]",
"[parameters('gatewayDatabase')]"
],
"credentialType": "Windows",
"dataSourceType": "sql"
},
"type": "gatewaySetting"
},
Any help is appreciated! :)

Try removing the gateway block from the Logic App definition and changing the connection definition to this:
{
"type": "Microsoft.Web/connections",
"apiVersion": "2016-06-01",
"name": "[parameters('connections_sql_name')]",
"location": "northeurope",
"properties": {
"displayName": "[parameters('connections_sql_displayName')]",
"customParameterValues": {
},
"parameterValues": {
"server": "[parameters('server')]",
"database": "[parameters('database')]",
"authType": "[parameters('authType')]",
"username": "[parameters('username')]",
"password": "[parameters('password')]",
"gateway": {
"id": "/subscriptions/{sub-id}/resourceGroups/{gateway-resource-group-name}/providers/Microsoft.Web/connectionGateways/{gateway-name}"
}
},
"api": {
"id": "[concat('/subscriptions/{sub-id}/providers/Microsoft.Web/locations/northeurope/managedApis/', parameters('connections_sql_name'))]"
}
}
}
The documentation isn't very helpful regarding the gateway property in connection resources.

Related

SQL Server creation failing on Azure due to policies in place

We are in the process of creating SQL Server using ARM Template but since there are policies in place we are unable to get SQL Server created following are the 2 policy exceptions:
##[error]Resource 'xxxxx' was disallowed by policy. Error Type: PolicyViolation, Policy Definition Name : SQL Server should use a virtual network service endpoint (NPD), Policy Assignment Name : NPD1a8a9dc8-aef3-421a-93. Error Type: PolicyViolation, Policy Definition Name : Auditing on SQL server should be enabled (NPD), Policy Assignment Name : NPD7885d0ef-a3de-44a3-9a.
Following is the ARM Template we are using and I am not sure why its failing as now we have VNet rules and auditing also enabled as part of the SQL Server creation:
{
"$schema": http://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#,
"contentVersion": "1.0.0.0",
"parameters": {
"serverName": {
"type": "string",
"metadata": {
"description": "The name of the new database server to create."
}
},
"location": {
"type": "string",
"metadata": {
"description": "The location of the database server."
}
},
"serverVersion": {
"type": "string",
"defaultValue" : "12.0"
},
"administratorLogin": {
"type": "string",
"metadata": {
"description": "The account name to use for the database server administrator."
}
},
"administratorLoginPassword": {
"type": "securestring",
"metadata": {
"description": "The password to use for the database server administrator."
}
},
"storageAccountName": {
"type": "string",
"metadata": {
"description": "The name of the new storage account to create."
}
},
"emailAddresses": {
"type": "string",
"defaultValue": "",
"metadata": {
"description": "Email address for alerts."
}
},
"privateEndpointName": {
"type": "string"
},
"vnetName": {
"type": "string"
},
"vnetRg": {
"type": "string",
"metadata": {
"description": "Resource Group Name of VNet"
}
},
"subnet1Name": {
"type": "string"
},
"storageType": {
"type": "string",
"defaultValue": "Standard_GRS",
"allowedValues": [
"Standard_LRS",
"Standard_ZRS",
"Standard_GRS",
"Standard_RAGRS",
"Premium_LRS"
]
},
"BUSINESS-OWNER": {
"type": "string"
},
"COST-CENTER": {
"type": "int"
},
"LIFECYCLE": {
"type": "string"
},
"APPLICATION": {
"type": "string"
},
"PROJECT-CODE": {
"type": "string"
},
"TECHNICAL-OWNER": {
"type": "string"
},
"GL-CODE": {
"type": "string"
}
},
"resources": [
{
"name": "[parameters('serverName')]",
"type": "Microsoft.Sql/servers",
"location": "[parameters('location')]",
"apiVersion": "2014-04-01-preview",
"properties": {
"administratorLogin": "[parameters('administratorLogin')]",
"administratorLoginPassword": "[parameters('administratorLoginPassword')]",
"version": "[parameters('serverVersion')]",
"minimalTlsVersion": "1.2",
"publicNetworkAccess": "Disabled"
},
"tags": {
"BUSINESS-OWNER": "[parameters('BUSINESS-OWNER')]",
"COST-CENTER": "[parameters('COST-CENTER')]",
"LIFECYCLE": "[parameters('LIFECYCLE')]",
"APPLICATION": "[parameters('APPLICATION')]",
"PROJECT-CODE": "[parameters('PROJECT-CODE')]",
"TECHNICAL-OWNER": "[parameters('TECHNICAL-OWNER')]",
"GL-CODE": "[parameters('GL-CODE')]"
},
"resources": [
{
"name": "sergiodb1",
"type": "databases",
"location": "[parameters('location')]",
"tags": {
"BUSINESS-OWNER": "[parameters('BUSINESS-OWNER')]",
"COST-CENTER": "[parameters('COST-CENTER')]",
"LIFECYCLE": "[parameters('LIFECYCLE')]",
"APPLICATION": "[parameters('APPLICATION')]",
"PROJECT-CODE": "[parameters('PROJECT-CODE')]",
"TECHNICAL-OWNER": "[parameters('TECHNICAL-OWNER')]",
"GL-CODE": "[parameters('GL-CODE')]"
},
"apiVersion": "2015-05-01-preview",
"dependsOn": [
"[parameters('serverName')]"
],
"properties": {
"edition": "Basic",
"collation": "SQL_Latin1_General_CP1_CI_AS"
}
},
{
"type": "Microsoft.Sql/servers/virtualNetworkRules",
"apiVersion": "2020-08-01-preview",
"name": "[concat(parameters('serverName'), '/allow-', parameters('subnet1Name'))]",
"dependsOn": [ "[resourceId('Microsoft.Sql/servers', parameters('serverName'))]" ],
"properties": {
"virtualNetworkSubnetId": "[resourceId(parameters('vnetRg'), 'Microsoft.Network/virtualNetworks/subnets', parameters('vnetName'), parameters('subnet1Name'))]",
"ignoreMissingVnetServiceEndpoint": false
}
},
{
"apiVersion": "2014-04-01-preview",
"type": "firewallrules",
"location": "[parameters('location')]",
"name": "AllowAllWindowsAzureIps",
"dependsOn": [
"[concat('Microsoft.Sql/servers/', parameters('serverName'))]"
],
"properties": {
"endIpAddress": "0.0.0.0",
"startIpAddress": "0.0.0.0"
}
},
{
"name": "Default",
"type": "auditingSettings",
"apiVersion": "2017-03-01-preview",
"location": "[parameters('location')]",
"dependsOn": [
"[concat('Microsoft.Sql/servers/', parameters('serverName'))]"
],
"properties": {
"State": "Enabled",
"storageEndpoint": "[concat('https://',parameters('storageAccountName'),'.blob.core.windows.net')]",
"storageAccountAccessKey": "[listKeys(resourceId('Microsoft.Storage/storageAccounts', parameters('storageAccountName')), providers('Microsoft.Storage', 'storageAccounts').apiVersions[0]).keys[0].value]",
"retentionDays": 365,
"auditActionsAndGroups": null,
"storageAccountSubscriptionId": "[subscription().subscriptionId]",
"isStorageSecondaryKeyInUse": false,
"isAzureMonitorTargetEnabled": false
}
},
{
"name": "DefaultSAP",
"type": "securityAlertPolicies",
"apiVersion": "2017-03-01-preview",
"dependsOn": [
"[concat('Microsoft.Sql/servers/', parameters('serverName'))]",
"[concat('Microsoft.Sql/servers/', parameters('serverName'), '/auditingSettings/Default')]"
],
"properties": {
"state": "Enabled",
"disabledAlerts": null,
"emailAddresses": "[array(parameters('emailAddresses'))]",
"emailAccountAdmins": true,
"storageEndpoint": "[concat('https://',parameters('storageAccountName'),'.blob.core.windows.net')]",
"storageAccountAccessKey": "[listKeys(resourceId('Microsoft.Storage/storageAccounts', parameters('storageAccountName')), providers('Microsoft.Storage', 'storageAccounts').apiVersions[0]).keys[0].value]",
"retentionDays": 365
}
},
{
"name": "VulnerabilityAssessment",
"type": "vulnerabilityAssessments",
"apiVersion": "2018-06-01-preview",
"dependsOn": [
"[concat('Microsoft.Sql/servers/', parameters('serverName'))]",
"[concat('Microsoft.Sql/servers/', parameters('serverName'), '/auditingSettings/Default')]",
"[concat('Microsoft.Sql/servers/', parameters('serverName'), '/securityAlertPolicies/DefaultSAP')]"
],
"properties": {
"storageContainerPath": "[concat('https://',parameters('storageAccountName'),'.blob.core.windows.net','/vulnerability-assessment')]",
"storageAccountAccessKey": "[listKeys(resourceId('Microsoft.Storage/storageAccounts', parameters('storageAccountName')), providers('Microsoft.Storage', 'storageAccounts').apiVersions[0]).keys[0].value]",
"recurringScans": {
"isEnabled": true,
"emailSubscriptionAdmins": true,
"emails": []
}
}
}
]
},
{
"type": "Microsoft.Network/privateEndpoints",
"apiVersion": "2020-06-01",
"name": "[parameters('privateEndpointName')]",
"location": "[parameters('location')]",
"dependsOn": [
"[parameters('serverName')]"
],
"properties": {
"subnet": {
"id": "[resourceId(parameters('vnetRg'), 'Microsoft.Network/virtualNetworks/subnets', parameters('vnetName'), parameters('subnet1Name'))]"
},
"privateLinkServiceConnections": [
{
"name": "[parameters('privateEndpointName')]",
"properties": {
"privateLinkServiceId": "[resourceId('Microsoft.Sql/servers',parameters('serverName'))]",
"groupIds": [
"sqlServer"
]
}
}
]
},
"tags": {
"BUSINESS-OWNER": "[parameters('BUSINESS-OWNER')]",
"COST-CENTER": "[parameters('COST-CENTER')]",
"LIFECYCLE": "[parameters('LIFECYCLE')]",
"APPLICATION": "[parameters('APPLICATION')]",
"PROJECT-CODE": "[parameters('PROJECT-CODE')]",
"TECHNICAL-OWNER": "[parameters('TECHNICAL-OWNER')]",
"GL-CODE": "[parameters('GL-CODE')]"
}
}
],
"outputs": {
}
}
Resolved: My ARM Template formatting had issues but most important was the REGION for VNet in which the Pvt endpoint was getting created was set to EASTUS2 while the SQL Server was was getting provisioned in EASTUS. After fixing the regions and the ARM template I was able to successfully deploy it. Following is the corrected ARM Template:
{
"$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters":{
"serverName": {
"type": "string",
"metadata": {
"description": "The name of the new database server to create."
}
},
"location": {
"type": "string",
"metadata": {
"description": "The location of the database server."
}
},
"serverVersion": {
"type": "string",
"defaultValue" : "12.0"
},
"administratorLogin": {
"type": "string",
"metadata": {
"description": "The account name to use for the database server administrator."
}
},
"administratorLoginPassword": {
"type": "securestring",
"metadata": {
"description": "The password to use for the database server administrator."
}
},
"storageAccountName": {
"type": "string",
"metadata": {
"description": "The name of the new storage account to create."
}
},
"emailAddresses": {
"type": "string",
"defaultValue": "",
"metadata": {
"description": "Email address for alerts."
}
},
"privateEndpointName": {
"type": "string"
},
"vnetName": {
"type": "string"
},
"vnetRg": {
"type": "string",
"metadata": {
"description": "Resource Group Name of VNet"
}
},
"subnet1Name": {
"type": "string"
},
"storageType": {
"type": "string",
"defaultValue": "Standard_GRS",
"allowedValues": [
"Standard_LRS",
"Standard_ZRS",
"Standard_GRS",
"Standard_RAGRS",
"Premium_LRS"
]
},
"BUSINESS-OWNER": {
"type": "string"
},
"COST-CENTER": {
"type": "int"
},
"LIFECYCLE": {
"type": "string"
},
"APPLICATION": {
"type": "string"
},
"PROJECT-CODE": {
"type": "string"
},
"TECHNICAL-OWNER": {
"type": "string"
},
"GL-CODE": {
"type": "string"
}
},
"variables": {
"databaseName": "[concat(parameters('serverName'),'/sample-db')]"
},
"resources": [
{
"type": "Microsoft.Sql/servers",
"apiVersion": "2020-02-02-preview",
"name": "[parameters('serverName')]",
"location": "[parameters('location')]",
"tags": {
"displayName": "[parameters('serverName')]",
"BUSINESS-OWNER": "xxx",
"COST-CENTER": "11",
"LIFECYCLE": "xx",
"APPLICATION": "xx",
"PROJECT-CODE": "xx",
"TECHNICAL-OWNER": "xxx",
"GL-CODE": "111"
},
"kind": "v12.0",
"properties": {
"administratorLogin": "[parameters('administratorLogin')]",
"administratorLoginPassword": "[parameters('administratorLoginPassword')]",
"version": "12.0",
"minimalTlsVersion": "1.2",
"publicNetworkAccess": "Disabled"
},
"resources": [
{
"type": "Microsoft.Sql/servers/databases",
"apiVersion": "2020-02-02-preview",
"name": "[variables('databaseName')]",
"location": "[parameters('location')]",
"dependsOn": [
"[resourceId('Microsoft.Sql/servers', parameters('serverName'))]"
],
"tags": {
"displayName": "[variables('databaseName')]",
"BUSINESS-OWNER": "xxx",
"COST-CENTER": "11",
"LIFECYCLE": "xx",
"APPLICATION": "xx",
"PROJECT-CODE": "xx",
"TECHNICAL-OWNER": "xxx",
"GL-CODE": "111"
},
"sku": {
"name": "Basic",
"tier": "Basic",
"capacity": 5
},
"properties": {
"collation": "SQL_Latin1_General_CP1_CI_AS",
"edition": "Basic",
"maxSizeBytes": 104857600,
"requestedServiceObjectiveName": "Basic",
"sampleName": "AdventureWorksLT"
}
},
{
"type": "Microsoft.Network/privateEndpoints",
"apiVersion": "2020-06-01",
"name": "[parameters('privateEndpointName')]",
"location": "[parameters('location')]",
"dependsOn": [
"[resourceId('Microsoft.Sql/servers', parameters('serverName'))]"
],
"tags": {
"BUSINESS-OWNER": "xxx",
"COST-CENTER": "11",
"LIFECYCLE": "xx",
"APPLICATION": "xx",
"PROJECT-CODE": "xx",
"TECHNICAL-OWNER": "xxx",
"GL-CODE": "111"
},
"properties": {
"subnet": {
"id": "[resourceId(parameters('vnetRg'), 'Microsoft.Network/virtualNetworks/subnets', parameters('vnetName'), parameters('subnet1Name'))]"
},
"privateLinkServiceConnections": [
{
"name": "[parameters('privateEndpointName')]",
"properties": {
"privateLinkServiceId": "[resourceId('Microsoft.Sql/servers',parameters('serverName'))]",
"groupIds": [
"sqlServer"
]
}
}
]
}
},
{
"name": "Default",
"type": "auditingSettings",
"apiVersion": "2017-03-01-preview",
"location": "[parameters('location')]",
"dependsOn": ["[resourceId('Microsoft.Sql/servers', parameters('serverName'))]"],
"properties": {
"State": "Enabled",
"storageEndpoint": "[concat('https://',parameters('storageAccountName'),'.blob.core.windows.net')]",
"storageAccountAccessKey": "[listKeys(resourceId('Microsoft.Storage/storageAccounts', parameters('storageAccountName')), providers('Microsoft.Storage', 'storageAccounts').apiVersions[0]).keys[0].value]",
"retentionDays": 365,
"auditActionsAndGroups": null,
"storageAccountSubscriptionId": "[subscription().subscriptionId]",
"isStorageSecondaryKeyInUse": false,
"isAzureMonitorTargetEnabled": false
}
},
{
"name": "DefaultSAP",
"type": "securityAlertPolicies",
"apiVersion": "2017-03-01-preview",
"dependsOn": ["[resourceId('Microsoft.Sql/servers', parameters('serverName'))]",
"[concat('Microsoft.Sql/servers/', parameters('serverName'), '/auditingSettings/Default')]"
],
"properties": {
"state": "Enabled",
"disabledAlerts": null,
"emailAddresses": "[array(parameters('emailAddresses'))]",
"emailAccountAdmins": true,
"storageEndpoint": "[concat('https://',parameters('storageAccountName'),'.blob.core.windows.net')]",
"storageAccountAccessKey": "[listKeys(resourceId('Microsoft.Storage/storageAccounts', parameters('storageAccountName')), providers('Microsoft.Storage', 'storageAccounts').apiVersions[0]).keys[0].value]",
"retentionDays": 365
}
}
]
}
]
}
Please make sure the subnet you are using on the template has assigned the Microsoft.Sql type name, meaning it is already a Virtual Service endpoint.
The following script can add the type name Microsoft.Sql to your subnet. But the script tries the add only if your subnet lacks the type name.
### 1. LOG into to your Azure account, needed only once per PS session. Assign variables.
$yesno = Read-Host 'Do you need to log into Azure (only one time per powershell.exe session)? [yes/no]'
if ('yes' -eq $yesno) { Connect-AzAccount }
# Assignments to variables used by the later scripts.
# You can EDIT these values, if necessary.
$SubscriptionName = 'yourSubscriptionName'
Select-AzSubscription -SubscriptionName "$SubscriptionName"
$ResourceGroupName = 'yourRGName'
$VNetName = 'yourVNetName'
$SubnetName = 'yourSubnetName'
$SubnetAddressPrefix = 'Obtain this value from the Azure portal.' # Looks roughly like: '10.0.0.0/24'
$ServiceEndpointTypeName_SqlDb = 'Microsoft.Sql' # Do NOT edit. Is official value.
### 2. Search for your virtual network, and then for your subnet.
# Search for the virtual network.
$vnet = $null
$vnet = Get-AzVirtualNetwork -ResourceGroupName $ResourceGroupName -Name $VNetName
if ($vnet -eq $null) {
Write-Host "Caution: No virtual network found by the name '$VNetName'."
return
}
$subnet = $null
for ($nn = 0; $nn -lt $vnet.Subnets.Count; $nn++) {
$subnet = $vnet.Subnets[$nn]
if ($subnet.Name -eq $SubnetName) { break }
$subnet = $null
}
if ($null -eq $subnet) {
Write-Host "Caution: No subnet found by the name '$SubnetName'"
Return
}
### 3. Is your subnet tagged as 'Microsoft.Sql' endpoint server type?
$endpointMsSql = $null
for ($nn = 0; $nn -lt $subnet.ServiceEndpoints.Count; $nn++) {
$endpointMsSql = $subnet.ServiceEndpoints[$nn]
if ($endpointMsSql.Service -eq $ServiceEndpointTypeName_SqlDb) {
$endpointMsSql
break
}
$endpointMsSql = $null
}
if ($null -eq $endpointMsSql) {
Write-Host "Good: Subnet found, and is already tagged as an endpoint of type '$ServiceEndpointTypeName_SqlDb'."
return
} else {
Write-Host "Caution: Subnet found, but not yet tagged as an endpoint of type '$ServiceEndpointTypeName_SqlDb'."
# Ask the user for confirmation.
$yesno = Read-Host 'Do you want the PS script to apply the endpoint type name to your subnet? [yes/no]'
if ('no' -eq $yesno) { return }
}
### 4. Add a Virtual Service endpoint of type name 'Microsoft.Sql', on your subnet.
$setParams = #{
Name = $SubnetName
AddressPrefix = $SubnetAddressPrefix
VirtualNetwork = $vnet
ServiceEndpoint = $ServiceEndpointTypeName_SqlDb
}
$vnet = Set-AzVirtualNetworkSubnetConfig #setParams
# Persist the subnet update.
$vnet = Set-AzVirtualNetwork -VirtualNetwork $vnet
for ($nn = 0; $nn -lt $vnet.Subnets.Count; $nn++) {
$vnet.Subnets[0].ServiceEndpoints # Display.
}

Running data flows in Azure Data Factory 4 times slower than running in Azure SSIS data flow

Here are the details of this performance test (very simple). I'm trying to understand why running data flows in the cloud native Azure Data Factory environment (Spark) is so much slower than running data flows hosted in Azure SSIS IR. My results show that running in latest ADFv2 is over 4 times slower than running the exact same data flow in Azure SSIS (even with a warm IR cluster already warmed up from previous run). I like all the new features of the v2 data flows but it hardly seems worth the performance hit unless I'm completely missing something. Eventually I'll be adding more complex data flows but wanted to understand base performance behavior.
Source:
1GB CSV stored in blob storage
Destination:
Azure SQL Server Database (one table and truncated before each run)
When using control flow in ADFv2 using a simple CopyActivity (no data flow)
91 seconds
When using native SSIS package with data flow (Azure Feature Pack to pull from same blob storage) running Azure SSIS with 8 cores.
76 seconds
Pure ADF Cloud Pipeline using DataFlow with warm Azure IR (cached from previous run) 8 (+ 8 Driver cores) with default partitioning (Spark)
(includes 96 seconds cluster startup which is another thing I don't understand since the TTL is 30 minutes on the IR and it was just ran 10 minutes prior)
360 seconds
Pipeline (LandWithCopy)
{
"name": "LandWithCopy",
"properties": {
"activities": [
{
"name": "CopyData",
"type": "Copy",
"dependsOn": [],
"policy": {
"timeout": "7.00:00:00",
"retry": 0,
"retryIntervalInSeconds": 30,
"secureOutput": false,
"secureInput": false
},
"userProperties": [],
"typeProperties": {
"source": {
"type": "DelimitedTextSource",
"storeSettings": {
"type": "AzureBlobStorageReadSettings",
"recursive": true,
"wildcardFileName": "data.csv",
"enablePartitionDiscovery": false
},
"formatSettings": {
"type": "DelimitedTextReadSettings"
}
},
"sink": {
"type": "AzureSqlSink",
"preCopyScript": "TRUNCATE TABLE PatientAR",
"disableMetricsCollection": false
},
"enableStaging": false,
"translator": {
"type": "TabularTranslator",
"mappings": [
{
"source": {
"name": "RecordAction",
"type": "String"
},
"sink": {
"name": "RecordAction",
"type": "String"
}
},
{
"source": {
"name": "UniqueId",
"type": "String"
},
"sink": {
"name": "UniqueId",
"type": "String"
}
},
{
"source": {
"name": "Type",
"type": "String"
},
"sink": {
"name": "Type",
"type": "String"
}
},
{
"source": {
"name": "TypeDescription",
"type": "String"
},
"sink": {
"name": "TypeDescription",
"type": "String"
}
},
{
"source": {
"name": "PatientId",
"type": "String"
},
"sink": {
"name": "PatientId",
"type": "String"
}
},
{
"source": {
"name": "PatientVisitId",
"type": "String"
},
"sink": {
"name": "PatientVisitId",
"type": "String"
}
},
{
"source": {
"name": "VisitDateOfService",
"type": "String"
},
"sink": {
"name": "VisitDateOfService",
"type": "String"
}
},
{
"source": {
"name": "VisitDateOfEntry",
"type": "String"
},
"sink": {
"name": "VisitDateOfEntry",
"type": "String"
}
},
{
"source": {
"name": "DoctorId",
"type": "String"
},
"sink": {
"name": "DoctorId",
"type": "String"
}
},
{
"source": {
"name": "DoctorName",
"type": "String"
},
"sink": {
"name": "DoctorName",
"type": "String"
}
},
{
"source": {
"name": "FacilityId",
"type": "String"
},
"sink": {
"name": "FacilityId",
"type": "String"
}
},
{
"source": {
"name": "FacilityName",
"type": "String"
},
"sink": {
"name": "FacilityName",
"type": "String"
}
},
{
"source": {
"name": "CompanyName",
"type": "String"
},
"sink": {
"name": "CompanyName",
"type": "String"
}
},
{
"source": {
"name": "TicketNumber",
"type": "String"
},
"sink": {
"name": "TicketNumber",
"type": "String"
}
},
{
"source": {
"name": "TransactionDateOfEntry",
"type": "String"
},
"sink": {
"name": "TransactionDateOfEntry",
"type": "String"
}
},
{
"source": {
"name": "InternalCode",
"type": "String"
},
"sink": {
"name": "InternalCode",
"type": "String"
}
},
{
"source": {
"name": "ExternalCode",
"type": "String"
},
"sink": {
"name": "ExternalCode",
"type": "String"
}
},
{
"source": {
"name": "Description",
"type": "String"
},
"sink": {
"name": "Description",
"type": "String"
}
},
{
"source": {
"name": "Fee",
"type": "String"
},
"sink": {
"name": "Fee",
"type": "String"
}
},
{
"source": {
"name": "Units",
"type": "String"
},
"sink": {
"name": "Units",
"type": "String"
}
},
{
"source": {
"name": "AREffect",
"type": "String"
},
"sink": {
"name": "AREffect",
"type": "String"
}
},
{
"source": {
"name": "Action",
"type": "String"
},
"sink": {
"name": "Action",
"type": "String"
}
},
{
"source": {
"name": "InsuranceGroup",
"type": "String"
},
"sink": {
"name": "InsuranceGroup",
"type": "String"
}
},
{
"source": {
"name": "Payer",
"type": "String"
},
"sink": {
"name": "Payer",
"type": "String"
}
},
{
"source": {
"name": "PayerType",
"type": "String"
},
"sink": {
"name": "PayerType",
"type": "String"
}
},
{
"source": {
"name": "PatBalance",
"type": "String"
},
"sink": {
"name": "PatBalance",
"type": "String"
}
},
{
"source": {
"name": "InsBalance",
"type": "String"
},
"sink": {
"name": "InsBalance",
"type": "String"
}
},
{
"source": {
"name": "Charges",
"type": "String"
},
"sink": {
"name": "Charges",
"type": "String"
}
},
{
"source": {
"name": "Payments",
"type": "String"
},
"sink": {
"name": "Payments",
"type": "String"
}
},
{
"source": {
"name": "Adjustments",
"type": "String"
},
"sink": {
"name": "Adjustments",
"type": "String"
}
},
{
"source": {
"name": "TransferAmount",
"type": "String"
},
"sink": {
"name": "TransferAmount",
"type": "String"
}
},
{
"source": {
"name": "FiledAmount",
"type": "String"
},
"sink": {
"name": "FiledAmount",
"type": "String"
}
},
{
"source": {
"name": "CheckNumber",
"type": "String"
},
"sink": {
"name": "CheckNumber",
"type": "String"
}
},
{
"source": {
"name": "CheckDate",
"type": "String"
},
"sink": {
"name": "CheckDate",
"type": "String"
}
},
{
"source": {
"name": "Created",
"type": "String"
},
"sink": {
"name": "Created",
"type": "String"
}
},
{
"source": {
"name": "ClientTag",
"type": "String"
},
"sink": {
"name": "ClientTag",
"type": "String"
}
}
]
}
},
"inputs": [
{
"referenceName": "PAR_Source_DS",
"type": "DatasetReference"
}
],
"outputs": [
{
"referenceName": "PAR_Sink_DS",
"type": "DatasetReference"
}
]
}
],
"annotations": []
}
}
Pipeline Data Flow (LandWithFlow)
{
"name": "WriteData",
"properties": {
"type": "MappingDataFlow",
"typeProperties": {
"sources": [
{
"dataset": {
"referenceName": "PAR_Source_DS",
"type": "DatasetReference"
},
"name": "GetData"
}
],
"sinks": [
{
"dataset": {
"referenceName": "PAR_Sink_DS",
"type": "DatasetReference"
},
"name": "WriteData"
}
],
"transformations": [],
"script": "source(output(\n\t\tRecordAction as string,\n\t\tUniqueId as string,\n\t\tType as string,\n\t\tTypeDescription as string,\n\t\tPatientId as string,\n\t\tPatientVisitId as string,\n\t\tVisitDateOfService as string,\n\t\tVisitDateOfEntry as string,\n\t\tDoctorId as string,\n\t\tDoctorName as string,\n\t\tFacilityId as string,\n\t\tFacilityName as string,\n\t\tCompanyName as string,\n\t\tTicketNumber as string,\n\t\tTransactionDateOfEntry as string,\n\t\tInternalCode as string,\n\t\tExternalCode as string,\n\t\tDescription as string,\n\t\tFee as string,\n\t\tUnits as string,\n\t\tAREffect as string,\n\t\tAction as string,\n\t\tInsuranceGroup as string,\n\t\tPayer as string,\n\t\tPayerType as string,\n\t\tPatBalance as string,\n\t\tInsBalance as string,\n\t\tCharges as string,\n\t\tPayments as string,\n\t\tAdjustments as string,\n\t\tTransferAmount as string,\n\t\tFiledAmount as string,\n\t\tCheckNumber as string,\n\t\tCheckDate as string,\n\t\tCreated as string,\n\t\tClientTag as string\n\t),\n\tallowSchemaDrift: true,\n\tvalidateSchema: false,\n\twildcardPaths:['data.csv']) ~> GetData\nGetData sink(input(\n\t\tRecordAction as string,\n\t\tUniqueId as string,\n\t\tType as string,\n\t\tTypeDescription as string,\n\t\tPatientId as string,\n\t\tPatientVisitId as string,\n\t\tVisitDateOfService as string,\n\t\tVisitDateOfEntry as string,\n\t\tDoctorId as string,\n\t\tDoctorName as string,\n\t\tFacilityId as string,\n\t\tFacilityName as string,\n\t\tCompanyName as string,\n\t\tTicketNumber as string,\n\t\tTransactionDateOfEntry as string,\n\t\tInternalCode as string,\n\t\tExternalCode as string,\n\t\tDescription as string,\n\t\tFee as string,\n\t\tUnits as string,\n\t\tAREffect as string,\n\t\tAction as string,\n\t\tInsuranceGroup as string,\n\t\tPayer as string,\n\t\tPayerType as string,\n\t\tPatBalance as string,\n\t\tInsBalance as string,\n\t\tCharges as string,\n\t\tPayments as string,\n\t\tAdjustments as string,\n\t\tTransferAmount as string,\n\t\tFiledAmount as string,\n\t\tCheckNumber as string,\n\t\tCheckDate as string,\n\t\tCreated as string,\n\t\tClientTag as string,\n\t\tFileName as string,\n\t\tPractice as string\n\t),\n\tallowSchemaDrift: true,\n\tvalidateSchema: false,\n\tdeletable:false,\n\tinsertable:true,\n\tupdateable:false,\n\tupsertable:false,\n\tformat: 'table',\n\tpreSQLs:['TRUNCATE TABLE PatientAR'],\n\tmapColumn(\n\t\tRecordAction,\n\t\tUniqueId,\n\t\tType,\n\t\tTypeDescription,\n\t\tPatientId,\n\t\tPatientVisitId,\n\t\tVisitDateOfService,\n\t\tVisitDateOfEntry,\n\t\tDoctorId,\n\t\tDoctorName,\n\t\tFacilityId,\n\t\tFacilityName,\n\t\tCompanyName,\n\t\tTicketNumber,\n\t\tTransactionDateOfEntry,\n\t\tInternalCode,\n\t\tExternalCode,\n\t\tDescription,\n\t\tFee,\n\t\tUnits,\n\t\tAREffect,\n\t\tAction,\n\t\tInsuranceGroup,\n\t\tPayer,\n\t\tPayerType,\n\t\tPatBalance,\n\t\tInsBalance,\n\t\tCharges,\n\t\tPayments,\n\t\tAdjustments,\n\t\tTransferAmount,\n\t\tFiledAmount,\n\t\tCheckNumber,\n\t\tCheckDate,\n\t\tCreated,\n\t\tClientTag\n\t),\n\tskipDuplicateMapInputs: true,\n\tskipDuplicateMapOutputs: true) ~> WriteData"
}
}
}
We are having the Same issues. Copy Activity without Data Flow is much faster than Data Flow. Our case is Copy Activity vs Data Flow. Not Sure if I'm doing anything wrong.
Our Scenario is just copy from Source to Destination 13 tables based on where Clause. We now have two copy activity which takes 1.5 minutes. So I was thinking may be create Data Flow and do one Source two Sinks. But it's running like 5 minutes to 8 Minutes depending on Cluster startup time. Hope we get an answer.

How to add Azure AD groups via ARM script to a Azure SQL database

I am creating a new Azure SQL with two databses and an elastic pool via ARM script. These are my scripts
azure.deploy.ps1
{
"$schema": "http://schema.management.azure.com/schemas/2014-04-01-preview/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"administratorLogin": {
"type": "string",
"metadata": {
"description": "The SQL Server administrator login"
}
},
"administratorLoginPassword": {
"type": "securestring",
"metadata": {
"description": "The SQL Server administrator login password."
}
},
"serverName": {
"type": "string",
"metadata": {
"description": "The SQL Server name."
}
},
"elasticPoolName": {
"type": "string",
"metadata": {
"description": "The Elastic Pool name."
}
},
"edition": {
"type": "string",
"defaultValue": "Standard",
"allowedValues": [
"Basic",
"Standard",
"Premium"
],
"metadata": {
"description": "The Elastic Pool edition."
}
},
"poolDtu": {
"type": "int",
"metadata": {
"description": "The Elastic Pool DTU."
}
},
"databaseDtuMin": {
"type": "int",
"defaultValue": 0,
"metadata": {
"description": "The Elastic Pool database DTU min."
}
},
"databaseDtuMax": {
"type": "int",
"metadata": {
"description": "The Elastic Pool database DTU max."
}
},
"databasesNames": {
"type": "array",
"defaultValue": [
"db1",
"db2"
],
"metadata": {
"description": "The SQL Databases names."
}
},
"databaseCollation": {
"type": "string",
"defaultValue": "SQL_Latin1_General_CP1_CI_AS",
"metadata": {
"description": "The SQL Database collation."
}
},
"location": {
"type": "string",
"defaultValue": "[resourceGroup().location]",
"metadata": {
"description": "Location for all resources."
}
}
},
"variables": {},
"resources": [
{
"apiVersion": "2014-04-01-preview",
"location": "[parameters('location')]",
"name": "[parameters('serverName')]",
"properties": {
"administratorLogin": "[parameters('administratorLogin')]",
"administratorLoginPassword": "[parameters('administratorLoginPassword')]",
"version": "12.0"
},
"type": "Microsoft.Sql/servers"
},
{
"apiVersion": "2014-04-01",
"dependsOn": [
"[concat('Microsoft.Sql/servers/', parameters('serverName'))]"
],
"location": "[parameters('location')]",
"name": "[concat(parameters('serverName'), '/', parameters('elasticPoolName'))]",
"properties": {
"edition": "[parameters('edition')]",
"dtu": "[parameters('poolDtu')]",
"databaseDtuMin": "[parameters('databaseDtuMin')]",
"databaseDtuMax": "[parameters('databaseDtuMax')]"
},
"type": "Microsoft.Sql/servers/elasticPools"
},
{
"type": "Microsoft.Sql/servers/databases",
"name": "[concat(parameters('serverName'), '/', parameters('databasesNames')[copyIndex()])]",
"location": "[parameters('location')]",
"apiVersion": "2014-04-01-preview",
"dependsOn": [
"[concat('Microsoft.Sql/servers/', parameters('serverName'))]",
"[concat('Microsoft.Sql/servers/', parameters('serverName') ,'/elasticpools/', parameters('elasticPoolName'))]"
],
"properties": {
"collation": "[parameters('databaseCollation')]",
"requestedServiceObjectiveName": "ElasticPool",
"elasticPoolName": "[parameters('elasticPoolName')]"
},
"copy": {
"name": "addDatabasesInElasticPool",
"count": "[length(parameters('databasesNames'))]"
}
},
{
"apiVersion": "2014-04-01-preview",
"dependsOn": [
"[concat('Microsoft.Sql/servers/', parameters('serverName'))]"
],
"location": "[parameters('location')]",
"name": "[concat(parameters('serverName'), '/', 'AllowAllWindowsAzureIps')]",
"properties": {
"endIpAddress": "0.0.0.0",
"startIpAddress": "0.0.0.0"
},
"type": "Microsoft.Sql/servers/firewallrules"
}
]
}
an this is the parameters file:
azure.deploy.parameters.ps1
{
"$schema": "http://schema.management.azure.com/schemas/2014-04-01-preview/deploymentParameters.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"administratorLogin": {
"value": "bogblogsqldbadmin"
},
"serverName": {
"value": "azrsqlsrv1"
},
"elasticPoolName": {
"value": "azrsqlsrve1"
},
"poolDtu": {
"value": 100
},
"databaseDtuMax": {
"value": 100
},
"databasesNames": {
"value": [ "asqldb11", "asqldb12" ]
}
}
}
I would like to use Azure AD and groups to authenticate users on these database. I would like to add those groups and the configuration for the use of Azure AD directly in my ARM scripts. How can i do that? Is that possible?
Below example may help:
{
"$schema": "http://schema.management.azure.com/schemas/2014-04-01-preview/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"SQL Administrator Login": {
"type": "String"
},
"SQL Administrator Password": {
"type": "SecureString"
},
"AAD Admin Login": {
"type": "String"
},
"AAD Admin ObjectID": {
"type": "String"
},
"AAD TenantId": {
"type": "String"
},
"Location (Region)": {
"type": "String"
},
"Server Name": {
"type": "String"
}
},
"variables": {},
"resources": [
{
"type": "Microsoft.Sql/servers",
"name": "[parameters('Server Name')]",
"apiVersion": "2014-04-01-preview",
"location": "[parameters('Location (Region)')]",
"properties": {
"administratorLogin": "[parameters('SQL Administrator Login')]",
"administratorLoginPassword": "[parameters('SQL Administrator Password')]",
"version": "12.0"
},
"resources": [
{
"type": "firewallrules",
"name": "AllowAllWindowsAzureIps",
"apiVersion": "2014-04-01-preview",
"location": "[parameters('Location (Region)')]",
"properties": {
"endIpAddress": "0.0.0.0",
"startIpAddress": "0.0.0.0"
},
"dependsOn": [
"[concat('Microsoft.Sql/servers/', parameters('Server Name'))]"
]
},
{
"type": "administrators",
"name": "activeDirectory",
"apiVersion": "2014-04-01-preview",
"location": "[parameters('Location (Region)')]",
"properties": {
"administratorType": "ActiveDirectory",
"login": "[parameters('AAD Admin Login')]",
"sid": "[parameters('AAD Admin ObjectID')]",
"tenantId": "[parameters('AAD TenantID')]"
},
"dependsOn": [
"[concat('Microsoft.Sql/servers/', parameters('Server Name'))]"
]
}
]
}
]
}

Deploying a marketplace Connector from Azure Resource Group template

I'm using the Azure Resource Group project template in Visual studio to deploy two API Apps and a Logic App. I want one of those API Apps to be a Blob Connector from the marketplace. What I need is the uri of the .zip package for the connector, as shown here:
{
"apiVersion": "2014-06-01",
"name": "MSDeploy",
"type": "Extensions",
"dependsOn": [
//........
],
"properties": {
"packageUri": "https://auxmktplceprod.blob.core.windows.net/packages/UmbracoCms.WebPI.7.2.5.zip",
"dbType": "SQL",
(source)
I tried this solution, but that cmdlet is now deprecated. Is there any way to get these URIs?
-Thanks!
I found a way of deploying custom api app with Marketplace apps.
Below is a sample script to just guide you
{
"$schema": "http://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"blobConnectorName": {
"type": "string",
"minLength": 1,
"defaultValue" : "mytestblobconnector"
},
"blobStorageAccount": {
"type": "string",
"minLength": 1,
"defaultValue" : "mystorage.blob.core.windows.net"
},
"blobStorageKey": {
"type": "securestring",
"minLength": 1,
"defaultValue" : "storgekey"
},
"blobContainerName": {
"type": "string",
"minLength": 1,
"defaultValue" : "mycontainer"
},
"gatewayName": {
"type": "string",
"minLength": 1,
"defaultValue" : "myblobconnectorgateway"
},
"logicAppName": {
"type": "string",
"minLength": 1,
"defaultValue" : "testinglogicapp"
},
"svcPlanName": {
"type": "string",
"minLength": 1,
"defaultValue" : "myresourcegrpserviceplan"
},
"sku": {
"type": "string",
"defaultValue": "Basic",
"allowedValues": [
"Free",
"Basic",
"Standard",
"Premium"
]
},
"svcPlanSize": {
"defaultValue": "0",
"type": "string",
"allowedValues": [
"0",
"1",
"2"
]
},
"gatewayToApiAppSecret": {
"defaultValue": "0000000000000000000000000000000000000000000000000000000000000000",
"type": "securestring"
}
},
"variables": {
"$packageId": "Microsoft.ApiApp",
"$nugetFeed": "http://apiapps-preview.nuget.org/api/v2/"
},
"resources": [
{
"type": "Microsoft.Web/serverfarms",
"apiVersion": "2015-04-01",
"name": "[parameters('svcPlanName')]",
"location": "[resourceGroup().location]",
"tags": {
"displayName": "AppServicePlan"
},
"properties": {
"name": "[parameters('svcPlanName')]",
"sku": "[parameters('sku')]",
"workerSize": "[parameters('svcPlanSize')]",
"numberOfWorkers": 1
}
},
{
"type": "Microsoft.Web/sites",
"apiVersion": "2015-04-01",
"name": "[parameters('gatewayName')]",
"location": "[resourceGroup().location]",
"kind": "gateway",
"tags": {
"displayName": "GatewayHost"
},
"resources": [
{
"type": "providers/links",
"apiVersion": "2015-01-01",
"name": "Microsoft.Resources/gateway",
"dependsOn": [
"[resourceId('Microsoft.Web/sites', parameters('gatewayName'))]"
],
"properties": {
"targetId": "[resourceId('Microsoft.AppService/gateways', parameters('gatewayName'))]"
}
}
],
"dependsOn": [
"[concat(resourceGroup().id, '/providers/Microsoft.Web/serverfarms/', parameters('svcPlanName'))]"
],
"properties": {
"name": "[parameters('gatewayName')]",
"gatewaySiteName": "[parameters('gatewayName')]",
"serverFarmId": "[resourceId('Microsoft.Web/serverfarms', parameters('svcPlanName'))]",
"siteConfig": {
"appSettings": [
{
"name": "ApiAppsGateway_EXTENSION_VERSION",
"value": "latest"
},
{
"name": "EmaStorage",
"value": "D:\\home\\data\\apiapps"
},
{
"name": "WEBSITE_START_SCM_ON_SITE_CREATION",
"value": "1"
}
]
}
}
},
{
"type": "Microsoft.AppService/gateways",
"apiVersion": "2015-03-01-preview",
"name": "[parameters('gatewayName')]",
"location": "[resourceGroup().location]",
"tags": {
"displayName": "Gateway"
},
"resources": [
{
"type": "providers/links",
"apiVersion": "2015-01-01",
"name": "Microsoft.Resources/gatewaySite",
"dependsOn": [
"[resourceId('Microsoft.AppService/gateways', parameters('gatewayName'))]"
],
"properties": {
"targetId": "[resourceId('Microsoft.Web/sites', parameters('gatewayName'))]"
}
},
{
"type": "tokens",
"apiVersion": "2015-03-01-preview",
"location": "[resourceGroup().location]",
"name": "[parameters('logicAppName')]",
"tags": {
"displayName": "AuthenticationToken"
},
"dependsOn": [
"[resourceId('Microsoft.AppService/gateways', parameters('gatewayName'))]"
]
}
],
"dependsOn": [
"[resourceId('Microsoft.Web/sites', parameters('gatewayName'))]"
],
"properties": {
"host": {
"resourceName": "[parameters('gatewayName')]"
}
}
},
{
"type": "Microsoft.Web/sites",
"apiVersion": "2015-04-01",
"name": "[parameters('blobConnectorName')]",
"location": "[resourceGroup().location]",
"kind": "apiApp",
"tags": {
"displayName": "APIAppHost",
"packageId": "AzureStorageBlobConnector"
},
"dependsOn": [
"[resourceId('Microsoft.Web/serverfarms', parameters('svcPlanName'))]",
"[resourceId('Microsoft.AppService/gateways', parameters('gatewayName'))]"
],
"resources": [
{
"type": "siteextensions",
"tags": {
"displayName": "APIAppExtension"
},
"apiVersion": "2015-02-01",
"name": "AzureStorageBlobConnector",
"dependsOn": [
"[resourceId('Microsoft.Web/sites', parameters('blobConnectorName'))]"
],
"properties": {
"type": "WebRoot",
"feed_url": "[variables('$nugetFeed')]"
}
},
{
"type": "providers/links",
"apiVersion": "2015-01-01",
"name": "Microsoft.Resources/apiApp",
"dependsOn": [
"[resourceId('Microsoft.Web/sites', parameters('blobConnectorName'))]"
],
"properties": {
"targetId": "[resourceId('Microsoft.AppService/apiapps', parameters('blobConnectorName'))]"
}
}
],
"properties": {
"gatewaySiteName": "[parameters('gatewayName')]",
"serverFarmId": "[resourceId('Microsoft.Web/serverfarms', parameters('svcPlanName'))]",
"siteConfig": {
"appSettings": [
{
"name": "EMA_MicroserviceId",
"value": "[parameters('blobConnectorName')]"
},
{
"name": "EMA_Secret",
"value": "[parameters('gatewayToAPIappSecret')]"
},
{
"name": "EMA_RuntimeUrl",
"value": "[concat('https://', reference(resourceId('Microsoft.Web/sites', parameters('gatewayName'))).hostNames[0])]"
},
{
"name": "WEBSITE_START_SCM_ON_SITE_CREATION",
"value": "1"
},
{
"name": "BlobConnector_ContainerUrl",
"value": "[concat('https://', parameters('blobStorageAccount'),'/',parameters('blobContainerName'))]"
},
{
"name": "BlobConnector_AccessKey",
"value": "[parameters('blobStorageKey')]"
}
],
"applicationLogs": {
"filesystem": {
"level": "Verbose"
},
"azureTableStorage": {
"level": "Off",
"sasUrl": null
},
"azureBlobStorage": {
"level": "Off",
"sasUrl": null,
"retentionInDays": null
}
}
}
}
},
{
"type": "Microsoft.AppService/apiapps",
"apiVersion": "2015-03-01-preview",
"name": "[parameters('blobConnectorName')]",
"location": "[resourceGroup().location]",
"tags": {
"displayName": "BlobConnector"
},
"resources": [
{
"type": "providers/links",
"apiVersion": "2015-01-01",
"name": "Microsoft.Resources/apiAppSite",
"dependsOn": [
"[resourceId('Microsoft.AppService/apiapps', parameters('blobConnectorName'))]"
],
"properties": {
"targetId": "[resourceId('Microsoft.Web/sites', parameters('blobConnectorName'))]"
}
}
],
"dependsOn": [
"[resourceId('Microsoft.Web/sites/siteextensions', parameters('blobConnectorName'), 'AzureStorageBlobConnector')]"
],
"properties": {
"package": {
"id": "AzureStorageBlobConnector"
},
"host": {
"resourceName": "[parameters('blobConnectorName')]"
},
"gateway": {
"resourceName": "[parameters('gatewayName')]"
},
"dependencies": [ ]
}
},
{
"type": "Microsoft.Logic/workflows",
"apiVersion": "2015-02-01-preview",
"name": "[parameters('logicAppName')]",
"location": "[resourceGroup().location]",
"tags": {
"displayName": "LogicApp"
},
"dependsOn": [
"[resourceId('Microsoft.AppService/apiApps', parameters('blobConnectorName'))]"
],
"properties": {
"sku": {
"name": "[parameters('sku')]",
"plan": {
"id": "[concat(resourceGroup().id, '/providers/Microsoft.Web/serverfarms/',parameters('svcPlanName'))]"
}
},
"definition": {
"$schema": "http://schema.management.azure.com/providers/Microsoft.Logic/schemas/2014-12-01-preview/workflowdefinition.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"token": {
"defaultValue": "[reference(resourceId('Microsoft.AppService/gateways/tokens', parameters('gatewayName'), parameters('logicAppName'))).token]",
"type": "String",
"metadata": {
"token": {
"name": "token"
}
}
},
"runworkflowmanually": {
"defaultValue": true,
"type": "Bool"
}
},
"triggers": { },
"actions": {
"azurestorageblobconnector": {
"type": "ApiApp",
"inputs": {
"apiVersion": "2015-01-14",
"host": {
"id": "[concat(resourceGroup().id, '/providers/Microsoft.AppService/apiApps/',parameters('blobConnectorName'))]",
"gateway": "[concat('https://', reference(resourceId('Microsoft.Web/sites', parameters('gatewayName'))).hostNames[0])]"
},
"operation": "UploadBlob",
"parameters": {
"BlobPath": "myfolder/test.txt",
"BlobContent": {
"Content": "TestMessage",
"ContentTransferEncoding": "None"
},
"Overwrite": true
},
"authentication": {
"type": "Raw",
"scheme": "Zumo",
"parameter": "#parameters('token')"
}
},
"conditions": [ ]
}
},
"outputs": {
}
},
"parameters": { }
}
}
]
}
Just search for "AzureStorageBlobConnector" in the above json to observe the usage which is the package id of the blob connector from marketplace. I found the package id for the blob connector from azure portal by deploying one manually and then checking its settings. Please feel free to post a comment for package id for other market place apps if you face any difficulty.

bigquery - Input contained no data

I'm testing bigquery platform with real traffic of my site (more than 80M of events by day).
I'm uploading gz files using java api, using insert jobs.
In some cases, i've receive this message: Input contained no data
{
"kind": "bigquery#job",
"etag": "\"******************\"",
"id": "*********",
"selfLink": "********",
"jobReference": {
"projectId": "********",
"jobId": "**************"
},
"configuration": {
"load": {
"schema": {
"fields": [
{
"name": "tms",
"type": "TIMESTAMP"
},
{
"name": "page",
"type": "STRING"
},
{
"name": "user_agent",
"type": "STRING"
},
{
"name": "print_id",
"type": "STRING"
},
{
"name": "referer",
"type": "STRING"
},
{
"name": "gtms",
"type": "TIMESTAMP"
},
{
"name": "cookies",
"type": "STRING"
},
{
"name": "ip",
"type": "STRING"
},
{
"name": "site",
"type": "STRING"
},
{
"name": "call_params",
"type": "STRING"
},
{
"name": "domains",
"type": "RECORD",
"mode": "REPEATED",
"fields": [
{
"name": "name",
"type": "STRING"
},
{
"name": "ads",
"type": "RECORD",
"mode": "REPEATED",
"fields": [
{
"name": "id",
"type": "STRING"
},
{
"name": "type",
"type": "STRING"
},
{
"name": "position",
"type": "STRING"
},
{
"name": "strategy",
"type": "STRING"
},
{
"name": "score",
"type": "STRING"
},
{
"name": "cpc",
"type": "STRING"
},
{
"name": "site",
"type": "STRING"
},
{
"name": "categ",
"type": "STRING"
},
{
"name": "cust",
"type": "STRING"
},
{
"name": "campaign",
"type": "STRING"
}
]
}
]
}
]
},
"destinationTable": {
"projectId": "**********",
"datasetId": "*******",
"tableId": "********"
},
"createDisposition": "CREATE_IF_NEEDED",
"writeDisposition": "WRITE_APPEND",
"sourceFormat": "NEWLINE_DELIMITED_JSON"
}
},
"status": {
"state": "DONE",
"errors": [
{
"reason": "invalid",
"message": "Input contained no data"
}
]
},
"statistics": {
"creationTime": "1416491042309",
"startTime": "1416491061440",
"endTime": "1416491076876",
"load": {
"inputFiles": "1",
"inputFileBytes": "0",
"outputRows": "0",
"outputBytes": "0"
}
}
}
And then of this, all my jobs return the same response.
Can anybody tell me what is the reason of this behaviour?
Thanks!!!!
Your job succeeded: there is no "errorResult" field in the status.
First, I understand this mistake: the return of errors and warnings in the job api is, frankly, as clear as mud.
Here's the quick overview:
status.errorResult is where job error is reported. If no errorResult is reported, the job succeeded.
status.errors is where individual errors and warnings are reported.
Please reference the documentation https://cloud.google.com/bigquery/docs/reference/v2/jobs and search for status.errorResult and status.errors.
Most people don't hit this problem since a job only encountering a warning is pretty rare.
Ok, the problem was very simple: the gz file.
Thanks!