I have a strange issue while querying from Presto (AWS EMR). I was using Presto 0.194 and everything was ok, after I upgraded to 0.224, I cannot run my queries. I'm using LDAP authentication for presto and also file base authorization for Hive using a authorization.json file. I'm using the same json file which was working fine in the old version. Any help would highly appreciated.
Error:
Query 20191005_104119_00006_3snge failed: Access Denied: View owner 'username' cannot create view that selects from ...
config.propertis:
coordinator=true
node-scheduler.include-coordinator=false
discovery.uri=http://IP.ap-southeast-1.compute.internal:8889
http-server.threads.max=500
discovery-server.enabled=true
sink.max-buffer-size=1GB
query.max-memory=30GB
query.max-memory-per-node=6532645258B
query.max-total-memory-per-node=7839174309B
query.max-history=40
query.min-expire-age=30m
http-server.http.port=8889
http-server.log.path=/var/log/presto/http-request.log
http-server.log.max-size=67108864B
http-server.log.max-history=5
log.max-size=268435456B
log.max-history=5
query.execution-policy=phased
optimizer.dictionary-aggregation=true
optimizer.optimize-metadata-queries=true
colocated-joins-enabled=true
http-server.authentication.type=PASSWORD
http-server.https.enabled=true
http-server.https.port=9443
http-server.https.keystore.path=/etc/presto/presto_keystore.jks
http-server.https.keystore.key=passw0rd
node-scheduler.max-splits-per-node=125
optimizer.use-mark-distinct=false
hive.properties:
hive.metastore-refresh-interval=1m
connector.name=hive-hadoop2
hive.metastore.uri=thrift://ip-10-0-2-141.ap-southeast-
1.compute.internal:9083
hive.metastore-cache-ttl=20m
hive.config.resources=/etc/hadoop/conf/core-
site.xml,/etc/hadoop/conf/hdfs-site.xml
hive.non-managed-table-writes-enabled = true
hive.s3-file-system-type = EMRFS
hive.hdfs.authentication.type = NONE
hive.hdfs.impersonation.enabled = true
hive.orc.bloom-filters.enabled=true
hive.recursive-directories=true
hive.s3select-pushdown.enabled=true
hive.security=file
security.config-file=/etc/presto/conf.dist/authorization.json
authorization.json:
{
"schemas": [
{
"user": "prestoSA",
"owner": true
},
{
"user": "marketing_jack",
"owner": true
},
{
"user": "system-apiquery",
"owner": true
},
{
"user": "redash",
"owner": true
},
{
"user": "system_.*",
"schema": "prestosync_.*",
"owner": true
},
{
"user": "system_.*",
"schema": "views_.*",
"owner": true
},
{
"user": "system_.*",
"schema": "raw_.*",
"owner": true
}
],
"tables": [
{
"user": "prestoSA",
"privileges": [
"SELECT",
"INSERT",
"DELETE",
"OWNERSHIP"
]
},
{
"user": "redash",
"privileges": [
"SELECT"
]
},
{
"schema": "raw_.*",
"user": "system_.*",
"privileges": [
"SELECT",
"INSERT",
"DELETE",
"OWNERSHIP"
]
},
{
"schema": "production_.*",
"user": "system_.*",
"privileges": [
"SELECT",
"INSERT",
"DELETE",
"OWNERSHIP"
]
},
{
"schema": "prestosync_.*",
"user": "system_.*",
"privileges": [
"SELECT",
"INSERT",
"DELETE",
"OWNERSHIP"
]
},
{
"schema": "views_.*",
"user": "system_.*",
"privileges": [
"SELECT",
"INSERT",
"DELETE",
"OWNERSHIP"
]
},
{
"schema": ".*dev",
"user": "developer_.*",
"privileges": [
"SELECT"
]
},
{
"schema": "raw_rin",
"user": "developer_.*",
"privileges": [
"SELECT"
]
},
{
"schema": ".*prod",
"user": "developer_.*",
"privileges": [
"SELECT"
]
},
{
"schema": "views_development_.*",
"user": "marketing_.*",
"privileges": [
"SELECT"
]
},
{
"schema": "views_prod",
"user": "marketing_.*",
"privileges": [
"SELECT"
]
},
{
"schema": "views_dev",
"user": "sales_.*",
"privileges": [
"SELECT"
]
},
{
"schema": "views_prod",
"user": "sales_.*",
"privileges": [
"SELECT"
]
},
{
"schema": "emr59_prod",
"user": "marketing_.*",
"privileges": [
"SELECT"
]
},
{
"schema": "views_dev",
"user": "management_.*",
"privileges": [
"SELECT"
]
},
{
"schema": "views_prod",
"user": "management_.*",
"privileges": [
"SELECT"
]
},
{
"schema": "views_dev",
"user": "management_.*",
"privileges": [
"SELECT"
]
},
{
"schema": "views_prod",
"user": "management_.*",
"privileges": [
"SELECT"
]
}
]
}
acceess-control.properties:
access-control.name=file
security.config-file=/etc/presto/conf.dist/rules.json
rules.json:
{
"catalogs": [
{
"user": "system_.*",
"catalog": "(mysql|system)",
"allow": true
},
{
"user": "prestoSA",
"catalog": "(mysql|system)",
"allow": true
},
{
"user": "redash",
"catalog": "(mysql|system)",
"allow": true
},
{
"user": "developer_.*",
"catalog": "(mysql|hive)",
"allow": true
},
{
"catalog": "hive",
"allow": true
},
{
"catalog": "system",
"allow": false
}
]
}
Error: Query 20191005_104119_00006_3snge failed: Access Denied: View owner 'username' cannot create view that selects from ...
This means that username does not have GRANT_SELECT privilege on a particular table or tables.
The particular change that affects you went in in 0.199 release:
https://github.com/prestosql/presto/commit/6ed1ed88083baef1d29171364297631962adf05d
This was a bug fix (creating view should require different privileges), so it is intentional (although inconvenient) that the change did not maintain backward compatibility.
BTW
For one-time troubleshooting-style questions which are unlikely to be beneficial for SO community I recommend using #troubleshooting channel on Presto Community Slack
Related
enter image description hereI am trying to create a JSON schema which is expecting inputs from 3 sources (file/Table/Kafka message), at a time any one of them will get the data from external source.
Issue :
I am using "anyOf" keyword in JSON but when I am applying "anyOf" and testing the negative scenario that is removing one of required field from the input properties, it's not giving any validation error, which ideally should have given that required field is missing.
**Expectation: **
Need to get input from any one of the source (file/table/message). If external party sends data in the file format, file portion from JSON schema should get executed, similarly if gets data from table or kafka message then relevant portion from JSON schema should get executed.
JSON SCHEMA CODE:
{
"$schema": "https://json-schema.org/draft/2019-09/schema",
"type": "object",
"title": "Job",
"properties": {
"unique_name": {
"type": "string",
"format": "regex",
"pattern": "[a-zA-Z0-9_]+"
},
"processing_instructions_file": {
"type": "string",
"examples": [
"/path/to/some/file/processing123.yaml"
]
},
"variables": {
"type": "array",
"additionalItems": true,
"items": {
"type": "object",
"properties": {
"name": {
"type": "string",
"examples": [
"environment_prefix"
]
},
"value": {
"type": "string",
"examples": [
"DEV"
]
}
},
"additionalProperties": true,
"required": [
"name",
"value"
]
}
},
"inputs": {
"type": "array",
"additionalItems": true,
"items": {
"type": "object",
"properties": {
"is_stream": {
"type": "boolean",
"examples": [
true
],
"default": false
}
},
"additionalProperties": true,
"anyOf": [
{
"type": "object",
"properties": {
"file": {
"type": "object",
"properties": {
"folder_structure": {
"type": "string",
"examples": [
"$environment_prefix\\$$yyyy-MMM-dd"
]
},
"naming_convention": {
"type": "string",
"examples": [
"STATIC_FILE_NAME:$$yyyy_MMM-dd"
]
},
"format type": {
"type": "string",
"enum": [
"CSV",
"EBCDIC"
],
"examples": [
"CSV"
]
},
"format_options": {
"type": "array",
"additionalItems": true,
"items": {
"type": "object",
"properties": {
"option_name": {
"type": "string",
"examples": [
"generate_record_id"
]
},
"option_value": {
"type": [
"string",
"boolean"
],
"examples": [
true
]
}
},
"additionalProperties": true,
"required": [
"option_name",
"option_value"
]
}
}
},
"additionalProperties": true,
"if": {
"properties": {
"format type": {
"type": "string",
"enum": [
"CSV",
"EBCDIC"
],
"examples": [
"CSV"
],
"const": "EBCDIC"
}
},
"required": [
"format type"
]
},
"then": {
"properties": {
"schema_location": {
"type": "string",
"examples": [
"some_path"
]
}
},
"required": [
"schema_location"
]
},
"else": {
"properties": {
"schema_location": {
"type": "string",
"examples": [
"some_path"
]
}
}
},
"required": [
"folder_structure",
"naming_convention",
"format type",
"format_options"
]
}
},
"additionalProperties": true
},
{
"type": "object",
"properties": {
"table": {
"type": "object",
"properties": {
"name": {
"type": "string",
"examples": [
"tbl_employees"
]
},
"jdbc_url": {
"type": "string",
"examples": [
"orcl#thinXXX"
]
},
"jdbc_library_jar": {
"type": "string",
"examples": [
"orcl.jar"
]
},
"schema": {
"type": "string",
"examples": [
"xxxxx"
]
},
"info_date_from_column_name": {
"type": "string",
"examples": [
"ASOF_DATE"
]
},
"info_date_to_column_name": {
"type": "string",
"examples": [
"END_DATE"
]
},
"primary_key_column_name": {
"type": "array",
"additionalItems": true,
"items": {
"type": "string"
}
}
},
"additionalProperties": true,
"required": [
"name",
"jdbc_url",
"schema"
]
}
},
"additionalProperties": true
},
{
"type": "object",
"properties": {
"message": {
"type": "object",
"properties": {
"topic_name": {
"type": "string",
"examples": [
"XXXXXX"
]
},
"group_id": {
"type": "string",
"examples": [
"XXXXX"
]
},
"port": {
"type": "integer",
"examples": [
8081
]
},
"kafka_cluster_ip": {
"type": "string",
"examples": [
"1.1.1.1"
]
},
"kafka_cluster_schema_registry": {
"type": "string",
"examples": [
"1.1.1.1"
]
}
},
"additionalProperties": true,
"required": [
"topic_name",
"group_id",
"port",
"kafka_cluster_ip",
"kafka_cluster_schema_registry"
]
}
},
"additionalProperties": true
}
]
}
},
"outputs": {
"type": "array",
"additionalItems": true,
"items": {
"type": "object",
"properties": {
"name": {
"type": "string",
"examples": [
"output_3"
]
},
"is_stream": {
"type": "boolean",
"examples": [
true
]
},
"spark_save_mode": {
"type": "string",
"examples": [
"append"
]
},
"file": {
"type": "object",
"properties": {
"root_path": {
"type": "string",
"examples": [
"some_path"
]
},
"folder_structure": {
"type": "string",
"examples": [
"$environment_prefix\\$$yyyy-MMM-dd"
]
},
"naming_convention": {
"type": "string",
"examples": [
"STATIC_FILE_NAME:$$yyyy_MMM-dd"
]
},
"partition_by": {
"type": "array",
"additionalItems": true,
"items": {
"type": "string"
}
},
"format": {
"type": "object",
"properties": {
"format_type": {
"type": "string",
"examples": [
"CSV"
]
},
"format_options": {
"type": "array",
"additionalItems": true,
"items": {
"type": "object",
"properties": {
"option_name": {
"type": "string",
"examples": [
"include_header"
]
},
"option_value": {
"type": [
"string",
"boolean"
],
"examples": [
true
]
}
},
"additionalProperties": true,
"required": [
"option_name",
"option_value"
]
}
}
},
"additionalProperties": true,
"required": [
"format_type",
"format_options"
]
}
},
"additionalProperties": true,
"required": [
"root_path",
"folder_structure",
"naming_convention",
"partition_by",
"format"
]
},
"mongo": {
"type": "object",
"properties": {
"mongo_spark_connector_options": {
"type": "array",
"additionalItems": true,
"items": {
"type": "object",
"properties": {
"option_name": {
"type": "string",
"examples": [
"format"
]
},
"option_value": {
"type": "string"
}
},
"additionalProperties": true,
"required": [
"option_name",
"option_value"
]
}
}
},
"additionalProperties": true,
"required": [
"mongo_spark_connector_options"
]
},
"table": {
"type": "object",
"properties": {
"name": {
"type": "string",
"examples": [
"tbl_employees"
]
},
"jdbc_url": {
"type": "string",
"examples": [
"orcl#thin:XXX"
]
},
"jdbc_library_jar": {
"type": "string",
"examples": [
"orcl.jar"
]
},
"schema": {
"type": "string",
"examples": [
"xxxxx"
]
},
"info_date_from_column_name": {
"type": "string",
"examples": [
"ASOF_DATE"
]
},
"info_date_to_column_name": {
"type": "string",
"examples": [
"END_DATE"
]
},
"primary_key_column_name": {
"type": "array",
"additionalItems": true,
"items": {
"type": "string"
}
}
},
"additionalProperties": true,
"required": [
"name",
"jdbc_url",
"jdbc_library_jar",
"schema",
"info_date_from_column_name",
"info_date_to_column_name",
"primary_key_column_name"
]
},
"message": {
"type": "object",
"properties": {
"topic_name": {
"type": "string",
"examples": [
"XXXXXX"
]
},
"group_id": {
"type": "string",
"examples": [
"XXXXX"
]
},
"port": {
"type": "integer",
"examples": [
8081
]
},
"kafka_cluster_ip": {
"type": "string",
"examples": [
"1.1.1.1"
]
},
"kafka_cluster_schema_registry": {
"type": "string",
"examples": [
"1.1.1.1"
]
}
},
"additionalProperties": true,
"required": [
"topic_name",
"group_id",
"port",
"kafka_cluster_ip",
"kafka_cluster_schema_registry"
]
}
},
"additionalProperties": true,
"required": [
"name",
"is_stream"
]
}
}
},
"additionalProperties": true,
"required": [
"unique_name",
"processing_instructions_file",
"inputs",
"outputs"
]
}
JSON DATA
{
"unique_name": "aXeTGImcM6PrCalRpGjLigbj1puXXvK",
"processing_instructions_file": "/path/to/some/file/processing123.yaml",
"variables": [
{
"name": "environment_prefix",
"value": "DEV"
}
],
"inputs": [
{
"is_stream": false,
"file": {
"folder_structure": "$environment_prefix\\$$yyyy-MMM-dd",
"naming_convention": "STATIC_FILE_NAME:$$yyyy_MMM-dd",
"format type": "CSV",
"format_options": [
{
"option_name": "generate_record_id",
"option_value": "Lorem"
}
],
"schema_location": "some_path"
}
}
],
"outputs": [
{
"name": "output_3",
"is_stream": true,
"spark_save_mode": "append",
"file": {
"root_path": "some_path",
"folder_structure": "$environment_prefix\\$$yyyy-MMM-dd",
"naming_convention": "STATIC_FILE_NAME:$$yyyy_MMM-dd",
"partition_by": [
"Lorem"
],
"format": {
"format_type": "CSV",
"format_options": [
{
"option_name": "include_header",
"option_value": "Lorem"
}
]
}
},
"mongo": {
"mongo_spark_connector_options": [
{
"option_name": "format",
"option_value": "Lorem"
}
]
},
"table": {
"name": "tbl_employees",
"jdbc_url": "orcl#thin:XXX",
"jdbc_library_jar": "orcl.jar",
"schema": "xxxxx",
"info_date_from_column_name": "ASOF_DATE",
"info_date_to_column_name": "END_DATE",
"primary_key_column_name": [
"Lorem"
]
},
"message": {
"topic_name": "XXXXXX",
"group_id": "XXXXX",
"port": 8081,
"kafka_cluster_ip": "1.1.1.1",
"kafka_cluster_schema_registry": "1.1.1.1"
}
}
]
}
I am trying to create Delivery plan in the Azure Devops project using Azure Devops REST Apis. I have used following method to create the same.
https://learn.microsoft.com/en-us/rest/api/azure/devops/work/plans/create?view=azure-devops-rest-6.0
POST https://dev.azure.com/{organization}/{project}/_apis/work/plans?api-version=6.0
and I am sending following data in the request body properties
{
"properties": {
"teamBacklogMappings": [
{
"teamId": "09d57738-697f-4433-abdd-b80a2bc6337b",
"categoryReferenceName": "Microsoft.RequirementCategory"
},
{
"teamId": "5df45eec-4108-474a-8d93-bc09c0b9037e",
"categoryReferenceName": "Microsoft.RequirementCategory"
},
{
"teamId": "e8ed402b-68e7-4140-96f2-07790a08788b",
"categoryReferenceName": "Microsoft.RequirementCategory"
},
{
"teamId": "14425694-efa5-454e-811c-e9e03d79198f",
"categoryReferenceName": "Microsoft.RequirementCategory"
},
{
"teamId": "b02690e9-1f48-421a-a918-3b23cc9a1b73",
"categoryReferenceName": "Microsoft.RequirementCategory"
}
],
"cardSettings": {
"fields": {
"showId": true,
"showAssignedTo": true,
"assignedToDisplayFormat": "avatarOnly",
"showState": true,
"showTags": true,
"showParent": false,
"showEmptyFields": true,
"showChildRollup": true,
"additionalFields": null,
"coreFields": [
{
"referenceName": "System.AssignedTo",
"displayName": "Assigned To",
"fieldType": "string",
"isIdentity": true
},
{
"referenceName": "System.Id",
"displayName": "ID",
"fieldType": "integer",
"isIdentity": false
},
{
"referenceName": "System.State",
"displayName": "State",
"fieldType": "string",
"isIdentity": false
},
{
"referenceName": "System.Tags",
"displayName": "Tags",
"fieldType": "plainText",
"isIdentity": false
}
]
}
},
"markers": [],
"styleSettings": [
{
"name": "BLOCKER",
"isEnabled": "True",
"filter": "[System.Tags] CONTAINS 'BLOCKER'",
"clauses": [
{
"fieldName": "System.Tags",
"logicalOperator": "AND",
"operator": "CONTAINS",
"value": "BLOCKER"
}
],
"settings": {
"background-color": "#E60017",
"title-color": "#000000"
}
},
{
"name": "New",
"isEnabled": "True",
"filter": "[System.State] = 'New'",
"clauses": [
{
"fieldName": "System.State",
"logicalOperator": "AND",
"operator": "=",
"value": "New"
}
],
"settings": {
"background-color": "#AAAAAA",
"title-color": "#000000"
}
},
{
"name": "Dev Completed",
"isEnabled": "True",
"filter": "[System.State] = 'Development Completed'",
"clauses": [
{
"fieldName": "System.State",
"logicalOperator": "AND",
"operator": "=",
"value": "Development Completed"
}
],
"settings": {
"background-color": "#D7E587",
"title-color": "#000000"
}
},
{
"name": "Deployed to QA",
"isEnabled": "True",
"filter": "[System.State] = 'Deployed to QA (SIT)'",
"clauses": [
{
"fieldName": "System.State",
"logicalOperator": "AND",
"operator": "=",
"value": "Deployed to QA (SIT)"
}
],
"settings": {
"background-color": "#C3D84C",
"title-color": "#000000"
}
},
{
"name": "Deployed to UAT",
"isEnabled": "True",
"filter": "[System.State] = 'Deployed to UAT'",
"clauses": [
{
"fieldName": "System.State",
"logicalOperator": "AND",
"operator": "=",
"value": "Deployed to UAT"
}
],
"settings": {
"background-color": "#60AF49",
"title-color": "#000000"
}
},
{
"name": "Deployed to PROD",
"isEnabled": "True",
"filter": "[System.State] = 'Completed'",
"clauses": [
{
"fieldName": "System.State",
"logicalOperator": "AND",
"operator": "=",
"value": "Completed"
}
],
"settings": {
"background-color": "#00643A",
"title-color": "#000000"
}
}
],
"tagStyleSettings": []
}
}
However Styling rules are not getting created in the project.
Got this done by using Update for the Delivery plan immediately after creating the same. Update it with same properties though you will have to add revision property to the request body.
https://learn.microsoft.com/en-us/rest/api/azure/devops/work/plans/update?view=azure-devops-rest-6.0
https://dev.azure.com/{organization}/{project}/_apis/work/plans/{id}?api-version=6.0
Create Delivery Plan doesn't accept style settings, from the UI you can notice this:
Just get the plan id and the response from the Create API and then use them in the Update API, this is the only way.
The following ARM template deploys: Virtual Network, Network Security Group, Storage Account, App Service Plan, Function App
When the settings for WEBSITE_CONTENTAZUREFILECONNECTIONSTRING and WEBSITE_CONTENTSHARE are omitted (commented out) the deployment succeeds but the function app configuration shows a warning.
When enabling the two settings, the deployment fails with a 403 Forbidden message.
New-AzResourceGroupDeployment : 17:04:05 - The deployment '20201209-170356' failed with error(s). Showing 1 out of 1 error(s).
Status Message: There was a conflict. The remote server returned an error: (403) Forbidden. (Code: BadRequest)
- There was a conflict. The remote server returned an error: (403) Forbidden. (Code:)
- (Code:BadRequest)
- (Code:)
CorrelationId: ec11767b-9f8f-4722-acca-e751e5c1bbe8
I have tried numerous settings on the NSG, adding service tags, allowing IPs associated with the function app. I have also tried allowing IPRules on the storage account firewall. The only setting that worked was to entirely disable the storage account firewall with 'Allow access from all networks', which is not an acceptable setting for the network.
The ARM template to demonstrate the error:
{
"$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {
},
"variables": {
"vnetName": "vnet1a",
"addressPrefixVnet": "10.17.0.0/20",
"addressPrefixSubnet": "10.17.4.0/24",
"nsgName_sb_functionapp": "[concat(variables('vnetName'), '-sb-functionapp-nsg')]",
"storageAccountName": "[concat(uniquestring(resourceGroup().id), 'sa1a')]",
"appServicePlanName": "[concat(uniquestring(resourceGroup().id), 'asp1a')]",
"functionAppName": "[concat(uniquestring(resourceGroup().id), 'asp1a')]"
},
"resources": [
{
"type": "Microsoft.Network/networkSecurityGroups",
"apiVersion": "2019-11-01",
"name": "[variables('nsgName_sb_functionapp')]",
"location": "[resourceGroup().location]",
"tags": {
"Purpose": "Function App"
},
"properties": {
"securityRules": []
}
},
{
"type": "Microsoft.Network/virtualNetworks",
"apiVersion": "2019-11-01",
"name": "[variables('vnetName')]",
"location": "[resourceGroup().location]",
"dependsOn": [
"[resourceId('Microsoft.Network/networkSecurityGroups', variables('nsgName_sb_functionapp'))]"
],
"tags": {
"Purpose": "Debug Function App and Storage Account Connectivity"
},
"properties": {
"addressSpace": {
"addressPrefixes": [
"[variables('addressPrefixVnet')]"
]
},
"subnets": [
{
"name": "sb-functionapp",
"properties": {
"addressPrefix": "[variables('addressPrefixSubnet')]",
"networkSecurityGroup": {
"id": "[resourceId('Microsoft.Network/networkSecurityGroups', variables('nsgName_sb_functionapp'))]"
},
"serviceEndpoints": [
{
"service": "Microsoft.Storage",
"locations": [
"*"
]
}
],
"delegations": [
{
"name": "delegation",
"properties": {
"serviceName": "Microsoft.Web/serverFarms"
}
}
],
"privateEndpointNetworkPolicies": "Enabled",
"privateLinkServiceNetworkPolicies": "Enabled"
}
}
],
"enableDdosProtection": false,
"enableVmProtection": false
}
},
{
"type": "Microsoft.Storage/storageAccounts",
"apiVersion": "2019-04-01",
"name": "[variables('storageAccountName')]",
"location": "[resourceGroup().location]",
"tags": {
"Purpose": "Debug Function App and Storage Account Connectivity"
},
"kind": "StorageV2",
"sku": {
"name": "Standard_GRS",
"tier": "Standard"
},
"properties": {
"networkAcls": {
"defaultAction": "Deny",
"bypass": "AzureServices",
"supportsHttpsTrafficOnly": true,
"ipRules": [],
"encryption": {
"keySource": "Microsoft.Storage",
"services": {
"file": {
"enabled": true
},
"blob": {
"enabled": true
}
}
},
"accessTier": "Hot",
"virtualNetworkRules": [
{
"id": "[concat(resourceId('Microsoft.Network/virtualNetworks', variables('vnetName')), '/subnets/sb-functionapp')]",
"ignoreMissingVNetServiceEndpoint": false
}
]
}
}
},
{
"type": "Microsoft.Web/serverfarms",
"apiVersion": "2018-02-01",
"name": "[variables('appServicePlanName')]",
"location": "[resourceGroup().location]",
"tags": {
"Purpose": "Debug Function App and Storage Account Connectivity"
},
"sku": {
"name": "EP1",
"tier": "ElasticPremium",
"size": "EP1",
"family": "EP",
"capacity": 1
},
"kind": "elastic",
"properties": {
"perSiteScaling": false,
"maximumElasticWorkerCount": 20,
"isSpot": false,
"reserved": false,
"isXenon": false,
"hyperV": false,
"targetWorkerCount": 0,
"targetWorkerSizeId": 0
}
},
{
"type": "Microsoft.Web/sites",
"apiVersion": "2018-11-01",
"name": "[variables('functionAppName')]",
"location": "[resourceGroup().location]",
"dependsOn": [
"[resourceId('Microsoft.Web/serverfarms', variables('appServicePlanName'))]"
],
"tags": {
"Purpose": "Debug Function App and Storage Account Connectivity"
},
"kind": "functionapp",
"properties": {
"enabled": true,
"hostNameSslStates": [
{
"name": "[concat(variables('functionAppName'), '.azurewebsites.net')]",
"sslState": "Disabled",
"hostType": "Standard"
},
{
"name": "[concat(variables('functionAppName'), '.scm.azurewebsites.net')]",
"sslState": "Disabled",
"hostType": "Repository"
}
],
"serverFarmId": "[resourceId('Microsoft.Web/serverfarms', variables('appServicePlanName'))]",
"reserved": false,
"isXenon": false,
"hyperV": false,
"scmSiteAlsoStopped": false,
"clientAffinityEnabled": true,
"clientCertEnabled": false,
"hostNamesDisabled": false,
"containerSize": 1536,
"dailyMemoryTimeQuota": 0,
"httpsOnly": true,
"redundancyMode": "None",
"siteConfig": {
"appSettings": [
{
"name": "FUNCTIONS_EXTENSION_VERSION",
"value": "~1"
},
{
"name": "WEBSITE_CONTENTAZUREFILECONNECTIONSTRING",
"value": "[concat('DefaultEndpointsProtocol=https;AccountName=', variables('storageAccountName'), ';AccountKey=', listKeys(resourceId('Microsoft.Storage/storageAccounts', variables('storageAccountName')), '2019-04-01').keys[0].value)]"
},
{
"name": "WEBSITE_CONTENTSHARE",
"value": "[variables('functionAppName')]"
},
{
"name": "WEBSITE_DNS_SERVER",
"value": "168.63.129.16"
},
{
"name": "WEBSITE_VNET_ROUTE_ALL",
"value": "1"
}
]
}
},
"resources": [
{
"type": "networkConfig",
"apiVersion": "2018-11-01",
"name": "virtualNetwork",
"location": "[resourceGroup().location]",
"dependsOn": [
"[resourceId('Microsoft.Web/sites', variables('functionAppName'))]"
],
"properties": {
"subnetResourceId": "[concat(resourceId('Microsoft.Network/virtualNetworks', variables('vnetName')), '/subnets/sb-functionapp')]",
"swiftSupported": true
}
}
]
},
{
"type": "Microsoft.Web/sites/config",
"apiVersion": "2018-11-01",
"name": "[concat(variables('functionAppName'), '/web')]",
"location": "[resourceGroup().location]",
"dependsOn": [
"[resourceId('Microsoft.Web/sites', variables('functionAppName'))]"
],
"tags": {
"Purpose": "Debug Function App and Storage Account Connectivity"
},
"properties": {
"numberOfWorkers": 1,
"defaultDocuments": [
"Default.htm",
"Default.html",
"Default.asp",
"index.htm",
"index.html",
"iisstart.htm",
"default.aspx",
"index.php"
],
"netFrameworkVersion": "v4.0",
"phpVersion": "5.6",
"requestTracingEnabled": false,
"remoteDebuggingEnabled": false,
"remoteDebuggingVersion": "VS2019",
"httpLoggingEnabled": false,
"logsDirectorySizeLimit": 35,
"detailedErrorLoggingEnabled": false,
"publishingUsername": "[concat('$', variables('functionAppName'))]",
"scmType": "VSTSRM",
"use32BitWorkerProcess": true,
"webSocketsEnabled": false,
"alwaysOn": false,
"managedPipelineMode": "Integrated",
"virtualApplications": [
{
"virtualPath": "/",
"physicalPath": "site\\wwwroot",
"preloadEnabled": true
}
],
"loadBalancing": "LeastRequests",
"experiments": {
"rampUpRules": [
]
},
"autoHealEnabled": false,
"cors": {
"allowedOrigins": [],
"supportCredentials": false
},
"localMySqlEnabled": false,
"ipSecurityRestrictions": [],
"scmIpSecurityRestrictions": [
{
"ipAddress": "Any",
"action": "Allow",
"priority": 1,
"name": "Allow all",
"description": "Allow all access"
}
],
"scmIpSecurityRestrictionsUseMain": false,
"http20Enabled": false,
"minTlsVersion": "1.2",
"ftpsState": "AllAllowed",
"reservedInstanceCount": 1
}
}
]
}
Command to deploy to existing resource group:
New-AzResourceGroupDeployment -Name (Get-Date).ToString('yyyyMMdd-HHmmss') -ResourceGroupName 'Test-FunctionApp-Storage-VNet' -TemplateFile .\DebugFunctionApp.json -Verbose
I have seen the question/answer at Function App Deployment Failed - The remote server returned an error: (403) Forbidden but it doesn't solve the problem I see.
The solution is to add another setting named WEBSITE_CONTENTOVERVNET and to set the value to "1".
The updated appSettings section looks like:
"siteConfig": {
"appSettings": [
{
"name": "FUNCTIONS_EXTENSION_VERSION",
"value": "~1"
},
{
"name": "WEBSITE_CONTENTAZUREFILECONNECTIONSTRING",
"value": "[concat('DefaultEndpointsProtocol=https;AccountName=', variables('storageAccountName'), ';AccountKey=', listKeys(resourceId('Microsoft.Storage/storageAccounts', variables('storageAccountName')), '2019-04-01').keys[0].value)]"
},
{
"name": "WEBSITE_CONTENTOVERVNET",
"value": "1"
},
{
"name": "WEBSITE_CONTENTSHARE",
"value": "[variables('functionAppName')]"
},
{
"name": "WEBSITE_DNS_SERVER",
"value": "168.63.129.16"
},
{
"name": "WEBSITE_VNET_ROUTE_ALL",
"value": "1"
}
]
}
The setting is document at https://learn.microsoft.com/en-us/azure/azure-functions/functions-app-settings#website_contentovervnet
For Premium plans only. A value of 1 enables your function app to scale when you have your storage account restricted to a virtual network. You should enable this setting when restricting your storage account to a virtual network.
I am trying to move the index.html to a subfolder (in this case a folder called "app") but I am unable to make it to work with CLI. Here is my aurelia.json
{
"name": "lv-app",
"type": "project:application",
"bundler": {
"id": "cli",
"displayName": "Aurelia-CLI"
},
"build": {
"targets": [
{
"id": "web",
"displayName": "Web",
"output": "app/scripts"
}
],
"options": {
"minify": "stage & prod",
"sourcemaps": "dev & stage"
},
"bundles": [
{
"name": "app-bundle.js",
"source": [
"[**/*.js]",
"**/*.{css,html}"
]
},
{
"name": "vendor-bundle.js",
"prepend": [
"node_modules/requirejs/require.js"
],
"dependencies": [
// dependencies omitted
]
}
],
"loader": {
"type": "require",
"configTarget": "vendor-bundle.js",
"includeBundleMetadataInConfig": "auto",
"plugins": [
{
"name": "text",
"extensions": [
".html",
".css"
],
"stub": true
}
]
}
},
"platform": {
"id": "web",
"displayName": "Web",
"baseDir": "./app"
},
"transpiler": {
"id": "babel",
"displayName": "Babel",
"fileExtension": ".js",
"options": {
"plugins": [
"transform-es2015-modules-amd"
]
},
"source": "src/**/*.js"
},
"markupProcessor": {
"id": "none",
"displayName": "None",
"fileExtension": ".html",
"source": "src/**/*.html"
},
"cssProcessor": {
"id": "postcss",
"displayName": "PostCSS",
"fileExtension": ".css",
"source": "src/**/*.css"
},
"paths": {
"root": "src",
"resources": "resources",
"elements": "resources/elements",
"attributes": "resources/attributes",
"valueConverters": "resources/value-converters",
"bindingBehaviors": "resources/binding-behaviors"
}
}
Al first I was having this error:
ERROR [Bundle] Error: Unable to update vendor-bundle path to
app/scripts/vendor-bundle.js, could not find existing reference to
replace
EDIT
But I had configured the baseDir incorrectly.
After changing "baseDir": "app/" for "baseDir": ". /app" I'm getting a different error. When I load the web in localhost the app-bundle.js doesn't load apparently because it's pointing to the wrong route as can be seen in the next picture
As you can see in the next picture the vendor-bundle.js is loading from localhost:9000/scripts no localhost:9000/app/scripts/
I have created an elasticsearch index. It searches everything correctly but when any new document is added it does not index it automatically. Everytime I have to restart the service for it to index the new document.
PUT /_river/mydocum1/_meta
{
"type": "fs",
"fs": {
"url": "\\\\file\\datum\\Depart\\Inet\\",
"update_rate": 3600,
"includes": [
"*.docx",
"*.xlsx",
"*.pdf",
"*.txt",
"*.doc",
"*.xls"
]
},
"index": {
"index": "test",
"type": "docum",
"bulk_size": 500
}
}
EDIT:
Here are the settings
{
"test": {
"settings": {
"index": {
"uuid": "9rofjAZySByrtehnjemkjudg",
"analysis": {
"filter": {
"synonym_filter": {
"type": "synonym",
"synonyms_path": "analysis/main_synonym.txt"
},
"nGram_filter": {
"max_gram": "20",
"min_gram": "2",
"type": "nGram",
"token_chars": [
"letter",
"digit",
"punctuation",
"symbol"
]
}
},
"analyzer": {
"search_analyzer_2": {
"type": "custom",
"filter": [
"lowercase",
"asciifolding",
"synonym_filter"
],
"tokenizer": "standard"
},
"index_analyzer": {
"type": "custom",
"filter": [
"lowercase",
"asciifolding",
"nGram_filter"
],
"tokenizer": "whitespace"
},
"search_analyzer": {
"type": "custom",
"filter": [
"lowercase",
"asciifolding",
"synonym_filter"
],
"tokenizer": "whitespace"
},
"index_analyzer_2": {
"type": "custom",
"filter": [
"lowercase",
"asciifolding"
],
"tokenizer": "whitespace"
},
"index_analyzer_3": {
"type": "custom",
"filter": [
"lowercase",
"asciifolding"
],
"tokenizer": "standard"
},
"nGram_analyzer": {
"type": "custom",
"filter": [
"lowercase",
"asciifolding",
"nGram_filter"
],
"tokenizer": "whitespace"
},
"index_analyzer_4": {
"type": "custom",
"filter": [
"lowercase",
"asciifolding"
],
"tokenizer": "keyword"
},
"whitespace_analyzer": {
"type": "custom",
"filter": [
"lowercase",
"asciifolding"
],
"tokenizer": "whitespace"
}
}
}
}
}
Does anyone know how do I solve it?