I am trying to create an index in elasticsearch with the following settings. I am facing the following error
error:
RemoteTransportException[[ys2order-stg2-01][inet[/64.101.206.15:9300]][indices:admin/mapping/put]];
nested: MapperParsingException[Analyzer [autocomplete] not found for
field [DESCRIPTION_AUTO]]; status: 400
{
"settings": {
"index": {
"analysis": {
"filter": {
"autocomplete_filter": {
"type": "edge_ngram",
"min_gram": 1,
"max_gram": 20
}
},
"analyzer": {
" autocomplete": {
"type": "custom",
"filter": [
"lowercase",
"autocomplete_filter"
],
"tokenizer": "standard"
}
}
},
"refresh_interval": -1,
"number_of_replicas": 1,
"number_of_shards": 4,
"index_analyzer": "default_index",
"search_analyzer": "default_search"
}
},
"mappings": {
"itemsnew": {
"properties": {
"DESCRIPTION_AUTO": {
"index_analyzer": "autocomplete",
"search_analyzer": "standard",
"type": "string"
}
}
}
}
}
You need to remove a space in your analyzer name:
"analyzer": {
" autocomplete": { <--- remove initial space here
"type": "custom",
"filter": [
"lowercase",
"autocomplete_filter"
],
"tokenizer": "standard"
}
....
"DESCRIPTION_AUTO": {
"index_analyzer": "autocomplete", <-- because there is no initial space here !!!
"search_analyzer": "standard",
"type": "string"
}
Related
I need to set datatype for Additional Column with Dynamic Content in Sink in ADF
By default its taking nvarchar(max) from Json obj but I need bigInt
Below is a Json Obj which create table with Additional column
{
"source": {
"type": "SqlServerSource",
"additionalColumns": [
{
"name": "ApplicationId",
"value": 3604509277250831000
}
],
"sqlReaderQuery": "SELECT * from Table A",
"queryTimeout": "02:00:00",
"isolationLevel": "ReadUncommitted",
"partitionOption": "None"
},
"sink": {
"type": "AzureSqlSink",
"writeBehavior": "insert",
"sqlWriterUseTableLock": false,
"tableOption": "autoCreate",
"disableMetricsCollection": false
},
"enableStaging": false,
"translator": {
"type": "TabularTranslator",
"typeConversion": true,
"typeConversionSettings": {
"allowDataTruncation": true,
"treatBooleanAsNumber": false
}
}
}
ADF Configuration
After create table Database - column with datatype
If I convert Dynamic content into Int
#int(pipeline().parameters.application.applicationId)
Then getting below warning
Please let me know how can I set Datatype in ADF
I also tried the same and getting same result.
By default its taking nvarchar(max) from Json obj but I need bigInt
To resolve this when you add additional column in your source data set and in Mapping click onimport schema it will import the schema of the source and also give you additional column in schema you have to change the type of the column as Int64 as shown in below image. in below image you can see after name there is additional means it is an additional column.
After this run your pipeline, It will create additional column with data type bigint .
{
"name": "pipeline2",
"properties": {
"activities": [
{
"name": "Copy data1",
"type": "Copy",
"dependsOn": [],
"policy": {
"timeout": "0.12:00:00",
"retry": 0,
"retryIntervalInSeconds": 30,
"secureOutput": false,
"secureInput": false
},
"userProperties": [],
"typeProperties": {
"source": {
"type": "JsonSource",
"additionalColumns": [
{
"name": "name",
"value": {
"value": "#pipeline().parameters.demo.age",
"type": "Expression"
}
}
],
"storeSettings": {
"type": "AzureBlobFSReadSettings",
"recursive": true,
"enablePartitionDiscovery": false
},
"formatSettings": {
"type": "JsonReadSettings"
}
},
"sink": {
"type": "AzureSqlSink",
"writeBehavior": "insert",
"sqlWriterUseTableLock": false,
"tableOption": "autoCreate",
"disableMetricsCollection": false
},
"enableStaging": false,
"translator": {
"type": "TabularTranslator",
"mappings": [
{
"source": {
"path": "$['taskId']"
},
"sink": {
"name": "taskId",
"type": "String"
}
},
{
"source": {
"path": "$['taskObtainedScore']"
},
"sink": {
"name": "taskObtainedScore",
"type": "String"
}
},
{
"source": {
"path": "$['multiInstance']"
},
"sink": {
"name": "multiInstance",
"type": "String"
}
},
{
"source": {
"path": "$['name']"
},
"sink": {
"name": "name",
"type": "Int64"
}
}
],
"collectionReference": ""
}
},
"inputs": [
{
"referenceName": "Json1",
"type": "DatasetReference"
}
],
"outputs": [
{
"referenceName": "AzureSqlTable1",
"type": "DatasetReference"
}
]
}
],
"parameters": {
"demo": {
"type": "object",
"defaultValue": {
"name": "John",
"age": 30,
"isStudent": true
}
}
},
"annotations": []
}
}
OUTPUT:
We are currently using GA4 data API and faced the issue when custom dimensions returns value "(not set)".
We were using the following article to set custom dimension for the session count, but we still receiving "(not set)" values.
Example of request:
{
"dateRanges": [
{
"startDate": "2021-09-01",
"endDate": "2021-09-05"
}
],
"offset": 0,
"limit": 100,
"dimensionFilter": {
"filter": {
"fieldName": "eventName",
"stringFilter": {
"matchType": 1,
"value": "screen_view",
"caseSensitive": true
}
}
},
"dimensions": [
{
"name": "customUser:applicationID"
},
{
"name": "customEvent:ga_session_number"
},
{
"name": "dateHour"
},
{
"name": "platform"
},
{
"name": "sessionSource"
},
{
"name": "sessionMedium"
},
{
"name": "sessionCampaignName"
},
{
"name": "deviceCategory"
}
],
"metrics": [
{
"name": "userEngagementDuration"
}
]
}
Does anybody have any idea why it may happen?
I have been trying to validate my request parameters using x-amazon-apigateway-request-validator, but unfortunately it is not working. Below is swagger file-
{
"swagger": "2.0",
"info": {
"title": "API Gateway - Request Validation Demo"
},
"schemes": [
"https"
],
"produces": [
"application/json"
],
"x-amazon-apigateway-request-validators" : {
"full" : {
"validateRequestBody" : true,
"validateRequestParameters" : true
},
"body-only" : {
"validateRequestBody" : true,
"validateRequestParameters" : false
}
},
"x-amazon-apigateway-request-validator" : "full",
"paths": {
"/orders": {
"post": {
"x-amazon-apigateway-request-validator": "body-only",
"parameters": [
{
"in": "body",
"name": "CreateOrders",
"required": true,
"schema": {
"$ref": "#/definitions/CreateOrders"
}
}
],
"responses": {
"200": {
"schema": {
"$ref": "#/definitions/Message"
}
},
"400" : {
"schema": {
"$ref": "#/definitions/Message"
}
}
},
"x-amazon-apigateway-integration": {
"responses": {
"default": {
"statusCode": "200",
"responseTemplates": {
"application/json": "{\"message\" : \"Orders successfully created\"}"
}
}
},
"requestTemplates": {
"application/json": "{\"statusCode\": 200}"
},
"passthroughBehavior": "never",
"type": "mock"
}
},
"get": {
"x-amazon-apigateway-request-validator": "full",
"parameters": [
{
"in": "header",
"name": "Account-Id",
"required": true
},
{
"in": "query",
"name": "type",
"required": false,
"schema": {
"$ref": "#/definitions/InputOrders"
}
}
],
"responses": {
"200" : {
"schema": {
"$ref": "#/definitions/Orders"
}
},
"400" : {
"schema": {
"$ref": "#/definitions/Message"
}
}
},
"x-amazon-apigateway-integration": {
"responses": {
"default": {
"statusCode": "200",
"responseTemplates": {
"application/json": "[{\"order-id\" : \"qrx987\",\n \"type\" : \"STOCK\",\n \"symbol\" : \"AMZN\",\n \"shares\" : 100,\n \"time\" : \"1488217405\",\n \"state\" : \"COMPLETED\"\n},\n{\n \"order-id\" : \"foo123\",\n \"type\" : \"STOCK\",\n \"symbol\" : \"BA\",\n \"shares\" : 100,\n \"time\" : \"1488213043\",\n \"state\" : \"COMPLETED\"\n}\n]"
}
}
},
"requestTemplates": {
"application/json": "{\"statusCode\": 200}"
},
"passthroughBehavior": "never",
"type": "mock"
}
}
}
},
"definitions": {
"CreateOrders": {
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "Create Orders Schema",
"type": "array",
"minItems" : 1,
"items": {
"type": "object",
"$ref" : "#/definitions/Order"
}
},
"Orders" : {
"type": "array",
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "Get Orders Schema",
"items": {
"type": "object",
"properties": {
"order_id": { "type": "string" },
"time" : { "type": "string" },
"state" : {
"type": "string",
"enum": [
"PENDING",
"COMPLETED"
]
},
"order" : {
"$ref" : "#/definitions/Order"
}
}
}
},
"Order" : {
"type": "object",
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "Schema for a single Order",
"required": [
"account-id",
"type",
"symbol",
"shares",
"details"
],
"properties" : {
"account-id": {
"type": "string",
"pattern": "[A-Za-z]{6}[0-9]{6}"
},
"type": {
"type" : "string",
"enum" : [
"STOCK",
"BOND",
"CASH"]
},
"symbol" : {
"type": "string",
"minLength": 1,
"maxLength": 4
},
"shares": {
"type": "number",
"minimum": 1,
"maximum": 1000
},
"details": {
"type": "object",
"required": [
"limit"
],
"properties": {
"limit": {
"type": "number"
}
}
}
}
},
"InputOrder" : {
"type": "object",
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "Schema for a Input Order",
"required": [
"type"
],
"properties" : {
"type": {
"type" : "string",
"enum" : [
"STOCK",
"BOND",
"CASH"]
}
}
},
"Message": {
"type": "object",
"properties": {
"message" : {
"type" : "string"
}
}
}
}
}
I am trying to validate my request parameters against some regex and enum values.
I am not sure if this is even possible or not. Can anybody please help me with this?
For HTTP parameter validation, API Gateway only supports marking one as 'required'. It does not support regex/enum values for parameters.
I have two structures very similar to pests-owners, this is my structure:
{
"name": "Contratti",
"fields": {
"rischicontratti": {
"collection": "RischiContratti",
"via": "Contratti"
},
"intervento": {
"collection": "Interventi",
"via": "Contratto"
},
"Numero": {
"type": "string"
},
"Impresa": {
"object": "Imprese"
},
"Oggetto": {
"type": "string"
}
},
{
"name": "RischiContratti",
"fields": {
"Contratti": {
"object": "Contratti"
},
"Rischio": {
"object": "Rischi"
}
}
},
{
"name": "Rischi",
"fields": {
"rischicontratti": {
"collection": "RischiContratti",
"via": "Rischio"
},
"rischi_interventi": {
"collection": "rischi_interventi",
"via": "Rischio"
},
"Rischio": {
"type": "string"
}
}
When I'm inserting a Contratto I want to also insert all related Rischi. Is there a way to post all in one call? Or do I have to post first the Contratto and then loop each Rischi?
I have a situation where I'll need to import a bunch of different data that may end up having conflicting data types. I have decided to convert everything to a string and then convert back later if the data is needed. I can't figure out how to do this with Elasticsearches (ES) dynamic mapping using the javascript client.
What ES says in their docs:
{
"mappings": {
"my_type": {
"dynamic_templates": [
{ "es": {
"match": "*_es",
"match_mapping_type": "string",
"mapping": {
"type": "string",
"analyzer": "spanish"
}
}},
{ "en": {
"match": "*",
"match_mapping_type": "string",
"mapping": {
"type": "string",
"analyzer": "english"
}
}}
]
}}}
in their docs it says " Match string fields whose name ends in _es".
"Match all other string fields": https://www.elastic.co/guide/en/elasticsearch/guide/current/custom-dynamic-mapping.html
This is what I've tried, but doesn't convert all to string (also tried without quotes around wildcard):
event.mappings = {
"mytype": {
"match": "*",
"mapping": {
"type": "string"
}
}
}
I've also tried "match_mapping_type" : "*".
I've tried: esClient.indices.putMapping({index:"myindex", type:"mytype", body:mybody})
in the response and outside of the .create function.
Any tips?
Your mapping should look like this
PUT /test
{
"mappings": {
"test": {
"dynamic_templates": [
{
"en": {
"match": "*",
"mapping": {
"type": "string"
}
}
}
]
}
}
}
Test data:
POST /test/test/1
{
"nr": 1,
"jsonDate":"2015-06-08T03:41:12-05:00",
"bool": true
}
The resulting mapping, as seen by ES:
{
"test": {
"mappings": {
"test": {
"dynamic_templates": [
{
"en": {
"mapping": {
"type": "string"
},
"match": "*"
}
}
],
"properties": {
"bool": {
"type": "string"
},
"jsonDate": {
"type": "string"
},
"nr": {
"type": "string"
}
}
}
}
}
}