I am trying perform match on tis json array. Scenario goes like this
Scenario : match lob
* def op =
"""
[
{
"_id": "1",
"_class":
"com.xxx.versionone.tir.enterprise.persistence.model.xxx",
"lobName": "abc",
"changeDate": "2016-11-04T11:41:40",
"changedBy": "abc",
"createdDate": "2014-07-01T11:47:23",
"lastSdpPublishDate": "2018-10-31T00:00:00"
},
{
"_id": {
"$oid": "57883a41e4b076d23a82e9e7"
},
"_class":
"com.xxx.versionone.tir.enterprise.persistence.model.xxx",
"lobName": "asda",
"changeDate": "2016-07-14T21:20:54",
"changedBy": "TXA858",
"createdDate": "2016-07-14T21:20:01",
"createdBy": "TXA858",
"lastSdpPublishDate": "2018-10-31T00:00:00"
}
]
"""
* match $op...lobName contains ["abc"]
I get this error:
com.intuit.karate.exception.KarateException: Character '.' on position 3 is not valid.
at com.intuit.karate.StepDefs.matchNamed(StepDefs.java:540)
at com.intuit.karate.StepDefs.matchContains(StepDefs.java:532)
JSON path deep scan is .. not ...
This should work,
* match $op..lobName contains ["abc"]
Json path operators
Related
For example, look at the schema:
"type": "array",
"items": {
"oneOf":[
{"type": "number"},
{"type": "string"}
]
}
}
The schema would then allow a list of numbers and the last element would be a string. and the string can be optional.
For example:
{
"list_of_items" : [ 3, 4, 5, "hello"]
}
is allowed.
{
"list_of_items" : [ 3, "hello" , 4, 5 ]
}
Is not allowed because text is only allowed on the last element.
From reading the documentation. I see how to do a constant amount of numbers and a text element.
Or all numbers,
But I don't see how to create n-1 numbers and text for the nth.
It could be that is is not something json-schema can do.
Thanks.
During my work flow, after extracting the data from API, the JSON has the following structure:
[
{
"fields":
[
{
"meta": {
"app_type": "ios"
},
"name": "app_id",
"value": 100
},
{
"meta": {},
"name": "country",
"value": "AE"
},
{
"meta": {
"name": "Top"
},
"name": "position",
"value": 1
}
],
"metrics": {
"click": 1,
"price": 1,
"count": 1
}
}
]
Then it is store as .jsonl and put on GCS. However, when I load it onto BigQuery for further extraction, the automatic schema inference return the following error:
Error while reading data, error message: JSON parsing error in row starting at position 0: Could not convert value to string. Field: value; Value: 100
I want to convert it in to the following structure:
app_type
app_id
country
position
click
price
count
ios
100
AE
Top
1
1
1
Is there a way to define manual schema on BigQuery to achieve this result? Or do I have to preprocess the jsonl file before put it to BigQuery?
One of the limitations in loading JSON data from GCS to BigQuery is that it does not support maps or dictionaries in JSON.
A invalid example would be:
"metrics": {
"click": 1,
"price": 1,
"count": 1
}
Your jsonl file should be something like this:
{"app_type":"ios","app_id":"100","country":"AE","position":"Top","click":"1","price":"1","count":"1"}
I already tested it and it works fine.
So wherever you process the conversion of the json files to jsonl files and storage to GCS, you will have to do some preprocessing.
Probably you have to options:
precreate target table with an app_id field as an INTEGER
preprocess jsonfile and enclose 100 into quotes like "100"
I have been struggling with this problem for a long time. I need to create a new JSON flowfile using QueryRecord by taking an array (field ref) from input JSON field refs and skip the object field as shown in example below:
Input JSON flowfile
{
"name": "name1",
"desc": "full1",
"refs": {
"ref": [
{
"source": "source1",
"url": "url1"
},
{
"source": "source2",
"url": "url2"
}
]
}
}
QueryRecord configuration
JSONTreeReader setup as Infer Schema and JSONRecordSetWriter
select name, description, (array[rpath(refs, '//ref[*]')]) as sources from flowfile
Output JSON (need)
{
"name": "name1",
"desc": "full1",
"references": [
{
"source": "source1",
"url": "url1"
},
{
"source": "source2",
"url": "url2"
}
]
}
But got error:
QueryRecord Failed to write MapRecord[{references=[Ljava.lang.Object;#27fd935f, description=full1, name=name1}] with schema ["name" : "STRING", "description" : "STRING", "references" : "ARRAY[STRING]"] as a JSON Object due to java.lang.ClassCastException: null
Try the following approach, in your case it shoud work:
1) Read your JSON field fully (I imitated it with GenerateFlowFile processor with your example)
2) Add EvaluateJsonPath processor which will put 2 header fileds (name, desc) into the attributes:
3) Add SplitJson processor which will split your JSON byt refs/ref/ groups (split by "$.refs.ref"):
4) Add ReplaceText processor which will add you header fields (name, desc) to the split lines (replace "[{]" value with "{"name":"${json.name}","desc":"${json.desc}","):
5) It`s done:
Full process in my demo case:
Hope this helps.
Solution!: use JoltTransformJSON to transform JSON by Jolt specification. About this specification.
I want to get a specific string from a column. How can I get that.
Here is the json with column name json with table name my_table
I want to fetch "extensionAttribute.simSerial": "310240000029929".
{
"name": "urn:imei:930000001801583",
"type": "DEVICE",
"sourceId": "P-n1000USCqT4",
"consumers": "CDM",
"crudStatus": {
"status": "SUCCESS",
"operation": "UPDATE"
},
"targetName": "urn:imei:930000001801583",
"deviceTypeId": "dgs11b74714f7020ctoogu3zfugc6",
"createdUserId": "a8aacc5d978d494eb54ae4243e714646",
"onboardStatus": "DONE",
"consrStatus": "",
"deviceTypeName": "performance_device_type_2",
"lwm2mPskSecret": "49443",
"createdUserName": "manager",
"bootstrapRequest": true,
"lwm2mPskIdentity": "urn:imei:930000001801583",
"boostrapPskSecret": "49443",
"deviceTypeVersion": 1,
"lastUpdatedUserId": "",
"lwm2mSecurityMode": "psk",
"consumersForUpdate": "",
"bootstrapPskIdentity": "urn:imei:930000001801583",
"pureCoapSecurityMode": "NONE",
"bootstrapSecurityMode": "psk",
"extensionAttribute.imsi": "310240000029929",
"extensionAttribute.msisdn": "310240000029929",
"extensionAttribute.simSerial": "310240000029929",
"extensionAttribute.msisdnStatus": "active"
}
Solution
You can do it as follows:
select column_name->>'extensionAttribute.simSerial' as simSerial from my_table;
where column_name is your column name in a table.
If you will have JSON with higher depth you can do something like:
column_name->'key_in_json'->'another_key_in_json'->>'last_key_in_json'
Manual
JSON Functions and Operators
PostgreSQL JSON Tutorial
I am facing issue while validate Nested JSON response in API Testing using Karate Framework.
JSON Response:
Feed[
{ "item_type": "Cake" ,
"title": "Birthday Cake",
"Services":
[
{
"id": "1",
"name": {
"first_name": "Rahul",
"last_name": "Goyal"
}
},
{
"id": "2",
"name":{
"first_name": "Hitendra",
"last_name": "garg"
}
}
]
},
{
"item_type":"Cycle",
"title": "used by"
},
{
"item_type": "College"
"dept":
[
{"branch": "EC"},
{"branch": "CSE"},
{"branch": "CIVIL"}
]
},
]
}
Now i need to validate response based on Item type. as we can see nested JSON is different for different item_type.
I have tried with below solution
Schema Design for Item_type value cake
def Feed_Cake_Service_name={first_name: '#string',last_name: '#string'}
def Feed_Cake_Services= {id: '#string',name:#(Feed_Cake_Service_name)}
def Feed_Cake={item_type:'#string',title: '#string',Services: '#[] Feed_Cake_Services'}
def Feed_Cake_Response= {Feed: '#[] Feed_Cake'}
Schema Design for item_type Cycle
def Feed_Cycle={item_type:'#string',title:'#string'}
Schema Design for item type College
def Feed_College_Dept_Branch={branch:'#string'}
def Feed_College={item_type:'#string',dept: '[] Feed_College_Dept_Branch'}
now if i want to verify only item type Cake then i have written match like below
match response contains Feed_Cake_Response
but here my test case is getting failed. because it is comparing for all item type.
so here i have two question
1.) How we can compare particular item type schema
2.) How we can include all item type in one match equation since any item type can come in JSON response , and i want to validate all
Thanks
I'll just give you one hint. For the rest, read the documentation please:
* def item = { item_type: '#string', title: '##string', dept: '##[]', Services: '##[]' }
* match each response == item