I have been struggling with this problem for a long time. I need to create a new JSON flowfile using QueryRecord by taking an array (field ref) from input JSON field refs and skip the object field as shown in example below:
Input JSON flowfile
{
"name": "name1",
"desc": "full1",
"refs": {
"ref": [
{
"source": "source1",
"url": "url1"
},
{
"source": "source2",
"url": "url2"
}
]
}
}
QueryRecord configuration
JSONTreeReader setup as Infer Schema and JSONRecordSetWriter
select name, description, (array[rpath(refs, '//ref[*]')]) as sources from flowfile
Output JSON (need)
{
"name": "name1",
"desc": "full1",
"references": [
{
"source": "source1",
"url": "url1"
},
{
"source": "source2",
"url": "url2"
}
]
}
But got error:
QueryRecord Failed to write MapRecord[{references=[Ljava.lang.Object;#27fd935f, description=full1, name=name1}] with schema ["name" : "STRING", "description" : "STRING", "references" : "ARRAY[STRING]"] as a JSON Object due to java.lang.ClassCastException: null
Try the following approach, in your case it shoud work:
1) Read your JSON field fully (I imitated it with GenerateFlowFile processor with your example)
2) Add EvaluateJsonPath processor which will put 2 header fileds (name, desc) into the attributes:
3) Add SplitJson processor which will split your JSON byt refs/ref/ groups (split by "$.refs.ref"):
4) Add ReplaceText processor which will add you header fields (name, desc) to the split lines (replace "[{]" value with "{"name":"${json.name}","desc":"${json.desc}","):
5) It`s done:
Full process in my demo case:
Hope this helps.
Solution!: use JoltTransformJSON to transform JSON by Jolt specification. About this specification.
Related
During my work flow, after extracting the data from API, the JSON has the following structure:
[
{
"fields":
[
{
"meta": {
"app_type": "ios"
},
"name": "app_id",
"value": 100
},
{
"meta": {},
"name": "country",
"value": "AE"
},
{
"meta": {
"name": "Top"
},
"name": "position",
"value": 1
}
],
"metrics": {
"click": 1,
"price": 1,
"count": 1
}
}
]
Then it is store as .jsonl and put on GCS. However, when I load it onto BigQuery for further extraction, the automatic schema inference return the following error:
Error while reading data, error message: JSON parsing error in row starting at position 0: Could not convert value to string. Field: value; Value: 100
I want to convert it in to the following structure:
app_type
app_id
country
position
click
price
count
ios
100
AE
Top
1
1
1
Is there a way to define manual schema on BigQuery to achieve this result? Or do I have to preprocess the jsonl file before put it to BigQuery?
One of the limitations in loading JSON data from GCS to BigQuery is that it does not support maps or dictionaries in JSON.
A invalid example would be:
"metrics": {
"click": 1,
"price": 1,
"count": 1
}
Your jsonl file should be something like this:
{"app_type":"ios","app_id":"100","country":"AE","position":"Top","click":"1","price":"1","count":"1"}
I already tested it and it works fine.
So wherever you process the conversion of the json files to jsonl files and storage to GCS, you will have to do some preprocessing.
Probably you have to options:
precreate target table with an app_id field as an INTEGER
preprocess jsonfile and enclose 100 into quotes like "100"
I want to get a specific string from a column. How can I get that.
Here is the json with column name json with table name my_table
I want to fetch "extensionAttribute.simSerial": "310240000029929".
{
"name": "urn:imei:930000001801583",
"type": "DEVICE",
"sourceId": "P-n1000USCqT4",
"consumers": "CDM",
"crudStatus": {
"status": "SUCCESS",
"operation": "UPDATE"
},
"targetName": "urn:imei:930000001801583",
"deviceTypeId": "dgs11b74714f7020ctoogu3zfugc6",
"createdUserId": "a8aacc5d978d494eb54ae4243e714646",
"onboardStatus": "DONE",
"consrStatus": "",
"deviceTypeName": "performance_device_type_2",
"lwm2mPskSecret": "49443",
"createdUserName": "manager",
"bootstrapRequest": true,
"lwm2mPskIdentity": "urn:imei:930000001801583",
"boostrapPskSecret": "49443",
"deviceTypeVersion": 1,
"lastUpdatedUserId": "",
"lwm2mSecurityMode": "psk",
"consumersForUpdate": "",
"bootstrapPskIdentity": "urn:imei:930000001801583",
"pureCoapSecurityMode": "NONE",
"bootstrapSecurityMode": "psk",
"extensionAttribute.imsi": "310240000029929",
"extensionAttribute.msisdn": "310240000029929",
"extensionAttribute.simSerial": "310240000029929",
"extensionAttribute.msisdnStatus": "active"
}
Solution
You can do it as follows:
select column_name->>'extensionAttribute.simSerial' as simSerial from my_table;
where column_name is your column name in a table.
If you will have JSON with higher depth you can do something like:
column_name->'key_in_json'->'another_key_in_json'->>'last_key_in_json'
Manual
JSON Functions and Operators
PostgreSQL JSON Tutorial
In jmeter, I want to pass dynamic parameters. For simple json its easy to put ${value1} but if json structure is complex like array or with multiple values then what is the proper method to pass parameter dynamically. Please refer below json.
Below is json with parameter :
{
"squadName": "Super hero squad",
"homeTown": "Metro City",
"formed": 2016,
"secretBase": "Super tower",
"active": true,
"members": [
{
"name": "Molecule Man",
"age": 29,
"secretIdentity": "Dan Jukes",
"powers": [
"Radiation resistance",
"Turning tiny",
"Radiation blast"
]
},
{
"name": "Madame Uppercut",
"age": 39,
"secretIdentity": "Jane Wilson",
"powers": [
"Million tonne punch",
"Damage resistance",
"Superhuman reflexes"
]
},
{
"name": "Eternal Flame",
"age": 1000000,
"secretIdentity": "Unknown",
"powers": [
"Immortality",
"Heat Immunity",
"Inferno",
"Teleportation",
"Interdimensional travel"
]
}
]
}
=======
Now I have used below method to send parameter through csv config file.
Is there any other simple method to pass parameter through variables in Jmeter for complex json (5-6 level with array data) ?
CSV DATA config is the best to parameterize your test data.
If you want to customize the way you want to pick values from CSV you can use BeanShell /JSR223 sampler
here is one article that shows how to pick random values from CSV data config.
From a form submission I receive two objects: the original values and the dirty values. I like to figure out how to create a diff to send to the server using the following rules:
id field of the root object should always be included
all changed primitive values should be included
all nested changes should be included as well.
if a nested value other than id changed, it should include id as well.
Original values:
{
"id":10,
"name": "tkvw"
"locale": "nl",
"address":{
"id":2,
"street": "Somewhere",
"zipcode": "8965",
},
"subscriptions":[8,9,10],
"category":{
"id":6
},
}
Example expected diff objects:
1) User changes field name to "Foo"
{
"id":10,
"name":"foo"
}
2) User changes field street on address node and category
{
"id":10,
"address":{
"id": 2,
"street":"Changed"
},
"category":{
"id":5
}
}
I do understand the basics of functional programming, but I just need a hint in the right direction (some meta code maybe).
Take a look at JSON Patch (rfc6902), JSON Patch is a format for describing changes to a JSON document. For example:
[
{ "op": "replace", "path": "/baz", "value": "boo" },
{ "op": "add", "path": "/hello", "value": ["world"] },
{ "op": "remove", "path": "/foo"}
]
You generate a patch by comparing to JS objects/arrays, and then you can apply the patch to the original object (on the server side for example) to reflect changes.
You can create a patch using the fast-json-patch lib.
const obj1 = {"id":10,"name":"tkvw","locale":"nl","address":{"id":2,"street":"Somewhere","zipcode":"8965"},"subscriptions":[8,9,10],"category":{"id":6}};
const obj2 = {"id":10,"name":"cats","locale":"nl","address":{"id":2,"street":"Somewhere","zipcode":"8965"},"subscriptions":[8,9,10,11],"category":{"id":7}};
const delta = jsonpatch.compare(obj1, obj2);
console.log('delta:\n', delta);
const doc = jsonpatch.applyPatch(obj1, delta).newDocument;
console.log('patched obj1:\n', doc);
<script src="https://cdnjs.cloudflare.com/ajax/libs/fast-json-patch/2.0.6/fast-json-patch.min.js"></script>
I want to create a path which accepts any number of IDs separated with commas. It should accept all the following:
GET /pets/1,2 # returns info about pets with ids 1, 2
GET /pets/4,10,12,124,2 # same, for pets with ids 4, 10, 12, 124, 2
The book "building apis you won't hate" gave me this idea.
How do I do that in swagger?
Swagger 2.0 supports the collectionFormat parameter. From the documentation:
Determines the format of the array if type array is used. Possible values are:
csv: comma separated values: foo,bar
ssv: space separated values: foo bar
tsv: tab separated values: foo\tbar
pipes: pipe separated values: foo|bar
multi: corresponds to multiple parameter instances instead of multiple values for a single instance foo=bar&foo=baz. This is valid only for parameters in "query" or "formData".
Default value is csv.
An example of use can be seen on the documentation:
{
"get": {
"description": "Returns pets based on ID",
"summary": "Find pets by ID",
"operationId": "getPetsById",
"produces": [
"application/json",
"text/html"
],
"responses": { ... }
},
"parameters": [
{
"name": "id",
"in": "path",
"description": "ID of pet to use",
"required": true,
"type": "array",
"items": {
"type": "string"
},
"collectionFormat": "csv"
}
]
}