I am evaluating Circe and couldn't find out how to use filter for arrays to transform a JSON. I read the guide on its website and API doc, still no clue. Help much appreciated.
Sample data:
{
"Department" : "HR",
"Employees" :[{ "name"： "abc", "age": 25 }, {"name":"def", "age" : 30 }]
}
Task:
How to use a filter for Employees to transform the JSON to another JSON, for example, all employees with age older than 50?
For some reason I can't filter from data source before JSON is generated, in case you ask.
Thanks
One possible way of doing this is by
val data = """{"Department" : "HR","Employees" :[{ "name": "abc", "age": 25 }, {"name":"def", "age":30}]}"""
def ageFilter(j:Json): Json = j.withArray { x =>
Json.fromValues(x.filter(_.hcursor.downField("age").as[Int].map(_ > 26).getOrElse(false)))
}
val y: Either[ParsingFailure, Json] = parse(data).map( _.hcursor.downField("Employees").withFocus(ageFilter).top.get)
println(s"$y")
Related
I have a table on AWS RDS PostgreSQL that stores JSON objects. For instance I have this registry:
{
"id": "87b05c62-4153-4341-9b58-e86bade25ffd",
"title": "Just Ok",
"rating": 2,
"gallery": [
{
"id": "1cb158af-0983-4bac-9e4f-0274b3836cdd",
"typeCode": "PHOTO"
},
{
"id": "aae64f19-22a8-4da7-b40a-fbbd8b2ef30b",
"typeCode": "PHOTO"
}
],
"reviewer": {
"memberId": "2acf2ea7-7a37-42d8-a019-3d9467cbdcd1",
},
"timestamp": {
"createdAt": "2011-03-30T09:52:36.000Z",
"updatedAt": "2011-03-30T09:52:36.000Z"
},
"isUserVerified": true,
}
And I would like to create a query for obtaining one of the gallery objects.
I have tried this but get both objects in the array:
SELECT jsonb_path_query(data->'gallery', '$[*]') AS content
FROM public.reviews
WHERE jsonb_path_query_first(data->'gallery', '$.id') ? '1cb158af-0983-4bac-9e4f-0274b3836cdd'
With this other query I get the first object:
SELECT jsonb_path_query_first(data->'gallery', '$[*]') AS content
FROM public.reviews
WHERE jsonb_path_query_first(data->'gallery', '$.id') ? '1cb158af-0983-4bac-9e4f-0274b3836cdd'
But filtering by the second array object id, I get no result:
SELECT jsonb_path_query_first(data->'gallery', '$[*]') AS content
FROM public.reviews
WHERE jsonb_path_query_first(data->'gallery', '$.id') ? 'aae64f19-22a8-4da7-b40a-fbbd8b2ef30b'
I have read the official documentation and tried other functions like jsonb_path_exists or jsonb_path_match on the where condition but was not able to make the query work.
Any help would be greatly appreciated. Thanks in advance.
I managed to get the query working as needed. Here is my proposal:
SELECT gallery
FROM public.reviews, jsonb_path_query(data->'gallery', '$[*]') as gallery
WHERE data->>'id' = '87b05c62-4153-4341-9b58-e86bade25ffd' and gallery->>'id' = 'aae64f19-22a8-4da7-b40a-fbbd8b2ef30b'
Hope it helps others.
I am using Elsaticsearch Spring Data. I have a custom repository that uses ElasticsearchOperations based on examples on docs. I need some aggregation query results and I successfully get the intended results. but I need to map those results to a model. But currently I'm unable to access contents of AggregationsContainer.
override fun getStats(startTime: Long, endTime: Long, pageable: Pageable): AggregationsContainer<*>?
{
val query: Query = NativeSearchQueryBuilder()
.withQuery(QueryBuilders.rangeQuery("time").from(startTime).to(endTime))
.withAggregations(AggregationBuilders.sum("discount").field("discount"))
.withAggregations(AggregationBuilders.sum("price").field("price"))
.withPageable(pageable)
.build()
val searchHits: SearchHits<Product> = operations.search(query, Product::class.java)
return searchHits.aggregations
}
I return the result of the following code:
val stats = repository.getTotalStats(before, currentTime, pageable)?.aggregations()
the result is :
{
"asMap": {
"discount": {
"name": "discount",
"metadata": null,
"value": 8000.0,
"valueAsString": "8000.0",
"type": "sum",
"fragment": true
},
"price": {
"name": "price",
"metadata": null,
"value": 9000.0,
"valueAsString": "9000.0",
"type": "sum",
"fragment": true
}
},
"fragment": true
}
How can I convert above output to an intended output model like following? as I tested contents of aggregations() are inaccessible and the type is Any :
{
"priceSum":9000.0,
"discountSum":8000
}
There is no data model in the Elasticsearch RestHighLevelClient classes for aggregations, and there is no on in Spring Data Elasticsearch. Therefore the original Aggregations object is returned to the caller (contained in that AggregationContainer, because that will change with new new client implementation, and then the container will hold a different object).
You have to parse this by yourself, I had something in the answer of another question (https://stackoverflow.com/a/63105356/4393565). The interesting thing for you is the last codeblock where the aggregations are passed. You basically have to iterate over the elements, cast them to the appropriate type and evaluate them.
Background:
I wish to locate the entire JSON document that has a condition where "state" = "new" and where length(Features.id) > 4
{
"id": "123"
"feedback": {
"Features": [
{
"state": "new"
"id": "12345"
}
]
}
}
This is what I have tried to do:
Since this is a nested document. My query looks like this:
A stackoverflow member has helped me to access the nested contents within the query, but is there a way to obtain the full document
I have used:
SELECT VALUE t.id FROM t IN f.feedback.Features where t.state = 'new' and length(t.id)>4
This will give me the ids.
My desire is to have access to the full document with this condition?
{
"id": "123"
"feedback": {
"Features": [
{
"state": "new"
"id": "12345"
}
]
}
}
Any help is appreciated
Try this
SELECT *
FROM f
WHERE
f.feedback.Features[0].state = 'new'
AND length(f.feedback.Features[0].id)>4
Here is the SELECT spec for CosmosDB for more details
https://learn.microsoft.com/en-us/azure/cosmos-db/sql-query-select
Also, check out "working with JSON" in CosmosDB notes
https://learn.microsoft.com/en-us/azure/cosmos-db/sql-query-working-with-json
If the Features array has more than 1 value, you can use EXISTS clause to search within them. See specs of EXISTS here with examples:
https://learn.microsoft.com/en-us/azure/cosmos-db/sql-query-subquery#exists-expression
I am trying to write a query to return all trains that have more than one etapesSupervision.
My table has a column called DETAIL, in this column I can find the JSON of my train.
"nomTrain": "EVOL99",
"compositionCourtLong": "LONG",
"sillons": [{
"numeroTrain": "EVOL99"
}],
"sillonsV4": [{
"refSillon": "sillons/4289505/2"
}],
"branchesStif": [{
"data": "49",
"data": "BP",
"data": "ORIGINE"
} ],
"etapesSupervision": [{
"data": "PR/0087-758896-00",
"data": "PR/0087-758607-BV",
"superviseur": "1287",
"uoSuperviseur": "B"
},
{
"data": "PR/0087-758607-BV",
"data": "PR/0087-001479-BV",
"superviseur": "1287",
"uoSuperviseur": "B"
}],
This is the query I wrote :
select * from course where CODE_LIGNE_COMMERCIALE='B'
--and ref = 'train/2018-11-12'
and instr(count(train.detail,'"etapesSupervision":'))> 1 ;
Using this, I return trains with only one etapesSupervision.
The thing is the column DETAIL is JSON, so I feel like I can't do a lot with it.
I tried also with like, but it doesn't work either.
Thank you for your comments.
This is the query that worked:
select data,data,data
from train
where
length(DETAIL) - length(replace(DETAIL,'uoSuperviseur',null)) > 20 ;
And this way I have only trains that have more than one supervisor.
Thanks again
I am new to PIG scripting and working with JSONs. I am in the need of parsing multi-level json files in PIG. Say,
{
"firstName": "John",
"lastName" : "Smith",
"age" : 25,
"address" :
{
"streetAddress": "21 2nd Street",
"city" : "New York",
"state" : "NY",
"postalCode" : "10021"
},
"phoneNumber":
[
{
"type" : "home",
"number": "212 555-1234"
},
{
"type" : "fax",
"number": "646 555-4567"
}
]
}
I am able to parse a single level json through JsonLoader() and do join and other operations and get the desired results as JsonLoader('name:chararray,field1:int .....');
Is it possible to parse the above mentioned JSON file using the built-in JsonLoader() function of PIG 0.10.0. If it is. Please explain me how it is done and accessing fields of the particular JSON?
You can handle nested json loading with Twitter's Elephant Bird: https://github.com/kevinweil/elephant-bird
a = LOAD 'file3.json' USING com.twitter.elephantbird.pig.load.JsonLoader('-nestedLoad')
This will parse the JSON into a map http://pig.apache.org/docs/r0.11.1/basic.html#map-schema the JSONArray gets parsed into a DataBag of maps.
It is possible by creating your own UDF. A simple UDF example is shown in below link
http://pig.apache.org/docs/r0.9.1/udf.html#udf-java
C = load 'path' using JsonLoader('firstName:chararray,lastName:chararray,age:int,address:(streetAddress:chararray,city:chararray,state:chararray,postalCode:chararray),
phoneNumber:{(type:chararray,number:chararray)}')