How to search array of objects RedisJson using RedisSearch - redis

i have added a json document into my redis db using JSON.set Below given is my Json SET command
{
"books": [
{
"title": "Peter Pan",
"price": 8.95
},
{
"title": "Moby Dick",
"price": 12.99
}
]
}
JSON.SET myDoc $ '{"books": [{"title": "Peter Pan", "price": 8.95},{"title": "A Thousand Suns", "price": 8.15}, {"title": "Moby Dick", "price": 12.99}]}'
Now i want to search this specific array of objects to fetch me the records which have price greater and equal to 8
i have tried creating an index but it always return me 0 records Below given is my index
FT.CREATE docIdx ON JSON SCHEMA $.myDoc.books.price AS price Numeric

in the way you propose the index, you're ignoring the array of elements that belong to the object book. One way could be:
FT.CREATE docIdx ON JSON SCHEMA $.myDoc.books[*].price AS price Numeric
or
FT.CREATE docIdx ON JSON SCHEMA $.myDoc.books[0:].price AS price Numeric
another recommendation it's to use the prefix since all documents with the same prefix will be indexed by Redisearch.
Something like:
FT.CREATE docIdx PREFIX 1 myDoc ON JSON SCHEMA $.books[*].price AS price Numeric
Notice, since you are trying to match the condition within the same document when you search for your price condition (between 8 and 9), the result will be the entire document "myDoc" with all the books, since the "Peter Pan" and "A Thousand Suns" matches the query condition. Such as:
> FT.SEARCH docIdx "#price:[8 12]"
1) "1"
2) "myDoc"
3) 1) "$"
2) "{\"books\":[{\"title\":\"Peter Pan\",\"price\":8.95},{\"title\":\"A Thousand Suns\",\"price\":8.15},{\"title\":\"Moby Dick\",\"price\":12.99}]}"

Related

Postgres jsonb_array_elements() returns "cannot extract elements from a scalar", even when given a valid JSON array?

I have a table in postgres called day, which contains a jsonb column called plan_activities.
I have a day record with day.id = 18 and plan_activities contains the following JSON:
[
{
"activity": "Gym",
"cardio": false,
"strength": true,
"quantity": 20,
"units": "mins",
"timeOfDay": "Evening",
"summary": "Gym - 20 mins - Evening",
"timeOfDayOrder": 4
},
{
"activity": "Walk",
"cardio": true,
"strength": false,
"quantity": 15,
"units": "minutes",
"timeOfDay": "morning",
"summary": "Walk - 15 minutes - Lunchtime",
"timeOfDayOrder": 1
}
]
When I execute the following query:
select jsonb_array_elements(day.plan_activities) as activities
from day
where day.id = 18;
I get the following error:
Failed to run sql query: cannot extract elements from a scalar
The JSON contains a valid JSON array as far as I can tell. What am I doing wrong?
My eventual goal if I can extract this list is to create separate records elsewhere, each of which contains all the fields plus a reference back to the day record.
This error happens when you try to treat a JSON scalar, like a single string or number, as an array.
-- ERROR: cannot extract elements from a scalar
select jsonb_array_elements('23'::jsonb);
One of the rows of your query does not contain a JSON array.
Check with select plan_activities from day where id = 18. Although id is normally a unique primary key and it should be impossible to have more than one row returned.
Another way this could happen is if the JSON structure was accidentally added as a single JSON string.
-- 1, 2, 3
select jsonb_array_elements('[1, 2, 3]'::jsonb);
-- Note the extra quotes.
-- ERROR: cannot extract elements from a scalar
select jsonb_array_elements('"[1, 2, 3]"'::jsonb);

Postgres jsonb field to array

I was going through the Postgres Jsonb documentation but was unable to find a solution for a small issue I'm having.
I've got a table : MY_TABLE
that has the following columns:
User, Name, Data and Purchased
One thing to note is that "Data" is a jsonb and has multiple fields. One of the fields inside of "Data" is "Attribute" but it is currently a string. How can I go about changing this to a list of strings?
I have tried using json_build_array but have not had any luck
So for example, I'd want my jsonb to look like :
{
"Id": 1,
"Attributes": ["Test"]
}
instead of
{
"Id": 1,
"Attributes": "Test"
}
I only care about the "Attributes" field inside of the Json, not any other fields.
I also want to ensure for some Attributes that have an empty string "Attributes": "", they get mapped to an empty list and not a list with an empty string ([] not [""])
You can use jsonb_set(), and some conditional logic for the empty string:
jsonb_set(
mycol,
'{Attributes}',
case when js ->> 'Attributes' <> ''
then jsonb_build_array(js ->> 'Attributes')
else '[]'::jsonb
end
)

Accessing values in JSON array

I am following the instruction in the documentation for how to access JSON values in CloudWatch Insights where the recomendation is as follows
JSON arrays are flattened into a list of field names and values. For example, to specify the value of instanceId for the first item in requestParameters.instancesSet, use requestParameters.instancesSet.items.0.instanceId.
ref
https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CWL_AnalyzeLogData-discoverable-fields.html
I am trying the following and getting nothing in return. The intellisense autofills up to processList.0 but no further
fields processList.0.vss
| sort #timestamp desc
| limit 1
The JSON I am woking with is
"processList": [
{
"vss": xxxxx,
"name": "aurora",
"tgid": xxxx,
"vmlimit": "unlimited",
"parentID": 1,
"memoryUsedPc": 16.01,
"cpuUsedPc": 0.01,
"id": xxxxx,
"rss": xxxxx
},
{
"vss": xxxx,
"name": "aurora",
"tgid": xxxxxx,
"vmlimit": "unlimited",
"parentID": 1,
"memoryUsedPc": 16.01,
"cpuUsedPc": 0.06,
"id": xxxxx,
"rss": xxxxx
}]
Have you tried the following?
fields ##timestamp, #processList.0.vss
| sort ##timestamp desc
| limit 5
It may be a syntax error. If not, please post a couple of records worth of the overall structure, with #timestamp included.
The reference link that you have posted also states the following.
CloudWatch Logs Insights can extract a maximum of 100 log event fields
from a JSON log. For extra fields that are not extracted, you can use
the parse command to parse these fields from the raw unparsed log
event in the message field.
https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CWL_AnalyzeLogData-discoverable-fields.html
For very large JSON messages, Insights intellisense may not be parsing all the fields into named fields. So, the solution is to use parse on the complete JSON string in the field where you expect your data field to be present. In your example and mine it is processList.
I was able to extract the value of specific cpuUsedPc under processList by using a query like the following.
fields #timestamp, cpuUtilization.total, processList
| parse processList /"name":"RDS processes","tgid":.*?,"parentID":.*?,"memoryUsedPc":.*?,"cpuUsedPc":(?<RDSProcessesCPUUsedPc>.*?),/
| sort #timestamp asc
| display #timestamp, cpuUtilization.total, RDSProcessesCPUUsedPc

Query data inside an attribute array in a json column in Postgres 9.6

I have a table say types, which had a JSON column, say location that looks like this:
{ "attribute":[
{
"type": "state",
"value": "CA"
},
{
"type": "distance",
"value": "200.00"
} ...
]
}
Each row in the table has the data, and all have the "type": "state" in it. I want to just extract the value of "type": "state" from every row in the table, and put it in a new column. I checked out several questions on SO, like:
Query for element of array in JSON column
Index for finding an element in a JSON array
Query for array elements inside JSON type
but could not get it working. I do not need to query on this. I need the value of this column. I apologize in advance if I missed something.
create table t(data json);
insert into t values('{"attribute":[{"type": "state","value": "CA"},{"type": "distance","value": "200.00"}]}'::json);
select elem->>'value' as state
from t, json_array_elements(t.data->'attribute') elem
where elem->>'type' = 'state';
| state |
| :---- |
| CA |
dbfiddle here
I mainly use Redshift where there is a built-in function to do this. So on the off-chance you're there, check it out.
redshift docs
It looks like Postgres has a similar function set:
https://www.postgresql.org/docs/current/static/functions-json.html
I think you'll need to chain three functions together to make this work.
SELECT
your_field::json->'attribute'->0->'value'
FROM
your_table
What I'm trying is a json extract by key name, followed by a json array extract by index (always the 1st, if your example is consistent with the full data), followed finally by another extract by key name.
Edit: got it working for your example
SELECT
'{ "attribute":[
{
"type": "state",
"value": "CA"
},
{
"type": "distance",
"value": "200.00"
}
]
}'::json->'attribute'->0->'value'
Returns "CA"
2nd edit: nested querying
#McNets is the right, better answer. But in this dive, I discovered you can nest queries in Postgres! How frickin' cool!
I stored the json as a text field in a dummy table and successfully ran this:
SELECT
(SELECT value FROM json_to_recordset(
my_column::json->'attribute') as x(type text, value text)
WHERE
type = 'state'
)
FROM dummy_table

Query and count on jsonb column

I'm new to the postgreSQL(9.5) Json world. Looking for help writing this query. Take this simplified table as an example.
CREATE TABLE activity_log (uri varchar,
data jsonb );
Example of data inside of 'data' column.
"{"ListingInputFilterBean":{"searchItems": [], "listingStatus": "ACTIVE"}"
"{"ListingInputFilterBean":{"searchItems": [{"name": "Dachshund", "type": "BREED"}], "listingStatus": "ACTIVE"}}"
"{"ListingInputFilterBean":{"searchItems": [{"name": "Lab", "type": "BREED"}, {"name": "Black Lab", "type": "CST"}], "listingStatus": "ACTIVE"}}"
The 'data' column is used to log specific sets of data for each URI call. In this case the searchItems array contain the items used in the search. I'm looking to write a query that finds the most searched for 'breed'. I'd like to count the number of times each 'name' is used when type is 'BREED'.
My initial approach was to pull back each of the 'searchItems'. Turn those into a row set using jsonb_to_recordset, but I quickly got in over my head when reading the documentation (sorry, I'm a noob).
Any suggestions on how to write that SQL?
WITH log_activity(data) AS ( VALUES
('{"ListingInputFilterBean":{"searchItems": [], "listingStatus": "ACTIVE"}}'::JSONB),
('{"ListingInputFilterBean":{"searchItems": [{"name": "Dachshund", "type": "BREED"}], "listingStatus": "ACTIVE"}}'::JSONB),
('{"ListingInputFilterBean":{"searchItems": [{"name": "Lab", "type": "BREED"}, {"name": "Black Lab", "type": "CST"}], "listingStatus": "ACTIVE"}}'::JSONB)
)
SELECT search_item->>'name',count(search_item->>'name')
FROM
log_activity la,
jsonb_array_elements(la.data#>'{ListingInputFilterBean,searchItems}') as search_item
WHERE search_item->>'type' = 'BREED'
GROUP BY search_item;
Result:
name | count
-----------+-------
Lab | 1
Dachshund | 1
(2 rows)
Here you just need to iterate over the list of searchItems and group only those entries, which do match your criteria. Steps are the following:
Get jsonb array of searchItems with #> operator, it will get JSON object at specified path;
Iterate over the list of elements retrieved from step 1 with jsonb_array_elements(), function which expands a JSON array to a set of JSON values;
count() names where searchItems' type = BREED, you can get actual text value with ->> operator;
UPDATE
With jsonb_to_recordset() it looks shorter, but you need explicitly define search_item columns' types:
SELECT search_item.name ,count(search_item.name)
FROM
log_activity la,
jsonb_to_recordset(la.data#>'{ListingInputFilterBean,searchItems}') as search_item(name text,type text)
WHERE search_item.type = 'BREED'
GROUP BY search_item.name;