How do I query using fields inside the new PostgreSQL JSON datatype? - sql

I am looking for some docs and/or examples for the new JSON functions in PostgreSQL 9.2.
Specifically, given a series of JSON records:
[
{name: "Toby", occupation: "Software Engineer"},
{name: "Zaphod", occupation: "Galactic President"}
]
How would I write the SQL to find a record by name?
In vanilla SQL:
SELECT * from json_data WHERE "name" = "Toby"
The official dev manual is quite sparse:
http://www.postgresql.org/docs/devel/static/datatype-json.html
http://www.postgresql.org/docs/devel/static/functions-json.html
Update I
I've put together a gist detailing what is currently possible with PostgreSQL 9.2.
Using some custom functions, it is possible to do things like:
SELECT id, json_string(data,'name') FROM things
WHERE json_string(data,'name') LIKE 'G%';
Update II
I've now moved my JSON functions into their own project:
PostSQL - a set of functions for transforming PostgreSQL and PL/v8 into a totally awesome JSON document store

Postgres 9.2
I quote Andrew Dunstan on the pgsql-hackers list:
At some stage there will possibly be some json-processing (as opposed
to json-producing) functions, but not in 9.2.
Doesn't prevent him from providing an example implementation in PLV8 that should solve your problem. (Link is dead now, see modern PLV8 instead.)
Postgres 9.3
Offers an arsenal of new functions and operators to add "json-processing".
The manual on new JSON functionality.
The Postgres Wiki on new features in pg 9.3.
The answer to the original question in Postgres 9.3:
SELECT *
FROM json_array_elements(
'[{"name": "Toby", "occupation": "Software Engineer"},
{"name": "Zaphod", "occupation": "Galactic President"} ]'
) AS elem
WHERE elem->>'name' = 'Toby';
Advanced example:
Query combinations with nested array of records in JSON datatype
For bigger tables you may want to add an expression index to increase performance:
Index for finding an element in a JSON array
Postgres 9.4
Adds jsonb (b for "binary", values are stored as native Postgres types) and yet more functionality for both types. In addition to expression indexes mentioned above, jsonb also supports GIN, btree and hash indexes, GIN being the most potent of these.
The manual on json and jsonb data types and functions.
The Postgres Wiki on JSONB in pg 9.4
The manual goes as far as suggesting:
In general, most applications should prefer to store JSON data as
jsonb, unless there are quite specialized needs, such as legacy
assumptions about ordering of object keys.
Bold emphasis mine.
Performance benefits from general improvements to GIN indexes.
Postgres 9.5
Complete jsonb functions and operators. Add more functions to manipulate jsonb in place and for display.
Major good news in the release notes of Postgres 9.5.

With Postgres 9.3+, just use the -> operator. For example,
SELECT data->'images'->'thumbnail'->'url' AS thumb FROM instagram;
see http://clarkdave.net/2013/06/what-can-you-do-with-postgresql-and-json/ for some nice examples and a tutorial.

With postgres 9.3 use -> for object access. 4 example
seed.rb
se = SmartElement.new
se.data =
{
params:
[
{
type: 1,
code: 1,
value: 2012,
description: 'year of producction'
},
{
type: 1,
code: 2,
value: 30,
description: 'length'
}
]
}
se.save
rails c
SELECT data->'params'->0 as data FROM smart_elements;
returns
data
----------------------------------------------------------------------
{"type":1,"code":1,"value":2012,"description":"year of producction"}
(1 row)
You can continue nesting
SELECT data->'params'->0->'type' as data FROM smart_elements;
return
data
------
1
(1 row)

Related

Using Bookshelf to execute a query on Postgres JSONB array elements

I have a postgres table with jsonb array elements and I'm trying to do sql queries to extract the matching elements. I have the raw SQL query running from the postgres command line interface:
select * from movies where director #> any (array ['70', '45']::jsonb[])
This returns the results I'm looking for (all records from the movies table where the director jsonb elements contain any of the elements in the input element).
In the code, the value for ['70, '45'] would be a dynamic variable ie. fixArr and the length of the array is unknown.
I'm trying to build this into my Bookshelf code but haven't been able to find any examples that address the complexity of the use case. I've tried the following approaches but none of them work:
models.Movies.where('director', '#> any', '(array' + JSON.stringify(fixArr) + '::jsonb[])').fetchAll()
ERROR: The operator "#> any" is not permitted
db.knex.raw('select * from movies where director #> any(array'+[JSON.stringify(fixArr)]+'::jsonb[])')
ERROR: column "45" does not exist
models.Movies.query('where', 'director', '#>', 'any (array', JSON.stringify(fixArr) + '::jsonb[])').fetchAll()
ERROR: invalid input syntax for type json
Can anyone help with this?
As you have noticed, knex nor bookshelf doesn't bring any support for making jsonb queries easier. As far as I know the only knex based ORM that supports jsonb queries etc. nicely is Objection.js
In your case I suppose better operator to find if jsonb column contains any of the given values would be ?|, so query would be like:
const idsAsString = ids.map(val => `'${val}'`).join(',');
db.knex.raw(`select * from movies where director \\?| array[${idsAsString}]`);
More info how to deal with jsonb queries and indexing with knex can be found here https://www.vincit.fi/en/blog/objection-js-postgresql-power-json-queries/
No, you're just running into the limitations of that particular query builder and ORM.
The easiest way is using bookshelf.Model.query and knex.raw (whereRaw, etc.). Alias with AS and subclass your Bookshelf model to add these aliased attributes if you care about such things.
If you want things to look clean and abstracted through Bookshelf, you'll just need to denormalize the JSONB into flat tables. This might be the best approach if your JSONB is mostly flat and simple.
If you end up using lots of JSONB (it can be quite performant with appropriate indexes) then Bookshelf ORM is wasted effort. The knex query builder is only not a waste of time insofar as it handles escaping, quoting, etc.

Search through jsonb column in ActiveRecord

My User model have jsonb column which names is raw. It looks like this:
{"client"=>{"tokens"=>["asdasd"]}}
Now I want to find a user by a token which is in raw["client"]["tokens"]. How can I do that?
I usually first craft such queries in a SQL console and then convert it to ActiveRecord.
You can navigate hash keys in Postgres. The query
SELECT
raw #> '{client,tokens}'
FROM users
will return just the tokens array from that path. Now we will need to check if it contains the value we are looking for. The query
SELECT
raw #> '{client,tokens}' ? 'asdasd'
FROM users
will select t for those row that have a matching token. Now you can move this to the WHERE section:
SELECT
*
FROM users
WHERE raw #> '{client,tokens}' ? 'asdasd'
And if this selects what you expect, then you can convert it to AR:
User.where("config #> '{client, tokens}' ? :token", token: 'asdasd')
Note that I can not use ? for parameter substitution and use :token instead. Also note that this only works in JSONB (Postgres 9.4+ https://www.postgresql.org/docs/9.4/static/functions-json.html#FUNCTIONS-JSONB-OP-TABLE)
Update:
You should(tm) (I have not tested this) get along with:
CREATE INDEX index_tokens ON users USING GIN ((raw #> '{client, tokens}'));
See https://www.postgresql.org/docs/9.6/static/datatype-json.html#JSON-INDEXING for more details
Update2:
Another way to query would be:
raw #> '{"client": {"tokens": ["asdasd"]}}'
Which should be able to use a simple GIN index on the raw column (which uses more space than the expression index described above).
CREATE INDEX index_user_raw ON users USING GIN (raw)
Again: details see the JSON INDEXING link above. And use a Query Visualizer to see the differences. I like http://tatiyants.com/pev

How do you index an array inside a JSON with an Oracle 12c query?

I have a table "move" with one column "move_doc" which is a CLOB. The json stored inside has the structure:
{
moveid : "123",
movedate : "xyz",
submoves: [
{
submoveid: "1",
...
},
{
submoveid : "2",
...
}
]
}
I know I can run an Oracle 12c query to access the submoves list with:
select move.move_doc.submoves from move move
How do I access particular submoves of the array? And the attributes inside a particular submove?
You have to use Oracle functions json_query and/or json_value like this:
SELECT json_value(move_doc, '$.submoves[0].submoveid' RETURNING NUMBER) FROM move;
returns 1.
SELECT json_query(move_doc, '$.submoves[1]') FROM move;
would return the second JSON element, i.e. something like
{
submoveid : "2",
...
}
json_value is used to retrieve a scalar value, json_query is used to retrieve JSON values. You might also want to have a look at json_table which returns an SQL result table and thus can be used in Joins.
See this Oracle Doc for more examples
Beda here from the Oracle JSON team.
We have added a new multi-value index in release 21c allowing you to index values from a JSON array. Obviously, 21c is brand new and you want to know how to do this in older releases: Functional indexes (using JSON_Value function) are limited to a single value per JSON document and therefore are not capable to index array values. But: there is a 'JSON search index' which indexes your entire JSON document and therefore also values in the array. Another solution is to use a materialized view usign JSON_Table. This will expand the array values into separate rows.Then you can add a regular B-Tree index on that column.
Sample code here:
JSON indexing with functional indexes and JSON search index
https://livesql.oracle.com/apex/livesql/file/content_HN507PELCEEJGVNW4Q61L34DS.html
JSON and materialized views
https://livesql.oracle.com/apex/livesql/file/content_HYMB1YBP4CPMG6T6MXY5G9X5L.html
From what I've looked, In Oracle, you can index the "whole array" as a single index entry, but not individual elements of an array.
NoSQL databases like MongoDB, Couchbase, Cassandra have "array/collection" indexes which can index individual elements or fields of objects within an array and query them.

SQL LIKE operator to find words in stored JSON

I have this JSON stored in DB:
Column name: json
- '{"brand":"1","year":"2008","model":"2","price":"2001212","category":"Category Example"}'
- '{"brand":"1","year":"2008","model":"2","price":"2001212","category":"Category Example2"}'
I want to make a search using Like operator to find all categories with "Category" word:
At this moment Im doing it this way, but only return a complete phrase:
select * from table where json like '%"category":"Category Example"%';
How can I build a query that returns all categories with "Category word"?
Updated:
I'm using MySQL
Thanks
While undeclared this looks like a Postgres question.
There are hardly any JSON-processing tool in the current version 9.2.
But a whole set of tools will be shipped with the upcoming Postgres 9.3 currently in beta.
I interpret your question as:
Find all rows where the json column contains one or more fields named 'category' holding a value that contains the string 'Category'.
One ore more? Not sure if Postgres enforces uniqueness, I don't have a 9.3 installation at hand.
With Postgres 9.3, your query could look like this:
SELECT *
FROM tbl
WHERE json->>'category' LIKE '%Category%'
->> .. "Get JSON object field as text"
Use ILIKE for a case insensitive search.
More in this related answer:
How do I query using fields inside the new PostgreSQL JSON datatype?
Can you use a library? The "common schema" library offers a function that does just what you need:
http://common-schema.googlecode.com/svn/trunk/common_schema/doc/html/extract_json_value.html
Maybe I asked a really bad question, because I could make the search using Regexp.
I found this solution. Maybe this is not the fastest way, but does what I need:
select * from table where json regexp '"category":"([^"]*)Category([^"]*)"';
Thanks
I hope this helps.
select * from table where json #> '{"category":"Category Example"}';

How to get elasticsearch to perform similar to SQL 'LIKE'

If using a SQL 'Like' statement to query data it will return data even if its only partially matched. For instance, if I'm searching for food and there's an item in my db called "raisins" when using SQL the query would return "raisins" even if my search only contained "rai". In elasticsearch, the query won't return a record unless the entire name (in this case "raisins") is specified. How can I get elasticsearch to behave similar to the SQL statement. I'm using Rails 3.1.1 and PostgreSQL. Thanks!
While creating index of model for elasticsearch use tokenizer on index which will fulfil your requirement. For. e.g.
tokenizer: {
:name_tokenizer => {type: "edgeNGram", max_gram: 100, min_gram: 3, side: "front"}
}
this will create tokens of size from 3 to 100 of your fields and as side is given as front it will check from the starting. You can get more details from here http://www.slideshare.net/clintongormley/terms-of-endearment-the-elasticsearch-query-dsl-explained