I have a postgres table with jsonb array elements and I'm trying to do sql queries to extract the matching elements. I have the raw SQL query running from the postgres command line interface:
select * from movies where director #> any (array ['70', '45']::jsonb[])
This returns the results I'm looking for (all records from the movies table where the director jsonb elements contain any of the elements in the input element).
In the code, the value for ['70, '45'] would be a dynamic variable ie. fixArr and the length of the array is unknown.
I'm trying to build this into my Bookshelf code but haven't been able to find any examples that address the complexity of the use case. I've tried the following approaches but none of them work:
models.Movies.where('director', '#> any', '(array' + JSON.stringify(fixArr) + '::jsonb[])').fetchAll()
ERROR: The operator "#> any" is not permitted
db.knex.raw('select * from movies where director #> any(array'+[JSON.stringify(fixArr)]+'::jsonb[])')
ERROR: column "45" does not exist
models.Movies.query('where', 'director', '#>', 'any (array', JSON.stringify(fixArr) + '::jsonb[])').fetchAll()
ERROR: invalid input syntax for type json
Can anyone help with this?
As you have noticed, knex nor bookshelf doesn't bring any support for making jsonb queries easier. As far as I know the only knex based ORM that supports jsonb queries etc. nicely is Objection.js
In your case I suppose better operator to find if jsonb column contains any of the given values would be ?|, so query would be like:
const idsAsString = ids.map(val => `'${val}'`).join(',');
db.knex.raw(`select * from movies where director \\?| array[${idsAsString}]`);
More info how to deal with jsonb queries and indexing with knex can be found here https://www.vincit.fi/en/blog/objection-js-postgresql-power-json-queries/
No, you're just running into the limitations of that particular query builder and ORM.
The easiest way is using bookshelf.Model.query and knex.raw (whereRaw, etc.). Alias with AS and subclass your Bookshelf model to add these aliased attributes if you care about such things.
If you want things to look clean and abstracted through Bookshelf, you'll just need to denormalize the JSONB into flat tables. This might be the best approach if your JSONB is mostly flat and simple.
If you end up using lots of JSONB (it can be quite performant with appropriate indexes) then Bookshelf ORM is wasted effort. The knex query builder is only not a waste of time insofar as it handles escaping, quoting, etc.
Related
Is it possible to generate a json object using the column name as keys automatically?
I have a table with many columns and I need to dump it into a json object.
I know I can do this using the JSON_OBJECT function but I was looking for a more condensed syntax that would allow me to do this without having to specify the name of all the columns
SELECT JSON_OBJECT("col_a", m.col_a, "col_b", m.col_b, "col_c", m.col_c, ...)
FROM largetable AS m
Something like this?
SELECT JSON_OBJECT(m.*)
FROM largetable AS m
I'm using MariaDB version 10.8.2
Json objects make sense in other languages like javascript, C#... There are many libraries to convert the result of a MariaDB query into a list of json objects in most languages.
Also, a good practice is to make the database engine do as little effort as possible when performing queries and processing the result in the application.
This is of course not possible, since the parser would not accept an odd number of parameters for the JSON_OBJECT function.
To do that in pure SQL, you can't do that within a single statement, since you need to retrieve the column names from information_schema first:
select #statement:=concat("SELECT JSON_OBJECT(", group_concat(concat("\"",column_name,"\"", ",", column_name)),") FROM mytable") from information_schema.columns where table_name="mytable" and table_schema="test";
prepare my_statement from statement;
execute my;
Much easier and faster is to convert the result in your application, for example in Python:
import mariadb, json
conn= mariadb.connect(db="test")
cursor= conn.cursor(dictionary=True)
cursor.execute("select * from mytable")
json_row= json.dumps(cursor.fetchone())
I am evaluating if we can migrate from plain JDBC to jOOQ for our project. Most of it looks promising, but I am wondering currently about one specific flow: nested rows. Let me explain.
Say you have the following two tables:
class(id, name)
student(id, name, class_id)
(We assume that a student can only be part of one class.)
Let's create a response type for these tables. I will be using these in the queries below.
create type type_student as(id integer, name text);
create type type_class as(id integer, name text, students type_student[]);
Now let's fetch all classes with its student by using nested rows:
select row(class.id, class.name, array
(
select row(student.id, student.name)::type_student
from student
where student.class_id = class.id
))::type_class
from class
A useful variant is to use only nested rows in arrays:
select class.id, class.name, array
(
select row(student.id, student.name)::type_student
from student
where student.class_id = class.id
) as students
from class
I am wondering if jOOQ has an elegant approach to parse such results containing nested rows?
Your usage of the word "parse" could mean several things, and I'll answer them all in case anyone finds this question looking for "jOOQ" / "parse" / "row".
Does the org.jooq.Parser support row value expressions?
Not yet (as of jOOQ 3.10 and 3.11). jOOQ ships with a SQL parser that parses (almost) anything that can be represented using the jOOQ API. This has various benefits, including:
Being able to reverse engineer DDL scripts for the code generator
Translating SQL between dialects (see an online version here: https://www.jooq.org/translate)
Unfortunately, it cannot parse row value expressions in the projection yet i.e. in the SELECT clause.
Does the jOOQ API support ("parse") row value expressions?
Yes, you can use them using the various DSL.row() constructors, mainly for predicates, but also for projections by wrapping them in a Field using DSL.rowField(). As of jOOQ 3.11, this is still a bit experimental as there are many edge cases in PostgreSQL itself, related to what is allowed and what isn't. But in principle, queries like yours should be possible
Does jOOQ support parsing the serialised version of a PostgreSQL record
PostgreSQL supports these anonymous record types, as well as named "composite" types. And arrays thereof. And nesting of arrays and composite types. jOOQ can serialise and deserialise these types if type information is available to jOOQ, i.e. if you're using the code generator. For instance, if your query is stored as a view
create view test as
select row(class.id, class.name, array
(
select row(student.id, student.name)::type_student
from student
where student.class_id = class.id
))::type_class
from class
Then, the code generator will produce the appropriate types, including:
TypeStudentRecord
TypeClassRecord
Which can be serialised as expected. In principle, this would be possible also without the code generator, but you'd have to create the above types yourself, manually, so why not just use the code generator.
Yes it does: https://www.jooq.org/doc/latest/manual/sql-building/table-expressions/nested-selects/
Field<Object> records =
create.select(student.id, student.name)
.from(student)
.where(student.class_id.eq(class.id)
.asField("students");
create.select(class.id, class.name, array, records)
.from(class)
.fetch();
The above example might not work directly as I have not tried, but just wanted to give a general idea.
Note: that the object records is not executed alone. When fetch is called in the second statement, JOOQ should create one SQL statement internally.
I am having a problem performing a where query against an array field in my Postgres database.
In my rails app i have a table called People. One column in this is called pets. Now this column contains array values, ie:
["dog", "cat", "fish"]
I would like to perform a query that returns all the people that have a pet dog for example.
The solution ive been using so far looks as such
People.where("\"pets\" #> '{\"" + checkedPet + "\"}'")
where checkedPet is a variable and could be "dog" or any other animal.
This works but i feel is vulnerable to a SQL injection problem?
Is this the case? If so what is a better and safer solution to avoid it?
According to ActiveRecord and PostgreSQL guide you can do the following:
People.where('? = any("pets")', checkedPet)
Or
People.where('"pets" #> ?', "{#{checkedPet}}")
I'm using Hibernate and Postgres and defined a character(1)[] column type.
So I don´t know how to make this criteria to find a value in the array.
Like this query
SELECT * FROM cpfbloqueado WHERE bloqueados #> ARRAY['V']::character[]
I am not familiar with Postgres and its types but you can define your own type using custom basic type mapping. That could simplify the query.
There are many threads here on SO regarding Postres array types and Hibernate, for instance, this one. Another array mapping example that could be useful is here. At last, here is an example of using Criteria with user type.
Code example could be
List result = session.createCriteria(Cpfbloqueado.class)
.setProjection(Projections.projectionList()
.add(Projections.property("characterColumn.attribute"), PostgresCharArrayType.class)
)
.setResultTransformer(Transformer.aliasToBean(Cpfbloqueado.class))
.add(...) // add where restrictions here
.list()
Also, if it is not important for the implementation, you can define max length in the entity model, annotating your field with #Column(length = 1).
Or if you need to store an array of characters with length of 1 it is possible to use a collection type.
I hope I got the point right, however, it would be nice if the problem domain was better described.
So you have array of single characters... Problem is that in PG that is not fixed length. I had this problem, but around 10 years ago. At that time I had that column mapped as string, and that way I was able to process internal data - simply slice by comma, and do what is needed.
If you hate that way, as I did... Look for columns with text[] type - that is more common, so it is quite easy to find out something. Please look at this sample project:
https://github.com/phstudy/jpa-array-converter-sample
I am having some trouble understanding CSqlDataProvider and how it works.
When I am using CActiveDataProvider, the results can be accessed as follows:
$data->userProfile['first_name'];
However, when I use CSqlDataProvider, I understand that the results are returned as an array not an object. However, the structure of the array is flat. In other words, I am seeing the following array:
$data['first_name']
instead of
$data['userProfile']['first_name']
But the problem here is what if I have another joined table (let's call it 'author') in my sql code that also contains a first_name field? With CActiveDataProvider, the two fields are disambiguated, so I can do the following to access the two fields:
$data->userProfile['first_name'];
$data->author['first_name'];
But with CSqlDataProvider, there doesn't seem to be anyway I can access the data as follows:
$data['userProfile']['first_name'];
$data['author']['first_name'];
So, outside of assigning a unique name to those fields directly inside my SQL, by doing something like this:
select author.first_name as author_first_name, userProfile.first_name as user_first_name
And then referring to them like this:
$data['author_first_name'];
$data['user_first_name']
is there anyway to get CSqlDataProvider to automatically structure the arrays so they are nested in the same way that CActiveDataProvider objects are? So that I can call them by using $data['userProfile']['first_name']
Or is there another class I should be using to obtain these kinds of nested arrays?
Many thanks!
As far as I can tell, no Yii DB methods break out JOIN query results in to 2D arrays like you are looking for. I think you will need to - as you suggest - alias the column names in your select statement.
MySql returns a single row of data when you JOIN tables in a query, and CSqlDataProvider returns exactly what MySql does: single tabular array representation indexed/keyed by the column names, just like your query returns.
If you want to break apart your results into a multi-dimensional array I would either alias the columns, or use a regular CActiveDataProvider (which you can still pass complex queries and joins in via CDbCritiera).