Former sequelize person here.
What is the best method to extract data from MariaDB using a raw Knex statement? Here are the two methods i have tried:
MariaDb:
SELECT JSON_OBJECT(
...........
) 'JSON_OBJECT'
jscript to interpret results:
for ( tmp of result[0] ) { console.log (JSON.parse(tmp.JSON_OBJECT)) ; }
and this MariaDb:
SELECT JSON_ARRAYAGG(
JSON_OBJECT(
..........
)
) 'JSON_ARRAYAGG'
jscript:
for ( tmp of JSON.parse(result[0][0]['JSON_ARRAYAGG'] )) { console.log(tmp) ; }
Both of these methods work, but there may be a much cleaner way rather than using JSON.parse.
Suggestions?
note: i certainly understand there is a controversy using raw - i have a number of very large sql statements that were in a previous application, and i would rather (for now) just use them as is rather than rewrite them in pure Knex.
EDIT: this may be "good enough" since it appears that knex raw does not necessarily return objects.
node:
SELECT JSON_ARRAYAGG(
JSON_OBJECT(
..........
)
) 'JSON_ARRAYAGG'
.....................................
const result = await knex.raw( sqlStatement, sqlQuery);
return result[0][0].JSON_ARRAYAGG ;
browser:
(async () => { try { let result = await app.service('raw-service').find({query: sqlQuery}) ; newResult = result; } catch (e) { console.log(e); } } )() ;
console.log(JSON.parse(newResult));
but at least the really ugly stuff is hidden away, inside of node.
SQL databases are designed to return rows of columns; any query where you route everything through json instead is not going to be "the best method" you ask for.
Your first try at least uses rows, so is better than the second.
That said, if you are just modifying existing code to work with Knex rather than rewriting it, whatever requires minimum changes will lead to fewer bugs and should be preferred.
To extract data from a MariaDB database using Knex, you can use the .select() method on a Knex query builder object. This method allows you to specify the columns that you want to retrieve from the database, as well as any conditions or filters using the .where() method.
Related
My model in Laravel has a linked_ids string field like this:
echo $model->linked_ids
1,2,3,4,5
I want to make a query that gets me all records with a given id in linked_ids.
Currently I have:
Model::where('linked_ids', 'LIKE', '%' . $model->id . '%');
but this selects me more than I want to (if ex: $model->id is 3 => selects: 1,32,67)\
How can I avoid this since I don't know what position the id will be nor will the ids be ordered? I would like to do this in eloquent but can also use something like DB::raw() to run sql queries.
Bad way to keep your ids but if you really can't change it, you could take advantage of LazyCollections and filter with php.
I'm sure there's a way to do it directly in MySQL (or whatever dbms you're using) but this is what I have.
$id = 3;
Model::cursor()
->filter(function ($model) use ($id) {
return in_array($id, explode(',', $model->linked_ids));
})
// then chain one of these methods
->first(); // returns the first match or null
->collect(); // returns an Illuminate\Support\Collection of the results after the filtering
->all(); // returns an array of Models after the filtering
->toArray(); // returns an array and transforms the models to arrays as well.
->toJson(); // returns a json string
Take notice that this will still do a SELECT * FROM table without any filtering (unless you chain some where methods before cursor() but it won't load any model into memory (which is usually the bottleneck for big queries in Laravel)
I have something like this, using Akka, Alpakka + Slick
Slick
.source(
sql"""select #${onlyTheseColumns.mkString(",")} from #${dbSource.table}"""
.as[Map[String, String]]
.withStatementParameters(rsType = ResultSetType.ForwardOnly, rsConcurrency = ResultSetConcurrency.ReadOnly, fetchSize = batchSize)
.transactionally
).map( doSomething )...
I want to update this plain sql query with skipping the first N-th element.
But that is very DB specific.
Is is possible to get the pagination bit generated by Slick? [like for type-safe queries one just do a drop, filter, take?]
ps: I don't have the Schema, so I cannot go the type-safe way, just want all tables as Map, filter, drop etc on them.
ps2: at akka level, the flow.drop works, but it's not optimal/slow, coz it still consumes the rows.
Cheers
Since you are using the plain SQL, you have to provide a workable SQL in code snippet. Plain SQL may not type-safe, but agile.
BTW, the most optimal way is to skip N-th element by Database, such as limit in mysql.
depending on your database engine, you could use something like
val page = 1
val pageSize = 10
val query = sql"""
select #${onlyTheseColumns.mkString(",")}
from #${dbSource.table}
limit #${pageSize + 1}
offset #${pageSize * (page - 1)}
"""
the pageSize+1 part tells you whether the next page exists
I want to update this plain sql query with skipping the first N-th element. But that is very DB specific.
As you're concerned about changing the SQL for different databases, I suggest you abstract away that part of the SQL and decide what to do based on the Slick profile being used.
If you are working with multiple database product, you've probably already abstracted away from any specific profile, perhaps using JdbcProfile. In that case you could place your "skip N elements" helper in a class and use the active slickProfile to decide on the SQL to use. (As an alternative you could of course check via some other means, such as an environment value you set).
In practice that could be something like this:
case class Paginate(profile: slick.jdbc.JdbcProfile) {
// Return the correct LIMIT/OFFSET SQL for the current Slick profile
def page(size: Int, firstRow: Int): String =
if (profile.isInstanceOf[slick.jdbc.H2Profile]) {
s"LIMIT $size OFFSET $firstRow"
} else if (profile.isInstanceOf[slick.jdbc.MySQLProfile]) {
s"LIMIT $firstRow, $size"
} else {
// And so on... or a default
// Danger: I've no idea if the above SQL is correct - it's just placeholder
???
}
}
Which you could use as:
// Import your profile
import slick.jdbc.H2Profile.api._
val paginate = Paginate(slickProfile)
val action: DBIO[Seq[Int]] =
sql""" SELECT cols FROM table #${paginate.page(100, 10)}""".as[Int]
In this way, you get to isolate (and control) RDBMS-specific SQL in one place.
To make the helper more usable, and as slickProfile is implicit, you could instead write:
def page(size: Int, firstRow: Int)(implicit profile: slick.jdbc.JdbcProfile) =
// Logic for deciding on SQL goes here
I feel obliged to comment that using a splice (#$) in plain SQL opens you to SQL injection attacks if any of the values are provided by a user.
I am using Postgres as main DB and REDIS for caching. I am working on caching mechanism for one db query which takes to much time (It's about 5-6 JOINS + nested SELECTS). For now I am caching results of this query using SET 'some key' JSON.stringify(query.result). This works fine, however I have one column that cannot be cached - it is called commentsCount. It has to be always up to date. As a temporary solution, I am querying db just for this one particular field like this:
app.get('/post/getBySlug/:slug',function(req,res,next){
var cacheKey = req.params.slug+'|'+req.params.language; // "my-post-slug|en-us" for example
cache.get(cacheKey, function(err, post){
throw err if err;
if(post) {
db.getPostCommentsCount({ where: { id: post.id }}).done(function(err,commentsCount){
throw err if err;
post.commentsCount = commentsCount;
res.json(post);
next()
})
} else {
db.getFullPostBySlug(req.params.slug, req.params.language).done(function(err, post){
throw err if err;
cache.set(cacheKey, post);
res.json(post);
next();
})
}
})
})
But it is still now what I want, because main DB is still queried. Is there any standard/good practise on storing counters in REDIS? My comment insert function looks like this:
START TRANSACTION
INSERT INTO "Comments" VALUES (...) // insert comments
UPDATE "Posts" SET "commentsCount" = "commentsCount" + 1 WHERE "Posts"."id" = 123456 // update counter on post
COMMIT TRANSACTION
I am using transaction because I dont want comment to be inserted without incrementing comments count. As a "side" question - is it better to make 2 sql queries in transaction or write a trigger to handle incrementing counter??
According to my query (I posted link to gist in comments):
We dont plan more than 2 languages (though it is possible)
I made those counters because I have to keep counters separate per language, be able to order by those separate counters and also be able to order by sum of the counters (total for all languages) - I found it hard to make query that would order by sum of columns from separate rows while still returning those rows... (At the begining counters were stored in language translations).
Generally this query looks for post where exists translation with specific 'slug' and 'language' (slug+language on post translation is unique index). Morover post has to be published (isPublished = boolean) and post.status has to be 'published' (status = enum) or post.iscomingSoon has to be true (isComingSoon = boolean). Do you have idea what index/ordering I could add to this query? Or should I just remove limit?
In every translation table I keep language as TEXT. It can be for example en-us or zh-cn etc. Do you think I should make it enum or maybe I should make another table to store languages and just keep language_id in translations?
Author actually can be null :)
I have been reading some recipes in the Perl Hacks book. Recipe #24 "Query Databases Dynamically without SQL" looked interesting. The idea is to use SQL-Abstract to generate the SQL statement for you.
The syntax to generate a select statement looks something like this:
my($stmt, #bind) = $sql->select($table, \#fields, \%where, \#order);
To illustrate further, an example could look like this (taken from the perldoc):
my %where = (
requestor => 'inna',
worker => ['nwiger', 'rcwe', 'sfz'],
status => { '!=', 'completed' }
);
my($stmt, #bind) = $sql->select('tickets', '*', \%where);
The above would give you something like this:
$stmt = "SELECT * FROM tickets WHERE
( requestor = ? ) AND ( status != ? )
AND ( worker = ? OR worker = ? OR worker = ? )";
#bind = ('inna', 'completed', 'nwiger', 'rcwe', 'sfz');
Which you could then use in DBI code like so:
my $sth = $dbh->prepare($stmt);
$sth->execute(#bind);
Now, sometimes the order of the columns in the WHERE clause is very important, especially if you want to make good use of indexes.
But, since the columns to the WHERE clause generator in SQL-Abstract are specified by means of a hash - and as is known, the order that data is retrieved out of perl hashes cannot be guaranteed - you seem to loose the ability to specify the order of the columns.
Am i missing something? Is there an alternate facility to guarantee the order that columns appear in the WHERE clause when using SQL-Abstract ?
I originally misinterpreted your question.
You can use -and to achieve the desired ordering.
For example:
#!/usr/bin/perl
use strict; use warnings;
use SQL::Abstract;
my $sql = SQL::Abstract->new;
my ($stmt, #bind) = $sql->select(
tickets => '*',
{
-and => [
requestor => 'inna',
status => { '!=', 'completed' },
worker => ['nwiger', 'rcwe', 'sfz'],
],
}
);
print "$stmt\n";
See Nested conditions, -and/-or prefixes.
This module cannot do everything -- it is meant as a convenience for constructing queries that will do the job "most of the time". Sometimes you still may need to write a query by hand. I use SQL::Abstract in my main $work::app and have never run into the situation that you describe. A good SQL engine will know which keys are indexed, and optimize the query to use those first, no matter the ordering you specify. Are you sure that your engine is not the same, and that the order you specify in the query is really significant?
If you really need to order your WHERE clauses in a special order, you may find it easier to write subqueries instead. SQL::Abstract can make this easier too.
Setting the DBIC_TRACE environment variable to true:
BEGIN { $ENV{DBIC_TRACE} = 1 }
generates very helpful output, especially showing the SQL query that is being executed, but the SQL query is all on one line.
Is there a way to push it through some kinda "sql tidy" routine to format it better, perhaps breaking it up over multiple lines? Failing that, could anyone give me a nudge into where in the code I'd need to hack to add such a hook? And what the best tool is to accept a badly formatted SQL query and push out a nicely formatted one?
"nice formatting" in this context simply means better than "all on one line". I'm not particularly fussed about specific styles of formatting queries
Thanks!
As of DBIx::Class 0.08124 it's built in.
Just set $ENV{DBIC_TRACE_PROFILE} to console or console_monochrome.
From the documentation of DBIx::Class::Storage
If DBIC_TRACE is set then trace information is produced (as when the
debug method is set). ...
debug Causes trace information to be emitted on the debugobj
object. (or STDERR if debugobj has not specifically been set).
debugobj Sets or retrieves the object used for metric collection.
Defaults to an instance of DBIx::Class::Storage::Statistics that is
compatible with the original method of using a coderef as a callback.
See the aforementioned Statistics class for more information.
In other words, you should set debugobj in that class to an object that subclasses DBIx::Class::Storage::Statistics. In your subclass, you can reformat the query the way you want it to be.
First, thanks for the pointers! Partial answer follows ....
What I've got so far ... first some scaffolding:
# Connect to our db through DBIx::Class
my $schema = My::Schema->connect('dbi:SQLite:/home/me/accounts.db');
# See also BEGIN { $ENV{DBIC_TRACE} = 1 }
$schema->storage->debug(1);
# Create an instance of our subclassed (see below)
# DBIx::Class::Storage::Statistics class
my $stats = My::DBIx::Class::Storage::Statistics->new();
# Set the debugobj object on our schema's storage
$schema->storage->debugobj($stats);
And the definition of My::DBIx::Class::Storage::Statistics being:
package My::DBIx::Class::Storage::Statistics;
use base qw<DBIx::Class::Storage::Statistics>;
use Data::Dumper qw<Dumper>;
use SQL::Statement;
use SQL::Parser;
sub query_start {
my ($self, $sql_query, #params) = #_;
print "The original sql query is\n$sql_query\n\n";
my $parser = SQL::Parser->new();
my $stmt = SQL::Statement->new($sql_query, $parser);
#printf "%s\n", $stmt->command;
print "The parameters for this query are:";
print Dumper \#params;
}
Which solves the problem about how to hook in to get the SQL query for me to "pretty-ify".
Then I run a query:
my $rs = $schema->resultset('SomeTable')->search(
{
'email' => $email,
'others.some_col' => 1,
},
{ join => 'others' }
);
$rs->count;
However SQL::Parser barfs on the SQL generated by DBIx::Class:
The original sql query is
SELECT COUNT( * ) FROM some_table me LEFT JOIN others other_table ON ( others.some_col_id = me.id ) WHERE ( others.some_col_id = ? AND email = ? )
SQL ERROR: Bad table or column name '(others' has chars not alphanumeric or underscore!
SQL ERROR: No equijoin condition in WHERE or ON clause
So ... is there a better parser than SQL::Parser for the job?