I found and have used a closure that temporarily turns hibernate.SQL logging to Trace to allow me to see the exact queries that are generated. However I would like to be able to have PostgresQL's explain run automatically instead of having to pull out queries individually for analysis.
logging closure:
(found here: http://www.intelligrape.com/blog/2011/10/21/log-sql-in-grails-for-a-piece-of-code/)
public static def execute(Closure closure) {
Logger sqlLogger = Logger.getLogger("org.hibernate.SQL");
Logger transactionLogger = Logger.getLogger("org.hibernate.transaction");
Level currentLevel = sqlLogger.level
Level transLevel = transactionLogger.level
sqlLogger.setLevel(Level.TRACE)
transactionLogger.setLevel(Level.TRACE)
def result = closure.call()
sqlLogger.setLevel(currentLevel)
transactionLogger.setLevel(transLevel)
result
}
usage:
def result
execute{
result=Dog.createCriteria().list{
eq("breed","Greyhound")
}
}
I would like something that can be used in a similar way.
Is this something I could do with a sub-class of Criteria or Hibernate.Restrictions ?
Or is there something I'm missing in the docs on how to modify the SQL statement that is sent to the DB from GORM?
Thanks for any info.
Assuming you don't want all queries, but only the longer-running ones you might find the "auto explain" add-on for PostgreSQL useful.
http://www.postgresql.org/docs/9.1/static/auto-explain.html
Related
I have something like this, using Akka, Alpakka + Slick
Slick
.source(
sql"""select #${onlyTheseColumns.mkString(",")} from #${dbSource.table}"""
.as[Map[String, String]]
.withStatementParameters(rsType = ResultSetType.ForwardOnly, rsConcurrency = ResultSetConcurrency.ReadOnly, fetchSize = batchSize)
.transactionally
).map( doSomething )...
I want to update this plain sql query with skipping the first N-th element.
But that is very DB specific.
Is is possible to get the pagination bit generated by Slick? [like for type-safe queries one just do a drop, filter, take?]
ps: I don't have the Schema, so I cannot go the type-safe way, just want all tables as Map, filter, drop etc on them.
ps2: at akka level, the flow.drop works, but it's not optimal/slow, coz it still consumes the rows.
Cheers
Since you are using the plain SQL, you have to provide a workable SQL in code snippet. Plain SQL may not type-safe, but agile.
BTW, the most optimal way is to skip N-th element by Database, such as limit in mysql.
depending on your database engine, you could use something like
val page = 1
val pageSize = 10
val query = sql"""
select #${onlyTheseColumns.mkString(",")}
from #${dbSource.table}
limit #${pageSize + 1}
offset #${pageSize * (page - 1)}
"""
the pageSize+1 part tells you whether the next page exists
I want to update this plain sql query with skipping the first N-th element. But that is very DB specific.
As you're concerned about changing the SQL for different databases, I suggest you abstract away that part of the SQL and decide what to do based on the Slick profile being used.
If you are working with multiple database product, you've probably already abstracted away from any specific profile, perhaps using JdbcProfile. In that case you could place your "skip N elements" helper in a class and use the active slickProfile to decide on the SQL to use. (As an alternative you could of course check via some other means, such as an environment value you set).
In practice that could be something like this:
case class Paginate(profile: slick.jdbc.JdbcProfile) {
// Return the correct LIMIT/OFFSET SQL for the current Slick profile
def page(size: Int, firstRow: Int): String =
if (profile.isInstanceOf[slick.jdbc.H2Profile]) {
s"LIMIT $size OFFSET $firstRow"
} else if (profile.isInstanceOf[slick.jdbc.MySQLProfile]) {
s"LIMIT $firstRow, $size"
} else {
// And so on... or a default
// Danger: I've no idea if the above SQL is correct - it's just placeholder
???
}
}
Which you could use as:
// Import your profile
import slick.jdbc.H2Profile.api._
val paginate = Paginate(slickProfile)
val action: DBIO[Seq[Int]] =
sql""" SELECT cols FROM table #${paginate.page(100, 10)}""".as[Int]
In this way, you get to isolate (and control) RDBMS-specific SQL in one place.
To make the helper more usable, and as slickProfile is implicit, you could instead write:
def page(size: Int, firstRow: Int)(implicit profile: slick.jdbc.JdbcProfile) =
// Logic for deciding on SQL goes here
I feel obliged to comment that using a splice (#$) in plain SQL opens you to SQL injection attacks if any of the values are provided by a user.
I have a parameterised named Query like this :
Query moveOutQuery = session.createSQLQuery(moveOutQueryStr.toString())
.addEntity(MyClass.class)
.setParameter("assignmentStatus", Constants.CHECKED_OUT)
I want to see the actual SQL query with parameters filled in. However while debugging I only get the following query:
Select * from my_assignment WHERE assignment_status in ( :assignmentStatus )
Why isn't the assignmentStatus being substituted for its real value?
Why isn't the assignmentStatus being substituted for its real value?
This is because NHibernate use query parameters to input values. This is efficient in many cases and also helpful against SQL Injection attack. Parameters are sent separately. You can find them at the bottom if SQL is logged as explained below.
You may log each SQL to file as explained below.
This is implemented through log4net.dll; you need to add reference.
Add namespaces as below:
using log4net;
using log4net.Appender;
using log4net.Core;
using log4net.Layout;
using log4net.Repository.Hierarchy;
Configure log4net in NHibernate as below:
Hierarchy hierarchy = (Hierarchy)LogManager.GetRepository();
hierarchy.Root.RemoveAllAppenders();
FileAppender fileAppender = new FileAppender();
fileAppender.Name = "NHFileAppender";
fileAppender.File = logFilePath;
fileAppender.AppendToFile = true;
fileAppender.LockingModel = new FileAppender.MinimalLock();
fileAppender.Layout = new PatternLayout("%d{yyyy-MM-dd HH:mm:ss}:%m%n%n");
fileAppender.ActivateOptions();
Logger logger = hierarchy.GetLogger("NHibernate.SQL") as Logger;
logger.Additivity = false;
logger.Level = Level.Debug;
logger.AddAppender(fileAppender);
hierarchy.Configured = true;
You also need to set ShowSql while configuration as below:
configuration.SetProperty(NHibernate.Cfg.Environment.ShowSql, "true");
configuration.SetProperty(NHibernate.Cfg.Environment.FormatSql, "true");
You need to call this code once at startup of your application. Output log includes values of parameters as well.
Following is the code:
session.CreateSQLQuery("SELECT * FROM MyEntity WHERE MyProperty = :MyProperty")
.AddEntity(typeof(MyEntity))
.SetParameter("MyProperty", "filterValue")
.UniqueResult<MyEntity>();
Following is the logged query:
2020-01-09 14:25:39:
SELECT
*
FROM
MyEntity
WHERE
MyProperty = #p0;
#p0 = 'filterValue' [Type: String (4000:0:0)]
As you can see, parameter value filterValue is listed at the bottom.
This works for all query APIs like IQueryOver, IQuery, ISQLQuery etc.
This logs both success and failed statements. You can play with FileAppender and Logger class to meet your additional requirements.
Also refer PatternLayout from documentation. More details can also be found here, here and here. This Q/A discusses the same.
Following Q/A may also help:
Get executed SQL from nHibernate
Using log4net to write to different loggers
How to log SQL calls with NHibernate to the console of Visual Studio?
As you see, this logs the parameter values at bottom of the query. If you want those logged embedded in the query, please refer to this article.
I have a RavenDB Index that I am using to delete a bunch of documents at one time. I would like to know home many documents were actually deleted after the operation is done, however I can't figure out how to get that information from a DatabaseCommands call. For a RavenQuery/DocumentQuery you can use the out Statistics, but I haven't found anything for the DatabaseCommands.
I did find a RetrieveDetails flag as part of the BulkOperationsOptions but I'm not sure how to actually see the details. Here is what my query looks like
var op = connection.Store.DatabaseCommands.DeleteByIndex(
"Store/ByExpiration",
new IndexQuery
{
Query = "Expiration:yes"
},
new BulkOperationOptions
{
AllowStale = false,
RetrieveDetails = true
});
op.WaitForCompletion();
At this point I am not sure how to get the details once the operation has completed. Has anyone else figured out how to get these details?
The result is returned as a RavenJToken in WaitForCompletion.
var result = op.WaitForCompletion();
The documentation isn't clear on this.
Hint: When I can't find examples in the documentation I look at the RavenDB source code unit tests for examples, e.g. the ShouldRetrieveOperationDetailsWhenTheyWereRequested unit test.
If I were retrieving the data I wanted from a plain sql query, the following would suffice:
select * from stvterm where stvterm_code > TT_STUDENT.STU_GENERAL.F_Get_Current_term()
I have a grails domain set up correctly for this table, and I can run the following code successfully:
def a = SaturnStvterm.findAll("from SaturnStvterm as s where id > 201797") as JSON
a.render(response)
return false
In other words, I can hardcode in the results from the Oracle function and have the HQL run correctly, but it chokes any way that I can figure to try it with the function. I have read through some of the documentation on Hibernate about using procs and functions, but I'm having trouble making much sense of it. Can anyone give me a hint as to the proper way to handle this?
Also, since I think it is probably relevant, there aren't any synonyms in place that would allow the function to be called without qualifying it as schema.package.function(). I'm sure that'll make things more difficult. This is all for Grails 1.3.7, though I could use a later version if needed.
To call a function in HQL, the SQL dialect must be aware of it. You can add your function at runtime in BootStrap.groovy like this:
import org.hibernate.dialect.function.SQLFunctionTemplate
import org.hibernate.Hibernate
def dialect = applicationContext.sessionFactory.dialect
def getCurrentTerm = new SQLFunctionTemplate(Hibernate.INTEGER, "TT_STUDENT.STU_GENERAL.F_Get_Current_term()")
dialect.registerFunction('F_Get_Current_term', getCurrentTerm)
Once registered, you should be able to call the function in your queries:
def a = SaturnStvterm.findAll("from SaturnStvterm as s where id > TT_STUDENT.STU_GENERAL.F_Get_Current_term()")
Setting the DBIC_TRACE environment variable to true:
BEGIN { $ENV{DBIC_TRACE} = 1 }
generates very helpful output, especially showing the SQL query that is being executed, but the SQL query is all on one line.
Is there a way to push it through some kinda "sql tidy" routine to format it better, perhaps breaking it up over multiple lines? Failing that, could anyone give me a nudge into where in the code I'd need to hack to add such a hook? And what the best tool is to accept a badly formatted SQL query and push out a nicely formatted one?
"nice formatting" in this context simply means better than "all on one line". I'm not particularly fussed about specific styles of formatting queries
Thanks!
As of DBIx::Class 0.08124 it's built in.
Just set $ENV{DBIC_TRACE_PROFILE} to console or console_monochrome.
From the documentation of DBIx::Class::Storage
If DBIC_TRACE is set then trace information is produced (as when the
debug method is set). ...
debug Causes trace information to be emitted on the debugobj
object. (or STDERR if debugobj has not specifically been set).
debugobj Sets or retrieves the object used for metric collection.
Defaults to an instance of DBIx::Class::Storage::Statistics that is
compatible with the original method of using a coderef as a callback.
See the aforementioned Statistics class for more information.
In other words, you should set debugobj in that class to an object that subclasses DBIx::Class::Storage::Statistics. In your subclass, you can reformat the query the way you want it to be.
First, thanks for the pointers! Partial answer follows ....
What I've got so far ... first some scaffolding:
# Connect to our db through DBIx::Class
my $schema = My::Schema->connect('dbi:SQLite:/home/me/accounts.db');
# See also BEGIN { $ENV{DBIC_TRACE} = 1 }
$schema->storage->debug(1);
# Create an instance of our subclassed (see below)
# DBIx::Class::Storage::Statistics class
my $stats = My::DBIx::Class::Storage::Statistics->new();
# Set the debugobj object on our schema's storage
$schema->storage->debugobj($stats);
And the definition of My::DBIx::Class::Storage::Statistics being:
package My::DBIx::Class::Storage::Statistics;
use base qw<DBIx::Class::Storage::Statistics>;
use Data::Dumper qw<Dumper>;
use SQL::Statement;
use SQL::Parser;
sub query_start {
my ($self, $sql_query, #params) = #_;
print "The original sql query is\n$sql_query\n\n";
my $parser = SQL::Parser->new();
my $stmt = SQL::Statement->new($sql_query, $parser);
#printf "%s\n", $stmt->command;
print "The parameters for this query are:";
print Dumper \#params;
}
Which solves the problem about how to hook in to get the SQL query for me to "pretty-ify".
Then I run a query:
my $rs = $schema->resultset('SomeTable')->search(
{
'email' => $email,
'others.some_col' => 1,
},
{ join => 'others' }
);
$rs->count;
However SQL::Parser barfs on the SQL generated by DBIx::Class:
The original sql query is
SELECT COUNT( * ) FROM some_table me LEFT JOIN others other_table ON ( others.some_col_id = me.id ) WHERE ( others.some_col_id = ? AND email = ? )
SQL ERROR: Bad table or column name '(others' has chars not alphanumeric or underscore!
SQL ERROR: No equijoin condition in WHERE or ON clause
So ... is there a better parser than SQL::Parser for the job?