Can I pretty-print the DBIC_TRACE output in DBIx::Class? - sql

Setting the DBIC_TRACE environment variable to true:
BEGIN { $ENV{DBIC_TRACE} = 1 }
generates very helpful output, especially showing the SQL query that is being executed, but the SQL query is all on one line.
Is there a way to push it through some kinda "sql tidy" routine to format it better, perhaps breaking it up over multiple lines? Failing that, could anyone give me a nudge into where in the code I'd need to hack to add such a hook? And what the best tool is to accept a badly formatted SQL query and push out a nicely formatted one?
"nice formatting" in this context simply means better than "all on one line". I'm not particularly fussed about specific styles of formatting queries
Thanks!

As of DBIx::Class 0.08124 it's built in.
Just set $ENV{DBIC_TRACE_PROFILE} to console or console_monochrome.

From the documentation of DBIx::Class::Storage
If DBIC_TRACE is set then trace information is produced (as when the
debug method is set). ...
debug Causes trace information to be emitted on the debugobj
object. (or STDERR if debugobj has not specifically been set).
debugobj Sets or retrieves the object used for metric collection.
Defaults to an instance of DBIx::Class::Storage::Statistics that is
compatible with the original method of using a coderef as a callback.
See the aforementioned Statistics class for more information.
In other words, you should set debugobj in that class to an object that subclasses DBIx::Class::Storage::Statistics. In your subclass, you can reformat the query the way you want it to be.

First, thanks for the pointers! Partial answer follows ....
What I've got so far ... first some scaffolding:
# Connect to our db through DBIx::Class
my $schema = My::Schema->connect('dbi:SQLite:/home/me/accounts.db');
# See also BEGIN { $ENV{DBIC_TRACE} = 1 }
$schema->storage->debug(1);
# Create an instance of our subclassed (see below)
# DBIx::Class::Storage::Statistics class
my $stats = My::DBIx::Class::Storage::Statistics->new();
# Set the debugobj object on our schema's storage
$schema->storage->debugobj($stats);
And the definition of My::DBIx::Class::Storage::Statistics being:
package My::DBIx::Class::Storage::Statistics;
use base qw<DBIx::Class::Storage::Statistics>;
use Data::Dumper qw<Dumper>;
use SQL::Statement;
use SQL::Parser;
sub query_start {
my ($self, $sql_query, #params) = #_;
print "The original sql query is\n$sql_query\n\n";
my $parser = SQL::Parser->new();
my $stmt = SQL::Statement->new($sql_query, $parser);
#printf "%s\n", $stmt->command;
print "The parameters for this query are:";
print Dumper \#params;
}
Which solves the problem about how to hook in to get the SQL query for me to "pretty-ify".
Then I run a query:
my $rs = $schema->resultset('SomeTable')->search(
{
'email' => $email,
'others.some_col' => 1,
},
{ join => 'others' }
);
$rs->count;
However SQL::Parser barfs on the SQL generated by DBIx::Class:
The original sql query is
SELECT COUNT( * ) FROM some_table me LEFT JOIN others other_table ON ( others.some_col_id = me.id ) WHERE ( others.some_col_id = ? AND email = ? )
SQL ERROR: Bad table or column name '(others' has chars not alphanumeric or underscore!
SQL ERROR: No equijoin condition in WHERE or ON clause
So ... is there a better parser than SQL::Parser for the job?

Related

tdbc::tokenize documentation and use

I'm using tdbc::odbc to connect to a Pervasive (btrieve type) database, and am unable to pass variables to the driver. A short test snippet:
set customer "100000"
set st [pvdb prepare {
INSERT INTO CUSTOMER_TEMP_EMPTY
SELECT * FROM CUSTOMER_MASTER
WHERE CUSTOMER = :customer
}]
$st execute
This returns:
[Pervasive][ODBC Client Interface]Parameter number out of range.
(binding the 'customer' parameter)
Works fine if I replace :customer with "100000", and I have tried using a variable with $, #, wrapping in apostrophes, quotes, braces. I believe that tdbc::tokenize is the answer I'm looking for, but the man page gives no useful information on its use. I've experimented with tokenize with no progress at all. Can anyone comment on this?
The tdbc::tokenize command is a helper for writing TDBC drivers. It's used for working out what bound variables are inside an SQL string so the binding map can be supplied to the low level driver or, in the case of particularly stupid drivers, string substitutions performed (I hope there's no drivers that need to do this; it'd be annoyingly difficult to get right). The parser knows enough to handle weird cases like things that look like bound variables in strings and comments (those aren't bound variables).
If we feed it (it's calling syntax is trivial) the example SQL you've got, we get this result:
{
INSERT INTO CUSTOMER_TEMP_EMPTY
SELECT * FROM CUSTOMER_MASTER
WHERE CUSTOMER = } :customer {
}
That's a list of three items (the last element just has a newline in it) that simplifies processing a lot; each item is either trivially a bound variable or trivially not.
Other examples (bear in mind in the second case that bound variables may also start with $ or #):
% tdbc::tokenize {':abc' = :abc = ":abc" -- :abc}
{':abc' = } :abc { = ":abc" -- :abc}
% tdbc::tokenize {foo + $bar - #grill}
{foo + } {$bar} { - } #grill
% tdbc::tokenize {foo + :bar + [:grill]}
{foo + } :bar { + [:grill]}
Note that the tokenizer does not fully understand SQL! It makes no attempt to parse the other bits; it's just looking for what is a bound variable.
I've no idea what use the tokenizer could be to you if you're not writing a DB driver.
Still could not get the driver to accept the variable, but looking at your first example of the tokenized return, I came up with:
set customer "100000"
set v [tdbc::tokenize "$customer"]
set query "INSERT INTO CUSTOMER_TEMP_EMPTY SELECT * FROM CUSTOMER_MASTER WHERE CUSTOMER = $v"
set st [pvdb prepare $query]
$st execute
as a test command, and it did indeed successfully pass the command through the driver

passing msqli to a function - can't suss out why it's not working

I've searched high and low for an answer on this, but I'm either missing something, or I just can't find anything in this context.
Background - trying to avoid spaghetti frenzy with a little casual project I'm starting; part of this will involve mainly just calls to a mysql database, displaying table content and so on. Simply put, like a CRM I guess.
I may be way off base here, but I want to be able to create my sql calls as functions, which will help if/when I tweak and tune, as well as creating a cleaner code for what I'm looking to do.
So, without further ado, I have this as a demomstration:
echo "<table>";
selectall('actions','content',$mysqli);
echo "</table><br><br>";
What this does is show all rows from my table of 'actions. "content" is just an example field name I'm passing through that I want to display, as it is the main human-relevant field name in that table. I'm also passing $mysqli through here for my function db call.
My function looks like this:
function selectall($s_table,$s_content,$mysqli){
$query = "SELECT * FROM " . $s_table;
$resource = $mysqli->query($query);
if ( !$resource ) throw new Exception($db->error);
while ( $row = $resource->fetch_assoc() ) {
$id = $row['id'];
echo "<tr><td>{$row[$s_content]}</td></tr>";
}
$resource->free();
$mysqli->close();
}
However.... it doesn't work, and it seems to throw a wobbly saying:
Warning: mysqli::query(): Couldn't fetch mysqli
This points to the action within the line $resource = $mysqli->query($query);
I know the function and everything is ok, as if I restate and declare $mysqli within the first line of the function, like so...
$mysqli = new mysqli(username password and so on in here);
... it works spot on.
$mysqli exists and works within the same code that is passing the variable within the function too.
This is early stages, so by shuffling the code around trying to poke the $mysqli pass into life I have perhaps made the code a little messier that intended, so try not to worry too much about that.
Anyone any ideas why it doesn't like this?
D'oh...
I had a
$mysqli->close();
in the lines above. Solved myself.
For reference, this is my function:
function selectall($s_table,$s_field,$mysqli){
if ($mysqli->connect_error) {die('Connect Error (' . $mysqli->connect_errno . ') '. $mysqli->connect_error);}
$s_table = preg_replace('/[^0-9a-zA-Z_]/', '', $s_table); // Cleans up the table name variable
$s_field = preg_replace('/[^0-9a-zA-Z_]/', '', $s_field); // Cleans up the field name variable
$query = "SELECT * FROM " . $s_table; // Adds passed table name to the select all
$resource = $mysqli->query($query);
if ( !$resource ) throw new Exception($db->error);
while ( $row = $resource->fetch_assoc() ) {
echo "<tr><td>{$row[$s_field]}</td></tr>"; // Content for each row of the select all
}
$resource->free();
$mysqli->close();
}
As you can see, I've also tried to protect the variables that enter the function.
This can be called via:
selectall('actions','content',$mysqli);
In this context, I want to view all the entries in the 'actions' table by the field name 'content'. This function, with some code above and below for a table, will create a new row for each entry.
I'll probably evolve a few, already created on that includes a delete button at the end of the line which is 'selectalldel'.
Open to comments on whether this actually is worthwhile, but thought I'd post up my corrected stupidity in case anyone finds this useful.

Groovy SQL Multiple ResultSets

I am calling a stored procedure from my Groovy code. The stored proc looks like this
SELECT * FROM blahblahblah
SELECT * FROM suchAndsuch
So basically, two SELECT statements and therefore two ResultSets.
sql.eachRow("dbo.testing 'param1'"){ rs ->
println rs
}
This works fine for a single ResultSet. How can I get the second one (or an arbitrary number of ResultSets for that matter).
You would need callWithAllRows() or its variant.
The return type of this method is List<List<GroovyRowResult>>.
Use this when calling a stored procedure that utilizes both output
parameters and returns multiple ResultSets.
This question is kind of old, but I will answer since I came across the same requirement recently and it maybe useful for future reference for me and others.
I'm working on a Spring application with SphinxSearch. When you run a query in sphinx, you get results, you need to run a second query to get the metadata for number of records etc...
// the query
String query = """
SELECT * FROM INDEX_NAME WHERE MATCH('SEARCHTERM')
LIMIT 0,25 OPTION MAX_MATCHES=25;
SHOW META LIKE 'total_found';
"""
// create an instance of our groovy sql (sphinx doesn't use a username or password, jdbc url is all we need)
// connection can be created from java, don't have to use groovy for it
Sql sql = Sql.newInstance('jdbc:mysql://127.0.0.1:9306/?characterEncoding=utf8&maxAllowedPacket=512000&allowMultiQueries=true','sphinx','sphinx123','com.mysql.jdbc.Driver')
// create a prepared statement so we can execute multiple resultsets
PreparedStatement ps = sql.getConnection().prepareStatement(query)
// execute the prepared statement
ps.execute()
// get the first result set and pass to GroovyResultSetExtension
GroovyResultSetExtension rs1 = new GroovyResultSetExtension(ps.getResultSet())
rs1.eachRow {
println it
}
// call getMoreResults on the prepared statement to activate the 2nd set of results
ps.getMoreResults()
// get the second result set and pass to GroovyResultSetExtension
GroovyResultSetExtension rs2 = new GroovyResultSetExtension(ps.getResultSet())
rs2.eachRow {
println it
}
Just some test code, this needs some improving on. You can loop the result sets and do whatever processing...
Comments should be self-explanatory, hope it helps others in the future!

Perl DBI modifying Oracle database by creating a VIEW

I wrote a Perl script to check the data in an Oracle database. Because the query process is very complex I chose to create a VIEW in the middle. Using this view the code could be largely simplified.
The Perl code run well when I used it to query the database starting from a file, like Perl mycode.pl file_a. The Perl code reads lines from file_a and creates/updates the view until the end of the input. The results I achieved are completely right.
The problem came when I simultaneously run
perl mycode.pl file_a
and
perl mycode.pl file_b
to access the same database. According to my observation, the VIEW used by the first process will be modified by the second process. These two processes were intertwined on the same view.
Is there any suggestion to make these two processes not conflict with one another?
The Perl code for querying database is normally like this, but the details in each real query is more complex.
my ($gcsta,$gcsto,$cms) = #t; #(details of #t is read from a line in file a or b)
my $VIEWSS = 'CREATE OR REPLACE VIEW VIEWSS AS SELECT ID,GSTA,GSTO,GWTA FROM TABLEA WHERE GSTA='.$gcsta.' AND GSTO='.$gcsto.' AND CMS='.$cms;
my $querying = q{ SELECT COUNT(*) FROM VIEWSS WHERE VIEWSS.ID=1};
my $inner_sth = $dbh->prepare($VIEWSS);
my $inner_rv = $inner_sth->execute();
$inner_sth = $dbh->prepare($querying);
$inner_rv = $inner_sth->execute();
You must
Create the view only once, and use it everywhere
Use placeholders in your SQL statements, and pass the actual parameters with the call to execute
Is this the full extent of your SQL? Probably not, but if so it really is fairly simple.
Take a look at this refactoring for some ideas. Note that is uses a here document to express the SQL. The END_SQL marker for the end of the text must have no whitespace before or after it.
If your requirement is more complex than this then please describe it to us so that we can better help you
my $stmt = $dbh->prepare(<<'END_SQL');
SELECT count(*)
FROM tablea
WHERE gsta = ? AND gsto = ? AND cms= ? AND id = 1
END_SQL
my $rv = $stmt->execute($gcsta, $gcsto, $cms);
If you must use a view then you should use placeholders in the CREATE VIEW as before, and make every set of changes into a transaction so that other processes can't interfere. This involves disabling AutoCommit when you create the database handle $dbh and adding a call to $dbh->commit when all the steps are complete
use strict;
use warnings;
use DBI;
my $dbh = DBI->connect('dbi:Oracle:mydbase', 'user', 'pass',
{ AutoCommit => 0, RaiseError => 1 } );
my $make_view = $dbh->prepare(<<'END_SQL');
CREATE OR REPLACE VIEW viewss AS
SELECT id, gsta, gsto, gwta
FROM tablea
WHERE gsta = ? AND gsto = ? AND cms= ? AND id = 1
END_SQL
my $get_count = $dbh->prepare(<<'END_SQL');
SELECT count(*)
FROM viewss
WHERE id = 1
END_SQL
while (<>) {
my ($gcsta, $gcsto, $cms) = split;
my $rv = $make_view->execute($gcsta, $gcsto, $cms);
$rv = $get_count->execute;
my ($count) = $get_count->fetchrow_array;
$dbh->commit;
}
Is the view going to be the same or different?
If the views are all the same then create it only once, or check if it exists with the all_views table : http://docs.oracle.com/cd/B12037_01/server.101/b10755/statviews_1202.htm#i1593583
You can easily create a view including your pid with the $$ variable to be the pid, but it wont be unique across computers, oracle has also some unique ids, see http://docs.oracle.com/cd/B14117_01/server.101/b10759/functions150.htm, for example, the SESSIONID.
But do you really need to do this? why dont you prepare a statement and then execute it? http://search.cpan.org/dist/DBI/DBI.pm#prepare
thanks,
mike

SQL Injection: is this secure?

I have this site with the following parameters:
http://www.example.com.com/pagination.php?page=4&order=comment_time&sc=desc
I use the values of each of the parameters as a value in a SQL query.
I am trying to test my application and ultimately hack my own application for learning purposes.
I'm trying to inject this statement:
http://www.example.com.com/pagination.php?page=4&order=comment_time&sc=desc' or 1=1 --
But It fails, and MySQL says this:
Warning: mysql_fetch_assoc() expects parameter 1 to be resource,
boolean given in /home/dir/public_html/pagination.php on line 132
Is my application completely free from SQL injection, or is it still possible?
EDIT: Is it possible for me to find a valid sql injection statement to input into one of the parameters of the URL?
The application secured from sql injection never produces invalid queries.
So obviously you still have some issues.
Well-written application for any input produces valid and expected output.
That's completely vulnerable, and the fact that you can cause a syntax error proves it.
There is no function to escape column names or order by directions. Those functions do not exist because it is bad style to expose the DB logic directly in the URL, because it makes the URLs dependent on changes to your database logic.
I'd suggest something like an array mapping the "order" parameter values to column names:
$order_cols = array(
'time' => 'comment_time',
'popular' => 'comment_score',
... and so on ...
);
if (!isset($order_cols[$_GET['order'])) {
$_GET['order'] = 'time';
}
$order = $order_cols[$_GET['order']];
Restrict "sc" manually:
if ($_GET['sc'] == 'asc' || $_GET['sc'] == 'desc') {
$order .= ' ' . $_GET['sc'];
} else {
$order .= ' desc';
}
Then you're guaranteed safe to append that to the query, and the URL is not tied to the DB implementation.
I'm not 100% certain, but I'd say it still seems vulnerable to me -- the fact that it's accepting the single-quote (') as a delimiter and then generating an error off the subsequent injected code says to me that it's passing things it shouldn't on to MySQL.
Any data that could possibly be taken from somewhere other than your application itself should go through mysql_real_escape_string() first. This way the whole ' or 1=1 part gets passed as a value to MySQL... unless you're passing "sc" straight through for the sort order, such as
$sql = "SELECT * FROM foo WHERE page='{$_REQUEST['page']}' ORDER BY data {$_REQUEST['sc']}";
... which you also shouldn't be doing. Try something along these lines:
$page = mysql_real_escape_string($_REQUEST['page']);
if ($_REQUEST['sc'] == "desc")
$sortorder = "DESC";
else
$sortorder = "ASC";
$sql = "SELECT * FROM foo WHERE page='{$page}' ORDER BY data {$sortorder}";
I still couldn't say it's TOTALLY injection-proof, but it's definitely more robust.
I am assuming that your generated query does something like
select <some number of fields>
from <some table>
where sc=desc
order by comment_time
Now, if I were to attack the order by statement instead of the WHERE, I might be able to get some results... Imagine I added the following
comment_time; select top 5 * from sysobjects
the query being returned to your front end would be the top 5 rows from sysobjects, rather than the query you try to generated (depending a lot on the front end)...
It really depends on how PHP validates those arguments. If MySQL is giving you a warning, it means that a hacker already passes through your first line of defence, which is your PHP script.
Use if(!preg_match('/^regex_pattern$/', $your_input)) to filter all your inputs before passing them to MySQL.