Log every SQL query to database in Rails - sql

I want to save to a log file some SQL query rails performs, (namely the CREATE, UPDATE and DELETE ones)
therefore I need to intercept all queries and then filter them maybe with some regexp and log them as needed.
Where would I put such a thing in the rails code?

Here a simplified version of what c0r0ner linked to, to better show it:
connection = ActiveRecord::Base.connection
class << connection
alias :original_exec :execute
def execute(sql, *name)
# try to log sql command but ignore any errors that occur in this block
# we log before executing, in case the execution raises an error
begin
File.open(Rails.root.join("/log/sql.txt"),'a'){|f| f.puts Time.now.to_s+": "+sql}
rescue Exception => e
;
end
# execute original statement
original_exec(sql, *name)
end
end

SQL logging in rails -
In brief - you need to override ActiveRecord execute method. There you can add any logic for logging.

As a note for followers, you can "log all queries" like Rails - See generated SQL queries in Log files and then grep the files for the ones you want, if desired.

If you are using mysql I would look into mysqlbinlog . It is going to track everything that potentially updates data. you can grep out whatever you need from that log easily.
http://dev.mysql.com/doc/refman/5.0/en/mysqlbinlog.html
http://dev.mysql.com/doc/refman/5.0/en/binary-log.html

SQL Server? If so...
Actually, I'd do this at the SQL end. You could set up a trace, and collect every query that comes through a connection with a particular Application Name. If you save it to a table, you can easily query that table later.

Slightly updated version of #luca's answer for at least Rails 4 (and probably Rails 5)
Place this in config/initializers/sql_logger.rb:
connection = ActiveRecord::Base.connection
class << connection
alias :original_exec :execute
def execute(sql, *name)
# try to log sql command but ignore any errors that occur in this block
# we log before executing, in case the execution raises an error
begin
File.open(Rails.root.join("log/sql.log"), 'a') do |file|
file.puts Time.now.to_s + ": " + sql
end
rescue Exception => e
"Error logging SQL: #{e}"
end
# execute original statement
original_exec(sql, *name)
end
end

Related

Initialize / activate SQL prior to GET DIAGNOSTICS

I have two service programs: mySrvPgm and myErr
mySrvPgm has a procedure which contains:
/free
...
exec sql INSERT INTO TABLE VALUES(:RECORD_FMT);
if sqlError() = *ON;
//handle error
endif;
...
/end-free
myErr has a procedure sqlError:
/free
exec sql GET DIAGNOSTICS CONDITION 1
:state = RETURNED_SQLSTATE;
...
/end-free
Background info: I am using XMLSERVICE to call the given procedure in mySrvPgm from PHP. I am not using a persistent connection. myErr is bound-by-reference via a binding directory used by mySrvPgm. Its activation is set to *IMMED, its activation group is set to *CALLER.
The problem: Assume there is an error on the INSERT statement in mySvrPgm. The first time sqlError() is called it will return SQLSTATE 00000 despite the error. All subsequent calls to sqlError() return the expected SQLSTATE.
A workaround: I added a procedure to myErr called initSQL:
/free
exec sql SET SCHEMA MYLIB;
/end-free
If I call initSQL() before the INSERT statement in mySrvPgm, sqlError() functions correctly. It doesn't have to be SET SCHEMA, it can be another GET DIAGNOSTICS statement. However, if it does not contain an executable SQL statement it does not help.
The question: I believe the myErr service program is activating properly and has the correct scope, but I am wondering if there is something more I need to do to activate the SQL part of it. Is there some way to set it up so SQL auto-initializes when the service program is activated, or do I have to execute an unneeded SQL statement in order to get it started?
There is some more background information available here.
Thank you for reading.
What version an release of the OS? Are you upto date on PTFs?
Honestly, seems to me that it's possibly a bug. Or the manual(s) need clarification.. I'd open a PMR.

SQL Server Multiple DDLs Ignoring Order And in a Single Transaction

I'm trying to run multiple DDLs (around 90) on an SQL Server.
The DDLs don't contain any changes to tables, only view, stored procedures, and functions. The DDLs might have inter-dependencies between them, one STP that calls another, for example.
I don't want to start organizing the files in the correct order, because it would take too long, and I want the entire operation to fail if any one of the scripts has an error.
How can I achieve this?
My idea so far, is to start a transaction, tell the SQL to ignore errors (which I don't know how to do) run all the scripts once, tell the SQL to start throwing errors again, run all the scripts again, and then commit if everything succeeds.
Is this a good idea?
How do I CREATE \ ALTER a stored procedure or view even though it has errors?
To clarify and address some concerns...
This is not intended for production. I just don't want to leave the DB I'm testing on broken.
What I would like to achieve is this: run a big group of scripts on the server, without taking the time to order them. But if any of the scripts has an error in it, I want to rollback the entire operation.
I don't care about isolation, I only want the operation to happen as a single transaction.
Organize the files in the correct order, test the procedure on a test environment, have a validation and acceptance test, then run it in production.
While running DDL in a transaction may seem possible, in practice is not. There are many DDL statements that don't mix well with transactions. You must put the application offline, take a database backup (or create a snapshot) before the schema changes, run the tested and verified upgrade procedure (your scripts), validate the result with acceptance tests and then turn the application back online. If something fails, revert to the backup created initially (with all the implications vis-a-vis any downstream log consumer like replication, log shipping or mirroring).
This is the correct way, and as far as I'm concerned the only way. I know you'll find plenty of advice on how to do this the wrong way.
We actually do something like this to deploy our database scripts to production. We do this in an application that connects to our databases. To add to the complication, we also have 600 databases that should have the same schema, but don't really. Here's our approach:
Merge all our scripts into one big file. Injecting go's in between every single file. This makes it look like there's one very long script. We do a simple ordering based on what the coders requested.
Split everything into "go blocks". Since go isn't legal sql, we split them up into multiple blocks that get executed one at a time.
Open a database connection.
Start a transaction.
for each go block:
Make sure the transaction is still active. (This is VERY important. I'll explain why in a bit.)
Run the code, recording the errors.
If there were any errors, rollback. Otherwise, commit.
In our multi database set up, we do this whole thing twice. Run through every database once, "testing" the code to make sure there are no errors on any database, and then go back and run them again "for real".
Now on to why you need to make sure the transaction is still active. There are some commands that will rollback your transaction on error! Imagine our surprise the first time we found this out... Everything before the error was rolled back, but everything after was committed. If there is an error, however, nothing in that same block gets committed, so it's all good.
Below is our core of our execution code. We use a wrapper around SqlClient, but it should look very similar to SqlClient.
Dim T = New DBTransaction(client)
For Each block In scriptBlocks
If Not T.RestartIfNecessary Then
exceptionCount += 1
Log("Could not (re)start the transaction for {0}. Not executing the rest of the script.", scriptName)
Exit For
End If
Debug.Assert(T.IsInTransaction)
Try
client.Text = block
client.ExecNonQuery()
Catch ex As Exception
exceptionCount += 1
Log(ex.Message + " on {0} executing: '{1}'", client.Connection.Database, block.Replace(vbNewLine, ""))
End Try
Next
If exceptionCount > 0 Then Log("There were {0} exceptions while executing {1}.", exceptionCount, scriptName)
If testing OrElse
exceptionCount > 0 Then
Try
T.Rollback()
Log("Rolled back all changes for {0} on {1}.", scriptName, client.Connection.Database)
Catch ex As Exception
Log("Could not roll back {0} on {1}: {2}", scriptName, client.Connection.Database, ex.Message)
If Debugger.IsAttached Then
Debugger.Break()
End If
End Try
Else
T.Commit()
Log("Successfully committed all changes for {0} on {1}.", scriptName, client.Connection.Database)
End If
Return exceptionCount
Class DBTransaction
Private _tName As String
Public ReadOnly Property name() As String
Get
Return _tName
End Get
End Property
Private _client As OB.Core2.DB.Client
Public Sub New(client As OB.Core2.DB.Client, Optional name As String = Nothing)
If name Is Nothing Then
name = "T" & Guid.NewGuid.ToString.Replace("-", "").Substring(0, 30)
End If
_tName = name
_client = client
End Sub
Public Function Begin() As Boolean
Return RestartIfNecessary()
End Function
Public Function RestartIfNecessary() As Boolean
Try
_client.Text = "IF NOT EXISTS (Select transaction_id From sys.dm_tran_active_transactions where name = '" & name & "') BEGIN BEGIN TRANSACTION " & name & " END"
_client.ExecNonQuery()
Return IsInTransaction()
Catch ex As Exception
Return False
End Try
End Function
Public Function IsInTransaction() As Boolean
_client.Text = "Select transaction_id From sys.dm_tran_active_transactions where name = '" & name & "'"
Dim scalar As String = _client.ExecScalar
Return scalar <> ""
End Function
Public Sub Rollback()
_client.Text = "ROLLBACK TRANSACTION " & name
_client.ExecNonQuery()
End Sub
Public Sub Commit()
_client.Text = "COMMIT TRANSACTION " & name
_client.ExecNonQuery()
End Sub
End Class
You have a good answer, here is "hack" answer. For the case "You cannot do this, but if you want it very much, then go on". I'm quite confident that you will not achieve what you are thinking of, therefore
DO FULL BACKUP!
Assuming there are no COMMIT or GO statements (explicit or !implicit!) in any of these files, the only thing you need to do is to run them in a single transaction. Combine them in one file, wrap in a transaction, and run.
How to combine 90 files in 1 file:
If sorting by name brings them in right order, then run this from folder with files in command prompt:
FOR /F "tokens=1" %G IN ('dir /b /-d /o:n *.sql') DO (
type %G >> Big_SQL_Script.sql && echo. >> Big_SQL_Script.sql
)
If order is random, then create a list of files dir /b /-d *.sql > File_Name_List.txt and order it manually. Then run:
FOR /F "tokens=1" %G IN (File_Name_List.txt) DO (
type %G >> Big_SQL_Script.sql && echo. >> Big_SQL_Script.sql
)
This way you can concatenate 90 files in automated order. Run and see what happens.
Good luck!

RSpec, Spork & Postgres Error: prepared statement “a1” already exists

In our PostgreSQL-backed Rails project, when running rspec using spork, sometimes we receive the following error:
ActiveRecord::StatementInvalid:
PG::Error: ERROR: prepared statement "a1" already exists
Originally, it occurred only a few times a day, but recently, it has begun happening every 3-4 test runs which slows our development efforts to a crawl.
Is there a way to reset/delete the prepared statements in PostgreSQL somewhere inside of our spec_helper.rb file?
After searching the PostgreSQL docs and a fair amount of trial and error testing it in different spots inside the spec_helper.rb, I finally figured out that I could add the following to delete all existing prepared statements and I have not since seen the error:
RSpec.configure do |config|
# ... other code
config.after(:suite) do
ActiveRecord::Base.connection.execute("DEALLOCATE ALL")
end
# ... other code
end
You can get this if you're doing something like:
class ActiveRecord::Base
mattr_accessor :shared_connection
##shared_connection = nil
def self.connection
##shared_connection || retrieve_connection
end
end
ActiveRecord::Base.shared_connection = ActiveRecord::Base.connection
(which was recommended here).
If you have that or similar, try removing it.

JRuby and HSQLDB: randomly missing rows

I have the following codes:
# my db connection is here:
# #connection = DriverManager.getConnection("jdbc:hsqldb:file:../db/rdata", "user", "user")
#the method recieves a array of hash {:rev,:time,:path,:result}
def create(list)
st=#connection.createStatement()
sql=""
list.each do |i|
sql.concat("INSERT INTO RESULTS_LOGIN (REVISION, TIMESTAMP, ARCHIVEPATH, RESULTS) VALUES (#{i[:rev]},'#{i[:time]}','#{i[:path]}','#{i[:result]}');\n")
end
p sql.lines.to_a.size
st.execute(sql)
st.close
end
Problem:
The sql.lines.to_a.size is 128, which means I have 128 rows to be inserted. However, after rerunning for many times, I found that everytime I got different rows in the db. For example, this time the RESULT_LOGIN table has 114 rows, and next time it was 99.... There are also successful runs, which made 128 rows in the table.
I can't understand why such thing happens. No errors, no exceptions, and what's strange is that in my code there is a method right after the "create" method, whcih executes "select * from RESULTS_LOGIN" and print it, like this:
create(list) # insert rows
read_all # print all rows
Everytime the "read_all" right after "create" prints out 128 rows and the correct data; However, if I run read_all seperately after closing and recreate the connection, or browse the db using HSQLDB's GUI management tool, there will be some rows missing....
For tests such as this, you need to add an extra setting either to URL or as an executed SQL statement. This is to ensure all unwritten data is persisted at the end of the test. See my answer to this question:
something funny with embedded hsql

Log every SQL query to database in Rails 3

This question is a follow up to this question, where should I place this code?
connection = ActiveRecord::Base.connection
class << connection
alias :original_exec :execute
def execute(sql, *name)
# try to log sql command but ignore any errors that occur in this block
# we log before executing, in case the execution raises an error
begin
file = File.open(RAILS_ROOT + "/log/sql.txt",'a'){|f| f.puts Time.now.to_s+": "+sql}
rescue Exception => e
;
end
# execute original statement
original_exec(sql, *name)
end
end
I have tried placing it inside of the model, but what happens is when I execute some sql query more then once it returns "stack level is to deep" error.
Put it in config/initializers. Most likely it's because of reloading classes each time in dev env. This code need to be executed only once though.