Rails raw SQL example - sql

How can I convert this code to raw sql and use in rails? Because When I deploy this code in heroku,there is a request timeout error.I think this will be faster if I use raw sql.
#payments = PaymentDetail.joins(:project).order('payment_details.created_at desc')
#payment_errors = PaymentError.joins(:project).order('payment_errors.created_at desc')
#all_payments = (#payments + #payment_errors)

You can do this:
sql = "Select * from ... your sql query here"
records_array = ActiveRecord::Base.connection.execute(sql)
records_array would then be the result of your sql query in an array which you can iterate through.

I know this is old... But I was having the same problem today and found a solution:
Model.find_by_sql
If you want to instantiate the results:
Client.find_by_sql("
SELECT * FROM clients
INNER JOIN orders ON clients.id = orders.client_id
ORDER BY clients.created_at desc
")
# => [<Client id: 1, first_name: "Lucas" >, <Client id: 2, first_name: "Jan">...]
Model.connection.select_all('sql').to_hash
If you just want a hash of values:
Client.connection.select_all("SELECT first_name, created_at FROM clients
WHERE id = '1'").to_hash
# => [
{"first_name"=>"Rafael", "created_at"=>"2012-11-10 23:23:45.281189"},
{"first_name"=>"Eileen", "created_at"=>"2013-12-09 11:22:35.221282"}
]
Result object:
select_all returns a result object. You can do magic things with it.
result = Post.connection.select_all('SELECT id, title, body FROM posts')
# Get the column names of the result:
result.columns
# => ["id", "title", "body"]
# Get the record values of the result:
result.rows
# => [[1, "title_1", "body_1"],
[2, "title_2", "body_2"],
...
]
# Get an array of hashes representing the result (column => value):
result.to_hash
# => [{"id" => 1, "title" => "title_1", "body" => "body_1"},
{"id" => 2, "title" => "title_2", "body" => "body_2"},
...
]
# ActiveRecord::Result also includes Enumerable.
result.each do |row|
puts row['title'] + " " + row['body']
end
Sources:
ActiveRecord - Findinig by
SQL.
Ruby on Rails - Active Record Result
.

You can execute raw query using ActiveRecord. And I will suggest to go with SQL block
query = <<-SQL
SELECT *
FROM payment_details
INNER JOIN projects
ON projects.id = payment_details.project_id
ORDER BY payment_details.created_at DESC
SQL
result = ActiveRecord::Base.connection.execute(query)

You can do direct SQL to have a single query for both tables. I'll provide a sanitized query example to hopefully keep people from putting variables directly into the string itself (SQL injection danger), even though this example didn't specify the need for it:
#results = []
ActiveRecord::Base.connection.select_all(
ActiveRecord::Base.send(:sanitize_sql_array,
["... your SQL query goes here and ?, ?, ? are replaced...;", a, b, c])
).each do |record|
# instead of an array of hashes, you could put in a custom object with attributes
#results << {col_a_name: record["col_a_name"], col_b_name: record["col_b_name"], ...}
end
Edit: as Huy said, a simple way is ActiveRecord::Base.connection.execute("..."). Another way is ActiveRecord::Base.connection.exec_query('...').rows. And you can use native prepared statements, e.g. if using postgres, prepared statement can be done with raw_connection, prepare, and exec_prepared as described in https://stackoverflow.com/a/13806512/178651
You can also put raw SQL fragments into ActiveRecord relational queries: http://guides.rubyonrails.org/active_record_querying.html
and in associations, scopes, etc. You could probably construct the same SQL with ActiveRecord relational queries and can do cool things with ARel as Ernie mentions in http://erniemiller.org/2010/03/28/advanced-activerecord-3-queries-with-arel/. And, of course there are other ORMs, gems, etc.
If this is going to be used a lot and adding indices won't cause other performance/resource issues, consider adding an index in the DB for payment_details.created_at and for payment_errors.created_at.
If lots of records and not all records need to show up at once, consider using pagination:
https://www.ruby-toolbox.com/categories/pagination
https://github.com/mislav/will_paginate
If you need to paginate, consider creating a view in the DB first called payment_records which combines the payment_details and payment_errors tables, then have a model for the view (which will be read-only). Some DBs support materialized views, which might be a good idea for performance.
Also consider hardware or VM specs on Rails server and DB server, config, disk space, network speed/latency/etc., proximity, etc. And consider putting DB on different server/VM than the Rails app if you haven't, etc.

I want to work with exec_query of the ActiveRecord class, because it returns the mapping of the query transforming into object, so it gets very practical and productive to iterate with the objects when the subject is Raw SQL.
Example:
values = ActiveRecord::Base.connection.exec_query("select * from clients")
p values
and return this complete query:
[{"id": 1, "name": "user 1"}, {"id": 2, "name": "user 2"}, {"id": 3, "name": "user 3"}]
To get only list of values
p values.rows
[[1, "user 1"], [2, "user 2"], [3, "user 3"]]
To get only fields columns
p values.columns
["id", "name"]

You can also mix raw SQL with ActiveRecord conditions, for example if you want to call a function in a condition:
my_instances = MyModel.where.not(attribute_a: nil) \
.where('crc32(attribute_b) = ?', slot) \
.select(:id)

Related

Writing SQL in Rails 4

I have a payment_request model and a payment_detail model. In the payment_request index I need to be able to search by first and last name which are stored in the payment_details table. I am newish to writing SQL and could use some help. I have what I believe to be the correct query below, but am not sure how to write that in my Rails controller so I can search by name.
SELECT first_name, last_name
FROM payment_details
LEFT OUTER JOIN payment_requests
ON payment_requests.id = payment_details.payment_request_id;
If you're using ActiveRecord models, you can skip all that and build that query with the ActiveRecord Querying Interface.
#payment_requests = PaymentRequest.joins(:payment_detail).where(payment_detail: {first_name: params[:first_name], last_name: params[:last_name]})
If you intent to show payment_details data on that index page, you should consider including that information in that query, so you avoid n+1 queries.
#payment_requests = PaymentRequest.includes(:payment_detail).where(payment_detail: { first_name: params[:first_name], last_name: params[:last_name]})
Note: You've got to have a complete match to use the above, so it may not be what you want.
I'd also recommend you use the Ransack gem to build complex queries. It would go something like this:
PaymentRequest.ransack(params[:q])
and in your views:
<%= f.search_field :payment_detail_first_name_or_payment_detail_last_name_cont %>
That would allow you to use just one field to query both columns.
You can do the following:
term_to_find = params[:search]
columns_to_search = %w( payment_details.first_name payment_details.last_name )
sql_conditions = []
columns_to_search.map |column_name|
sql_conditions.push("#{column_name} ILIKE :term_to_find")
end
PaymentRequest.includes(:payment_details)
.where(sql_conditions.join(' OR '), term_to_find: "%#{term_to_find}%")
This will find results containing the string you searched. Example: you typed "bob" in the search, it could find "bobby" or even "Mr. Bob" (the ILIKE makes the search case-insensitive)

SQL injections in Rails 4 issue

I'm trying to learn about SQL injections and have tried to implement these, but when I put this code in my controller:
params[:username] = "johndoe') OR admin = 't' --"
#user_query = User.find(:first, :conditions => "username = '#{params[:username]}'")
I get the following error:
Couldn't find all Users with 'id': (first, {:conditions=>"username = 'johndoe') OR admin = 't' --'"}) (found 0 results, but was looking for 2)
I have created a User Model with the username "johndoe", but I am still getting no proper response. BTW I am using Rails 4.
You're using an ancient Rails syntax. Don't use
find(:first, :condition => <condition>) ...
Instead use
User.where(<condtion>).first
find accepts a list of IDs to lookup records for. You're giving it an ID of :first and an ID of condition: ..., which aren't going to match any records.
User.where(attr1: value, attr2: value2)
or for single items
User.find_by(attr1: value, attr2: value)
Bear in mind that while doing all this, it would be valuable to check what the actual sql statement is by adding "to_sql" to the end of the query method (From what I remember, find_by just does a LIMIT by 1)

doctrine native sql not accepting parameter list

I'm trying to do native SQL in Doctrine. Basically I have 2 parameters:
CANDIDATE_ID - user for who we delete entries,
list of FILE_ID to keep
So I make
$this->getEntityManager()->getConnection()->
executeUpdate( "DELETE FROM FILE WHERE CANDIDATE_ID = :ID AND NOT ID IN :KEEPID",
array(
"ID" => $candidate->id,
"KEEPID" => array(2) )
);
But Doctrine fails:
Notice: Array to string conversion in D:\xampp\htdocs\azk\vendor\doctrine\dbal\lib\Doctrine\DBAL\Connection.php on line 786
Is this bug in Doctrine? I'm making somewhere else select with IN but with QueryBuilder and it's working. Maybe someone could suggest better way of deleting entries, with QueryBuilder for example?
$stmt = $conn->executeQuery('SELECT * FROM articles WHERE id IN (?)',
array(array(1, 2, 3, 4, 5, 6)),
array(\Doctrine\DBAL\Connection::PARAM_INT_ARRAY)
);
From Doctrine's documentation.
You can't pass an array of IDs to a parameter. You can do this for scalar values, but even if this had a 'toString', it wouldn't be what you want.
String concatenation is one method,
"DELETE FROM FILE WHERE CANDIDATE_ID = :ID AND NOT ID IN (". implode(",", $list_of_ids) .")"
But this method goes straight around parameters, and therefore suffers in terms of readability, and is limited to a certain maximum line length, which can vary between databases.
Another approach is to write a function returning a table result, which takes a string of IDs as a parameter.
You could also solve this with a join to a table containing the IDs to keep.
It's a problem I've seen many times with few good answers, but it's usually caused by a misunderstanding in the way the database is modelled. This is a 'code smell' for database access.

Rails3: SQL execution with hash substitution like .where()

With a simple model like that
class Model < ActiveRecord::Base
# ...
end
we can do queries like that
Model.where(["name = :name and updated_at >= :D", \
{ :D => (Date.today - 1.day).to_datetime, :name => "O'Connor" }])
Where the values in the hash will be substituted into the final SQL statement with proper escaping depending on the underlying database engine.
I would like to know a similar feature for SQL execution like:
ActiveRecord::Base.connection.execute( \
["update models set name = :name, hired_at = :D where id = :id;"], \
{ :id => 73465, :D => DateTime.now, :name => "O'My God" }] \
) # THIS CODE IS A FANTASY. NOT WORKING.
(Please do not solve the example with loading a Model object, modifying and then saving! The example is only an illustration for the feature I would like to have / know. Concentrate on the subject!)
The original problem is that I want to insert large amount (many thousand lines) of data into the database. I want to use some features of the SQL abstraction of the ActiveRecord framework but I don't want to use model objects based on ActiveRecord::Base because they are damn slow! (8 queries per second for my current problem.)
query = ActiveRecord::Base.connection.raw_connection.prepare("INSERT INTO users (name) VALUES(:name)")
query.execute(:name => 'test_name')
query.close
Extending the #peufeu solution with concrete code example for bulk insert:
users_places = []
users_values = []
timestamp = Time.now.strftime('%Y-%m-%d %H:%M:%S')
params[:users].each do |user|
users_places << "(?,?,?,?)"
users_values << user[:name] << user[:punch_line] << timestamp << timestamp
end
bulk_insert_users_sql_arr = ["INSERT INTO users (name, punch_line, created_at, updated_at) VALUES #{users_places.join(", ")}"] + users_values
begin
sql = ActiveRecord::Base.send(:sanitize_sql_array, bulk_insert_users_sql_arr)
ActiveRecord::Base.connection.execute(sql)
rescue
"something went wrong with the bulk insert sql query"
end
Here is the reference to sanitize_sql_array method in ActiveRecord::Base, it generates the proper query string by escaping the single quotes in the strings. For example the punch_line "Don't let them get you down" will become "Don\'t let them get you down".
Yes you could do raw SQL, but checkout the ar-extensions gem that helps with batch inserts:
https://github.com/zdennis/ar-extensions
Here's a post on it, and various other techniques:
http://www.coffeepowered.net/2009/01/23/mass-inserting-data-in-rails-without-killing-your-performance/
For INSERTs, batching them using a long VALUES clause (as shown by Simon's link) is the fastest way (unless you want to generate a text file and load it in your database with MySQL's LOAD DATA INFILE). But you have to be very careful about escaping your text values (which is not done in the example).
I was asking "what database are you using" because it does matter for mass UPDATEs.
For instance, you can do this on postgres (and I believe SQL Server changing "columnX" to "colX" ):
UPDATE foo
JOIN (VALUES (1,2),(3,4),... long list) v ON (foo.id=v.column1)
SET foo.bar = v.column2
And you can update a load of rows using a single statement, very fast.
If you don't need Ruby to perform some Ruby-specific magic on your data, the fastest way to transfer data from one DB to a different one is to export as a text file (CSV or tab separated), load it on the other DB (LOAD DATA INFILE on MySQL), perhaps in a temporary table, and bulk process using SQL.
EDIT : Here's how I do this in Python :
sql = [ "INSERT INTO foo (column list) VALUES " ]
values = []
for tuple in tuple_list:
append "(?,?,?,?)" to sql
extend values list with tuple
Then join sql into a string, you get "INSERT INTO foo (column list) VALUES (?,?,?,?),(?,?,?,?),(?,?,?,?)" with the "(?,?,?,?)" repeated as many times as you have lines to insert.
Then "values" contains a list of (a1,b1,c1,d1,a2,b2,c2,d2,a3,b3,c3,d3) with an,bn,cn,dn being the tuples you want to insert for line n. Each one corresponds to a placeholder in the sql string.
Then pass this to the usual "execute query with parameters" function which will handle quoting and escaping as usual.
I encountered a similar issue recently when tying to insert 100K+ records into a MySQL database for a Rails 4 app using mysql2 gem. The data included characters that had to be sanitized prior to insert.
The solution I ended going with was a slightly modified version of Option 3 described at https://www.coffeepowered.net/2009/01/23/mass-inserting-data-in-rails-without-killing-your-performance/
Here's the relevant code block from the above link:
TIMES = 10000
inserts = []
TIMES.times do
inserts.push "(3.0, '2009-01-23 20:21:13', 2, 1)"
end
sql = "INSERT INTO user_node_scores (`score`, `updated_at`, `node_id`, `user_id`) VALUES #{inserts.join(", ")}"
The modification I made was using the public method ActiveRecord::Base.sanitize() on values that required it.
inserts = []
created = Time.now.strftime "%Y-%m-%d %H:%M:%S"
params[:audits].each do |audit|
inserts.push "(#{audit.user_id), #{created}," + ActiveRecord::Base.sanitize(audit.comment) + ", #{audit.status})"
end
sql = "INSERT INTO user_audits (`user_id`, `created_at`, `comment`, `status`) VALUES #{inserts.join(", ")}"

searching for and ranking results

I'm trying to write a relatively simple algorithm to search for a string on several attributes
Given some data:
Some data:
1: name: 'Josh', location: 'los angeles'
2: name: 'Josh', location: 'york'
search string: "josh york"
The results should be [2, 1] because that query string hits the 2nd record twice, and the 1st record once.
It's safe to assume case-insensitivity here.
So here's what I have so far, in ruby/active record:
query_string = "josh new york"
some_attributes = [:name, :location]
results = {}
query_string.downcase.split.each do |query_part|
some_attributes.each do |attribute|
find(:all, :conditions => ["#{attribute} like ?", "%#{query_part}%"]).each do |result|
if results[result]
results[result] += 1
else
results[result] = 1
end
end
end
end
results.sort{|a,b| b[1]<=>a[1]}
The issue I have with this method is that it produces a large number of queries (query_string.split.length * some_attributes.length).
Can I make this more efficient somehow by reducing the number of queries ?
I'm okay with sorting within ruby, although if that can somehow be jammed into the SQL that'd be nice too.
Why aren't you using something like Ferret? Ferret is a Ruby + C extension to make a full text index. Since you seem to be using ActiveRecord, there's also acts_as_ferret.