searching for and ranking results - sql

I'm trying to write a relatively simple algorithm to search for a string on several attributes
Given some data:
Some data:
1: name: 'Josh', location: 'los angeles'
2: name: 'Josh', location: 'york'
search string: "josh york"
The results should be [2, 1] because that query string hits the 2nd record twice, and the 1st record once.
It's safe to assume case-insensitivity here.
So here's what I have so far, in ruby/active record:
query_string = "josh new york"
some_attributes = [:name, :location]
results = {}
query_string.downcase.split.each do |query_part|
some_attributes.each do |attribute|
find(:all, :conditions => ["#{attribute} like ?", "%#{query_part}%"]).each do |result|
if results[result]
results[result] += 1
else
results[result] = 1
end
end
end
end
results.sort{|a,b| b[1]<=>a[1]}
The issue I have with this method is that it produces a large number of queries (query_string.split.length * some_attributes.length).
Can I make this more efficient somehow by reducing the number of queries ?
I'm okay with sorting within ruby, although if that can somehow be jammed into the SQL that'd be nice too.

Why aren't you using something like Ferret? Ferret is a Ruby + C extension to make a full text index. Since you seem to be using ActiveRecord, there's also acts_as_ferret.

Related

Writing SQL in Rails 4

I have a payment_request model and a payment_detail model. In the payment_request index I need to be able to search by first and last name which are stored in the payment_details table. I am newish to writing SQL and could use some help. I have what I believe to be the correct query below, but am not sure how to write that in my Rails controller so I can search by name.
SELECT first_name, last_name
FROM payment_details
LEFT OUTER JOIN payment_requests
ON payment_requests.id = payment_details.payment_request_id;
If you're using ActiveRecord models, you can skip all that and build that query with the ActiveRecord Querying Interface.
#payment_requests = PaymentRequest.joins(:payment_detail).where(payment_detail: {first_name: params[:first_name], last_name: params[:last_name]})
If you intent to show payment_details data on that index page, you should consider including that information in that query, so you avoid n+1 queries.
#payment_requests = PaymentRequest.includes(:payment_detail).where(payment_detail: { first_name: params[:first_name], last_name: params[:last_name]})
Note: You've got to have a complete match to use the above, so it may not be what you want.
I'd also recommend you use the Ransack gem to build complex queries. It would go something like this:
PaymentRequest.ransack(params[:q])
and in your views:
<%= f.search_field :payment_detail_first_name_or_payment_detail_last_name_cont %>
That would allow you to use just one field to query both columns.
You can do the following:
term_to_find = params[:search]
columns_to_search = %w( payment_details.first_name payment_details.last_name )
sql_conditions = []
columns_to_search.map |column_name|
sql_conditions.push("#{column_name} ILIKE :term_to_find")
end
PaymentRequest.includes(:payment_details)
.where(sql_conditions.join(' OR '), term_to_find: "%#{term_to_find}%")
This will find results containing the string you searched. Example: you typed "bob" in the search, it could find "bobby" or even "Mr. Bob" (the ILIKE makes the search case-insensitive)

SQL injections in Rails 4 issue

I'm trying to learn about SQL injections and have tried to implement these, but when I put this code in my controller:
params[:username] = "johndoe') OR admin = 't' --"
#user_query = User.find(:first, :conditions => "username = '#{params[:username]}'")
I get the following error:
Couldn't find all Users with 'id': (first, {:conditions=>"username = 'johndoe') OR admin = 't' --'"}) (found 0 results, but was looking for 2)
I have created a User Model with the username "johndoe", but I am still getting no proper response. BTW I am using Rails 4.
You're using an ancient Rails syntax. Don't use
find(:first, :condition => <condition>) ...
Instead use
User.where(<condtion>).first
find accepts a list of IDs to lookup records for. You're giving it an ID of :first and an ID of condition: ..., which aren't going to match any records.
User.where(attr1: value, attr2: value2)
or for single items
User.find_by(attr1: value, attr2: value)
Bear in mind that while doing all this, it would be valuable to check what the actual sql statement is by adding "to_sql" to the end of the query method (From what I remember, find_by just does a LIMIT by 1)

how do i get a rails WHERE query to be more specific? "Like" works, = doesn't

I am very new to Rails and am trying to create a search field to query my database. When I use the WHERE method and use "like", it works, but it's far too broad. If a person's last name is Smith, and I search for "i", it will return Smith as that string contains an "i". How can I make it more specific? I've searched around, read the docs and thought the following should work, but it doesn't return records.
this works
def self.search(query)
where("last_name like ? OR email like ?", "%#{query}%", "%#{query}%")
end
this does not
def self.search(query)
where("last_name = ? OR email = ?", "%#{query}%", "%#{query}%")
end
You're going to want to look more into SQL, as this is very specifically an SQL question. The % operator in SQL returns a fuzzy match. You want a strict match. So
def self.search(query)
where("last_name = ? OR email = ?", query, query)
end
or more succinctly:
scope :search, ->(query) { where('last_name = :query OR email = :query', query: query) }

Rails Active Record Search - Name includes a word

Im trying to pull all records from a Project model that includes in the project_name the word 'Fox'. I can do an active record search and return specific project_names, like 'Brown Fox':
#projects = Project.where("project_name like ?", "Brown Fox")
But if I want to return all the names that INCLUDE 'Fox', this does not work unless the complete project name is 'Fox':
#projects = Project.where("project_name like ?", "Fox")
How do I do a search that returns all the objects with the word 'Fox' in the name?
Try using:
variable = "Fox"
Project.where("project_name like ?", "%#{variable}%")
You can use the SQL % operator:
#projects = Project.where("project_name like ?", "%Fox%")
Note that if you want your query to return results ignoring the word case, you can use PostgreSQL ilike instead of like.
Did you try ransack ?
With ransack you can do something like
#projects = Project.search(:project_name_cont => "Fox")
If you think it is too much for what you need. you can use the % operator as MurifoX said
Here's a version that will allow you to handle any number of input words and to search for all of them within a name. I was looking for this answer and didn't find the more complicated case, so here it is:
def self.search(pattern)
if pattern.blank? # blank? covers both nil and empty string
all
else
search_functions = []
search_terms = pattern.split(' ').map{|word| "%#{word.downcase}%"}
search_terms.length.times do |i|
search_functions << 'LOWER(project_name) LIKE ?'
end
like_patterns = search_functions.join(' and ')
where("#{like_patterns}", *search_terms)
end
end

Rails raw SQL example

How can I convert this code to raw sql and use in rails? Because When I deploy this code in heroku,there is a request timeout error.I think this will be faster if I use raw sql.
#payments = PaymentDetail.joins(:project).order('payment_details.created_at desc')
#payment_errors = PaymentError.joins(:project).order('payment_errors.created_at desc')
#all_payments = (#payments + #payment_errors)
You can do this:
sql = "Select * from ... your sql query here"
records_array = ActiveRecord::Base.connection.execute(sql)
records_array would then be the result of your sql query in an array which you can iterate through.
I know this is old... But I was having the same problem today and found a solution:
Model.find_by_sql
If you want to instantiate the results:
Client.find_by_sql("
SELECT * FROM clients
INNER JOIN orders ON clients.id = orders.client_id
ORDER BY clients.created_at desc
")
# => [<Client id: 1, first_name: "Lucas" >, <Client id: 2, first_name: "Jan">...]
Model.connection.select_all('sql').to_hash
If you just want a hash of values:
Client.connection.select_all("SELECT first_name, created_at FROM clients
WHERE id = '1'").to_hash
# => [
{"first_name"=>"Rafael", "created_at"=>"2012-11-10 23:23:45.281189"},
{"first_name"=>"Eileen", "created_at"=>"2013-12-09 11:22:35.221282"}
]
Result object:
select_all returns a result object. You can do magic things with it.
result = Post.connection.select_all('SELECT id, title, body FROM posts')
# Get the column names of the result:
result.columns
# => ["id", "title", "body"]
# Get the record values of the result:
result.rows
# => [[1, "title_1", "body_1"],
[2, "title_2", "body_2"],
...
]
# Get an array of hashes representing the result (column => value):
result.to_hash
# => [{"id" => 1, "title" => "title_1", "body" => "body_1"},
{"id" => 2, "title" => "title_2", "body" => "body_2"},
...
]
# ActiveRecord::Result also includes Enumerable.
result.each do |row|
puts row['title'] + " " + row['body']
end
Sources:
ActiveRecord - Findinig by
SQL.
Ruby on Rails - Active Record Result
.
You can execute raw query using ActiveRecord. And I will suggest to go with SQL block
query = <<-SQL
SELECT *
FROM payment_details
INNER JOIN projects
ON projects.id = payment_details.project_id
ORDER BY payment_details.created_at DESC
SQL
result = ActiveRecord::Base.connection.execute(query)
You can do direct SQL to have a single query for both tables. I'll provide a sanitized query example to hopefully keep people from putting variables directly into the string itself (SQL injection danger), even though this example didn't specify the need for it:
#results = []
ActiveRecord::Base.connection.select_all(
ActiveRecord::Base.send(:sanitize_sql_array,
["... your SQL query goes here and ?, ?, ? are replaced...;", a, b, c])
).each do |record|
# instead of an array of hashes, you could put in a custom object with attributes
#results << {col_a_name: record["col_a_name"], col_b_name: record["col_b_name"], ...}
end
Edit: as Huy said, a simple way is ActiveRecord::Base.connection.execute("..."). Another way is ActiveRecord::Base.connection.exec_query('...').rows. And you can use native prepared statements, e.g. if using postgres, prepared statement can be done with raw_connection, prepare, and exec_prepared as described in https://stackoverflow.com/a/13806512/178651
You can also put raw SQL fragments into ActiveRecord relational queries: http://guides.rubyonrails.org/active_record_querying.html
and in associations, scopes, etc. You could probably construct the same SQL with ActiveRecord relational queries and can do cool things with ARel as Ernie mentions in http://erniemiller.org/2010/03/28/advanced-activerecord-3-queries-with-arel/. And, of course there are other ORMs, gems, etc.
If this is going to be used a lot and adding indices won't cause other performance/resource issues, consider adding an index in the DB for payment_details.created_at and for payment_errors.created_at.
If lots of records and not all records need to show up at once, consider using pagination:
https://www.ruby-toolbox.com/categories/pagination
https://github.com/mislav/will_paginate
If you need to paginate, consider creating a view in the DB first called payment_records which combines the payment_details and payment_errors tables, then have a model for the view (which will be read-only). Some DBs support materialized views, which might be a good idea for performance.
Also consider hardware or VM specs on Rails server and DB server, config, disk space, network speed/latency/etc., proximity, etc. And consider putting DB on different server/VM than the Rails app if you haven't, etc.
I want to work with exec_query of the ActiveRecord class, because it returns the mapping of the query transforming into object, so it gets very practical and productive to iterate with the objects when the subject is Raw SQL.
Example:
values = ActiveRecord::Base.connection.exec_query("select * from clients")
p values
and return this complete query:
[{"id": 1, "name": "user 1"}, {"id": 2, "name": "user 2"}, {"id": 3, "name": "user 3"}]
To get only list of values
p values.rows
[[1, "user 1"], [2, "user 2"], [3, "user 3"]]
To get only fields columns
p values.columns
["id", "name"]
You can also mix raw SQL with ActiveRecord conditions, for example if you want to call a function in a condition:
my_instances = MyModel.where.not(attribute_a: nil) \
.where('crc32(attribute_b) = ?', slot) \
.select(:id)