I want to do something like "Find all Books where book.pages.count < books.max_pages".
So the models are:
class Book
has_many :pages
end
class Page
belongs_to :book
end
I know I can find books w/ a set number of pages. eg:
# Get books w/ < 5 pages.
Book.joins(:pages).group("books.id").having("count(pages.id) < ?", 5)
Is there a good way to do this with a dynamic page count? eg:
Book.joins(:pages).group("books.id").select(.having("count(pages.id) <= book.max_pages")
If not I can always just store something inside the Book model (eg book.is_full = false until a save causes it to be full), but this is a bit less flexible if max_pages gets updated.
You could create a scope like this:
def self.page_count_under(amount)
joins(:pages)
.group('books.id')
.having('COUNT(pages.id) < ?', amount)
end
UPDATE
This should work if max_pages is an attribute of the Book model.
def self.page_count_under_max
joins(:pages)
.group('books.id')
.having('COUNT(pages.id) < books.max_pages')
end
Use counter_cache!
http://guides.rubyonrails.org/association_basics.html 4.1.2.3 :counter_cache
In Rails 4, I have a vanilla has_and_belongs_to_many relationship, say between Cars and Colors.
Given a set of colors I'd like to find the cars that have exactly those colors. The closest I can get is: Car.joins(:colors).where('colors.id' => colors), but that returns all Cars with any of the colors.
I'd like to do this entirely in ActiveRecord since both tables are liable to be huge, so something like Car.joins(:colors).where('colors.id' => colors).to_a.select{|car| car.colors == colors} is less than ideal.
Any idea how to accomplish this?
I was able to get it with having and some gnarly string interpolated SQL. I've made this into a scope you can use like so:
# Car.with_exact(colors: colors)
class ActiveRecord::Base
class << self
def with_exact(associations={})
scope = self
associations.each do |association_name, set|
association = reflect_on_association(association_name)
scope = scope.joins(association_name)
.group("#{table_name}.id")
.having("COUNT(\"#{association.join_table}\".*) = ?", set.count)
.having(%{
COUNT(\"#{association.join_table}\".*) = SUM(
CASE WHEN \"#{association.join_table}\".\"#{association.association_foreign_key}\"
IN (#{set.to_a.map(&:quoted_id).join(', ')})
THEN 1 ELSE 0 END
)
}.squish)
end
scope
end
end
end
I have an app that records calls. Each call can have multiple units associated with it. Part of my app has a reports section which basically just does a query on the Call model for different criteria. I've figured out how to write some scopes that do what I want and chain them to the results of my reporting search functionality. But I can't figure out how to search by "unit". Below are relevant excerpts from my code:
Call.rb
has_many :call_units
has_many :units, through: :call_units
#Report search logic
def self.report(search)
search ||= { type: "all" }
# Determine which scope to search by
results = case search[:type]
when "open"
open_status
when "canceled"
cancel
when "closed"
closed
when "waitreturn"
waitreturn
when "wheelchair"
wheelchair
else
scoped
end
#Search results by unit name, this is what I need help with. Scope or express otherwise?
results = results. ??????
results = results.by_service_level(search[:service_level]) if search[:service_level].present?
results = results.from_facility(search[:transferred_from]) if search[:transferred_from].present?
results = results.to_facility(search[:transferred_to]) if search[:transferred_to].present?
# If searching with BOTH a start and end date
if search[:start_date].present? && search[:end_date].present?
results = results.search_between(Date.parse(search[:start_date]), Date.parse(search[:end_date]))
# If search with any other date parameters (including none)
else
results = results.search_by_start_date(Date.parse(search[:start_date])) if search[:start_date].present?
results = results.search_by_end_date(Date.parse(search[:end_date])) if search[:end_date].present?
end
results
end
Since I have an association for units already, I'm not sure if I need to make a scope for units somehow or express the results somehow in the results variable in my search logic.
Basically, you want a scope that uses a join so you can use a where criteria in against the associated model? Is that correct?
So in SQL you're looking for something like
select * from results r
inner join call_units c on c.result_id = r.id
inner join units u on u.call_unit_id = c.id
where u.name = ?
and the scope would be (from memory, I haven't debugged this) something like:
scope :by_unit_name, lambda {|unit_name|
joins(:units).where('units.unit_name = ?', unit_name)
}
units.name isn't a column in the db. Changing it to units.unit_name didn't raise an exception and seems to be what I want. Here's what I have in my results variable:
results = results.by_unit_name(search[:unit_name]) if search[:unit_name].present?
When I try to search by a different unit name no results show up. Here's the code I'm using to search:
<%= select_tag "search[unit_name]", options_from_collection_for_select(Unit.order(:unit_name), :unit_name, :unit_name, selected: params[:search].try(:[], :unit_name)), prompt: "Any Unit" %>
I'm trying to run a query of about 50,000 records using ActiveRecord's find_each method, but it seems to be ignoring my other parameters like so:
Thing.active.order("created_at DESC").limit(50000).find_each {|t| puts t.id }
Instead of stopping at 50,000 I'd like and sorting by created_at, here's the resulting query that gets executed over the entire dataset:
Thing Load (198.8ms) SELECT "things".* FROM "things" WHERE "things"."active" = 't' AND ("things"."id" > 373343) ORDER BY "things"."id" ASC LIMIT 1000
Is there a way to get similar behavior to find_each but with a total max limit and respecting my sort criteria?
The documentation says that find_each and find_in_batches don't retain sort order and limit because:
Sorting ASC on the PK is used to make the batch ordering work.
Limit is used to control the batch sizes.
You could write your own version of this function like #rorra did. But you can get into trouble when mutating the objects. If for example you sort by created_at and save the object it might come up again in one of the next batches. Similarly you might skip objects because the order of results has changed when executing the query to get the next batch. Only use that solution with read only objects.
Now my primary concern was that I didn't want to load 30000+ objects into memory at once. My concern was not the execution time of the query itself. Therefore I used a solution that executes the original query but only caches the ID's. It then divides the array of ID's into chunks and queries/creates the objects per chunk. This way you can safely mutate the objects because the sort order is kept in memory.
Here is a minimal example similar to what I did:
batch_size = 512
ids = Thing.order('created_at DESC').pluck(:id) # Replace .order(:created_at) with your own scope
ids.each_slice(batch_size) do |chunk|
Thing.find(chunk, :order => "field(id, #{chunk.join(',')})").each do |thing|
# Do things with thing
end
end
The trade-offs to this solution are:
The complete query is executed to get the ID's
An array of all the ID's is kept in memory
Uses the MySQL specific FIELD() function
Hope this helps!
find_each uses find_in_batches under the hood.
Its not possible to select the order of the records, as described in find_in_batches, is automatically set to ascending on the primary key (“id ASC”) to make the batch ordering work.
However, the criteria is applied, what you can do is:
Thing.active.find_each(batch_size: 50000) { |t| puts t.id }
Regarding the limit, it wasn't implemented yet: https://github.com/rails/rails/pull/5696
Answering to your second question, you can create the logic yourself:
total_records = 50000
batch = 1000
(0..(total_records - batch)).step(batch) do |i|
puts Thing.active.order("created_at DESC").offset(i).limit(batch).to_sql
end
Retrieving the ids first and processing the in_groups_of
ordered_photo_ids = Photo.order(likes_count: :desc).pluck(:id)
ordered_photo_ids.in_groups_of(1000, false).each do |photo_ids|
photos = Photo.order(likes_count: :desc).where(id: photo_ids)
# ...
end
It's important to also add the ORDER BY query to the inner call.
Rails 6.1 adds support for descending order in find_each, find_in_batches and in_batches.
One option is to put an implementation tailored for your particular model into the model itself (speaking of which, id is usually a better choice for ordering records, created_at may have duplicates):
class Thing < ActiveRecord::Base
def self.find_each_desc limit
batch_size = 1000
i = 1
records = self.order(created_at: :desc).limit(batch_size)
while records.any?
records.each do |task|
yield task, i
i += 1
return if i > limit
end
records = self.order(created_at: :desc).where('id < ?', records.last.id).limit(batch_size)
end
end
end
Or else you can generalize things a bit, and make it work for all the models:
lib/active_record_extensions.rb:
ActiveRecord::Batches.module_eval do
def find_each_desc limit
batch_size = 1000
i = 1
records = self.order(id: :desc).limit(batch_size)
while records.any?
records.each do |task|
yield task, i
i += 1
return if i > limit
end
records = self.order(id: :desc).where('id < ?', records.last.id).limit(batch_size)
end
end
end
ActiveRecord::Querying.module_eval do
delegate :find_each_desc, :to => :all
end
config/initializers/extensions.rb:
require "active_record_extensions"
P.S. I'm putting the code in files according to this answer.
You can iterate backwards by standard ruby iterators:
Thing.last.id.step(0,-1000) do |i|
Thing.where(id: (i-1000+1)..i).order('id DESC').each do |thing|
#...
end
end
Note: +1 is because BETWEEN which will be in query includes both bounds but we need include only one.
Sure, with this approach there could be fetched less than 1000 records in batch because some of them are deleted already but this is ok in my case.
As remarked by #Kirk in one of the comments, find_each supports limit as of version 5.1.0.
Example from the changelog:
Post.limit(10_000).find_each do |post|
# ...
end
The documentation says:
Limits are honored, and if present there is no requirement for the batch size: it can be less than, equal to, or greater than the limit.
(setting a custom order is still not supported though)
I was looking for the same behaviour and thought up of this solution. This DOES NOT order by created_at but I thought I would post anyways.
max_records_to_retrieve = 50000
last_index = Thing.count
start_index = [(last_index - max_records_to_retrieve), 0].max
Thing.active.find_each(:start => start_index) do |u|
# do stuff
end
Drawbacks of this approach:
- You need 2 queries (first one should be fast)
- This guarantees a max of 50K records but if ids are skipped you will get less.
You can try ar-as-batches Gem.
From their documentation you can do something like this
Users.where(country_id: 44).order(:joined_at).offset(200).as_batches do |user|
user.party_all_night!
end
Using Kaminari or something other it will be easy.
Create batch loader class.
module BatchLoader
extend ActiveSupport::Concern
def batch_by_page(options = {})
options = init_batch_options!(options)
next_page = 1
loop do
next_page = yield(next_page, options[:batch_size])
break next_page if next_page.nil?
end
end
private
def default_batch_options
{
batch_size: 50
}
end
def init_batch_options!(options)
options ||= {}
default_batch_options.merge!(options)
end
end
Create Repository
class ThingRepository
include BatchLoader
# #param [Integer] per_page
# #param [Proc] block
def batch_changes(per_page=100, &block)
relation = Thing.active.order("created_at DESC")
batch_by_page do |next_page|
query = relation.page(next_page).per(per_page)
yield query if block_given?
query.next_page
end
end
end
Use the repository
repo = ThingRepository.new
repo.batch_changes(5000).each do |g|
g.each do |t|
#...
end
end
Adding find_in_batches_with_order did solve my usecase, where I was having ids already but need batching and ordering. It was inspired by #dirk-geurs solution
# Create file config/initializers/find_in_batches_with_order.rb with follwing code.
ActiveRecord::Batches.class_eval do
## Only flat order structure is supported now
## example: [:forename, :surname] is supported but [:forename, {surname: :asc}] is not supported
def find_in_batches_with_order(ids: nil, order: [], batch_size: 1000)
relation = self
arrangement = order.dup
index = order.find_index(:id)
unless index
arrangement.push(:id)
index = arrangement.length - 1
end
ids ||= relation.order(*arrangement).pluck(*arrangement).map{ |tupple| tupple[index] }
ids.each_slice(batch_size) do |chunk_ids|
chunk_relation = relation.where(id: chunk_ids).order(*order)
yield(chunk_relation)
end
end
end
Leaving Gist here https://gist.github.com/the-spectator/28b1176f98cc2f66e870755bb2334545
I had the same problem with a query with DISTINCT ON where you need an ORDER BY with that field, so this is my approach with Postgres:
def filtered_model_ids
Model.joins(:father_model)
.select('DISTINCT ON (model.field) model.id')
.order(:field)
.map(&:id)
end
def processor
filtered_model_ids.each_slice(BATCH_SIZE).lazy.each do |batch|
Model.find(batch).each do |record|
# Code
end
end
end
My code
batch_size = 100
total_count = klass.count
offset = 0
processed_count = 0
while processed_count < total_count
relation = klass.order({ active_at: :asc, created_at: :desc }).offset(offset).limit(batch_size)
relation.each do |record|
record.process
end
processed_count += batch_size
end
Do it in one query and avoid iterating:
User.offset(2).order('name DESC').last(3)
will product a query like this
SELECT "users".* FROM "users" ORDER BY name ASC LIMIT $1 OFFSET $2 [["LIMIT", 3], ["OFFSET", 2]
I currently have a scope where I am attempting to find last record created in an association and select it if a particular boolean value is false
IE Foo has_many Bar's and Bar's has a boolean column named bazzed
scope :no_baz, joins(:bars).order("bars.id DESC").limit(1).where("bars.bazzed = 'f'")
The problem with this is that rails turns this query into something like this
SELECT "foos".* FROM "foos" INNER JOIN "bars" ON "bars"."foo_id" = "foos"."id" WHERE (bars.bazzed = 'f') ORDER BY bars.id DESC LIMIT 1
the problem lies that rails is calling the order and limit after the where clause, what i'm looking for is to do the order and limit first to try and find the last bar that has bazzed set to false.
Is there a native AR way to perform the query I am attempting to accomplish?
EDIT
I am trying to grab the foo's that have a bar where the last bar they have has bazzed set to false and only if the last bar that that foo has has a false bazzed.
Ok, I would suggest this for the query on the "foo" model:
Foo.bars.where("bars.bazzed = ?", 'f').all( :order => "created_at DESC").first
Note: 'f' can be replaced by false, depending on the value you use in your "bazzed" column, of course.
[Edit]
Ok, as I think I better understand the problem, here is a suggestion, but for a public method and not a scoped query.
def no_baz
all_no_baz_foos = Array.new
Foo.all.each do |foo|
last_bar = foo.bars.all.order("bars.id DESC").first
if last_bar.bazzed == 'f'
all_no_baz_foos << foo
end
end
return all_no_baz_foos
end
This method will return an Array with all the no_baz_foos record in it. As I did not test my code, you may have to change few things for it to work, but I think you get the idea.
For the "scope" method, I just can't find a way to chain correctly the queries to have the desired result. If anyone else knows how to achieve that using a scope, I'll be glad to hear the solution too.
Using a class method for now but the problem with that lies that it returns an array object and not an active record relation which is what i'm trying to return. Still attempting to get the query correctly done.