Heroku rails queuing mongoid - ruby-on-rails-3

Question

I need to proccess a csv file with 10,000 records in it. It takes about 4 seconds per 100 records, and I'm cool with it.
I want to be able to split the 10,000 record to 100 jobs, that will be executed when server is not busy(no order needed).
I'm working with heroku, and if I can, I'll be happy to distribute the handling to a few nodes.
What's best practice here? How to handle the mongo connection? How to split the job, and create this tasks that will run in future?
Don't need a full solution just some guidance please.

Answer 1

I'd have the same suggestion as Sergio. Try the Sidekiq background worker gem. There's the kiqstand middleware that makes it work with Mongoid.
Rough possible sketch:
# Gemfile
gem 'kiqstand'
gem 'sidekiq'
# Procfile
web: ...
worker: bundle exec sidekiq -e $RACK_ENV -C config/sidekiq.yml
# config/sidekiq.yml
#
# Configuration file for Sidekiq.
# Options here can still be overridden by cmd line args.
---
:verbose: false
:namespace: sidekiq
:concurrency: 25
:queues:
- [often, 7]
- [default, 5]
- [seldom, 3]
# config/initializers/sidekiq.rb
Sidekiq.configure_server do |config|
# Require kiqstand middleware
config.server_middleware do |chain|
chain.add Kiqstand::Middleware
end
end
# app/workers/import_worker.rb
# The actual Sidekiq worker that performs the input
class ImportWorker
include Sidekiq::Worker
def perform(my_var)
# processing the csv
end
end
Sidekiq and Kiqstand should handle the Mongoid Mongodb connections. For splitting the tasks you could create a second worker that feeds the first one. As the arguments sent to ImportWorker.perform_async(my_var) will be serialized and stored in Redis they should be small, just a row reference or so in your case.
Hope that gives a few pointers.

Answer 2

also have a look at the smarter_csv gem - which is designed to read CSV files in batches, so you can hand them off to a resque or sidekiq worker..
https://github.com/tilo/smarter_csv