writing a caching version of Mechanize - ruby-on-rails-3

Question

I'd like a caching version of Mechanize. The idea is that #get(uri...) checks to see if that uri has been previously fetched, and if so, fetch the response from the cache rather than hitting the web. If not in the cache, it hits the web and saves the response in the cache.
My naive approach doesn't work. (I probably don't need to mention that CachedWebPage is a subclass of ActiveRecord::Base):
class CachingMechanize < Mechanize
def get(uri, parameters = [], referer = nil, headers = {})
page = if (record = CachedWebPage.find_by_uri(uri.to_s))
record.contents
else
super.tap {|contents| CachedWebPage.create!(:uri => uri, :contents => contents)}
end
yield page if block_given?
page
end
end
This fails because the object returned by Mechanize#get() is a complex, circular structure that neither YAML nor JSON want to serialize for storage into the database.
I realize that what I want is to capture the low-level contents before Mechanize parses it.
Is there clean way to do this? I think I can use Mechanize's post_connect hook to access the raw page coming in, but I don't see how to subsequently pass the cached raw page to Mechanize for parsing.
Is there some package I should be using that does web page caching already?

Answer 1

It turns out the solution was simple, albeit not entirely clean. It's a simple matter to cache the results of Mechanize#get() like this:
class CachingMechanize < Mechanize
def get(uri, parameters = [], referer = nil, headers = {})
WebCache.with_web_cache(uri.to_s) { super }
end
end
... where with_web_cache() uses YAML to serialize and cache the object returned by super.
My problem was that by default, Mechanize#get() returns a Mechanize::Page object containing some lambda object, which cannot be dumped and loaded by YAML. The fix was to eliminate those lambdas, which turned out to be rather simple. Full code follows.
class CachingMechanize < Mechanize
def initialize(*args)
super
sanitize_scheme_handlers
end
def get(uri, parameters = [], referer = nil, headers = {})
WebCache.with_web_cache(uri.to_s) { super }
end
# private
def sanitize_scheme_handlers
scheme_handlers['http'] = SchemeHandler.new
scheme_handlers['https'] = scheme_handlers['http']
scheme_handlers['relative'] = scheme_handlers['http']
scheme_handlers['file'] = scheme_handlers['http']
end
class SchemeHandler
def call(link, page) ; link ; end
end
end
the moral: don't try to YAML.dump and YAML.load objects containing lambda or proc
This goes beyond just this example: if you see a YAML error that reads:
TypeError: allocator undefined for Proc
Check to see if there's a lambda or proc in the object you're trying to serialize and deserialize. If you are able (as I was in this case) to replace the lambda with a method call to an object, you should be able to work around the problem.
Hope this helps someone else.
Update
In response to #Martin's request for the definition of WebCache, here 'tis:
# Simple model for caching pages fetched from the web. Assumes
# a schema like this:
#
# create_table "web_caches", :force => true do |t|
# t.text "key"
# t.text "value"
# t.datetime "expires_at"
# t.datetime "created_at", :null => false
# t.datetime "updated_at", :null => false
# end
# add_index "web_caches", ["key"], :name => "index_web_caches_on_key", :unique => true
#
class WebCache < ActiveRecord::Base
serialize :value
# WebCache.with_web_cache(key) {
# ...body...
# }
#
# Searches the web_caches table for an entry with a matching key. If
# found, and if the entry has not expired, the value for that entry is
# returned. If not found, or if the entry has expired, yield to the
# body and cache the yielded value before returning it.
#
# Options:
# :expires_at sets the expiration date for this entry upon creation.
# Defaults to one year from now.
# :expired_prior_to overrides the value of 'now' when checking for
# expired entries. Mostly useful for unit testing.
#
def self.with_web_cache(key, opts = {})
serialized_key = YAML.dump(key)
expires_at = opts[:expires_at] || 1.year.from_now
expired_prior_to = opts[:expired_prior_to] || Time.zone.now
if (r = self.where(:key => serialized_key).where("expires_at > ?", expired_prior_to)).exists?
# cache hit
r.first.value
else
# cache miss
yield.tap {|value| self.create!(:key => serialized_key, :value => value, :expires_at => expires_at)}
end
end
# Prune expired entries. Typically called by a cron job.
def self.delete_expired_entries(expired_prior_to = Time.zone.now)
self.where("expires_at < ?", expired_prior_to).destroy_all
end
end