I have setup an couchdb database to be used with a ruby on rails application using couchrest.
Now the database has around 100K records and I want to add a field to store timestamp in integer format.
Trying to update it in a loop through causes request time out.
What is the best way to add a new field/property to the couchdb ?
Code
pages = Page.all
pages.each do |p|
p.update_attributes(:created_at_to_i => p.created_at.to_i)
puts p.created_at_to_i.inspect
end
Trace
>> Page.update_timestamp
RestClient::RequestTimeout: Request Timeout
from /usr/local/lib/ruby/gems/1.8/gems/rest-client-1.6.7/lib/restclient/request.rb:184:in `transmit'
from /usr/local/lib/ruby/gems/1.8/gems/rest-client-1.6.7/lib/restclient/request.rb:64:in `execute'
from /usr/local/lib/ruby/gems/1.8/gems/rest-client-1.6.7/lib/restclient/request.rb:33:in `execute'
from /usr/local/lib/ruby/gems/1.8/gems/couchrest-1.1.2/lib/couchrest/rest_api.rb:89:in `execute'
from /usr/local/lib/ruby/gems/1.8/gems/couchrest-1.1.2/lib/couchrest/rest_api.rb:45:in `get'
from /usr/local/lib/ruby/gems/1.8/gems/couchrest-1.1.2/lib/couchrest/database.rb:260:in `view'
from /usr/local/lib/ruby/gems/1.8/gems/couchrest-1.1.2/lib/couchrest/design.rb:52:in `view_on'
from /usr/local/lib/ruby/gems/1.8/gems/couchrest_model-1.1.2/lib/couchrest/model/views.rb:142:in `fetch_view'
from /usr/local/lib/ruby/gems/1.8/gems/couchrest_model-1.1.2/lib/couchrest/model/views.rb:135:in `fetch_view_with_docs'
from /usr/local/lib/ruby/gems/1.8/gems/couchrest_model-1.1.2/lib/couchrest/model/views.rb:104:in `view'
from /usr/local/lib/ruby/gems/1.8/gems/couchrest_model-1.1.2/lib/couchrest/model/document_queries.rb:12:in `all'
from /var/www/monitoring_couch/app/models/page.rb:309:in `update_timestamp'
from (irb):2
To add a new property you need to load each document, modify it, and save it back to the database. You could do it using the bulk update api, but for just 100k I would think it's not worth the extra hassle.
Please post your update script and the traceback you get when the timeout occurs.
Related
After upgrading rails from 4.2 to 5.2 my test gets stuck on a request while it is working in development server I'm getting following failure on running test suit.
Failures:
1) cold end overview shows cold end stats
Failure/Error: example.run
RuntimeError:
Requests did not finish in 60 seconds
# /home/asnad/.rvm/gems/ruby-2.5.0/gems/capybara-2.18.0/lib/capybara/server.rb:94:in `rescue in wait_for_pending_requests'
# /home/asnad/.rvm/gems/ruby-2.5.0/gems/capybara-2.18.0/lib/capybara/server.rb:91:in `wait_for_pending_requests'
# /home/asnad/.rvm/gems/ruby-2.5.0/gems/capybara-2.18.0/lib/capybara/session.rb:130:in `reset!'
# /home/asnad/.rvm/gems/ruby-2.5.0/gems/capybara-2.18.0/lib/capybara.rb:314:in `block in reset_sessions!'
# /home/asnad/.rvm/gems/ruby-2.5.0/gems/capybara-2.18.0/lib/capybara.rb:314:in `reverse_each'
# /home/asnad/.rvm/gems/ruby-2.5.0/gems/capybara-2.18.0/lib/capybara.rb:314:in `reset_sessions!'
# /home/asnad/.rvm/gems/ruby-2.5.0/gems/capybara-2.18.0/lib/capybara/rspec.rb:22:in `block (2 levels) in <top (required)>'
# ./spec/spec_helper.rb:43:in `block (3 levels) in <top (required)>'
# /home/asnad/.rvm/gems/ruby-2.5.0/gems/database_cleaner-1.6.2/lib/database_cleaner/generic/base.rb:16:in `cleaning'
# /home/asnad/.rvm/gems/ruby-2.5.0/gems/database_cleaner-1.6.2/lib/database_cleaner/base.rb:98:in `cleaning'
# /home/asnad/.rvm/gems/ruby-2.5.0/gems/database_cleaner-1.6.2/lib/database_cleaner/configuration.rb:86:in `block (2 levels) in cleaning'
# /home/asnad/.rvm/gems/ruby-2.5.0/gems/database_cleaner-1.6.2/lib/database_cleaner/configuration.rb:87:in `cleaning'
# ./spec/spec_helper.rb:37:in `block (2 levels) in <top (required)>'
# ------------------
# --- Caused by: ---
# Timeout::Error:
# execution expired
# /home/asnad/.rvm/gems/ruby-2.5.0/gems/capybara-2.18.0/lib/capybara/server.rb:92:in `sleep'
Top 1 slowest examples (62.59 seconds, 97.0% of total time):
cold end overview shows cold end stats
62.59 seconds ./spec/features/cold_end_overview_spec.rb:13
Finished in 1 minute 4.51 seconds (files took 4.15 seconds to load)
1 example, 1 failure
my spec_helper.rb has configurations
RSpec.configure do |config|
config.include FactoryBot::Syntax::Methods
config.around(:each) do |example|
DatabaseCleaner[:active_record].clean_with(:truncation)
DatabaseCleaner.cleaning do
if example.metadata.key?(:js) || example.metadata[:type] == :feature
# VCR.configure { |c| c.ignore_localhost = true }
WebMock.allow_net_connect!
VCR.turn_off!
VCR.eject_cassette
example.run
else
# WebMock.disable_net_connect!
VCR.turn_on!
cassette_name = example.metadata[:full_description]
.split(/\s+/, 2)
.join('/')
.underscore.gsub(/[^\w\/]+/, '_')
# VCR.configure { |c| c.ignore_localhost = false }
VCR.use_cassette(cassette_name) { example.run }
VCR.turn_off!
WebMock.allow_net_connect!
end
end
end
config.expect_with :rspec do |expectations|
expectations.include_chain_clauses_in_custom_matcher_descriptions = true
end
config.mock_with :rspec do |mocks|
mocks.verify_partial_doubles = true
end
config.filter_run :focus
config.run_all_when_everything_filtered = true
config.example_status_persistence_file_path = "spec/examples.txt"
if config.files_to_run.one?
config.default_formatter = 'doc'
end
# Print the 10 slowest examples and example groups at the
# end of the spec run, to help surface which specs are running
# particularly slow.
config.profile_examples = 10
# Run specs in random order to surface order dependencies. If you find an
# order dependency and want to debug it, you can fix the order by providing
# the seed, which is printed after each run.
# --seed 1234
config.order = :random
# Seed global randomization in this process using the `--seed` CLI option.
# Setting this allows you to use `--seed` to deterministically reproduce
# test failures related to randomization by passing the same `--seed` value
# as the one that triggered the failure.
Kernel.srand config.seed
end
# Selenium::WebDriver.logger.level = :debug
# Selenium::WebDriver.logger.output = 'selenium.log'
Capybara.register_driver :selenium_chrome_headless do |app|
capabilities = Selenium::WebDriver::Remote::Capabilities.chrome(chromeOptions: { args: %w[headless no-sandbox disable-dev-shm-usage disable-gpu window-size=1200,1500] }, loggingPrefs: { browser: 'ALL' })
Capybara::Selenium::Driver.new(app, browser: :chrome, desired_capabilities: capabilities)
end
Chromedriver.set_version '2.39'
Capybara.javascript_driver = :selenium_chrome_headless
Capybara::Screenshot.prune_strategy = :keep_last_run
in my spec the line sign_in current_user takes too much time actually it redirects to a page and do not get response even long time while it is working on development environment.
what can be the reason if you need anything else please comment.
I've just arrived here myself after upgrading from 4.2 to 5.1 and now 5.2, and I'm seeing the same thing in my testing, when I have frozen a test in mid-request with binding.pry, I get the message Requests did not finish in 60 seconds. What a great story, skip to the end for tl;dr (I may have figured it out.)
Now I have upgraded all of the gems, incrementally so I can preserve the capacity to bisect and observe the source of interesting changes like this one. I only noticed this new 60 second timeout after changing over from chromedriver-helper which reported it had been deprecated to the new webdrivers gem that's taking over, but that seems to be not related as I searched webdrivers for any timeout or 60 second value, and only found references to an unrelated Pull Request #60 (fixes Issue #59).
I checked my gem source directory for this message, Requests did not finish in 60 seconds, and found it was not in fact an older version of Capybara, but that it has been raised from versions dating back to at least 3.9.0, and in the most current version 3.24.0 in lib/capybara/server.rb.
The object used there is a Timer which you can find an interface to it here, in the helper:
https://github.com/teamcapybara/capybara/blob/320ee96bb8f63ac9055f7522961a1e1cf8078a8a/lib/capybara/helpers.rb#L79
This particular message is raised out of the method wait_for_pending_requests which passes a hard 60 into the :expire_in named parameter, then after sends any errors that were encountered in the server thread. This means the time is not configurable, probably 60 seconds is a reasonable length of time to wait for a web request in progress to complete, although it's a bit inconvenient for my test.
That method is only called in one place, reset!, which you can find defined here in capybara/session.rb: https://github.com/teamcapybara/capybara/blob/320ee96bb8f63ac9055f7522961a1e1cf8078a8a/lib/capybara/session.rb#L126
The reset! method is an interesting one that comes with some documentation about how it's used. #server&.wait_for_pending_requests looks like it might call wait_for_pending_requests if it has an active server thread in a request, and then raise_server_error! which similarly acts only if #server&.error is truthy.
Now we find that reset! comes with two aliases, this message reset! is received whenever Capybara calls cleanup! or reset_session!. At this point we can probably understand what happened, but it's still a little bit mysterious when I've been using chromedriver-helper and selenium testing for several years, but never recall seeing this 60 second timeout before. I'm hesitant to point the finger at webdriver, but I don't have any other answers for why this timeout is new. I haven't done really anything that could account for it but upgrade to this gem, and any other gems, plus clearing out deprecation warnings.
It seems possible that in Rails 5.1+, capybara calls reset! a lot more, maybe more than in between test examples. Especially when you read that documentation of the method and think about what Single-Page focus there has been now, and consider all of the things that reset! documentation tells you it doesn't reset, clear browser cache/HTML 5 local storage/IndexedDB/Web SQL database/etc — or maybe I'm imagining it, and this isn't new. But I'm imagining there are a lot of ways that it can call reset! and not land in this timeout code, that are likely to be driver-dependent.
Did you change to webdrivers gem by any chance when you did your Rails upgrade?
Edit: I reverted to chromedriver-helper just to be sure, and that wasn't it. What's actually happening is my test is failing in one thread, but the server has left a binding.pry session open. Capybara has moved onto the next test, and thus to get a fresh session it has called reset!, but 60 seconds later I am still in my pry session, and the server is still not ready to serve a root request. I have a feeling that the threading behavior of capybara has changed, in my memory a pry session opened during a server request would block the test from failing until it had returned. But that's apparently not what's happening anymore.
How did you arrive here? I have no idea unfortunately, but this is a fair description of what's happening when that message is received.
https://devcenter.heroku.com/articles/request-timeout
30 Seconds and a timeout error fires according to their documentation.
I'm uploading and parsing a CSV file to save to my database. One of those files is 1.7MB in size and has 37000 rows.
This process takes a bit long to process, certainly more than 30 seconds.
What can I do in these cases? What options do I have?
require 'csv'
class DatabaseImporterController < ApplicationController
def index
end
def import
# Receive the uploaded CSV file and import to the database.
csv_file = params[:csv_file].tempfile
i = 0
CSV.foreach(csv_file) do |row|
# Structure for CSV file: Year, Make, Model, Trim
if i > 0 then
make = Make.find_or_create_by_name(row[1])
model = make.model.create(:year => row[0], :name => row[2], :trim => row[3])
end
i += 1
end
redirect_to :action => 'list'
end
def list
#models = Model.all
end
end
Instead of processing your CSV file in the controller have it push a notification to a queue with the location of the uploaded file. Then have a worker dyno handle that processing.
You'll pay a little bit more, especially if you're trying to stick with the free single dyno tier, but this is a scalable design (which is why I'd imagine there's a 30 second timeout on HTTP processing).
An alternative is to push the data directly into a table and execute a stored procedure asynchronously. This pushes the work off to Postgres to handle off the HTTP thread and may place your request under the 30 second time limit, though with larger files you may breech this cap anyway.
Before you bother restructuring your entire application you'll want to run a test to ensure that Heroku has not disabled libpq-asynch.
The big cost here in your code above is Make.find_or_create_by_name which is invoking 37,000 separate - as per your example input - SELECT and possibly an INSERT for each row in your CSV. If libpq-asynch is not an option you'll have to create a stored procedure that will perform this functionality in batches of 100 or 1000 rows at a time - that way your controller code isn't making so many round trips to the database. Postgres supports arrays in the classical ordinal index style as well as arrays of row types so this is actually much less painful than it sounds.
I recollect getting log files that were nicely ordered, so that you could follow one request, then the next, and so on.
Now, the log files are, as my 4 year old says "all scroggled up", meaning that they are no longer separate, distinct chunks of text. Loggings from two requests get intertwined/mixed up.
For instance:
Started GET /foobar
...
Completed 200 OK in 2ms (Views: 0.4ms | ActiveRecord: 0.8ms)
Patient Load (wait, that's from another request that has nothing to do with foobar!)
[ blank space ]
Something else
This is maddening, because I can't tell what's happening within one single request.
This is running on Passenger.
I tried to search for the same answer but couldn't find any good info. I'm not sure if you should fix server or rails code.
If you want more info about the issue here is the commit that removed old way of logging https://github.com/rails/rails/commit/04ef93dae6d9cec616973c1110a33894ad4ba6ed
If you value production log readability over everything else you can use the
PassengerMaxInstancesPerApp 1
configuration. It might cause some scaling issues. Alternatively you could stuff something like this in application.rb:
process_log_filename = Rails.root + "log/#{Rails.env}-#{Process.pid}.log"
log_file = File.open(process_log_filename, 'a')
Rails.logger = ActiveSupport::BufferedLogger.new(log_file)
Yep!, they have made some changes in the ActiveSupport::BufferedLogger so it is not any more waiting until the request has ended to flush the logs:
http://news.ycombinator.com/item?id=4483390
https://github.com/rails/rails/commit/04ef93dae6d9cec616973c1110a33894ad4ba6ed
But they have added the ActiveSupport::TaggedLogging which is very funny and you can stamp every log with any kind of mark you want.
In your case could be good to stamp the logs with the request UUID like this:
# config/application.rb
config.log_tags = [:uuid]
Then even if the logs are messed up you still can follow which of them correspond to the request you are following up.
You can make more funny things with this feature to help you in your logs study:
How to log user_name in Rails?
http://zogovic.com/post/21138929607/running-time-in-rails-logs
Well, for me the TaggedLogging solution is a no go, I can live with some logs getting lost if the server crashes badly, but I want my logs to be perfectly ordered. So, following advice from the issue comments I'm applying this to my app:
# lib/sequential_logs.rb
module ActiveSupport
class BufferedLogger
def flush
#log_dest.flush
end
def respond_to?(method, include_private = false)
super
end
end
end
# config/initializers/sequential_logs.rb
require 'sequential_logs.rb'
Rails.logger.instance_variable_get(:#logger).instance_variable_get(:#log_dest).sync = false
As far as I can say this hasn't affected my app, it is still running and now my logs make sense again.
They should add some quasi-random reqid and write it in every line regarding one single request. This way you won't get confused.
I haven't used it, but I believe Lumberjack's unit_of_work method may be what you're looking for. You call:
Lumberjack.unit_of_work do
yield
end
And all logging done either in that block or in the yielded block are tagged with a unique ID.
Often I need to throw a custom(ized) error. Like when a resource cannot be found due to a mismatch in parameters or so.
I prefer to throw existing errors or, throw an error that is inherited from an existing error. That way, I don't introduce Error classes that were already defined and could have been used perfectly (DRY). But it also allows to keep wording and style the same, by inheriting and simply changing a word or two to clarify the difference with the original Error.
For example:
Foo.new
Foo.some_external_id = nil
Foo.fetch_external_resource
# => InvalidOptions: Calling Foo#fetch_external_resource with nil is invalid
I am quite sure such errors are already defined. In fact, after reading trough many lines of code, I found my MongoID driver has Mongoid::Errors::InvalidOptions: Calling Document#find with nil is invalid.
Is there a list of available Error classes in Ruby Core and Ruby on Rails? Is there a way to get such a list for your current project?
Is it smart at all to re-use and/or inherit existing errors, or should I maintain my own, custom set instead?
While this list may change, and it's best to still use Ryan's solution, the list for Rails 4 (limited to Rails classes and not other gems):
AbstractController::ActionNotFound
AbstractController::DoubleRenderError
AbstractController::Error
AbstractController::Helpers::ClassMethods::MissingHelperError
ActionController::ActionControllerError
ActionController::BadRequest
ActionController::InvalidAuthenticityToken
ActionController::MethodNotAllowed
ActionController::MissingFile
ActionController::NotImplemented
ActionController::ParameterMissing
ActionController::RedirectBackError
ActionController::RenderError
ActionController::RoutingError
ActionController::SessionOverflowError
ActionController::UnknownController
ActionController::UnknownFormat
ActionController::UnknownHttpMethod
ActionController::UnpermittedParameters
ActionController::UrlGenerationError
ActionDispatch::Cookies::CookieOverflow
ActionDispatch::IllegalStateError
ActionDispatch::Journey::Router::RoutingError
ActionDispatch::ParamsParser::ParseError
ActionDispatch::RemoteIp::IpSpoofAttackError
ActionDispatch::Session::SessionRestoreError
ActionView::Helpers::NumberHelper::InvalidNumberError
ActiveModel::ForbiddenAttributesError
ActiveModel::MissingAttributeError
ActiveRecord::ActiveRecordError
ActiveRecord::AdapterNotFound
ActiveRecord::AdapterNotSpecified
ActiveRecord::AssociationTypeMismatch
ActiveRecord::AttributeAssignmentError
ActiveRecord::ConfigurationError
ActiveRecord::ConnectionNotEstablished
ActiveRecord::ConnectionTimeoutError
ActiveRecord::DangerousAttributeError
ActiveRecord::DeleteRestrictionError
ActiveRecord::DuplicateMigrationNameError
ActiveRecord::DuplicateMigrationVersionError
ActiveRecord::EagerLoadPolymorphicError
ActiveRecord::HasAndBelongsToManyAssociationForeignKeyNeeded
ActiveRecord::HasManyThroughAssociationNotFoundError
ActiveRecord::HasManyThroughAssociationPointlessSourceTypeError
ActiveRecord::HasManyThroughAssociationPolymorphicSourceError
ActiveRecord::HasManyThroughAssociationPolymorphicThroughError
ActiveRecord::HasManyThroughCantAssociateNewRecords
ActiveRecord::HasManyThroughCantAssociateThroughHasOneOrManyReflection
ActiveRecord::HasManyThroughCantDissociateNewRecords
ActiveRecord::HasManyThroughNestedAssociationsAreReadonly
ActiveRecord::HasManyThroughSourceAssociationNotFoundError
ActiveRecord::HasOneThroughCantAssociateThroughCollection
ActiveRecord::IllegalMigrationNameError
ActiveRecord::ImmutableRelation
ActiveRecord::InvalidForeignKey
ActiveRecord::InverseOfAssociationNotFoundError
ActiveRecord::IrreversibleMigration
ActiveRecord::MultiparameterAssignmentErrors
ActiveRecord::NestedAttributes::TooManyRecords
ActiveRecord::PendingMigrationError
ActiveRecord::PreparedStatementInvalid
ActiveRecord::ReadOnlyAssociation
ActiveRecord::ReadOnlyRecord
ActiveRecord::RecordInvalid
ActiveRecord::RecordNotDestroyed
ActiveRecord::RecordNotFound
ActiveRecord::RecordNotSaved
ActiveRecord::RecordNotUnique
ActiveRecord::Rollback
ActiveRecord::SerializationTypeMismatch
ActiveRecord::StaleObjectError
ActiveRecord::StatementInvalid
ActiveRecord::SubclassNotFound
ActiveRecord::Tasks::DatabaseAlreadyExists
ActiveRecord::Tasks::DatabaseNotSupported
ActiveRecord::ThrowResult
ActiveRecord::TransactionIsolationError
ActiveRecord::Transactions::TransactionError
ActiveRecord::UnknownAttributeError
ActiveRecord::UnknownMigrationVersionError
ActiveRecord::UnknownPrimaryKey
ActiveRecord::WrappedDatabaseException
ActiveSupport::JSON::Encoding::CircularReferenceError
ActiveSupport::MessageVerifier::InvalidSignature
ActiveSupport::SafeBuffer::SafeConcatError
ActiveSupport::XMLConverter::DisallowedType
There's a mostly adequate solution here: http://www.ruby-forum.com/topic/158088
Since this question is unanswered, yet coming up at the top of Google search results, I decided to wrap Frederick Cheung's solution up in a rake task and post it here.
Drop the following in lib/tasks/exceptions.rake
namespace :exceptions do
task :list => :environment do
exceptions = []
ObjectSpace.each_object(Class) do |k|
exceptions << k if k.ancestors.include?(Exception)
end
puts exceptions.sort { |a,b| a.to_s <=> b.to_s }.join("\n")
end
end
Run it with:
bundle exec rake exceptions:list
If you're still on Rails 2, or not using Bundler, leave off the bundle exec
This list is probably adequate, but not exhaustive. For example, ActiveResource defines several exceptions such as ActiveResource::ConnectionError and ActiveResource::TimeoutError that are not appearing when I run this task. Maybe someone else can enlighten me as to why.
A shorter option is available in Rails thanks to ActiveSupport:
puts Exception.descendants.sort_by(&:name)
You can also check out the source to see how they handle it.
I am currently writing a rails app using bleeding edge stuff. Rails3, rSpec2, Ruby 1.9.2 and Geokit 1.5.0. When i try to geocode addresses that have special characters that are not in ASCII-8Bit i get this error:
incompatible character encodings:
UTF-8 and ASCII-8BIT
The Trace is like this:
1) Spot Basic Validations should calculate lat and lng
Failure/Error: spot = Spot.create!({
incompatible character encodings: UTF-8 and ASCII-8BIT
# /Users/nilsriedemann/.rvm/gems/ruby-1.9.2-rc2/gems/geokit-1.5.0/lib/geokit/geocoders.rb:435:in `do_geocode'
# /Users/nilsriedemann/.rvm/gems/ruby-1.9.2-rc2/gems/geokit-1.5.0/lib/geokit/geocoders.rb:126:in `geocode'
# ./app/models/spot.rb:26:in `geocode_address'
# /Users/nilsriedemann/.rvm/gems/ruby-1.9.2-rc2/gems/activesupport-3.0.0.rc/lib/active_support/callbacks.rb:409:in `_run_validation_callbacks'
# /Users/nilsriedemann/.rvm/gems/ruby-1.9.2-rc2/gems/activemodel-3.0.0.rc/lib/active_model/validations/callbacks.rb:53:in `run_validations!'
# /Users/nilsriedemann/.rvm/gems/ruby-1.9.2-rc2/gems/activemodel-3.0.0.rc/lib/active_model/validations.rb:168:in `valid?'
# /Users/nilsriedemann/.rvm/gems/ruby-1.9.2-rc2/gems/activerecord-3.0.0.rc/lib/active_record/validations.rb:55:in `valid?'
# /Users/nilsriedemann/.rvm/gems/ruby-1.9.2-rc2/gems/activerecord-3.0.0.rc/lib/active_record/validations.rb:75:in `perform_validations'
# /Users/nilsriedemann/.rvm/gems/ruby-1.9.2-rc2/gems/activerecord-3.0.0.rc/lib/active_record/validations.rb:49:in `save!'
# /Users/nilsriedemann/.rvm/gems/ruby-1.9.2-rc2/gems/activerecord-3.0.0.rc/lib/active_record/attribute_methods/dirty.rb:30:in `save!'
# /Users/nilsriedemann/.rvm/gems/ruby-1.9.2-rc2/gems/activerecord-3.0.0.rc/lib/active_record/transactions.rb:242:in `block in save!'
# /Users/nilsriedemann/.rvm/gems/ruby-1.9.2-rc2/gems/activerecord-3.0.0.rc/lib/active_record/transactions.rb:289:in `block in with_transaction_returning_status'
# /Users/nilsriedemann/.rvm/gems/ruby-1.9.2-rc2/gems/activerecord-3.0.0.rc/lib/active_record/connection_adapters/abstract/database_statements.rb:139:in `transaction'
# /Users/nilsriedemann/.rvm/gems/ruby-1.9.2-rc2/gems/activerecord-3.0.0.rc/lib/active_record/transactions.rb:204:in `transaction'
# /Users/nilsriedemann/.rvm/gems/ruby-1.9.2-rc2/gems/activerecord-3.0.0.rc/lib/active_record/transactions.rb:287:in `with_transaction_returning_status'
# /Users/nilsriedemann/.rvm/gems/ruby-1.9.2-rc2/gems/activerecord-3.0.0.rc/lib/active_record/transactions.rb:242:in `save!'
# /Users/nilsriedemann/.rvm/gems/ruby-1.9.2-rc2/gems/activerecord-3.0.0.rc/lib/active_record/validations.rb:34:in `create!'
# ./spec/models/spot_spec.rb:13:in `block (2 levels) in <top (required)>'
I used # coding: utf-8 in all of my related files (specs, factories and model). Yet i get this error when i use an address like "Elsassers Straße 27".
Any hints? I thought Geokit was already compatible with 1.9.1 and therefore with all this new encoding thing.
Using CGI.escape is not a good idea, as it gives unexpected results. Try "Oslo, Norway" with and without CGI.escape, you'll see what I mean.
A better solution is to use Iconv on the location:
ic = Iconv.new('US-ASCII//IGNORE', 'UTF-8')
utf8location = ic.iconv(location)
Cheers!
EDIT: I had a suggestion by Wes Gamble for a edit here, which I think is relevant:
Using //IGNORE will remove any non-ASCII characters. But in many (most) cases, you may want to transliterate certain characters such as umlauts (e.g. "Zürich" will become "Zurich") or carons (e.g "Niš" will become "Nis") in order to successfully geocode them. If you ignore non-ASCII characters, then "Zürich" will become "Zrich" and "Niš" will become "Ni", neither of which will successfully geocode.
For this you want to use
ic = Iconv.new('US-ASCII//TRANSLIT', 'UTF-8')
Note that the conversion will throw an exception if the transliteration cannot be completed so make sure you handle that.
CGI.escape seems to be more accurate than Geokit::Inflector::url_escape.
Here are the results of encoding "Elsassers Straße 27"
>> CGI.escape(address)
=> "Elsassers+Stra%C3%9Fe+27"
While
>> Geokit::Inflector::url_escape(address)
=> "Elsassers+Stra%C3e+27"
The letter ß should show as c39F (as per http://www.utf8-chartable.de/unicode-utf8-table.pl)
In addition, debug statement was blowing up (I knew there was a reason to check if debug logging is enabled :)
So, here is my solution for GoogleGeocoder3, I guess others will have a similar problem
module Geokit
module Geocoders
class GoogleGeocoder3 < Geocoder
def self.do_geocode(address, options = {})
bias_str = options[:bias] ? construct_bias_string_from_options(options[:bias]) : ''
address_str = address.is_a?(GeoLoc) ? address.to_geocodeable_s : address
#use CGI.escape instead of Geokit::Inflector::url_escape
url ="http://maps.google.com/maps/api/geocode/json?sensor=false&address=#{CGI.escape(address_str)}#{bias_str}"
res = self.call_geocoder_service(url)
return GeoLoc.new if !res.is_a?(Net::HTTPSuccess)
json = res.body
# escape results of json
logger.debug "Google geocoding. Address: #{address}. Result: #{CGI.escape(json)}"
return self.json2GeoLoc(json, address)
end
end
end
end
Are you using Postgres and pg gem v0.8? Upgrade to 0.9
I know it a very very late answer, but I have written a Google geocoder for the Geokit gem that handles all of this Incompatibility errors. This Geocoder uses the newest V3 API of Google's geocoding service. The advantage is that now it does not parse XML but rather JSON which is faster, paired with the required gem Yajl (a super fast json parser for ruby) is way faster. My benchmarks show about 1.5x times faster than the old way.
https://github.com/rubymaniac/geokit-gem
I had the same problem and I solved this by adding CGI.escape() like this:
geo = Geokit::Geocoders::MultiGeocoder.geocode(CGI.escape(address))