I'm getting "invalid byte sequence in UTF-8" on page requests (permalinks) and I have no idea why nor can I reproduce it but I do get a lot of exceptions like this:
A ArgumentError occurred in products#index:
invalid byte sequence in UTF-8
activesupport (3.0.4) lib/active_support/core_ext/object/blank.rb:68:in `=~'
-------------------------------
Request:
-------------------------------
* URL : http://www.mysite.com/category/category-name-\x8E~ice
* Parameters: {"page"=>1, "controller"=>"products", "action"=>"index", "category"=>"category-name-\x8E~ice"}
The string at the end should not be there ("-\x8E~ice"). Any idea why that shows up or what can I do to debug/reproduce it ?
Thanks
we created a rails middleware that filters out all the strange encodings that can not be handled within our app.
the problem that we encounter is that there are requests that have strange encodings, for example Cp1252 / Windows-1252. when ruby 1.9 tries to match those strings against utf-8 regexps it blows up.
i tried various ways of dealing with this problem by using iconv, but it looks like solutions that work on my mac don't work on the servers. so the simplest approach is probably the best...
I've just posted a new gem called UTF8Cleaner which is heavily based on #phoet and #pithyless' work. It include a Railtie, so you can just drop it in to your Gemfile and forget about those "invalid byte sequence" errors.
https://github.com/singlebrook/utf8-cleaner
Similar to #phoet, I also used a Rails Middleware to solve similar encoding issues.
Tested on Ruby 1.9.3 (no Iconv):
https://gist.github.com/3639014
If you are using apache (and mod_rails) you can prevent these invalid url requests from hitting your Rails application completely by following this answer:
https://stackoverflow.com/questions/13512727/how-can-i-configure-apache-to-respond-400-when-request-contains-an-invalid-byte/13527812#13527812
Related
I have the following situation: My webservice is receiving JSON data and creating models (typical REST scenario). Sometimes I get a
Encoding::CompatibilityError Exception: incompatible character encodings: ASCII-8BIT and UTF-8
error message when saving the records, which can only be (or is) bound to two attributes. Firing up the debugger, setting ANY of those two attributes to an empty string and saving works, like so:
model = Model.new(params[:model])
model.save! # Fails with above error message
model = Model.new(params[:model])
model.attribute1 = ""
model.save! # Works
model = Model.new(params[:model])
model.attribute2 = ""
model.save! # Works too!
Now the params are parsed from the http request, how can they be dependent on each other?
Anyone with the same scenario?
Edit:
We've found the reason for the compability error: https://github.com/jruby/activerecord-jdbc-adapter/issues/229 As it seems, the JDBC adapter has some errors with utf-8 encoding, something which has been fixed for a long time in traditional rubies.
As added in the edit to my original question, the problem is a bug in the JDBC adapter of JRuby (which I forgot to add as a constraint, my bad!): https://github.com/jruby/activerecord-jdbc-adapter/issues/229
This is the query generated by Mongomapper:
MONGODB mydatabase['users'].find({:name=>"bob"}).limit(-1)
But this is not valid in the mongo console since the correct syntax is
db.users.find({:name=>"bob"}).limit(-1)
If I just use the generated one, I got this error in the console
Thu Jan 12 03:01:23 ReferenceError: mydatabase is not defined (shell):1
Is there any way to make it correct? This causes my rails application broken.
You can't use symbols in the MongoDB console as they are ruby and not javascript :-) Try this:
db.users.find({name: "bob"}).limit(-1)
It is not mongodb's issue. 406 is pretty much relating to the controller call.
I need to use:
render :json => #user
rather than
respond_to
I am building a Rails app where I serialize a hash to JSON, and then encode the hash using the gem crypt19 and the blowfish algorithm. I'm using Rails 3.0.9, Ruby 1.9.2 p180, the latest crypt19 v1.2.1 and I'm developing on Windows7. In my Rails controller, I do:
require 'crypt/blowfish'
h=Hash.new
h["thing"]="12345"
h["thang"]="abcdefghijklmnopqrstuvwxyz"
blowfish=Crypt::Blowfish.new("SECRET")
encrypted_thingy=blowfish.encrypt_string(h.to_json)
I assign encrypted_thingy to a value in the model (which is a string),
#my_model.string_thing=encrypted_thingy
#my_model.save
but when I save the model it throws an Argument Error exception where the model is saved.
invalid byte sequence in US-ASCII
(And when I assign it a plain old string, #my_model="xxxxxxxx", everything works fine.
My eventual plan is to store encrypted_thingy in the database via the model, and then later decrypt it, parse out JSON, and get the values for "thing" and "thang".
Searching the 'net suggested that I need to change the encoding, but it is not clear how I do that with the result of the crypt19/blowfish encoding.
Is there any way to store this encrypted string as a string just like any other string I store?
Or is there a way to apply some magic (along with reversible magic) to turn that funky string into a real string which I can pass around in an email?
I was able to make it work. There is a gem called "hex_string" which converts binary-ish things with strange encodings into byte strings.
First I had to do
gem install hex_string
Then the code looked like this
require 'crypt/blowfish'
require 'hex_string'
h=Hash.new
h["thing"]="12345"
h["thang"]="abcdefghijklmnopqrstuvwxyz"
blowfish=Crypt::Blowfish.new("SECRET")
encrypted_thingy=blowfish.encrypt_string(h.to_json).to_hex_string.split(" ").join
The "encrypted_thingy" was now a string which I could pass around easily in my web app, store in a database, debug, etc without worrying about character encoding, etc.
To reverse the process, I did this:
decrypted_string= blowfish.decrypt_string(encrypted_thingy.to_byte_string)
The decrypted string could then be JSON-parsed to extract the data in the original hash.
I would like to alter the Message-ID header that is in the header portion of an email sent from a Ruby on Rails v3 application using ActionMailer.
I am using Sendmail on localhost for mail delivery.
Do I configure this in Sendmail or ActionMailer?
Where do I configure this (if it is ActionMailer): a file in config/ folder or a file in app/mailers/ folder?
Teddy's answer is good, except that if you actually want each message to have a different ID, you need to make the default a lambda. In the first block of code in his answer, it calculates the message-ID once, at init, and uses the same one for every message.
Here's how I'm doing this in my app:
default "Message-ID" => lambda {"<#{SecureRandom.uuid}##{Rails.application.config.mailgun_domain}>"}
... with the domain taken from a custom app config variable (and using SecureRandom.uuid, which is a little more straightforward than a SHA-2 based on the timestamp IMO.)
I usually prefer generating the message-id with a UUID. Assuming you have the uuid gem:
headers['Message-ID'] = "<#{ UUID.generate }#example.com>"
Also you should note that according to RFC 2822 the message-id should be placed inside "<" and ">"
In Rails 4+ (or just Ruby 2.0+) the folowing syntax is working correctly:
default "Message-ID" => ->(v){"<#{Digest::SHA2.hexdigest(Time.now.to_i.to_s)}#yourdomain.com>"}
Tested this with MailCatcher.
I figured this out. The easiest way to do is to use the default method at the top of the mailer class file.
Example:
require 'digest/sha2'
class UserMailer < ActionMailer::Base
default "Message-ID"=>"#{Digest::SHA2.hexdigest(Time.now.to_i.to_s)}#yourdomain.com"
# ... the rest of your mailer class
end
However, I found this difficult to test, so I wrote a private method and used the sent_at time instead of Time.now:
def message_id_in_header(sent_at=Time.now)
headers["Message-ID"] = "#{Digest::SHA2.hexdigest(sent_at.to_i.to_s)}#yourdomain.com"
end
And I simply called that method before calling the mail method. This made it easy to pass a sent_at parameter from my test and verify a match in email.encoded.
#jasoncrawford is almost right. The problem is that the mailgun_domain attribute may not be able at development environment, so it is better to access the ActionMailer configs.
default "Message-ID" => lambda {"<#{SecureRandom.uuid}##{ActionMailer::Base.smtp_settings[:domain]}>"}
I access to my webservice like that:
http://localhost/SuiPService/SuiPDataService.svc/GetShowCodeFiltered?&showName='auto'
It works fine, but when I try:
http://localhost/SuiPService/SuiPDataService.svc/GetShowCodeFiltered?&showName='auto''
it failed, ok this I understand, but if I encode the url like that:
http://localhost/SuiPService/SuiPDataService.svc/GetShowCodeFiltered?&showName='auto%27'
I get the error Bad Request - Error in query syntax. too
What is the solution?
The reason it's failing is because %27 is equal to '.
Everything is encoded before being sent to the web server, even if the URL box doesn't say so.
This will become hard to maintain and possibly confuse your users. I'd change it so you aren't padding the variable with ' and that way you can use http://localhost/SuiPService/SuiPDataService.svc/GetShowCodeFiltered?&showName=auto' if you need to have a ' after it.
Also, if you need the '' around auto. Consider doing this on the server side.
It looks like your using this to build an SQL query...
See here for the reason PHP deprecated it for that exact reason: http://en.wikipedia.org/wiki/Magic_quotes
Hope this helps,
Jeffrey Kevin Pry