encoding error when uploading image via paperclip - ruby-on-rails-3

I'm trying to upload an image to paperclip and save it to s3. However, I get the following error in my console
!! Unexpected error while processing request: invalid byte sequence in UTF-8
There are a few responses on StackOverflow about how to resolve this problem, though most point to the original solution being an update to Rack. However, I'm using Ruby 1.9.3 and Rails 3.1.3, and believe I don't have Rack (I haven't installed it as a Gem, should I??).
The filenames that I've been trying are fairly simple, so I'm assuming the issue is in the actual file, but I'm not sure how to debug which upload variable the error is coming from. Rails isn't putting any of these errors in the log files, so I can't seem to get more details.
My controller is fairly simple, just like the example on the paperclip github documentation
def create
wine_photo = WinePhoto.create(params[:wine_photo])
return render :json => wine_photo
end
though originally I used the more common
wine_photo - WinePhoto.new(params[:wine_photo])
if wine_photo.save
return render :json => wine_photo
else
return render :json => wine_photo.errors
end
My model (which I doubt is very helpful) is
class WinePhoto true
validates_with AttachmentPresenceValidator, :attributes => :photo
belongs_to :wine
belongs_to :user
def photo_url
photo.url
end
end
based on this response on stackoverflow, Ruby Invalid Byte Sequence in UTF-8, I've tried the below in my controller
def create
wine_photo = WinePhoto.new(params[:wine_photo])
wine_photo.photo = IO.read(wine_photo.photo).force_encoding("ISO-8859-1").encode("utf-8", replace: nil)
...
but still got the error.
Any suggestions on how to get past this encoding issue? Is there a way to confirm that the error is coming from the file being uploaded?
My upload code (ajax) is
save_photo: function(){
var file = document.getElementById('file_api').files[0];
console.log(file);
var xhr = new XMLHttpRequest();
if (xhr.upload && file.type == "image/jpeg" ) {
// create progress bar
var o = document.getElementById("progress");
var progress = o.appendChild(document.createElement("p"));
progress.appendChild(document.createTextNode("upload " + file.name));
// progress bar
xhr.upload.addEventListener("progress", function(e) {
var pc = parseInt(100 - (e.loaded / e.total * 100));
progress.style.backgroundPosition = pc + "% 0";
}, false);
// file received/failed
xhr.onreadystatechange = function(e) {
if (xhr.readyState == 4) {
progress.className = (xhr.status == 200 ? "success" : "failure");
}
};
// start upload
xhr.open("POST", document.getElementById("add_photo").action, true);
xhr.setRequestHeader("X_FILENAME", file.name);
xhr.send(file);
}
}
and the params for file are
File {webkitRelativePath: "", lastModifiedDate: Thu Nov 10 2011 09:40:39 GMT+1100 (AUS Eastern Summer Time), name: "WP_000012.jpg", type: "image/jpeg", size: 1344450}

After two days of messing with this, it turns out the problem was in my ajax upload, rails wasn't getting the file field at all.
I followed this blog post to get the ajax upload working,
https://github.com/newbamboo/example-ajax-upload/blob/master/public/index.html
and changed the name in my input file to < input type="file" name="wine_photo[photo]" > where before the name was just photo.
the html form now also has
<input name="utf8" type="hidden" value="✓">

Related

Paperclip not displaying image (rails api)

I have a user module and I have generated paperclip attachment: profile_pic
user.rb:
has_attached_file :profile_pic,
style: { :medium => "300x300>", thumb: "100x100>" },
default_url: "/images/:style/missing.png"
controller:
image_base = params[:manager][:profile_pic]
if image_base != nil
image = Paperclip.io_adapters.for(image_base)
image.original_filename = params[:manager][:file_name]
current_user.profile_pic = image
current_user.errors.delete(:profile_pic)
current_user.save
end
config/initializers/paperclip.rb:
Paperclip::DataUriAdapter.register
It's not showing any errors but If I am trying to display the image, it gives me following error:
ActionController::RoutingError (No route matches [GET] "/system/managers/profile_pics/000/000/008/original/icon_new.png"):
When I am trying in console like:
user.profile_pic.display
/system/managers/profile_pics/000/000/008/original/icon_new.png?
1556082410 => nil
Picture was saved in public folder
The default folder is the following, so Paperclip uploaded to the right place:
:rails_root/public/system/:class/:attachment/:id_partition/:style/:filename
But it looks like you are having troubles accessing the right location. Have you tried user.profile_pic.url? I tried on my working example, both .display and .url are the same but I use an AWS S3 bucket.
user.profile_pic.url should be where to find the file on the web.
user.profile_pic.path should be where to find the file on the file system.
The problem is with production mode and Solved this by adding the following line in production.rb
config.public_file_server.enabled = true

Accept an image via Rails 5 API and Active Storage

I am trying to send images to my Rails app and then store them via Active Storage.
I tried Base64 and direct upload and researched for hours but nothing really works.
Can somebody point me to a good way?
My last attempt was to use Base64 like so:
def attach_preview
page = Page.first
content = JSON.parse(request.body.read)
decoded_data = Base64.decode64(content["file_content"].force_encoding("UTF-8"))
begin
file = Tempfile.new('test')
file.write decoded_data
#page.thumbnail = file
filename = "foooo"
page.thumbnail.attach(io: File.read(file), filename: filename)
if page.save
render :json => {:message => "Successfully uploaded the profile picture."}
else
render :json => {:message => "Failed to upload image"}
end
ensure
file.close
file.unlink
end
end
But this results in a "\xAB" from ASCII-8BIT to UTF-8 error.
Dont really care if its Base64 or something else, I just need a way :-)
This works, I use IO directly since ActiveStorage needs it anyway.
def attach_thumbnail
content = JSON.parse(request.body.read.force_encoding("UTF-8"))
decoded_data = Base64.decode64(content["file_content"])
io = StringIO.new
io.puts(decoded_data)
io.rewind
#page.thumbnail.attach(io: io, filename: 'base.png')
#page.save
render json: {
success: #page.thumbnail.attached?,
thumbnail_url: url_for(#page.thumbnail),
page: #page
}
end

Rails 3.2 SSRS and SavonRD

I'm integrating my Rails 3.2 app with SSRS for reporting using Savonrb to do the Soap call to the SSRS web service. Everything is finally working and I can successfully make the following calls:
LoadReport
SetExecutionParameters
Render
The render call takes a format parameter (in my case PDF) and returns an XML response. I thought the call would result in the report being rendered in a pdf, but it doesn't. What do I need to do with the response object to render the report?
code below:
response = #client.call(:load_report, message: {report: "/path/to/report"} )
response = #client.call(:set_execution_parameters, message: { "Parameters" => param1} )
response = #client.call(:render, message: {"Format" => "PDF"})
Thanks!
The answer is that I need to take the xml string that was returned from the render call and decode it to Base64, write it to a file, and then user send_file to stream the file to the browser. This works now, code below:
response = #newClient.call(:render, message: {"Format" => "PDF"})
#results = response.to_hash
#report = #results[:render_response][:result]
#extension = #results[:render_response][:extension]
#mime_type = #results[:render_response][:mime_type]
#report_name = "MyReportName.pdf" # this should be dynamically generated from a param passed in the call to run_report
File.open(#report_name, "wb+") do |f|
f.puts Base64.decode64(#report)
end
#file = File.open(#report_name, 'r')
send_file #file, :filename => #report_name, :type => "application/pdf; charset=utf-8", :disposition => "attachment"

Rails - Uploading large files directly to S3 with Jquery File Upload (hosted on Heroku )

I'm using Heroku, which means I have to upload multiple large files to S3 directly.. I'm using Rails 3.2.11, and Ruby 1.9.3. I do not wish to use carrierwave or paperclip gems, or really change much at this point - I just need to get this what I have working.
Before trying to move to S3, if I ran my app locally, I could upload multiple large files to the local file system. When I ran it on Heroku, small files upload but large ones failed. Hence the switch to S3..
I tried several tweaks, and also this link below, but it's just too much of a change to what I have that already working with the local server's file system (and Heroku as well, but Heroku just can't handle large files ..)
Tried: https://devcenter.heroku.com/articles/direct-to-s3-image-uploads-in-rails
I've tried some of the other examples here on Stack Overflow but they are too much of a change for what works locally, and well, I don't grasp everything they are doing.
Now, what happens when I do try to upload images?
It's as if the file upload works - the preview images are successfully created, but nothing is ever uploaded to Amazon s3, and I don't receive any kind of error messages (like s3 authentication failure or anything.. nothing)
What do I need to change in order to get the files over to my s3 storage, and what can I write out to console to detect problems, if any, connecting to my s3?
My form:
<%= form_for #status do |f| %>
{A FEW HTML FIELDS USED FOR A DESCRIPTION OF THE FILES - NOT IMPORTANT FOR THE QUESTION}
File:<input id="fileupload" multiple="multiple" name="image"
type="file" data-form-data = <%= #s3_direct_post.fields%>
data-url= <%= #s3_direct_post.url %>
data-host =<%=URI.parse(#s3_direct_post.url).host%> >
<%= link_to 'submit', "#", :id=>'submit' , :remote=>true%>
<% end %>
My jquery is:
....
$('#fileupload').fileupload({
formData: {
batch: createUUID(),
authenticity_token:$('meta[name="csrf-token"]').attr('content')
},
dataType: 'json',
acceptFileTypes: /(\.|\/)(gif|jpe?g|png)$/i,
maxFileSize: 5000000, // 5 MB
previewMaxWidth: 400,
previewMaxHeight: 400,
previewCrop: true,
add: function (e, data) {
tmpImg.src = URL.createObjectURL(data.files[0]) ; // create image preview
$('#'+ fn + '_inner' ).append(tmpImg);
...
My controller:
def index
#it's in the index just to simplify getting it working
#s3_direct_post = S3_BUCKET.presigned_post(key: "uploads/#{SecureRandom.uuid}/${filename}", success_action_status: '201', acl: 'public-read')
end
The element that is generated for the form is (via Inspect Element):
<input id="fileupload" multiple="multiple" name="image"
data-form-data="{"key"=>"uploads/34a64607-8d1b-4704-806b-159ecc47745e/${filename}"," "success_action_status"="
>"201"," "acl"=">"public-read"," "policy"=">"[encryped stuff - no need to post]","
"x-amz-credential"=">"
[AWS access key]/[some number]/us-east-1/s3/aws4_request"
," "x-amz-algorithm"=">"AWS4-HMAC-SHA256"
," "x-amz-date"=">"20150924T234656Z"
," "x-amz-signature"=">"[some encrypted stuff]"}"
data-url="https://nunyabizness.s3.amazonaws.com" data-host="nunyabizness.s3.amazonaws.com" type="file">
Help!
With S3 there actually is no easy out of the box solutions to upload files, because Amazon is a rather complex instrument.
I had a similar issue back in the day and spend two weeks trying to figure out how S3 works, and now use a working solution for uploading files onto S3. I can tell you a solution that works for me, I never tried the one proposed by Heroku.
A plugin of choice I use is Plupload, since it is the only component I actually managed to get working, apart from simple direct S3 uploads via XHR, and offers the use of percentage indicators and in-browser image resizing, which I find completely mandatory for production applications, where some users have 20mb images that they want to upload as their avatar.
Some basics in S3:
Step 1
Amazon bucket needs correct configuration in its CORS file to allow external uploads in the first place. The Heroku totorial already told you how to put the configuration in the right place.
http://docs.aws.amazon.com/AmazonS3/latest/dev/cors.html
Step 2
Policy data is needed, otherwise your client will not be able to access the corresponding bucket file. I find generating policies to be better done via Ajax calls, so that, for example, admin gets the ability to upload files into the folders of different users.
In my example, cancan is used to manage security for the given user and figaro is used to manage ENV variables.
def aws_policy_image
user = User.find_by_id(params[:user_id])
authorize! :upload_image, current_user
options = {}
bucket = Rails.configuration.bucket
access_key_id = ENV["AWS_ACCESS_KEY_ID"]
secret_access_key = ENV["AWS_SECRET_ACCESS_KEY"]
options[:key] ||= "users/" + params[:user_id] # folder on AWS to store file in
options[:acl] ||= 'private'
options[:expiration_date] ||= 10.hours.from_now.utc.iso8601
options[:max_filesize] ||= 10.megabytes
options[:content_type] ||= 'image/' # Videos would be binary/octet-stream
options[:filter_title] ||= 'Images'
options[:filter_extentions] ||= 'jpg,jpeg,gif,png,bmp'
policy = Base64.encode64(
"{'expiration': '#{options[:expiration_date]}',
'conditions': [
{'x-amz-server-side-encryption': 'AES256'},
{'bucket': '#{bucket}'},
{'acl': '#{options[:acl]}'},
{'success_action_status': '201'},
['content-length-range', 0, #{options[:max_filesize]}],
['starts-with', '$key', '#{options[:key]}'],
['starts-with', '$Content-Type', ''],
['starts-with', '$name', ''],
['starts-with', '$Filename', '']
]
}").gsub(/\n|\r/, '')
signature = Base64.encode64(
OpenSSL::HMAC.digest(
OpenSSL::Digest::Digest.new('sha1'),
secret_access_key, policy)).gsub("\n", "")
render :json => {:access_key_id => access_key_id, :policy => policy, :signature => signature, :bucket => bucket}
end
I went as far as put this method into the application controller, although you could find a better place for it.
Path to this function should be put into the route, of course.
Step 3
Frontend, get plupload: http://www.plupload.com/ make some link to act as the upload button:
<a id="upload_button" href="#">Upload</a>
Make a script that configures the plupload initialization.
function Plupload(config_x, access_key_id, policy, signature, bucket) {
var $this = this;
$this.config = $.extend({
key: 'error',
acl: 'private',
content_type: '',
filter_title: 'Images',
filter_extentions: 'jpg,jpeg,gif,png,bmp',
select_button: "upload_button",
multi_selection: true,
callback: function (params) {
},
add_files_callback: function (up, files) {
},
complete_callback: function (params) {
}
}, config_x);
$this.params = {
runtimes: 'html5',
browse_button: $this.config.select_button,
max_file_size: $this.config.max_file_size,
url: 'https://' + bucket + '.s3.amazonaws.com/',
flash_swf_url: '/assets/plupload/js/Moxie.swf',
silverlight_xap_url: '/assets/plupload/js/Moxie.xap',
init: {
FilesRemoved: function (up, files) {
/*if (up.files.length < 1) {
$('#' + config.select_button).fadeIn('slow');
}*/
}
},
multi_selection: $this.config.multi_selection,
multipart: true,
// resize: {width: 1000, height: 1000}, // currently causes "blob" problem
multipart_params: {
'acl': $this.config.acl,
'Content-Type': $this.config.content_type,
'success_action_status': '201',
'AWSAccessKeyId': access_key_id,
'x-amz-server-side-encryption': "AES256",
'policy': policy,
'signature': signature
},
// Resize images on clientside if we can
resize: {
preserve_headers: false, // (!)
width: 1200,
height: 1200,
quality: 70
},
filters: [
{
title: $this.config.filter_title,
extensions: $this.config.filter_extentions
}
],
file_data_name: 'file'
};
$this.uploader = new plupload.Uploader($this.params);
$this.uploader.init();
$this.uploader.bind('UploadProgress', function (up, file) {
$('#' + file.id + ' .percent').text(file.percent + '%');
});
// before upload
$this.uploader.bind('BeforeUpload', function (up, file) {
// optional: regen the filename, otherwise the user will upload image.jpg that will overwrite each other
var extension = file.name.split('.').pop();
var file_name = extension + "_" + (+new Date);
up.settings.multipart_params.key = $this.config.key + '/' + file_name + '.' + extension;
up.settings.multipart_params.Filename = $this.config.key + '/' + file_name + '.' + extension;
file.name = file_name + '.' + extension;
});
// shows error object in the browser console (for now)
$this.uploader.bind('Error', function (up, error) {
console.log('Expand the error object below to see the error. Use WireShark to debug.');
alert_x(".validation-error", error.message);
});
// files added
$this.uploader.bind('FilesAdded', function (up, files) {
$this.config.add_files_callback(up, files, $this.uploader);
// p(uploader);
// uploader.start();
});
// when file gets uploaded
$this.uploader.bind('FileUploaded', function (up, file) {
$this.config.callback(file);
up.refresh();
});
// when all files are uploaded
$this.uploader.bind('UploadComplete', function (up, file) {
$this.config.complete_callback(file);
up.refresh();
});
}
Plupload.prototype.init = function () {
//
}
Step 4
The implemetation of the general multi-purpose file uploader function:
ImageUploader = {
init: function (user_id, config, callback) {
$.ajax({
type: "get",
url: "/aws_policy_image",
data: {user_id: user_id},
error: function (request, status, error) {
alert(request.responseText);
},
success: function (msg) {
// set aws credentials
callback(config, msg);
}
});
},
},
// local functions
photo_uploader: function (user_id) {
var container = "#photos .unverified_images" // for example;
var can_render = false;
this.init(user_id,
{
select_button: "upload_photos",
callback: function (file) {
file.aws_id = file.id;
file.id = "0";
file.album_title = "userpics"; // I use this param to manage photo directory
file.user_id = user_id;
//console.log(file);
[** your ajax code here that saves the image object in the database via file variable you get here **]
});
},
add_files_callback: function (up, files, uploader) {
$.each(files, function (index, value) {
// do something like adding a progress bar html
});
uploader.start();
},
complete_callback: function (files) {
can_render = true;
}
}, function (config, msg) {
config.key = "users/" + user_id;
// Most important part:
window.photo_uploader = new Plupload(config, msg.access_key_id, msg.policy, msg.signature, msg.bucket);
});
}
can_render variable is useful so that you can make the application only then re-render the page, when the uploader is actually done.
And to make the button work from somewhere else call:
ImageUploader.photo_uploader(user_id);
And the button will act as a Plupload uploader button.
What is important is that Policy is made in a way so that noone can upload the photo into someone else's directory.
It would be great to have a version that does the same not via ajax callbacks, but with web hooks, this is something I want to do in the future.
Again, this is not a perfect solution, but something that works good enough from my experience for the purpose of uploading images and videos onto amazon.
Note in case someone asks why I have this complex object-oriented structure of uploader objects, the reason is that my application has all different kinds of uploaders that behave differently and they need to have an initializer with common behavior. The way I did it I can write an initializer for, say, videos, with minimum amount of code, that will do similar things to the existing image uploader.

Paperclip- validate pdfs with content_type='application/octet-stream'

I was using paperclip for file upload. with validations as below:
validates_attachment_content_type :upload, :content_type=>['application/pdf'],
:if => Proc.new { |module_file| !module_file.upload_file_name.blank? },
:message => "must be in '.pdf' format"
But, my client complained today that he is not able to upload pdf. After investigating I come to know from request headers is that the file being submitted had content_type=application/octet-stream.
Allowing application/octet-stream will allow many type of files for upload.
Please suggest a solution to deal with this.
Seems like paperclip doesn't detect content type correctly. Here is how I was able to fix it using custom content-type detection and validation (code in model):
VALID_CONTENT_TYPES = ["application/zip", "application/x-zip", "application/x-zip-compressed", "application/pdf", "application/x-pdf"]
before_validation(:on => :create) do |file|
if file.media_content_type == 'application/octet-stream'
mime_type = MIME::Types.type_for(file.media_file_name)
file.media_content_type = mime_type.first.content_type if mime_type.first
end
end
validate :attachment_content_type
def attachment_content_type
errors.add(:media, "type is not allowed") unless VALID_CONTENT_TYPES.include?(self.media_content_type)
end
Based on the above, here's what I ended up with which is compatible with PaperClip 4.2 and Rails 4:
before_post_process on: :create do
if media_content_type == 'application/octet-stream'
mime_type = MIME::Types.type_for(media_file_name)
self.media_content_type = mime_type.first.to_s if mime_type.first
end
end
For paperclip 3.3 and Rails 3, I did this a bit differently
before_validation on: :create do
if media_content_type == 'application/octet-stream'
mime_type = MIME::Types.type_for(media_file_name)
self.media_content_type = mime_type.first if mime_type.first
end
end
validates_attachment :media, content_type: { content_type: VALID_CONTENT_TYPES }
By the way, i needed to do this because testing with Capybara and phantom js using attach_file did not generate the correct mime type for some files.