I have an app with a lot of images uploaded via paperclip and stored on S3. I'm having some trouble where S3 is telling my iOS app that a few of the image keys don't exist (although I see that they indeed DO exist when I take a look at my S3 bucket). One of my theories is that this is being caused by the filenames, so I'd like to simplify my paperclip path.
My existing path is:
:path => "/:class/:style/:id_:basename.:extension"
I'd like it to be
:path => "/:class/:id/:style.:extension"
which is much cleaner.
My problem is that I'm not sure how to go about doing this. My first thought was to change the path format string in the model and then reprocess! all of the attachments, but now I realize that paperclip needs to use the original path to get at the original uploaded images before it can reprocess and save the images to a new path.
Is there a simple, quick way to make this change?
Thanks!
You cannot 'rename' an object in S3. However, there is a copy command that will duplicate the object within S3. After you duplicate the object, delete the original.
Related
Background:
I have implemented user-defined cropping on image uploads roughly as-per Ryan Bates Railscast #182.
This works when set to the :file storage method, but not when set to :s3. S3 storage was working fine before adding the intermediate cropping step.
From the server log, it appears to be looking for the source file locally:
[paperclip] An error was received while processing: #<Paperclip::Errors::NotIdentifiedByImageMagickError: /profiles/pictures/000/001/543/original/headshot.jpg is not recognized by the 'identify' command.>
This file is present on S3, but not locally by this point, as the upload is processed before being cropped (as well as after).
My question:
How can I bring the file down from S3 to the local server before the second process step?
N.B. I have looked at other answers on SO already.
Paperclip looking for file locally for reprocessing when using S3 – seems relevant, but the only answer refers to downgrading Paperclip. I can’t do that, and besides, that answer is neither upvoted nor accepted.
Error reprocessing in Paperclip 2.3.5 – this is about an older version of Paperclip.
Other thoughts:
It has occurred to me that another approach would be to store the file locally until it has been cropped, and then use DelayedJob or something similar to upload it to S3 later on. This will be more work though, so I’d rather avoid it for now.
In order to better understand what's happening, it would be cool to see your model set up. Specifically I'm looking for the "has_attached_file" setup.
Just to cover the basics of what I'm looking for: here's an example
has_attached_file :picture,
path: <optional, default is fine.>
url: ':s3_alias_url',
s3_protocol: 'https',
s3_host_alias: 'cdn.<something>.com' (or, s3.amazonaws.com/bucketname/,
storage: :s3,
s3_credentials: Proc.new{ |a| a.instance.credentials }
When you reprocess an image, it should be brought down into a temp file and processed there, then reuploaded with these settings.
based on the profiles/pictures/000/001/543/original/headshot.jpg it almost looks like it's grabbing your path variable, but not going to your s3 bucket to get that image. so I would check the storage value, specifically.
With more info, I can update my answer appropriately.
Wanting to use Orchard 1.7 with Media storage on S3 (as I'm deploying to AppHarbor)
So far I'm looking at the S3 Storage provider But its a bit out of date.
Has anyone done this ? is there a better way to use S3 with the new media manager?
I've got images uploading to s3, but they don't display when I click the folder.
here is the Gist of my updated S3Provider
Missing methods for create file, rename folder, get file, and Get storage path. any help on how to complete these would be appreciated.... however stepping through the debugger in VS this doesn't seem to be the root cause of my displaying images issue above.
Edit
Looks like the file is up loading to s3 but not to the database, due to the GetFile method throwing an error...
Edit 2
Added some code to the Get file method. Now that works; (gist updated) Can up load images. However the thumbnails are still not working, they just come back as empty tags ...Think this is because the media manager is using the Open get method - which is supposed to open a file so you can write a stream to it. Don't know how to achieve this with S3... any ideas welcome
As Part of the AWSSKD NuGet package version 1.5.28.3 you can access a S3FileInfo object. I've used this in my S3 Storage File and updated the S3 Storage provider.
This seem to work, need to do a bit more testing on it.
NOTE: I had to add some code on the GetFile Method to ensure the permissions where set correctly otherwise the updating of thumbnails overwrote permissions on the file.... I'm sure there is a better way to do this.
I know this is a question that may have been asked before (at least in Python), but I am still struggling to get this right. I compare my local folder structure and content with what I have stored in my Amazon S3 bucket. The directories not exisiting on S3, but which are found locally, are to be created in my S3 bucket. It seems that Amazon S3 does not have the concept of a folder, but rather a folder is identified as an empty file of size 0. My question is, how can I easily create a folder in objective-c by putting an empty file (with name correspoding to the folder name) on S3 (I use ASIHTTP for my get and put events)? I want to create the directory explicitly and not implicitly by copying a new file to a non-exisiting folder. I appreciate your help on this.
It seems that Amazon S3 does not have the concept of a folder, but rather a folder is identified as an empty file of size 0
The / character is often used as a delimiter, when keys are used as pathnames. To make a folder called bar in the parent folder foo, create a key with the name /foo/bar/.
Amazon now has an AWS SDK for Objective C. The S3PutObjectRequest class has the method -initWithKey:inBucket:.
I'm using Paperclip (2.3) to handle image uploads on a Rails 3.0.3 app running on Ubuntu. Paperclip is handling the uploads as advertised BUT the RackMultipart* files that are created in the application's /tmp folder persist -- that is, they simply accumulate rather than deleting themselves. I realize that I could use tmpreaper to delete old tmpfiles but I'd really like to find a more elegant (and scalable) solution.
I had a previous issue with temp files (i.e. RackMultipart* files) accumulating in the Rails app's root directory (instead of in /tmp). I resolved this by explicitly setting the temp path in my environment.rb file like so:
ENV['TMPDIR'] = Rails.root.join('tmp')
Is there another environment variable that needs to be set to make sure that the tempfiles are handled properly -- i.e. deleted once they've been saved in the model? I'm not sure if this is a problem with Paperclip or my Rails setup.
I've searched high and low but have made little progress on this. I'd be grateful for any leads.
Sincere thanks.
PS - I'm using currently using S3 for storage. This doesn't seem to be tied to the problem though -- I had the same problem when I was storing the files locally.
The TempFileReaper is the Rack middleware thought to handle this issue.
http://www.rubydoc.info/github/rack/rack/Rack/TempfileReaper
Including this line in the application.rb solves the problem:
config.middleware.use Rack::TempfileReaper
I don't know if this is anymore elegant but this is what I am doing after the file is saved"
tempfile = params[:file].tempfile.path
if File::exists?(tempfile)
File::delete(tempfile)
end
UPDATE: Problem should be resolved in rack-1.6.0.beta2. I see it's already being used in Rails 4.2.0.rc2.
Below workaround served me well for almost a year:
I've added this at the end of controller action that accepts file uploads:
Thread.new { GC.start }
This triggers Garbage Collection of unused Rack::Request objects which also deletes associated temp files. Note it doesn't sweep temp file of current request, but it does remove previous files, and prevents them from accumulating.
We're using Amazon S3 for file storage and recently found out that we need to keep some sort of directory structure. Since S3 doesn't allow that, we know we can name the files according to their structure for storage. For example...
abc/123/draft.doc
What I want to know is if I want to provide a public link to this particular file is there anyway that the file can simply be draft.doc instead of abc/123/draft.doc ?
I feel stupid. After some more investigation I realized that by creating a GET url to the resource, I get exactly what I need.