I have just switched from carrierwave_backgrounder to carrierwave_direct. I have carrierwave_direct set up and functioning. That is, the main file is being uploaded and can be displayed in the view. However, my uploader versions are not being created.
Following is my job:
class ProcessReceiptJob < ApplicationJob
queue_as :process_receipt
def perform(expense_id, key)
expense = Expense.find expense_id
uploader = expense.receipt
expense.key = key
expense.remote_receipt_url = uploader.direct_fog_url(with_path: true)
expense.save!
# expense.recreate_versions!
end
after_perform do |job|
expense = Expense.find(job.arguments.first)
expense.update_column :receipt_processing, false
end
end
When exactly does carrierwave_direct process the versions---or, when is carrierwave instructed to process the versions? I'm assuming that loading the original image using expense.remote_receipt_url, and then calling save! triggers the uploader to process the versions. Is that correct?
In any case, my original image is being uploaded via a background job---however, the versions are not being created/uploaded.
Do I need to "recreate_versions" even thought they don't previously exist? Do I need to somehow explicitly process versions after pointing to the source file or should that be handled automagically?
I was not saving the model after assigning it :key BEFORE sending it the background worker. I was sending the key to the background worker as an argument and then saving the model in processing the job. This was the problem. It is mentioned in the docs the need to save the model after assigning it :key in the success action.
So, I had to update_attributes(key: params[:key]) and THEN call my background job (where incidentally the model is saved again).
Related
We had a third party create a python based image thumbnail script that we set up to trigger on an S3 ObjectCreated event. We then imported a collection of close to 5,000 images after testing the script, but the sheer volume of the image files ended up filling up the lambda test space during the import and only about 12% of the images ended up having thumbnails created for them.
We need to manually create thumbnails for the other 88%. While I have a php based script I can run from EC2, it's somewhat slow. It occurs to me that I could create them 'on demand' and could avoid having to create thumbnails for all of the files that didn't auto-create already during the import.
Some of the files may never be accessed again by a customer - the existing lambda thumbnailer already has a slight delay that I account for in the javascript setTimeout retry loop, but before invoking this loop, I could conceivably check if it's a recent upload -- e.g. within the last 10 seconds -- whenever a thumbnail is not found then trigger the lambda manually before starting the retry loop.
But to do this, I need to have the ability to trigger the Lambda script with the parameters similar to the event trigger. It appears as though their script is only accessing the bucket name and key from the event values:
bucket = event['Records'][0]['s3']['bucket']['name']
key = urllib.parse.unquote_plus(event['Records'][0]['s3']['object']['key'], encoding='utf-8')
Being unfamiliar with lambda and still somewhat new to the sdk, I am not sure how I do a lambda trigger that would include those values for the python script.
I can use either the php sdk or the javascript sdk. (or even the cli)
Any help is appreciated.
I think I figured it out, copying the data structure in the python references to create a bare-bones payload and triggering it as event:
$lambda = $awsSvc->getAwsSdkCached()->createLambda();
// bucket = event['Records'][0]['s3']['bucket']['name']
// key = urllib.parse.unquote_plus(event['Records'][0]['s3']['object']['key'], encoding='utf-8')
$bucket = "mybucket";
$key = "somefolder/someimage.jpg";
$payload_json = sprintf('{"Records":[{"s3":{"bucket":{"name":"%s"},"object":{"key":"%s"}}}]}', $bucket, $key);
$params = array(
'FunctionName' => 'ThumbnailGenerator',
'InvocationType' => 'Event',
'LogType' => 'Tail',
'Payload' => $payload_json
);
$result = $lambda->invoke($params);
I have set up a very simple rails 5 project to narrow down my problem:
https://github.com/benedikt-voigt/capybara_js_demo
In this project the data mutation done by the Capybara JS is not deleted, neither by Rails nor by the Database cleaner I added.
The following great blog argues, that no DatabaseCleaner is needed:
http://brandonhilkert.com/blog/7-reasons-why-im-sticking-with-minitest-and-fixtures-in-rails
but this works only for fixtures, not for the mutation done by an out-of-thread Capybara test.
I added the Database cleaner, but this also needed work.
Does anybody has a sample setup?
From a quick look at your test I would say it's leaving data because the data is actually being added after DatabaseCleaner cleans. The click_on call occurs asynchronously, so when your assert_no_content call happens there's no guarantee the app has handled the request yet or the page has changed yet and since the current page doesn't have the text 'Name has already been taken' on it the assertion passes and the database gets cleaned. While that is happening the click gets processed by the app and the new data is created after the cleaning has occurred. You need to check/wait for content that will appear on the page after the click - something like
page.assert_text('New Foo Created')
You should only be asserting there is no content once you already know the page has changed, or you're expecting something to disappear from the current page.
I solved now the problem by setting the DB connection to one
class ActiveRecord::Base
mattr_accessor :shared_connection
##shared_connection = nil
def self.connection
##shared_connection || ConnectionPool::Wrapper.new(:size => 1) { retrieve_connection }
end
end
ActiveRecord::Base.shared_connection = ActiveRecord::Base.connection
as describe here:
https://mattbrictson.com/minitest-and-rails
I uploaded the working repo here:
https://github.com/benedikt-voigt/capybara_js_demo_working
I have ArtworkUploader and i want to create a duplicate of the artwork image in same directory. Help me to solve this.
My Uploader:
class ArtworkUploader < CarrierWave::Uploader::Base
def store_dir
if model
"uploads/#{model.class.to_s.underscore}/#{model.id}/#{mounted_as}"
end
end
def filename
"artwork.png"
end
end
I tried with console but it doesn't work. What am i missing here?
Console:
> u = User.find(5)
> u.artwork.create(name: "testing.png", file: u.artwork.path)
> NoMethodError: undefined method `create!' for /uploads/5/artwork/Artwork:ArtworkUploader
there are 2 way I can think of you can do that
a) VIA Versioning :
create version of your file
version: copy_file do
process :create_a_copy
end
and now just define create_a_copy method inside your uploader which will just return the same file
This way you can have the copy of the file .I didn't understand your custom filename stuff but the way you have define it your for uploader filename method you can do that the same for version as well something like this
version: copy_file do
process :create_a_copy
def filename
"testing.png"
end
end
NOTE:
Not sure of the filename stuff for version file since I done it long back, but I believe the above setting of different filename method would work.
Adavantage :
All File and copy are associated in one uploader
No extra column is required on database (which is required in approach b)
Now the above approach too have some caveats in it
Slightly more complex
Single delete issue, delete to original uploader would delete its copy as well
b) VIA Separate Column :
They other way you can achieve that is defining a separate column called artwork_copy and mount the same uploader with just a slightly change in your uploader like this
def filename
if self.mounted_as == :artwork
"artwork.png"
else
"testing.png"
end
end
And the way you attach the file (give that your file is stored in locally)
u = User.find(5)
u.artwork_copy = File.open(u.artwork) ## Just cross check
u.save
There is an another way you do this via do the above same
u = User.find(5)
u.artwork_copy.store!(File.open(u.artwork))
Now you it pretty obvious what the advantage/disadvantage of the approach b mention above
Hope this make sense
The idea is:
Perform some time consuming action in background.
Have the results from that action be propagated back to the controller using a callback.
Store the result in an in memory session store.
Have the result in session be used and available from that point onward.
Controller receives the results in the callback:
# controller callback, saves result to session
# this method is executed as expected
# session id matches the one from other controller actions
def on_counter_value_calculated(context, new_value)
#counter = new_value
session[:counter] = #counter
end
However, stored session is lost in subsequent calls:
# although the same session is targeted (same id)
# the saved value is not the same
def index
#counter = session[:counter] || 0
end
I've created a small Rails project that demonstrates the issue:
https://github.com/elvanja/controller_callbak_store_in_session
Any input appreciated.
Checked Rails code, and if I understand correctly, session in fact comes from request:
# actionpack-3.2.11/lib/action_controller/metal.rb:132
delegate :session, :to => "#_request"
It seems session is valid only within request cycle context and although it can be accessed, the changes are not saved, as demonstrated by the project.
Hence, this will not work. As suggested # Ruby on Rails Google Group and Ruby Rogues, the best way to deal with this is to use Sidekiq, DelayedJob, Resque or similar frameworks.
EDIT: Access `session` within rails controller thread is actually the reason (background processing in the example is done in a separate thread).
Formerly: Running a model method on a paperclip attachment after create or update (paperclip callbacks don't seem to work)
Edit (later that day)
I figured out my problem. The processor apparently works with the file that is updated, but doesn't save any files until after processing. I changed my Zip::ZipFile to open 'file' rather than 'attachment.path' since the attachment path doesn't actually hold anything yet. This fixed the first problem. Now I'm having other problems that I'll need to track down. But the answer below is mostly correct.
Edit (1/31/2011):
So I have taken the advice to create a processor for my attachment which will perform all the necessary actions. So far, it looks like it should work; the processor starts and does all the initialization stuff, apparently. However, when I get the point where I want to access the zip file that gets uploaded, I get an error saying that the file cannot be found. The code for my processor is below:
class Extractor < Processor
attr_accessor :resolution, :whiny
def initialize(file, options = {}, attachment = nil)
super
#file = file
#whiny = options[:whiny].nil? ? true : options[:whiny]
#basename = File.basename(#file.path, File.extname(#file.path))
#attachment = attachment
#instance = attachment.instance
end
def make
# do your conversions here, you've got #file, #attachment and #basename to work with
export_path = attachment.path.gsub('.zip', '_content')
Zip::ZipFile.open(attachment.path) { |zip_file|
zip_file.each { |image|
image_path = File.join(export_path, image.name)
FileUtils.mkdir_p(File.dirname(image_path))
unless File.exist?(image_path)
zip_file.extract(image, image_path)
# ..stuff that it does..
end
}
}
# clean up source files, but leave the zip
FileUtils.remove_dir(export_path)
# you return a file handle which is the processed result
dst = File.open result_file_path
end
end
And here is the contents of the error that I get:
Zip::ZipError in GalleriesController#create
File /home/joshua/railscamp/moments_on_three/public/assets/archives/delrosario.zip not found
Rails.root: /home/joshua/railscamp/moments_on_three
Application Trace | Framework Trace | Full Trace
config/initializers/extractor.rb:16:in `make'
app/controllers/galleries_controller.rb:32:in `new'
app/controllers/galleries_controller.rb:32:in `create'
Request
Parameters:
{"utf8"=>"✓",
"authenticity_token"=>"0s4L4MrlqjDTMjzjgkUdvUxeHklZNOIShDhT6fgOICY=",
"gallery"=>{"name"=>"DelRosario",
"public"=>"0",
"owner_id"=>"1",
"shoot_date(1i)"=>"2011",
"shoot_date(2i)"=>"1",
"shoot_date(3i)"=>"31",
"category_id"=>"1",
"archive"=>#<ActionDispatch::Http::UploadedFile:0x00000004148d78 #original_filename="delrosario.zip",
#content_type="application/zip",
#headers="Content-Disposition: form-data; name=\"gallery[archive]\"; filename=\"delrosario.zip\"\r\nContent-Type: application/zip\r\n",
#tempfile=#<File:/tmp/RackMultipart20110131-9745-14u347v>>},
"commit"=>"Create Gallery"}
From what I can tell it's looking for the file in the right place, but the file doesn't seem to be uploaded yet to access it. As far as I'm aware, Paperclip is smart enough to know and wait for the attachment to upload before it tries to process it. Can anyone spot what I'm doing wrong here?
Thanks a lot.
Old stuff:
I'm developing a photo gallery app using Rails 3 and Paperclip. The Admin is able to create a gallery and upload a zip file containing a bunch of images.
What I want to happen:
Enter gallery info and zip file to upload into the form.
Hit 'Create Gallery' button.
Form posts, gallery saves, and zip file gets uploaded.
After zip file is uploaded, run the method :extract_photos (btw, this code
works).
4.a. At the end of this method, zip file is destroyed.
Admin is redirected to gallery page with all the photos in it (where
gallery has_many photos).
I've tried to make this work several different ways.
Before, I created a controller method which would allow the Admin to click a link which ran the :extract_photos method. This worked on my computer, but for some reason the server had trouble routing this on the client's computer. So it's a no go. Plus I thought it was an ugly way of doing it.
Recently, I tried using callback methods. after_save didn't work because it apparently interrupts the form POST and the file doesn't get uploaded and the :extract_photos method can't find the file.
I checked out callback methods on the Paperclip github page, and it talks about the callbacks:
Before and after the Post Processing
step, Paperclip calls back to the
model with a few callbacks, allowing
the model to change or cancel the
processing step. The callbacks are
"before_post_process" and
"after_post_process" (which are called
before and after the processing of
each attachment), and the
attachment-specific
"beforepost_process" and
"afterpost_process". The callbacks are
intended to be as close to normal
ActiveRecord callbacks as possible, so
if you return false (specifically -
returning nil is not the same) in a
before filter, the post processing
step will halt. Returning false in an
after filter will not halt anything,
but you can access the model and the
attachment if necessary.
I've tried using before_post_process and after_post_process, but it can't find the file to run the process, so the file obviously isn't getting uploaded by the time those methods are getting called (which I think is strange). Additionally, when I try beforepost_process and afterpost_process, I get a NoMethodError.
So how do I call a method on an attachment when it is created or updated, but after the file is uploaded and in the right place?
UPDATE
I tried the code below, moving my extraction method code into the make method of the processor. I've gotten farther than I have did before with trying to write a processor, but it's still a no-go. The process throws an exception as soon as I try and open the uploaded file for processing, saying the file doesn't exist. The naming scheme is correct and everything, but still nothing is getting uploaded before the process is getting triggered. Does anyone have any idea why this is happening?
You can write your own processor to accomplish this.
in your model when declaring the paperclip stuff add a custom processor
has_attached_file :my_attachment, {
:styles => {:original => {:processors => [:my_processor]}}
}.merge(PAPERCLIP_SETTINGS)
then write your own processor and put it config/initializers:
module Paperclip
class MyProcessor < Processor
attr_accessor :resolution, :whiny
def initialize(file, options = {}, attachment = nil)
super
#file = file
#whiny = options[:whiny].nil? ? true : options[:whiny]
#basename = File.basename(#file.path, File.extname(#file.path))
#attachment = attachment
end
def make
# do your conversions here, you've got #file, #attachment and #basename to work with
# you return a file handle which is the processed result
dst = File.open result_file_path
end
end
end
I am using a custom processor to things similar to what you are doing with lots of processing and converting of the file in the middle and it seems to work well.