I'd like ExceptionNotifier to send out an email when an exception happens in a delayed job, just like for other exceptions. How can I achieve that?
I do this with Rails 3.2.6, delayed_job 3.0.3 and exception_notification 2.6.1 gem
# In config/environments/production.rb or config/initializers/delayed_job.rb
# Optional but recommended for less future surprises.
# Fail at startup if method does not exist instead of later in a background job
[[ExceptionNotifier::Notifier, :background_exception_notification]].each do |object, method_name|
raise NoMethodError, "undefined method `#{method_name}' for #{object.inspect}" unless object.respond_to?(method_name, true)
end
# Chain delayed job's handle_failed_job method to do exception notification
Delayed::Worker.class_eval do
def handle_failed_job_with_notification(job, error)
handle_failed_job_without_notification(job, error)
# only actually send mail in production
if Rails.env.production?
# rescue if ExceptionNotifier fails for some reason
begin
ExceptionNotifier::Notifier.background_exception_notification(error)
rescue Exception => e
Rails.logger.error "ExceptionNotifier failed: #{e.class.name}: #{e.message}"
e.backtrace.each do |f|
Rails.logger.error " #{f}"
end
Rails.logger.flush
end
end
end
alias_method_chain :handle_failed_job, :notification
end
It's probably a good idea to load this code in all environments to catch errors after bundle update etc before they reach production. I do this by having a config/initializers/delayed_job.rb file but you could duplicate the code for each config/environments/* environment.
Another tip is to tune the delayed job config a bit as default you may get a lot of duplicate exception mails when job fails.
# In config/initializers/delayed_job_config.rb
Delayed::Worker.max_attempts = 3
Update I had some problems with the delayed_job daemon silently exiting and it turned out to be when ExceptionNotifier fails to send mail and no one rescued the exception. Now the code rescues and log them.
Adding to #MattiasWadman answer, since exception_notification 4.0 there's a new way to handle manual notify. So instead of:
ExceptionNotifier::Notifier.background_exception_notification(error)
use
ExceptionNotifier.notify_exception(error)
Another way to handle exceptions (put as an initializer):
class DelayedErrorHandler < Delayed::Plugin
callbacks do |lifecycle|
lifecycle.around(:invoke_job) do |job, *args, &block|
begin
block.call(job, *args)
rescue Exception => e
# ...Process exception here...
raise e
end
end
end
end
Delayed::Worker.plugins << DelayedErrorHandler
alias_method_chain no longer exists in Rails 5.
Here's the new (proper) way to do this using Ruby 2's prepend
# In config/initializers/delayed_job.rb
module CustomFailedJob
def handle_failed_job(job, error)
super
ExceptionNotifier.notify_exception(error, data: {job: job})
end
end
class Delayed::Worker
prepend CustomFailedJob
end
For exception_notification 3.0.0 change:
ExceptionNotifier::Notifier.background_exception_notification(error)
to:
ExceptionNotifier::Notifier.background_exception_notification(error).deliver
simpler and updated answer:
# Chain delayed job's handle_failed_job method to do exception notification
Delayed::Worker.class_eval do
def handle_failed_job_with_notification job, error
handle_failed_job_without_notification job, error
ExceptionNotifier.notify_exception error,
data: {job: job, handler: job.handler} rescue nil
end
alias_method_chain :handle_failed_job, :notification
end
And test on console with:
Delayed::Job.enqueue (JS=Struct.new(:a){ def perform; raise 'here'; end }).new(1)
Related
Iam using delayed job in a rails application. I want to notify an error to airbake whenever a delayed job fails. I checked on github and leant about the failure method.
I want to send the last_error attribute of failed delayed job to airbrake. Something like this:
class ParanoidNewsletterJob < NewsletterJob
def perform
end
def failure
Airbrake.notify(:message => self.last_error, :error_class => self.handler)
end
end
But it gives me the following runtime error:
undefined method `last_error' for #<struct ParanoidNewsletterJob>
Please help me figure out how I can notify Airbrake the last_error of a failed delayed_job.
Many Thanks!!
You should be able to pass the job to the failure method, and then extract the last_error from the job. i.e.
def failure(job)
Airbrake.notify(:message => job.last_error, :error_class => job.handler)
end
this should work fine
def failure(job)
Airbrake.notify(:message => job.error, :error_class => job.error.class, :backtrace => job.error.backtrace)
end
There are two ways you can achieve what you want:
A job specific method which only applies to the type of job you want by implementing the failure method with the job as the parameter. The job will contain error and last_error. And this is what other answers are about.
A global option where a plugin can be developed to apply it to any job type created. This is desired if all jobs need to be monitored. The plugin can be registered and perform actions around various events in the lifecycle of a job. For example, below is a plugin to update the last_error if we want to process it before storing to database
One example below:
require 'delayed_job'
class ErrorDelayedJobPlugin < Delayed::Plugin
def self.update_last_error(event, job)
begin
unless job.last_error.nil?
job.last_error = job.last_error.gsub("\u0000", '') # Replace null byte
job.last_error = job.last_error.encode('UTF-8', invalid: :replace, undef: :replace, replace: '')
end
rescue => e
end
end
callbacks do |lifecycle|
lifecycle.around(:failure) do |worker, job, *args, &block|
update_last_error(:around_failure, job)
block.call(worker, job)
end
end
end
Basically it will be called when any failure occurs for any job. For details on how this callback thing work, you can refer to A plugin to update last_error in Delayed Job.
I have this custom job which is located in /lib/jobs/MessageNotificationJob.rb
class MessageNotificationJob < Struct.new(:user_id, :message_id)
def perform
#user = User.find(user_id)
#message = Message.find(message_id)
if !message.reciever_open
MessagesMailer.message_notification(#user, #message ).deliver
end
end
end
I call it from a method in the MessagesController with
Delayed::Job.enqueue(MessageNotificationJob.new(#user.id, #msg.id))
In my application.rb I have
config.autoload_paths += Dir["#{config.root}/lib/**/"]
But I get the error
NameError - uninitialized constant MessagesController::MessageNotificationJob:
How do I fix this? I've tried several variations, with this as the latest one as a solution to a similar question. I am missing something?
I haven't used the latest DelayedJob, but it looks an awful like the delayed job doesn't have the MessageNotificationJob class loaded when it runs. Try the following:
# file: config/initializers/custom.rb
require 'message_notification_job'
See also this wiki entry -- I think newer versions of DJ improved the error message (rather than silently failing), but the underlying cause is the same.
I have an iteration in which I am calling set_table_name for some model. The idea is that in each iteration, the model changes its table. Sometimes, the table won't exist, in that case, an error of this kind will happen:
Mysql2::Error: Table 'db_name.table_name_xyz' doesn't exist
I want the iteration to keep running and not abort because of the error. I've wrapped the set_table_name line of code with a begin and rescue, but no exception seems to be raised as the script instantly aborts on error (it doesn't execute the rescue code). Here's the code:
((start_year)..(start_actual_year)).each do |year|
begin
Data.set_table_name("Secciones#{year}#{year + 1}")
rescue Exception => e
next
end
end
Can I rescue this kind of error? What should I do? Thanks!
Are you sure you are catching the exception correctly? Here's more info if needed.
begin
# Your code here.
rescue Exception => e
# Output the exception that it is catching.
puts e.message
ensure
# The ensure block is used to close a db connection if needed.
end
I have two rake tasks that I'd like to run nightly. I'd like them to log to one file. I thought this would do the trick (got it here: Rake: logging any task executing):
application.rb
module Rake
class Task
alias_method :origin_invoke, :invoke if method_defined?(:invoke)
def invoke(*args)
#logger = Logger.new('rake_tasks_log.log')
#logger.info "#{Time.now} -- #{name} -- #{args.inspect}"
origin_invoke(args)
end
end
end
and then in the rakefile:
task :hello do
#logger.warn "Starting Hello task"
puts "Hello World!"
puts "checking connection "
checkConnection
puts "done checking"
#logger.debug "End hello rake task"
end
But when I run the task I get:
private method 'warn' called for nil:NilClass
I've tried a couple of flavors of that call to logging (#, ##, no #) to no avail. Read several threads on here about it. The
rubyonrails.org site doesn't mention logging in rake tasks. The tasks that I'm invoking are fairly complex (about 20-40 mins to complete) so I'll really want to know what went wrong if they fail. I'd prefer for DRY reasons to only create the logger object once.
Unless you're wrapping everything in giant begin/rescue's and catching errors that way, the best way to log errors is to catch all output from stderr and stdout with something like:
rake your:job 2>&1 >> /var/log/rake.log
You could also set your Rails environment to use the system logger as well.
I ended up solving this (or at least well enough) by making a "log" task and depending on that in other tasks. Not really ideal, since that means having to include that dependency in any new task, but I have only a few tasks so this will do fine. I'm aware that there is a "file" task but it didn't seem to want to work in Windows, so I chose this because it seems to be more cross platform and it's more explicit.
I need a logger object because I am passing that object into some method calls in the [...] sections. There's enough begin/rescue/end in there that writing to the output stream wouldn't work (I think).
#log_file = "log/tasks.log"
directory "log"
task :check_log => ["log"] do
log = #log_file
puts 'checking log existence'
if not FileTest.exists? ("./#{log}")
puts 'creating log file'
File.open(log, 'w')
end
end
task :check_connection => [:check_log] do
begin
conn = Mongo::Connection.new
[...]
end
end
task :nightly_tasks => [:check_connection, :environment ] do
for i in 1..2
logger.warn "#########################"
end
[...]
logger.warn "nightly tasks complete"
end
def logger
##logger ||= Logger.new( File.join(Rails.root, #log_file) )
end
I have an rspec test that I expect to fail, but it is passing because the code that it is testing rescues the exception that rspec raises. Here's an example of the situation:
class Thing do
def self.method_being_tested( object )
# ... do some stuff
begin
object.save!
rescue Exception => e
# Swallow the exception and log it
end
end
end
In the rspec file:
describe "method_being_tested" do
it "should not call 'save!' on the object passed in" do
# ... set up the test conditions
mock_object.should_not_receive( :save! )
Thing.method_being_tested( mock_object )
end
end
I knew that the execution was reaching the "object.save!" line of the method being tested, and the test should therefore be failing, but the test passes. Using the debugger in the rescue block, I find the following:
(rdb:1) p e # print the exception object "e"
#<RSpec::Mocks::MockExpectationError: (Mock "TestObject_1001").save!
expected: 0 times
received: 1 time>
So basically the test is failing but, but the failure is being suppressed by the very code it is trying to test. I cannot figure out a viable way to stop this code from swallowing Rspec exceptions without somehow compromising the code. I don't want the code to explicitly check if the exception is an Rspec exception, because that is bad design (tests should be written for code, code should never be written for tests). But I also can't check that the exception is any particular type that I DO want it to catch, because I want it to catch ANYTHING that could be raised in a normal production environment.
Someone must have had this problem before me! Please help me find a solution.
Assuming the code is correct as-is:
describe "method_being_tested" do
it "should not call 'save!' on the object passed in" do
# ... set up the test conditions
calls = 0
mock_object.stub(:save!) { calls += 1 }
expect {Thing.method_being_tested(mock_object)}.to_not change{calls}
end
end
If there's no need to catch absolutely all exceptions including SystemExit, NoMemoryError, SignalException etc (input from #vito-botta):
begin
object.save!
rescue StandardError => e
# Swallow "normal" exceptions and log it
end
StandardError is the default exception level caught by rescue.
from rspec-mock:
module RSpec
module Mocks
class MockExpectationError < Exception
end
class AmbiguousReturnError < StandardError
end
end
end
Do you really need to catch Exception? Could you catch StandardError instead?
Catching all exceptions is generally a bad thing.
I would refactor it like so:
class Thing do
def self.method_being_tested!( object )
# ... do some stuff
return object.save
end
end
If you want to ignore the exception thrown by save! there is no point in calling save! in the first place. You just call save and inform the calling code accordingly.