Rails 3.2.x: How to change logging levels without restarting the application - ruby-on-rails-3

I would like to change the logging levels of a running Rails 3.2.x application without restarting the application. My intent is to use it to do short-time debugging and information gathering before reverting it to the usual logging level.
I also understand that the levels in ascending order are debug, info, warn, error, and fatal, and that production servers log info and higher, while development logs debug and higher.
I understand that if I run
Rails.logger.level=:debug #or :info, :warn, :error, :fatal
Will this change the logging level immediately?
If so, can I do this by writing a Rake task to adjust the logging level, or do I need to support this by adding a route? For example in config/routes.rb:
match "/set_logging_level/:level/:secret" => "logcontroller#setlevel"
and then setting the levels in the logcontroller. (:level is the logging level, and :secret which is shared between client and server, is something to prevent random users from tweaking the log levels)
Which is more appropriate, rake task or /set_logging_level?

Why don't you use operating system signals for that? For example on UNIX user1 and user2 signals are free to use for your application:
config/initializers/signals.rb:
trap('USR1') do
Rails.logger.level = Logger::DEBUG
end
trap('USR2') do
Rails.logger.level = Logger::WARN
end
Then just do this:
kill -SIGUSR1 pid
kill -SIGUSR2 pid
Just make sure you dont override signals of your server - each server leverages various signals for things like log rotation, child process killing and terminating and so on.

In Rails console, you can simply do:
Rails.logger.level = :debug
Now all executed code will run with this log level

As you have to change the level in the running rails instance, a simple rake task will not work.
I would go with the dedicated route.
instead of a shared secret I would use the app's standard user authentication (if your app has users) and restrict access to admin/super user.

In your controller LogController try this
def setlevel
begin
Rails.logger.level = Logger.const_get(params[:level].upcase)
rescue
logger.info("Logging level #{params[:level]} not supported")
end
end

You can also use gdb to attach to the running process, set the Rails.logger to debug level and then detach. I have created the following 1 liner to do this for my puma process:
gdb attach $(pidof puma) -ex 'call(rb_eval_string("Rails.logger.level = Logger::DEBUG"))' -ex detach -ex quit
NOTE: pidof will return multiple pids, in descending order. So if you have multiple processes with the same name this will only run on the first one returned by pidof. The others will be discarded by the "gdb attach" command with the message: "Excess command line arguments ignored. (26762)". However you can safely ignore it if you only care about the first process returned by pidof.

Using rufus-scheduler, I created this schedule:
scheduler.every 1.second do
file_path = "#{Rails.root}/tmp/change_log_level.#{Process.pid}"
if File.exists? file_path
log_level = File.open(file_path).read.strip
case log_level
when "INFO"
Rails.logger.level = Logger::INFO
Rails.logger.info "Changed log_level to INFO"
when "DEBUG"
Rails.logger.level = Logger::DEBUG
Rails.logger.info "Changed log_level to DEBUG"
end
File.delete file_path
end
end
Then, log level can be changed by creating a file under tmp/change_log_level.PID, where pid is the process id of the rails process. You can create a rake/capistrano task to detect and create these files, allowing you to quickly switch log level of your running production server.
Just remember to start rufus in the worker threads, if you are using unicorn or similar.

Related

How to start and stop multiple weblogic managed servers at one go through WLST

I am writing a code to start , stop, undeploy and deploy my application on weblogc.
My components need to be deployed on few managed servers.
When I do new deployments manually I can start and stop the servers in parallel, by ticking multiple boxes and selecting start and stop from the dop down. See below.
but when trying from WLST, i could do that in one server at a time.
ex:
start(name='ServerX',type='Server',block='true')
start(name='ServerY',type='Server',block='true')
shutdown(name='ServerX',entityType='Server',ignoreSessions='true',timeOut=600,force='true',block='true')
shutdown(name='ServerY',entityType='Server',ignoreSessions='true',timeOut=600,force='true',block='true')
Is there a way I can start stop multiple servers in once command?
Instead of directly starting and stopping servers, you create tasks, then wait for them to complete.
e.g.
tasks = []
for server in cmo.getServerLifeCycleRuntimes():
# to shut down all servers
if (server.getName() != ‘AdminServer’ and server.getState() != ‘RUNNING’ ):
tasks.append(server.start())
#or to start them up:
#if (server.getName() != ‘AdminServer’ and server.getState() != ‘SHUTDOWN’ ):
# tasks.append(server.shutdown())
#wait for tasks to complete
while len(tasks) > 0:
for task in tasks:
if task.getStatus() != ‘TASK IN PROGRESS’ :
tasks.remove(task)
java.lang.Thread.sleep(5000)
I know this is an old post, today I was reading this book "Advanced WebLogic Server Automation" written by Martin Heinzl so in the page 282 I found this.
def startCluster(clustername):
try:
start(clustername, 'Cluster')
except Exception, e:
print 'Error while starting cluster', e
dumpStack()
I tried it and it started managed servers in parallel.
Just keep in mind the AdminServer must be started first and your script must connect to the AdminServer before trying it.
Perhaps this would not be useful for you as the servers should be in a cluster, but I wanted to share this :)

Sidekiq stop one single, running job

So I need to stop a running Job in Sidekiq (3.1.2) programmatically, not a scheduled one. I did read the API documentation but didn't really find anything about cancelling running jobs. Is this possible with sidekiq?
When this is not directly possible, my idea was to circumvent this, by raising an exception in the job when I call the signal, then deleting the job from the retryset. This is clearly not optimal though.
Thanks in advance
Correct, the only way to stop a job is for the job to stop itself. Your application must implement that logic.
https://github.com/mperham/sidekiq/wiki/FAQ#how-do-i-cancel-a-sidekiq-job
If you know the long running job's Thread ID, its possible to terminate it from another task:
class ThreadLightly
include Sidekiq::Worker
def perform(tid)
puts "I'm %s, and I'll be terminating TID: %s..." % [self.class, tid]
Thread.list.each {|t|
if t.object_id.to_s == tid
puts "Goodbye %s!" % t
t.exit
end
}
end
end
You can trigger it from the sidekiq_pusher:
bundle exec ./pusher.rb ThreadLightly $YOURJOBSTHREADID
You'll need to log the Thread.current.object_id from each job since the UI dosn't show it. Also, if you run distributed sidekiqs, you'll need to run this task until it runs on the same instance.

Rails 3.2.2 log files unordered, requests intertwined

I recollect getting log files that were nicely ordered, so that you could follow one request, then the next, and so on.
Now, the log files are, as my 4 year old says "all scroggled up", meaning that they are no longer separate, distinct chunks of text. Loggings from two requests get intertwined/mixed up.
For instance:
Started GET /foobar
...
Completed 200 OK in 2ms (Views: 0.4ms | ActiveRecord: 0.8ms)
Patient Load (wait, that's from another request that has nothing to do with foobar!)
[ blank space ]
Something else
This is maddening, because I can't tell what's happening within one single request.
This is running on Passenger.
I tried to search for the same answer but couldn't find any good info. I'm not sure if you should fix server or rails code.
If you want more info about the issue here is the commit that removed old way of logging https://github.com/rails/rails/commit/04ef93dae6d9cec616973c1110a33894ad4ba6ed
If you value production log readability over everything else you can use the
PassengerMaxInstancesPerApp 1
configuration. It might cause some scaling issues. Alternatively you could stuff something like this in application.rb:
process_log_filename = Rails.root + "log/#{Rails.env}-#{Process.pid}.log"
log_file = File.open(process_log_filename, 'a')
Rails.logger = ActiveSupport::BufferedLogger.new(log_file)
Yep!, they have made some changes in the ActiveSupport::BufferedLogger so it is not any more waiting until the request has ended to flush the logs:
http://news.ycombinator.com/item?id=4483390
https://github.com/rails/rails/commit/04ef93dae6d9cec616973c1110a33894ad4ba6ed
But they have added the ActiveSupport::TaggedLogging which is very funny and you can stamp every log with any kind of mark you want.
In your case could be good to stamp the logs with the request UUID like this:
# config/application.rb
config.log_tags = [:uuid]
Then even if the logs are messed up you still can follow which of them correspond to the request you are following up.
You can make more funny things with this feature to help you in your logs study:
How to log user_name in Rails?
http://zogovic.com/post/21138929607/running-time-in-rails-logs
Well, for me the TaggedLogging solution is a no go, I can live with some logs getting lost if the server crashes badly, but I want my logs to be perfectly ordered. So, following advice from the issue comments I'm applying this to my app:
# lib/sequential_logs.rb
module ActiveSupport
class BufferedLogger
def flush
#log_dest.flush
end
def respond_to?(method, include_private = false)
super
end
end
end
# config/initializers/sequential_logs.rb
require 'sequential_logs.rb'
Rails.logger.instance_variable_get(:#logger).instance_variable_get(:#log_dest).sync = false
As far as I can say this hasn't affected my app, it is still running and now my logs make sense again.
They should add some quasi-random reqid and write it in every line regarding one single request. This way you won't get confused.
I haven't used it, but I believe Lumberjack's unit_of_work method may be what you're looking for. You call:
Lumberjack.unit_of_work do
yield
end
And all logging done either in that block or in the yielded block are tagged with a unique ID.

how do I get all my rake tasks to write to the same log file?

I have two rake tasks that I'd like to run nightly. I'd like them to log to one file. I thought this would do the trick (got it here: Rake: logging any task executing):
application.rb
module Rake
class Task
alias_method :origin_invoke, :invoke if method_defined?(:invoke)
def invoke(*args)
#logger = Logger.new('rake_tasks_log.log')
#logger.info "#{Time.now} -- #{name} -- #{args.inspect}"
origin_invoke(args)
end
end
end
and then in the rakefile:
task :hello do
#logger.warn "Starting Hello task"
puts "Hello World!"
puts "checking connection "
checkConnection
puts "done checking"
#logger.debug "End hello rake task"
end
But when I run the task I get:
private method 'warn' called for nil:NilClass
I've tried a couple of flavors of that call to logging (#, ##, no #) to no avail. Read several threads on here about it. The
rubyonrails.org site doesn't mention logging in rake tasks. The tasks that I'm invoking are fairly complex (about 20-40 mins to complete) so I'll really want to know what went wrong if they fail. I'd prefer for DRY reasons to only create the logger object once.
Unless you're wrapping everything in giant begin/rescue's and catching errors that way, the best way to log errors is to catch all output from stderr and stdout with something like:
rake your:job 2>&1 >> /var/log/rake.log
You could also set your Rails environment to use the system logger as well.
I ended up solving this (or at least well enough) by making a "log" task and depending on that in other tasks. Not really ideal, since that means having to include that dependency in any new task, but I have only a few tasks so this will do fine. I'm aware that there is a "file" task but it didn't seem to want to work in Windows, so I chose this because it seems to be more cross platform and it's more explicit.
I need a logger object because I am passing that object into some method calls in the [...] sections. There's enough begin/rescue/end in there that writing to the output stream wouldn't work (I think).
#log_file = "log/tasks.log"
directory "log"
task :check_log => ["log"] do
log = #log_file
puts 'checking log existence'
if not FileTest.exists? ("./#{log}")
puts 'creating log file'
File.open(log, 'w')
end
end
task :check_connection => [:check_log] do
begin
conn = Mongo::Connection.new
[...]
end
end
task :nightly_tasks => [:check_connection, :environment ] do
for i in 1..2
logger.warn "#########################"
end
[...]
logger.warn "nightly tasks complete"
end
def logger
##logger ||= Logger.new( File.join(Rails.root, #log_file) )
end

Is it possible to terminate an already running delayed job using Ruby Threading?

Let's say I have delayed_job running in the background. Tasks can be scheduled or run immediately(some are long tasks some are not)
If a task is too long, a user should be able to cancel it. Is it possible in delayed job? I checked the docs and can't seem to find a terminate method or something. They only provide a catch to cancel delayed job itself(thus cancelling all tasks...I need to just cancel a certain running task)
UPDATE
My boss(who's a great programmer btw) suggested to use Ruby Threading for this feature of ours. Is this possible? Like creating new threads per task and killing that thread while it's running?
something like:
t1 = Thread.new(task.run)
self.delay.t1.join (?) -- still reading on threads so correct me if im wrong
then to stop it i'll just use t1.stop (?) again don't know yet
Is this possible? Thanks!
It seems that my boss hit the spot so here's what we did(please tell us if there's some possibility this is bad practice so I can bring it up):
First, we have a Job model that has def execute! (which runs what it's supposed to do).
Next, we have delayed_job worker in the background, listening for new jobs. Now when you create a job, you can schedule it to run immediately or run every certain day (we use rufus for this one)
When a job is created, it checks if its supposed to run immediately. If it is, it adds itself to the delayed job queue. The execute function creates a Thread, so each job has its own thread.
User in the ui can see if a job is running(if there's a started_at and no finished_at). If it IS running, there's a button to cancel it. Canceling it just sets the job's canceled_at to Time.now.
While the job is running it also checks itself if it has a canceled_at or if Time.now is > finished_at. If so, kill the thread.
Voila! We've tested it for one job and it seems to work. Now the only problem is scaling...
If you see any problems with this please do so in the comments or give more suggestions if ever :) I hope this helps some one too!
Delayed::Job is an < ActiveRecord::Base model, so you can query it just like you normally would like Delayed::Job.all(:conditions => {:last_error => nil}).
Delayed::Job objects have a payload field which contain a serialized version of the method or job that you're attempting to run. This object is accessed by their '#payload_object' method, which loads the object in question.
You can combine these two capabilities to make queriable job workers, for instance, if you have a User model, and the user has a paperclip'ed :avatar, then you can make a method to delete unprocessed jobs like so:
class User < ActiveRecord::Base
has_attached_file :avatar, PaperclipOptions.new(:avatar)
before_create :'process_avatar_later'
def process_avatar_later
filename = Rails.root.join('tmp/avatars_for_processing/',self.id)
open(filename, 'w') do |file| file <<self.avatar.to_file end
Delayed::Job.enqueue(WorkAvatar.new(self.id, filename))
self.avatar = nil
end
def cancel_future_avatar_processing
WorkAvatar.future_jobs_for_user(self.id).each(&:destroy)
#ummm... tell them to reupload their avatar, I guess?
end
class WorkAvatar < Struct.new(:user_id, :path)
def user
#user ||= User.find(self.user_id)
end
def self.all_jobs
Delayed::Job.scoped(:conditions => 'payload like "%WorkAvatar%"')
end
def self.future_jobs_for_user(user_id)
all_jobs.scoped(:conditions => {:locked_at => nil}).select do |job|
job.payload_object.user_id == user_id
end
end
def perform
#user.avatar = File.open(path, 'rb')
#user.save()
end
end
end
It's possible someone has made a plugin make queryable objects like this. Perhaps searching on GitHub would be fruitful.
Note also that you'd have to work with any process monitoring tools you might have to cancel any running job worker processes that are being executed if you want to cancel a job that has locked_at and locked_by set.
You can wrap the task into a Timeout statement.
require 'timeout'
class TaskWithTimeout < Struct.new(:parameter)
def perform
Timeout.timeout(10) do
# ...
end
rescue Timeout::Error => e
# the task took longer than 10 seconds
end
end
No, there's no way to do this. If you're concerned about a runaway job you should definitely wrap it in a timeout as Simone suggests. However, it sounds like you're in search of something more but I'm unclear on your end goal.
There will never be a way for a user to have a "cancel" button since this would involve finding a method to directly communicate with the worker running process running the job. It would be possible to add a signal handler to the worker so that you could do something like kill -USR1 pid to have it abort the job it's currently working and move on. Would this accomplish you goal?