My Rails app runs on Heroku using Resque backed by RedisCloud.
Somehow Resque has gotten totally hosed. A few days ago it stopped showing what was currently working (Overview and Working tabs on resque web). I tried to solve it by "cleaning up" what appeared to me to be stale keys in Redis.
Bad move! Now, failed jobs stopped showing up on the Failed tab (and no failed key in Redis).
At this point I would like to do a clean reset, but how? I basically wiped all keys, following the process of this gem: resque-reset. What else can be done? Where is state kept besides Redis? Of course I also did heroku restart.
But it's still hosed. That is, resque web never shows what's working or what's failed.
Related
In the configuration file example for Puma, it says the following for the on_restart function:
Code to run before doing a restart. This code should close log files,
database connections, etc.
Do I need to implement this for a Rails app, to close connections to the db and the logfile, or is that taken care of automatically? If not, how do I actually do all that?
No you don't, Rails takes care of reloading your code automatically. But this code reloading support is limited. For example changes to application.rb are not applied until you restart the app server.
But I would recommend Phusion Passenger over Puma. Phusion Passenger is a lot easier to setup, especially when you hit production. Phusion Passenger integrates into Apache and Nginx directly and provides advanced features like dynamic worker management. Phusion Passenger is very mature, stable and performant and used by the likes of New York Times, Symantec, AirBnB, etc.
I've found that using Redis as my Rails.cache provider causes an error page upon the first request every time my Rails/Puma server is restarted. The error I got was:
Redis::InheritedError (Tried to use a connection from a child process
without reconnecting. You need to reconnect to Redis after forking.)
To get around this error, I didn't add anything to on_restart, but did have to add code to on_worker_boot ( I am running Puma with workers=4 ):
puma-config.rb
on_worker_boot do
puts "Reconnecting Rails.cache"
Rails.cache.reconnect
end
I use the delayed job gem to handle my email deliveries. It is working fine in the development and I am very happy with it. However after I deployed to the server, when I use command:
RAILS_ENV=production script/delayed_job start
it will be working. I've checked the log file and database, everything is fine and I can receive the mails just as I expected. However, when I exit from the server, nothing is going to happen.
I've checked my database by using sequel pro and seen that the delayed job has created a row in the DB and after the time in the run_at column, the row would disappear, but no mails can be received. When I log in again, the delayed job process is still running, and the log is nothing strange, but I just cannot receive and email that I suppose to. I can't keep my self log in all the time. Without the delayed job, I can use the traditional way and it's working properly but slow. Why the delayed job failed after I log out of the server?
This is my delayed job setting in the config/initializers/delay_job.rb
require "bcrypt"
Delayed::Worker.max_attempts = 5
Delayed::Worker.delay_jobs = !Rails.env.test?
Delayed::Worker.destroy_failed_jobs = false
P.S. I am not sure is it anything to do with the standalone passenger as I have to use different version of rails so I have to use a standalone passenger with port 3002.
I think I've found the solution.
After reading through this https://github.com/collectiveidea/delayed_job/wiki/Common-problems#wiki-jobs_are_silently_removed_from_the_database
I soon realized I might miss the "require bcrypt" in the configuration file.
I use RVM and have many gemsets, but just this particular gemset has the gem bcrypt-ruby. The delayed job might use the global or default gemset after I log out the system, so I install bcrypt-ruby in all the gemsets and restart the standalone passenger and it works!.
But still, I dont really know the connection between bcrypt and the delayed job.
We use Foreman to start all of our web processes in development.
A while back, I tried to get the ruby-debugger gem working with this setup, but I couldn't, so I abandoned my effort. Along the way, I must have changed some setting or another, and now when I try to look at the server log in real time when I make a request to my local environment, nothing gets printed out. I have to kill foreman in order to see any output from the request.
This is really slowing down my development, as I have to make a request, kill foreman to get information about what went wrong, then start up and try again.
Any ideas how to get my server log to spit out everything as I'm making requests?
I had the same problem. It's solved here:
It's as simple as adding a line
$stdout.sync = true
To your config/environments/development.rb file, then restarting foreman.
Worked for me and makes my life much easier.
I am getting a 404 error when trying to push my database to Heroku via Taps
(1.9.2#[app_name]_db) heroku db:push --app [app_name]
Loaded Taps v0.3.24
Auto-detected local database: sqlite://db/development.sqlite3
Warning: Data in the app '[app-name]' will be overwritten and will not be recoverable.
! WARNING: Destructive Action
! This command will affect the app: [app-name]
! To proceed, type "[app-name]" or re-run this command with --confirm [app-name]
> [app-name]
Sending schema
Schema: 0% | | ETA: --:--:--
Saving session to push_201209251425.dat..
!!! Caught Server Exception
HTTP CODE: 404
The db:push command used to work fine, then I made some changes to my database by rolling back the migrations, editing them, and then re-migrating. Now I can deploy the app just fine, but the database will not push -- I don't know if this is related to editing the migrations or not.
The app works fine on my machine, and I wanted to eliminate any discrepancies between Heroku's copy and my own, so I created a new app and pushed to that. Same thing: the Heroku app works but will not receive db:push; it errors out with the same 404 above.
Is this a Heroku service temporarily down, or has changing my app caused the 404?
Edit: heroku logs do not show any error message
Heroku support was taking too long to respond, so I found a workaround that communicates with my EC2 instance directly by using the Taps gem.
Go to Heroku dashboard for your database. For me this was at
https://postgres.heroku.com/databases/[my-database-name]
though I navigated by going through Addons.
Click on 'URL' in 'Connection Settings', should give you something like
postgres://[username]:[password]#ec2-[ip_address_numbers].compute-1.amazonaws.com:[port]/[database_name]
Copy this value down, I'll reference it here as [EC2_URL]
Get Taps installed on 1.9.2 gemset if you don't already have it (not sure if 1.9.3 will work, didn't test it)
Set up localhost taps server to facilitate transaction by running in terminal:
taps server postgres://[local_machine_username]#localhost/[name_from_database.yml] [some_username] [some_password]
(note the spaces before username and password)
Then you can process the transaction yourself through another terminal window:
taps pull [EC2_URL] http://[some_username]:[some_password]#localhost:5000
It should run and pull all your data from the local development db to the Amazon instance. You can also do vice versa, or choose a different database, etc. Or not, I'm not a cop.
There are some problems with heroku db commands and ruby 1.9.2 (I have this version).
db:pull ends with "Unable to fetch tables information from"
db:push ends with "!!! Caught Server Exception HTTP CODE: 404"
There is a work-around for this problem. Switch to ruby 1.8.7 (I am using rvm for this) for a moment just to do db operations on heroku and after finish switch ruby back.
I do the same process (have Heroku convert my sqlite database to Postgres), and I was getting this problem yesterday as well. It seems to be working now, so I'm believe it was an issue with Heroku.
I have a worker doing some processing 24/7. However, sometimes the code crashes and it needs to be restarted (even if I catch the exception, I have to restart the worker in order for it to work).
What do you do when this happens or am I doing something wrong and this shouldn't happen at all? Does your dynos/workers crash or it is just me?
thanks
Heroku is supposed to restart a worker every time it crashes. As far as I know, you don't have to select or configure anything. Whatever is in your jobs:work task will be executed as soon as it fails.
In the event that you are heavily dependent on background jobs in your web app. You could create a rake task that finds the last record to be updated and execute a background job to update it. Or perhaps automate the rake task to find the rest of the records that need updating, since the last crash.
Alternatively, you force worker restart manually as indicated in this article (using delayed_job):
heroku workers 0;
heroku workers 1;
Or perhaps you can restart a specific worker by doing (mentioned in this article):
heroku restart worker.1
By the way, try the 1.9 stack. Make sure your app is 1.9.2 compatible, before doing so. Hopefully crashes are less frequent there:
heroku stack:migrate bamboo-mri-1.9.2
In the event, that such issues still arise. Best to contact Heroku support. They are very responsive at what they do.
Latest command to restart a specific heroku web worker (2014):
heroku ps:restart web.1
(tested on Cedar stack)
At times, for instance in case of DB crashes, the worker may not restart automatically. you would need to do this.
heroku restart web.1
It worked for me.