I'm using the Spawn gem in a rails 3 app - it's the rails3-adapted fork at https://github.com/rfc2822/spawn
My app is deployed on heroku, and when i tried to spawn i get this failure:
app[web.1]: ### ../controllers/messages_controller.rb:10:in `create_message': About to spawn
app[web.1]: spawn> parent PID = 1
app[web.1]: spawn> child PID = 49
app[web.1]: ### ../controllers/messages_controller.rb:17:in `create_message': After spawn
app[web.1]: Task Load (1.2ms) SELECT "tasks".* FROM "tasks" WHERE "tasks"."id" = 80 LIMIT 1
app[web.1]: PGError: server closed the connection unexpectedly
app[web.1]: This probably means the server terminated abnormally
app[web.1]: before or while processing the request.
I have this option in my config/database.yml, following the recommendation of the spawn documentation:
reconnect: true
Is it connected to this do you think?
Bit at a loss with this... before i go investigating, does anyone know what's causing this?
cheers, max
I ended up using the girl_friday gem instead, which is a simple forked-queue system. It worked great for me.
Related
We recently started having trouble with chef-client dying in the middle of a run after taking a lot more time stuck on various parts of the run-list that normally proceeded much quicker. I've been on my home wifi and my colleague has been on the work wifi, which has been having some connectivity problems of its own.
If your ssh connection gets interrupted to a machine while chef-client is running, does that crash the run in seemingly inexplicable ways? I am using PutTY to connect from my Win7 and my colleague is using the Apple Terminal App.
All the machines we've been running this on are Ubuntu 12.04 (in EC2) and have plenty of disk space left over - they're only utilizing ~1GB with ~5GB free.
Here is the output of the log from /var/log/chef/client.log (set with the log_location directive in /etc/chef/client.rb as described here).
[2014-01-08T00:27:07+00:00] WARN: Nodejs user is nodejs
[2014-01-08T00:27:07+00:00] WARN: Cloning resource attributes for group[nodejs] from prior resource (CHEF-3694)
[2014-01-08T00:27:07+00:00] WARN: Previous group[nodejs]: /var/chef/cache/cookbooks/nodejs/recipes/default.rb:26:in `from_file'
[2014-01-08T00:27:07+00:00] WARN: Current group[nodejs]: /var/chef/cache/cookbooks/spicoli-app/recipes/default.rb:38:in `from_file'
[2014-01-08T00:27:07+00:00] WARN: Cloning resource attributes for user[nodejs] from prior resource (CHEF-3694)
[2014-01-08T00:27:07+00:00] WARN: Previous user[nodejs]: /var/chef/cache/cookbooks/nodejs/recipes/default.rb:34:in `from_file'
[2014-01-08T00:27:07+00:00] WARN: Current user[nodejs]: /var/chef/cache/cookbooks/spicoli-app/recipes/default.rb:46:in `from_file'
[2014-01-08T00:27:30+00:00] WARN: Environment is _default
[2014-01-08T00:27:30+00:00] WARN: Nodejs user is nodejs
[2014-01-08T02:04:54+00:00] ERROR: Running exception handlers
[2014-01-08T02:04:54+00:00] ERROR: Exception handlers complete
[2014-01-08T02:04:54+00:00] FATAL: Stacktrace dumped to /var/chef/cache/chef-stacktrace.out
[2014-01-08T02:04:55+00:00] ERROR: Input/output error - <STDOUT>
[2014-01-08T02:04:57+00:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)
And the error stacktrace just has this:
Generated at 2014-01-08 02:04:54 +0000
Errno::EIO: Input/output error - <STDOUT>
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.8.0/lib/chef/formatters/base.rb:91:in `write'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.8.0/lib/chef/formatters/base.rb:91:in `puts'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.8.0/lib/chef/formatters/base.rb:91:in `puts'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.8.0/lib/chef/formatters/error_descriptor.rb:61:in `display_section'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.8.0/lib/chef/formatters/error_descriptor.rb:44:in `block (2 levels) in display'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.8.0/lib/chef/formatters/error_descriptor.rb:43:in `each'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.8.0/lib/chef/formatters/error_descriptor.rb:43:in `block in display'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.8.0/lib/chef/formatters/error_descriptor.rb:42:in `each'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.8.0/lib/chef/formatters/error_descriptor.rb:42:in `display'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.8.0/lib/chef/formatters/base.rb:130:in `display_error'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.8.0/lib/chef/formatters/base.rb:161:in `resource_failed'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.8.0/lib/chef/formatters/doc.rb:159:in `resource_failed'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.8.0/lib/chef/event_dispatch/dispatcher.rb:29:in `block in resource_failed'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.8.0/lib/chef/event_dispatch/dispatcher.rb:29:in `each'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.8.0/lib/chef/event_dispatch/dispatcher.rb:29:in `resource_failed'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.8.0/lib/chef/resource.rb:637:in `rescue in run_action'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.8.0/lib/chef/resource.rb:643:in `run_action'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.8.0/lib/chef/runner.rb:49:in `run_action'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.8.0/lib/chef/runner.rb:81:in `block (2 levels) in converge'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.8.0/lib/chef/runner.rb:81:in `each'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.8.0/lib/chef/runner.rb:81:in `block in converge'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.8.0/lib/chef/resource_collection.rb:98:in `block in execute_each_resource'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.8.0/lib/chef/resource_collection/stepable_iterator.rb:116:in `call'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.8.0/lib/chef/resource_collection/stepable_iterator.rb:116:in `call_iterator_block'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.8.0/lib/chef/resource_collection/stepable_iterator.rb:85:in `step'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.8.0/lib/chef/resource_collection/stepable_iterator.rb:104:in `iterate'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.8.0/lib/chef/resource_collection/stepable_iterator.rb:55:in `each_with_index'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.8.0/lib/chef/resource_collection.rb:96:in `execute_each_resource'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.8.0/lib/chef/runner.rb:80:in `converge'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.8.0/lib/chef/client.rb:433:in `converge'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.8.0/lib/chef/client.rb:500:in `do_run'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.8.0/lib/chef/client.rb:199:in `block in run'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.8.0/lib/chef/client.rb:193:in `fork'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.8.0/lib/chef/client.rb:193:in `run'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.8.0/lib/chef/application.rb:208:in `run_chef_client'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.8.0/lib/chef/application/client.rb:312:in `block in run_application'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.8.0/lib/chef/application/client.rb:304:in `loop'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.8.0/lib/chef/application/client.rb:304:in `run_application'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.8.0/lib/chef/application.rb:66:in `run'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.8.0/bin/chef-client:26:in `<top (required)>'
/usr/bin/chef-client:23:in `load'
/usr/bin/chef-client:23:in `<main>'
Which is a really generic error! But it does seem to indicate an interruption to STDOUT output, which kind of makes sense with a client disconnection.
Edit: As requested, here are the contents of the client.rb file (names obfuscated, naturally.)
$ cat /etc/chef/client.rb
log_level :auto
log_location "/var/log/chef/client.log"
chef_server_url "https://api.opscode.com/organizations/myapp"
validation_client_name "my-validator"
node_name "my-app-node"
Edit 2: Attempt using sudo su -s /bin/bash root -c "screen chef-client"
Screen terminated while I was at lunch and recorded a timeout on the ShellOut command for npm install. This was after chef-client was sitting stuck on this operation for over an hour.
[2014-01-09T16:39:07+00:00] WARN: Environment is _default
[2014-01-09T16:39:07+00:00] WARN: Nodejs user is nodejs
[2014-01-09T18:16:28+00:00] ERROR: Running exception handlers
[2014-01-09T18:16:28+00:00] ERROR: Exception handlers complete
[2014-01-09T18:16:28+00:00] FATAL: Stacktrace dumped to /var/chef/cache/chef-stacktrace.out
[2014-01-09T18:16:31+00:00] ERROR: execute[npm-install-app] (spicoli-app::default line 110) had an error: Mixlib::ShellOut::CommandTimeout: command timed out:
---- Begin output of npm --registry http://my.npm.repo.amazonaws.com:5984/registry/_design/app/_rewrite install --cache /home/nodejs/.npm --tmp /home/nodejs/tmp
--- snip: install messages from npm ---
[2014-01-09T18:16:33+00:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)
This is a totally different error than before. The stacktrace.out file also explicitly mentions ShellOut, so it is entirely different as well. Most oddly, when I run the same npm command from the command line, in finishes in under a minute.
So I'm not sure there is a way to further diagnose the previous failure, but I would welcome other suggestions. For input on this new failure, I asked this followup question.
If your ssh connection gets interrupted to a machine while chef-client is running, does that crash the run in seemingly inexplicable ways?
Well, the stacktrace seems to imply that something like that is happening. The message says "Errno::EIO: Input/output error - <STDOUT>" which is consistent with what I'd expect to see if STDOUT was going over an SSH channel that had been closed.
I suggest 2 things:
Run chef-client with all console output redirected to a file; e.g. add > /tmp/log 2>&1 to the end of the command. (The redirection needs to happen on the remote machine.)
Add -l debug to the command to increase the level of logging, as covered in Opscode's technical FAQ. This could reveal clues that are currently being hidden.
Looking at your second update, this has the hallmarks of some kind of firewall or network related problem.
My Ruby on Rails app was running smoothly until yesterday.
When ever I try to open my Heroku App it gives me a "Application Error" message and when I check my logs I get this message below.
[36m2013-09-10T17:42:34.393159+00:00 app[web.1]: [0m Connecting to database
specified by DATABASE_URL
[36m2013-09-10T17:42:34.846457+00:00 app[web.1]: [0m Exiting
[36m2013-09-10T17:42:34.849786+00:00 app[web.1]: [0m /app/vendor/bundle/ruby/2.
0.0/gems/devise-3.1.0/lib/devise/rails/routes.rb:440:in `raise_no_secret_key':
Devise.secret_key was not set. Please add the following to your Devise
initializer: (RuntimeError)
Has anyone encountered this? What does the error mean?
A discussion about that error here:
https://github.com/plataformatec/devise/issues/2554
Follow the error log instructions in your post. Add the following to your Devise initializer:
config.secret_key = '-- secret key --'
I have a Rails app running in heroku.
I wanna watermark an original AUDIO-file with another AUDIO-file through a merging/mixing (amerge or amix) filter. Also fades and looping should work.
FFMPEG must be compiled for this feature to run on heroku.
My steps I got from this GIST which uses Vuclan
https://gist.github.com/liufengyun/5055354
Locally I use the following term to convert with Carrierwave which is an uploader:
"-i Original.aiff -i public/watermark/#{ENV['WATERMARK_FILENAME']}.aiff -loop 1 -filter_complex '[1]afade=t=in:ss=0:d=1[4];[4]afade=t=out:st=#{self.evaluate_length-2}:d=2[5];[5][0]amix=duration=shortest[out]' -map [out]"
This works with the ffmpeg built in on my mac.
My compiled version for heroku does contain amerge, amix and the libavformat library.
This is the log which I got from heroku when I go to my upload form and uplaod a file:
(all this works, but only locally).
Started POST "/sounds" for 84.113.97.32 at 2013-07-15 21:14:33 +0000
2013-07-15T21:14:33.175631+00:00 app[web.1]: Running transcoding...
2013-07-15T21:14:33.175631+00:00 app[web.1]: ffmpeg -y -i /tmp/sounds/1373922873-2-5714/watermark_loop-95.aiff -i public/watermark/1khz.aiff -loop 1 -filter_complex '[1]afade=t=in:ss=0:d=1[4];[4]afade=t=out:st=-2.0:d=2[5];[5][0]amix=duration=shortest[out]' -map [out] /tmp/sounds/1373922873-2-5714/watermark_loop-95.aiff
2013-07-15T21:14:33.175631+00:00 app[web.1]:
2013-07-15T21:14:33.241221+00:00 app[web.1]: Failed encoding...
2013-07-15T21:14:33.241221+00:00 app[web.1]: ffmpeg -y -i /tmp/sounds/1373922873-2-5714/watermark_loop-95.aiff -i public/watermark/1khz.aiff -loop 1 -filter_complex '[1]afade=t=in:ss=0:d=1[4];[4]afade=t=out:st=-2.0:d=2[5];[5][0]amix=duration=shortest[out]' -map [out] /tmp/sounds/1373922873-2-5714/watermark_loop-95.aiff
2013-07-15T21:14:33.241221+00:00 app[web.1]:
2013-07-15T21:14:33.241221+00:00 app[web.1]: ffmpeg: error while loading shared libraries: libavformat.so.54: cannot open shared object file: No such file or directory
2013-07-15T21:14:33.241221+00:00 app[web.1]:
2013-07-15T21:14:33.241221+00:00 app[web.1]: Errors: encoded file is invalid.
2013-07-15T21:14:33.241221+00:00 app[web.1]:
2013-07-15T21:14:33.277487+00:00 app[web.1]:
2013-07-15T21:14:33.277487+00:00 app[web.1]: FFMPEG::Error (Failed encoding.Errors: encoded file is invalid. Full output: ffmpeg: error while loading shared libraries: libavformat.so.54: cannot open shared object file: No such file or directory
2013-07-15T21:14:33.277487+00:00 app[web.1]: ):
2013-07-15T21:14:33.277487+00:00 app[web.1]: app/uploaders/sound_uploader.rb:39:in `watermarking'
2013-07-15T21:14:33.277487+00:00 app[web.1]:
It's my 6th attempt to compile, get ffmpeg running correctly and watermarking my sounds on heroku.
Please help me out of this neverending story.
I have forgotten to add all subpaths to the LD_LIBRARY_PATH
So I changed my heroku variable from
LD_LIBRARY_PATH: vendor/ffmpeg/lib:/usr/local/lib
to
LD_LIBRARY_PATH: vendor/x264/lib:vendor/mp3lame/lib:vendor/ffmpeg/lib:/usr/local/lib
via
heroku config:set LD_LIBRARY_PATH=vendor/x264/lib:vendor/mp3lame/lib:vendor/ffmpeg/lib:/usr/local/lib
I have made a gist with detailed compiling instructions for FFMPEG on HEROKU
https://gist.github.com/klangfeld/6009744
We have built an open-source, free Google Reader Alternative and have live users using the platform. It is hosted at http://reader.pykih.com and code is at http://github.com/pykih/reader
Until, last night my app was working fine and today morning it is down due to timeout.
We use DelayedJobs for all feed imports
This timeout is occurring on the homepage where there is no external service call, emails being set, etc. except may be push to GA
Heroku Support (as per them) is asleep right now and we will get an answer only on Monday. :-(
We went through a lot of past questions on Stack over flow about this issue but none seem to answer our issue.
Can someone please advice?
2013-06-08T04:25:32.596864+00:00 app[worker.1]: [Worker(host:b0f05380-0265-49de-b2c1-df13df269cd6 pid:2)] Starting job worker
2013-06-08T04:25:34.225938+00:00 app[web.1]: Disconnected from ActiveRecord
2013-06-08T04:25:34.225544+00:00 app[web.1]: I, [2013-06-08T04:25:34.225214 #2] INFO -- : listening on addr=0.0.0.0:33707 fd=7
2013-06-08T04:25:34.250988+00:00 heroku[web.1]: State changed from starting to up
2013-06-08T04:25:35.246195+00:00 app[web.1]: Disconnected from ActiveRecord
2013-06-08T04:25:35.254585+00:00 app[web.1]: Connected to ActiveRecord
2013-06-08T04:25:35.375559+00:00 app[web.1]: I, [2013-06-08T04:25:35.375194 #5] INFO -- : worker=0 ready
2013-06-08T04:25:36.263576+00:00 app[web.1]: Disconnected from ActiveRecord
2013-06-08T04:25:36.273848+00:00 app[web.1]: Connected to ActiveRecord
2013-06-08T04:25:36.390283+00:00 app[web.1]: I, [2013-06-08T04:25:36.389861 #8] INFO -- : worker=1 ready
2013-06-08T04:25:37.281171+00:00 app[web.1]: I, [2013-06-08T04:25:37.280925 #2] INFO -- : master process ready
2013-06-08T04:25:37.291365+00:00 app[web.1]: Connected to ActiveRecord
2013-06-08T04:25:37.416442+00:00 app[web.1]: I, [2013-06-08T04:25:37.415844 #11] INFO -- : worker=2 ready
2013-06-08T04:25:46.993985+00:00 app[web.1]: Started GET "/" for 120.63.8.230 at 2013-06-08 04:25:46 +0000
2013-06-08T04:26:16.919340+00:00 heroku[router]: at=error code=H12 desc="Request timeout" method=GET path=/ host=reader.pykih.com fwd="120.63.8.230" dyno=web.1 connect=2ms service=30000ms status=503 bytes=0
2013-06-08T04:26:23.974357+00:00 app[web.1]: E, [2013-06-08T04:26:17.347459 #2] ERROR -- : worker=0 PID:5 timeout (31s > 30s), killing
2013-06-08T04:26:23.975290+00:00 app[web.1]: E, [2013-06-08T04:26:17.376903 #2] ERROR -- : reaped #<Process::Status: pid 5 SIGKILL (signal 9)> worker=0
2013-06-08T04:26:23.975290+00:00 app[web.1]: Disconnected from ActiveRecord
2013-06-08T04:26:23.975611+00:00 app[web.1]: Connected to ActiveRecord
2013-06-08T04:26:23.975611+00:00 app[web.1]: I, [2013-06-08T04:26:18.524590 #14] INFO -- : worker=0 ready
Heroku fixed itself 11 hours later. V annoying.
Although read many articles about this problem, I still couldn't solve those error,
Here is a part of the heroku logs, if it's less-information, I would update others code of the red mine source code.
2012-11-05T14:02:54+00:00 app[web.1]: Plugins in vendor/plugins (/app/vendor/plugins) are no longer allowed. Please, put your Redmine plugins in the `plugins` directory at the root of your Redmine directory (/app/plugins)
2012-11-05T14:02:54+00:00 app[web.1]: => Booting WEBrick
2012-11-05T14:02:54+00:00 app[web.1]: => Rails 3.2.8 application starting in production on http://0.0.0.0:40115
2012-11-05T14:02:54+00:00 app[web.1]: => Call with -d to detach
2012-11-05T14:02:54+00:00 app[web.1]: => Ctrl-C to shutdown server
2012-11-05T14:02:54+00:00 app[web.1]: Exiting
2012-11-05T14:02:55+00:00 heroku[web.1]: Process exited with status 1
2012-11-05T14:02:55+00:00 heroku[web.1]: State changed from starting to crashed
2012-11-05T14:02:56+00:00 heroku[router]: Error H10 (App crashed) -> GET ceciits.herokuapp.com/ dyno= queue= wait= service= status=503 bytes=
2012-11-05T14:06:04+00:00 heroku[web.1]: State changed from crashed to down
2012-11-05T14:06:04+00:00 heroku[web.1]: Unidling
2012-11-05T14:06:04+00:00 heroku[web.1]: State changed from down to starting
2012-11-05T14:06:07+00:00 heroku[web.1]: Starting process with command `bundle exec rails server -p 3103`
2012-11-05T14:06:12+00:00 app[web.1]: Plugins in vendor/plugins (/app/vendor/plugins) are no longer allowed. Please, put your Redmine plugins in the `plugins` directory at the root of your Redmine directory (/app/plugins)
2012-11-05T14:06:12+00:00 app[web.1]: => Booting WEBrick
2012-11-05T14:06:12+00:00 app[web.1]: => Rails 3.2.8 application starting in production on http://0.0.0.0:3103
2012-11-05T14:06:12+00:00 app[web.1]: => Call with -d to detach
2012-11-05T14:06:12+00:00 app[web.1]: => Ctrl -C to shutdown server
2012-11-05T14:06:12+00:00 app[web.1]: Exiting
2012-11-05T14:06:13+00:00 heroku[web.1]: Process exited with status 1
2012-11-05T14:06:13+00:00 heroku[web.1]: State changed from starting to crashed
I can't figure where is the reason that leads app crashed actually,
Thanks for answering...
Just out of curiosity, did you happen to generate your secret key?
Try running:
heroku run rake generate_secret_token
Not sure if it will help, but I know I've had an issue with this before.