Recently I inherited a Rails application that has been deployed to production many times. I have previously deployed to a staging environment. Now, it fails to deploy to either. However, another Rails application that deploys to the same servers with the same account successfully deploys using rvm and capistrano.
I am receiving the following error:
* executing "if [ -d /path/to/app/shared/cached-copy ]; then svn switch -q --username svnusername --password <filtered> --no-auth-cache -r111111 https://svn.server.local/svn/projects/app/trunk /path/to/app/shared/cached-copy; else svn checkout -q --username svnusername --password <filtered> --no-auth-cache -r111111 https://svn.server.local/svn/projects/app/trunk /path/to/app/shared/cached-copy; fi"
servers: ["myserver-prod01.private.local"]
[myserver-prod01.private.local] executing command
** [myserver-prod01.private.local:: out]
** [myserver-prod01.private.local:: out] $rvm_path (/home/appuser/.rvm/) does not exist.
** [myserver-prod01.private.local:: out] /usr/local/rvm/scripts/rvm: line 174: rvm_is_a_shell_function: command not found
** [myserver-prod01.private.local:: out] /usr/local/rvm/scripts/rvm: line 185: __rvm_teardown: command not found
** [myserver-prod01.private.local:: out] /usr/local/rvm/bin/rvm-shell: line 83: rvm: command not found
** [myserver-prod01.private.local:: out] Error: RVM was unable to use 'ruby-1.9.3-current#appuser'
command finished in 554ms
*** [deploy:update_code] rolling back
* executing "rm -rf /path/to/app/releases/20130425150643; true"
servers: ["myserver-prod01.private.local"]
[myserver-prod01.private.local] executing command
** [out :: myserver-prod01.private.local]
** [out :: myserver-prod01.private.local] $rvm_path (/home/appuser/.rvm/) does not exist.
** [out :: myserver-prod01.private.local] /usr/local/rvm/scripts/rvm: line 174: rvm_is_a_shell_function: command not found
** [out :: myserver-prod01.private.local] /usr/local/rvm/scripts/rvm: line 185: __rvm_teardown: command not found
** [out :: myserver-prod01.private.local] /usr/local/rvm/bin/rvm-shell: line 83: rvm: command not found
** [out :: myserver-prod01.private.local] Error: RVM was unable to use 'ruby-1.9.3-current#appuser'
command finished in 209ms
** [deploy:update_code] exception while rolling back: Capistrano::CommandError, failed: "env PATH=/opt/toolkit/extra-dev-current/root/usr/bin:$PATH:/usr/database/bin LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8 RAILS_ENV=production rvm_path=$HOME/.rvm/ /usr/local/rvm/bin/rvm-shell 'ruby-1.9.3-current#appuser' -c 'rm -rf /path/to/app/releases/20130425150643; true'" on myserver-prod01.private.local
failed: "env PATH=/opt/toolkit/extra-dev-current/root/usr/bin:$PATH:/usr/database/bin LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8 RAILS_ENV=production rvm_path=$HOME/.rvm/ /usr/local/rvm/bin/rvm-shell 'ruby-1.9.3-current#appuser' -c 'if [ -d /path/to/app/shared/cached-copy ]; then svn switch -q --username svnusername --password <filtered> --no-auth-cache -r111111 https://svn.server.local/svn/projects/app/trunk /path/to/app/shared/cached-copy; else svn checkout -q --username svnusername --password <filtered> --no-auth-cache -r111111 https://svn.server.local/svn/projects/app/trunk /path/to/app/shared/cached-copy; fi'" on myserver-prod01.private.local
I have checked the server. RVM is installed and working.
This is an rvm/rvm-capistrano version mismatch. Check the version of rvm installed on the server and compare to the version of rvm-capistrano installed with bundle. If your server has rvm 1.18.x, then lock the version of rvm in your Gemfile to 1.2.x. rvm-capistrano 1.3.x requires rvm 1.19.x.
I am answering my own question because I had to figure this out the hard way. It is obvious in retrospect, but not when you first see the error. This is a case of failure to specify product versions in the Gemfile. Generally, our extensive test suite catches such problems, but our test suite does not cover deploy so we missed this one until it was too late.
Related
I installed zabbix-sender but when execute zabbix-sender i get command not found.
[root#db1 zabbix]# yum info zabbix-sender
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
Installed Packages
Name : zabbix-sender
Arch : x86_64
Version : 3.4.6
Release : 1.el7
Size : 1.0 M
Repo : installed
From repo : _ftp
Summary : Zabbix Sender
URL : http://www.zabbix.com/
License : GPLv2+
Description : Zabbix sender command line utility
when i use thi command
[root#db2 zabbix]# zabbix_sender -z zabbix -s db2 -k mysql[Binlog_cache_disk_use] -o 2000 -vv
-bash: zabbix_sender: command not found
Why does this happen?
[root#db1 zabbix]# !r
rpm -lq zabbix-sender
/usr/bin/zabbix_sender
/usr/share/doc/zabbix-sender-3.4.6
/usr/share/doc/zabbix-sender-3.4.6/AUTHORS
/usr/share/doc/zabbix-sender-3.4.6/COPYING
/usr/share/doc/zabbix-sender-3.4.6/ChangeLog
/usr/share/doc/zabbix-sender-3.4.6/NEWS
/usr/share/doc/zabbix-sender-3.4.6/README
/usr/share/man/man1/zabbix_sender.1.gz
[root#db1 zabbix]# zabbix-sender -z zabbix -s db1 -k mysql[Uptime] -o 1540 -vv
-bash: zabbix-sender: command not found
[root#db1 zabbix]#
checked again
I understand why does this happen! I remove Zabbix-sender and install again. when using Zabbix-sender -z Zabbix -s db1 -k mysql[Uptime] -o 1540 -vv there is no problem and working but after execute prel MySQL-check.pl root password then Zabbix-sender it seems to be removed. Does anyone know why? I using this template link
List files provided by rpm package:
$ rpm -ql zabbix-sender
/usr/bin/zabbix_sender
...
Try /usr/bin/zabbix_sender - do you have correct PATH setting (is /usr/bin/ included there)?
I am having trouble deploying using Capistrano using public key authentication. On windows, I have it configured to start an SSH agent automatically when I open my terminal.
Agent pid 4476
Enter passphrase for /c/Users/Lea/.ssh/id_rsa:
Identity added: /c/Users/Lea/.ssh/id_rsa (/c/Users/Lea/.ssh/id_rsa)
id_rsa is in my authorized_keys file on the server, and I use it all the time to ssh into it using ssh lea#web.3.
My Capfile is as follows:
require 'rubygems'
require 'railsless-deploy'
# application name
set :application, "site.com"
# multi-stage deploy
task :production do
set :branch, "master"
set :app_environment, "production"
role :web, "web.3", :primary => true
set :deploy_to, "/var/www/vhosts/site/site.com/"
end
task :dev do
set :branch, `git rev-parse HEAD`
set :app_environment, "development"
role :web, "web.3", :primary => true
set :deploy_to, "/var/www/vhosts/site/dev.site.com/"
end
# deploys remotely on SSH using deploy only key
set :repository, "git#bitbucket.org:us/site.git"
set :scm, :git
set :git_enable_submodules, 1
set :deploy_via, :remote_cache
# release configuration
set :use_sudo, false
set :keep_releases, 2
after "deploy:update", "deploy:cleanup"
# the web server user
set :user, "lea"
namespace :deploy do
task :migrate do
# do nothing
end
task :finalize_update, :except => { :no_release => true } do
transaction do
#run "chmod -R g+w #{release_path}"
run "echo '#{app_environment}' > #{release_path}/ENVIRONMENT"
end
end
task :restart, :except => { :no_release => true } do
# don't need to restart
end
end
When I run the deployment, it asks again for my id_rsa passphrase. Why does it ask when I already have the ssh agent running and the passphrase entered?
Following is the log of the cap dev deploy command. You can see where it asks my passphrase. Also note when I ssh into the server, it starts an ssh-agent there as well and loads a deployment_rsa key used for git (you can see these messages in the log).
$ cap dev deploy
DL is deprecated, please use Fiddle
* 2013-09-12 13:19:30 executing `dev'
* 2013-09-12 13:19:30 executing `deploy'
* 2013-09-12 13:19:30 executing `deploy:update'
** transaction: start
* 2013-09-12 13:19:30 executing `deploy:update_code'
updating the cached checkout on all servers
* executing "if [ -d /var/www/vhosts/site/dev.site.com/shared/cache
d-copy ]; then cd /var/www/vhosts/site/dev.site.com/shared/cached-cop
y && git fetch -q origin && git fetch --tags -q origin && git reset -q --hard 33
09af4ac302a6c2dc46bcf36e877abbd8472988\\\n && git submodule -q init && git submo
dule -q sync && export GIT_RECURSIVE=$([ ! \"`git --version`\" \\< \"git version
1.6.5\" ] && echo --recursive) && git submodule -q update --init $GIT_RECURSIVE
&& git clean -q -d -x -f; else git clone -q git#bitbucket.org:us/v
entek.git /var/www/vhosts/site/dev.site.com/shared/cached-copy && cd
/var/www/vhosts/site/dev.site.com/shared/cached-copy && git checkout
-q -b deploy 3309af4ac302a6c2dc46bcf36e877abbd8472988 && git submodule -q init &
& git submodule -q sync && export GIT_RECURSIVE=$([ ! \"`git --version`\" \\< \"
git version 1.6.5\" ] && echo --recursive) && git submodule -q update --init $GI
T_RECURSIVE; fi"
servers: ["web.3"]
Enter passphrase for c:/Users/Lea/.ssh/id_rsa:
[web.3] executing command
** [web.3 :: out] Agent pid 11336
** [web.3 :: err] Identity added: /home/lea/.ssh/deployment_rsa (/home/lea/.ssh
/deployment_rsa)
command finished in 2300ms
copying the cached version to /var/www/vhosts/site/dev.site.com/r
eleases/20130912191939
* executing "cp -RPp /var/www/vhosts/site/dev.site.com/shared/cache
d-copy /var/www/vhosts/site/dev.site.com/releases/20130912191939 && (
echo 3309af4ac302a6c2dc46bcf36e877abbd8472988\\\n > /var/www/vhosts/us/dev.site.com/releases/20130912191939/REVISION)"
servers: ["web.3"]
[web.3] executing command
** [out :: web.3] Agent pid 11442
*** [err :: web.3] Identity added: /home/lea/.ssh/deployment_rsa (/home/lea/.ssh
/deployment_rsa)
command finished in 751ms
* 2013-09-12 13:19:39 executing `deploy:finalize_update'
* executing "echo 'development' > /var/www/vhosts/site/dev.site.com
/releases/20130912191939/ENVIRONMENT"
servers: ["web.3"]
[web.3] executing command
** [out :: web.3] Agent pid 11451
*** [err :: web.3] Identity added: /home/lea/.ssh/deployment_rsa (/home/lea/.ssh
/deployment_rsa)
command finished in 610ms
* 2013-09-12 13:19:40 executing `deploy:create_symlink'
* executing "rm -f /var/www/vhosts/site/dev.site.com/current && ln
-s /var/www/vhosts/site/dev.site.com/releases/20130912191939 /var/www
/vhosts/site/dev.site.com/current"
servers: ["web.3"]
[web.3] executing command
** [out :: web.3] Agent pid 11460
*** [err :: web.3] Identity added: /home/lea/.ssh/deployment_rsa (/home/lea/.ssh
/deployment_rsa)
command finished in 621ms
** transaction: commit
triggering after callbacks for `deploy:update'
* 2013-09-12 13:19:41 executing `deploy:cleanup'
* executing "ls -xt /var/www/vhosts/site/dev.site.com/releases"
servers: ["web.3"]
[web.3] executing command
[err :: web.3] Identity added: /home/lea/.ssh/deployment_rsa (/home/lea/.ssh/dep
loyment_rsa)
command finished in 1186ms
** keeping 2 of 7 deployed releases
* executing "rm -rf /var/www/vhosts/site/dev.site.com/releases/2013
0906181120 /var/www/vhosts/site/dev.site.com/releases/20130912185329
/var/www/vhosts/site/dev.site.com/releases/20130912185937 /var/www/vhosts/site/dev.site.com/releases/20130912191939 /var/www/vhosts/us/dev.site.com/releases/11469"
servers: ["web.3"]
[web.3] executing command
** [out :: web.3] Agent pid 11476
*** [err :: web.3] Identity added: /home/lea/.ssh/deployment_rsa (/home/lea/.ssh
/deployment_rsa)
command finished in 750ms
$
Now, my major problem is not with the passphrase. Every time I run capistrano it fails to authenticate 2 times for every deployment. I see this in the ssh log on the server, but no indication is given in Capistrano:
11:58:44 web3 sshd[1134]: Failed password for lea from [ip] port 42421 ssh2
11:58:56 web3 sshd[1134]: Failed password for lea from [ip] port 42421 ssh2
The server is running fail2ban which blocks my IP (for 10 minutes) after 5 failed authentications, meaning I get blocked out after running capistrano 3 times. This is a huge an unacceptable problem, and I have no idea why this would occur. Do you have any advice for how to troubleshoot this problem, or a solution?
Thanks!
I ended up solving this problem myself. I was being locked out of the server because Fail2ban was an old version.
When connecting to SSH, sshd does a reverse DNS lookup. My reverse DNS on the office internet was failing, and it was printing an error into the /var/logs/secure log file.
Address x.x.x.x maps to server.domain.com, but this does not map back to the address - POSSIBLE BREAK-IN ATTEMPT!
Fail2ban was recognizing this as a failed connection, and blocking my IP because of it. It was never a problem when connecting manually because that is infrequent, but when Capistrano makes several connections in a row it was triggering it.
I used the info here: https://github.com/fail2ban/fail2ban/pull/64 to solve the problem by removing the regular expression from the fail2ban config file.
Step 1:
Do you really need a passphrase for your keys? This kind of risk today is mitigated by full disk encryption products or use of truecrypt-ed USB sticks. Less PITA, and still passes your security manager's best practices.
That said:
http://blog.blenderbox.com/2013/02/20/ssh-agent-forwarding-with-github/
Try adding
ssh_options[:forward_agent] = true
to the capfile, not Deploy.rb
I'm confused about what to do to get Capistrano to update.
I've committed my changes to git. Well hell I'll write all the steps I took.
git commit -a
git push
(all files successfully pushed to the remote git repository. all changes noted)
cap deploy
But it doesn't deploy the latest version of the site.
In fact it deploys the oldest version.
Cap Deploy Response
triggering load callbacks
* 2013-07-13 17:09:08 executing `deploy:update'
** transaction: start
* 2013-07-13 17:09:08 executing `deploy:update_code'
executing locally: "git ls-remote ssh://ubuntu#54.229.78.34/~/liquid_admin.git master"
command finished in 3150ms
* executing "git clone -b master --depth 1 ssh://ubuntu#54.229.78.34/~/liquid_admin.git /home/ubuntu/liquid_admin/releases/20130713150911 && cd /home/ubuntu/liquid_admin/releases/20130713150911 && git checkout -b deploy d609108bf81df3cb558f7536c3cee98d852b4ec5 && git submodule init && git submodule sync && export GIT_RECURSIVE=$([ ! \"`git --version`\" \\< \"git version 1.6.5\" ] && echo --recursive) && git submodule update --init $GIT_RECURSIVE && rm -Rf /home/ubuntu/liquid_admin/releases/20130713150911/.git && (echo d609108bf81df3cb558f7536c3cee98d852b4ec5 > /home/ubuntu/liquid_admin/releases/20130713150911/REVISION)"
servers: ["54.229.78.34"]
[54.229.78.34] executing command
** [54.229.78.34 :: out] Cloning into '/home/ubuntu/liquid_admin/releases/20130713150911'...
** [54.229.78.34 :: out] remote: Counting objects: 276, done.
remote: Compressing objects: 1% (3/239)
** [54.229.78.34 :: out] remote: Compressing objects: 2% (5/239)
** [54.229.78.34 :: out] remote: Compressing objects: 3% (8/239)
** [54.229.78.34 :: out] remote: Compressing objects: 4% (10/239)
** [54.229.78.34 :: out] remote: Compressing objects: 5% (12/239)
** [54.229.78.34 :: out] remote: Compressing objects: 6% (15/239)
(then a hell of a lot more of those then...)
** [54.229.78.34 :: out] Resolving deltas: 100% (58/58), done.
** [54.229.78.34 :: out] Switched to a new branch 'deploy'
command finished in 5206ms
* 2013-07-13 17:09:19 executing `deploy:finalize_update'
triggering before callbacks for `deploy:finalize_update'
* 2013-07-13 17:09:19 executing `bundle:install'
* executing "cd /home/ubuntu/liquid_admin/releases/20130713150911 && bundle install --gemfile /home/ubuntu/liquid_admin/releases/20130713150911/Gemfile --path /home/ubuntu/liquid_admin/shared/bundle --deployment --quiet --without development test"
servers: ["54.229.78.34"]
[54.229.78.34] executing command
command finished in 2138ms
* executing "chmod -R -- g+w /home/ubuntu/liquid_admin/releases/20130713150911 && rm -rf -- /home/ubuntu/liquid_admin/releases/20130713150911/public/system && mkdir -p -- /home/ubuntu/liquid_admin/releases/20130713150911/public/ && ln -s -- /home/ubuntu/liquid_admin/shared/system /home/ubuntu/liquid_admin/releases/20130713150911/public/system && rm -rf -- /home/ubuntu/liquid_admin/releases/20130713150911/log && ln -s -- /home/ubuntu/liquid_admin/shared/log /home/ubuntu/liquid_admin/releases/20130713150911/log && rm -rf -- /home/ubuntu/liquid_admin/releases/20130713150911/tmp/pids && mkdir -p -- /home/ubuntu/liquid_admin/releases/20130713150911/tmp/ && ln -s -- /home/ubuntu/liquid_admin/shared/pids /home/ubuntu/liquid_admin/releases/20130713150911/tmp/pids"
servers: ["54.229.78.34"]
[54.229.78.34] executing command
command finished in 756ms
* executing "find /home/ubuntu/liquid_admin/releases/20130713150911/public/images /home/ubuntu/liquid_admin/releases/20130713150911/public/stylesheets /home/ubuntu/liquid_admin/releases/20130713150911/public/javascripts -exec touch -t 201307131509.22 -- {} ';'; true"
servers: ["54.229.78.34"]
[54.229.78.34] executing command
** [out :: 54.229.78.34] find:
** [out :: 54.229.78.34] `/home/ubuntu/liquid_admin/releases/20130713150911/public/images'
** [out :: 54.229.78.34] : No such file or directory
** [out :: 54.229.78.34]
** [out :: 54.229.78.34] find:
** [out :: 54.229.78.34] `/home/ubuntu/liquid_admin/releases/20130713150911/public/stylesheets'
** [out :: 54.229.78.34] : No such file or directory
** [out :: 54.229.78.34]
** [out :: 54.229.78.34] find:
** [out :: 54.229.78.34] `/home/ubuntu/liquid_admin/releases/20130713150911/public/javascripts'
** [out :: 54.229.78.34] : No such file or directory
** [out :: 54.229.78.34]
command finished in 767ms
* 2013-07-13 17:09:23 executing `deploy:create_symlink'
* executing "sudo -p 'sudo password: ' rm -f /home/ubuntu/liquid_admin/current && sudo -p 'sudo password: ' ln -s /home/ubuntu/liquid_admin/releases/20130713150911 /home/ubuntu/liquid_admin/current"
servers: ["54.229.78.34"]
[54.229.78.34] executing command
command finished in 837ms
** transaction: commit
UPDATE
I did "cap deploy:update" and it updated some of the files. For example my database.yml was updated. But none of the new views, new controllers, or new models are there...
UPDATE 2
It seems to have only changed files that existed during my first deployment. So "posts" and "home" and all that is changed... but any new controllers, models, or views that I made after that were not deployed.
Cap deploy is generally deploying the master.. Did you make changes in a different branch and forgot about merging it back?
I think we updated rubber a few days back to 1.15.0 this problem now shows up in deploy.rb . Here is the block:
if Rubber::Util.has_asset_pipeline?
# load asset pipeline tasks, and reorder them to run after
# rubber:config so that database.yml/etc has been generated
load 'deploy/assets'
callbacks[:after].delete_if {|c| c.source == "deploy:assets:precompile"}
callbacks[:before].delete_if {|c| c.source == "deploy:assets:symlink"}
before "deploy:assets:precompile", "deploy:assets:symlink"
after "rubber:config", "deploy:assets:precompile"
end
The problem is when I comment it out the deploy fails saying:
* executing "sudo -p 'sudo password: ' bash -l -c 'cd /mnt/foo-production/current && RUBBER_ENV=\"production\" RAILS_ENV=\"production\" bundle exec rake rubber:config'"
servers: ["app01.foo.com"]
[app01.foo.com] executing command
** [out :: app01.foo.com] rake aborted!
** [out :: app01.foo.com] Don't know how to build task 'rubber:config'
** [out :: app01.foo.com]
** [out :: app01.foo.com] (See full trace by running task with --trace)
command finished in 17128ms
failed: "/bin/bash -l -c 'sudo -p '\\''sudo password: '\\'' bash -l -c '\\''cd /mnt/foo-production/current && RUBBER_ENV=\"production\" RAILS_ENV=\"production\" bundle exec rake rubber:config'\\'''" on app01.foo.com
Any idea why? Thanks!
I'm using ror 3.1 rc4, somehow when I deploy into a production server, the directories for images, stylesheets, and javascript are not found, and deployment fails. I do have the necessary code in deploy.rb
namespace :deploy do
task :start do ; end
task :stop do ; end
desc "Restarting mod_rails with restart.txt"
task :restart, :roles => :app, :except => { :no_release => true } do
run "touch #{current_path}/tmp/restart.txt"
end
task :precompile do
run "cd #{release_path}; RAILS_ENV=production rake assets:precompile"
end
end
after 'deploy:update_code', 'deploy:precompile'
And here is the error I get
executing "find /var/www/nattyvelo/releases/20110624033801/public/images /var/www/nattyvelo/releases/20110624033801/public/stylesheets /var/www/nattyvelo/releases/20110624033801/public/javascripts -exec touch -t 201106240338.03 {} ';'; true"
servers: ["66.228.39.243"]
[66.228.39.243] executing command
** [out :: 66.228.39.243] find: `/var/www/nattyvelo/releases/20110624033801/public/images'
** [out :: 66.228.39.243] : No such file or directory
** [out :: 66.228.39.243] find: `/var/www/nattyvelo/releases/20110624033801/public/stylesheets'
** [out :: 66.228.39.243] : No such file or directory
** [out :: 66.228.39.243] find: `/var/www/nattyvelo/releases/20110624033801/public/javascripts'
** [out :: 66.228.39.243] : No such file or directory
command finished in 705ms
triggering after callbacks for `deploy:update_code'
* executing `bundle:install'
* executing "ls -x /var/www/nattyvelo/releases"
servers: ["66.228.39.243"]
[66.228.39.243] executing command
command finished in 595ms
* executing "cd /var/www/nattyvelo/releases/20110624033801 && bundle install --gemfile /var/www/nattyvelo/releases/20110624033801/Gemfile --path /var/www/nattyvelo/shared/bundle --deployment --quiet --without development test"
servers: ["66.228.39.243"]
[66.228.39.243] executing command
** [out :: 66.228.39.243] bash: bundle: command not found
command finished in 604ms
*** [deploy:update_code] rolling back
* executing "rm -rf /var/www/nattyvelo/releases/20110624033801; true"
There's two errors happening here.
The first is that there is no longer a public/images, public/stylesheets or public/javascripts folder within a Rails 3.1 application. They have all been moved into app/assets. However, if you run rake assets:precompile then there will be a public/assets folder. This is where the static assets for your application will be served out of.
Whatever it is in your deploy script that is referencing these three folders needs to stop doing so or otherwise you'll continue to get this error.
The second error is that, just like the two other people before me have kind of suggested, you need to have the Bundler gem installed on the server.
You have a PATH problem by the look of it:
** [out :: 66.228.39.243] bash: bundle: command not found
You need to fix your PATH environment variable.
You probably have to install bundler in the production server.
sudo gem install bundler