chef-solo hangs at the end when installing redis as if chef is waiting for some event to occur. Here is output when I had to kill it with ctrl+c.
[2013-05-14T15:55:27+00:00] ERROR: Running exception handlers
[2013-05-14T15:55:27+00:00] ERROR: Exception handlers complete
Chef Client failed. 8 resources updated
[2013-05-14T15:55:27+00:00] FATAL: Stacktrace dumped to /home/ubuntu/cache/chef-stacktrace.out
[2013-05-14T15:55:27+00:00] FATAL: Chef::Exceptions::MultipleFailures: Multiple failures occurred:
* SystemExit occurred in chef run: service[redis] (redis::default line 107) had an error: SystemExit: exit
* Chef::Exceptions::Exec occurred in delayed notification: service[redis] (redis::default line 83) had an error: Chef::Exceptions::Exec: /sbin/start redis returned 1, expected 0
I am new to chef and unable to figure out why this is happening. Has anyone noticed this behaviour before?
Here is my recipe file
package "build-essential" do
action :install
end
user node[:redis][:user] do
action :create
system true
shell "/bin/false"
end
directory node[:redis][:dir] do
owner node[:redis][:user]
group node[:redis][:user]
mode "0755"
action :create
end
directory node[:redis][:data_dir] do
owner node[:redis][:user]
group node[:redis][:user]
mode "0755"
action :create
end
directory node[:redis][:log_dir] do
owner node[:redis][:user]
group node[:redis][:user]
mode "0755"
action :create
end
remote_file "#{Chef::Config[:file_cache_path]}/redis-2.6.10.tar.gz" do
source "http://redis.googlecode.com/files/redis-2.6.10.tar.gz"
action :create_if_missing
end
# Adding 'make test' causes the install to freeze for some reason.
bash "compile_redis_source" do
cwd Chef::Config[:file_cache_path]
code <<-EOH
tar zxf redis-2.6.10.tar.gz
cd redis-2.6.10
make && sudo make install
# to give permissions to the executables that it copied to.
chown -R redis:redis /usr/local/bin
EOH
creates "/usr/local/bin/redis-server"
end
service "redis" do
provider Chef::Provider::Service::Upstart
subscribes :restart, resources(:bash => "compile_redis_source")
supports :restart => true, :start => true, :stop => true
end
template "redis.conf" do
path "#{node[:redis][:dir]}/redis.conf"
source "redis.conf.erb"
owner node[:redis][:user]
group node[:redis][:user]
mode "0644"
notifies :restart, resources(:service => "redis")
end
template "redis.upstart.conf" do
path "/etc/init/redis.conf"
source "redis.upstart.conf.erb"
owner node[:redis][:user]
group node[:redis][:user]
mode "0644"
notifies :restart, resources(:service => "redis")
end
service "redis" do
action [:enable, :start]
end
There are 2 service "redis" resource statements, is that a problem? or how does chef workout in this case, does it merge into a single resource when running?
I am using upstart and here is the redis.upstart.conf.erb file. Not sure if anything is wrong with this. Does the order of the statement matter in this file?
#!upstart
description "Redis Server"
emits redis-server
# run when the local FS becomes available
start on local-filesystems
stop on shutdown
setuid redis
setgid redis
expect fork
# Respawn unless redis dies 10 times in 5 seconds
#respawn
#respawn limit 10 5
# start a default instance
instance $NAME
env NAME=redis
#instance $NAME
# run redis as the correct user
#setuid redis
#setgid redis
# run redis with the correct config file for this instance
exec /usr/local/bin/redis-server /etc/redis/redis.conf
respawn
#respawn limit 10 5
I think Dmytro was on the right path, but not exactly.
I see that you are using Upstart as the service provider in Chef. Please check your Upstart config for redis-server for any expect statement. If you have an expect fork or expect daemon statement in there, it means that when starting redis-server, Upstart will be waiting for the Redis service to fork once or twice respectively. If you have daemonize no in the redis.conf, Redis process will never fork, and therefore Upstart just hangs at the execution of the init script.
Your redis is not failing to start, it simply runs in the foreground.
I had similar problem with one of the Redis cookbooks I was using. In the redis.conf.erb file it had configuration option
daemonize no
Some other cookbooks have this option configurable by attribute. So, your fix would depend on the cookbook you are using. Either edit your redis.conf.erb file or find how that attribute is configured and set it to yes.
Related
I need a step by step procedure to uninstall RUNDECK. i am facing some STACK overflow issue which i wasn't able to resolve so i want to uninstall and install it from scratch
Stack error:
[2020-06-05 18:48:44.098] ERROR StackTrace --- [tp1284944245-71] Full Stack Trace:
org.grails.taglib.GrailsTagException: [views/layouts/base.gsp:184] Error executing tag <g:render>: [views/common/_sidebar.gsp:128] Error executing tag <g:ifMenuItems>: Method 'java.util.Set com.dtolabs.rundeck.core.authorization.providers.EnvironmentalContext.forProject(java.lang.String)' must be InterfaceMethodref constant
at org.grails.gsp.GroovyPage.throwRootCause(GroovyPage.java:473)
at org.grails.gsp.GroovyPage.invokeTag(GroovyPage.java:415)
at jdk.internal.reflect.GeneratedMethodAccessor217.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:98)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.in```
WAR based instance:
Make sure that the Rundeck process is down, identify the process
doing ps aux| grep -i rundeck and use kill -9 <PID> to shut
down.
Wipe the instance, you can delete all directory (and content)
defined in %RDECK_BASE%. All configurations and files are inside
this directory. If your system has a init script to launch rundeck, ensure that script doesn't have any reference to rundeck.
Re-install following this.
RPM-based (CentOS, RHEL, Fedora) instance:
Shutdown the Rundeck service: # systemctl stop rundeckd.
Make sure that the process is down: # systemctl status rundeckd.
Remove the package, do # yum remove rundeck.
Some files keep on the system, check and wipe the following paths:
/etc/rundeck, /var/lib/rundeck and /var/log/rundeck.
Re-install following this.
DEB-based (Debian, Ubuntu, Mint) instance:
Shutdown the Rundeck service: # systemctl stop rundeckd.
Make sure that the process is down doing # systemctl status rundeckd.
Remove the package, do # apt-get purge rundeck
Some files keep on the system, check and wipe the following paths:
/etc/rundeck, /var/lib/rundeck and /var/log/rundeck.
Reinstall following this.
Anyway, I recommend to make a backup of your instance / configurations before wiping it.
For testing the best option is to run Rundeck docker image, it saves a lot of time.
About the error, check your Rundeck version, maybe you're facing this issue.
I'm provisioning a vagrant vm using chef_solo, using bento/centos7.2. I'm installing httpd through the httpd cookbook with the following code.
# Install Apache and starts it
httpd_service 'mine' do
mpm 'prefork'
action [:create, :start]
listen_ports ['80', '443']
run_user node['webserver']['run_user'] # the value is 'vagrant'
end
# Add the site configuration
httpd_config 'mine' do
instance 'mine'
source 'mine.conf.erb'
notifies :restart, 'httpd_service[mine]'
end
# Create document rood directory
directory "#{node['webserver']['document_root']}" do
recursive true
end
# Homepage
template "#{node['webserver']['document_root']}/index.php" do
source 'index.php.erb'
mode '0644'
owner node['webserver']['user']
group node['webserver']['group']
action :create_if_missing
end
# Install the mod_php Apache module.
httpd_module 'php' do
instance 'mine'
end
# Install memcached
package 'memcached'
# Install php-pecl-memcache
package 'php-pecl-memcache' do
action :install
notifies :restart, 'httpd_service[mine]'
end
# Install php-pecl-memcached
package 'php-pecl-memcached' do
action :install
notifies :restart, 'httpd_service[mine]'
end
# Install php-mysql
package 'php-mysql' do
action :install
notifies :restart, 'httpd_service[mine]'
end
# Install php-pdo
package 'php-pdo' do
action :install
notifies :restart, 'httpd_service[mine]'
end
# Install php-intl
package 'php-intl' do
action :install
notifies :restart, 'httpd_service[mine]'
end
# Install php-mssql
package 'php-mssql' do
action :install
notifies :restart, 'httpd_service[mine]'
end
Funny thing is, When I run kitchen converge everything is good and httpd works perfectly (InSpec test pass, and I manually verify with kitchen login).
But when I run on vagrant, it is killed from the very beginning, and it dies after each request. I don't know why, nor I know if there is any log I can read to debug this behavior. Any ideas?
UDPATE:
This is the log found in /var/log/httpd-mine/error_log
This is the log in sudo journalctl
I just completed my capistrano installation for the first time. Most of everything is left to default settings, I configured my server, its authentification, and the remote folder, as well as the access to my git repository.
I use capistrano to deploy php code to my server.
cap staging deploy and cap production deploy function, but they run every command twice. It sometimes causes problems when those tasks are executed too quickly on the server, returning error codes, which stops the deploying process.
an example of my output when running cap staging deploy
DEBUG[47ecea59] Running /usr/bin/env if test ! -d ~/www/test_server/repo; then echo "Directory does not exist '~/www/test_server/repo'" 1>&2; false; fi on ftp.cluster013.ovh.net
DEBUG[47ecea59] Command: if test ! -d ~/www/test_server/repo; then echo "Directory does not exist '~/www/test_server/repo'" 1>&2; false; fi
DEBUG[c450e730] Running /usr/bin/env if test ! -d ~/www/test_server/repo; then echo "Directory does not exist '~/www/test_server/repo'" 1>&2; false; fi on ftp.cluster013.ovh.net
DEBUG[c450e730] Command: if test ! -d ~/www/test_server/repo; then echo "Directory does not exist '~/www/test_server/repo'" 1>&2; false; fi
It does the same with every single task, except the one I defined myself (in my deploy.rb, I defined a :set_distant_server task that moves around files with server info)
I am pretty sure I missed something during the initial configuration.
Here is my capfile, still to default settings :
# Load DSL and Setup Up Stages
require 'capistrano/setup'
# Includes default deployment tasks
require 'capistrano/deploy'
# Includes tasks from other gems included in your Gemfile
# require 'capistrano/rvm'
# require 'capistrano/rbenv'
# require 'capistrano/chruby'
#require 'capistrano/bundler'
#require 'capistrano/rails/assets'
#require 'capistrano/rails/migrations'
# Loads custom tasks from `lib/capistrano/tasks' if you have any defined.
Dir.glob('lib/capistrano/tasks/*.rake').each { |r| import r }
Followed by my deploy.rb file:
# config valid only for Capistrano 3.1
lock '3.2.1'
set :scm, :git
set :application, 'Application name'
# I use token authentification
set :repo_url, 'https://XXXXXXXXXXX:#XXXXXXX.git'
set :role, 'web'
# Default value for :log_level is :debug
set :log_level, :debug
set :tmp_dir, 'www/test_server/tmp'
set :keep_releases, 8
role :deploy_server, "XXXuser_name#XXXX_server"
task :set_distant do
on roles(:deploy_server) do
execute 'echo ------------******* STAGING *******------------'
execute 'cp ~/www/test_server/current/access_distant.php ~/www/test_server/current/access.php'
execute 'cp ~/www/test_server/current/session_distant.php ~/www/test_server/current/session.php'
end
end
after "deploy:finished", :set_distant
Here is my staging.rb, much shorter:
server 'XXX_server', user: 'XXXuser_name', roles: %w{web}, port: 22, password: 'XXXpassword'
set :deploy_to, '~/www/test_server'
set :branch, 'staging'
And my production.rb, very similar:
server 'XXX_server', user: 'XXXuser_name', roles: %w{web}, port: 22, password: 'XXXpassword'
set :deploy_to, '~/www/beta/'
I'm pretty sure I missed a step in all the prerequisites to make it run nicely. I am new to ruby, to gems, and didn't use shell for a very long time.
Does anyone see why those commands are run twice, and how I could fix it?
In advance, many many thanks.
Additional info:
Ruby version: ruby -v
ruby 2.1.2p95 (2014-05-08 revision 45877) [x86_64-darwin13.0]
Capistrano version: cap -V
Capistrano Version: 3.2.1 (Rake Version: 10.1.0)
I did not create a Gemfile or set it up, I understood it was not needed in Capistrano 3. Anyway, I would not know how to do it.
I was having this same issue and realized I didn't need both
role :web
and
server '<server>'
I got rid of role :web and that got rid of the 2nd execution.
What possible reasons can Sidekiq prevent from processing jobs in the queue? The queue is full. The log file sidekiq.log indicates no activity at all. Thus the queue is full but the log is empty, and Sidekiq does not seem to process items. There seem to no worker processing jobs. Restarting Redis or flush it with FLUSHALL or FLUSHDB has no effect. Sidekiq has been started with
bundle exec sidekiq -L log/sidekiq.log
and produces the following log file:
2013-05-30..Booting Sidekiq 2.12.0 using redis://localhost:6379/0 with options {}
2013-05-30..Running in ruby 1.9.3p374 (2013-01-15 revision 38858) [i686-linux]
2013-05-30..See LICENSE and the LGPL-3.0 for licensing details.
2013-05-30..Starting processing, hit Ctrl-C to stop
How can you find out what went wrong? Are there any hidden log files?
The reason was in our case: Sidekiq may look for the wrong queue. By default Sidekiq uses a queue named "default". We used two different queue names, and defined them in config/sidekiq.yml
# configuration file for Sidekiq
:queues:
- queue_name_1
- queue_name_2
The problem is that this config file is not automatically loaded by default in your development environment (unlike database.yml or thinking_sphinx.yml for instance) by a simple bundle exec sidekiq command. Thus we wrote our jobs in two certain queues, and Sidekiq was waiting for jobs in a third queue (the default one). You have to pass the path to the config file as a parameter through the -Cor --config option:
bundle exec sidekiq -C ./config/sidekiq.yml
or you can pass the queue names directly (no spaces allowed here after the comma):
bundle exec sidekiq -q queue_name_1,queue_name_2
To find the problem out it is helpful to pass the option -v or --verbose at the command line, too, or to use :verbose: true in the sidekiq.yml file. Everything which is defined in a config file is of course useless if the config file is not loaded.. Therefore make sure you are using the right config file first.
If you have a config/sidekiq.yml check that all the queues are defined there, check this sample file: https://github.com/mperham/sidekiq/blob/master/examples/config.yml
If you are passing queue names in the command line or Procfile, something similar to
bin/sidekiq -q queue1 -q queue2
bundle exec sidekiq -q queue1 -q queue2
check that all your queues are defined there.
In case you are not sure about the names of your queues, you can figure it out with the following script:
require "sidekiq/api"
stats = Sidekiq::Stats.new
stats.queues
# {"production_mailers"=>25, "production_default"=>1}
Then, you can do things with the queues:
queue = Sidekiq::Queue.new("production_mailers")
queue.count
queue.clear
It took me hours to find out that I had set config.active_job.queue_name_prefix = "xxxxx_#{Rails.env}". The queue names in the settings look the same, but sidekiq looks for the queue with prefix.
Wrong setting
app/jobs/my_job.rb
class MyJob < ApplicationJob
queue_as :default
end
config/sidekiq.yml
:queues:
- default
Correct setting
app/jobs/my_job.rb
class MyJob < ApplicationJob
queue_as :default
end
config/sidekiq.yml
:queues:
- xxxxx_development_default
- xxxxx_production_default
My problem was I had a configure_server but not configure_client in my initialiser, you must have both:
Sidekiq.configure_server do |config|
config.redis = { url: ENV.fetch('SIDEKIQ_REDIS_URL', 'redis://127.0.0.1:6379/1') }
end
Sidekiq.configure_client do |config|
config.redis = { url: ENV.fetch('SIDEKIQ_REDIS_URL', 'redis://127.0.0.1:6379/1') }
end
In my case, sidekiq was fine in development, but stuck in staging. It was human error on the capistrano's deploy configuration. I set the path for sidekiq.yml incorrectly in the Capfile (shared instead of current).
It failed silently:
# Capfile
# WRONG:
set :sidekiq_config, -> { File.join(shared_path, 'config', 'sidekiq.yml') }
^^^^^^^^^^^
# RIGHT:
set :sidekiq_config, -> { File.join(current_path, 'config', 'sidekiq.yml') }
flushing redis worked for me.
WARNING: THIS WILL REMOVE ALL DATA IN YOUR REDIS DATABASE.
redis-cli flushall
I was banging my head against a brick wall on this for a while, my issue was that sidekiq required a newer version of redis-server. I ran "bundle exec sidekiq" and that revealed the error. Once I updated to a newer version of redis-server it was fine.
I just had this issue. Turns out I had made a syntax error in my sidekiq.yml
Spent at least two hours on this as well because queues and configuration and web UI were all fine ... the jobs were just not processed.
My issue was that the sidekiq-server was not running in my docker-compose setup even though it should have been started in the command-section here:
sidekiq:
depends_on:
- 'proddb'
- 'redis'
build: rails-app
--> command: bundle exec sidekiq --environment ${RAILS_ENV} -C config/sidekiq.yml
volumes:
- './rails-app:/project'
- '/project/tmp' # don't mount tmp directory
environment:
- REDIS_URL_SIDEKIQ=${REDIS_URL_SIDEKIQ}
networks:
- backend
My problem was I did not config my initializers/sidekiq.rb properly but even with the correct config, sidekiq was still not running enqueued jobs. I had to run spring stop on top of that and restarted everything and it solved my issue.
I encountered a similar problem wherein the logs would show entries such as INFO Rails : queueing TestWorker (TestWorker). However, the jobs would never get processed, and none of the answers in this question solved the issue.
The tl;dr to my solution is that Sidekiq's Testing Client was getting unexpectedly triggered.
I eventually deduced that there is some "magic" going on underneath the surface that makes it difficult to discretely determine where/when/how the above testing trigger was getting configured, based on the following anecdote...
Running bundle exec sidekiq -C config/sidekiq.yml -e development had the result that Sidekiq::Testing.fake? == true
However, running bundle exec sidekiq -C config/sidekiq.yml -e development_2 had the result that Sidekiq::Testing.fake? == false
^ The only difference between these 2 commands is that I renamed the development environment in sidekiq.yml to development_2, i.e. the same/equivalent environment was running with both commands (at least, presumably it would be the same environment if it wasn't for this inane "magic" under the hood).
I updated sidekiq.rb to explicitly toggle Sidekiq::Testing via the following:
sidekiq_testing_fake = false # set this using env var, etc.
if sidekiq_testing_fake
Sidekiq::Testing.fake!
elsif Sidekiq.constants.include?(:Testing)
Sidekiq::Testing.disable!
end
My issue was that I had both a redis-server running and Redis.app's redis-server running, I killed the redis-server (and kept the Redis.app one)
I'm struggling to get delayed_job working under rails 3.0.9 (ruby 1.9.2). The only way I have succeeded to run is to tape manualy the command rake jobs:work.
But I want that to be automatically started when the rails application is starting.
I have installed monit under ubuntu and I configured it to launch a file located in my app. This fails looks like:
check process delayed_job with pidfile /home/me/myapp/tmp/pids/delayed_job.pid
start program = "/home/me/myapp/script/delayed_job start"
stop program = "/home/me/myapp/script/delayed_job stop"
And I added the environment setting in the delayed_job script file:
#!/usr/bin/env ruby
ENV['RAILS_ENV'] = "development"
require File.expand_path(File.join(File.dirname(__FILE__), '..', 'config', 'environment'))
require 'delayed/command'
Delayed::Command.new(ARGV).daemonize
When I run the command "sudo monit start delayed_job" I get the following error:
/usr/lib/ruby/1.8/rubygems/custom_require.rb:31:in `gem_original_require': no such file to load -- bundler/setup (LoadError)
So I guess it is because sudo is using a wrong version of ruby environment
I tried then the solution of:
rvm monit delayed_job
by adding rvm -S in the start program / stop program lines.
But it still failing with the error : rvm command not found
my rvm dir is located in my home dir /home/me/.rvm
I tried to find workarounds in (sudo changes PATH - why?) to change the PATH environment variable by adding
/usr/bin/env PATH=/home/me/.rvm/bin:$PATH
The command "sudo monit start delayed_job" succeeded! and the worker started.
But the issue is: When I launch sudo /etc/init.d/monit start and when I look to the syslog I still get 'delayed_job' failed to start
So I don't know how to investigate more, how to get more verbose errors for monit.
I have finally succeeded to solve this issue.
I modified the monit file like this:
check process delayed_job with pidfile /home/me/myapp/tmp/pids/delayed_job.pid
start program = "/bin/su - me -c 'cd /home/me/myapp/; script/delayed_job start'"
stop program = "/bin/su - me -c 'cd /home/me/myapp/; script/delayed_job stop'"
I have also downgraded the daemons gem because it seems that there are problems with the latest version. So I'm using now daemons v 1.0.10
I also modified the rights of the log file /home/me/myapp/log/delayed_job.log, because it seems that is was created before my root and my user had no access to it (I had problems to test the command "script/delayed_job start" with "me" user)
This i s the only line that worked for me that read the ENV properly
start program = "/usr/local/rvm/bin/rvm-shell -c 'cd /var/www/[APP]/current/; RAILS_ENV=production bundle exec bin/delayed_job start'"
Hope it helps!