monit - Can I restart a process if system memory is too much? - monit

I have this rule for system:
check system $HOST
if memory usage > 90% for 3 cycles then alert
and this rule for a process:
check process my_process matching "..."
restart program = "..."
I would like that if system memory is more than 90% to restart the process my_process.
Is this possible with monit?
I tried variants of if memory usage > 90% for 3 cycles then restart my_process but always the syntax is not recognized on monit reload.

Back to your sample, you can use something like this.
check system $HOST
if memory usage > 90% for 3 cycles then
exec "/bin/bash -c '/usr/local/bin/monit restart my_process'"
To restart a service named "my_process".
With regards, Lutz

The command (start, stop, restart, ...) execute the proper command for the service itself.
With regards,
Lutz

Related

How to run forever without sudo on Ec2

The title is pretty much tells what the question is about. I am trying to use forever to start a script on Ec2 but it does not work unless I use sudo.
If I start without sudo, I get
warn: --minUptime not set. Defaulting to: 1000ms
warn: --spinSleepTime not set. Your script will exit if it does not stay up for at least 1000ms
info: Forever processing file: ci.js
But when I do forever list
info: No forever processes running
You should run forever list under same user you've started forever (it seems like you are doing that right).
Try to check ps aux | grep node after you do forever start. Maybe you haven't started any process (because of errors in command line or in your NodeJS file) so forever list returns empty list.
p.s. I've checked forever on my machine and it is behaving exactly as you said - if i run it under my 'ubuntu' user -> list of running is empty even though process is alive... Seem like a bug in forever.

Linux loop script: Cannot allocate memory

I face a huge problem with a script on linux.
I work on an apache2 server and I have to execute a php script every seconds to update the database (yes, every second, I'm sure).
To do that, I created this script :
#!/bin/bash
while [ -f "MONFICHIER" ]
do
php fichier.php >> log.txt
sleep 1
done
exit 0
This script runs for a while and stops and I get this error message : "fork: Cannot allocate memory"
Actually everything works fine and after a while, a plenty of defunct processes are generated and it is because these processes that the memory is full.
About the php file it is a index.php file to the CodeIgniter Framework with parameters for the fonction to call. Finally it updates the database by checking data.
I'm sorry but I can't provide the source code (confidential) but the function is fast (less than a second)
Did any body have this problem?
Thanks!
Either your PHP script takes more than one second to execute and you have a lot of parallel php invocation, or it takes a lot of memory. Either way, nobody can help you with only the source of the shell script.

Setting memory consumption limits with Upstart

I've recently become quite fond of Upstart. Previously I've been using God, Monit and Bluepill but I don't really like these solutions so I'm giving Upstart a try.
I've been using the Foreman gem to generate some basic Upstart configuration files for my processes in /etc/init. However, these generated files only handle the respawning of a crashed process. I was wondering whether it's possible to tell Upstart to restart a process that's consuming for example > 150mb of memory, as you would with Monit, God or Bluepill.
I read through the Upstart docs and this looks like the thing I'm looking for. Though I have no clue how to config something like this.
What I basically want is quite simple. I want to restart my web process if the memory usage is > 150mb ram. These are the files I have:
|-- myapp-web-1.conf
|-- myapp-web-2.conf
|-- myapp-web-3.conf
|-- myapp-web.conf
|-- myapp.conf
And their contents are:
myapp.conf
pre-start script
bash << "EOF"
mkdir -p /var/log/myapp
chown -R deployer /var/log/myapp
EOF
end script
myapp-web.conf
start on starting myapp
stop on stopping myapp
myapp-web-1.conf / myapp-web-2.conf / myapp-web-3.conf
start on starting myapp-web
stop on stopping myapp-web
respawn
exec su - deployer -c 'cd /var/applications/releases/20110607140607; cd myapp && bundle exec unicorn -p $PORT >> /var/log/myapp/web-1.log 2>&1'
Any help much appreciated!
Appending this to the end of myapp-web-*.conf will cause any allocation calls trying to allocate more than 150mb of memory to return ENOMEM:
limit rss 157286400 157286400
The process might crash at this point, or it might not. That's up to the process!
Here's a test for this in the Upstart Source.
From the Upstart docs, the limits come from the rlimit system call options. (http://upstart.ubuntu.com/cookbook/#limit)
Since Linux 2.4+ setting the rss (Resident Set Size) has no effect.
An alternative already suggested in other answers is as which sets the virtual memory Address Space size limits. This will have a very different effect of setting 'real' memory limits.
limit as <soft limit> <hard limit>
Excerpt from man pages for setrlimit:
RLIMIT_AS
The maximum size of the process's virtual memory (address space) in bytes. This limit affects calls to brk(2), mmap(2), and mremap(2),
which fail with the error ENOMEM upon exceeding this limit. Also automatic stack expansion will fail (and generate a SIGSEGV that
kills the process if no alternate stack has been made available via sigaltstack(2)). Since the value is a long, on machines with a
32-bit long either this limit is at most 2 GiB, or this resource is unlimited.

Fork shell script (not &)

I'm accessing a webserver via PHP. I want to update some info in the Apache configs, so I start a shell script that makes the changes. Then I want to stop and restart Apache.
Problem: as soon as I stop Apache, my process stops and my shell script, being a child process, is killed. Apache never restarts. This also happens with Apache restart.
Is there a way to fork an independent, non-child process for the shell script, so I can restart Apache?
Thx,
Mr B
You can use disown:
disown [-ar] [-h] [jobspec ...]
Without options, each jobspec is removed from the table of active jobs. If the `-h' option is given, the job is not removed from the table, but is marked so that SIGHUP is not sent to the job if the shell receives a SIGHUP. If jobspec is not present, and neither the `-a' nor `-r' option is supplied, the current job is used. If no jobspec is supplied, the `-a' option means to remove or mark all jobs; the `-r' option without a jobspec argument restricts operation to running jobs.
./myscript.sh &
disown
./myscript.sh will continue running even if the script that started it dies.
Take a look at nohup, may fit you needs.
let's say you have a script called test.sh
for i in $(seq 100); do
echo $i >> test.temp
sleep 1;
done
if you run nohup ./test.sh & you can kill the shell and the process stay alive.

valgrind on server process

hi i am new to valgrind. I know how to run valgrind on executable files from command line. But how do you run valgrind on server processes like apache/myqld/traffic server etc ..
I want to run valgrind on traffic server (http://incubator.apache.org/projects/trafficserver.html) to detect some memory leaks taking place in the plugin I have written.
Any suggestions ?
thanks,
pigol
You have to start the server under Valgrind's control. Simply take the server's normal start command, and prepend it with valgrind.
Valgrind will attach to every process your main "server" process spawns. When each thread or process ends, Valgrind will output its analysis, so I'd recommend piping that to a file (not sure if it comes out on stderr or stdout.)
If your usual start command is /usr/local/mysql/bin/mysqld, start the server instead with valgrind /usr/local/mysql/bin/mysqld.
If you usually start the service with a script (like /etc/init.d/mysql start) you'll probably need to look inside the script for the actual command the script executes, and run that instead of the script.
Don't forget to pass the --leak-check=full option to valgrind to get the memory leak report.