I've recently become quite fond of Upstart. Previously I've been using God, Monit and Bluepill but I don't really like these solutions so I'm giving Upstart a try.
I've been using the Foreman gem to generate some basic Upstart configuration files for my processes in /etc/init. However, these generated files only handle the respawning of a crashed process. I was wondering whether it's possible to tell Upstart to restart a process that's consuming for example > 150mb of memory, as you would with Monit, God or Bluepill.
I read through the Upstart docs and this looks like the thing I'm looking for. Though I have no clue how to config something like this.
What I basically want is quite simple. I want to restart my web process if the memory usage is > 150mb ram. These are the files I have:
|-- myapp-web-1.conf
|-- myapp-web-2.conf
|-- myapp-web-3.conf
|-- myapp-web.conf
|-- myapp.conf
And their contents are:
myapp.conf
pre-start script
bash << "EOF"
mkdir -p /var/log/myapp
chown -R deployer /var/log/myapp
EOF
end script
myapp-web.conf
start on starting myapp
stop on stopping myapp
myapp-web-1.conf / myapp-web-2.conf / myapp-web-3.conf
start on starting myapp-web
stop on stopping myapp-web
respawn
exec su - deployer -c 'cd /var/applications/releases/20110607140607; cd myapp && bundle exec unicorn -p $PORT >> /var/log/myapp/web-1.log 2>&1'
Any help much appreciated!
Appending this to the end of myapp-web-*.conf will cause any allocation calls trying to allocate more than 150mb of memory to return ENOMEM:
limit rss 157286400 157286400
The process might crash at this point, or it might not. That's up to the process!
Here's a test for this in the Upstart Source.
From the Upstart docs, the limits come from the rlimit system call options. (http://upstart.ubuntu.com/cookbook/#limit)
Since Linux 2.4+ setting the rss (Resident Set Size) has no effect.
An alternative already suggested in other answers is as which sets the virtual memory Address Space size limits. This will have a very different effect of setting 'real' memory limits.
limit as <soft limit> <hard limit>
Excerpt from man pages for setrlimit:
RLIMIT_AS
The maximum size of the process's virtual memory (address space) in bytes. This limit affects calls to brk(2), mmap(2), and mremap(2),
which fail with the error ENOMEM upon exceeding this limit. Also automatic stack expansion will fail (and generate a SIGSEGV that
kills the process if no alternate stack has been made available via sigaltstack(2)). Since the value is a long, on machines with a
32-bit long either this limit is at most 2 GiB, or this resource is unlimited.
Related
I have this rule for system:
check system $HOST
if memory usage > 90% for 3 cycles then alert
and this rule for a process:
check process my_process matching "..."
restart program = "..."
I would like that if system memory is more than 90% to restart the process my_process.
Is this possible with monit?
I tried variants of if memory usage > 90% for 3 cycles then restart my_process but always the syntax is not recognized on monit reload.
Back to your sample, you can use something like this.
check system $HOST
if memory usage > 90% for 3 cycles then
exec "/bin/bash -c '/usr/local/bin/monit restart my_process'"
To restart a service named "my_process".
With regards, Lutz
The command (start, stop, restart, ...) execute the proper command for the service itself.
With regards,
Lutz
I am trying to run an application which uses pagemap in gem5 FS mode.
But I am not able to use pagemap in gem5. It throws below error -
"assert(pagemap>=0) failed"
The line of code is:
int pagemap = open("/proc/self/pagemap", O_RDONLY);
assert(pagemap >= 0);
Also, If I try to run my application on gem5 terminal with sudo ,it throws error-
sudo command not found
How can I use sudo in gem5 ??
These problems are not gem5 specific, but rather image / Linux specific, and would likely happen on any simulator or real hardware. So I recommend that you remove gem5 from the equation completely, and ask a Linux or image specific question next time, saying exactly what image your are using, kernel configs, and provide a minimal C example that reproduces the problem: this will greatly improve the probability that you will get help.
I have just done open("/proc/self/pagemap", O_RDONLY) successfully with: this program and on this fs.py setup on aarch64, see also these comments.
If /proc/<pid>/pagemap is not present for any file, do the following:
ensure that procfs is mounted on /proc. This is normally done with an fstab entry of type:
proc /proc proc defaults 0 0
but your init script needs to use fstab as well.
Alternatively, you can mount proc manually with:
mount -t proc proc proc/
you will likely want to ensure that /sys and /dev are mounted as well.
grep the kernel to see if there is some config controlling the file creation.
These kinds of things are often easy to find without knowing anything about the kernel.
If I do:
git grep '"pagemap'
to find the pagemap string, which is likely the creation point, on v4.18 this leads me to fs/proc/base.c, which contains:
#ifdef CONFIG_PROC_PAGE_MONITOR
REG("pagemap", S_IRUSR, proc_pagemap_operations),
#endif
so make sure CONFIG_PROC_PAGE_MONITOR is set.
sudo: most embedded / simulator images don't have it, you just login as root directly and can do anything by default without it. This can be seen by the conventional # in the prompt instead of $.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
This question does not appear to be about a specific programming problem, a software algorithm, or software tools primarily used by programmers. If you believe the question would be on-topic on another Stack Exchange site, you can leave a comment to explain where the question may be able to be answered.
Closed 4 years ago.
Improve this question
I frequently get the following Dropbox error. The error message's proposal fixes the error, but I'm trying to figure out what it's doing to my system, and perhaps if there is a root cause at play.
Unable to monitor entire Dropbox folder hierarchy. Please run
echo fs.inotify.max_user_watches=100000 | sudo tee -a /etc/sysctl.conf; sudo sysctl -p`
and restart Dropbox to fix the problem.
Note: I strongly encourage you to actually DO the steps and not just read them if you want to learn about Linux!
If I type apropos inotify in a shell to see which manpage are about "inotify" I get these results:
$ apropos inotify
inotify (7) - monitoring filesystem events
inotify_add_watch (2) - add a watch to an initialized inotify instance
inotify_init (2) - initialize an inotify instance
inotify_init1 (2) - initialize an inotify instance
inotify_rm_watch (2) - remove an existing watch from an inotify instance
upstart-file-bridge (8) - Bridge between Upstart and inotify
apropos finds manpages. apropos is your friend. Remember apropos.
The first one looks promising, so lets try opening that with man inotify. You should now get the documentation of ifnotify. It says:
NAME
inotify - monitoring filesystem events
DESCRIPTION
The inotify API provides a mechanism for monitoring filesystem events.
Inotify can be used to monitor individual files, or to monitor directo‐
ries. When a directory is monitored, inotify will return events for
the directory itself, and for files inside the directory.
So now we've learned what inotify roughly does. Let's see if it also has something useful to
say about your error.
We can search in manpages by pressing /<term><enter>. So lets try: /max_user_watches<enter> That brings us to this section:
/proc/sys/fs/inotify/max_user_watches
This specifies an upper limit on the number of watches that can
be created per real user ID.
The /proc/sys/fs/inotify/max_user_watches file is the same as the fs.inotify.max_user_watches setting in /etc/sysctl.conf. They're just two different ways to access the same Linux kernel parameter.
You can press q to exit.
I can see what the current value is by using:
$ cat /proc/sys/fs/inotify/max_user_watches
524288
or:
$ sysctl fs.inotify.max_user_watches
fs.inotify.max_user_watches = 524288
They both use the same underlying value in the Linux kernel.
Note that this value is actually five times larger than what Dropbox recommends! This is on my Ubuntu 15.10 system.
So now we have learned that:
inotify is a Linux system to monitor changes to files and directories.
That we can set how many files or directories a user is allowed to watch simultaneously.
That Dropbox gives the error "Unable to monitor entire Dropbox folder hierarchy".
From this information, It seems to be the case that Dropbox is unable to watch enough files and directories for changes because the fs.inotify.max_user_watches is too low.
What I recommend is:
Check the /etc/sysctl.conf with any text editor as root. Make sure there are no fs.inotify.max_user_watches=100000 lines here. If there are, remove them.
Reboot the system to restore the default value.
Check what the value of fs.inotify.max_user_watches is as described above.
Double that by running sudo echo 'fs.inotify.max_user_watches=XXX' >> /etc/sysctl.conf && sudo sysctl -p /etc/sysctl.conf. You don't need to reboot after this change.
Hope this error is now fixed. Time will tell. If not, try doubling the value again.
If that doesn't fix it, there may be another problem, or maybe you just need to increase it even more. It depends a bit on what the default value is for your system.
Note: Newer versions of systemd no longer load the /etc/sysctl.conf file, it will only load files from the /etc/sysctl.d/ directory. Using a file in the /etc/sysctl.d directory should already be supported by most Linux distros, so I recommend you use that for future-proofing.
If you're wondering "why is there a limit in the first place?" then consider what would happen if a program would watch a million files. Would that still work? And what about a billion? 10 billion? 10 trillion? etc.
At some point, your system will run out of resources and crash. See here on some "fun" ways to do that ;-)
I am currently setting up a web service powered by apache and running on CENTOS 6.4.
This service uses perl scripts (cgi-bin) launching in particular external homemade fortran compiled binaries.
Here is the issue: when I boot my server, everything goes well except that one of my binary crashes systematically (with a kernel segfault) when called by my perl scripts.
If I restart manually the httpd service (at the command line: service httpd restart), the issue is totally fixed.
I examined apache/system logs and nothing suspicious can be found.
It appears that the problem occurs only when httpd is launched by /etc/rc[0-6].d startup directives. I tried to change the launch order of http (S85httpd by default) to any other position without success.
To summarize, my web service is only functional (with no external binary crash) when httpd is launched at the command line once the server has fully booted up!
[EDIT] This issue is now resolved:
My fortran binary handles very large arrays and complex functions requiring an unlimited stack size.
Despite that the stack size limit was defined on a system-wide basis (in /etc/security/limits.conf), for any reason it appears that the "apache/perl/fortran binary" ensemble was not aware of that (causing my binary to crash each time it was called).
At the contrary, when I manually restarted apache at the shell prompt, the stacksize limit was correctly passed (.bashrc with 'ulimit -S -s unlimited').
As a workaround, I used BSD::Resource module (http://metacpan.org/pod/BSD::Resource) to define stacksize directly in my perl script by using e.g. setrlimit(RLIMIT_STACK, $softlimit, $hardlimit);
Thus, this new stack size limit is now directly passed from my perl script to my binary.
I've run into similar problems before. Maybe one way to solve this is to put the binary on a 'delayed start', so that it starts after everything else on your system is running. One way to do this is to put an at job in your /etc/rc.local script, to start the binary in X minutes.
Is there a way to determine the unshared memory size of a process on SunOS? Specifically the size of a child httpd process running under mod_perl if that helps.
Try pmap -x - see Solaris Internals: Process Memory Map: pmap -x for an example and details.