This is sort of related to a previous, yet so far unsuccessful question of mine. I have a daemon that is placed in the LaunchAgents folder (on Mac) and it should run perpetually in the background, but after a couple of days it just stops for no apparent reason. I have no idea why and thus my question:
What are the reasons that a daemon might randomly stop?
Thanks for the help!
A Daemon is just a long lasting (forked) process. The reason a Daemon crashes is the same any other program crashes:
attempting to read or write memory
that is not allocated for reading or
writing by that application
(segmentation fault) or x86 specific
(general protection fault)
attempting to execute privileged or
invalid instructions
attempting to perform I/O operations
on hardware devices to which it does
not have permission to access
passing invalid arguments to system
calls
attempting to access other system
resources to which the application
does not have permission to access
(bus error)
attempting to execute machine
instructions with bad arguments
(depending on CPU architecture):
divide by zero, operations on denorms
or NaN values, memory access to
unaligned addresses, etc.
Since it's a LaunchAgent, it runs as part of your login session, and hence will be killed if you log out.
On the other hand, if it's dying before you log out, and you can't find/fix whatever is causing it to crash/exit, or you can tell launchd to automatically restart it by adding
<key>KeepAlive</key>
</true>
to its .plist
Related
Background
A while ago I wrote a Raku module to test out some ideas I had for making external process pipelines (e.g., grep raku | wc -l). I followed the traditional way of creating Pipes and setting up I/O redirections for the external processes.
Since there's no fork available in Raku, nor is it recommended due to threads in MoarVM, I resorted to using Proc::Async to start a wrapper process for each external command in a pipeline, and the wrapper would then sets up I/O redirections to connect the command processes and the pipes, and then close other FDs before exec-ing the real command.
Problem
This all seems to work well except there's still the possibility that other parts of the program using the module might start a sub process (e.g., via run, shell, Proc, or Proc::Async) in another thread after the pipe FDs are created but before they are closed, thus leaking them and causing the command processes in the pipeline to block on read / write.
Question
How would you prevent FDs from leaking to any child processes started in another thread in Raku?
Thanks
I'm trying to generate a full heapdump from Weblogic 10.3.6 due to an OutOfMemoryError generated by a Web Application deployed on the Server.
I've setted the following start script:
-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/path/to/heapdump
When the OutOfMemoryError occurs, Weblogic generates an empty hprof file (0 bytes size) in /path/to/heapdump folder, and nothing happens: the Server remains in RUNNING mode, even if is not reachable anymore.
The java process is still alive, but with 0% of processor.
Even the server.out log seems completely frozen, without any trace of the OutOfMemoryError.
What's wrong with the configuration?
Probably you can use Java Flight Recorder to save events and check which objects are generating OOM.
(any profiler should work as well).
Been there :( . I remember at the time that we've found it was somewhat logical since there was not enough memory for normal operation, the JVM could not automagically find enough memory to create a heapdump either. If memory serves me well, at that time we did 2 things to debug the memory leak. First we were "lucky" enough that the problem was happening fairly regularly so a close manual monitoring was possible (monitoring of the gc.log looking for repeated FullGC and monitoring of the performance tab in the console). Knowing when the onset of the problem was starting we were doing some kill -3 to get the dump manually. We also used jstack {PID} (JDK 1.6 on Linux) with some luck. With those, at the time, the devs were able to identify the memory leak. Hope that helps.
Okay, your configuration looks alright.. you might want to check if the weblogic process user has the rights to edit the heap dump file.
You can take heap dump by Java tools :
JAVA_HOME/bin/jmap -dump:format=b,file=path_of_the_file
OR
%JROCKIT_HOME%\bin\jrcmd hprofdump filename=path_of_the_file
I have a service that scans network folders using a parallel.for method.
However recently I am finding if I stop the service then while windows says the service is stopped the process is still running in task manager. However it is at 0 cpu and the memory does not change. If I try and end the task (even a force in command prompt) it just says access denied and i have to reboot the server.
What would be the best way to make sure everything terminates?
I thought of adding a global Boolean that in the stop procedure it turns true and part of my parallel code will check for that and call s.stop.
Thank you
In brief, when your service is stopped, it needs to cancel all pending and running operations, then wait for those operation to actually finish. Check out the MSDN reference for Task Cancellation.
I have created a web application in Apache Cocoon.This website is running properly but after every 3-4 days, it stops responding. It doesn't run until and unless, we restart the tomcat service. In the catalina.2011-05-09.log file, it shows following error:-
"May 9, 2011 3:17:34 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/webresources] is still processing a request that has yet to finish. This is very likely to create a memory leak. You can control the time allowed for requests to finish by using the unloadDelay attribute of the standard Context implementation."
I am not been able to understand the cause of this problem. Can someone suggest me how to resolve this issue?
You are using a library that is starting one or more threads and is not properly shutting them down or releasing other resources captured by the thread. This often happens with things like Apache HTTP components (I get this error with Http Components) and anything that uses separate threads internally. What libraries are you using in your Cocoon application?
It is telling you the issue:
[...] is still processing a request that has yet to finish
You need to find out what that request is/is going to. One easy way is to have something like PsiProbe installed.
Also, it's not a bad idea to restart Tomcat every night. It can help alleviate these kinds of issues until you find the root cause.
I am currently experiencing some instability in my session variables and believe the app pool is where the error is coming from. What I cannot find is a list of possible culprits for the issue. What can cause the app pool to recycle on its own, other than a scheduled recycle?
Common reasons why your application pool may unexpectedly recycle
EDIT: Full Text in the event that the link goes 404:
If your application crashes, hangs and deadlocks it will cause/require the application pool to recycle in order to be resolved, but sometimes your application pool inexplicably recycles for no obvious reason. This is usually a configuration issue or due to the fact that you're performing file system operations in the application directory.
For the sake of elimination I thought I'd list the most common reasons.
Application pool settings
If you check the properties for the application pool you'll see a number of settings for recycling the application pool. In IIS6 they are:
Recycle worker processes (in minutes)
Recycle worker process (in requests)
Recycle worker processes at the following times
Maximum virtual memory
Maximum used memory
These settings should be pretty self explanatory, but if you want to read more, please take a look at this MSDN article
The processModel element of machine.config
If you're running IIS5 or the IIS5 isolation mode you'll have to look at the processModel element. The Properties you should pay the closest attention to are:
memoryLimit
requestLimit
timeout
memoryLimit
The default value of memoryLimit is 60. This value is only of interest if you have fairly little memory on a 32 bit machine. 60 stands for 60% of total system memory. So if you have 1 GB of memory the worker process will automatically restart once it reaches a memory usage of 600 MB. If you have 8 GB, on the other hand, the process would theoretically restart when it reaches 4,8 GB, but since it is a 32 bit process it will never grow that big. See my post on 32 bit processes for more information why.
requestLimit
This setting is "infinite" by default, but if it is set to 5000 for example, then ASP.NET will launch a new worker process once it's served 5000 requests.
timeout
The default timeout is "infinite", but here you can set the lifetime of the worker process. Once the timeout is reached ASP.NET will launch a new worker process, so setting this to "00:05:00" would recycle the application every five minutes.
Other properties
There are other properties within the processModel element that will cause your application pool to recycle, like responseDeadlockInterval. But these other settings usually depend on something going wrong or being out of the ordinary to trigger. If you have a deadlock then that's your main concern. Changing the responseDeadlockInterval setting wouldn't do much to resolve the situation. You'd need to deal with the deadlock itself.
Editing and updating
ASP.NET 2.0 depends on File Change Notifications (FCN) to see if the application has been updated. Depending on the change the application pool will recycle. If you or your application is adding and removing directories to the application folder, then you will be restarting your application pool every time, so be careful with those temporary files.
Altering the following files will also trigger an immediate restart of the application pool:
web.config
machine.config
global.asax
Anything in the bin directory or it's sub-directories
Updating the .aspx files, etc. causing a recompile will eventually trigger a restart of the application pool as well. There is a property of the compilation element under system.web that is called numRecompilesBeforeAppRestart. The default value is 20. This means that after 20 recompiles the application pool will recycle.
A workaround to the sub-directory issue
If your application really depends on adding and removing sub-directories you can use linkd to create a directory junction. Here's how:
Create a directory you'd like to exclude from the FCN, E.g. c:\inetpub\wwwroot\WebApp\MyDir
Create a separate folder somewhere outside the wwwroot. E.g. c:\MyExcludedDir
use linkd to link the two: linkd c:\inetpub\wwwroot\WebApp\MyDir c:\MyExcludedDir
Any changes made in the c:\inetpub\wwwroot\WebApp\MyDir will actually occur in c:\MyExcludedDir so they will go unnoticed by the FCN.
Is recycling the application pool really that bad?
You really shouldn't have to recycle the application pool, but if you're dealing with a memory leak in your application and need to buy time to fix it, then by all means recycling the application pool could be a good idea.
What about session state?
Well, if you're running in-process session state, then obviously it's going to be reset each and every time the application pool is recycled. If you need to brush up on your state server options, then I recommend taking a look at this entry.