Just curious, Is there a way to measure the heap memory consumed by each individual object from inside the code in realtime?
(I know I can use VisualVM and Eclipse Memory Analyzer as answered here, but is there a way to do this from the inside?)
Since you don't want to you the free tools available, you can look into the Hprof (freely available)code, try to change it in your own way. Hprof internally uses JVMTI . It will work.
Related
Recently, I've been digging in to JVM heap dumps using Eclipse MAT. I like it, but the one feature that I seem to use the most is the Dominator Tree. Eclipse's example screenshot:
Anyways, I find that a lot of the time, I usually get the most value out of just looking at that table and getting the first few entries. Since the turnaround time for getting this is:
Create Heap Dump (jcmd <pid> GC.heap_dump)
Download/Pull heapdump to a location (MAT isn't installed on our servers)
Run Eclipse MAT's ParseHeapDump.sh tool to build the various trees
Open MAT, click Dominator Tree icon.
Analyze
Is there a way to get this equivalent information off of a running JVM programmatically? I'd like to run some kind of gather_dominators.sh <pid> script on a host and get the Top X Objects from a JVM, but I don't know where to start.
If by "running jvm" You meant - "getting the info without doing stop-world heap-dump" then the obvious answer is: in order to do such thing without "full-scan" - the data needs to be collected throughout system life-time by tapping creation/release of each object and by maintaining the statistics. You could achieve such things by instrumentation or by using a ready-made custom agents (jol/jamm/etc). Note that many GCs are already doing similar work to collect (and print) statistics. IIRC - newer JVMs even keep track of such info within the class-metadata area (so getting statistics is instant).
https://github.com/google/allocation-instrumenter
(google-allocation-instumenter)
http://blog.javabenchmark.org/2013/07/compute-java-object-memory-footprint-at.html (with JAMM)
https://github.com/jbellis/jamm (JAMM src)
In Java, what is the best way to determine the size of an object? (JOL/etc here)
http://www.javaworld.com/article/2074458/core-java/estimating-java-object-sizes-with-instrumentation.html (short DYI guide)
https://www.youtube.com/results?search_query=Understanding+Java+GC
(webCast on how GC traverses objects for similar purposes)
On other hand - if You're fine to grab a heap-dump (which should be fine on any production system with any proper node-redundancy in place, designed for handling unavoidable Sun-JVM stop-world GC pauses), then Jhat, MAT-api, YourKit and Jol are probably Your best friends:
Programmatically analyze java heap dump file
How to analyse the heap dump using jmap in java
It is important to note that currently-existing heap-dump format loses the info about actual sizes of objects, so all tools (MAT/etc) are just trying to GUESS it properly:
http://shipilev.net/blog/2014/heapdump-is-a-lie/ (What Heap Dumps Are Lying To You About, by Aleksey Shipilёv)
HTH :)
I know this is kind of a weird question, but how do you...allocate physical memory? I know using New will make a new object but it's not allocating what I'm looking for. Here's kind of what I'm looking for: http://www.soft.tahionic.com/download-memalloc/index.html
That program allocates memory in the way I want to. How would I go about allocating around...say 500 MB? Or will VB.NET not allow this because of it's memory management? I tried googling about memorystreams and unmanagedmemorystreams but I'm not sure how to start. I also tried making large arrays but that seems kind of...unprofessional. I've only been using VB.NET for a year or so. Can someone help me get started? By the way, I just joined. Nice to meet you all!
You can allocate and free a specified block of unmanaged memory like this:
Dim handle As IntPtr = Marshal.AllocHGlobal(size)
Marshal.FreeHGlobal(handle)
See the MSDN for more info. You could alternatively use the Marshal.AllocCoTaskMem method and free it with Marshal.FreeCoTaskMem.
utilize the windows api functions such as HeapAlloc using pinvoke
I've recently converted my iOS project to ARC. I have two targets in my project. One is the application itself, and the other is a set of GHUnit tests. I have approximately 200 tests that do quite a lot of work in terms of creating and modifying Core Data objects. The Core Data store used by the tests is an in memory store, and is thrown away once the tests are finished (i.e. it is not persisted anywhere).
When my tests have been running a while (they never reach the exact same point before the error is thrown, but it is always around the same tests) the application crashes with an EXC_BAD_ACCESS (Code=2, address=...)
The output in the console is as follows:
I've followed the instructions here in this answer, and set my main.m file of the GHUnit target to use the -fno-objc-arc compiler flag, but that doesn't seem to have helped.
I don't really understand what these errors mean, and searching for them doesn't seem to have helped. My only guess is that I am running out of memory, but I'm not sure why or how, considering ARC should be releasing objects for me.
I'd really appreciate any help anyone can give me to fix this! If you have any questions just leave me a comment and I will get back to you asap!
Thanks!
Chris,
First, As you have a memory exhaustion problem, you should look at your tests running under the Instruments allocation tool. Remember to turn on VM automatic snapshots. Then you should mark the heap multiple times as the tests execute.
Second, while this may be related to ARC, it quite possibly isn't. In general, ARC apps, because they can automatically release objects sooner, have a smaller footprint than MRR apps. The move to a new compiler with different options may just be uncovering a pre-existing problem.
Third, because you are using an in-memory database, my first test would be to just change it to a SQLite DB. It can have a much smaller footprint. (While you may choose to return to in-memory DBs later, we're trying to find the cause of your memory exhaustion. An in-memory DB can use a lot of RAM. Hence, lets take is out of the equation.
Once you've done the 1st and 3rd tasks above, please report back your results.
Andrew
I'm using phantomjs with casperjs to run multiple tests and it looks like each instance of PhantomJS takes ~106M of RAM. Is it possible to reduce that amount? Is there a simple way to run tests in multiple "tabs"?
Edit: As pointed by #newfurniturey, release is now deprecated. We must use close:
http://phantomjs.org/api/webpage/method/close.html
Don't know if that helps, but the release function could be a good tail :
http://phantomjs.org/api/webpage/method/release.html
Releases memory heap associated with this page. Do not use the page
instance after calling this.
Due to some technical limitation, the web page object might not be
completely garbage collected. This is often encountered when the same
object is used over and over again. Calling this function may stop the increasing heap allocation.
:)
I'd like to have a small (not doing too damn much) daemon running on a little server, watching a directory for new files being added to it (and any directories in the main one), and calling another Clojure program to deal with that new file.
Ideally, each file would be added to a queue (a list represented by a ref in Clojure?) and the main process would take care of those files in the queue on a FIFO basis.
My question is: is having a JVM up running this little program all the time too much a resource hog? And do you have any suggestions as to how go about doing this?
Thank you very much!
EDIT: Another question I should ask: should I run this as its own instance (using less memory) and have it launch a new JVM when a file is seen, or have it on the same JVM the Clojure code that will process the file?
As long as it is running fine now and it has no memory leaks it should be fine.
From the daemon terminology I gather it is on a unix clone, and in this case best is to start it from an init script, or from the rc.local script. Unfortunately details differ from OS to OS to be more specific.
Limit the memry using -Xmx=64m or something to make sure it fails before taking down the rest of the services. Play a bit with the number to find the lowest reliable size.
Also, since clojures claim to fame is its ability to deal with concurrency it make a lot of sense to only run one JVM with all functionality running on it in multiple threads. The overhead of spawning new processes is already very big and if it is a JVM which needs to JIT and warm up its memory management, doubly so. On a resource constrained machine could pose a problem. and on a resource rich machine this is a waste.
I always found that the JVM is not made to quickly run something script like and exit again. It is really not made for that use case in my opinion
.