In our app running on Jdk 8 we use VisualVM to track the usage of loaded classes and the usage of the metaspace.
At some point in time while our app is running we see that the number of loaded classes don't increase any more but the metaspace still increases in it's size while our program is running. So what else apart from classes is stored in metaspace, that could cause that?
While your program is running, some parts of your code may be determined as "hot" by HotSpot's JIT compiler. This will cause those parts to be transformed/compiled to native code, and also some other code may be inlined into it. This native code representation has to go somewhere, and it goes into the same place as other class metadata - the Metaspace.
It explains continuous growth you're seeing: hot parts are determined over time using a simple metric of how much times did that piece of code got executed. Over time more and more code pieces will be JIT'ed as they'll hit threshold set by -XX:CompileThreshold (defaults to 10000)
i am not sure but hier (http://java.dzone.com/articles/java-8-permgen-metaspace) i fund this
Garbage collection of the dead classes and classloaders is triggered once the class metadata usage reaches the “MaxMetaspaceSize”.
maybe is this the cause for increasing metaspace size.
Related
I am currently working on an embedded firmware development which uses FreeRTOS running on an STM32F777II microcontroller. Resource wise, I have around 10 tasks (total sum of stack size will be under 40 KByte) at the same priority, around 4 queues of 1KByte each, 4 binary semaphores. I know this would be an incomplete question without posting the actual code, but I really do not have any specific portion in my firmware that I think will be worth sharing related to my issue. I have a ton of business logic in my code which I cannot fully share as well.
I have a struct which consists of multiple char and int arrays of a specific length. 4 of the tasks uses these structures each. Each structure consumes around 15KByte of space and is defined in the global space of the FreeRTOS environment, not local to a task. The structs are allocated statically only and not dynamically on runtime. And since I initialize few members of the struct when declaring, so they go to the .data section only if I am not mistaken. Until now, there had been absolutely no problem whatsoever in my code and it worked 100% without any issue at all. Now I recently had a requirement where I had to add the same stuct to 2 more tasks. So, I added this 15KByte stuct to one of my tasks, basically just allocated and initialized and did not do any processing in any of the tasks. Observed no problems, nothing, no data corruption, nothing. Now when I allocated one more struct variable of the same type only, what I observe is data corruption in a lot of other places in my project. Some of the queues stopped working correctly and showed garbage data when read. Some of the other buffers also showed data corruption. I am really not sure why just one more variable allocation of this struct is triggering a lot of data corruption at other places in my project. If I remove this one allocation, everything goes back to normal. My MCU has 512KB of RAM and as per the IDE's build analyzer feature, it showed below 40% RAM usage, so what is triggering this issue, any suggestions to try? Could be because of some overlapping of .data or .bss sections or something? I did not observe any stack overflows or hard faults in the system during this.
For a quick resolution,
I randomly just disabled the D-cache by commenting out the function:
SCB_EnableDCache();
and voila, everything started to function correctly as it should without any instances of data corruption.
For long run and correct resolution:
Looks like there are some latent issues with my coding. I need to review the memory use, and regions of memory with different properties. Look at the buses, review any DMA usage, and MPU memory settings. Also, review the correct usage of volatile memory directives, thread-safe operation, and cache-coherency issues. Also, Use memory fencing and cache flushing as appropriate.
More details: Level 1 cache on STM32F7 Series and STM32H7 Series
I had three projects open. One of them - Spark - was very large. Upon closing spark there was NO difference in memory usage - as reported by os/x activity monitor. Note: all projects are opened within the same Intellij instance.
It is in fact using just over 4GB. And I only now have two projects open. Those two projects only take up 1.5GB if I shut down Intellij and start it up again.
So .. what to do to "encourage" Intellij to release the memory it is using? It is running very very slowly (can not keep up with my typing for example)
Update I just closed the larger of the two remaining projects. STILL no reduction in memory usage. The remaining project is a single python file. So Intellij should be using under 512Meg at this point!
Following up on #PeterGromov's answer it seems that is were difficult to obtain the memory back. In addition #KevinKrumwiede mentioned the XX:MaxHeapFreeRatio which appears to be an avenue.
Here are a couple of those ideas taken bit farther from Does GC release back memory to OS?
The HotSpot JVM does release memory back to the OS, but does so
reluctantly.
You can make it more aggressive by setting -XX:GCTimeRatio=19
-XX:MinHeapFreeRatio=20 -XX:MaxHeapFreeRatio=30 which will allow it to spend more CPU time on collecting and constrain the amount of
allocated-but-unused heap memory after a GC cycle.
Additionally with Java 9 -XX:-ShrinkHeapInSteps option can be be used
to apply the shrinking caused by the previous two options more
aggressively. Relevant OpenJDK bug.
Do note that shrinking ability and behavior depends on the chosen
garbage collector. For example G1 only gained the ability to yield
back unused chunks in the middle of the heap with jdk8u20.
So if heap shrinking is needed it should be tested for a particular
JVM version and GC configuration.
and from How to free memory in Java?
To extend upon the answer and comment by Yiannis Xanthopoulos and Hot
Licks (sorry, I cannot comment yet!), you can set VM options like this
example:
-XX:+UseG1GC -XX:MinHeapFreeRatio=15 -XX:MaxHeapFreeRatio=30 In my jdk 7 this will then release unused VM memory if more than 30% of the heap
becomes free after GC when the VM is idle. You will probably need to
tune these parameters.
While I didn't see it emphasized in the link below, note that some
garbage collectors may not obey these parameters and by default java
may pick one of these for you, should you happen to have more than one
core (hence the UseG1GC argument above).
I am going to add the -XX:MaxHeapFreeRatio to IJ and report back if it were to help.
Our application presently only runs on Java7 so the first approach above is not yet viable - but there is hope since our app is moving to jdk8 soon.
https://www.jetbrains.com/help/idea/status-bar.html
I used this:
Shows the current heap level and memory usage. Visibility of this section in the Status bar is defined by the Show memory indicator check box in the Appearance page of the Settings/Preferences dialog. It is not shown by default.
Click the memory indicator to run the garbage collector.
The underlying Java virtual machine supports only growing of its heap. So even if after closing all projects the IDE doesn't need all of it, it's still allocated and counted as used in the OS.
I have this app written in VB.Net with winforms that shows some stats and pictures on a bigscreen monitor. I also monitor the memory usage of sad app by using this.
Process.WorkingSet64
I know windows does not always report the correct usage but I just wanted to know if I didn't have any little memory leaks which I had but are solved now. But the first week the memory usage was around 100MB and the second week the memory usage showed around 50MB.
So why did it all of a sudden drop while still running the exact same code?
I can hardly imagine that the garbage collector kicked in this late since the app refreshes every 10 seconds and it has ample time in between those periods to do it's thing.
Or perhaps there is just better way to get memory usage for a process that is more reliable.
Process.WrokingSet64 doesn't report the memory usage, it omits the memory that is swapped to disk:
The value returned by this property represents the current size of working set memory used by the process. The working set of a process is the set of memory pages currently visible to the process in physical RAM memory. These pages are resident and available for an application to use without triggering a page fault. (MSDN)
Even if your system was never low on free memory, you may have minimized the application window, which caused Windows to trim its working set.
If you want to look for memory leaks you should probably use Process.PrivateMemorySize64 instead. Your shared memory is going to contain just executable code and it's going to remain more or less constant throughout the life of the process, so you should focus on the private memory.
just as the topic suggests I've come across a slight issue with boost::serialization when serializing a huge amount of data to a file. The problem consists of the memory footprint of the serialization part of the application taking around 3 to 3.5 times the memory of my objects being serialized.
It is important to note that the data structure I have is a three dimensional vector of base class pointers and a pointer to that structure. Like this:
using namespace std;
vector<vector<vector<MyBase*> > >* data;
This is later serialised with a code analog to this one:
ar & BOOST_SERIALIZATION_NVP(data);
boost/serialization/vector.hpp is included.
Classes being serialised all inherit from "MyBase".
Now, since the start of my project I've used different archives for serialization from typical binary_archive, text, xml and finally polymorphic binary/xml/text. Every single one of these acts exactly the same way.
Typically this wouldn't be a problem if I had to serialize small amounts of data but the number of classes I have are in the milions (ideally around 10 milion) and the memory usage as I've been able to test it shows consistently that the memory allocated by boost::serialization part of the code is around 2/3 of the application whole memory footprint while writing the file.
This amounts to around 13.5 GB of RAM taken for 4 milion objects where the objects themselves take 4.2GB. Now this is as far as I've been able to take my code since I don't have access to a machine with more than 8GB of physical RAM. I should also note that this is a 64bit application being run on a Windows 7 professional x64 edition but the situation is similar on an Ubuntu box.
Anyone has any idea how I would go about troubleshooting this as it is unacceptable for me to have such high memory requirements for an application that will not use as much memory while running as it does while serializing.
Deserialization isn't as bad, as it allocates around 1.5 times the needed memory. This is something I could live with.
Tried turning tracking off with boost::archive::archive_flags::no_tracking but it acts exactly the same.
Anyone have any idea what I should do?
Using valgrind I found that the main reason of memory consumption is a map inside the library to track pointers. If you are certain that you do not need pointer tracking ( it means you are sure that there is no pointer aliasing) disable tracking. You can find here the main concepts of disable tracking. In short you must do something like this:
BOOST_CLASS_TRACKING(vector<vector<vector<MyBase*> > >, boost::serialization::track_never)
In my question I wrote a version of this macro that you could disable tracking of a template class. This must have a significant impact on your memory consumption.
Also notice that there are pointers inside any containers If you want tracking never you must disable tracking of them too. Currently I could not find any way to do this properly.
I have a WebSphere Portal application running four instances on a single box and after about 7 days of runtime there is only 130-150mb of address space free in native memory (using PMAP). Somewhere in another 7-10 days the figure drops well below 100mb (which we deem dangerous and we start to recycle the JVM). If we don't do the recycle, the JVM will eventually crash with a SIGSEGV signal.
We've done some initial investigation into class counts and the size of JIT code. Class counts grow, but slowly from 50k onwards...about a couple hundred per day. JITC sizes get to about 210 MB after 7 days and grow about 1 MB per day after that so. In our previous experience we don't find these to be sinister values.
What we need to to be able to break down what is in the native heap, whether it is threads (all thread counts appear normal and we have fixed thread pools), String pools, constant pools, bytecode, or whatever else.
One lead we are trying now is reducing the reflection threshold to 0 (shutting off the bytecode accessors for reflectively created classes). This app uses a lot of pointcutting and a lot of reflection, so we're hoping there's a good chance this helps.
Any advice is welcome.
Might be a bit of back-and-forth, but have you GC logged and ensured it's not growing Java heap over time? Looked at your perm space? The SIGSEGV is an interesting one though, I'd expect a more JVM-ish crash for any in-Java mem issues.
After lengthy investigation, this ended up being a WebSphere bug: PK72252: CALLS TO CLASSLOADER.GETRESOURCEASSTREAM ARE SLOW. Fixed in 6.0.2.33.