It is not clear what the difference is between MaxDirectMemorySize and Dstorage.diskCache.bufferSize. They both seem to be off heap. Are they redundant when both are specified?
Docs : The size of direct memory consumed by OrientDB is limited by
the size of the disk cache (variable storage.diskCache.bufferSize). https://orientdb.com/docs/2.2/Embedded-Server.html
The docs seem to imply that they refer to the same space, but that the driectmemorysize is limited by the buffer size. Is this correct?
MaxDirectMemorySize is a JVM setting that limits all direct byte buffer allocations within that JVM instance.
storage.diskCache.bufferSize is an application setting that limits direct byte buffer allocations for the purpose of IO caching in orientdb.
Related
Summary:
I have a Java application that uses akka streams that's using more memory than I have specified the jvm to use. The below values are what I have set through the JAVA_OPTS.
maximum heap size (-Xmx) = 700MB
metaspace (-XX) = 250MB
stack size (-Xss) = 1025kb
Using those values and plugging them into the formula below, one would assume the application would be using around 950MB. However that is not the case and it's using over 1.5GB.
Max memory = [-Xmx] + [-XX:MetaspaceSize] + number_of_threads * [-Xss]
Question: Thoughts on how this is possible?
Application overview:
This java application uses alpakka to connect to pubsub and consumes messages. It utilizes akka stream's parallelism where it performs logic on the consumed messages and then it produces those messages to a kafka instance. See the heap dump below. Note, the heap is only 912.9MB so something is taking up 587.1MB and getting the memory usage over 1.5GB
Why is this a problem?
This application is deployed on a kubernetes cluster and the POD has a memory limit specified to 1.5GB. So when the container, where the java application is running, consumes more that 1.5GB the container is killed and restarted.
The short answer is that those do not account for all the memory consumed by the JVM.
Outside of the heap, for instance, memory is allocated for:
compressed class space (governed by the MaxMetaspaceSize)
direct byte buffers (especially if your application performs network I/O and cares about performance, it's virtually certain to make somewhat heavy use of those)
threads (each thread has a stack governed by -Xss ... note that if mixing different concurrency models, each model will tend to allocate its own threads and not necessarily provide a means to share threads)
if native code is involved (e.g. perhaps in the library Alpakka is using to interact with pubsub?), that can allocate arbitrary amounts of memory outside of the heap)
the code cache (typically 48MB)
the garbage collector's state (will vary based on the GC in use, including the presence of any tunable options)
various other things that generally aren't going to be that large
In my experience you're generally fairly safe with a heap that's at most (pod memory limit minus 1 GB), but if you're performing exceptionally large I/Os etc. you can pretty easily get OOM even then.
Your JVM may ship with support for native memory tracking which can shed light on at least some of that non-heap consumption: most of these allocations tend to happen soon after the application is fully loaded, so running with a much higher resource limit and then stopping (e.g. via SIGTERM with enough time to allow it to save results) should give you an idea of what you're dealing with.
My ignite server have 128G RAM, with Xmx 10G off-heap 70G, when start, the log shows:
[11:30:27,376][INFO][main][IgniteKernal] Performance suggestions for grid (fix if possible)
[11:30:27,377][INFO][main][IgniteKernal] To disable, set -DIGNITE_PERFORMANCE_SUGGESTIONS_DISABLED=true
[11:30:27,377][INFO][main][IgniteKernal] ^-- Set max direct memory size if getting 'OOME: Direct buffer memory' (add '-XX:MaxDirectMemorySize=<size>[g|G|m|M|k|K]' to JVM options)
I have search web, and i found this article said it is not necessary to configure MaxDirectMemorySize, http://apache-ignite-users.70518.x6.nabble.com/Do-we-require-to-set-MaxDirectMemorySize-JVM-parameter-td21200.html
and some articles says the default MaxDirectMemorySize will be same as Xmx, so what should i configure for this option, i am just confused, and if it is no useful, why ignite dump that suggestion log to fix this problem?
This is not an indication of failure, you can just ignore this suggestion unless your node/cluster failing due to OOM in direct buffer memory. This is an option to give you ability to control how many direct memory could be allocated, otherwise it is controlled using default direct memory policy by JVM you are using. Ignite only checks if it's set in JVM options.
Do you experience any issues with OOME in direct buffer memory in your app?
Regards.
Direct buffer memory is used by some file operations (like read and write) when the program calls functions from the NIO libray.
Due to a bug, its value if not specified and you set Xmx ... it is copied from Xmx.
Direct buffer memory default is 64 Mb (if you don't set it ans also don't set Xmx).
I suggest MaxDirectMemorySize=64 or 256 Mb
Bigger values: perhaps you don't see errors, but I doubt you get better performance.
How to set the parameter - setRAMBufferSizeMB? Is depending on the RAM size of the Machine? Or Size of Data that needs to be Indexed? Or any other parameter? could someone please suggest an approach for deciding the value of setRAMBufferSizeMB.
So, what we have about this parameter in Lucene javadoc:
Determines the amount of RAM that may be used for buffering added
documents and deletions before they are flushed to the Directory.
Generally for faster indexing performance it's best to flush by RAM
usage instead of document count and use as large a RAM buffer as you
can. When this is set, the writer will flush whenever buffered
documents and deletions use this much RAM.
The maximum RAM limit is inherently determined by the JVMs available
memory. Yet, an IndexWriter session can consume a significantly larger
amount of memory than the given RAM limit since this limit is just an
indicator when to flush memory resident documents to the Directory.
Flushes are likely happen concurrently while other threads adding
documents to the writer. For application stability the available
memory in the JVM should be significantly larger than the RAM buffer
used for indexing.
By default, Lucene uses 16 Mb as this parameter (this is the indication to me, that you shouldn't have that much big parameter to have fine indexing speed). I would recommend you to tune this parameter by setting it let's say to 500 Mb and checking how well your system behave. If you will have crashes, you could try some smaller value like 200 Mb, etc. until your system will be stable.
Yes, as it stated in the javadoc, this parameter depends on the JVM heap, but for Python, I think it could allocate memory without any limit.
Usually I set -Xms512m and -Xmx1g so that when JVM starts it allocates 512MB and gradually increases heap to 1GB as necessary. But I see these values set to same say 1g in a dedicated server instance. Is there any advantage for the having both set to the same value?
Well there are couple of things.
Program will start with -Xms value and if the value is lesser it will eventually force GC to occur more frequently
Once the program reaches -Xms heap, jvm request OS for additional memory and eventually grabs -Xmx that requires additional time leading to performance issue, you might as well set it to that at the beginning avoiding jvm to request additional memory.
It is very nicely answered here - https://developer.jboss.org/thread/149559?_sscc=t
From Oracle Java SE 8 docs:
https://docs.oracle.com/javase/8/docs/technotes/guides/vm/gctuning/sizing.html
By default, the virtual machine grows or shrinks the heap at each
collection to try to keep the proportion of free space to live objects
at each collection within a specific range. This target range is set
as a percentage by the parameters -XX:MinHeapFreeRatio=<minimum> and
-XX:MaxHeapFreeRatio=<maximum>, and the total size is bounded below by -Xms<min> and above by -Xmx<max>. Setting -Xms and -Xmx to the same value increases predictability by removing the most important sizing
decision from the virtual machine. However, the virtual machine is
then unable to compensate if you make a poor choice.
if the value of -Xms and -Xmx is same JVM will not have to adjust the heap size and that means less work by JVM and more time to your application. but if the chosen value is a poor choice for -Xms then some of the memory allocated will never be used because the heap will never shrink and if it is a poor choice for -Xmx you will get OutOfMemoryError.
AFAIK One more reason, is that expansion of heap is a stop-the-world event; setting those to the same value will prevent that.
There are some advantages.
if you know the size is going to grow to the maximum, e.g. in a benchmark, you may as well start with the size you know you need.
you can get better performance giving the program more memory that it might naturally give itself. YMWV
In general, I would make the Xms a value I am confident it will use, and the double this for head room for future use cases or situations we haven't tested for. i.e. a size we don't expect but it might use.
In short, the maximum is the point you would rather the program fail than use any more.
Application will suffer frequent GC with lower -Xms value.
Every time asking for more memory from OS with consume time.
Above all, if your application is performance critical then you would certainly want to avoid memory pages swapping out to/from disk as this will cause GC consuming more time. To avoid this, memory can be locked. But if Xms and Xmx are not same then memory allocated after initial allocation will not be locked.
I'm creating a list of elements inside a task in the following way:
l = (dllist*)pvPortMalloc(sizeof(dllist));
dllist is 32 byte big.
My embedded system has 60kB SRAM so I expected my 200 element list can be handled easily by the system. I found out that after allocating space for 8 elements the system is crashing on the 9th malloc function call (256byte+).
If possible, where can I change the heap size inside freeRTOS?
Can I somehow request the current status of heap size?
I couldn't find this information in the documentation so I hope somebody can provide some insight in this matter.
Thanks in advance!
(Yes - FreeRTOS pvPortMalloc() returns void*.)
If you have 60K of SRAM, and configTOTAL_HEAP_SIZE is large, then it is unlikely you are going to run out of heap after allocating 256 bytes unless you had hardly any heap remaining before hand. Many FreeRTOS demos will just keep creating objects until all the heap is used, so if your application is based on one of those, then you would be low on heap before your code executed. You may have also done something like use up loads of heap space by creating tasks with huge stacks.
heap_4 and heap_5 will combine adjacent blocks, which will minimise fragmentation as far as practical, but I don't think that will be your problem - especially as you don't mention freeing anything anywhere.
Unless you are using heap_3.c (which just makes the standard C library malloc and free thread safe) you can call xPortGetFreeHeapSize() to see how much free heap you have. You may also have xPortGetMinimumEverFreeHeapSize() available to query how close you have ever come to running out of heap. More information: http://www.freertos.org/a00111.html
You could also define a malloc() failed hook (http://www.freertos.org/a00016.html) to get instant notification of pvPortMalloc() returning NULL.
For the standard allocators you will find a config option in FreeRTOSConfig.h .
However:
It is very well possible you run out of memory already, depending on the allocator used. IIRC there is one that does not free() any blocks (free() is just a dummy). So any block returned will be lost. This is still useful if you only allocate memory e.g. at startup, but then work with what you've got.
Other allocators might just not merge adjacent blocks once returned, increasing fragmentation much faster than a full-grown allocator.
Also, you might loose memory to fragmentation. Depending on your alloc/free pattern, you quickly might end up with a heap looking like swiss cheese: Many holes between allocated blocks. So while there is still enough free memory, no single block is big enough for the size required.
If you only allocate blocks that size there, you might be better of using your own allocator or a pool (blocks of fixed size). Thaqt would be statically allocated (e.g. array) and chained as a linked list during startup. Alloc/free would then just be push/pop on a stack (or put/get on a queue). That would also be very fast and have complexity O(1) (interrupt-safe if properly written).
Note that normal malloc()/free() are not interrupt-safe.
Finally: Do not cast void *. (Well, that's actually what standard malloc() returns and I expect that FreeRTOS-variant does the same).