I want to know how much memory occupies for a HSQLDB in-memory database and further I would like to increase the size of a memory(RAM) so that I can load more data into hsqldb database for processing.
The memory used is part of the Java heap. You can use Java API calls to check how much heap space is used and how much is left. A small HSQLDB database can work in about 32MB of heap space.
You use the Java -Xmx setting to increase the allocated heap when you start your program that embeds HSQLDB, or a standalone HSQLDB server.
Related
Ignite version : 2.12
OS : Windows 10
I am trying to understand ignites heap usage.
I started Ignite server with below command and no special vm args. As suggested by https://ignite.apache.org/docs/latest/quick-start/java
ignite.bat -v ..\examples\config\example-ignite.xml
Post that started analyzing heap usage of same with visualvm tool and the heap usage looks like this
Next thing that I tried is increase the heap memory and restart the server.
Surprisingly Now ignite is consuming even more memory as seen in this graph
I Know the GC is working its way to clear the heap, but why does ignite memory consumption increases with increase in heap space ?
How will this impact a server with ~40-60G memory, how much memory I can expect to be consumed by Ignite?
I'm planning to use ignite as in memory cache along with Cassandra as DB.
Just like Cassandra, Hadoop or Kafka, Ignite is a Java middleware that uses the Java Heap for various needs. But your data is always stored in an off-heap memory that allows utilizing all available memory space without worrying about garbage collection. This gives Ignite complete control over how the data is managed, and ensures the long-term performance of the system.
Ignite uses a page memory model for storing everything, including user data, indices, meta information, etc. This allows Ignite to utilize memory management, improve performance and it also can use the whole disk without any data modifications.
In other words, you might think that direct page memory access is being performed by memory pointers (outside of JVM), but some internal tasks like bootstrapping Ignite itself, performing local SQL processing tasks, etc. do require JVM heap because Ignite itself is written in Java.
Check this and that pages for details.
How will this impact a server with ~40-60G memory, how much memory I
can expect to be consumed by Ignite?
You would need 40-60 GB of RAM + something for JVM itself (Java heap), recommended values might differ, but 2GB of Java heap should be enough.
How to set the parameter - setRAMBufferSizeMB? Is depending on the RAM size of the Machine? Or Size of Data that needs to be Indexed? Or any other parameter? could someone please suggest an approach for deciding the value of setRAMBufferSizeMB.
So, what we have about this parameter in Lucene javadoc:
Determines the amount of RAM that may be used for buffering added
documents and deletions before they are flushed to the Directory.
Generally for faster indexing performance it's best to flush by RAM
usage instead of document count and use as large a RAM buffer as you
can. When this is set, the writer will flush whenever buffered
documents and deletions use this much RAM.
The maximum RAM limit is inherently determined by the JVMs available
memory. Yet, an IndexWriter session can consume a significantly larger
amount of memory than the given RAM limit since this limit is just an
indicator when to flush memory resident documents to the Directory.
Flushes are likely happen concurrently while other threads adding
documents to the writer. For application stability the available
memory in the JVM should be significantly larger than the RAM buffer
used for indexing.
By default, Lucene uses 16 Mb as this parameter (this is the indication to me, that you shouldn't have that much big parameter to have fine indexing speed). I would recommend you to tune this parameter by setting it let's say to 500 Mb and checking how well your system behave. If you will have crashes, you could try some smaller value like 200 Mb, etc. until your system will be stable.
Yes, as it stated in the javadoc, this parameter depends on the JVM heap, but for Python, I think it could allocate memory without any limit.
Usually I set -Xms512m and -Xmx1g so that when JVM starts it allocates 512MB and gradually increases heap to 1GB as necessary. But I see these values set to same say 1g in a dedicated server instance. Is there any advantage for the having both set to the same value?
Well there are couple of things.
Program will start with -Xms value and if the value is lesser it will eventually force GC to occur more frequently
Once the program reaches -Xms heap, jvm request OS for additional memory and eventually grabs -Xmx that requires additional time leading to performance issue, you might as well set it to that at the beginning avoiding jvm to request additional memory.
It is very nicely answered here - https://developer.jboss.org/thread/149559?_sscc=t
From Oracle Java SE 8 docs:
https://docs.oracle.com/javase/8/docs/technotes/guides/vm/gctuning/sizing.html
By default, the virtual machine grows or shrinks the heap at each
collection to try to keep the proportion of free space to live objects
at each collection within a specific range. This target range is set
as a percentage by the parameters -XX:MinHeapFreeRatio=<minimum> and
-XX:MaxHeapFreeRatio=<maximum>, and the total size is bounded below by -Xms<min> and above by -Xmx<max>. Setting -Xms and -Xmx to the same value increases predictability by removing the most important sizing
decision from the virtual machine. However, the virtual machine is
then unable to compensate if you make a poor choice.
if the value of -Xms and -Xmx is same JVM will not have to adjust the heap size and that means less work by JVM and more time to your application. but if the chosen value is a poor choice for -Xms then some of the memory allocated will never be used because the heap will never shrink and if it is a poor choice for -Xmx you will get OutOfMemoryError.
AFAIK One more reason, is that expansion of heap is a stop-the-world event; setting those to the same value will prevent that.
There are some advantages.
if you know the size is going to grow to the maximum, e.g. in a benchmark, you may as well start with the size you know you need.
you can get better performance giving the program more memory that it might naturally give itself. YMWV
In general, I would make the Xms a value I am confident it will use, and the double this for head room for future use cases or situations we haven't tested for. i.e. a size we don't expect but it might use.
In short, the maximum is the point you would rather the program fail than use any more.
Application will suffer frequent GC with lower -Xms value.
Every time asking for more memory from OS with consume time.
Above all, if your application is performance critical then you would certainly want to avoid memory pages swapping out to/from disk as this will cause GC consuming more time. To avoid this, memory can be locked. But if Xms and Xmx are not same then memory allocated after initial allocation will not be locked.
I am tuning my PostgreSQL db effective_cache_size. The PostgreSQL documentation references the expected available memory in PostgreSQL buffer caches to calculate the expected memory available for disk caching. How do I estimate this? Is the shared_buffers the only memory allocated for the buffer caching?
effective_cache_size represents the total memory of the machine minus what you know is used for something else than disk caching.
From Greg's Smith 5-Minute Introduction to PostgreSQL Performance:
effective_cache_size should be set to how much memory is leftover for
disk caching after taking into account what's used by the operating
system, dedicated PostgreSQL memory, and other applications
shared_buffers is considered in this sentence as "dedicated PostgreSQL memory", but other than that, it's not correlated to effective_cache_size.
On Linux if you run free when your system is at its typical memory usage (all applications running and caches are warm), the cached field gives a good value for effective_cache_size.
If you use monitoring tools that produce graphs, you can look at the cached size for long period of times at a glance.
One typical suggestion for a dedicated Postgres server is to set effective_cache_size to about 3/4 of your available RAM. A good tool to use for setting sane defaults is pgtune, which can be found here: https://github.com/gregs1104/pgtune
I'm currently using sun's java 1.6 on a SL4 cluster.
For some reason, the 1.6 JVM is starting up with an impossibly large heap, and cannot start:
java -version
Error occurred during initialization of VM
Could not reserve enough space for object heap
Could not create the Java virtual machine.
If I start it with e.g. -Xmx1800M, then it works OK. So, I'm wondering where the default heap size is set, and more importantly how to change it?
The machine has 8GB of physical memory, and I believe that sun's server JVM is supposed to start with a default of half the memory up to 512M, but this is clearly not the case, as it's trying to allocate over 1800M.
EDIT: I realise that it's possible to use _JAVA_OPTIONS, but this feels a bit clunky; I was expecting a properties file somewhere, but so far I've been unable to find it.
There is no properties file for this. According to Garbage Collector Ergonomics:
initial heap size:
Larger of 1/64th of the machine's
physical memory on the machine or some
reasonable minimum. Before J2SE 5.0,
the default initial heap size was a
reasonable minimum, which varies by
platform. You can override this
default using the -Xms command-line
option.
maximum heap size:
Smaller of 1/4th of the physical
memory or 1GB. Before J2SE 5.0, the
default maximum heap size was 64MB.
You can override this default using
the -Xmx command-line option.
Note: The boundaries and fractions given for the heap size are correct
for J2SE 5.0. They are likely to be
different in subsequent releases as
computers get more powerful.
Given you have 8GB of RAM, default maximum heap size should be 1GB assuming you're using Java 6.
There's no standard properties file. The (Sun) JVM has the default values hardcoded in it.