I want to understand when a cache is created with native persistence enabled, will it store the data in the defined data region/RAM and in the disk at the same time? Is there any way I can restrict the disk utilization for storing the data?
Additionally, in a cluster of 3 due to any reason the disk got full for one of the nodes and there is not enough memory available, what will be the impact on the cluster?
Yes, data will be stored both in RAM and on the disk. I does not have to fit in RAM at the same time.
If you run out of disk space, your persistent store will likely be corrupted.
Related
How can I limit maximum size on disk when using Ignite Persistence? For example, my data set in a database is 5TB. I want to cache maximum of 50GB of data in memory with no more than 500GB on disk. Any reasonable eviction policy like LRU for on-disk data will work for me. Parameter maxSize controls in-memory size and I will set it to 50GB. What should I do to limit my disk storage to 500GB then? Looking for something like maxPersistentSize and not finding it there.
Thank you
There is no direct parameter to limit the complete disk usage occupied by the data itself. As you mentioned in the question, you can control in-memory regon allocation, but when a data region is full, data pages are going to be flushed and loaded on demand to/from the disk, this process is called page replacement.
On the other hand, page eviction works only for non-persistent cluster preventing it from running OOM. Personally, I can't see how and why that eviction might be implemented for the data stored on disk. I'm almost sure that other "normal" DBs like Postgres or MySQL do not have this option either.
I suppose you might check the following options:
You can limit WAL and WAL archive max sizes. Though these items are rather utility ones, they still might occupy a lot of space [1]
Check if it's possible to use expiry policies on your data items, in this case, data will be cleared from disk as well [2]
Use monitoring metrics and configure alerting to be aware if you are close to the disk limits.
I understand the native persistence mode of Apache Ignite allows to store as much as possible data in memory - and the potential remaining data on disk.
Is it possible to manually choose which table I want to store in memory and which I want to store EXCLUSIVELY on disk? If I want to save costs, should I just give Ignite a lot of disk space and just a small amount of memory? What if I know some tables should return results as fast as possible while other tables have lower priorities in terms of speed (even if they are accessed more often)? Is there any feature to prioritize data storage into memory at table level or any other level?
You can define two different data regions - one with small amount of memory and enabled persistence and second without persistence, but with bigger max memory size: https://apacheignite.readme.io/docs/memory-configuration
You can't have a cache (which contains rows for a table) to be stored exclusively on disk.
When you add a row to table it gets stored in Durable Memory, which is always located in RAM. Later it may be flushed to disk via Checkpointing process, which will use checkpoint page buffer, which is also in RAM. So you can have a separate region with low memory usage (see another answer) but you can't have data exclusively on disk.
When you access data it will always be pulled from disk to Durable Memory, too.
Redis is a database in-memory but persistent on disk meanwhile.
Q1: So I wonder does this mean that when redis server starts, it will automatically load all the data on the disk into memory?
Q2: And when writing data to redis, will it both update in the memory and the disk?
Can anyone please help me answer my two questions?
Q1: So I wonder does this mean that when redis server starts, it will
automatically load all the data on the disk into memory?
Yes, depending on the configuration, Redis performs snapshots of memory to disk and, when Redis is restarted it can take latest snapshot and take it to memory again automatically.
Q2: And when writing data to redis, will it both update in the memory
and the disk?
Redis prioritizes writes on memory and writes to disk are done in a separate thread. The answer then is yes, it writes data to both memory and disk, but it might happen that a server failure may produce a data loss since it's not mandatory to Redis to persist data to disk.
Check official docs about persistence to learn more about the topic.
I am facing some scaling issues with my redis instances and was wondering if there's a way to configure redis to save data only to disk (and not hold it in memory). That way I could just increase disk space and not RAM.
Right now my instances are getting stuck and just hang when they reach the memory limit.
Thanks!
No - Redis, atm, is an in-memory database. That means that all data that it manages resides first and foremost in RAM.
Based on the
http://redis.io/topics/faq
Redis is an in-memory but persistent on disk database.
So may I know redis save key/value in memory or in disk? or both?
When writing value in Redis, it write into memory and disk at the same time?
Thanks for the concept.
depending on how you configure it, redis can periodically back up the existing state to disk, but otherwise, everything is in memory.
Redis will atomically snapshot its memory state to disk if so configured. See this part of the docs for more info:
http://redis.io/topics/persistence
So you can have different levels of durability. For the most part, when you get a key, it is out of memory and when you set a key it is also in memory. The data is written to disk independently of read/write operations.