Expiry policy on off-heap entries is not working as expected. I am seeing a linear increase in off-heap entries and after some time sudden dip in off-heap size. Is only marking of cache entry done after expiry time and collected later in bulk? I am not finding any documentation regarding the same.
We need to enable eager TTL in the cache configuration. It will create a thread in the background and removes the expired entries.
There are two modes of expiry policy in Ignite: eager and non-eager: https://www.gridgain.com/docs/latest/developers-guide/configuring-caches/expiry-policies#eager-ttl
In case of an eager TTL entries are expired periodically by a background thread. When a non-eager TTL is used, then entries are only expired upon access. So, it's possible for expired entries to be stored in off-heap for some time until they get read.
Also keep in mind that JVM can allocate more memory than is immediately required. The amount of memory allocated by the Java process is not directly related to the amount of data stored in Ignite.
Related
When using BinaryObject for off-heap in-memory-only cache values, do we need to do anything to protect against the cache entry being evicted or expired while accessing fields via BinaryObject::field(String)?
For example, if the cache's data region has the default memory-size eviction (90% full?), or if the cache uses a creation expiry policy, and the region happens to evict entries or the cache expire entries while the code is making several calls to BinaryObject::field(String). Does Ignite automatically ensure that BinaryObject won't access invalid off-heap memory (throwing an exception perhaps), or can the developer use locking / transactions or a "touched" expiry to help prevent this?
Thanks!
BinaryObject instances returned from the Ignite API and accessed by the user code are copies. They do not reference Ignite storage memory directly.
You can work with BinaryObject even after the corresponding cache entry gets evicted.
When a cache object becomes eligible for eviction, it will be removed from memory.
Ignite has several Eviction policies:
Random-LRU
Random-2-LRU
Explained here
This means that if you recently used a cache value(or are using it at the moment that the eviction is taking place), either in BinaryObject representation or not, it will not be evicted. All Eviction algorithms use a LEAST RECENTLY USED algorithm.
We are expiring 1000 keys per sec by average while taking snapshot we happened to see the size of the dump is comparatively lesser than the primary since because the snapshot is excluding the expired key's memory. Since expiring keys holding much memory in our platform, is there any way that we can make redis to release the memory held for expired keys periodically. (we are using 2.8.21 engine) or latest redis engine versions will resolve this problem little efficiently.
Please guide me to the correct platform if stackoverflow is not appropriate for my question.
Reclaim memory guide : https://docs.redislabs.com/latest/ri/memory-optimizations/reclaim-expired-keys-memory-faster/ ( but need suggestion, whether upgrading will help a lot or doing scan will be good as mentioned in the doc)
Expired keys are removed from memory:
Passively: when you try to access it and the key is found to be timed out. This is how doing a full SCAN would help you, it forces a passive removal across all the keyspace.
Actively: every 100 ms, it tries to remove from memory expired keys at random, never investing more than 1 ms per cycle at it, until it estimates that less than 25% of expired keys remain. The logic is not that trivial, see activeExpireCycle (2.8.21 version).
Upgrading may help as there new features/configuration settings, like activedefrag.
Please see Redis filling up memory fast, running --bigkeys free it up for solutions including eviction policy and active expiring frequency.
I'm assuming that during replica resynchronisation (full or partial), the master will attempt to send data as fast as possible to the replica. Wouldn't this mean the replica output buffer on the master would rapidly fill up since the speed the master can write is likely to be faster than the throughput of the network? If I have client-output-buffer-limit set for replicas, wouldn't the master end up closing the connection before the resynchronisation can complete?
Yes, Redis Master will close the connection and the synchronization will be started from beginning again. But, please find some details below:
Do you need to touch this configuration parameter and what is the purpose/benefit/cost of it?
There is a zero (almost) chance it will happen with default configuration and pretty much moderate modern hardware.
"By default normal clients are not limited because they don't receive data
without asking (in a push way), but just after a request, so only asynchronous clients may create a scenario where data is requested faster than it can read." - the chunk from documentation .
Even if that happens, the replication will be started from beginning but it may lead up to infinite loop when slaves will continuously ask for synchronization over and over. Redis Master will need to fork whole memory snapshot (perform BGSAVE) and use up to 3 times of RAM from initial snapshot size each time during synchronization. That will be causing higher CPU utilization, memory spikes network utilization (if any) and IO.
General recommendations to avoid production issues tweaking this configuration parameter:
Don't decrease this buffer and before increasing the size of the buffer make sure you have enough memory on your box.
Please consider total amount of RAM as snapshot memory size (doubled for copy-on-write BGSAVE process) plus the size of any other buffers configured plus some extra capacity.
Please find more details here
I am experiencing a very strange case happen in production with redis, a large amount of redis keys are evicted unexpectedly even though memory not reach max configuration.
Current redis setting is max mem = 7GB, volatile-ttl.
Most of the keys are set a TTL when store to Redis.
Below graph showing a large drop in redis key eventhough memory at the time was only 3.5GB (<< 7GB)
According to my understanding, Redis would evict keys only when memory reach max-mem. And even when it does, it will only drop keys gradually according to the need for inserting new keys.
Thank you very much!
I am a newbie to Cassandra - I have been searching for information related to commits and crash recovery in Cassandra on a single node. And, hoping someone can clarify the details.
I am testing Cassandra - so, set it up on a single node. I am using stresstool on datastax to insert millions of rows. What happens if there is an electrical failure or system shutdown? Will all the data that was in Cassandra's memory get written to disk upon Cassandra restart (I guess commitlog acts as intermediary)? How long is this process?
Thanks!
Cassandra's commit log gives Cassandra durable writes. When you write to Cassandra, the write is appended to the commit log before the write is acknowledged to the client. This means every write that the client receives a successful response for is guaranteed to be written to the commit log. The write is also made to the current memtable, which will eventually be written to disk as an SSTable when large enough. This could be a long time after the write is made.
However, the commit log is not immediately synced to disk for performance reasons. The default is periodic mode (set by the commitlog_sync param in cassandra.yaml) with a period of 10 seconds (set by commitlog_sync_period_in_ms in cassandra.yaml). This means the commit log is synced to disk every 10 seconds. With this behaviour you could lose up to 10 seconds of writes if the server loses power. If you had multiple nodes in your cluster and used a replication factor of greater than one you would need to lose power to multiple nodes within 10 seconds to lose any data.
If this risk window isn't acceptable, you can use batch mode for the commit log. This mode won't acknowledge writes to the client until the commit log has been synced to disk. The time window is set by commitlog_sync_batch_window_in_ms, default is 50 ms. This will significantly increase your write latency and probably decrease the throughput as well so only use this if the cost of losing a few acknowledged writes is high. It is especially important to store your commit log on a separate drive when using this mode.
In the event that your server loses power, on startup Cassandra replays the commit log to rebuild its memtable. This process will take seconds (possibly minutes) on very write heavy servers.
If you want to ensure that the data in the memtables is written to disk you can run 'nodetool flush' (this operates per node). This will create a new SSTable and delete the commit logs referring to data in the memtables flushed.
You are asking something like
What happen if there is a network failure at the time data is being loaded in Oracle using SQL*Loader ?
Or what happens Sqoop stops processing due to some condition while transferring data?
Simply whatever data is being transferred before electrical failure or system shutdown, it will remain the same.
Coming to second question, when ever the memtable runs out of space, i.e when the number of keys exceed certain limit (128 is default) or when it reaches the time duration (cluster clock), it is being stored into sstable, immutable space.