In my application I am implementing a server-side cache using Redis (for mySQL database). When data change in the database, I want to completely clear the cache to invalidate the old data.
However, I would like to see some statistics about how often are different keys queried in Redis, so that I can sort of manually pre-fetch frequently queried data for them to be available immediately after clearing the cache.
Is there any way how to see these statistics directly in Redis? Or what is a common solution to this problem?
You can use the object command.
OBJECT FREQ returns the logarithmic access frequency counter of
the object stored at the specified key. This subcommand is available
when maxmemory-policy is set to an LFU policy.
https://redis.io/commands/object
redis-cli --hotkeys can do the help for redis-cli version 4.x and above.
Related
fairly new to Redis and working on a Azure Cache for Redis implementation.
In the Azure documentation around Redis trouble shooting it's stated that there can be long-running commands and that the redis command documentation shows the time complexity of all commands.
I couldn't find anything around the config set maxmemory-policy command.
Is my assumption correct that setting/changing the maxmemory-policy itself is not an expensive command (unlike e.g. resharding/rebalancing a cluster)?
(I know, "expensive" does not really have a proper definition here :) )
Thanks for any hints or answers!
Yes, the config set command itself is not an expensive command. It iterates the config array which contains about 200 items, to find the config, and set it. That's all.
However, after the setting, Redis might need to free memory for evicted items periodically or for each command. That's a cost.
Could you please answer these 2 questions and correct me if wrong.
I assume Both Redis Database and Redis Cache are stored in Memory and not in Disk. Am I correct?
If Yes, What are the major difference between both. I am assuming both are stored in memory and it should not make much difference between them both. I mean the speed should be the same as they are in memory only. Do we still need Cache again?
Could you please tell me what are the differences and advantages between the both.
Second Question: Can the server restart remove all data in the Redis database? Cache must be deleted for sure I believe.
Thanks
Not sure what do you mean?
Redis is a product first of all - its an in-memory data structures store.
Depending on its configurations it can be targeted to different use cases:
Database
Cache
Even message broker
If you're coming from the cloud world, cloud providers can call this "Cache" and this means that they offer a redis that is pre-configured to be used as a cache (remove the oldest records when the memory becomes next to be fully utilized, etc).
But after you'll you will work with some kind of redis client that will interact with remote redis server.
I have multiple servers that all store set members in a shared Redis cache. When the cache fills up, I need to persist the data to disk to free up RAM. I then plan to parse the dumped data such that I will be able to combine all of the values that belong to a given key in MongoDB.
My first plan was to have each server process attempt an sadd operation. If the request fails because Redis has reached maxmemory, I planned to query for each of my set keys, and write each to disk.
However, I am wondering if there is a way to use one of the inbuilt persistence methods in Redis to write the Redis data to disk and delete the key/value pairs after writing. If this is possible I could just parse the rdb dump and work with the data in that fashion. I'd be grateful for any help others can offer on this question.
Redis' persistence is meant to be used for whatever's in the RAM. Put differently, you can't persist what ain't in RAM.
To answer your question: no, you can't use persistence to "offload" data from RAM.
I am using 5 databases in my redis server. I want to evict keys belonging to a particular DB using LRU mechanism. Is it possible ?
I read this: how-to-make-redis-choose-lru-eviction-policy-for-only-some-of-the-keys.
But all my databases are using time to live for their entries. So cant use volatile-lru policy.
I tried volatile-ttl policy but other databases are having less ttl for their keys. So they will get evicted which I dont want.
That's one of the effects of using numbered/shared database - they all share the same configuration and resources. You should consider using separate Redis servers, one for each of your databases, to have better control over what gets evicted and when. Even more importantly, using dedicated instances allows you to better utilize the cores that you server has.
I have a very large set of keys, 200M keys, with small values, <100 bytes, to store and I'm trying to use Redis. The problem is such that I have 10 Redis DB to split the keys over, but currently I'm on a single server with those 10 Redis DB. By a Redis DB I mean using SELECT. From my calculations it looks like I'm going to blow out memory. I think I'll need over 4TB of memory for this case! What are my options? First, my calculation is based on 10000 keys with 100 byte values taking 220MB of RAM (this is from a table I found). So simply put (2*10^8 / 10^4) * 220MB = 4.4TB.
If my calculation looks correct, what are my options? I've read on different posts that Redis VM is no longer an option. Can I use a Redis cluster? This still appears to require too many servers to be practical. I understand I could switch to another DB, but I'd like that to be the last resort option.
Firstly, using shared databases (i.e. the SELECT command) isn't a recommended practice since all of these databases are essentially managed by the same Redis process. It is preferable having 10 separate Redis processes (even on the same server) in order to avoid contention (more info here).
Next, there are ways to reduce the memory footprint of your database. You could, for example, perform client-side compression (see here) or consider other optimizations such as using Hashes to keep multiple values (as described here).
That said, a Redis server is ultimately bound by the amount of RAM that the host provides. Once you've reached that limit you'll need to shard your database and use a Redis cluster. Since you're already using multiple databases this shouldn't pose a big challenge as your code should already be compatible with that to a degree. Sharding can be done in one of three approaches: client, proxy or Redis Cluster. Client-side sharding can be implemented in your code or by the Redis client that you're using (if the client library that you're using supports that). Redis Cluster (v3) is expected to be released in the very near future and already has a stable release candidate. As for proxy-based sharding, there are several open source solutions out there, including Twitter's twemproxy, Netflix's dynomite and codis. Additional information about sharding and partitioning can be found here.
Disclaimer: I work at Redis Labs. Lastly, AFAIK there's only one Redis-as-a-Service provider that already provides built-in support for clustering Redis. Redis Labs' Redis Cloud is a fully-managed service that can scale seamlessly to any required capacity. Our clusters support both the '{}' hashtag standard as well as sharding by RegEx - more about this can be found here.
You can use LMDB with Dynomite to store data beyond your memory capacity. LMDB uses both disk and memory to store data. Dynomite make LMDB to be distributed.
We have done a POC with this combo and they work nicely together.
For more information, please check out our open issue here:
https://github.com/Netflix/dynomite/issues/254