background understandingļ¼in Redis cluster mode, every key-value will be distributed to a steady hashslot by CRC16(key) mod 16384.
redis cluster contains 16384 slots, my quetions is: in redis cluster mode, we just can save 16384 key-value(i think it is impossible, but i don't known why)? or one slot mappings many key-value, the algorithms before is just calculate the specific slot?
thank you for answerring sinserely.
A Slot can have multiple keys value the above algorithm just specifies the slot where the data needs to be stored or in which slot and depending on the number of shards the slots are distributed and the data in saved in one of them and automatically deleted when the ttl is over.
Related
I'm learning Redis and one thing I'm not certain about is when multiple redis clusters are present, if data partition will be done so that each cluster stores part of the data, my understanding is as below.
When multiple ElastiCache redis clusters are present, a key will be hashed using consistent hashing or similar algorithm and the request will be routed to any of the cluster in equal chances. Once the request is routed to a cluster then it will be hashed again to determine which of the 16364 slot it will be in and based on where the slot is allocated, the request will be on a particular shard of that cluster.
To summarize, when multiple clusters are present and cluster mode is enabled, there are two rounds of request level load balancing are done, one is to determine which cluster is to be used and the other is to determine which shard the request to be routed to.
In terms of data partition, each cluster will have a partition of the entire data and within each cluster, the data will be further split into multiple shards.
Please let me know if my understanding is correct, thanks!
According to Redis cluster specification, a Redis cluster has 16384 slots. I was wondering if any one has successfully tried configuring a Redis cluster to use a different number of slots.
The context of my question is: I have an application that's backed by a cluster-mode Redis cluster. The application sends many multi-get operations to Redis (using MGET command). In cluster mode, MGET is only supported for keys that belong to the same slot, so keys passed into the multi-get operation needs to be grouped by slot, and one MGET command is issued per involved slot.
The number of keys involved in MGET is always much less than 16384, worst case is ~2000 keys, which means most of the multi-get operations will ends up with hundreds of MGET commands with a single key each.
For slots that are on the same node, I can pipeline multiple MGETs to improve performance. But I've been wondering if I can avoid this problem completely by configuring a much lower slot number.
For example, if the number of slots is 128, with a multi-get operation containing 1024 keys (assume these keys are distributed over 128 slots perfectly evenly), it will end up with 128 MGET commands with 8 keys each.
If I group these by nodes and pipeline them, I think the performance will be better than using 16384 slots (1024 perfectly-evenly distributed keys will result in 1024 MGET commands with single key each)
As #sazzad noted, the number of slots is hardcoded (see cluster.h:#define CLUSTER_SLOTS) in Redis. It can't be changed by any means other than modifying and recompiling Redis (nor would I recommend using such an untested build).
As for the question itself, are you optimizing something real or is it just for fun? Although MGET is more efficient than just GETs, as long as you pipeline the batch responsibly the difference usually doesn't matter. The cluster clearly states that it is a subset of Redis, and as such multi-key operations behave differently. On the other hand, the way to squeeze more performance in the cluster is by adding nodes and spreading the load.
I am experiencing a very strange case happen in production with redis, a large amount of redis keys are evicted unexpectedly even though memory not reach max configuration.
Current redis setting is max mem = 7GB, volatile-ttl.
Most of the keys are set a TTL when store to Redis.
Below graph showing a large drop in redis key eventhough memory at the time was only 3.5GB (<< 7GB)
According to my understanding, Redis would evict keys only when memory reach max-mem. And even when it does, it will only drop keys gradually according to the need for inserting new keys.
Thank you very much!
Actually I'm using a configuration of Redis Master-Slaves with HAProxy for Wordpress to have High Avaibility. This configuration is nice and works perfect (I'm able to remove any server to maintenance without downtime). The problem of this configuration is that only one Redis server is getting all the traffic and the others are just waiting if that server dies, so in a very high load webpage can be a problem, and add more servers is not a solution because always only one will be master.
With this in mind, I'm thinking if maybe I can just use a Redis Cluster to allow to read/write on all nodes but I'm not really sure if it will works on my setup.
My setup is limited to three nodes the most of times, and I've read in some places that Redis cluster minimal setup is three nodes, but six is recommended. This is rational because this setup allow to have Slaves nodes that will become Masters if her Master dies, and then all data will be kept, but what happend if data don't cares?. I mean, on my setups the data is just cached objects, so if don't exists it just create it again so:
The data will be lost (don't care), and the other nodes will get the objects from clients again, to serve it on later requests (like happen if a Flush the data).
The nodes will answer that data doesn't exists and will reject to cache because the object would have to be on other node that is dead.
Someone know it?
Thanks!!
When a master dies, the Redis cluster goes to a down state and any command involving a key served by the failed instance will fail.
This may differ from some other distributed software because Redis Cluster is not the kind of program that every master holds all data. In fact, the key space is horizontally partitioned and each key is served by only one master.
This is mentioned in the specification:
The key space is split into 16384 slots...
a single hash slot will be served by a single node...
The base algorithm used to map keys to hash slots is the following:
HASH_SLOT = CRC16(key) mod 16384
When you setup a cluster, you certainly ask each node to serve a set of slots, and each slot can only be served by one node. If one node dies, you lose the slots on this node unless you have a slave failover to serve them, so that any command involving keys mapped to these slots will fail.
I have multiple redis instances. I made a cluster using different port. Now I want to transfer the data from pre-existing redis instances to the cluster. I know how to transfer data from one instance to the cluster but when the instances are greater than one, I am not able to do it.
You need to define some sort of sharding strategy for your redis cluster. Database Sharding So basically you need to have a certain consistent hashing strategy which will decide given a key, the shard or the redis instance in your cluster the key will go to. You need to have a certain script for this data migration that will have an array of all the redis instances in your cluster.
Then for a given key which you read from the standalone redis, you will use the hashing mechanism to find out the sharding index or the redis instance from the list you maintained earlier to use and accordingly you will write the data in that cluster node. My assumption in all this is that you have an in house redis cluster setup as opposed to the one which Redis Labs provide.