I am trying to insert multiple key/values at once on Redis (some values are sets, some are hashes) and I get this error: ERR CROSSSLOT Keys in request don't hash to the same slot.
I'm not doing this from redis-cli but from some Go code that needs to write multiple key/values to a redis cluster. I see other places in the code where multiple key values are done this way and I don't understand why mine don't work. What are the hash requirements to not have this error?
Thanks
In a cluster topology, the keyspace is divided into hash slots. Different nodes will hold a subset of hash slots.
Multiple keys operations, transactions, or Lua scripts involving multiple keys are allowed only if all the keys involved are in hash slots belonging to the same node.
Redis Cluster implements all the single key commands available in the
non-distributed version of Redis. Commands performing complex
multi-key operations like Set type unions or intersections are
implemented as well as long as the keys all belong to the same node.
You can force the keys to belong to the same node by using Hash Tags
ERR CROSSSLOT Keys in request don't hash to the same slot
As the error message suggests, only if all of the keys belong to same slot will the operation succeed. Otherwise, you will see this failure message. This error would be seen even though both/all slots belongs to the same node. The check is very strict and, as per the code, all keys should hash to same slot.
Related
Problem:
I need to efficiently delete keys from my Redis Cache using a wildcard pattern. I don't need atomicity; eventual consistency is acceptable.
Tech stack:
.NET 6 (async all the way through)
StackExchange.Redis 2.6.66
Redis Server 6.2.6
I currently have ~500k keys in Redis.
I'm not able to use RedisJSON for various reasons
Example:
I store the following 3 STRING types with keys:
dailynote:getitemsforuser:region:sw:user:123
dailynote:getitemsforuser:region:fl:user:123
dailynote:getitemsforuser:region:sw:user:456
...
where each STRING stores JSON like so:
> dump dailynote:getitemsforuser:region:fl:user:123
"{\"Name\":\"john\",\"Age\":22}"
The original solution used the KeysAsync method to retrieve the list of keys to delete via a wildcard pattern. Since the Redis Server is 6.x, the SCAN feature is being used by KeysAsync internally by the StackExchange.Redis nuget.
Original implementation used a wildcard pattern dailynote:getitemsforuser:region:*. As one would expect, this solution didn't scale well and we started seeing RedisTimeoutExceptions.
I'm aware of the "avoid this in PROD if you can" and have seen Marc Gravell respond to a couple other questions/issues on SO and StackExchange.Redis GitHub. The only potential alternative I could think of is to use a Redis SET to "track" each RedisKey and then retrieve the list of values from the SET (which are the keys I need to remove). Then delete the SET as well as the returned keys.
Potential Solution?:
Create a Redis SET with a key of dailynote:getitemsforuser with a value which is the key of the form dailynote:getitemsforuser:region:XX...
The SET would look like:
dailynote:getitemsforuser (KEY)
dailynote:getitemsforuser:region:sw:user:123 (VALUE)
dailynote:getitemsforuser:region:fl:user:123 (VALUE)
dailynote:getitemsforuser:region:sw:user:456 (VALUE)
...
I would still have each individual STRING type as well:
dailynote:getitemsforuser:region:sw:user:123
dailynote:getitemsforuser:region:fl:user:123
dailynote:getitemsforuser:region:sw:user:456
...
when it is time to do the "wildcard" remove, I get the members of the dailynote:getitemsforuser SET, then call RemoveAsync passing the members of the set as the RedisKey[]. Then call RemoveAsync with the key of the SET (dailynote:getitemsforuser)
I'm looking for feedback on how viable of a solution this is, alternative ideas, gotchas, and suggestions for improvement. TIA
UPDATE
Added my solution I went with below...
The big problem with both KEYS and SCAN with Redis is that they require a complete scan of the massive hash table that stores every Redis key. Even if you use a pattern, it still needs to check each entry in that hash table to see if it matches.
Assuming you are calling SADD when you are also setting the value in your key—and thus avoiding the call to SCAN—this should work. It is worth noting that calls to SMEMBERS to get all the members of a Set can also cause issues if the Set is big. Redis—being single-threaded—will block while all the members are returned. You can mitigate this by using SSCAN instead. StackExchange.Redis might do this already. I'm not sure.
You might also be able to write a Lua script that reads the Set and UNLINKs all the keys atomically. This would reduce network but could tie Redis up if this takes too long.
I ended up using the solution I suggested above where I use a Redis SET with a known/fixed key to "track" each of the necessary keys.
When a key that needs to be tracked is added, I call StackExchange.Redis.IDatabase.SetAddAsync (SADD) while calling StackExchange.Redis.IDatabase.HashSetAsync (HSET) for adding the "tracked" key (along with its TTL).
When it is time to remove the "tracked" key, I first call StackExchange.Redis.IDatabase.SetScanAsync (SSCAN) (with a page size of 250) iterating on the IAsyncEnumerable and call StackExchange.Redis.IDatabase.KeyDeleteAsync (HDEL) on chunks of the members of the SET. I then call StackExchange.Redis.IDatabase.KeyDeleteAsync on the actual key of the SET itself.
Hope this helps someone else.
Currently we're using single redis instance(String datastructure is used, here i am able to fetch data using "get key" and "mget key1 key2"). To scale the redis now created redis cluster here i get this error: ERR CROSSSLOT Keys in request don't hash to the same slot.
If I give hashtag then all data is storing in one slot. All other slots are unused.
I am new to redis, Please suggest a solution how to make use of all the slots. Thanks in advance.
RTFM: https://redislabs.com/blog/redis-clustering-best-practices-with-keys/
Keys are hashed. Using a hash forces the same hash result (aka slot) so that multiple command operations can work.
I am new to redis, I do not know the meaning of "keyspace" and "key space" in redis terminology which I encountered in redis official website. Can someone help me to clear that? Thanks.
These terms refer to the internal dictionary that Redis manages, in which all keys are stored. The keyspace of a Redis database is managed by a single server in the case of a single instance deployment, and is divided to exclusive slot ranges managed by different nodes when using cluster mode.
In a key-value database, all keys can be in one node or divided in multiple nodes. Suppose I am storing telephone dictionary as key-value store with name as key and phone number as a value. If I store names A-L on one node and M-Z on another node, I divide my database into two key spaces. When I run query to search number of Smith, I need to search only second key space or node. This divides the query on multiple nodes and divide the work giving faster result. This could be shared-nothing model of working.
I have a Redis Cluster consisting of multiple nodes. I want to update 3 different keys in a single atomic operation. My Lua script is like:
local u1 = redis.call('incrby', KEYS[1], ARGV[1])
local u2 = redis.call('incrby', KEYS[2], ARGV[1])
local u3 = redis.call('incrby', KEYS[3], ARGV[1])
And I fired it with:
EVAL script 3 key1 key2 key3 arg
But I got the error message:
WARN Resp(AppErr CROSSSLOT Keys in request don't hash to the same slot)
The above operations cannot be done, and the updates will fail. It seems I cannot modify the keys in different nodes with a single Lua script. But according to the doc:
All Redis commands must be analyzed before execution to determine
which keys the command will operate on. In order for this to be true
for EVAL, keys must be passed explicitly. This is useful in many ways,
but especially to make sure Redis Cluster can forward your request to
the appropriate cluster node.
Note this rule is not enforced in order
to provide the user with opportunities to abuse the Redis single
instance configuration, at the cost of writing scripts not compatible
with Redis Cluster.
So I think as long as I follow the key passing rule, the script should be compatible with Redis Cluster. I wonder what's the problem here and what should I do to update all keys in a single script.
I'm afraid you've misunderstood the documentation. (And I agree that it's not very clear.)
Redis operations, whether commands or Lua scripts, can only work when all the keys are on the same server. The purpose of the key passing rule is to allow Cluster servers to figure out where to send the script and to fail fast if all the keys don't come from the same server (which is what happened in your case).
So it's your responsibility to make sure that all the keys you want to operate on are located on the same server. The way to do that is to use hash tags to force keys to hash to the same slot. See the documentation for more details on that.
I have a Redis Cluster. I am using JedisCluster client to connect to my Redis.
My application is a bit complex and I want to basically control to which partition data from my application goes. For example, my application consists of sub-module A, B, C. Then I want that all data from sub-module A should go to partition 1 for example. Similarly data from sub-module B should go to partition 2 for example and so on.
I am using JedisCluster, but I don't find any API to write to a particular partition on my cluster. I am assuming I will have same partition names on all my Redis nodes and handling which data goes to which node will be automatically handled but to which partition will be handled by me.
I tried going through the JedisCluster lib at
https://github.com/xetorthio/jedis/blob/b03d4231f4412c67063e356a7c3acf9bb7e62534/src/main/java/redis/clients/jedis/JedisCluster.java
but couldn't find anything. Please help?
Thanks in advance for the help.
That's not how Redis Cluster works. With Redis Cluster, each node (partition) has a defined set of keys (slots) that it's handling. Writing a key to a master node which is not served by the master results in rejection of the command.
From the Redis Cluster Spec:
Redis Cluster implements a concept called hash tags that can be used in order to force certain keys to be stored in the same node.
[...]
The key space is split into 16384 slots, effectively setting an upper limit for the cluster size of 16384 master nodes (however the suggested max size of nodes is in the order of ~ 1000 nodes).
Each master node in a cluster handles a subset of the 16384 hash slots.
You need to define at Cluster configuration-level which master node is exclusively serving a particular slot or a set of slots. The configuration results in data locality.
The slot is calculated from the key. The good news is that you can enforce a particular slot for a key by using Key hash tags:
There is an exception for the computation of the hash slot that is used in order to implement hash tags. Hash tags are a way to ensure that multiple keys are allocated in the same hash slot. This is used in order to implement multi-key operations in Redis Cluster.
Example:
{user1000}.following
The content between {…} is used to calculate the slot. Key hash tags allow you to group keys on particular nodes and enforce the same data locality when using arbitrary hash tags.
You can also go a step further by using known hash tags that map to slots (you'd need either precalculate a table or see this Gist). By using known hash tags that map to a specific slot you're able to select the slot and so the master node on which the data is located.
Everything else is handled by your Redis client.