How do I copy a sorted set in redis to a regular, unsorted set? Is there a redis command that can do this? I can manually iterate through the sorted set and manually insert in the unsorted set, but it seems like there might be a better way to do this.
I don't think there is any command to do this directly.
But you can write simple lua script to do it on server instead downloading the sorted sets content to client and then pushing it back to new set.
Redis commands:
SCRIPT LOAD "for i,v in ipairs(redis.call('zrange', KEYS[1], 0, -1)) do redis.call('sadd', KEYS[2], v) end"
ZADD zset 1 first
ZADD zset 2 second
ZADD zset 3 third
EVALSHA dd1c22a22108d758b93c26eb92d1ef2933cec314 2 zset set
SMEMBERS set
Result:
"dd1c22a22108d758b93c26eb92d1ef2933cec314"
(integer) 0
(integer) 0
(integer) 0
(nil)
1) "second"
2) "first"
3) "third"
SCRIPT LOAD defines the script and returns its sha hash, EVALSHA than executes. Arguments are 2 to indicate that 2 key names follows, first is sorted set to copy from, second is set to copy to.
Related
I am trying to run the code from the redis transactions page. Specifically, this part:
WATCH zset
element = ZRANGE zset 0 0
MULTI
ZREM zset element
EXEC
If I try to do it from the cli, line by line, I get this:
localhost:6380> zadd set 1 a
(integer) 1
localhost:6380> WATCH zset
localhost:6380> element = ZRANGE zset 0 0
(error) ERR unknown command 'element'
OK
which probably means I'm doing something wrong? I remember working with lua about 9 years ago, so this doesn't really look like lua either to me.
How does someone run that snippet? Is it only some kind of pseudocode?
As #Dinei said, the example given is pseudocode.
Let's look at it (I added line numbers for us to refer to):
1 WATCH zset
2 element = ZRANGE zset 0 0
3 MULTI
4 ZREM zset element
5 EXEC
The point of the exercise is to solve the race condition that would occur if we only read the key (with ZRANGE, in line 2), and then modify the key (with ZREM in line 4). I assume you understand the problem if we didn't use the "CAS" semantics, so no need to get into it.
As pointed out, redis-cli just gives you the ability to run redis commands and see their replies, but not save values in variables, etc.
So the idea of the example is that in line 2, we are "saving" the result of the "read" operation, into a pseudo-variable element.
Then, in line 4, we are using that value in our "set" operation, and of course lines 1, 3 and 5 are just the "CAS" commands to ensure there is no race condition.
Presumably the actual usage of such commands would be done from a redis client in a programming language that would allow us to save the return value of the ZRANGE and then use it later in the ZREM command.
But if you wanted to run it in redis-cli, you'd see this, where we pretend that our client-side code would have read and saved "a" that was returned from zrange and then passed that value to the zrem command:
127.0.0.1:6379> zadd zset 1 a
(integer) 1
127.0.0.1:6379> watch zset
OK
127.0.0.1:6379> zrange zset 0 0
1) "a"
127.0.0.1:6379> multi
OK
127.0.0.1:6379> zrem zset a
QUEUED
127.0.0.1:6379> exec
1) (integer) 1
127.0.0.1:6379>
Yes, it is some kind of pseudocode.
redis-cli only accepts Redis commands, it is not a full-fledged editor nor supports direct Lua scripting (neither variables like the element variable in the pseudocode).
I remember working with lua about 9 years ago, so this doesn't really look like lua either to me.
This is not Lua, it is pseudocode. Actually, the Redis Transactions page you linked to does not refer to Lua at all (and that's why #Piglet's comment in your post makes sense).
However, it is possible to execute Lua scripts by using Redis' EVAL command.
I'd like to periodically scan thru a redis instance for keys that changed since the last scan. in between the scans i don't want to process the keys.
eg one key could get a thousand updates between scans. i care for the most recent value only when doing the next periodic scan.
There is no built-in way in Redis to achieve that (yet).
You could, for example, recode your app and add some sort of a way to track updates. For example, wherever you're calling SET foo bar, also call ZADD updated <timestamp> foo. Then, you can use the 'updated' Sorted Set to retrieve updated keys.
Alternatively, you can try using RedisGears to automate the tracking part (for starters). Assuming that you have RedisGears running (i.e. docker run -it -p 6379:6379 redislabs/redisgears), you can do something like the following:
$ cat gear.py
def addToUpdatedZset(x):
import time
now = time.time()
execute('ZADD', 'updated', now, x['key'])
return x
GB().filter(lambda x: x['key'] != 'updated').foreach(addToUpdatedZset).register('*')
$ redis-cli RG.PYEXECUTE "$(cat gear.py)"
OK
$ redis-cli
127.0.0.1:6379> KEYS *
(empty list or set)
127.0.0.1:6379> SET foo bar
OK
127.0.0.1:6379> KEYS *
1) "updated"
2) "foo"
127.0.0.1:6379> ZRANGE updated 0 -1 WITHSCORES
1) "foo"
2) "1559339877.1392548"
127.0.0.1:6379> SET baz qux
OK
127.0.0.1:6379> KEYS *
1) "updated"
2) "baz"
3) "foo"
127.0.0.1:6379> ZRANGE updated 0 -1 WITHSCORES
1) "foo"
2) "1559339877.1392548"
3) "baz"
4) "1559339911.5493586"
In Redis, keys user* will print all keys starting with user.
For example:
keys user*
1) "user2"
2) "user1"
Now, I want all keys that don't start with user to be printed.
How could I do that?
IMPORTANT: always use SCAN instead of (the evil) KEYS
Redis' pattern matching is somewhat functionally limited (see the implementation of stringmatchlen in util.c) and does not provide that which you seek ATM. That said, consider the following possible routes:
Extend stringmatchlen to match your requirements, possibly submitting it as a PR.
Consider what you're trying to do - fetching a subset of keys is always going to be inefficient unless you index them, consider tracking the names of all non-user keys (i.e.g. in a Redis Set) instead.
If you are really insistent on scanning the entire keyspace and match against negative patterns, one way to accomplish that is with a little bit of Lua magic.
Consider the following dataset and script:
127.0.0.1:6379> dbsize
(integer) 0
127.0.0.1:6379> set user:1 1
OK
127.0.0.1:6379> set use:the:force luke
OK
127.0.0.1:6379> set non:user a
OK
Lua (save this as scanregex.lua):
local re = ARGV[1]
local nt = ARGV[2]
local cur = 0
local rep = {}
local tmp
if not re then
re = ".*"
end
repeat
tmp = redis.call("SCAN", cur, "MATCH", "*")
cur = tonumber(tmp[1])
if tmp[2] then
for k, v in pairs(tmp[2]) do
local fi = v:find(re)
if (fi and not nt) or (not fi and nt) then
rep[#rep+1] = v
end
end
end
until cur == 0
return rep
Output - first time regular matching, 2nd time the complement:
foo#bar:~$ redis-cli --eval scanregex.lua , "^user"
1) "user:1"
foo#bar:~$ redis-cli --eval scanregex.lua , "^user" 1
1) "use:the:force"
2) "non:user"
#Karthikeyan Gopall you nailed it in your comment above and this saved me a bunch of time. Thanks!
Here's how you can use it in various combinations to get what you want:
redis.domain.com:6379[1]> set "hello" "foo"
OK
redis.domain.com:6379[1]> set "hillo" "bar"
OK
redis.domain.com:6379[1]> set "user" "baz"
OK
redis.domain.com:6379[1]> set "zillo" "bash"
OK
redis.domain.com:6379[1]> scan 0
1) "0"
2) 1) "zillo"
2) "hello"
3) "user"
4) "hillo"
redis.domain.com:6379[1]> scan 0 match "[^u]*"
1) "0"
2) 1) "zillo"
2) "hello"
3) "hillo"
redis.domain.com:6379[1]> scan 0 match "[^u^z]*"
1) "0"
2) 1) "hello"
2) "hillo"
redis.domain.com:6379[1]> scan 0 match "h[^i]*"
1) "0"
2) 1) "hello"
According to redis keys documentation the command supports glob style patterns, not regular expressions.
and if you look at the documentation, you'll see that the "!" character is not special as opposites to regular expressions.
Here is a simple test I ran in my own db:
redis 127.0.0.1:6379> set a 0
OK
redis 127.0.0.1:6379> set b 1
OK
redis 127.0.0.1:6379> keys *
1) "a"
2) "b"
redis 127.0.0.1:6379> keys !a
(empty list or set) // I expected "b" here
redis 127.0.0.1:6379> keys !b
(empty list or set) // I expected "a" here
redis 127.0.0.1:6379> keys [!b]
1) "b"
redis 127.0.0.1:6379> keys [b]
1) "b"
redis 127.0.0.1:6379> keys [ab]
1) "a"
2) "b"
redis 127.0.0.1:6379> keys ![b]
(empty list or set)
So I just don't think what you are trying to achieve is possible via the keys command.
Besides, the keys command is not very suitable for production environment as it locks your whole redis database.
I would recommend getting all the keys with the scan command, store them locally, and then remove them using LUA
Here's a trick to achieve this with native redis commands (no need for Lua scripts or anything).
If you are able to control the timing of when you insert the new keys (the ones you want to keep, deleting all other stuff like in your question), you can:
Before setting the new keys, set the expiration to all existing keys (by pattern or everything) to expire right now (see how)
Load the new keys
Redis will automatically delete all the older keys and you will be left just with the new ones you want.
You also can print all keys and pass it to grep. For example:
redis-cli -a <password> keys "*" | grep -v "user"
redis 127.0.0.1:6379> zadd somekey 12.54 value
(integer) 1
redis 127.0.0.1:6379> zrevrange somekey 0 -1 withscores
1) "value"
2) "12.539999999999999"
How do I keep the original number format in redis?
Redis relies on client for number precision conversion, so this might not be possible to achieve using Redis default cli using python.
A similar thread for same :
http://www.manning-sandbox.com/thread.jspa?messageID=159190
I'm currently working with redis using "set" structure.
I want to know if it's possible to clean automatically empty "set" ?
Else find a cron/process to clean periodically empty "set"
UPDATE:
More generic question, there is a diff (memory usage) between "(nil)" and "(empty list or set)"
example:
sadd x 1
srem x
smembers x
(empty list or set)
or
sadd x 1
del x
smembers x
(nil)
This is already automatic. When a set is empty, it is removed from the namespace.
> flushall
OK
> sadd x 1 2 3
(integer) 3
> keys *
1) "x"
> srem x 1 2 3
(integer) 3
> keys *
(empty list or set)
You do not have to do anything specific to benefit from this behavior.
To answer your second question, (nil) or (empty list or set) is just an interpretation of the client program. In the Redis server, in both cases, the entry has been physically removed, and the associated memory freed.