Is there a way to do this in redis ?
SET counter 0
INCR counter
SET KEY:{counter} "Content of line 1"
INCR counter
SET KEY:{counter} "Different content of line 2"
My example code should be substituted (i.e., transformed at runtime by the redis-cli) into:
SET counter 0
INCR counter
SET KEY:1 "Content of line 1"
INCR counter
SET KEY:2 "Different content of line 2"
etc.
My problem is NOT how to auto-increment the counter.
My problem is syntax: How to include a generic {wildcard} into something like:
SET keyname:{currentcounter} "value" ...
Any help is appreciated. Thanks a lot !
bernie
If you are using redis 2.6+ then you can use lua scripting along with EVAL command like the following:
eval "local c = redis.call('incr', KEYS[1]);
return redis.call('set', KEYS[2] .. ':' .. c, ARGV[1])"
2 counter KEY "Content of line 1"
I broke it up onto multiple lines to make it easier to read.
EDIT
Sorry, I was away on business for a few days. Here is a sample showing that it works.
redis 127.0.0.1:6379> flushdb
OK
redis 127.0.0.1:6379> eval "local c = redis.call('incr', KEYS[1]); return redis.call('set', KEYS[2] .. ':' .. c, ARGV[1])" 2 counter KEY "Content of line 1"
OK
redis 127.0.0.1:6379> keys *
1) "KEY:1"
2) "counter"
redis 127.0.0.1:6379> get counter
"1"
redis 127.0.0.1:6379> get KEY:1
"Content of line 1"
redis 127.0.0.1:6379> eval "local c = redis.call('incr', KEYS[1]); return redis.call('set', KEYS[2] .. ':' .. c, ARGV[1])" 2 counter KEY "Content of the next thing"
OK
redis 127.0.0.1:6379> keys *
1) "KEY:1"
2) "KEY:2"
3) "counter"
redis 127.0.0.1:6379> get counter
"2"
redis 127.0.0.1:6379> get KEY:2
"Content of the next thing"
Nope, SET/GET commands don't support this.
You can do similar things using LUA scripts in redis, or even simpler; you can issue the commands as redis expect them using trivial programming/scripting
Related
I am trying to run the code from the redis transactions page. Specifically, this part:
WATCH zset
element = ZRANGE zset 0 0
MULTI
ZREM zset element
EXEC
If I try to do it from the cli, line by line, I get this:
localhost:6380> zadd set 1 a
(integer) 1
localhost:6380> WATCH zset
localhost:6380> element = ZRANGE zset 0 0
(error) ERR unknown command 'element'
OK
which probably means I'm doing something wrong? I remember working with lua about 9 years ago, so this doesn't really look like lua either to me.
How does someone run that snippet? Is it only some kind of pseudocode?
As #Dinei said, the example given is pseudocode.
Let's look at it (I added line numbers for us to refer to):
1 WATCH zset
2 element = ZRANGE zset 0 0
3 MULTI
4 ZREM zset element
5 EXEC
The point of the exercise is to solve the race condition that would occur if we only read the key (with ZRANGE, in line 2), and then modify the key (with ZREM in line 4). I assume you understand the problem if we didn't use the "CAS" semantics, so no need to get into it.
As pointed out, redis-cli just gives you the ability to run redis commands and see their replies, but not save values in variables, etc.
So the idea of the example is that in line 2, we are "saving" the result of the "read" operation, into a pseudo-variable element.
Then, in line 4, we are using that value in our "set" operation, and of course lines 1, 3 and 5 are just the "CAS" commands to ensure there is no race condition.
Presumably the actual usage of such commands would be done from a redis client in a programming language that would allow us to save the return value of the ZRANGE and then use it later in the ZREM command.
But if you wanted to run it in redis-cli, you'd see this, where we pretend that our client-side code would have read and saved "a" that was returned from zrange and then passed that value to the zrem command:
127.0.0.1:6379> zadd zset 1 a
(integer) 1
127.0.0.1:6379> watch zset
OK
127.0.0.1:6379> zrange zset 0 0
1) "a"
127.0.0.1:6379> multi
OK
127.0.0.1:6379> zrem zset a
QUEUED
127.0.0.1:6379> exec
1) (integer) 1
127.0.0.1:6379>
Yes, it is some kind of pseudocode.
redis-cli only accepts Redis commands, it is not a full-fledged editor nor supports direct Lua scripting (neither variables like the element variable in the pseudocode).
I remember working with lua about 9 years ago, so this doesn't really look like lua either to me.
This is not Lua, it is pseudocode. Actually, the Redis Transactions page you linked to does not refer to Lua at all (and that's why #Piglet's comment in your post makes sense).
However, it is possible to execute Lua scripts by using Redis' EVAL command.
I'd like to periodically scan thru a redis instance for keys that changed since the last scan. in between the scans i don't want to process the keys.
eg one key could get a thousand updates between scans. i care for the most recent value only when doing the next periodic scan.
There is no built-in way in Redis to achieve that (yet).
You could, for example, recode your app and add some sort of a way to track updates. For example, wherever you're calling SET foo bar, also call ZADD updated <timestamp> foo. Then, you can use the 'updated' Sorted Set to retrieve updated keys.
Alternatively, you can try using RedisGears to automate the tracking part (for starters). Assuming that you have RedisGears running (i.e. docker run -it -p 6379:6379 redislabs/redisgears), you can do something like the following:
$ cat gear.py
def addToUpdatedZset(x):
import time
now = time.time()
execute('ZADD', 'updated', now, x['key'])
return x
GB().filter(lambda x: x['key'] != 'updated').foreach(addToUpdatedZset).register('*')
$ redis-cli RG.PYEXECUTE "$(cat gear.py)"
OK
$ redis-cli
127.0.0.1:6379> KEYS *
(empty list or set)
127.0.0.1:6379> SET foo bar
OK
127.0.0.1:6379> KEYS *
1) "updated"
2) "foo"
127.0.0.1:6379> ZRANGE updated 0 -1 WITHSCORES
1) "foo"
2) "1559339877.1392548"
127.0.0.1:6379> SET baz qux
OK
127.0.0.1:6379> KEYS *
1) "updated"
2) "baz"
3) "foo"
127.0.0.1:6379> ZRANGE updated 0 -1 WITHSCORES
1) "foo"
2) "1559339877.1392548"
3) "baz"
4) "1559339911.5493586"
In Redis, keys user* will print all keys starting with user.
For example:
keys user*
1) "user2"
2) "user1"
Now, I want all keys that don't start with user to be printed.
How could I do that?
IMPORTANT: always use SCAN instead of (the evil) KEYS
Redis' pattern matching is somewhat functionally limited (see the implementation of stringmatchlen in util.c) and does not provide that which you seek ATM. That said, consider the following possible routes:
Extend stringmatchlen to match your requirements, possibly submitting it as a PR.
Consider what you're trying to do - fetching a subset of keys is always going to be inefficient unless you index them, consider tracking the names of all non-user keys (i.e.g. in a Redis Set) instead.
If you are really insistent on scanning the entire keyspace and match against negative patterns, one way to accomplish that is with a little bit of Lua magic.
Consider the following dataset and script:
127.0.0.1:6379> dbsize
(integer) 0
127.0.0.1:6379> set user:1 1
OK
127.0.0.1:6379> set use:the:force luke
OK
127.0.0.1:6379> set non:user a
OK
Lua (save this as scanregex.lua):
local re = ARGV[1]
local nt = ARGV[2]
local cur = 0
local rep = {}
local tmp
if not re then
re = ".*"
end
repeat
tmp = redis.call("SCAN", cur, "MATCH", "*")
cur = tonumber(tmp[1])
if tmp[2] then
for k, v in pairs(tmp[2]) do
local fi = v:find(re)
if (fi and not nt) or (not fi and nt) then
rep[#rep+1] = v
end
end
end
until cur == 0
return rep
Output - first time regular matching, 2nd time the complement:
foo#bar:~$ redis-cli --eval scanregex.lua , "^user"
1) "user:1"
foo#bar:~$ redis-cli --eval scanregex.lua , "^user" 1
1) "use:the:force"
2) "non:user"
#Karthikeyan Gopall you nailed it in your comment above and this saved me a bunch of time. Thanks!
Here's how you can use it in various combinations to get what you want:
redis.domain.com:6379[1]> set "hello" "foo"
OK
redis.domain.com:6379[1]> set "hillo" "bar"
OK
redis.domain.com:6379[1]> set "user" "baz"
OK
redis.domain.com:6379[1]> set "zillo" "bash"
OK
redis.domain.com:6379[1]> scan 0
1) "0"
2) 1) "zillo"
2) "hello"
3) "user"
4) "hillo"
redis.domain.com:6379[1]> scan 0 match "[^u]*"
1) "0"
2) 1) "zillo"
2) "hello"
3) "hillo"
redis.domain.com:6379[1]> scan 0 match "[^u^z]*"
1) "0"
2) 1) "hello"
2) "hillo"
redis.domain.com:6379[1]> scan 0 match "h[^i]*"
1) "0"
2) 1) "hello"
According to redis keys documentation the command supports glob style patterns, not regular expressions.
and if you look at the documentation, you'll see that the "!" character is not special as opposites to regular expressions.
Here is a simple test I ran in my own db:
redis 127.0.0.1:6379> set a 0
OK
redis 127.0.0.1:6379> set b 1
OK
redis 127.0.0.1:6379> keys *
1) "a"
2) "b"
redis 127.0.0.1:6379> keys !a
(empty list or set) // I expected "b" here
redis 127.0.0.1:6379> keys !b
(empty list or set) // I expected "a" here
redis 127.0.0.1:6379> keys [!b]
1) "b"
redis 127.0.0.1:6379> keys [b]
1) "b"
redis 127.0.0.1:6379> keys [ab]
1) "a"
2) "b"
redis 127.0.0.1:6379> keys ![b]
(empty list or set)
So I just don't think what you are trying to achieve is possible via the keys command.
Besides, the keys command is not very suitable for production environment as it locks your whole redis database.
I would recommend getting all the keys with the scan command, store them locally, and then remove them using LUA
Here's a trick to achieve this with native redis commands (no need for Lua scripts or anything).
If you are able to control the timing of when you insert the new keys (the ones you want to keep, deleting all other stuff like in your question), you can:
Before setting the new keys, set the expiration to all existing keys (by pattern or everything) to expire right now (see how)
Load the new keys
Redis will automatically delete all the older keys and you will be left just with the new ones you want.
You also can print all keys and pass it to grep. For example:
redis-cli -a <password> keys "*" | grep -v "user"
This is my first question here on stackoverflow.
Dear community, thanks a lot for your combined knowledge and expertise !
I'm new to Redis so please bear with me, as I'm sure there is a simple solution.
redis-server --version
=> Redis server v=2.6.14 sha=00000000:0 malloc=libc bits=64
redis-cli --version
=> redis-cli 2.6.14
I have read "How to use Redis mass insertion?"
How to use Redis mass insertion?
I have googled and read a lot of praise for the redis INCR feature. But I don't really understand everything and it does not help me to do this redis-internally only.
My goal:
I want to import 'n' lines of text into redis and retrive them later in exactly this order.
For each line I set a unique key, like key:1, key:2, key:3 etc.
By using an increasing counter as part of the key I am able to later retrieve the lines in the same order in which they were stored in redis.
Now (without redis mass insert) I solve this easily
by using an awk script to generate redis-cli calls, like:
cat data.txt | awk -f myscript.awk | bash
"data.txt" looks like this:
This is the first line.
This here is the much longer second line.
"myscript.awk" looks like this:
#!/usr/bin/awk -f
### This section is being applied before any line is read:
BEGIN {
# Set counter initially to Zero
print "redis-cli SET counter 0"
}
### This section is being applied per line read into awk:
{
# Increase counter by One
print "redis-cli INCR counter"
# Read current counter from redis into an variable
print "MYCOUNTER=`redis-cli GET counter`"
# Use this variable as counter for the key
print "redis-cli SET key:$MYCOUNTER \"" $0 "\""
# Retrive stored value from redis for illustration
print "redis-cli GET key:$MYCOUNTER"
}
The output of "cat data.txt | awk -f myscript.awk | bash" is:
OK
(integer) 1
OK
"This is the first line."
(integer) 2
OK
"This here is the much longer second line."
So everything is fine.
But instead of calling "redis-cli" twice per imported line
I want to use the redis "mass insert" feature. Here I need your help:
How would I do something like this in redis only ?
SET counter 0
=> OK
INCR counter
=> (integer) 1
GET counter
=> "1"
SET KEY:{counter} "Content of line 1"
=> OK
INCR counter
=> (integer) 2
GET counter
=> "2"
SET KEY:{counter} "Different content of line 2"
=> OK
etc., etc.
The "GET counter" lines are just for illustration.
Any help is appreciated. Thanks again !
bernie
Use a list for this. There's no reason to use a new key for each line. All the list commands are here, but the one you want is RPUSH. You can rpush many values at once in the same line, so you'll just do that:
RPUSH some_key line1 line2 ... lineN
Then to retrieve:
LRANGE some_key 0 -1
Quick and easy!
How do I copy a sorted set in redis to a regular, unsorted set? Is there a redis command that can do this? I can manually iterate through the sorted set and manually insert in the unsorted set, but it seems like there might be a better way to do this.
I don't think there is any command to do this directly.
But you can write simple lua script to do it on server instead downloading the sorted sets content to client and then pushing it back to new set.
Redis commands:
SCRIPT LOAD "for i,v in ipairs(redis.call('zrange', KEYS[1], 0, -1)) do redis.call('sadd', KEYS[2], v) end"
ZADD zset 1 first
ZADD zset 2 second
ZADD zset 3 third
EVALSHA dd1c22a22108d758b93c26eb92d1ef2933cec314 2 zset set
SMEMBERS set
Result:
"dd1c22a22108d758b93c26eb92d1ef2933cec314"
(integer) 0
(integer) 0
(integer) 0
(nil)
1) "second"
2) "first"
3) "third"
SCRIPT LOAD defines the script and returns its sha hash, EVALSHA than executes. Arguments are 2 to indicate that 2 key names follows, first is sorted set to copy from, second is set to copy to.