I want to use Redis's HSCAN command in my assignment but I have no idea how it works. Redis's official page (http://redis.io/commands/hscan) for this command gives me blank page.
I am getting continuous input data and saving them instantaneously in multiple hashes in Redis and I would like to iterate through all of them at later point of time.
I'm using following command to save my data
HMSET myhash ordertype "neworder" order_ts "1234" act_type "order_ack" ack_ts "1240"
HMSET myhash2 ordertype "neworder" order_ts "2234" act_type "order_ack" ack_ts "2240"
Can anyone give me some examples of how to use HSCAN?
In my case I would like to get following output
1) myhash
2) myhash2
3) myhash3
.
.
.
.
Commands
Start a full hash scan with:
HSCAN myhash 0
Start a hash scan with fields matching a pattern with:
HSCAN myhash 0 MATCH order_*
Start a hash scan with fields matching a pattern and forcing the scan command to do more scanning with:
HSCAN myhash 0 MATCH order_* COUNT 1000
Note
Don't forget that MATCH can return little to no element for each iteration, as explained in the documentation:
It is important to note that the MATCH filter is applied after
elements are retrieved from the collection, just before returning data
to the client. This means that if the pattern matches very little
elements inside the collection, SCAN will likely return no elements in
most iterations.
And that's why you can use COUNT to force more scanning for each iteration.
[Update] As Didier Spezia specified, you'll need Redis 2.8+ to use the *SCAN commands.
As mentioned by you. you need to get the output of hash keys
myhash
myhash2
myhash3
HSCAN is not for this purpose. HSCAN is to scan the fields of a particular HASH. so you can scan the fields of myhash or myhash2. But if you want to find the keys on the basis of patterns you have two options.
Create a SET with the HASH keys
SADD hashkeys "myhash" "myhash1" "myhash2"
SMEMBERS hashkeys
1) "myhash"
2) "myhash2"
3) "myhash1"
If you are using 2.8 I would recommend you use sscan instead of smembers
sscan hashkeys 0 match my*
USE ONLY SCAN COMMAND
scan 0 MATCH myhash* count 1000
USE key matching (NOT RECOMMENDED AS it will block the redis server)
keys myhash*
So long story cut sort to fetch the keys you can use SMEMBERS, SSCAN or KEYS. Ofcourse best is SSCAN if you are using redis 2.8
Related
I am trying to run the code from the redis transactions page. Specifically, this part:
WATCH zset
element = ZRANGE zset 0 0
MULTI
ZREM zset element
EXEC
If I try to do it from the cli, line by line, I get this:
localhost:6380> zadd set 1 a
(integer) 1
localhost:6380> WATCH zset
localhost:6380> element = ZRANGE zset 0 0
(error) ERR unknown command 'element'
OK
which probably means I'm doing something wrong? I remember working with lua about 9 years ago, so this doesn't really look like lua either to me.
How does someone run that snippet? Is it only some kind of pseudocode?
As #Dinei said, the example given is pseudocode.
Let's look at it (I added line numbers for us to refer to):
1 WATCH zset
2 element = ZRANGE zset 0 0
3 MULTI
4 ZREM zset element
5 EXEC
The point of the exercise is to solve the race condition that would occur if we only read the key (with ZRANGE, in line 2), and then modify the key (with ZREM in line 4). I assume you understand the problem if we didn't use the "CAS" semantics, so no need to get into it.
As pointed out, redis-cli just gives you the ability to run redis commands and see their replies, but not save values in variables, etc.
So the idea of the example is that in line 2, we are "saving" the result of the "read" operation, into a pseudo-variable element.
Then, in line 4, we are using that value in our "set" operation, and of course lines 1, 3 and 5 are just the "CAS" commands to ensure there is no race condition.
Presumably the actual usage of such commands would be done from a redis client in a programming language that would allow us to save the return value of the ZRANGE and then use it later in the ZREM command.
But if you wanted to run it in redis-cli, you'd see this, where we pretend that our client-side code would have read and saved "a" that was returned from zrange and then passed that value to the zrem command:
127.0.0.1:6379> zadd zset 1 a
(integer) 1
127.0.0.1:6379> watch zset
OK
127.0.0.1:6379> zrange zset 0 0
1) "a"
127.0.0.1:6379> multi
OK
127.0.0.1:6379> zrem zset a
QUEUED
127.0.0.1:6379> exec
1) (integer) 1
127.0.0.1:6379>
Yes, it is some kind of pseudocode.
redis-cli only accepts Redis commands, it is not a full-fledged editor nor supports direct Lua scripting (neither variables like the element variable in the pseudocode).
I remember working with lua about 9 years ago, so this doesn't really look like lua either to me.
This is not Lua, it is pseudocode. Actually, the Redis Transactions page you linked to does not refer to Lua at all (and that's why #Piglet's comment in your post makes sense).
However, it is possible to execute Lua scripts by using Redis' EVAL command.
I'm new to Redis. How can I get the memory footprint of a specific key in redis?
db0
1) "unacked_mutex"
2) "_kombu.binding.celery"
3) "_kombu.binding.celery.pidbox"
4) "_kombu.binding.celeryev"
I just want to get the memory footprint of one specific key like "_kombu.binding.celery" , or one specific db like db0 , how can I get it?
redis_version: 2.8.17
Any commentary is very welcome. great thanks.
You are running a very old version of redis. The MEMORY command is not available in that version, so there is no precise way of getting at this information. However, you can approximate this information using the DUMP command.
Simply call DUMP _kombu.binding.celery and save the results to a file. The result is some characters and escape sequences. When you load this file into an environment like node, you can look at the length of the string and multiply by 2 to get the number of bytes. This is not precise, but it will give you a generally close approximation.
Here's what you could do:
in Redis:
$ redis-cli
127.0.0.1:6379> hset c 123 456
(integer) 0
127.0.0.1:6379> dump c
"\r\x12\x12\x00\x00\x00\r\x00\x00\x00\x02\x00\x00\xfe{\x03\xc0\xc8\x01\xff\t\x00\x10\xd4L \x908\x8b2"
in Node:
$ node
> a="\r\x12\x12\x00\x00\x00\r\x00\x00\x00\x02\x00\x00\xfe{\x03\xc0\xc8\x01\xff\t\x00\x10\xd4L \x908\x8b2"
'\r\u0012\u0012\u0000\u0000\u0000\r\u0000\u0000\u0000\u0002\u0000\u0000þ{\u0003ÀÈ\u0001ÿ\t\u0000\u0010ÔL 82'
> a.length
30
This is close to half of the actual amount that redis provides with MEMORY USAGE:
127.0.0.1:6379> MEMORY USAGE c
(integer) 63
MEMORY USAGE _kombu.binding.celery would give you the number of bytes that a key and value require to be stored in RAM.
Here is the doc for the command.
I have a lot of analytics data that I'm adding to redis. I plan on incrementally moving the data out of redis and into my database.
I know I can use KEYS [the_key]:* to get all keys that match. For example, I can do that to get the following:
127.0.0.1:6379> KEYS c_Track:*
1) "c_Track:6c93a5c1-77e9-4c4a-9232-bf182713a02e"
2) "c_Track:2c9d99c2-af37-4de9-ac64-b48f339e97a9"
3) "c_Track:9e7fd190-86d9-4b4a-9a70-7bf4c7768eef"
4) "c_Track:7f2d2e98-7440-4fd7-a80a-2af309ab15a4"
Is there a recommended way to get these values easily? I can get the keys, but how can I get all the values as well? I can loop through the keys to get the values, but is there some one-shot method for doing this?
Also I know I shouldn't use keys, but this is just an example. Thanks
Thanks
Also I know I shouldn't use keys
So don't. Use SCAN instead.
is there some one-shot method for doing this?
No, not as a core Redis command, but given the need this is fairly simple to achieve with a server-side Lua script. For example, assuming that your values are strings, you could do something like the following:
local cursor = tonumber(ARGV[1])
local pattern = ARGV[2]
local scan = redis.call('SCAN', cursor, 'MATCH', pattern)
for i, v in ipairs(scan[2]) do
local val = redis.call('GET', v)
scan[2][i] = { v, val }
end
return scan
Assuming that this script is saved under "scan.lua", you can run it as follows:
$ redis-cli SET foo bar
OK
$ redis-cli SET baz qaz
OK
$ redis-cli --eval scan.lua , 0 "*"
1) "0"
2) 1) 1) "baz"
2) "qaz"
2) 1) "foo"
2) "bar"
To scan your entire keyspace, call the script with the returned cursor until it returns 0.
Notes:
1) If your keys are of different types, you should change the script accordingly (e.g. https://github.com/itamarhaber/redis-lua-scripts/blob/master/scanfetch.lua).
2) While this script goes against the common recommendation of generating key names inside a script, it is still safe to run as SCAN returns keys that are in the server's keyspace (whether single-instance or clustered).
In Redis, keys user* will print all keys starting with user.
For example:
keys user*
1) "user2"
2) "user1"
Now, I want all keys that don't start with user to be printed.
How could I do that?
IMPORTANT: always use SCAN instead of (the evil) KEYS
Redis' pattern matching is somewhat functionally limited (see the implementation of stringmatchlen in util.c) and does not provide that which you seek ATM. That said, consider the following possible routes:
Extend stringmatchlen to match your requirements, possibly submitting it as a PR.
Consider what you're trying to do - fetching a subset of keys is always going to be inefficient unless you index them, consider tracking the names of all non-user keys (i.e.g. in a Redis Set) instead.
If you are really insistent on scanning the entire keyspace and match against negative patterns, one way to accomplish that is with a little bit of Lua magic.
Consider the following dataset and script:
127.0.0.1:6379> dbsize
(integer) 0
127.0.0.1:6379> set user:1 1
OK
127.0.0.1:6379> set use:the:force luke
OK
127.0.0.1:6379> set non:user a
OK
Lua (save this as scanregex.lua):
local re = ARGV[1]
local nt = ARGV[2]
local cur = 0
local rep = {}
local tmp
if not re then
re = ".*"
end
repeat
tmp = redis.call("SCAN", cur, "MATCH", "*")
cur = tonumber(tmp[1])
if tmp[2] then
for k, v in pairs(tmp[2]) do
local fi = v:find(re)
if (fi and not nt) or (not fi and nt) then
rep[#rep+1] = v
end
end
end
until cur == 0
return rep
Output - first time regular matching, 2nd time the complement:
foo#bar:~$ redis-cli --eval scanregex.lua , "^user"
1) "user:1"
foo#bar:~$ redis-cli --eval scanregex.lua , "^user" 1
1) "use:the:force"
2) "non:user"
#Karthikeyan Gopall you nailed it in your comment above and this saved me a bunch of time. Thanks!
Here's how you can use it in various combinations to get what you want:
redis.domain.com:6379[1]> set "hello" "foo"
OK
redis.domain.com:6379[1]> set "hillo" "bar"
OK
redis.domain.com:6379[1]> set "user" "baz"
OK
redis.domain.com:6379[1]> set "zillo" "bash"
OK
redis.domain.com:6379[1]> scan 0
1) "0"
2) 1) "zillo"
2) "hello"
3) "user"
4) "hillo"
redis.domain.com:6379[1]> scan 0 match "[^u]*"
1) "0"
2) 1) "zillo"
2) "hello"
3) "hillo"
redis.domain.com:6379[1]> scan 0 match "[^u^z]*"
1) "0"
2) 1) "hello"
2) "hillo"
redis.domain.com:6379[1]> scan 0 match "h[^i]*"
1) "0"
2) 1) "hello"
According to redis keys documentation the command supports glob style patterns, not regular expressions.
and if you look at the documentation, you'll see that the "!" character is not special as opposites to regular expressions.
Here is a simple test I ran in my own db:
redis 127.0.0.1:6379> set a 0
OK
redis 127.0.0.1:6379> set b 1
OK
redis 127.0.0.1:6379> keys *
1) "a"
2) "b"
redis 127.0.0.1:6379> keys !a
(empty list or set) // I expected "b" here
redis 127.0.0.1:6379> keys !b
(empty list or set) // I expected "a" here
redis 127.0.0.1:6379> keys [!b]
1) "b"
redis 127.0.0.1:6379> keys [b]
1) "b"
redis 127.0.0.1:6379> keys [ab]
1) "a"
2) "b"
redis 127.0.0.1:6379> keys ![b]
(empty list or set)
So I just don't think what you are trying to achieve is possible via the keys command.
Besides, the keys command is not very suitable for production environment as it locks your whole redis database.
I would recommend getting all the keys with the scan command, store them locally, and then remove them using LUA
Here's a trick to achieve this with native redis commands (no need for Lua scripts or anything).
If you are able to control the timing of when you insert the new keys (the ones you want to keep, deleting all other stuff like in your question), you can:
Before setting the new keys, set the expiration to all existing keys (by pattern or everything) to expire right now (see how)
Load the new keys
Redis will automatically delete all the older keys and you will be left just with the new ones you want.
You also can print all keys and pass it to grep. For example:
redis-cli -a <password> keys "*" | grep -v "user"
I have a string type key value store in redis having keys like this--
/url-pattern/url-slug-1
/url-pattern/url-slug-2
/url-pattern/url-slug-3
/url-pattern/url-slug-4 ...
I can retrieve all the keys of /url-pattern/ using a wild card query like this --
keys /url-pattern/*
I would like to retrieve the values of all keys corresponding to this wildcard /url-pattern/*
I tried this
mget /url-pattern/*
1) (nil)
but it doesnt returned the array as expected.
How can I retrieve the values of all keys corresponding to /url-pattern/*
I also want to do a sum on the values, but I think there is no such thing called SUM() in redis
MGET accepts multiple arguments where each a key name. It does not do key name patterns.
What you could do is first fetch all the relevant key names (do not use KEYS, use SCAN instead) and then fetch their values with an MGET.
Here is an updated answer for 2015.
If you can upgrade Redis above 2.8, the SCAN command with MATCH will work for this. Before that version, not so much, and do NOT use the KEYS command except in a development environment.
http://redis.io/commands/scan
Example on command line:
$ redis-cli
127.0.0.1:6379> scan match V3.0:*
(error) ERR invalid cursor
127.0.0.1:6379> scan 0 match V3.0:*
1) "0"
2) 1) "V3.0:UNITTEST55660BC7E0C5B"
2) "V3.0:shop.domain.com:route"
3) "V3.0:UNITTEST55660BC4A2548"
127.0.0.1:6379> scan 0 match V1.0:*
1) "0"
2) (empty list or set)
127.0.0.1:6379> scan 0 match V3.0:*
1) "0"
2) 1) "V3.0:UNITTEST55660BC7E0C5B"
2) "V3.0:shop.domain.com:route"
3) "V3.0:UNITTEST55660BC4A2548"
Example in PHP:
// Initialize our iterator to NULL
$iterate = null;
// retry when we get no keys back
$redis->setOption(Redis::OPT_SCAN, Redis::SCAN_RETRY);
while ($arr_keys = $redis->scan($iterate, 'match:*')) {
foreach ($arr_keys as $str_key) {
echo "Here is a key: $str_key\n";
}
echo "No more keys to scan!\n";
}
Note, php code is not tested and from the core documentation for example here. Production code would need to be modified depending on the keys needed to look up.
For those on Ubuntu here are the instructions to upgrade php5-redis:
Download the 2.2.7 package here: http://pecl.php.net/package/redis
$ php -i | grep Redis
Redis Support => enabled
Redis Version => 2.2.4
Follow instructions in README to phpize, configure, make install
Create a symlink for command line cli package: cd /etc/php5/cli/conf.d && sudo ln -s ../../mods-available/redis.ini 20-redis.ini
$ php -i | grep Redis
Redis Support => enabled
Redis Version => 2.2.7
There is NO command available in REDIS which can return values from wildcard keys.
If you see the documentation for KEYS command: http://redis.io/commands/keys, it says
Consider KEYS as a command that should only be used in production
environments with extreme care. It may ruin performance when it is
executed against large databases. This command is intended for
debugging and special operations. Don't use KEYS in your regular
application code.
I don't know your business use case, but looks like you may have to use different data structure for this requirement. You can use list or set to store similar url patterns.