Related
I have a lot of analytics data that I'm adding to redis. I plan on incrementally moving the data out of redis and into my database.
I know I can use KEYS [the_key]:* to get all keys that match. For example, I can do that to get the following:
127.0.0.1:6379> KEYS c_Track:*
1) "c_Track:6c93a5c1-77e9-4c4a-9232-bf182713a02e"
2) "c_Track:2c9d99c2-af37-4de9-ac64-b48f339e97a9"
3) "c_Track:9e7fd190-86d9-4b4a-9a70-7bf4c7768eef"
4) "c_Track:7f2d2e98-7440-4fd7-a80a-2af309ab15a4"
Is there a recommended way to get these values easily? I can get the keys, but how can I get all the values as well? I can loop through the keys to get the values, but is there some one-shot method for doing this?
Also I know I shouldn't use keys, but this is just an example. Thanks
Thanks
Also I know I shouldn't use keys
So don't. Use SCAN instead.
is there some one-shot method for doing this?
No, not as a core Redis command, but given the need this is fairly simple to achieve with a server-side Lua script. For example, assuming that your values are strings, you could do something like the following:
local cursor = tonumber(ARGV[1])
local pattern = ARGV[2]
local scan = redis.call('SCAN', cursor, 'MATCH', pattern)
for i, v in ipairs(scan[2]) do
local val = redis.call('GET', v)
scan[2][i] = { v, val }
end
return scan
Assuming that this script is saved under "scan.lua", you can run it as follows:
$ redis-cli SET foo bar
OK
$ redis-cli SET baz qaz
OK
$ redis-cli --eval scan.lua , 0 "*"
1) "0"
2) 1) 1) "baz"
2) "qaz"
2) 1) "foo"
2) "bar"
To scan your entire keyspace, call the script with the returned cursor until it returns 0.
Notes:
1) If your keys are of different types, you should change the script accordingly (e.g. https://github.com/itamarhaber/redis-lua-scripts/blob/master/scanfetch.lua).
2) While this script goes against the common recommendation of generating key names inside a script, it is still safe to run as SCAN returns keys that are in the server's keyspace (whether single-instance or clustered).
My redis collection contains many keys
I want to be able to flush them all except all the keys that start with:
"configurations::"
is this possible?
You can do this
redis-cli KEYS "*" | grep -v "configurations::" | xargs redis-cli DEL
List all keys into the redis, remove from the list keys that contains "configurations::" and delete them from the redis
Edit
As #Sergio Tulentsev notice it keys is not for use in production. I used this python script to remove keys on prodution redis. I stoped replication from master to slave before call the script.
#!/usr/bin/env python
import redis
import time
pattern = "yourpattern*"
poolSlave = redis.ConnectionPool(host='yourslavehost', port=6379, db=0)
redisSlave = redis.Redis(connection_pool=poolSlave)
poolMaster = redis.ConnectionPool(host='yourmasterhost', port=6379, db=0)
redisMaster = redis.Redis(connection_pool=poolMaster)
cursor = '0'
while cursor != 0:
cursor, data = redisSlave.scan(cursor, pattern, 1000)
print "cursor: "+str(cursor)
for key in data:
redisMaster.delete(key)
print "delete key: "+key
# reduce call per second on production server
time.sleep(1)
The SCAN & DEL approach (as proposed by #khanou) is the best ad-hoc solution. Alternatively, you could keep an index of all your configurations:: key names with a Redis Set (simply SADD the key's name to it whenever you create a new configurations:: key). Once you have this set you can SSCAN it to get all the relevant key names more efficiently (don't forget to SREM from it whenever you DEL though).
Yes, it's possible. Enumerate all the keys, evaluate each one and delete if it fits the criteria for deletion.
There is no built-in redis command for this, if this is what you were asking.
It might be possible to cook up a Lua script that will do this (and it'll look to your app that it's a single command), but still it's the same approach under the hood.
I have a string type key value store in redis having keys like this--
/url-pattern/url-slug-1
/url-pattern/url-slug-2
/url-pattern/url-slug-3
/url-pattern/url-slug-4 ...
I can retrieve all the keys of /url-pattern/ using a wild card query like this --
keys /url-pattern/*
I would like to retrieve the values of all keys corresponding to this wildcard /url-pattern/*
I tried this
mget /url-pattern/*
1) (nil)
but it doesnt returned the array as expected.
How can I retrieve the values of all keys corresponding to /url-pattern/*
I also want to do a sum on the values, but I think there is no such thing called SUM() in redis
MGET accepts multiple arguments where each a key name. It does not do key name patterns.
What you could do is first fetch all the relevant key names (do not use KEYS, use SCAN instead) and then fetch their values with an MGET.
Here is an updated answer for 2015.
If you can upgrade Redis above 2.8, the SCAN command with MATCH will work for this. Before that version, not so much, and do NOT use the KEYS command except in a development environment.
http://redis.io/commands/scan
Example on command line:
$ redis-cli
127.0.0.1:6379> scan match V3.0:*
(error) ERR invalid cursor
127.0.0.1:6379> scan 0 match V3.0:*
1) "0"
2) 1) "V3.0:UNITTEST55660BC7E0C5B"
2) "V3.0:shop.domain.com:route"
3) "V3.0:UNITTEST55660BC4A2548"
127.0.0.1:6379> scan 0 match V1.0:*
1) "0"
2) (empty list or set)
127.0.0.1:6379> scan 0 match V3.0:*
1) "0"
2) 1) "V3.0:UNITTEST55660BC7E0C5B"
2) "V3.0:shop.domain.com:route"
3) "V3.0:UNITTEST55660BC4A2548"
Example in PHP:
// Initialize our iterator to NULL
$iterate = null;
// retry when we get no keys back
$redis->setOption(Redis::OPT_SCAN, Redis::SCAN_RETRY);
while ($arr_keys = $redis->scan($iterate, 'match:*')) {
foreach ($arr_keys as $str_key) {
echo "Here is a key: $str_key\n";
}
echo "No more keys to scan!\n";
}
Note, php code is not tested and from the core documentation for example here. Production code would need to be modified depending on the keys needed to look up.
For those on Ubuntu here are the instructions to upgrade php5-redis:
Download the 2.2.7 package here: http://pecl.php.net/package/redis
$ php -i | grep Redis
Redis Support => enabled
Redis Version => 2.2.4
Follow instructions in README to phpize, configure, make install
Create a symlink for command line cli package: cd /etc/php5/cli/conf.d && sudo ln -s ../../mods-available/redis.ini 20-redis.ini
$ php -i | grep Redis
Redis Support => enabled
Redis Version => 2.2.7
There is NO command available in REDIS which can return values from wildcard keys.
If you see the documentation for KEYS command: http://redis.io/commands/keys, it says
Consider KEYS as a command that should only be used in production
environments with extreme care. It may ruin performance when it is
executed against large databases. This command is intended for
debugging and special operations. Don't use KEYS in your regular
application code.
I don't know your business use case, but looks like you may have to use different data structure for this requirement. You can use list or set to store similar url patterns.
Is there a way to print the number of keys in Redis?
I am aware of
keys *
But that seems slightly heavy weight. - Given that Redis is a key value store maybe this is the only way to do it. But I would still like to see something along the lines of
count keys *
You can issue the INFO command, which returns information and statistics about the server. See here for an example output.
As mentioned in the comments by mVChr, you can use info keyspace directly on the redis-cli.
redis> INFO
# Server
redis_version:6.0.6
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:b63575307aaffe0a
redis_mode:standalone
os:Linux 5.4.0-1017-aws x86_64
arch_bits:64
multiplexing_api:epoll
atomicvar_api:atomic-builtin
gcc_version:9.3.0
process_id:2854672
run_id:90a5246f10e0aeb6b02cc2765b485d841ffc924e
tcp_port:6379
uptime_in_seconds:2593097
uptime_in_days:30
hz:10
configured_hz:10
lru_clock:4030200
executable:/usr/local/bin/redis-server
DBSIZE returns the number of keys and it's easier to parse.
Downside: if a key has expired it may still count.
http://redis.io/commands/dbsize
The DBSIZE command returns the number of keys
> DBSIZE
WARNING: Do not run this on a production machine.
On a Linux box:
redis-cli KEYS "*" | wc -l
Note: As mentioned in comments below, this is an O(N) operation, so on a large DB with many keys you should not use this. For smaller deployments, it should be fine.
Since Redis 2.6, lua is supported, you can get number of wildcard keys like this
eval "return #redis.call('keys', 'prefix-*')" 0
see eval command
Go to redis-cli and use below command
info keyspace
It may help someone
Displays the DB's name along with keys count:
redis-cli info keyspace
# Keyspace
db0:keys=12995,expires=0,avg_ttl=0
db12:keys=5524396,expires=5,avg_ttl=45201
Display the no. of keys in the selected database:
redis-cli dbsize
(integer) 12995
Displays whole Redis's stats:
redis-cli info
dbsize() returns the total number of keys.
You can quickly estimate the number of keys matching a given pattern by sampling keys at random, then checking what fraction of them matches the pattern.
Example in python; counting all keys starting with prefix_:
import redis
r = redis.StrictRedis(host = 'localhost', port=6379)
iter=1000
print 'Approximately', r.dbsize() * float(sum([r.randomkey().startswith('prefix_') for i in xrange(iter)])) / iter
Even iter=100 gives a decent estimate in my case, yet is very fast, compared to keys prefix_.
An improvement is to sample 1000 keys on every request, but keep the total count, so that after two requests you'll divide by 2000, after three requests you'll divide by 3000. Thus, if your application is interested in the total number of matching keys fairly often, then every time it will get closer and closer to the true value.
After Redis 2.6, the result of INFO command are splitted by sections. In the "keyspace" section, there are "keys" and "expired keys" fields to tell how many keys are there.
eval "local count = redis.call('scan', 0, 'match', 'key:*:key', 'count', 10000) if count ~= 0 then return #count[2] end " 0
eval "local count = redis.call('sscan', 'key.key:all', 0, 'match', '*', 'count', 1000000) if count ~= 0 then return #count[2] end " 0
Is there a Redis command for fetching all keys in the database? I have seen some python-redis libraries fetching them. But was wondering if it is possible from redis-client.
Try to look at KEYS command. KEYS * will list all keys stored in redis.
EDIT: please note the warning at the top of KEYS documentation page:
Time complexity: O(N) with N being the number of keys in the database, under the assumption that the key names in the database and the given pattern have limited length.
UPDATE (V2.8 or greater): SCAN is a superior alternative to KEYS, in the sense that it does not block the server nor does it consume significant resources. Prefer using it.
Updated for Redis 2.8 and above
As noted in the comments of previous answers to this question, KEYS is a potentially dangerous command since your Redis server will be unavailable to do other operations while it serves it. Another risk with KEYS is that it can consume (dependent on the size of your keyspace) a lot of RAM to prepare the response buffer, thus possibly exhausting your server's memory.
Version 2.8 of Redis had introduced the SCAN family of commands that are much more polite and can be used for the same purpose.
The CLI also provides a nice way to work with it:
$ redis-cli --scan --pattern '*'
It can happen that using redis-cli, you connect to your remote redis-server, and then the command:
KEYS *
is not showing anything, or better, it shows:
(empty list or set)
If you are absolutely sure that the Redis server you use is the one you have the data, then maybe your redis-cli is not connecting to the Redis correct database instance.
As it is mentioned in the Redis docs, new connections connect as default to the db 0.
In my case KEYS command was not retrieving results because my database was 1. In order to select the db you want, use SELECT.
The db is identified by an integer.
SELECT 1
KEYS *
I post this info because none of the previous answers was solving my issue.
-->Get all keys from redis-cli
-redis 127.0.0.1:6379> keys *
-->Get list of patterns
-redis 127.0.0.1:6379> keys d??
This will produce keys which start by 'd' with three characters.
-redis 127.0.0.1:6379> keys *t*
This wil get keys with matches 't' character in key
-->Count keys from command line by
-redis-cli keys * |wc -l
-->Or you can use dbsize
-redis-cli dbsize
Get All Keys In Redis
Get all keys using the --scan option:
$ redis-cli --scan --pattern '*'
List all keys using the KEYS command:
$ redis-cli KEYS '*'
Take a look at following Redis Cheat Sheet.
To get a subset of redis keys with the redis-cli i use the command
KEYS "prefix:*"
Yes, you can get all keys by using this
var redis = require('redis');
redisClient = redis.createClient(redis.port, redis.host);
redisClient.keys('*example*', function (err, keys) {
})
SCAN doesn't require the client to load all the keys into memory like KEYS does. SCAN gives you an iterator you can use. I had a 1B records in my redis and I could never get enough memory to return all the keys at once.
Here is a python snippet to get all keys from the store matching a pattern and delete them:
import redis
r = redis.StrictRedis(host='localhost', port=6379, db=0)
for key in r.scan_iter("key_pattern*"):
print key
redis-cli -h <host> -p <port> keys *
where * is the pattern to list all keys
KEYS pattern
Available since 1.0.0.
Time complexity: O(N) with N being the number
of keys in the database, under the assumption that the key names in
the database and the given pattern have limited length.
Returns all keys matching pattern.
Warning : This command is not recommended to use because it may ruin performance when it is executed against large databases instead of KEYS you can use SCAN or SETS.
Example of KEYS command to use :
redis> MSET firstname Jack lastname Stuntman age 35
"OK"
redis> KEYS *name*
1) "lastname"
2) "firstname"
redis> KEYS a??
1) "age"
redis> KEYS *
1) "lastname"
2) "age"
3) "firstname"
In order to get all the keys available in redis server, you should open redis-cli and type:
KEYS *
In order to get more help please visit this page:
This Link
If your redis is a cluster,you can use this script
#!/usr/bin/env bash
redis_list=("172.23.3.19:7001,172.23.3.19:7002,172.23.3.19:7003,172.23.3.19:7004,172.23.3.19:7005,172.23.3.19:7006")
arr=($(echo "$redis_list" | tr ',' '\n'))
for info in ${arr[#]}; do
echo "start :${info}"
redis_info=($(echo "$info" | tr ':' '\n'))
ip=${redis_info[0]}
port=${redis_info[1]}
echo "ip="${ip}",port="${port}
redis-cli -c -h $ip -p $port set laker$port '湖人总冠军'
redis-cli -c -h $ip -p $port keys \*
done
echo "end"
For the ones that wants a typescript helper (using ioredis)
import Redis from 'ioredis';
import { from, Observable, of } from 'rxjs';
import { first, mergeMap } from 'rxjs/operators';
export function scanKeysFromRedis(redisStore: Redis.Redis, key: string,
target: number = 0, keys: string[] = []): Observable<string[]> {
return from(redisStore.scan(target, 'MATCH', key)).pipe(
first(),
mergeMap((_keys) => {
const _target = Number(_keys[0]);
if (_target !== 0) {
return scanKeysFromRedis(redisStore, key, _target, [...keys, ..._keys[1]]);
}
return of([...keys, ..._keys[1]]);
}),
);
}
and call it with: scanKeysFromRedis(store, 'hello');
If you are using Laravel Framework then you can simply use this:
$allKeyList = Redis::KEYS("*");
print_r($allKeyList);
In Core PHP:
$redis = new Redis();
$redis->connect('hostname', 6379);
$allKeyList = $redis->keys('*');
print_r($allKeyList);
You can simply connect to your redis server using redis-cli, select your database and type KEYS *, please remember it will give you all the keys present in selected redis database.
We should be using --scan --pattern with redis 2.8 and later.
You can try using this wrapper on top of redis-cli.
https://github.com/VijayantSoni/redis-helper