I am facing a quite strange issue in our deployment
After certain time in operation
I could not add a particular list with a particular keyname or listname to Redis using RPUSH.
Example
RPUSH client-send-process-servername TEST
I am unable to add that key to Redis database.
Anyways the output after executing that command i get
(Integer) 1
But i could not see the list
redis-cli keys *client-send*
Return empty list
However when this problem appears
I am able successfully execute the following command.
RPUSH client-send-process-ser TEST
And
redis-cli keys *client-send*
NOTE: one ASTRICK before n aftet client-send string
Which is not displayed here
It is listing the Queue
"client-send-process-ser"
However strangely i could not add a List with specific key say
client-send-process-servername
Any ideas to debug further.. where to look and what to look.
Redis Server Version is 2.8
I tired enabling debug logs in redis and tried to use redis-monitor command.
However nothing relevant could be found explaining this issue. I am eager to find a solution. Please if some one could help me to pursue further would be a great help.
It seems that this problem is not reproducible. However I've tested it in my local pc and found that it is working fine.
Example:
➜ ~ redis-cli
127.0.0.1:6379> RPUSH client-send-process-servername TEST
(integer) 1
127.0.0.1:6379> keys *client-send*
1) "client-send-process-servername"
127.0.0.1:6379> keys client-send*
1) "client-send-process-servername"
127.0.0.1:6379> RPUSH client-send-process-ser TEST
(integer) 1
127.0.0.1:6379> keys client-send*
1) "client-send-process-ser"
2) "client-send-process-servername"
127.0.0.1:6379>
If there is any option, you can try updating your redis server. current updated versions is 5.0.7 and yours is 2.8.
I have a lot of analytics data that I'm adding to redis. I plan on incrementally moving the data out of redis and into my database.
I know I can use KEYS [the_key]:* to get all keys that match. For example, I can do that to get the following:
127.0.0.1:6379> KEYS c_Track:*
1) "c_Track:6c93a5c1-77e9-4c4a-9232-bf182713a02e"
2) "c_Track:2c9d99c2-af37-4de9-ac64-b48f339e97a9"
3) "c_Track:9e7fd190-86d9-4b4a-9a70-7bf4c7768eef"
4) "c_Track:7f2d2e98-7440-4fd7-a80a-2af309ab15a4"
Is there a recommended way to get these values easily? I can get the keys, but how can I get all the values as well? I can loop through the keys to get the values, but is there some one-shot method for doing this?
Also I know I shouldn't use keys, but this is just an example. Thanks
Thanks
Also I know I shouldn't use keys
So don't. Use SCAN instead.
is there some one-shot method for doing this?
No, not as a core Redis command, but given the need this is fairly simple to achieve with a server-side Lua script. For example, assuming that your values are strings, you could do something like the following:
local cursor = tonumber(ARGV[1])
local pattern = ARGV[2]
local scan = redis.call('SCAN', cursor, 'MATCH', pattern)
for i, v in ipairs(scan[2]) do
local val = redis.call('GET', v)
scan[2][i] = { v, val }
end
return scan
Assuming that this script is saved under "scan.lua", you can run it as follows:
$ redis-cli SET foo bar
OK
$ redis-cli SET baz qaz
OK
$ redis-cli --eval scan.lua , 0 "*"
1) "0"
2) 1) 1) "baz"
2) "qaz"
2) 1) "foo"
2) "bar"
To scan your entire keyspace, call the script with the returned cursor until it returns 0.
Notes:
1) If your keys are of different types, you should change the script accordingly (e.g. https://github.com/itamarhaber/redis-lua-scripts/blob/master/scanfetch.lua).
2) While this script goes against the common recommendation of generating key names inside a script, it is still safe to run as SCAN returns keys that are in the server's keyspace (whether single-instance or clustered).
I have seen this pass results to another command in redis
and using via command line this command works well :
src/redis-cli keys '*' | xargs src/redis-cli mget
However how can we achieve the same effect via Lettuce (i started trying out 4.0.2.Final)
Also a solution to this is particularly important in the following scenario :
Say we are using geolocation capabilities, and we add a set of locations of "my-location-category"
using GEOADD
GEOADD "category-1" 8.6638775 49.5282537 "location-id:1" 8.3796281 48.9978127 "location-id:2" 8.665351 49.553302 "location-id:3"
Next, say we do a GeoRadius to get locations within 10 km radius of 8.6582361 49.5285495 for "category-1"
Now when we get "location-id:1" & "location-id:3"
Given that I already set values for above keys "location-id:1" & "location-id:3"
I want to pipe commands to do the GEORADIUS as well as do mget on all the matching results.
Does Redis provide feature to do that?
and / or how can we achieve this via the Lettuce client library without first manually iterating through results of GEORADIUS and then do manual mget.
That would be more efficient performance for the program that uses it.
Does anyone know how we can do this ?
Update
This is the piped command for the scenario I discussed above :
src/redis-cli GEORADIUS "category-1" 8.6582361 49.5285495 10 km | xargs src/redis-cli mget
Now we need to know how to do this via Lettuce
IMPORTANT: never use KEYS, always use SCAN instead if you must.
This isn't really a question about Lettuce nor Java so I can actually answer it :)
What you're trying to do is use the results from a read operation (GEORADIUS) as input (key names) for another read operation (MGET). This type of flow can't be pipelined, well, just because of that - pipelining means that you don't need the answers for operations right away but in you case you do.
However.
Since you're reading String keys with MGET, you might as well just denormalize everything (remember, we're NoSQL) and store the contents of these keys in the Sorted Set's members, e.g.:
GEOADD "category-1" 8.6638775 49.5282537 "location-id:1:moredata:evenmoredata:{maybe a JSON document here}:orperhapsmsgpack"
This will allow you to get the locations and their "data" with one GEORADIUS call. Of course, any updates to location:1's data will need to be done across all categories.
A note about Lua scripts: while a Lua script could definitely save on the back and forth in this case, any such script will be against best practices/not cluster safe.
After digging around and studying Lua script, my conclusion is that removing round-trips in such a way can only be done via Lua scripts as suggested by Itamar Haber.
I ended up creating a lua script file (myscript.lua) as below
local locationKeys = redis.call('GEORADIUS', 'category-1', '8.6582361', '49.5285495', '10', 'km' )
if unpack(locationKeys) == nil then
return nil
else
return redis.call('MGET', unpack(locationKeys))
end
** of course we should be sending in parameters to this... this is just a poc :)
now you can execute it via command
src/redis-cli EVAL "$(cat myscript.lua)" 0
Then to reduce the network-overhead of sending across the entire script to Redis for execution, we have the option of registering the script with Redis.
Redis will give us a sha1 digested code for future references for that script, which can be used for next calls to that script.
This can be done as below :
src/redis-cli SCRIPT LOAD "$(cat myscript.lua)"
this should give back a sha1 code something like this : 49730aa2ed3034ee48f818e486tpbdf1b500b19e
next calls can be done using this code
eg
src/redis-cli evalsha 49730aa2ed3034ee48f818e486b2bdf1b500b19e 0
The sad part however here is that the sha1 digest is remembered only so long as the instance of redis is running. If it is restarted, that the sha1 digest is lost. Then you do the SCRIPT LOAD once again. And if nothing changes in the script, then the sha1-digest code will be the same.
Ideally while using through client api, we should first attempt evalsha, if that returns a "No matching script" error, then as a fallback do script load, and procure the sha1 code once again, and create an internal map of that and use that sha1 code for further calls.
This can well be done via Lettuce. I could find the methods for those. Hope this gives a good insight into solution for the problem.
My redis collection contains many keys
I want to be able to flush them all except all the keys that start with:
"configurations::"
is this possible?
You can do this
redis-cli KEYS "*" | grep -v "configurations::" | xargs redis-cli DEL
List all keys into the redis, remove from the list keys that contains "configurations::" and delete them from the redis
Edit
As #Sergio Tulentsev notice it keys is not for use in production. I used this python script to remove keys on prodution redis. I stoped replication from master to slave before call the script.
#!/usr/bin/env python
import redis
import time
pattern = "yourpattern*"
poolSlave = redis.ConnectionPool(host='yourslavehost', port=6379, db=0)
redisSlave = redis.Redis(connection_pool=poolSlave)
poolMaster = redis.ConnectionPool(host='yourmasterhost', port=6379, db=0)
redisMaster = redis.Redis(connection_pool=poolMaster)
cursor = '0'
while cursor != 0:
cursor, data = redisSlave.scan(cursor, pattern, 1000)
print "cursor: "+str(cursor)
for key in data:
redisMaster.delete(key)
print "delete key: "+key
# reduce call per second on production server
time.sleep(1)
The SCAN & DEL approach (as proposed by #khanou) is the best ad-hoc solution. Alternatively, you could keep an index of all your configurations:: key names with a Redis Set (simply SADD the key's name to it whenever you create a new configurations:: key). Once you have this set you can SSCAN it to get all the relevant key names more efficiently (don't forget to SREM from it whenever you DEL though).
Yes, it's possible. Enumerate all the keys, evaluate each one and delete if it fits the criteria for deletion.
There is no built-in redis command for this, if this is what you were asking.
It might be possible to cook up a Lua script that will do this (and it'll look to your app that it's a single command), but still it's the same approach under the hood.
Is there a Redis command for fetching all keys in the database? I have seen some python-redis libraries fetching them. But was wondering if it is possible from redis-client.
Try to look at KEYS command. KEYS * will list all keys stored in redis.
EDIT: please note the warning at the top of KEYS documentation page:
Time complexity: O(N) with N being the number of keys in the database, under the assumption that the key names in the database and the given pattern have limited length.
UPDATE (V2.8 or greater): SCAN is a superior alternative to KEYS, in the sense that it does not block the server nor does it consume significant resources. Prefer using it.
Updated for Redis 2.8 and above
As noted in the comments of previous answers to this question, KEYS is a potentially dangerous command since your Redis server will be unavailable to do other operations while it serves it. Another risk with KEYS is that it can consume (dependent on the size of your keyspace) a lot of RAM to prepare the response buffer, thus possibly exhausting your server's memory.
Version 2.8 of Redis had introduced the SCAN family of commands that are much more polite and can be used for the same purpose.
The CLI also provides a nice way to work with it:
$ redis-cli --scan --pattern '*'
It can happen that using redis-cli, you connect to your remote redis-server, and then the command:
KEYS *
is not showing anything, or better, it shows:
(empty list or set)
If you are absolutely sure that the Redis server you use is the one you have the data, then maybe your redis-cli is not connecting to the Redis correct database instance.
As it is mentioned in the Redis docs, new connections connect as default to the db 0.
In my case KEYS command was not retrieving results because my database was 1. In order to select the db you want, use SELECT.
The db is identified by an integer.
SELECT 1
KEYS *
I post this info because none of the previous answers was solving my issue.
-->Get all keys from redis-cli
-redis 127.0.0.1:6379> keys *
-->Get list of patterns
-redis 127.0.0.1:6379> keys d??
This will produce keys which start by 'd' with three characters.
-redis 127.0.0.1:6379> keys *t*
This wil get keys with matches 't' character in key
-->Count keys from command line by
-redis-cli keys * |wc -l
-->Or you can use dbsize
-redis-cli dbsize
Get All Keys In Redis
Get all keys using the --scan option:
$ redis-cli --scan --pattern '*'
List all keys using the KEYS command:
$ redis-cli KEYS '*'
Take a look at following Redis Cheat Sheet.
To get a subset of redis keys with the redis-cli i use the command
KEYS "prefix:*"
Yes, you can get all keys by using this
var redis = require('redis');
redisClient = redis.createClient(redis.port, redis.host);
redisClient.keys('*example*', function (err, keys) {
})
SCAN doesn't require the client to load all the keys into memory like KEYS does. SCAN gives you an iterator you can use. I had a 1B records in my redis and I could never get enough memory to return all the keys at once.
Here is a python snippet to get all keys from the store matching a pattern and delete them:
import redis
r = redis.StrictRedis(host='localhost', port=6379, db=0)
for key in r.scan_iter("key_pattern*"):
print key
redis-cli -h <host> -p <port> keys *
where * is the pattern to list all keys
KEYS pattern
Available since 1.0.0.
Time complexity: O(N) with N being the number
of keys in the database, under the assumption that the key names in
the database and the given pattern have limited length.
Returns all keys matching pattern.
Warning : This command is not recommended to use because it may ruin performance when it is executed against large databases instead of KEYS you can use SCAN or SETS.
Example of KEYS command to use :
redis> MSET firstname Jack lastname Stuntman age 35
"OK"
redis> KEYS *name*
1) "lastname"
2) "firstname"
redis> KEYS a??
1) "age"
redis> KEYS *
1) "lastname"
2) "age"
3) "firstname"
In order to get all the keys available in redis server, you should open redis-cli and type:
KEYS *
In order to get more help please visit this page:
This Link
If your redis is a cluster,you can use this script
#!/usr/bin/env bash
redis_list=("172.23.3.19:7001,172.23.3.19:7002,172.23.3.19:7003,172.23.3.19:7004,172.23.3.19:7005,172.23.3.19:7006")
arr=($(echo "$redis_list" | tr ',' '\n'))
for info in ${arr[#]}; do
echo "start :${info}"
redis_info=($(echo "$info" | tr ':' '\n'))
ip=${redis_info[0]}
port=${redis_info[1]}
echo "ip="${ip}",port="${port}
redis-cli -c -h $ip -p $port set laker$port '湖人总冠军'
redis-cli -c -h $ip -p $port keys \*
done
echo "end"
For the ones that wants a typescript helper (using ioredis)
import Redis from 'ioredis';
import { from, Observable, of } from 'rxjs';
import { first, mergeMap } from 'rxjs/operators';
export function scanKeysFromRedis(redisStore: Redis.Redis, key: string,
target: number = 0, keys: string[] = []): Observable<string[]> {
return from(redisStore.scan(target, 'MATCH', key)).pipe(
first(),
mergeMap((_keys) => {
const _target = Number(_keys[0]);
if (_target !== 0) {
return scanKeysFromRedis(redisStore, key, _target, [...keys, ..._keys[1]]);
}
return of([...keys, ..._keys[1]]);
}),
);
}
and call it with: scanKeysFromRedis(store, 'hello');
If you are using Laravel Framework then you can simply use this:
$allKeyList = Redis::KEYS("*");
print_r($allKeyList);
In Core PHP:
$redis = new Redis();
$redis->connect('hostname', 6379);
$allKeyList = $redis->keys('*');
print_r($allKeyList);
You can simply connect to your redis server using redis-cli, select your database and type KEYS *, please remember it will give you all the keys present in selected redis database.
We should be using --scan --pattern with redis 2.8 and later.
You can try using this wrapper on top of redis-cli.
https://github.com/VijayantSoni/redis-helper