Is there a way to get around this when using RediSearch FT.SEARCH?
I basically want to get all the possible results for a search. I can page through them using the LIMIT offset count parameters but it errors out when I hit an offset higher than 10000.
It appears that when using redis/redis-stack-server you are limited to the offset number. I stood up an enterprise redis instance on azure and was able to use the limit with an offset higher than 100000.
You can change that 10000 ceiling by setting the MAXSEARCHRESULTS property. You can set it on the module installation, see https://redis.io/docs/stack/search/configuring/ like:
loadmodule ./redisearch.so MAXSEARCHRESULTS 100000
or at runtime on the CLI or via your client like:
127.0.0.1:6379> FT.CONFIG GET MAXSEARCHRESULTS
1) 1) MAXSEARCHRESULTS
2) 10000
127.0.0.1:6379> FT.CONFIG SET MAXSEARCHRESULTS 100000
OK
127.0.0.1:6379> FT.CONFIG GET MAXSEARCHRESULTS
1) 1) MAXSEARCHRESULTS
2) 100000
127.0.0.1:6379>
Related
I've recently discovered that redis has a property graph model implementation called redis graph and it's amazing.
One thing that I really miss for my use-case though, is the ability to "watch" the data. In typical redis data structures I can enable Keyspace notifications or client tracking and be notified on the data mutations I'm interested in, pull data from the server or mark my local cache as "dirty".
I don't know how that would work for a property graph since relations are much more complex (and the key feature for that matter), but is there a way to watch or synchronize with data stored in redis graph?
Keyspace notifications can be enabled for modules like this:
redis.cloud> CONFIG SET notify-keyspace-events AKE
The 'A' part includes modules—if the module publishes anything. Unfortunately, I tried this with RedisGraph, and it doesn't.
You can reproduce my test below. In one terminal I launched redis-cli and did this:
127.0.0.1:6379> PSUBSCRIBE *
Reading messages... (press Ctrl-C to quit)
1) "psubscribe"
2) "*"
3) (integer) 1
In another I did this:
127.0.0.1:6379> GRAPH.QUERY test 'CREATE (n:foo { alfa: 12, bravo: "test" })'
1) 1) "Labels added: 1"
2) "Nodes created: 1"
3) "Properties set: 2"
4) "Cached execution: 0"
5) "Query internal execution time: 0.204701 milliseconds"
127.0.0.1:6379> GRAPH.QUERY test 'MATCH (n) RETURN n'
1) 1) "n"
2) 1) 1) 1) 1) "id"
2) (integer) 0
2) 1) "labels"
2) 1) "foo"
3) 1) "properties"
2) 1) 1) "alfa"
2) (integer) 12
2) 1) "bravo"
2) "test"
3) 1) "Cached execution: 0"
2) "Query internal execution time: 1.106191 milliseconds"
127.0.0.1:6379> GRAPH..DELETE test
"Graph removed, internal execution time: 0.064498 milliseconds"
The first terminal returned nothing in response to this.
No. Unfortunately, currently, there is no way to trigger a keyspace notification whenever nodes or relationships are created, removed, or updated.
We plan to add such functionality in the future, but there is no specific date we can share right now.
I am trying to use Redis Rate limiting patter as specified in https://redis.io/commands/incr under "Pattern: Rate limiter 1". But how can I scale this in case I want to do rate limiting across multiple servers. Like I have service deployed across 5 servers behind load balancer and I want total requests per api key across 5 servers should not cross x/sec. As per redis pattern I mentioned , the problem is that if I have my rate limiter running in multiple servers itself, then the two different request to two different rate limiter servers,can do "get key" at same time and read same value, before anyone updates it, which can probably allow more requests to go.How can I handle this?I can obviously put get in MULTI block, but I think it will make things a lot more slow.
INCR replies with the updated value. So it can be used as both write and read command.
FUNCTION LIMIT_API_CALL(ip)
ts = CURRENT_UNIX_TIME()
keyname = ip+":"+ts
MULTI
INCR(keyname)
EXPIRE(keyname,10)
EXEC
current = RESPONSE_OF_INCR_WITHIN_MULTI
IF current > 10 THEN
ERROR "too many requests per second"
ELSE
PERFORM_API_CALL()
END
You need to run LUA script that will check rate-limiting and increase/decrease/reset the counter(s).
You can find a simple example in Larval framework here
https://github.com/laravel/framework/blob/8.x/src/Illuminate/Redis/Limiters/DurationLimiter.php
/**
* Get the Lua script for acquiring a lock.
*
* KEYS[1] - The limiter name
* ARGV[1] - Current time in microseconds
* ARGV[2] - Current time in seconds
* ARGV[3] - Duration of the bucket
* ARGV[4] - Allowed number of tasks
*
* #return string
*/
protected function luaScript()
{
return <<<'LUA'
local function reset()
redis.call('HMSET', KEYS[1], 'start', ARGV[2], 'end', ARGV[2] + ARGV[3], 'count', 1)
return redis.call('EXPIRE', KEYS[1], ARGV[3] * 2)
end
if redis.call('EXISTS', KEYS[1]) == 0 then
return {reset(), ARGV[2] + ARGV[3], ARGV[4] - 1}
end
if ARGV[1] >= redis.call('HGET', KEYS[1], 'start') and ARGV[1] <= redis.call('HGET', KEYS[1], 'end') then
return {
tonumber(redis.call('HINCRBY', KEYS[1], 'count', 1)) <= tonumber(ARGV[4]),
redis.call('HGET', KEYS[1], 'end'),
ARGV[4] - redis.call('HGET', KEYS[1], 'count')
}
end
return {reset(), ARGV[2] + ARGV[3], ARGV[4] - 1}
LUA;
}
The accepted answer is incorrect. It leads to inaccurate counter value. Let's look a the following example:
5 clients executing 5 concurrent requests to Redis. The current state of the counter is 10, and while the limit is also 10.
The 5 concurrent requests will increment the counter to 15 while declining each of the requests. Rather, the value should remain 10 to reflect on the correct number of times clients were "allowed".
Solution:
we effectively needs to combine two separate atomic operations into a single atomic operation. That's where LUA script comes in. It's simply a modification on Redis it self to introduce another code path that "executes a get, and then a set" atomically. And it does so because Redis is single thread.
My setting is Redis master-slave replication. I am sure the slaves are read only because when I connect to slave and try to write data, "(error) READONLY You can't write against a read only slave." is returned.
However, when I check the slowlog there are SET commands, eg:
127.0.0.1:6379> slowlog get 1
1) 1) (integer) 1360
2) (integer) 1544276677
3) (integer) 10653
4) 1) "SET"
2) "some value"
Anyone can explain this? Thanks in advance.
The Redis replica is replaying commands sent from the master, so the SET command must have originated from it.
It is unclear why that command ended in the slowlog, but it could be because of any number of reasons (IO or CPU blockage). If this happened once I wouldn't worry about it, but if it is pathological you may want to review your replica's infrastructure and configuration.
I am using Redis for cache in my application which is configured in spring beans, spring-data-redis 1.7.1, jedis 2.9.0.
I would like to know how to set the race condition ttl in the configuration.
Please comment if you have any suggestions.
If I understand you right, you want the same as that Ruby repo, but in Java.
For that case you may want to put a technical lock key along the one you need.
get yourkey
(nil)
get <yourkey>::lock
// if (nil) then calculate, if t then wait. assuming (nil) here
setex <yourkey>::lock 30 t
OK
// calcultions
set <yourkey> <result>
OK
del <yourkey>::lock
(integer) 1
Here with setex you set a lock key with TTL of 30 sec. You can put another TTL if you want.
There is one problem with the code above - some time will pass before checking a lock and aquiring it. To properly aquire a lock EVAL can be used: eval "local lk=KEYS[1]..'::lock' local lock=redis.call('get',lk) if (lock==false) then redis.call('setex',lk,KEYS[2],'t') return 1 else return 0 end" 2 <yourkey> 30 This either returns 0 if there is no lock or puts a lock and returns 1.
When I run the info command in redis-cli against a redis 3.2.4 server, it shows me this for expires:
expires=223518
However, when I then run a keys * command and ask for the ttl for each key, and only print out keys with a ttl > 0, I only see a couple hundred.
I thought that the expires is a count of the number of expiring keys but I am not even within an order of magnitude of this number.
Can someone clarify exactly what expires is meant to convey? Does this include both to-be-expired and previously expired but not yet evicted keys?
Update:
Here is how I counted the number of keys expiring:
task count_tmp_keys: :environment do
redis = Redis.new(timeout: 100)
keys = redis.keys '*'
ct_expiring = 0
keys.each do |k|
ttl = redis.ttl(k)
if ttl > 0
ct_expiring += 1
puts "Expiring: #{k}; ttl is #{ttl}; total: #{ct_expiring}"
STDOUT.flush
end
end
puts "Total expiring: #{ct_expiring}"
puts "Done at #{Time.now}"
end
When I ran this script it shows I have a total expiring of 78
When I run info, it says db0:keys=10237963,expires=224098,avg_ttl=0
Because 224098 is so much larger than 78, I am very confused. Is there perhaps a better way for me to obtain a list of all 225k expiring keys?
Also, how is it that my average ttl is 0? Wouldn't you expect it to be nonzero?
UPDATE
I have new information and a simple, 100% repro of this situation locally!
To repro: setup two redis processes locally on your laptop. Make one a slave of the other. On the slave process, set the following:
config set slave-serve-stale-data yes
config set slave-read-only no
Now, connect to the slave (not the master) and run:
set foo 1
expire foo 10
After 10 seconds, you will no longer be able to access foo, but info command will still show that you have 1 key expiring with an average ttl of 0.
Can someone explain this behavior?
expires contains existing keys with TTL which will expire, not including already expired keys.
Example ( with omission of extra information from info command for brevity ):
127.0.0.1:6379> flushall
OK
127.0.0.1:6379> SETEX mykey1 1000 "1"
OK
127.0.0.1:6379> SETEX mykey2 1000 "2"
OK
127.0.0.1:6379> SETEX mykey3 1000 "3"
OK
127.0.0.1:6379> info
# Keyspace
db0:keys=3,expires=3,avg_ttl=992766
127.0.0.1:6379> SETEX mykey4 1 "4"
OK
127.0.0.1:6379> SETEX mykey5 1 "5"
OK
127.0.0.1:6379> info
# Keyspace
db0:keys=3,expires=3,avg_ttl=969898
127.0.0.1:6379> keys *
1) "mykey2"
2) "mykey3"
3) "mykey1"
127.0.0.1:6379>
Given that in your situation you are asking about key expiry on slaves, per https://github.com/antirez/redis/issues/2861:
keys on a slave are not actively expired, and thus the avg_ttl is
never calculated
And per https://groups.google.com/forum/#!topic/redis-db/NFTpdmpOPnc:
avg_ttl is never initialized on a slave and thus it can be what ever
arbitrary value resides in memory at that place.
Thus, it is to be expected that the info command behaves differently on slaves.
The expires just returns the size of keys that will expire not the time.
The source code of 3.2.4
long long keys, vkeys;
keys = dictSize(server.db[j].dict);
vkeys = dictSize(server.db[j].expires);
if (keys || vkeys) {
info = sdscatprintf(info,
"db%d:keys=%lld,expires=%lld,avg_ttl=%lld\r\n",
j, keys, vkeys, server.db[j].avg_ttl);
}
It just calculate the size of server.db[j].expires. (note j is the database index).