How to Xtrim (trim redis stream) in the safe way - redis

Question

Redis Xtrim is to keep a number of records and remove the older ones.
suppose I am having 100 records, I want to keep the newest 10 ones. (suppose java/scala API)
redis = new Jedis(...)
redis.xlen("mystream") // returns 100
// new records arrive stream at here
redis.xtrim("mystream", 10)
however, new records arrive between the execution of xlen and xtrim. So for example now there are 105 records, it will keep 95 - 105, however I want to keep start from 90. So it also trim 90 - 95, which is bad.
Any ideas?

Answer 1

Option 1:
If you always want to keep only the 10 newest ones, use:
XADD mystream MAXLEN 10 * field value ...
By moving trimming to the addition, you keep it always at 10. You can forget about separate logic to trim.
In Jedis,
xadd(String key, StreamEntryID id, Map<String,String> hash, long maxLen, boolean approximateLength)
Option 2:
Optimistic, use a transaction.
WATCH mystream
XLEN mystream
MULTI
XTRIM mystream MAXLEN 10
EXEC
Here, if the stream has been modified, the transaction will fail, EXEC returns null, you then retry.
Jedis transaction example
Option 3:
Use a Lua Script to adjust your XTRIM target number. With this, you can execute the XLEN, do your logic to decide on the trimming you want, and then do XTRIM, all Redis-server-side, atomically, so no chance for your stream changing in-between.
Here some eval examples with Jedis.
Update:
A Lua Script like the following should do the trick: atomically evaluate the current number of myStreams entries and then call XTRIM accordingly.
EVAL "local streamLen = redis.call('XLEN', KEYS[1]) \n if streamLen >= tonumber(ARGV[1]) then \n return redis.call('XTRIM', KEYS[1], 'MAXLEN', streamLen - tonumber(ARGV[1])) \n else return redis.error_reply(KEYS[1]..' has less than '..ARGV[1]..' items') end" 1 myStream 90
Let's take a look at the Lua script:
local streamLen = redis.call('XLEN', KEYS[1])
if streamLen >= tonumber(ARGV[1]) then
return redis.call('XTRIM', KEYS[1], 'MAXLEN', streamLen - tonumber(ARGV[1]))
else
return redis.error_reply(KEYS[1]..' has less than '..ARGV[1]..' items')
end
I posted more detail on a separate question on How to delete or remove a given number of entries from a Redis stream?