Redis Streams inconsistent behavior of blocking XREAD after XDEL - redis

Calling XREAD after XDEL will not block on the stream, but return immediately. Expected behavior is for XREAD to block again.
127.0.0.1:6379> XADD my-stream * field1 string1
"1554300150697-0"
127.0.0.1:6379> XREAD BLOCK 5000 STREAMS my-stream 1554300150697-0
(nil)
(5.07s)
127.0.0.1:6379> XADD my-stream * field2 string2
"1554300285984-0"
127.0.0.1:6379> XREAD BLOCK 5000 STREAMS my-stream 1554300150697-0
1) 1) "my-stream"
2) 1) 1) "1554300285984-0"
2) 1) "field2"
2) "string2"
127.0.0.1:6379> XDEL my-stream 1554300285984-0
(integer) 1
127.0.0.1:6379> XLEN my-stream
(integer) 1
127.0.0.1:6379> XREAD BLOCK 5000 STREAMS my-stream 1554300150697-0
1) 1) "my-stream"
2) (empty list or set)
127.0.0.1:6379>
As you can see above, the first time XREAD is called it blocks for 5s - expected.
The second call to XREAD returns immediately, giving the new entry - expected.
The third call to XREAD return immediately with (empty list or set) - not expected!
Expected: The command should block for 5s.
I'm not sure if this is a bug or if there's something that I'm missing out. Please advise.
Thank you

It looks like you're running into this known bug.
See the second comment in particular, which points out that the partial fix supplied does not fix the issue you're running into:
It's not an entire fix for the blocking issue, since it only fixes the blocking behaviour for empty streams.
If the stream still contains some entries, but none with a larger ID than requested by the last-received-id parameter, then the request is still answered synchronously with an empty result list.

Looking through 5.0.4's source code I've found a way to (re)set ->last_id member through an undocumented command: XSETID
Although in the source code https://github.com/antirez/redis/blob/f72f4ea311d31f7ce209218a96afb97490971d39/src/t_stream.c#L1837 it says the syntax is XSETID <stream> <groupname> <id>, it's in fact XSETID <stream> <id>(there's an open issue on this one: https://github.com/antirez/redis/issues/5519 but I hope they'd add a new command for groups, like XGROUPSETID, and let this one as it is), which was exactly what I was looking for, so doing:
XSETID my-stream 1554300150697-0
would make:
127.0.0.1:6379> XREAD BLOCK 5000 STREAMS my-stream 1554300150697-0
(nil)
(5.08s)
127.0.0.1:6379>
to work as expected - it will block.
For anyone using this solution(which is more like a workaround in my opinion): Please use it with caution because in a high throughput machine/system/environment Redis could generate/add a new my-stream entry with the same ID as the deleted one 1554300285984-0 leading to possible duplicate data on client's side.

Related

Redis inconsistency between bigkeys and llen

When I scan entire redis instance using redis-cli --bigkeys following shortened result is returned
-------- summary -------
Sampled 241145 keys in the keyspace!
Total key length in bytes is 13013217 (avg len 53.96)
Biggest string found 'celery-task-meta-52b14b66-b924-4c40-b7dc-7d5b9b633470' has 6510 bytes
**Biggest list found 'celery9' has 156519 items**
Biggest set found '_kombu.binding.celeryev' has 52 members
Biggest hash found 'unacked' has 544 fields
Biggest zset found 'unacked_index' has 550 members
As you can see my biggest list is celery9 with length 156519. I am using only one keyspace
127.0.0.1:6379> info keyspace
# Keyspace
db0:keys=256672,expires=256659,avg_ttl=1701804
But when I connect to redis instance using redis-cli or even with redis connector from python and run following commands
127.0.0.1:6379> get celery9
(nil)
127.0.0.1:6379> llen celery9
(integer) 0
127.0.0.1:6379>
nil or zero is returned as if there was no key celery9.
So the question is, how to get correct length of this key? All others keys are working perfectly

Not sure how to run the CAS (compare and swap) code snippet from Redis documentation page

I am trying to run the code from the redis transactions page. Specifically, this part:
WATCH zset
element = ZRANGE zset 0 0
MULTI
ZREM zset element
EXEC
If I try to do it from the cli, line by line, I get this:
localhost:6380> zadd set 1 a
(integer) 1
localhost:6380> WATCH zset
localhost:6380> element = ZRANGE zset 0 0
(error) ERR unknown command 'element'
OK
which probably means I'm doing something wrong? I remember working with lua about 9 years ago, so this doesn't really look like lua either to me.
How does someone run that snippet? Is it only some kind of pseudocode?
As #Dinei said, the example given is pseudocode.
Let's look at it (I added line numbers for us to refer to):
1 WATCH zset
2 element = ZRANGE zset 0 0
3 MULTI
4 ZREM zset element
5 EXEC
The point of the exercise is to solve the race condition that would occur if we only read the key (with ZRANGE, in line 2), and then modify the key (with ZREM in line 4). I assume you understand the problem if we didn't use the "CAS" semantics, so no need to get into it.
As pointed out, redis-cli just gives you the ability to run redis commands and see their replies, but not save values in variables, etc.
So the idea of the example is that in line 2, we are "saving" the result of the "read" operation, into a pseudo-variable element.
Then, in line 4, we are using that value in our "set" operation, and of course lines 1, 3 and 5 are just the "CAS" commands to ensure there is no race condition.
Presumably the actual usage of such commands would be done from a redis client in a programming language that would allow us to save the return value of the ZRANGE and then use it later in the ZREM command.
But if you wanted to run it in redis-cli, you'd see this, where we pretend that our client-side code would have read and saved "a" that was returned from zrange and then passed that value to the zrem command:
127.0.0.1:6379> zadd zset 1 a
(integer) 1
127.0.0.1:6379> watch zset
OK
127.0.0.1:6379> zrange zset 0 0
1) "a"
127.0.0.1:6379> multi
OK
127.0.0.1:6379> zrem zset a
QUEUED
127.0.0.1:6379> exec
1) (integer) 1
127.0.0.1:6379>
Yes, it is some kind of pseudocode.
redis-cli only accepts Redis commands, it is not a full-fledged editor nor supports direct Lua scripting (neither variables like the element variable in the pseudocode).
I remember working with lua about 9 years ago, so this doesn't really look like lua either to me.
This is not Lua, it is pseudocode. Actually, the Redis Transactions page you linked to does not refer to Lua at all (and that's why #Piglet's comment in your post makes sense).
However, it is possible to execute Lua scripts by using Redis' EVAL command.

Rpush not adding particular key

I am facing a quite strange issue in our deployment
After certain time in operation
I could not add a particular list with a particular keyname or listname to Redis using RPUSH.
Example
RPUSH client-send-process-servername TEST
I am unable to add that key to Redis database.
Anyways the output after executing that command i get
(Integer) 1
But i could not see the list
redis-cli keys *client-send*
Return empty list
However when this problem appears
I am able successfully execute the following command.
RPUSH client-send-process-ser TEST
And
redis-cli keys *client-send*
NOTE: one ASTRICK before n aftet client-send string
Which is not displayed here
It is listing the Queue
"client-send-process-ser"
However strangely i could not add a List with specific key say
client-send-process-servername
Any ideas to debug further.. where to look and what to look.
Redis Server Version is 2.8
I tired enabling debug logs in redis and tried to use redis-monitor command.
However nothing relevant could be found explaining this issue. I am eager to find a solution. Please if some one could help me to pursue further would be a great help.
It seems that this problem is not reproducible. However I've tested it in my local pc and found that it is working fine.
Example:
➜ ~ redis-cli
127.0.0.1:6379> RPUSH client-send-process-servername TEST
(integer) 1
127.0.0.1:6379> keys *client-send*
1) "client-send-process-servername"
127.0.0.1:6379> keys client-send*
1) "client-send-process-servername"
127.0.0.1:6379> RPUSH client-send-process-ser TEST
(integer) 1
127.0.0.1:6379> keys client-send*
1) "client-send-process-ser"
2) "client-send-process-servername"
127.0.0.1:6379>
If there is any option, you can try updating your redis server. current updated versions is 5.0.7 and yours is 2.8.

How can I get the memory footprint of a specific key in redis?

I'm new to Redis. How can I get the memory footprint of a specific key in redis?
db0
1) "unacked_mutex"
2) "_kombu.binding.celery"
3) "_kombu.binding.celery.pidbox"
4) "_kombu.binding.celeryev"
I just want to get the memory footprint of one specific key like "_kombu.binding.celery" , or one specific db like db0 , how can I get it?
redis_version: 2.8.17
Any commentary is very welcome. great thanks.
You are running a very old version of redis. The MEMORY command is not available in that version, so there is no precise way of getting at this information. However, you can approximate this information using the DUMP command.
Simply call DUMP _kombu.binding.celery and save the results to a file. The result is some characters and escape sequences. When you load this file into an environment like node, you can look at the length of the string and multiply by 2 to get the number of bytes. This is not precise, but it will give you a generally close approximation.
Here's what you could do:
in Redis:
$ redis-cli
127.0.0.1:6379> hset c 123 456
(integer) 0
127.0.0.1:6379> dump c
"\r\x12\x12\x00\x00\x00\r\x00\x00\x00\x02\x00\x00\xfe{\x03\xc0\xc8\x01\xff\t\x00\x10\xd4L \x908\x8b2"
in Node:
$ node
> a="\r\x12\x12\x00\x00\x00\r\x00\x00\x00\x02\x00\x00\xfe{\x03\xc0\xc8\x01\xff\t\x00\x10\xd4L \x908\x8b2"
'\r\u0012\u0012\u0000\u0000\u0000\r\u0000\u0000\u0000\u0002\u0000\u0000þ{\u0003ÀÈ\u0001ÿ\t\u0000\u0010ÔL 82'
> a.length
30
This is close to half of the actual amount that redis provides with MEMORY USAGE:
127.0.0.1:6379> MEMORY USAGE c
(integer) 63
MEMORY USAGE _kombu.binding.celery would give you the number of bytes that a key and value require to be stored in RAM.
Here is the doc for the command.

REDIS - INTERsection with multiple sets

Im kind of new to redis, so this question might be stupid, but Im trying to wrap around SINTER command here.
The problem is I have multiple SET, and Im trying to create an INTER on them.
When I do it individually its getting me the results, but when I pass multiple args its returning empty. I read documentation on INTER, it tells if any supplied arg is an empty SET it would return nothing, but all my SET is non empty could any one help me please!
# below statement should return {'758', '762', '752'}
127.0.0.1:6379> SINTER Asset:all Asset:id:2275 Asset:id:2280 Asset:id:2269
(empty list or set)
127.0.0.1:6379> SINTER Asset:all Asset:id:2275
"758"
127.0.0.1:6379> SINTER Asset:all Asset:id:2280
"762"
127.0.0.1:6379> SINTER Asset:all Asset:id:2269
"752"
But all other commands like SUNION, SDIFF are working fine.
got the required behaviour by performing SUNION on sets and then SINTER on newly created set
127.0.0.1:6379> SUNION ~unionset Asset:id:2275 Asset:id:2280 Asset:id:2269
127.0.0.1:6379> SINTER Asset:all ~unionset
"758"
"762"
"752"