I just realized that the XACK do not auto delete message when only one consumer group exist.
I thought that when all consumer groups ack the same message, the message will be deleted by Redis-server, but seemed that this is not the case.
So, the Redis stream memory increases infinitely because of no messages will be deleted.
Maybe the only way to preventing this is manually XDEL message? But how can I know all consumer groups have acked the message?
Need some help, thanks!
Redis streams are primarily an append-only data structure. It's possible to remove an entry using the XDEL command, however that doesn't necessarily free up the memory used by the entry:
> XDEL mystream 1538561700640-0
(integer) 1
You could also cap the stream with an arbitrary threshold using the MAXLEN option to XADD or use the XTRIM command explicitly:
> XADD mystream MAXLEN 1000 * value 1
1526654998691-0
...
> XLEN mystream
(integer) 1000
But how can I know all consumer groups have acked the message?
You can inspect the list of pending messages for each consumer group using the XPENDING command:
> XPENDING mystream mygroup
1) (integer) 1
2) 1526984818136-0
3) 1526984818136-0
4) 1) 1) "consumer-1"
2) "1"
Related
I created a new Redis steam using the following command.
XGROUP CREATE A mygroup $ MKSTREAM
I added the below mentioned data
xadd A * X 1
xadd A * X 2
xadd A * X 3
xadd A * X 4
I am reading the data using the following command.
XREADGROUP GROUP mygroup Alice COUNT 1 STREAMS A 0
Its returning an empty array
1) 1) "A"
2) (empty array)
I am using Redis version 6.2.1. Kindly help me to debug the error.
When you use XREADGROUP command to read message, you should specify > as ID, instead of 0.
Reference from the doc:
The special > ID, which means that the consumer want to receive only messages that were never delivered to any other consumer. It just means, give me new messages.
Any other ID, that is, 0 or any other valid ID or incomplete ID (just the millisecond time part), will have the effect of returning entries that are pending for the consumer sending the command with IDs greater than the one provided. So basically if the ID is not >, then the command will just let the client access its pending entries: messages delivered to it, but not yet acknowledged. Note that in this case, both BLOCK and NOACK are ignored.
If ID is not >, you can only read pending messages, however, in your case, there's no pending message, since you have not consume anything.
Can they be reassigned, put back onto the queue?
What good way is there to ensure nothing pending is there for too long?
You can use xpending command:
XPENDING mystream group55 - + 10 consumer-123
What happens with the pending messages when the consumer is deleted?
According to XGROUP DELCONSUMER, all pending messages will be deleted.
You can easily verify it by yourself:
// Create stream and fill it with some messages
XADD mystream * msg 1
XADD mystream * msg 2
XADD mystream * msg 3
// Create group for stream and set read position to start
XGROUP CREATE mystream mygroup 0
// Read 2 messages
XREADGROUP GROUP mygroup myconsumer COUNT 2 STREAMS mystream >
// Verify that messages we read are pending
XPENDING mystream mygroup - + 10 myconsumer
// Delete Consumer
XGROUP DELCONSUMER mystream mygroup myconsumer
// Verify that pending messages are gone
XPENDING mystream mygroup - + 10 myconsumer
// Verify that new consumer receives third message only
XREADGROUP GROUP mygroup mynewconsumer STREAMS mystream >
AFAIK there is no explicit way to put pending messages back to stream, but you can reassign them with XCLAIM and XAUTOCLAIM to another consumer and then safely delete consumer.
Using Redis stream we can have pending items which aren't finished by some consumers.
I can find such items using xpending command.
Let we have two pending items:
1) 1) "1-0"
2) "local-dev"
3) (integer) 9599
4) (integer) 1
2) 1) "2-0"
2) "local-dev"
3) (integer) 9599
4) (integer) 1
The problem that by using xpending we can set filters based on id only. I have a couple of service nodes (A, B) which make zombie check: XPENDING mystream test_group - 5 1
Each of them receives "1-0" item and they make xclaim and only one of them (for example A) becomes the owner and starts processing this item. But B runs xpending again to get new items but it receives again "1-0" because it hasn't been processed yet (A is working) and it looks like all my queue is blocked.
Is there any solution how I can avoid it and process pending items concurrently?
You want to see the documentation, in particular Recovering from permanent failures.
The way this is normally used is:
You allow the same consumer to consume its messages from PEL after recovering.
You only XCLAIM from another consumer when a reasonably large time elapsed, that suggests the original consumer is in permanent failure.
You use delivery count to detect poison pills or death letters. If a message has been retried many times, maybe it's better to report it to an admin for analysis.
So normally all you need is to see the oldest age in PEL from other consumers for the Permanent Failure Recovery logic, and you consume one by one.
I'm wondering if there is a command or parameter with which I can get the message for the RecordId specified e.g.
XREADGROUP GROUP mygroup myconsumer COUNT 1 STREAMS mystream 12345-0
I want the message with the ID 12345-0, but it seems I get the first message after 12345-0.
I cannot use XRANGE since it doesn't update the deliveryCount and lastDeliveryTime and it doesn't seem to understand the concept of consumer groups.
I'm also aware of
XREADGROUP GROUP mygroup myconsumer STREAMS mystream 0
which gives me all pending messages, but this would update the deliveryCount for all messages and I don't want that.
Redis itself doesn't provide the function you ask for. So instead you might have to
use something like
XREADGROUP GROUP mygroup myconsumer COUNT 1 STREAMS mystream 12344-99999
instead of "12345-0"
The entry id returned by Redis Stream is in the format of millisecondsTime-sequenceNumber.
Since it's unlikely you insert 99999 items in one milisecond, you can be sure that you get the correct item.
In the redis protocol specification, under the "Multi-bulk replies section":
A Multi bulk reply is used to return an array of other replies. Every element of a Multi Bulk Reply can be of any kind, including a nested Multi Bulk Reply.
However, I can't figure out a way to get Redis to return such output. Can anyone provide an example?
Only certain commands (especially those returning list of values) return multi-bulk replies, you can try by using LRANGE for example but you can check the command reference for more details.
Usually multi-bulk replies are only 1-level deep but some Redis commands can return nested multi-bulk replies (max 2 levels), notably EXEC (depending on the commands executed while inside the transaction context) and both EVAL / EVALSHA (depending on the value returned by the Lua script).
Here is an example using EXEC:
redis 127.0.0.1:6379> MULTI
OK
redis 127.0.0.1:6379> LPUSH metavars foo foobar hoge
QUEUED
redis 127.0.0.1:6379> LRANGE metavars 0 -1
QUEUED
redis 127.0.0.1:6379> EXEC
1) (integer) 4
2) 1) "hoge"
2) "foobar"
3) "foo"
4) "metavars"
The second element of the multi-bulk reply to EXEC is a multi-bulk itsef.
PS: I added a clarification in the comments regarding the actual maximum level of nesting of multi-bulk replies when using Lua scripts. tl;dr: there's basically no limit.