Can they be reassigned, put back onto the queue?
What good way is there to ensure nothing pending is there for too long?
You can use xpending command:
XPENDING mystream group55 - + 10 consumer-123
What happens with the pending messages when the consumer is deleted?
According to XGROUP DELCONSUMER, all pending messages will be deleted.
You can easily verify it by yourself:
// Create stream and fill it with some messages
XADD mystream * msg 1
XADD mystream * msg 2
XADD mystream * msg 3
// Create group for stream and set read position to start
XGROUP CREATE mystream mygroup 0
// Read 2 messages
XREADGROUP GROUP mygroup myconsumer COUNT 2 STREAMS mystream >
// Verify that messages we read are pending
XPENDING mystream mygroup - + 10 myconsumer
// Delete Consumer
XGROUP DELCONSUMER mystream mygroup myconsumer
// Verify that pending messages are gone
XPENDING mystream mygroup - + 10 myconsumer
// Verify that new consumer receives third message only
XREADGROUP GROUP mygroup mynewconsumer STREAMS mystream >
AFAIK there is no explicit way to put pending messages back to stream, but you can reassign them with XCLAIM and XAUTOCLAIM to another consumer and then safely delete consumer.
Related
When using StreamMessageListenerContainer a subscription for a consumer group can be created by calling:
receive(consumer, readOffset, streamListener)
Is there a way to configure the container/subscription so that it will always attempt to re-process any PENDING messages before moving on to polling for new messages?
The goal would be to keep retrying any message that wasn't acknowledged until it succeeds, to ensure that the stream of events is always processed in exactly the order it was produced.
My understanding is if we specify the readOffset as '>' then on every poll it will use '>' and it will never see any messages from the PENDING list.
If we provide a specific message id, then it can see messages from the PENDING list, but the way the subscription updates the lastMessageId is like this:
pollState.updateReadOffset(raw.getId().getValue());
V record = convertRecord(raw);
listener.onMessage(record);
So even if the listener throws an exception, or just doesn't acknowledge the message id, the lastMessageId in pollState is still updated to this message id and won't be seen again on the next poll.
I created a new Redis steam using the following command.
XGROUP CREATE A mygroup $ MKSTREAM
I added the below mentioned data
xadd A * X 1
xadd A * X 2
xadd A * X 3
xadd A * X 4
I am reading the data using the following command.
XREADGROUP GROUP mygroup Alice COUNT 1 STREAMS A 0
Its returning an empty array
1) 1) "A"
2) (empty array)
I am using Redis version 6.2.1. Kindly help me to debug the error.
When you use XREADGROUP command to read message, you should specify > as ID, instead of 0.
Reference from the doc:
The special > ID, which means that the consumer want to receive only messages that were never delivered to any other consumer. It just means, give me new messages.
Any other ID, that is, 0 or any other valid ID or incomplete ID (just the millisecond time part), will have the effect of returning entries that are pending for the consumer sending the command with IDs greater than the one provided. So basically if the ID is not >, then the command will just let the client access its pending entries: messages delivered to it, but not yet acknowledged. Note that in this case, both BLOCK and NOACK are ignored.
If ID is not >, you can only read pending messages, however, in your case, there's no pending message, since you have not consume anything.
I'm wondering if there is a command or parameter with which I can get the message for the RecordId specified e.g.
XREADGROUP GROUP mygroup myconsumer COUNT 1 STREAMS mystream 12345-0
I want the message with the ID 12345-0, but it seems I get the first message after 12345-0.
I cannot use XRANGE since it doesn't update the deliveryCount and lastDeliveryTime and it doesn't seem to understand the concept of consumer groups.
I'm also aware of
XREADGROUP GROUP mygroup myconsumer STREAMS mystream 0
which gives me all pending messages, but this would update the deliveryCount for all messages and I don't want that.
Redis itself doesn't provide the function you ask for. So instead you might have to
use something like
XREADGROUP GROUP mygroup myconsumer COUNT 1 STREAMS mystream 12344-99999
instead of "12345-0"
The entry id returned by Redis Stream is in the format of millisecondsTime-sequenceNumber.
Since it's unlikely you insert 99999 items in one milisecond, you can be sure that you get the correct item.
I just realized that the XACK do not auto delete message when only one consumer group exist.
I thought that when all consumer groups ack the same message, the message will be deleted by Redis-server, but seemed that this is not the case.
So, the Redis stream memory increases infinitely because of no messages will be deleted.
Maybe the only way to preventing this is manually XDEL message? But how can I know all consumer groups have acked the message?
Need some help, thanks!
Redis streams are primarily an append-only data structure. It's possible to remove an entry using the XDEL command, however that doesn't necessarily free up the memory used by the entry:
> XDEL mystream 1538561700640-0
(integer) 1
You could also cap the stream with an arbitrary threshold using the MAXLEN option to XADD or use the XTRIM command explicitly:
> XADD mystream MAXLEN 1000 * value 1
1526654998691-0
...
> XLEN mystream
(integer) 1000
But how can I know all consumer groups have acked the message?
You can inspect the list of pending messages for each consumer group using the XPENDING command:
> XPENDING mystream mygroup
1) (integer) 1
2) 1526984818136-0
3) 1526984818136-0
4) 1) 1) "consumer-1"
2) "1"
Up until now, my RabbitMQ consumer clients have used a prefetch value of 1. I'm looking to increase the value in order to gain performance. If I set the value to 2, will the RabbitMQ server send each consumer 2 messages at once such that I will need to parse the two messages and store the second one in a List until the first is processed and acknowledged? Or will the API handle this behind the scenes?
I'm using the Java AMQP client library:
ConnectionFactory factory = new ConnectionFactory();
...
Connection connection = factory.newConnection();
Channel channel = connection.createChannel();
channel.basicQos(2);
QueueingConsumer consumer = new QueueingConsumer(channel);
channel.basicConsume(CONSUME_QUEUE_NAME, false, consumer);
while (!Thread.currentThread().isInterrupted()) {
try {
QueueingConsumer.Delivery delivery = consumer.nextDelivery();
String m = new String(delivery.getBody(), "UTF-8");
// Will m contain two messages? Will I have to each message and keep track of them within a List?
...
}
The api handles this behind the scenes, so there are no worries there for you.
Regarding which message gets where, RMQ will just deliver by using round robin, that is if you have the queue: 1 2 3 4 5 6 and consumer1 and consumer2.
consumer1 will have 1 3 5
consumer2 will have 2 4 6
Should the connection die to any of your consumers the prefetched messages will be redelivered to the active consumers using the same delivery method.
This should be interesting reading and a good starting point to figure more exactly what happens:
Tutorial no.2 which I'm sure you've read
Reliability
The api internally queue messages in a blocking queue.
Setting the prefetch count more than 1 is actually a good idea since your worker need not wait for each and every message to arrive. It can read up to N messages (where N is the prefetch count). It can start working on a message as soon as it has finished the previous one.
Also, you have the option to acknowledge multiple messages at once instead of acknowledging individually.
channel.basicAck(lastDeliveryTag, true);
boolean true indicates to acknowledge all the messages upto and including the supplied lastDeliveryTag