RabbitMQ: identify messages from same prefetch - rabbitmq

Question

My producer looks like this:
$connection = new AMQPStreamConnection(
$settings['amqp']['host'],
$settings['amqp']['port'],
$settings['amqp']['username'],
$settings['amqp']['password']
);
$channel = $connection->channel();
$channel->queue_declare($settings['amqp']['queue'], false, true, false, false);
$msg = array();
$msg['time'] = time();
$msg = new AMQPMessage(json_encode($msg), array('delivery_mode' => AMQPMessage::DELIVERY_MODE_PERSISTENT));
$channel->basic_publish($msg, '', $settings['amqp']['queue']);
$channel->close();
$connection->close();
And my consumer looks like this:
$connection = new AMQPStreamConnection(
$settings['amqp']['host'],
$settings['amqp']['port'],
$settings['amqp']['username'],
$settings['amqp']['password']
);
$channel = $connection->channel();
$channel->queue_declare($settings['amqp']['queue'], false, true, false, false);
$callback = function($message) {
//echo $message->body . "\n";
//echo $message->delivery_info['delivery_tag'] . "\n";
var_dump($message) . "\n";
$message->delivery_info['channel']->basic_ack($message->delivery_info['delivery_tag']);
};
$channel->basic_qos(null, 30, null);
$channel->basic_consume($settings['amqp']['queue'], '', false, false, false, false, $callback);
try {
while(count($channel->callbacks)) {
$channel->wait();
}
} catch (Exception $e){
die();
}
Since I'm setted basic_qos to null, 30, null my consumers retrieve blocks of 30 messages from the cluster.
My goal is identify these 30 as a group. And the next 30 as a diferent group.
Something like that:
ATTENTION: Example PHP + BASH !!!
$callback = function($message) {
mkdir -p /tmp/$message->delivery_info['prefetch_group']
echo $message->body > /tmp/$message->delivery_info['prefetch_group']/$message->delivery_info['delivery_tag']
};
At the end. I will have one dir for each block with the content of the messages.
Of course $message->delivery_info['prefetch_group'] doesn't exist. And I didn't found anything usefull into $message
It would be ideal send the basis_ack for each block (30 messages) only if all the process is done.
Any idea to deal with this?

Answer 1

There is nothing stopping you from processing 30 messages before receiving another 30 messages.
Wait to send the acknowledgement until after you have processed the 30th message
Be sure to set the multiple flag to "true"; this will acknowledge all messages received up to the 30th message
Obviously you'll need to keep the messages in the order they were received (they come one at a time from the broker). This can be done by sorting on DeliveryTag.
As a caveat, if for example the message processor fails/crashes with the 30th message, and you don't acknowledge the other 29, then you're going to end up getting those re-delivered to you, so this may be something to consider in your design.
Also note from the spec:
The server may send less data in advance than allowed by the client's specified prefetch windows but it MUST NOT send more.

Answer 2

After some more research I have done this.
Using basic_get instead basic_consume like that:
CODE NOT PROBED !!!. Writed like an example.
16 $connection = new AMQPStreamConnection(
17 $settings['amqp']['host'],
18 $settings['amqp']['port'],
19 $settings['amqp']['username'],
20 $settings['amqp']['password']
21 );
22
23 $channel = $connection->channel();
24 $channel->queue_declare($settings['amqp']['queue'], false, true, false, false);
25
26 $messages = Array();
27 for ($i=0; $i < 30; $i++) {
28 $messages[] = $channel->basic_get($settings['amqp']['queue']);
29 }
30
31 $now = time();
32 foreach($messages as $message) {
33 // DO WHATEVER YOU WANT WITH EACH MSG
34 // system('mkdir -p /tmp/' . $now);
// system('echo ' . $message->body . '> /tmp/' . $now . '/' . $message->delivery_info['delivery_tag']);
35 echo $message->body . "\n";
36 }
37
38 // system('tar /tmp/' . $now . '.gz cvfz /tmp/' . $now);
39
40 foreach($messages as $message) {
41 $channel->basic_ack($message->delivery_info['delivery_tag']);
42 }
43
44 $channel->close();
45 $connection->close();
This in only a POC. Would be necessary:
Be sure there is a message before push to $messages (Line 28)
Control many possible errors.
Daemonize the consumer in some way.
I need read more about basic_get vs basic_consume but this solutions seems good to me.