Spring cloud Kafka limit message consumption per time unit - kotlin

Question

I have a consumer application with spring cloud stream. This application is consuming messages from a queue (Kafka), and per each message the app makes 4 different HTTP calls to 4 different servers, one of then very slow (10 seconds to answer). When the queue is full of messages, like 6000, the app crashes because several reasons (1 - netty runs out of direct memory, 2 - we are using reactor and the thread pool gets empty).
Is there some way to limit the consumption velocity on consumer side either through spring-cloud-stream or kafka? Something like maximum messages per seconds would be nice.
Here you can see the configuration for kafka (application.yml)
spring:
kafka:
bootstrap-servers: my-cloud-kafka-instance
admin:
ssl:
protocol: SSL
properties:
security.protocol: SSL
cloud:
stream:
bindings:
input:
group: my-group
destination: my-destination
content-type: application/json
And here is my consumer (in kotlin):
#Controller
#EnableBinding(Processor::class)
class MyConsumer(
myDependendies
) {
#StreamListener(Processor.INPUT)
fun myMethod(
#Headers headers: Map<String, String>,
#Payload myMessage: Message
) {
myBussinessLogic
}
}

Answer 1

Pollable Message Source allows consumers to control consumption rates. For example, to illustrate briefly, we first define an interface:
public interface PolledProcessor {
#Input
PollableMessageSource destIn();
#Output
MessageChannel destOut();
}
Use examples:
#Autowired
private PolledProcessor polledProcessor;
#Scheduled(fixedDelay = 5_000)
public void poll() {
polledProcessor.destIn().poll(message -> {
byte[] bytes = (byte[]) message.getPayload();
String payload = new String(bytes);
logger.info("Received: " + payload);
polledProcessor.destOut().send(MessageBuilder.withPayload(payload.toUpperCase())
.copyHeaders(message.getHeaders())
.build());
});
}
Reference resources:
https://spring.io/blog/2018/02/27/spring-cloud-stream-2-0-polled-consumers