How do I design SpringBoot Pagination Restful API with Kafka topic? - api

Question

I am trying to build a pagination restful API that fetches data from the Kafka topic.
For example, inside my Kafka topics, I have 1 billion messages whose data structure is like the following:
class Record {
String ID;
JsonObject studentInfo;
}
How do I get the paginated query result for a specific student id? For example, I want to get 200 records of the student whose id is 0123 and this student might or might not have 200 records on the Kafka topic.
My intuitive approach was to poll data from the Kafka topic, keep the offset on the topic and keep reading the data on the Kafka topic until I have 200 specific student records or reach the end of the Kafka topic. However, I am not sure if this is the right approach I should take.

Answer 1

The Confluent REST Proxy already does what you want, so I would recommend using that, rather than reinventing the wheel
GET /consumers/(string:group_name)/instances/(string:instance)/records
Fetch data for the topics or partitions specified using one of the subscribe/assign APIs
Where, rather than number of records to poll, you give it a timeout (e.g. consumer.poll(Duration timeout)), or max_bytes (consumer config fetch.max.bytes, I think).
Re-GET that API endpoint to get the next "batch" (i.e. page) of records
https://docs.confluent.io/platform/current/kafka-rest/api.html
for a specific student id?
You wouldn't. That's not how Kafka works. If this is a feature you really need, then you can use Interactive Queries feature from Kafka Streams, which Spring has an InteractiveQueryService class that can help with this.
Or, as mentioned in the comments, dump your topic to a database, indexed by ID, then build an API endpoint that will query and paginate from that.