Reactor lazy pagination with expand - kotlin

Question

Based on How to collect paginated API responses using spring boot WebClient?
I created the following crawler class
class GitlabCrawler(private val client: WebClient, private val token: String) {
fun fetchCommits(project: URI): Flux<Commit> {
return fetchCommitsInternal(project).expand { cr: ClientResponse? ->
val nextUrl = getNextUrl(cr)
nextUrl?.let { fetchCommitsInternal(URI.create(it)) }
?: Mono.empty<ClientResponse>()
}.limitRate(1)
.flatMap { cr: ClientResponse? -> cr?.bodyToFlux(Commit::class.java) ?: Flux.empty() }
}
private fun getNextUrl(cr: ClientResponse?):String? {
// TODO replace with proper link parsing
return cr?.headers()?.header(HttpHeaders.LINK)?.firstOrNull()
?.splitToSequence(",")
?.find { it.endsWith("rel=\"next\"") }
?.let { it.substring(it.indexOf('<') + 1, it.lastIndexOf('>')) }
}
private fun fetchCommitsInternal(url: URI): Mono<ClientResponse> {
return client.get()
.uri(url)
.accept(MediaType.APPLICATION_JSON_UTF8)
.header("Private-Token", token)
.exchange()
}
}
data class Commit(
val id: String,
val message: String,
#JsonProperty("parent_ids") val parentIds: List<String>,
#JsonProperty("created_at") val createdAt: String)
I'd like to avoid unnecessary request, but it performs more request than needed to fulfill the request.
gitlabCrawler.fetchCommits(URI.create("https://...")).take(15).collectList().block()
Would only need one request as each page contains 20 entries, but it starts the second page request. It seems to always request one more page than necessary. I tried using limitRate but that doesn't seem to have an effect.
Is there a way to make it lazy, i.e., only request the next page when the current is exhausted?

Answer 1

Are you positive that it actually performs the request? fetchCommitInternal being invoked means that WebFlux "prepared" the request, not necessarily that it was executed (ie. subscribed).
The following simplification of your use case shows the difference:
private static Tuple2<Integer, Flux<Integer>> nextPage(int index, int pageSize) {
System.out.println("prepared a request for page " + index);
return Tuples.of(index, Flux.range((pageSize * (index - 1)) + 1, pageSize));
}
#Test
public void expandLimitedRequest() {
int pageSize = 5;
Flux.just(nextPage(1, pageSize))
.doOnSubscribe(sub -> System.out.println("requested first page"))
.expand(page -> {
int currentPage = page.getT1();
if (currentPage < 3) {
int nextPage = currentPage + 1;
return Flux.just(nextPage(nextPage, pageSize))
.doOnSubscribe(sub -> System.out.println("requested page " + nextPage));
}
return Flux.empty();
})
.doOnNext(System.out::println)
.flatMap(Tuple2::getT2)
.doOnNext(System.out::println)
.take(8)
.blockLast();
}
Prints:
prepared a request for page 1
requested first page
[1,FluxRange]
1
2
3
4
5
prepared a request for page 2
requested page 2
[2,FluxRange]
6
7
8
prepared a request for page 3
As you can see, it prepares the request for page 3 but never execute it (because the take downstream cancels the expand before that).