Controller returning reactive paged result from total item count as Mono, items as Flux - kotlin

Question

I've got 2 endpoints returning the same data in two different JSON-formats.
The first endpoint returns a JSON-array, and starts the response right away.
#Get("/snapshots-all")
fun allSnapshots(
#Format("yyyy-MM-dd") cutoffDate: LocalDate
): Flux<PersonVm> = snapshotDao.getAllSnapshots(cutoffDate)
The next endpoint that returns a paged result, is more sluggish. It starts the response when both streams are completed. It also requires a whole lot more of memory than the previous endpoint, even though the previous endpoint returns all records from BigQuery.
#Get("/snapshots")
fun snapshots(
#Format("yyyy-MM-dd") cutoffDate: LocalDate,
pageable: Pageable
): Mono<Page<PersonVm>> = Mono.zip(
snapshotDao.getSnapshotCount(cutoffDate),
snapshotDao.getSnapshots(
cutoffDate,
pageable.size,
pageable.offset
).collectList()
).map {
CustomPage(
items = it.t2,
totalNumberOfItems = it.t1,
pageable = pageable
)
}
(Question update) BigQuery is at the bottom of this endpoint. The strength of BigQuery compared to e.g. Postgres, is querying huge tables. The weakness is relatively high latency for simple queries. Hence I'm running the queries in parallel in order to keep latency for the endpoint at a minimum. Running the queries in sequence, will add at least a second to the total processing time.
Question is: Is there a possible rewrite of the chain that will speed up the /snapshots endpoint?
Solution requirements (question update after suggested approaches)
The consumer of this endpoint is external to the project, and every endpoint in this project is documented at a detailed level. Hence, pagination may only occur one time in the returned JSON. Else feel free to suggest new types for returning pagination along with the PersonVm collection.
If it turns out that another solution is impossible, that's an answer as well.
SnapshotDao#getSnapshotCount returns a Mono<Long>
SnapshotDao#getSnapshots returns a Flux<PersonVm>
PersonVm is defined like this:
#Introspected
data class PersonVm(
val volatilePersonId : UUID,
val cases: List<PublicCaseSnapshot>
)
CustomPage is defined like this:
#Introspected
data class CustomPage<T>(
private val items: List<T> = listOf(),
private val totalNumberOfItems: Long,
private val pageable: Pageable
) : Page<T> {
override fun getContent(): MutableList<T> = items.toMutableList()
override fun getTotalSize(): Long = totalNumberOfItems
override fun getPageable(): Pageable = pageable
}
PublicCaseSnapshot is a complex structure, and left out for brevity. It should not be required for solving this issue.
Code used during test of suggested approach from #Denis
In this approach, chain starts with SnapshotDao#getSnapshotCount, and is mapped into an HttpResponse instance with response body containing the Flux<PersonVm>, and total item count in header.
Queries will now run in sequence, and numerous comparison tests between below code and existing code, showed that the original code performs better (by approx. 1 second). Different page sizes were used during the tests, and BigQuery was warmed up by running same query multiple times. Best results were recorded.
Please note that in cases where time spent on total item count query is negligible (or total item count is cacheable) and pagination is not required to be part of the JSON, this should be considered as a viable approach.
#Get("/snapshots-with-total-count-in-header")
fun snapshotsWithTotalCountInHeader(
#Format("yyyy-MM-dd") cutoffDate: LocalDate,
pageable: Pageable
): Mono<HttpResponse<Flux<PersonVm>>> = snapshotDao.getSnapshotCount(cutoffDate)
.map { totalItemCount ->
HttpResponse.ok(
snapshotDao.getSnapshots(
cutoffDate,
pageable.size,
pageable.offset
)
).apply {
headers.add("total-item-count", totalItemCount.toString())
}
}

Answer 1

You need to rewrite the method to return a publisher of the items. I can see a few options here:
Return the pagination information in the header. Your method will have return type Mono<HttpResponse<Flux<PersonVm>>>.
Return the pagination information on every item: Flux<Tuple<PageInfo, PersonVm>>