Kotlin - Chunk sequence based on size and time - kotlin

Question

I have a never ending stream as a sequence.
What I am aiming for is to take a batch from the sequence both based on time and size.
What I mean is if my sequence has 2250 messages right now I want to send 3 batches ( 1000, 1000, 250).
Also if till the next 5 minute I still have not accumulated a 1000 messages I will send it anyway with whatever has accumulated so far.
sequence
.chunked(1000)
.map { chunk ->
// do something with chunk
}
What I was expecting to have is something like .chunked(1000, 300) which 300 is second for when I want to send every 5 minutes.
Thanks in advance

Answer 1

Kotlin Sequence is a synchronous concept and is not supposed to be used in any kind of time-limited fashion. If you ask the sequence for the next element then it blocks invoker thread until it produces the next element and there is no way to cancel it.
However, kotlinx.coroutines library introduces the concept of Channel which is a rough analogue of a sequence for an asynchronous world, where operation may take some time to complete and they don't block threads while doing so. You can read more in this guide.
It does not provide a ready-to-use chunked operator, but makes it straightforward to write one. You can use the following code:
import kotlinx.coroutines.experimental.channels.*
import kotlinx.coroutines.experimental.selects.*
fun <T> ReceiveChannel<T>.chunked(size: Int, time: Long) =
produce<List<T>>(onCompletion = consumes()) {
while (true) { // this loop goes over each chunk
val chunk = mutableListOf<T>() // current chunk
val ticker = ticker(time) // time-limit for this chunk
try {
whileSelect {
ticker.onReceive {
false // done with chunk when timer ticks, takes priority over received elements
}
this#chunked.onReceive {
chunk += it
chunk.size < size // continue whileSelect if chunk is not full
}
}
} catch (e: ClosedReceiveChannelException) {
return#produce // that is normal exception when the source channel is over -- just stop
} finally {
ticker.cancel() // release ticker (we don't need it anymore as we wait for the first tick only)
if (chunk.isNotEmpty()) send(chunk) // send non-empty chunk on exit from whileSelect
}
}
}
As you can see from this code, it embeds some non-trivial decisions on what to do in corner cases. What should we do if timer expires but current chunk is still empty? This code start new time interval and does not send the previous (empty) chunk. Do we finish current chunk on timeout after last element, measure time from the first element, or measure time from the beginning of chunk? This code does the later.
This code is completely sequential -- its logic is easy to follow in a step-by-step way (there is not concurrency inside the code). One can adjust it to any project-speicfic requirements.