How to avoid using sendBlocking in kotlin channel producer? - kotlin

Question

I'm using a kotlin channel in order to migrate a database: I have 1 producer and multiple processors which write to database. The producer just sends the batches of documents to channel:
fun CoroutineScope.produceDocumentBatches(mongoCollection: MongoCollection<Document>) = produce<List<Document>> {
var batch = arrayListOf<Document>()
for ((counter, document) in mongoCollection.find().withIndex()) {
if ((counter + 1) % 100 == 0) {
sendBlocking(batch)
batch = arrayListOf()
}
batch.add(document)
}
if (batch.isNotEmpty()) sendBlocking(batch) }
}
This is how my processors look like:
private fun CoroutineScope.processDocumentsAsync(
documentDbCollection: MongoCollection<Document>,
channel: ReceiveChannel<List<Document>>,
numberOfProcessedDocuments: AtomicInteger
) = launch(Dispatchers.IO) {
// do processing
}
And this is how I use them in the script:
fun run() = runBlocking {
val producer = produceDocumentBatches(mongoCollection)
(1..64).map { processDocumentsAsync(documentDbCollection, producer, count) }
}
So is it fine to use sendBlocking with regards to performance? If I use just send I create many suspending functions inside one coroutine because writes to database are much slower than reads and I get java.lang.OutOfMemoryError: Java heap space. Do I understand correctly that producer blocks Main thread but it's fine for performance because all consumers are executed on IO threads?

Answer 1

Maybe my understanding is not correct, but I think you are not creating many suspending functions inside a coroutine, not sure if it even makes sense to say something like that. As you haven't defined a capacity when using produce function, default value is zero and it is using a RendezvousChannel. It is going to suspend till another coroutine invokes receive. I don't think you need sendBlocking.
My guess is that you are using two many consumers and has two many Document instances in memory, it could be the reason for the OutOfMemoryError. What is the size of each Document? What are your jvm heap configurations?
My suggestions:
Use send instead of sendBlocking
Decrease the number of consumers to a much smaller number and see if the OutOfMemory persists.
Use clear() to erase the ArrayList instead of creating a new instance.
If it works fine after those suggestions, try to increase the number of consumers and check if everything still works fine.