Parallel execution of jobs using coroutines - kotlin

Question

I want to execute multiple jobs in parallel using coroutines. This is the piece of code that I came up with.
I have 2 queries:
How do I ensure completion callbacks happen in the caller thread?
Code has become more like the callback pattern I used to have with
normal threads. Please suggest changes in design to achieve the
coroutines readability advantage.
class ParallelExecutor {
suspend fun <OUTPUT> execute(
jobs: List<suspend () -> OUTPUT>,
onTimeout: (jobIndex: Int) -> OUTPUT,
onFailure: (jobIndex: Int, exception: Throwable) -> OUTPUT,
onCompletion: suspend (jobIndex: Int, result: OUTPUT) -> Unit,
timeout: Long,
onFullCompletion: suspend () -> Unit = {},
invokeDispatcher: CoroutineDispatcher = Dispatchers.Default
) {
withContext(invokeDispatcher) {
var counter = 0
val listenJobs = mutableListOf<Deferred<OUTPUT>>()
jobs.forEachIndexed { index, job ->
val listenJob = async {
try {
job()
} catch (e: Exception) {
onFailure(index, e)
}
}
listenJobs.add(listenJob)
}
listenJobs.forEachIndexed { index, job ->
launch {
val output = try {
withTimeout(timeout) {
job.await()
}
} catch (e: TimeoutCancellationException) {
onTimeout(index)
}
onCompletion(index, output)
if (++counter == listenJobs.size) {
onFullCompletion()
}
}
}
}
}
}

Answer 1

It seems to me that you can simplify your code quite a bit. You don't need a two-step idiom that first launches all the async jobs and then launches more jobs to await on them. You can just launch the jobs and delegate to the callbacks within the same block. That way the callbacks will naturally be invoked on the caller's dispatcher and only the job itself can be called within the changed context with the invokeDispatcher.
onFullCompletion looks like a piece of code that belongs on the caller side, below the execute call. Since execute doesn't throw any exceptions, you don't need any try-finally to get it.
suspend fun <OUTPUT> execute(
jobs: List<suspend () -> OUTPUT>,
onTimeout: (jobIndex: Int) -> OUTPUT,
onFailure: (jobIndex: Int, exception: Throwable) -> OUTPUT,
onCompletion: suspend (jobIndex: Int, result: OUTPUT) -> Unit,
timeout: Long,
invokeDispatcher: CoroutineDispatcher = Dispatchers.Default
) {
coroutineScope {
jobs.mapIndexed { index, job ->
launch {
val output = try {
withTimeout(timeout) {
withContext(invokeDispatcher) {
job()
}
}
} catch (e: TimeoutCancellationException) {
onTimeout(index)
} catch (e: Exception) {
onFailure(index, e)
}
onCompletion(index, output)
}
}
}
}

Answer 2

Made some improvements that should answer your queries.
class ParallelExecutor {
suspend fun <OUTPUT> execute(
jobs: List<suspend () -> OUTPUT>,
onTimeout: (jobIndex: Int) -> OUTPUT,
onFailure: (jobIndex: Int, exception: Throwable) -> OUTPUT,
onCompletion: suspend (jobIndex: Int, result: OUTPUT) -> Unit,
timeout: Long,
invokeDispatcher: CoroutineDispatcher = Dispatchers.Default
) {
supervisorScope {
val listenJobs = jobs.map { job ->
async(invokeDispatcher) {
withTimeout(timeout) {
job()
}
}
}
listenJobs.forEachIndexed { index, job ->
launch {
val output = try {
job.await()
} catch (e: TimeoutCancellationException) {
onTimeout(index)
} catch (e: Exception) {
onFailure(index, e)
}
onCompletion(index, output)
}
}
}
}
}
Jobs are now cancelled when timeout is reached.
Completion callbacks are now called in the caller's dispatcher.
The race condition when deciding when to call onFullCompletion has been fixed.
Removed some methods that you didn't really need.
If you feel this is more like the callback pattern, then you simply shouldn't use callbacks. Coroutines are designed such that you write this sort of code at use site with minimal boilerplate, so functions like this aren't necessary and look weird (IMHO).