Can anyone tell me why this program doesn't finish if you run it?
import kotlinx.coroutines.*
import java.util.concurrent.Executors
fun main(args: Array<String>) {
val dispatcher = Executors.newSingleThreadExecutor().asCoroutineDispatcher()
val scope = CoroutineScope(dispatcher)
scope.launch { }
scope.cancel()
}
I was writing a bit of database code that I wanted to dedicate a thread to and noticed this behavior. If I just use Dispatchers.IO instead of making my own then it finishes right away, but I wanted a dedicated single thread in this case.
Looks like #Slaw was right, it has to be manually closed. I guess its a somewhat arbitrary design choice?
import kotlinx.coroutines.*
import java.util.concurrent.Executors
fun main(args: Array<String>) {
val dispatcher = Executors.newSingleThreadExecutor().asCoroutineDispatcher()
val scope = CoroutineScope(dispatcher)
scope.launch { }
scope.cancel()
dispatcher.close() // missing this
}
A bit unfortunate since the obsolete newSingleThreadContext() didn't require this.
Related
I writing a KafkaConsumer in Kotlin using the reactive framework, the problem is the whole project structure is based on Kotlin coroutines and now the kafka consumer follows a Flux publisher pipeline,
I got it to work with runBlocking however I am aware this is not a good idea to have blocking code in our project,
I tried using #KafkaListener (fails when adding suspend modifier),
import com.github.avrokotlin.avro4k.Avro
import kotlinx.coroutines.runBlocking
import org.apache.avro.generic.GenericRecord
import org.apache.kafka.clients.consumer.ConsumerRecord
import org.springframework.boot.CommandLineRunner
import org.springframework.kafka.core.reactive.ReactiveKafkaConsumerTemplate
import org.springframework.stereotype.Component
#Component
class KafkaConsumer(
val slackNotificationService: SlackNotificationService,
val consumerTemplate: ReactiveKafkaConsumerTemplate<String, GenericRecord>
) : CommandLineRunner {
suspend fun sendNotification(record: ConsumerRecord<String, GenericRecord>) {
val tagNotification = Avro.default.fromRecord(TagNotification.serializer(), record.value())
slackNotificationService.notifyUsers(tagNotification)
}
override fun run(vararg args: String?) {
consumerTemplate
.receiveAutoAck()
.subscribe {
runBlocking {
sendNotification(it)
}
}
}
}
I can successfully receive the kafka messages and all the rest of the project is working fine, but I couldn't figure out to create this non blocking bridge here,
Does anyone know if there's a better way to handle this?
Thank you in advance :)
If you want to invoke sendNotification() asynchronously then create a CoroutineScope and launch coroutines with it:
class KafkaConsumer(
...
private val coroutineScope = CoroutineScope(Dispatchers.Default)
...
.subscribe {
coroutineScope.launch {
sendNotification(it)
}
}
If KafkaConsumer may be destroyed/shutdown then it is advised to invoke coroutineScope.cancel() when it happens.
Purpose: I want to code an Endpoint which consume another Endpoint taking advantage of light Coroutines assuming I am coding a light assyncronous EndPoint client.
My background: first time trying to use Kotlin Coroutine. I have studied last days and search around. I found numerous article explaining how use Coroutine in Android and I found few others explaining how use Coroutine in a main function. Unfortunatelly I didn't find articles explaining how code a Controller endpoint with coroutines and it ringed a bell in my mind if I am doing something not recommended.
Current situation: I successfully created few approaches using Coroutines but I am wondering which is the most appropriate for a traditional GET. On top of that, I am wondering how deal with Exception properly.
Main question: which one from approaches bellow is recommended and which Exception treatment should I care of?
Related secondary question: what is the diference between
fun someMethodWithRunBlocking(): String? = runBlocking {
return#runBlocking ...
}
and
suspend fun someMethodWithSuspendModifier(): String?{
return ...
}
All tentatives bellow are working and returning the json response but I don't know if "runBlocking" on endpoint method and returning "return#runBlocking" can cause me some negative drawback.
Controller (endpoint)
package com.tolearn.controller
import com.tolearn.service.DemoService
import io.micronaut.http.MediaType
import io.micronaut.http.annotation.Controller
import io.micronaut.http.annotation.Get
import io.micronaut.http.annotation.Produces
import kotlinx.coroutines.Deferred
import kotlinx.coroutines.*
import kotlinx.coroutines.runBlocking
import java.net.http.HttpResponse
import javax.inject.Inject
#Controller("/tolearn")
class DemoController {
#Inject
lateinit var demoService: DemoService
//APPROACH 1:
//EndPoint method with runBlocking CoroutineScope
//Using Deferred.await
//Using return#runBlocking
#Get("/test1")
#Produces(MediaType.TEXT_PLAIN)
fun getWithRunBlockingAndDeferred(): String? = runBlocking {
val jobDeferred: Deferred<String?> = async{
demoService.fetchUrl()
}
jobDeferred.await()
return#runBlocking jobDeferred.await()
}
//APPROACH 2:
//EndPoint method with runBlocking CoroutineScope
//Using return#runBlocking
#Get("/test2")
#Produces(MediaType.TEXT_PLAIN)
fun getWithReturnRunBlocking(): String? = runBlocking {
return#runBlocking demoService.fetchUrl()
}
//APPROACH 3:
//EndPoint method with suspend modifier calling a suspend modifier function
//No runBlocking neither CoroutineScope at all
#Get("/test3")
#Produces(MediaType.TEXT_PLAIN)
suspend fun getSuspendFunction(): String? {
return demoService.fetchUrlWithoutCoroutineScope()
}
}
Service used to call another Rest Endpoint
package com.tolearn.service
import kotlinx.coroutines.coroutineScope
import java.net.URI
import java.net.http.HttpClient
import java.net.http.HttpRequest
import java.net.http.HttpResponse
import java.time.Duration
import javax.inject.Singleton
#Singleton
class DemoService {
suspend fun fetchUrl(): String? = coroutineScope {
val client: HttpClient = HttpClient.newBuilder()
.version(HttpClient.Version.HTTP_2)
.followRedirects(HttpClient.Redirect.NEVER)
.connectTimeout(Duration.ofSeconds(20))
.build()
val request = HttpRequest.newBuilder()
.uri(URI.create("http://localhost:3000/employees"))
.build()
val response = client.sendAsync(request, HttpResponse.BodyHandlers.ofString());
print(response.get().body())
return#coroutineScope response.get().body()
}
suspend fun fetchUrlWithoutCoroutineScope(): String? {
val client: HttpClient = HttpClient.newBuilder()
.version(HttpClient.Version.HTTP_2)
.followRedirects(HttpClient.Redirect.NEVER)
.connectTimeout(Duration.ofSeconds(20))
.build()
val request = HttpRequest.newBuilder()
.uri(URI.create("http://localhost:3000/employees"))
.build()
val response = client.sendAsync(request, HttpResponse.BodyHandlers.ofString());
return response.get().body()
}
}
In case it matters, here are the build.gradle
plugins {
id("org.jetbrains.kotlin.jvm") version "1.4.10"
id("org.jetbrains.kotlin.kapt") version "1.4.10"
id("org.jetbrains.kotlin.plugin.allopen") version "1.4.10"
id("com.github.johnrengelman.shadow") version "6.1.0"
id("io.micronaut.application") version "1.2.0"
}
version = "0.1"
group = "com.tolearn"
repositories {
mavenCentral()
jcenter()
}
micronaut {
runtime("netty")
testRuntime("junit5")
processing {
incremental(true)
annotations("com.tolearn.*")
}
}
dependencies {
implementation("io.micronaut:micronaut-validation")
implementation("org.jetbrains.kotlin:kotlin-stdlib-jdk8:${kotlinVersion}")
implementation("org.jetbrains.kotlin:kotlin-reflect:${kotlinVersion}")
implementation("io.micronaut.kotlin:micronaut-kotlin-runtime")
implementation("io.micronaut:micronaut-runtime")
implementation("javax.annotation:javax.annotation-api")
implementation("io.micronaut:micronaut-http-client")
implementation("org.jetbrains.kotlinx:kotlinx-coroutines-core:1.4.2")
implementation("org.jetbrains.kotlinx:kotlinx-coroutines-jdk8:1.4.2")
runtimeOnly("ch.qos.logback:logback-classic")
runtimeOnly("com.fasterxml.jackson.module:jackson-module-kotlin")
}
application {
mainClass.set("com.tolearn.ApplicationKt")
}
java {
sourceCompatibility = JavaVersion.toVersion("11")
}
tasks {
compileKotlin {
kotlinOptions {
jvmTarget = "11"
}
}
compileTestKotlin {
kotlinOptions {
jvmTarget = "11"
}
}
}
You generally want to avoid using runBlocking in production code outside of the "main" entry point function and tests. This is stated in the runBlocking documentation.
For coroutines, it's important to understand the difference between blocking and suspending.
Blocking
Blocking code prevents the entire thread from continuing. Remember that coroutines are meant for asynchronous programming, not multi-threading. Therefore, assume that two or more coroutines can run on the same thread. Now, what happens when you block the thread? None of them can run.
This is dangerous. Blocking code can absolutely ruin your asynchronous programming. Sometimes you have to use blocking code, such as when you work with files. Kotlin has special ways to deal with it, such as the IO Dispatcher, which will run the code on its own, isolated thread so that it doesn't disrupt your other coroutines.
Suspending
This is the heart of coroutines. The idea is that when your coroutine is suspended, it tells the coroutine scope that another coroutine can execute in the meantime. The suspending piece is completely abstracted with how the asynchronous mechanisms work. That part is up to the implementation of the scope and dispatcher.
Suspending doesn't happen automatically. Some frameworks, like KTor use coroutines in their API, such that often you will find functions that are suspending.
If you have long-running operations that are not inherently suspending, you can convert them using something like what I mentioned in the "Blocking" section.
So What's Better?
Well, that depends on this line:
demoService.fetchUrl()
Is fetchUrl() suspending or blocking? If it's blocking, then all of your proposals are roughly the same (and not recommended). If it's suspending, then your third option is best.
If it's blocking, then the best way to deal with it is to create a coroutine scope and wrap it in something that makes it suspending, such as withContext, and return that from your suspending function.
However, this is only the case if these functions are being called from within a coroutine. I'm not familiar with Micronaut. If that framework is automatically calling your methods and not using coroutines, then there is no point introducing them in this class at all.
Lets see this code:
import kotlinx.coroutines.*
fun main() = runBlocking {
val deferreds: List<Deferred<Int>> = (1..3).map {
async {
delay(50L * it)
it
}
}
val sum = awaitAll(deferreds[0], deferreds[1], deferreds[2]).sum()
println("$sum")
}
Above we call awaitAll with a varargs. This is how the source code for awaitAll looks:
public suspend fun <T> awaitAll(vararg deferreds: Deferred<T>): List<T> =
if (deferreds.isEmpty()) emptyList() else AwaitAll(deferreds).await()
But apparently we can also call awaitAll on the list of promises:
import kotlinx.coroutines.*
fun main() = runBlocking {
val deferreds: List<Deferred<Int>> = (1..3).map {
async {
delay(50L * it)
it
}
}
val sum2 = deferreds.awaitAll().sum()
println("$sum2")
}
How is this possible if awaitAll is not an extension function?
Looking at the source code for kotlinx.coroutines.awaitAll it looks like the awaitAll method is both a global function and an extension method of the Collection<Deferred<T>> template class.
Either should work just as well and do basically the same thing, it's up to you to use the one which you prefer and which fits your standard the best!
I am trying to run the following code but it's not running because the compiler doesn't know which version of the async method to call. How do I tell it which one to call?
v
ar counter=0
val workerA=asyncIncrement(5000)
val workerB=asyncIncrement(100)
workerA.await()
workerB.await()
print("counter = $counter")
fun asyncIncrement(by:Int)=async{
for(i in 1..by){
counter++
}
}
Just copy and paste the code into a scratch file or wherever and you should see the same compiler error
From Kotlin 1.3 you need to call async on a scope. In this example I have chosen
the GlobalScope. But it doesn't matter which scope you choose, you always have to explicitly import the asyncextension function;
import kotlinx.coroutines.GlobalScope
import kotlinx.coroutines.async
import kotlinx.coroutines.runBlocking
import java.util.concurrent.atomic.AtomicInteger
fun main(args: Array<String>) {
val counter=AtomicInteger(0)
fun asyncIncrement(by:Int)= GlobalScope.async{
for(i in 1..by){
counter.incrementAndGet()
}
}
val workerA=asyncIncrement(5000)
val workerB=asyncIncrement(100)
runBlocking {
workerA.await()
workerB.await()
}
print("counter = $counter")
}
BTW:
I changed the variable counter from int to an AtomicInteger because the two async blocks may run in different threads. And I introduced runBlocking, because await has to run in a suspend function.
I have a collection of objects, which I need to perform some transformation on. Currently I am using:
var myObjects: List<MyObject> = getMyObjects()
myObjects.forEach{ myObj ->
someMethod(myObj)
}
It works fine, but I was hoping to speed it up by running someMethod() in parallel, instead of waiting for each object to finish, before starting on the next one.
Is there any way to do this in Kotlin? Maybe with doAsyncTask or something?
I know when this was asked over a year ago it was not possible, but now that Kotlin has coroutines like doAsyncTask I am curious if any of the coroutines can help
Yes, this can be done using coroutines. The following function applies an operation in parallel on all elements of a collection:
fun <A>Collection<A>.forEachParallel(f: suspend (A) -> Unit): Unit = runBlocking {
map { async(CommonPool) { f(it) } }.forEach { it.await() }
}
While the definition itself is a little cryptic, you can then easily apply it as you would expect:
myObjects.forEachParallel { myObj ->
someMethod(myObj)
}
Parallel map can be implemented in a similar way, see https://stackoverflow.com/a/45794062/1104870.
Java Stream is simple to use in Kotlin:
tasks.stream().parallel().forEach { computeNotSuspend(it) }
If you are using Android however, you cannot use Java 8 if you want an app compatible with an API lower than 24.
You can also use coroutines as you suggested. But it's not really part of the language as of now (August 2017) and you need to install an external library. There is very good guide with examples.
runBlocking<Unit> {
val deferreds = tasks.map { async(CommonPool) { compute(it) } }
deferreds.forEach { it.await() }
}
Note that coroutines are implemented with non-blocking multi-threading, which mean they can be faster than traditional multi-threading. I have code below benchmarking the Stream parallel versus coroutine and in that case the coroutine approach is 7 times faster on my machine. However you have to do some work yourself to make sure your code is "suspending" (non-locking) which can be quite tricky. In my example I'm just calling delay which is a suspend function provided by the library. Non-blocking multi-threading is not always faster than traditional multi-threading. It can be faster if you have many threads doing nothing but waiting on IO, which is kind of what my benchmark is doing.
My benchmarking code:
import kotlinx.coroutines.experimental.CommonPool
import kotlinx.coroutines.experimental.async
import kotlinx.coroutines.experimental.delay
import kotlinx.coroutines.experimental.launch
import kotlinx.coroutines.experimental.runBlocking
import java.util.*
import kotlin.system.measureNanoTime
import kotlin.system.measureTimeMillis
class SomeTask() {
val durationMS = random.nextInt(1000).toLong()
companion object {
val random = Random()
}
}
suspend fun compute(task: SomeTask): Unit {
delay(task.durationMS)
//println("done ${task.durationMS}")
return
}
fun computeNotSuspend(task: SomeTask): Unit {
Thread.sleep(task.durationMS)
//println("done ${task.durationMS}")
return
}
fun main(args: Array<String>) {
val n = 100
val tasks = List(n) { SomeTask() }
val timeCoroutine = measureNanoTime {
runBlocking<Unit> {
val deferreds = tasks.map { async(CommonPool) { compute(it) } }
deferreds.forEach { it.await() }
}
}
println("Coroutine ${timeCoroutine / 1_000_000} ms")
val timePar = measureNanoTime {
tasks.stream().parallel().forEach { computeNotSuspend(it) }
}
println("Stream parallel ${timePar / 1_000_000} ms")
}
Output on my 4 cores computer:
Coroutine: 1037 ms
Stream parallel: 7150 ms
If you uncomment out the println in the two compute functions you will see that in the non-blocking coroutine code the tasks are processed in the right order, but not with Streams.
You can use RxJava to solve this.
List<MyObjects> items = getList()
Observable.from(items).flatMap(object : Func1<MyObjects, Observable<String>>() {
fun call(item: MyObjects): Observable<String> {
return someMethod(item)
}
}).subscribeOn(Schedulers.io()).observeOn(AndroidSchedulers.mainThread()).subscribe(object : Subscriber<String>() {
fun onCompleted() {
}
fun onError(e: Throwable) {
}
fun onNext(s: String) {
// do on output of each string
}
})
By subscribing on Schedulers.io(), some method is scheduled on background thread.
To process items of a collection in parallel you can use Kotlin Coroutines. For example the following extension function processes items in parallel and waits for them to be processed:
suspend fun <T, R> Iterable<T>.processInParallel(
dispatcher: CoroutineDispatcher = Dispatchers.IO,
processBlock: suspend (v: T) -> R,
): List<R> = coroutineScope { // or supervisorScope
map {
async(dispatcher) { processBlock(it) }
}.awaitAll()
}
This is suspend extension function on Iterable<T> type, which does a parallel processing of items and returns some result of processing each item. By default it uses Dispatchers.IO dispatcher to offload blocking tasks to a shared pool of threads. Must be called from a coroutine (including a coroutine with Dispatchers.Main dispatcher) or another suspend function.
Example of calling from a coroutine:
val myObjects: List<MyObject> = getMyObjects()
someCoroutineScope.launch {
val results = myObjects.processInParallel {
someMethod(it)
}
// use processing results
}
where someCoroutineScope is an instance of CoroutineScope.
Or if you want to just launch and forget you can use this function:
fun <T> CoroutineScope.processInParallelAndForget(
iterable: Iterable<T>,
dispatcher: CoroutineDispatcher = Dispatchers.IO,
processBlock: suspend (v: T) -> Unit
) = iterable.forEach {
launch(dispatcher) { processBlock(it) }
}
This is an extension function on CoroutineScope, which doesn't return any result. It also uses Dispatchers.IO dispatcher by default. Can be called using CoroutineScope or from another coroutine.
Calling example:
someoroutineScope.processInParallelAndForget(myObjects) {
someMethod(it)
}
// OR from another coroutine:
someCoroutineScope.launch {
processInParallelAndForget(myObjects) {
someMethod(it)
}
}
where someCoroutineScope is an instance of CoroutineScope.