why is it not possible to call Deferred::await in the Sequence::map function like it is possible in List::map?
I made a small example
fun example() = runBlocking {
val list = listOf(1, 2, 3, 4)
list.map { async { doSomething(it) } }
.map { it.await() }
list.asSequence()
.map { async { doSomething(it) } }
.map { it.await() } // Error: Kotlin: Suspension functions can be called only within coroutine body
}
As you can see the last statement does not compile.
It's because list.map is an inline fun, which it can be because it's an eager operation that returns a new list. An inline fun can tolerate a suspend block because it's more like a macro: it gets expanded into the call site. As long as the call site is within a suspend block, this works fine.
sequence.map, on the other hand, is lazy and it just adds another lambda to the pipeline of operations that will be performed eventually, when you pull items from the sequence.
The closest match to a lazy sequence in the suspendable world is Flow, but it has a different model where you get all the data pushed to you in one go, whereas you can pull items from a lazy sequence one by one.
Adding to Marko's answer, which is correct:
Because the sequence is lazy, the await() call might happen after runBlocking is done, essentially. Neither the async call nor the await will happen until you start pulling elements off the list.
And in the case of the code above, nothing pulls the elements out of the sequence, so neither of the mapping operations on the sequence would happen inside the runBlocking block.
Related
I am working on the task of rewriting classes from RxJava to Coroutines and ran into the problem of finding an analogue of the ignoreElements() operator. What can replace it in case of using Coroutines?
For example, in this case. I replace Completable with Flow<Unit>:
private fun observeHistory(): Completable =
sensorDataService.observeRangeData()
.doOnNext {
historyRepo.add(
HistoryData(...)
)
}
.ignoreElements()
And got this
private fun observeHistory(): Flow<Unit> = flow {
sensorDataService.observeRangeData()
.onEach {
historyRepo.add(
HistoryData(...)
)
}
}
```
It depends on what RxJava code you are migrating. The RxJava ignoreElements() operator return Completable and suppresses all of the items emitted, in other words, it emits nothing.
So, if you have something like this in RxJava:
fun saveData(): Completable {
return yourObservable
.doOnComplete {
// save data
}
.ignoreElements()
}
then in Kotlin Coroutines you can use just suspend method without a return value:
suspend fun saveData(data: String) {
// save data
}
but if you want to keep a reactive approach you can convert it to Kotlin Flow but with Flow, you don't have an alternative to emit nothing, feel free to return your source type (you can return Unit type but it looks pointless then maybe you should reconsider why you use Flow here):
fun saveData(data: String): Flow<String> {
return yourFlow
.onCompletion {
// save data
}
}
and then you can just ignore the value during collecting:
saveData(data)
.collect()
Technically one equivalent of observable.ignoreElements() would be flow.filter { false }: it returns a flow with 0 elements (skipping all elements of the source flow), and completes when the source flow completes. That said, the resulting Flow<T> would not be an ideal representation of what you're actually returning.
A better return type to express this would be Deferred<Unit>, which better matches Rx's Completable, for instance with someScope.async(LAZY) { flow.collect() } - but that would be even less idiomatic.
A more idiomatic approach to solve the problem in your example is to use a suspend function instead, because this is what best represents a single async value (but in a sequential way thanks to coroutines):
private suspend fun observeHistory() {
sensorDataService.observeRangeData().collect {
historyRepo.add(HistoryData(...))
}
}
Then if you want to launch this as a background process from the calling code, you can use launch { observeHistory() } in a proper coroutine scope, which you would need anyway if you had returned a flow (because collecting the flow is suspending too).
I am calling a non-inlined library function, providing my own function f that needs to suspend because it receives from a channel. When the library function is called I simply want to wait for it to complete in the current context. The library function will always call f within its body (specifically, the library function does pre, f, then post, where f must be called between pre and post, so it could have been an inlined function). However within f the outer coroutine context no longer applies.
My first thought was to surround the suspending call with runBlocking, but this causes a deadlock, because the (potentially single) thread is now blocked until receive completes, which prevents the producer from progressing.
import kotlinx.coroutines.*
import kotlinx.coroutines.channels.*
// Some non-inlined library function that calls given function f immediately with some library provided value
fun libraryFunction(f: (Int) -> Int) =
f(5)
fun main() {
// Some outer loop that may sometimes only have 1 thread
runBlocking(newSingleThreadContext("thread")) {
// An existing channel
val channel = produce {
send("whatwewant")
}
// Our code
libraryFunction { libraryProvidedValue ->
println(libraryProvidedValue)
runBlocking {
println(channel.receive())
}
// A return value the library needs
7
}
}
}
What is the best way to solve this issue? Can using the inner runBlocking be prevented?
In other words: is there anyway to pinky-promise that we are, in fact, still within the same coroutine in the lambda function?
As an additional illustration, the following works:
import kotlinx.coroutines.*
import kotlinx.coroutines.channels.*
// Some non-inlined library function that calls given function f immediately with some library provided value
fun libraryFunction(f: (Int) -> Int) =
f(5)
fun main() {
// Some outer loop that may sometimes only have 1 thread
runBlocking(newSingleThreadContext("thread")) {
// An existing channel
val channel = produce {
send("whatwewant")
}
// <inline the start of the body of libraryFunction here> (which gives libraryProvidedValue)
// Our code
println(libraryProvidedValue)
println(channel.receive())
// <inline the end of the body of libraryFunction here>
}
}
Update: In this case, the only real issue seems to be that the compiler is not aware of the surrounding coroutine being the same in the lambda function body - but the code is entirely possible to run successfully if it did (as can be seen by inlining the library function). In essence, there is nothing wrong with the flow of this code in particular (because the library function calls the lambda function within its body), but it is a shortcoming in the lack of guarantees the compiler can determine. While runBlocking makes the compiler happy, it has unwanted side effects (notably, the nested blocking part, which makes communication with the outside difficult due to the outer runBlocking blocking up the potentially only thread).
Because of this, I decided to rewrite my entire code surrounding this in a style that uses Deferred with an await at the top level, instead of suspend functions. I would regard this as very un-kotlin-ic, and it comes with potential problems of its own (like resource leaks), but it works for my scenario.
Note that this still does not answer the question posed in any way, but I wanted to note it as an alternative decision to make for future users faced with a similar problem.
Unfortunately there is only one way to achieve what you want - to use runBlocking, but it has consequences like you described. suspend functions should be called from a coroutine (runBlocking in this case) or another suspend function. So to achieve this without using runBlocking libraryFunction function must accept a suspend f function and be suspend itself:
suspend fun libraryFunction(f: suspend (Int) -> Int) = f(5)
I would suggest first to receive the value from the channel, and then call the libraryFunction:
runBlocking(newSingleThreadContext("thread")) {
// An existing channel
val channel = produce {
send("whatwewant")
}
// Our code
val receivedValue = channel.receive()
libraryFunction { libraryProvidedValue ->
println(libraryProvidedValue)
println(receivedValue)
// A return value the library needs
7
}
}
I am having some trouble launching coroutines in parallel.
I have some code that makes a network request in a coroutine, this network request returns a collection of intermediate results, which I would like to fetch the complete result for each item in this collection in parallel (i.e. not sequentially).
Here is the relevant API's:
data class IntermediateResult(val id: Long)
data class FinalResult(val id: Long, val name: String)
suspend fun getIntermediateResults() : Collection<IntermediateResult>
suspend fun getFinalResult(id: Long) : FinalResult
And here is the code that is consuming the API's (it's inside a ViewModel):
fun example() {
viewModelScope.launch(Dispatchers.IO) {
val intermediateResults = getIntermediateResults()
intermediateResults.forEach { intermediateResult ->
viewModelScope.launch(Dispatchers.IO) {
val finalResult = getFinalResult(intermediateResult.id)
// Update cache and post updated cache to the UI via live data
}
}
}
}
It was my understanding that launching a new coroutine inside the forEach loop would cause the final result to be fetched for each item in a parallel fashion (i.e. it won't "wait" on the getFinalResult before continuing to the next iteration of the forEach) but, what I am observing is that it's happening sequentially (i.e. the call to getFinalResult in the forEach loop is causing the code to "wait" until the result comes back before starting the next iteration).
In other words, I don't care about the final results coming back in the order of iteration over the intermediate results.
If it's relevant (I'm fairly sure it's not) the implementation of the 2 suspend functions is using suspendCoroutine to wrap up old callback style code into suspend functions.
We often need some request validation before handling it. With arrow v 0.8 a typical message handler looked like:
fun addToShoppingCart(request: AddToShoppingCartRequest): IO<Either<ShoppingCardError, ItemAddedEvent>> = fx {
request
.pipe (::validateShoppingCard)
.flatMap { validatedRequest ->
queryShoppingCart().bind().map { validatedRequest to it } // fun queryShoppingCart(): IO<Either<DatabaseError, ShoppingCart>>
}
.flatMap { (validatedRequest, shoppingCart) ->
maybeAddToShoppingCart(shoppingCart, validatedRequest) // fun maybeAddToShoppingCart(...): Either<DomainError, ShoppingCart>
}
.flatMap { updatedShoppingCart ->
storeShoppingCart(updatedShoppingCart).bind() // fun storeShoppingCart(ShoppingCart): IO<Either<DatabaseError, Unit>>
.map {
computeItemAddedEvent(updatedShoppingCart)
}
}
.mapLeft(::computeShoppingCartError)
}
This seems to be a convenient and expressive definition of a workflow. I tried to define similar function in arrow v 0.10.5:
fun handleDownloadRequest(strUrl: String): IO<Either<BadUrl, MyObject>> = IO.fx {
parseUrl(strUrl) // fun(String): Either<BadUrl,Url>
.map {
!effect{ downloadObject(it) } // suspended fun downloadObject(Url): MyObject
}
}
Which results in a compiler error "Suspension functions can be called only within coroutine body". The reason is both map and flatMap functions of Either and Option are not inline.
Indeed, the blog post about fx says
"Soon you will find that you cannot call suspend functions inside the
functions declared for Either such as the ones mentioned above, and
other fan favorites like map() and handleErrorWith(). For that you
need a concurrency library!"
So the question is why is it so and what is the idiomatic way of such composition?
The idiomatic way is
fun handleDownloadRequest(strUrl: String): IO<Either<BadUrl, MyObject>> =
parseUrl(strUrl)
.fold({
IO.just(it.left()) // forward the error
}, {
IO { downloadObject(it) }
.attempt() // get an Either<Throwable, MyObject>
.map { it.mapLeft { /* Throwable to BadURL */ } } // fix the left side
})
Personally I wouldn't go to the deep end of IO with that one, and rewrite as a suspend function instead
suspend fun handleDownloadRequest(strUrl: String): Either<BadUrl, MyObject> =
parseUrl(strUrl)
.fold(::Left) { // forward the error
Either.catch({ /* Throwable to BadURL */ }) { downloadObject(it) }
}
What happened is, in 0.8.X the functions for Either used to be inlined. An unintended side-effect of this was that you could call suspend functions anywhere. While this is nice, it can lead to exceptions thrown (or jumping threads or deadlocks 🙈) in the middle of a map or a flatMap, which is terrible for correctness. It's a crutch.
In 0.9 (or was it 10?) we removed that crutch and made it into something explicit in the API: Either.catch. We kept fold as inlined because it's the same as when, so there was no real correctness tradeoff there.
So, the recommended thing is to use suspend everywhere and only reach for IO when trying to do threading, parallelism, cancellation, retries and scheduling, or anything really advanced.
For basic use cases suspend and Either.catch is enough. To call into a suspend function at the edge of your program or where you need to bridge with these advanced behaviors then use IO.
If you want to continue using Either you can define suspend/inline versions of regular functions at your own risk; or wait until IO<E, A> in 0.11 where you can use effectEither and effectMapEither.
I'm trying to poll a paginated API and provide new items to the user as they appear.
fun connect(): Sequence<T> = buildSequence {
while (true) {
// result is a List<T>
val result = dataSource.getFirstPage()
yieldAll(/* the new data in `result` */)
// Block the thread for a little bit
}
}
Here's the sample usage:
for (item in connect()) {
// do something as each item is made available
}
My first thought was to use the delay function, but I get this message:
Restricted suspended functions can only invoke member or extension suspending functions on their restricted coroutine scope
This is the signature for buildSequence:
public fun <T> buildSequence(builderAction: suspend SequenceBuilder<T>.() -> Unit): Sequence<T>
I think this message means that I can only use the suspend functions in SequenceBuilder: yield and yieldAll and that using arbitrary suspend function calls aren't allowed.
Right now I'm using this to block the sequence building by one second after every time the API is polled:
val resumeTime = System.nanoTime() + TimeUnit.SECONDS.toNanos(1)
while (resumeTime > System.nanoTime()) {
// do nothing
}
This works, but it really doesn't seem like a good solution. Has anybody encountered this issue before?
Why does it not work? Some research
When we look at buildSequence, we can see that it takes an builderAction: suspend SequenceBuilder<T>.() -> Unit as its argument. As a client of that method, you'll be able to hand on a suspend lambda that has SequenceBuilder as its receiver (read about lambda with receiver here).
The SequenceBuilder itself is annotated with RestrictSuspension:
#RestrictsSuspension
#SinceKotlin("1.1")
public abstract class SequenceBuilder<in T> ...
The annotation is defined and commented like this:
/**
* Classes and interfaces marked with this annotation are restricted
* when used as receivers for extension `suspend` functions.
* These `suspend` extensions can only invoke other member or extension
* `suspend` functions on this particular receiver only
* and are restricted from calling arbitrary suspension functions.
*/
#SinceKotlin("1.1") #Target(AnnotationTarget.CLASS) #Retention(AnnotationRetention.BINARY)
public annotation class RestrictsSuspension
As the RestrictSuspension documentation tells, in the case of buildSequence, you can pass a lambda with SequenceBuilder as its receiver but with restricted possibilities since you'll only be able to call "other member or extension suspend functions on this particular receiver". That means, the block passed to buildSequence may call any method defined on SequenceBuilder (like yield, yieldAll). Since, on the other hand, the block is "restricted from calling arbitrary suspension functions", using delay does not work. The resulting compiler error verifies it:
Restricted suspended functions can only invoke member or extension suspending functions on their restricted coroutine scope.
Ultimately, you need to be aware that the buildSequence creates a coroutine that is an example of a synchronous coroutine. In your example, the sequence code will be executed in the same thread that consumes the sequence by calling connect().
How to delay the sequence?
As we learned, The buildSequence creates a synchronous sequence. It's fine to use regular Thread blocking here:
fun connect(): Sequence<T> = buildSequence {
while (true) {
val result = dataSource.getFirstPage()
yieldAll(result)
Thread.sleep(1000)
}
}
But, do you really want an entire thread to be blocked? Alternatively, you can implement asynchronous sequences as described here. As a result, using delay and other suspending functions will be valid.
Just for an alternate solution...
If what you're really trying to do is asynchronously produce elements, you can use Flows which are basically asynchronous sequences.
Here is a quick table:
Sync
Async
Single
Normal valuefun example(): String
suspendingsuspend fun example(): Stringorfun example(): Deferred<String>
Many
Sequencefun example(): Sequence<String>
Flowfun example(): Flow<String>
You can convert your Sequence<T> to a Flow<T> by replacing the sequence { ... } builder with the flow { ... } builder and then replace yield/yieldAll with emit/emitAll:
fun example(): Flow<String> = flow {
(1..5).forEach { getString().let { emit(it) } }
}
suspend fun getString(): String = { ... }
So, for your example:
fun connect(): Flow<T> = flow {
while (true) {
// Call suspend function to get data from dataSource
val result: List<T> = dataSource.getFirstPage()
emitAll(result)
// _Suspend_ for a little bit
delay(1000)
}
}