How does Kotlin flow created with BroadcastChannel.asFlow() context preservation work? - kotlin

Here is an example to illustrate my confusion:
fun main() = runBlocking(Dispatchers.Default + CoroutineName("Main")) {
val broadcaster = BroadcastChannel<Int>(Channel.BUFFERED)
val flow = withContext(CoroutineName("InitialFlowCreation")) {
broadcaster.asFlow()
.map {
println("first mapping in context: $coroutineContext")
it * 10
}
.broadcastIn(CoroutineScope(Dispatchers.Default + CoroutineName("BroadcastIn")))
.asFlow()
}
val updatedFlow = withContext(CoroutineName("UpdatedFlowCreation")) {
flow.map {
println("second mapping in context: $coroutineContext")
it * 10
}
.flowOn(Dispatchers.Default + CoroutineName("FlowOn"))
}
launch(CoroutineName("Collector")) {
updatedFlow.collect {
println("Collecting $it in context: $coroutineContext")
}
}
delay(1_000)
launch(CoroutineName("OriginalBroadcast")) {
for (i in 1..10) {
broadcaster.send(i)
println("Sent original broadcast from: $coroutineContext")
delay(1_000)
}
}
return#runBlocking
}
This produces the following output (truncated):
Sent original broadcast from: [CoroutineName(OriginalBroadcast), StandaloneCoroutine{Active}#3a14b06a, DefaultDispatcher]
first mapping in context: [CoroutineName(InitialFlowCreation), UndispatchedCoroutine{Completed}#40202c08, DefaultDispatcher]
second mapping in context: [CoroutineName(UpdatedFlowCreation), UndispatchedCoroutine{Completed}#6cf04ddc, DefaultDispatcher]
Collecting 100 in context: [CoroutineName(Collector), StandaloneCoroutine{Active}#6ac9d4b5, DefaultDispatcher]
The documentation states things in various places that causes me to be confused by this result.
In Flow we have "Use channelFlow if the collection and emission of a flow are to be separated into multiple coroutines. It encapsulates all the context preservation work and allows you to focus on your domain-specific problem, rather than invariant implementation details. It is possible to use any combination of coroutine builders from within channelFlow." I know I'm not actually using the channelFlow function but a ChannelFlow is being created internally when we call broadcastIn so the same principals should apply.
I thought the first invocation of map would be run in the "OriginalBroadcast" context and the second would either be run in the "BroadcastIn" context or the "Collector" context but instead they are both run in the context where they are called. I don't understand why this is happening, shouldn't the context of map be where it is collected in order to be broadcast or the context where it is finally collected, not the context where map is called? Also the call to flowOn has no effect. What context preservation work is being encapsulated here?
Also am I correct that in a chain of flow.broadcastIn(...).asFlow().map{...}.broadcastIn(...).asFlow() the two BroadcastChannels created will not be fused? Trying to make sure I'm not missing something.
I guess what I'm really looking for is inclusive documentation of in what situation Channels are fused, how they are fused, and what context the operators that are called between ChannelFlow operators will run in.

The context preservation only applies to operations on flows, e.g. the code in flow { ... } builder works in the same context that calls collect(). The context is not preserved when operating via channels by the very nature of channels. Channels are communication primitives that are designed for communication between different coroutines.
It means that when you call broadcaster.send in one coroutine it will be received in another coroutine, in a coroutine that collects from the corresponding flow.
The documentation on channelFlow simply means that you don't have to worry about context preservation violation, which is non-trivial to ensure if you were to write such a primitive yourself.

Related

Thread-safe access to the same variable from different flows (Kotlin)

Is this code thread safe? Do I need a synchronized block or something like that? source1 and source2 endless Kotlin Flow
viewModelScope.launch {
var listAll = mutableListOf<String>()
var list1 = mutableListOf<String>()
var list2 = mutableListOf<String>()
launch {
source1.getNames().collect { list ->
list1 = list
listAll = mutableListOf()
listAll.addAll(list1)
listAll.addAll(list2)
//then consume listAll as StateFlow or return another flow with emit(listAll)
}
}
launch {
source2.getNames().collect { list ->
list2 = list
listAll = mutableListOf()
listAll.addAll(list2)
listAll.addAll(list1)
//then consume listAll as StateFlow or return another flow with emit(listAll)
}
}
}
This code is not thread safe.
However, it is called from viewModelScope.launch which runs on Dispatchers.Main by default. So your inner launch blocks will be called sequentially. This means that after all you will get the result which is produced by second launch block.
To achieve asynchronous behavior, you want to use viewModelScope.launch(Dispatchers.Default).
Your code will probably fire concurrent modification exception in that case.
To synchronize it, you may want to use Java's Collections.synchronizedList which blocks the list while one thread is performing operations with it, so the other thread are not able to perform modifications.
Or perform synchronizing manually using Mutex.
val mutex = Mutex()
viewModelScope.launch(Dispatchers.Default) {
launch {
mutex.withLock {
... // Your code
}
}
launch {
mutex.withLock {
... // Your code
}
}
}
Read official Kotlin guide to shared mutable state
After all, I am struggling to imagine real life example in which you will actually use that code. You probably don't need asynchronous behavior, you will be fine without using two launch blocks. Or you should rethink your design to avoid need of manual synchronization of two coroutines.

How to propagate closing to a chain of flows in kotlin

I am using kotlin and I wanted to stream over a possibly huge resultset using flows. I found some explanations around the web:
Callbacks and Kotlin Flows
Use Flow for asynchronous data streams
I implemented it and it works fine. I also needed to batch the results before sending them to an external services, so I implemented a chunked operation on flows. Something like that:
fun <T> Flow<T>.chunked(chunkSize: Int): Flow<List<T>> {
return callbackFlow {
val listOfResult = mutableListOf<T>()
this#chunked.collect {
listOfResult.add(it)
if (listOfResult.size == chunkSize) {
trySendBlocking(listOfResult.toList())
listOfResult.clear()
}
}
if (listOfResult.isNotEmpty()) {
trySendBlocking(listOfResult)
}
close()
}
}
To be sure that everything was working fine, I created some integration tests:
first flow + chuncked to consume all rows, passed
using the first flow (the one created from the jdbc repository) and
applying take operator just to consider few x items. It passed correctly.
using first flow + chunked operator + take operator, it hangs forever
So the last test showed that there was something wrong in the implementation.
I investigated a lot without finding nothing useful but, dumping the threads, I found a coroutine thread blocked in the trySendBlocking call on the first flow, the one created in the jdbc repository.
I am wondering in which way the chunked operator is supposed to propagate the closing to the upstream flow since it seems this part is missing.
In both cases I am propagating downstream the end of data with a close() call but I took a look the take operator and I saw it is triggering back the closing with an emitAbort(...)
Should I do something similar in the callbackFlow{...}?
After a bit of investigation, I was able to avoid the locking adding a timeout on the trySendBlocking inside the repository but I didnĀ“t like that. At the end, I realized that I could cast the original flow (in the chunked operator) to a SendChannel and close it if the downstream flow is closed:
trySendBlocking(listOfResult.toList()).onSuccess {
LOGGER.debug("Sent")
}.onFailure {
LOGGER.warn("An error occurred sending data.", it)
}.onClosed {
LOGGER.info("Channel has been closed")
(originalFlow as SendChannel<*>).close(it)
}
Is this the correct way of closing flows backwards? Any hint to solve this issue?
Thanks!
You shouldn't use trySendBlocking instead of send. You should never use a blocking function in a coroutine without wrapping it in withContext with a Dispatcher that can handle blocking code (e.g. Dispatchers.Default). But when there's a suspend function alternative, use that instead, in this case send().
Also, callbackFlow is more convoluted than necessary for transforming a flow. You should use the standard flow builder instead (and so you'll use emit() instead of send()).
fun <T> Flow<T>.chunked(chunkSize: Int): Flow<List<T>> = flow {
val listOfResult = mutableListOf<T>()
collect {
listOfResult.add(it)
if (listOfResult.size == chunkSize) {
emit(listOfResult.toList())
listOfResult.clear()
}
}
if (listOfResult.isNotEmpty()) {
emit(listOfResult)
}
}

Axonframework, how to use MessageDispatchInterceptor with reactive repository

I have read the set-based consistency validation blog and I want to validate through a dispatch interceptor. I follow the example, but I use reactive repository and it doesn't really work for me. I have tried both block and not block. with block it throws error, but without block it doesn't execute anything. here is my code.
class SubnetCommandInterceptor : MessageDispatchInterceptor<CommandMessage<*>> {
#Autowired
private lateinit var privateNetworkRepository: PrivateNetworkRepository
override fun handle(messages: List<CommandMessage<*>?>): BiFunction<Int, CommandMessage<*>, CommandMessage<*>> {
return BiFunction<Int, CommandMessage<*>, CommandMessage<*>> { index: Int?, command: CommandMessage<*> ->
if (CreateSubnetCommand::class.simpleName == (command.payloadType.simpleName)){
val interceptCommand = command.payload as CreateSubnetCommand
privateNetworkRepository
.findById(interceptCommand.privateNetworkId)
// ..some validation logic here ex.
// .filter { network -> network.isSubnetOverlap() }
.switchIfEmpty(Mono.error(IllegalArgumentException("Requested subnet is overlap with the previous subnet.")))
// .block() also doesn't work here it throws error
// block()/blockFirst()/blockLast() are blocking, which is not supported in thread reactor-
}
command
}
}
}
Subscribing to a reactive repository inside a message dispatcher is not really recommended and might lead to weird behavior as underling ThreadLocal (used by Axox) is not adapted to be used in reactive programing
Instead, check out Axon's Reactive Extension and reactive interceptors section.
For example what you might do:
reactiveCommandGateway.registerDispatchInterceptor(
cmdMono -> cmdMono.flatMap(cmd->privateNetworkRepository
.findById(cmd.privateNetworkId))
.switchIfEmpty(
Mono.error(IllegalArgumentException("Requested subnet is overlap with the previous subnet."))
.then(cmdMono)));

Should I emit from a coroutine when collecting from a different flow?

I have a use case where I need to trigger on a specific event collected from a flow and restart it when it closes. I also need to emit all of the events to a different flow. My current implementation looks like this:
scope.launch {
val flowToReturn = MutableSharedFlow<Event>()
while (true) {
client
.connect() // returns Flow<Event>
.catch { ... } // ignore errors
.onEach { launch { flowToReturn.emit(it) } } // problem here
.filterIsInstance<Event.Some>()
.collect { someEvent ->
doStuff(someEvent)
}
}
}.start()
The idea is to always reconnect when the client disconnects (collect then returns and a new iteration begins) while having the outer flow lifecycle separate from the inner (connection) one. It being a shared flow with potentially multiple subscribers is a secondary concern.
As the emit documentation states it is not thread-safe. Should I call it from a new coroutine then? My concern is that the emit will suspend if there are no subscribers to the outer flow and I need to run the downstream pipeline regardless.
The MutableSharedFlow.emit() documentation say that it is thread-safe. Maybe you were accidentally looking at FlowCollector.emit(), which is not thread-safe. MutableSharedFlow is a subtype of FlowCollector but promotes emit() to being thread-safe since it's not intended to be used as a Flow builder receiver like a plain FlowCollector. There's no reason to launch a coroutine just to emit to your shared flow.
There's no reason to call start() on a coroutine Job that was created with launch because launch both creates the Job and starts it.
You will need to declare flowToReturn before your launch call to be able to have it in scope to return from this outer function.

How to emit Flow value from different function? Kotlin Coroutines

I have a flow :
val myflow = kotlinx.coroutines.flow.flow<Message>{}
and want to emit values with function:
override suspend fun sendMessage(chat: Chat, message: Message) {
myflow.emit(message)
}
But compiler does not allow me to do this, is there any workarounds to solve this problem?
You can use StateFlow for such use case.
Here's a sample code.
import kotlinx.coroutines.*
import kotlinx.coroutines.flow.*
val chatFlow = MutableStateFlow<String>("")
fun main() = runBlocking {
// Observe values
val job = launch {
chatFlow.collect {
print("$it ")
}
}
// Change values
arrayOf("Hey", "Hi", "Hello").forEach {
delay(100)
sendMessage(it)
}
delay(1000)
// Cancel running job
job.cancel()
job.join()
}
suspend fun sendMessage(message: String) {
chatFlow.value = message
}
You can test this code by running below snippet.
<iframe src="https://pl.kotl.in/DUBDfUnX3" style="width:600px;"></iframe>
The answer of Animesh Sahu is pretty much correct. You can also return a Channel as a flow (see consumeAsFlow or asFlow on a BroadcastChannel).
But there is also a thing called StateFlow currently in development by Kotlin team, which is, in part, meant to implement a similar behavior, although it is unknown when it is going to be ready.
EDIT: StateFlow and SharedFlow have been released as part of a stable API (https://blog.jetbrains.com/kotlin/2020/10/kotlinx-coroutines-1-4-0-introducing-stateflow-and-sharedflow/). These tools can and should be used when state management is required in an async execution context.
Use a SharedStateFlow it has got everything you need.
Initialization of your flow:
val myFlow = MutableSharedFlow<Message>()
and now it should just work as you were trying earlier with:
override suspend fun sendMessage(chat: Chat, message: Message) {
myFlow.emit(message)
}
Flow is self contained, once the block (lambda) inside the flow is executed the flow is over, you've to do operations inside and emit them from there.
Here is the similar github issue, says:
Afaik Flow is designed to be a self contained, replayable, cold stream, so emission from outside of it's own scope wouldn't be part of the contract. I think what you're looking for is a Channel.
And IMHO you're probably looking at the Channels, or specifically a ConflatedBroadcastChannel for multiple receivers. The difference between a normal channel and a broadcast channel is that multiple receivers can listen to a broadcast channel using openSubscription function which returns a ReceiveChannel associated with the BroadcastChannel.