When I'm using Reactive Streams (https://github.com/reactor/reactor-core) with a custom Publisher in combination with the publishOn function, I always get an NPE. What is wrong with my code? Do I use the Publisher in a wrong way?
Flux.from(MyPublisher())
.publishOn(Schedulers.single())
.subscribe { println("<-- $it received") }
class MyPublisher : Publisher<Int> {
override fun subscribe(sub: Subscriber<in Int>) {
while (true) {
Thread.sleep(300)
sub.onNext(1)
}
}
}
Exception is:
Exception in thread "main" java.lang.NullPointerException
at reactor.core.publisher.FluxPublishOn$PublishOnSubscriber.onNext(FluxPublishOn.java:212)
at org.guenhter.kotlin.hello.MyPublisher.subscribe(HelloWorld.kt:18)
at reactor.core.publisher.FluxSource.subscribe(FluxSource.java:52)
at reactor.core.publisher.FluxPublishOn.subscribe(FluxPublishOn.java:96)
at reactor.core.publisher.Flux.subscribe(Flux.java:6447)
at reactor.core.publisher.Flux.subscribeWith(Flux.java:6614)
at reactor.core.publisher.Flux.subscribe(Flux.java:6440)
at reactor.core.publisher.Flux.subscribe(Flux.java:6404)
at reactor.core.publisher.Flux.subscribe(Flux.java:6347)
at org.guenhter.kotlin.hello.HelloWorldKt.main(HelloWorld.kt:11)
Publisher is defined by the "reactive-streams" standard and has a number of requirements. One of these requirements is that Subscriber.onSubscribe HAS to be called before any of the other methods in order to follow the protocol.
Since you haven't done this, it means something probably is not initialized properly, causing the NPE inside of the reactor class.
However even if you fix this problem the standard is designed to be reactive which means it only emits data when the subscriber asks for it. Since you will be sending it data regardless that will probably cause an exception later down the line. Use Flux.create to create an emitter that can properly handle requests instead of creating your own Publisher implementation.
Related
I have a function in my ViewModel in which I subscribe to some updates, I want to write a test that will check that after the subscribe is triggered, the specific function is called from the subscribe.
Here is how the function looks:
fun subscribeToTablesUpdates() {
dataManager.getTablesList()
.subscribeOn(Schedulers.io())
.observeOn(AndroidSchedulers.mainThread())
.subscribe { tablesList ->
updateTablesState(tablesList)
}
}
And this is the test that I wrote:
#Test
fun subscribeToTablesListTest() {
val mockedTablesList = mockk<List<Table>()
every {
viewModel.dataManager.getTablesList()
} returns Observable.just(mockedTablesList)
viewModel.subscribeToTablesUpdates()
verify {
viewModel.updateTablesState(mockedTablesList)
}
}
The issue is that I receive assertion exception without any another info and I don't know how to fix that.
Edit 1: subscribeToTableUpdates() is calling from the init block of ViewModel.
So basically the test itself was done right, but there were linking issue. Since the function of the VM was called from the init block the subscription happened only once, and that created a situation when at the time when I mocked the data service, the observer was already subscribed to the other service. Since the init block is called only once, there is no way to change the implementation of the data service to that observer.
After all this investigation the one thing which I successfully forgot came to my mind again: extract every external dependencies to constructors, so further you could substitute it for the test without any problems like this.
I am using kotlin and I wanted to stream over a possibly huge resultset using flows. I found some explanations around the web:
Callbacks and Kotlin Flows
Use Flow for asynchronous data streams
I implemented it and it works fine. I also needed to batch the results before sending them to an external services, so I implemented a chunked operation on flows. Something like that:
fun <T> Flow<T>.chunked(chunkSize: Int): Flow<List<T>> {
return callbackFlow {
val listOfResult = mutableListOf<T>()
this#chunked.collect {
listOfResult.add(it)
if (listOfResult.size == chunkSize) {
trySendBlocking(listOfResult.toList())
listOfResult.clear()
}
}
if (listOfResult.isNotEmpty()) {
trySendBlocking(listOfResult)
}
close()
}
}
To be sure that everything was working fine, I created some integration tests:
first flow + chuncked to consume all rows, passed
using the first flow (the one created from the jdbc repository) and
applying take operator just to consider few x items. It passed correctly.
using first flow + chunked operator + take operator, it hangs forever
So the last test showed that there was something wrong in the implementation.
I investigated a lot without finding nothing useful but, dumping the threads, I found a coroutine thread blocked in the trySendBlocking call on the first flow, the one created in the jdbc repository.
I am wondering in which way the chunked operator is supposed to propagate the closing to the upstream flow since it seems this part is missing.
In both cases I am propagating downstream the end of data with a close() call but I took a look the take operator and I saw it is triggering back the closing with an emitAbort(...)
Should I do something similar in the callbackFlow{...}?
After a bit of investigation, I was able to avoid the locking adding a timeout on the trySendBlocking inside the repository but I didn´t like that. At the end, I realized that I could cast the original flow (in the chunked operator) to a SendChannel and close it if the downstream flow is closed:
trySendBlocking(listOfResult.toList()).onSuccess {
LOGGER.debug("Sent")
}.onFailure {
LOGGER.warn("An error occurred sending data.", it)
}.onClosed {
LOGGER.info("Channel has been closed")
(originalFlow as SendChannel<*>).close(it)
}
Is this the correct way of closing flows backwards? Any hint to solve this issue?
Thanks!
You shouldn't use trySendBlocking instead of send. You should never use a blocking function in a coroutine without wrapping it in withContext with a Dispatcher that can handle blocking code (e.g. Dispatchers.Default). But when there's a suspend function alternative, use that instead, in this case send().
Also, callbackFlow is more convoluted than necessary for transforming a flow. You should use the standard flow builder instead (and so you'll use emit() instead of send()).
fun <T> Flow<T>.chunked(chunkSize: Int): Flow<List<T>> = flow {
val listOfResult = mutableListOf<T>()
collect {
listOfResult.add(it)
if (listOfResult.size == chunkSize) {
emit(listOfResult.toList())
listOfResult.clear()
}
}
if (listOfResult.isNotEmpty()) {
emit(listOfResult)
}
}
I have read the set-based consistency validation blog and I want to validate through a dispatch interceptor. I follow the example, but I use reactive repository and it doesn't really work for me. I have tried both block and not block. with block it throws error, but without block it doesn't execute anything. here is my code.
class SubnetCommandInterceptor : MessageDispatchInterceptor<CommandMessage<*>> {
#Autowired
private lateinit var privateNetworkRepository: PrivateNetworkRepository
override fun handle(messages: List<CommandMessage<*>?>): BiFunction<Int, CommandMessage<*>, CommandMessage<*>> {
return BiFunction<Int, CommandMessage<*>, CommandMessage<*>> { index: Int?, command: CommandMessage<*> ->
if (CreateSubnetCommand::class.simpleName == (command.payloadType.simpleName)){
val interceptCommand = command.payload as CreateSubnetCommand
privateNetworkRepository
.findById(interceptCommand.privateNetworkId)
// ..some validation logic here ex.
// .filter { network -> network.isSubnetOverlap() }
.switchIfEmpty(Mono.error(IllegalArgumentException("Requested subnet is overlap with the previous subnet.")))
// .block() also doesn't work here it throws error
// block()/blockFirst()/blockLast() are blocking, which is not supported in thread reactor-
}
command
}
}
}
Subscribing to a reactive repository inside a message dispatcher is not really recommended and might lead to weird behavior as underling ThreadLocal (used by Axox) is not adapted to be used in reactive programing
Instead, check out Axon's Reactive Extension and reactive interceptors section.
For example what you might do:
reactiveCommandGateway.registerDispatchInterceptor(
cmdMono -> cmdMono.flatMap(cmd->privateNetworkRepository
.findById(cmd.privateNetworkId))
.switchIfEmpty(
Mono.error(IllegalArgumentException("Requested subnet is overlap with the previous subnet."))
.then(cmdMono)));
I have a dilemma about handling exceptions in kotlin and coroutines. Will be thankful for any articles or your personal experience.
https://kotlinlang.org/docs/exceptions.html#checked-exceptions kotlin documentation says that exceptions were mistake so there are no checked exceptions in kotlin.
If you have multilayer architecture it is pain in the ass to handle an exception from bottom layer at the top one, cause there are no warnings or compile level checks for what exception you should wait for.
As an option you can catch exception asap, wrap it with some Result class and propagate it back as a return value.
return try {
Result.Success(api.call())
} catch(ex: IOException) {
Result.NetworkError
}
This solution works well until you get hands on coroutines with async/await.
If an exception is thrown inside the first async then whole scope becomes dead.
coroutineScope {
val r1 = async { }
val r2 = async { }
r1.await()
r2.await()
}
But if you use solution with return value, then both async will complete. Even if one of them completed with wrapped error. In 99% cases this behavior doesn't make sense.
So I run into situation interesting situation - there are no compile level/lint checks for catching/throwing checked exceptions so I can't use exceptions properly. But if I get rid of exceptions coroutines start act weird.
Here is an example to illustrate my confusion:
fun main() = runBlocking(Dispatchers.Default + CoroutineName("Main")) {
val broadcaster = BroadcastChannel<Int>(Channel.BUFFERED)
val flow = withContext(CoroutineName("InitialFlowCreation")) {
broadcaster.asFlow()
.map {
println("first mapping in context: $coroutineContext")
it * 10
}
.broadcastIn(CoroutineScope(Dispatchers.Default + CoroutineName("BroadcastIn")))
.asFlow()
}
val updatedFlow = withContext(CoroutineName("UpdatedFlowCreation")) {
flow.map {
println("second mapping in context: $coroutineContext")
it * 10
}
.flowOn(Dispatchers.Default + CoroutineName("FlowOn"))
}
launch(CoroutineName("Collector")) {
updatedFlow.collect {
println("Collecting $it in context: $coroutineContext")
}
}
delay(1_000)
launch(CoroutineName("OriginalBroadcast")) {
for (i in 1..10) {
broadcaster.send(i)
println("Sent original broadcast from: $coroutineContext")
delay(1_000)
}
}
return#runBlocking
}
This produces the following output (truncated):
Sent original broadcast from: [CoroutineName(OriginalBroadcast), StandaloneCoroutine{Active}#3a14b06a, DefaultDispatcher]
first mapping in context: [CoroutineName(InitialFlowCreation), UndispatchedCoroutine{Completed}#40202c08, DefaultDispatcher]
second mapping in context: [CoroutineName(UpdatedFlowCreation), UndispatchedCoroutine{Completed}#6cf04ddc, DefaultDispatcher]
Collecting 100 in context: [CoroutineName(Collector), StandaloneCoroutine{Active}#6ac9d4b5, DefaultDispatcher]
The documentation states things in various places that causes me to be confused by this result.
In Flow we have "Use channelFlow if the collection and emission of a flow are to be separated into multiple coroutines. It encapsulates all the context preservation work and allows you to focus on your domain-specific problem, rather than invariant implementation details. It is possible to use any combination of coroutine builders from within channelFlow." I know I'm not actually using the channelFlow function but a ChannelFlow is being created internally when we call broadcastIn so the same principals should apply.
I thought the first invocation of map would be run in the "OriginalBroadcast" context and the second would either be run in the "BroadcastIn" context or the "Collector" context but instead they are both run in the context where they are called. I don't understand why this is happening, shouldn't the context of map be where it is collected in order to be broadcast or the context where it is finally collected, not the context where map is called? Also the call to flowOn has no effect. What context preservation work is being encapsulated here?
Also am I correct that in a chain of flow.broadcastIn(...).asFlow().map{...}.broadcastIn(...).asFlow() the two BroadcastChannels created will not be fused? Trying to make sure I'm not missing something.
I guess what I'm really looking for is inclusive documentation of in what situation Channels are fused, how they are fused, and what context the operators that are called between ChannelFlow operators will run in.
The context preservation only applies to operations on flows, e.g. the code in flow { ... } builder works in the same context that calls collect(). The context is not preserved when operating via channels by the very nature of channels. Channels are communication primitives that are designed for communication between different coroutines.
It means that when you call broadcaster.send in one coroutine it will be received in another coroutine, in a coroutine that collects from the corresponding flow.
The documentation on channelFlow simply means that you don't have to worry about context preservation violation, which is non-trivial to ensure if you were to write such a primitive yourself.