I consider why there is no Retrofit2 adapter for Flow type like FlowCallAdapter and FlowCallAdapterFactory?
I found that retrofit supports suspend functions, and http request are one-shot so it better fits to have suspend functions rather then Flow return type, but Flow is more equivalent of RxJava and reactive programming frameworks and better fit in this schema of programming
How I can do something like this withoud Retrofit returning Flow and instead having suspend function
rules
.filter { it.isAsync }
.asFlow()
.flatMapMerge {
val rule = it
rule.validateAsync(input)
.filter { !it }
.map { rule }
}
.scan(mutableListOf<String>()) { acc, rule ->
acc.add(rule.errorMessage)
acc
}
.flowOn(Dispatchers.IO)
If I am using suspend I need to do something like this generally
return flow { emit(service.validate(email = value)) }
Because HTTP does not return a sequence of responses, only one response at a time. They replied they are not going to support that
https://github.com/square/retrofit/issues/3497
But you can implement your own CallAdapter to use flow
Related
I have an external interface which I cannot change:
interface ExternalApi {
fun onDataReceived(data: String)
}
I need to start consuming data and send it to flow. Data order is a necessity. I'd like to have a cold flow, but I couldn't find a version of cold flow with emit function, so I used hot flow + replay set to Max value as a workaround. Here was my first try:
class FlowProblem {
val flow: MutableSharedFlow<String> = MutableSharedFlow(replay = Int.MAX_VALUE)
fun startConsuming() {
object : ExternalApi {
override fun onDataReceived(data: String) {
flow.emit(data)
}
}
}
}
Unfortunately it doesn't work as emit function is a suspended function. However this is an external interface and I cannot add suspend modifier. I tried to also do something like this:
override fun onDataReceived(data: String) {
val coroutineScope = CoroutineScope(Job())
coroutineScope.launch {
flow.emit(data)
}
}
but for me it's kind a silly to create new coroutine only in order to move data to flow. I'm also wondering about data order.
What should I do? Maybe flow/channel is not suitable here and I should pick something another?
Thanks IR42, callbackFlow was exactly what I needed.
I can see the following example working in Spring WebFlux handler for a flow builder:
suspend fun getDummyFlow(req: ServerRequest): ServerResponse {
val flow = flow<String> { // flow builder
for (i in 1..3) {
delay(1000) // pretend we are doing something useful here
emit("<p>Hello $i</p>") // emit next value
}
}
return ServerResponse
.ok()
.contentType(MediaType.TEXT_HTML)
.bodyAndAwait(flow)
}
Yet, I need to build a flow with a MutableSharedFlow which is not working in Spring Web Flux. Here it is an example:
suspend fun getDummyFlow(req: ServerRequest): ServerResponse {
return coroutineScope {
val flow = MutableSharedFlow<String>()
launch {
for (i in 1..3) {
delay(1000) // pretend we are doing something useful here
flow.emit("<p>Hello $i</p>") // emit next value
}
}
ServerResponse
.ok()
.contentType(MediaType.TEXT_HTML)
.bodyAndAwait(
flow
.asSharedFlow()
.take(3)
)
}
My implementation is based on the example of SharedFlow documentation.
Yet, any HTTP GET request to this endpoint stays pending and waiting for a response, whereas the former example with flow builder receives the response progressively and fine.
I have already traced my code in debug and I see .bodyAndAwait(..) being called and then emit() in both cases.
Summary
My goal is to process and aggregate data from multiple servers efficiently while handling possible errors. For that, I
have a sequential version that I want to speed up. As I am using Kotlin, coroutines seem the way to go for this
asynchronous task. However, I'm quite new to this, and can't figure out how to do this idiomatic. None of my attempts
satisfied my requirements completely.
Here is the sequential version of the core function that I am currently using:
suspend fun readDataFromServers(): Set<String> = coroutineScope {
listOfServers
// step 1: read data from servers while logging errors
.mapNotNull { url ->
runCatching { makeRequestTo(url) }
.onFailure { println("err while accessing $url: $it") }
.getOrNull()
}
// step 2: do some element-wise post-processing
.map { process(it) }
// step 3: aggregate data
.toSet()
}
Background
In my use case, there are numServers I want to read data from. Each of them usually answers within successDuration,
but the connection attempt may fail after timeoutDuration with probability failProb and throw an IOException. As
downtimes are a common thing in my system, I do not need to retry anything, but only log it for the record. Hence,
the makeRequestTo function can be modelled as follows:
suspend fun makeRequestTo(url: String) =
if (random.nextFloat() > failProb) {
delay(successDuration)
"{Some response from $url}"
} else {
delay(timeoutDuration)
throw IOException("Connection to $url timed out")
}
Attempts
All these attempts can be tried out in the Kotlin playground. I don't know how long this link stays alive; maybe I'll need to upload this as a gist, but I liked that people can execute the code directly.
Async
I tried using async {makeRequestTo(it)} after listOfServers and awaiting the results in the following mapNotNull
similar
to this post
. While this collapses the communication time to timeoutDuration, all following processing steps have to wait for that
long before they can continue. Hence, some composition of Deferreds was required here, which is discouraged in
Kotlin (or at least should be avoided in favor of suspending
functions).
suspend fun readDataFromServersAsync(): Set<String> = supervisorScope {
listOfServers
.map { async { makeRequestTo(it) } }
.mapNotNull { kotlin.runCatching { it.await() }.onFailure { println("err: $it") }.getOrNull() }
.map { process(it) }
.toSet()
}
Loops
Using normal loops like below fulfills the functional requirements, but feels a bit more complex than it should be.
Especially the part where shared state must be synchronized makes me to not trust this code and any future modifications
to it.
val results = mutableSetOf<String>()
val mutex = Mutex()
val logger = CoroutineExceptionHandler { _, exception -> println("err: $exception") }
for (server in listOfServers) {
launch(logger) {
val response = makeRequestTo(server)
val processed = process(response)
mutex.withLock {
results.add(processed)
}
}
}
return#supervisorScope results
I'm wandering if there is a clean way to launch a series of flows in Kotlin and then, after their resolution, perform further operations based on whether they succeeded or not
For example's sake I need to read all integers from a DB (returning them into a flow), check if they are even or odd against an external API (also returning a flow), and then remove the odd ones from the DB
In code it would be something like this
fun findEven() {
db.readIntegers()
.map { listOfInt ->
listOfInt.asFlow()
.flatMapMerge { singleInt ->
httpClient.apiCallToCheckForOddity(singleInt)
.catch {
// API failure when number is even
}
.map {
// API success when number is odd
db.remove(singleInt).collect()
}
}.collect()
}.collect()
}
But the problem I see with this code is the access to the DB deleting entries done in parallel, and I think a better solution would be to run all API calls and somewhere collect all that failed and all that succeeded, so to be able to do a bulk insertion in the DB only once instead of having multiple coroutines do that on their own
In my opinion, it's kind of an anti-pattern to produce side effects in map, filter, etc. A side effect like removing items from a database should be a separate step (collect in the case of a Flow, and forEach in the case of a List) for clarity.
The nested flow is also kind of convoluted, since you can directly modify the list as a List.
I think you can do it like this, assuming the API can only check one item at a time.
suspend fun findEven() {
db.readIntegers()
.map { listOfInt ->
listOfInt.filter { singleInt ->
runCatching {
httpClient.apiCallToCheckForOddity(singleInt)
}.isSuccess
}
}
.collect { listOfOddInt ->
db.removeAll(listOfOddInt)
}
}
Parallel version, if the API call returns the parameter. (By the way, Kotlin APIs should not throw exceptions on non-programmer errors).
suspend fun findEven() {
db.readIntegers()
.map { listOfInt ->
coroutineScope {
listOfInt.map { singleInt ->
async {
runCatching {
httpClient.apiCallToCheckForOddity(singleInt)
}
}
}.awaitAll()
.mapNotNull(Result<Int>::getOrNull)
}
}
.collect { listOfOddInt ->
db.removeAll(listOfOddInt)
}
}
I have a Flux of strings that should be converted to a Flux of dto. Parsing can be finished with an error and by the business rules I just need to skip such entries
If I use "Kotlin's" null - I got NPE because by design reactor doesn't accept nulls in .map
fun toDtoFlux(source:Flux<String>):Flux<Dto>{
source.map(Parser::parse)
.filter(it!=null)
}
object Parser{
fun parse(line:String):Dto?{
..
}
}
I can use Optional. But it is not a Kotlin way.
fun toDtoFlux(source:Flux<String>):Flux<Dto>{
source.map(Parser::parse)
.filter(Optional.isPresent)
.map(Optional::get)
}
object Parser{
fun parse(line:String):Optional<Dto>{
..
}
}
What is the most idiomatic way to handle such cases in Kotlin?
You can create an extension function:
fun <T, U> Flux<T>.mapNotNull(mapper: (T) -> U?): Flux<U> =
this.flatMap { Mono.justOrEmpty(mapper(it)) }
Then you can use it like this:
fun main() {
Flux.just("a", "b", "c")
.mapNotNull { someNullableMapFunction(it) }
.doOnNext { println(it) } // prints "a" and "c"
.blockLast()
}
fun someNullableMapFunction(it: String): String? {
if (it == "b") {
return null
}
return it
}
UPDATE
Based on Simon's comment extension function implementation might be more idiomatic (and performant?) in Reactor this way:
fun <T, U> Flux<T>.mapNotNull(mapper: (T) -> U?): Flux<U> =
this.handle { item, sink -> mapper(item)?.let { sink.next(it) } }
The solutions I see :
Using Reactor API
I'd suggest you to use Reactor API to address such case, and make your parser return a Mono. The empty Mono represents the absence of result. With that, you can use flatMap instead of chaining map/filter/map.
It may seem a little overkill like that, but it will allow any parser implementation to do async stuff in the future if needed (fetching information from third-party service, waiting validation from user, etc.).
And it also provide a powerful API to manage parsing errors, as you can define backoff/custom error policies on parsing result.
That would change your example like that :
fun interface Parser {
fun parse(record: String): Mono<Dto>;
}
fun Parser.toDtoFlux(source:Flux<String>): Flux<Dto> {
source.flatMap(this::parse)
}
Using sealed class
Kotlin offers other ways of managing result options, inspired by functional programming. One way is to use sealed classes to desing a set of common cases to handle upon parsing. It allows to model rich results, giving parser users multiple choices to handle errors.
sealed class ParseResult
class Success(val value: Dto) : ParseResult
class Failure(val reason : Exception) : ParseResult
object EmptyRecord : ParseResult
fun interface Parser {
fun parse(raw: String) : ParseResult
}
fun Parser.toDtoFlux(source:Flux<String>): Flux<Dto> {
return source.map(this::parse)
.flatMap { when (it) {
is Success -> Mono.just(it.value)
is Failure -> Mono.error(it.reason) // Or Mono.empty if you don't care
is EmptyRecord -> Mono.empty()
}}
}