This question is loosely related to this question, but there were no answers. The answer from Bob Dalgleish is close, but doesn't support the potential error coming from a Single (which I think that OP actually wanted as well).
I'm basically looking for a way to "filter on error" - but don't think this exists when the lookup is RX based. I am trying to take a list of values, run them through a lookup, and skip any result that returns a lookup failure (throwable). I'm having trouble figuring out how to accomplish this in a reactive fashion.
I've tried various forms of error handling operators combined with mapping. Filter only works for raw values - or at least I couldn't figure out how to use it to support what I'd like to do.
In my use case, I iterate a list of IDs, requesting data for each from a remote service. If the service returns 404, then the item doesn't exist anymore. I should remove non-existing items from the local database and continue processing IDs. The stream should return the list of looked up values.
Here is a loose example. How do I write getStream() so that canFilterOnError passes?
import io.reactivex.Single
import io.reactivex.schedulers.Schedulers
import org.junit.Test
class SkipExceptionTest {
private val data: Map<Int, String> = mapOf(
Pair(1, "one"),
Pair(2, "two"),
Pair(4, "four"),
Pair(5, "five")
)
#Test
fun canFilterOnError() {
getStream(listOf(1, 2, 3, 4, 5))
.subscribeOn(Schedulers.trampoline())
.observeOn(Schedulers.trampoline())
.test()
.assertComplete()
.assertNoErrors()
.assertValueCount(1)
.assertValue {
it == listOf(
"one", "two", "four", "five"
)
}
}
fun getStream(list: List<Int>): Single<List<String>> {
// for each item in the list
// get it's value via getValue()
// if a call to getValue() results in a NotFoundException, skip that value and continue
// mutate the results using mutate()
TODO("not implemented")
}
fun getValue(id: Int): Single<String> {
return Single.fromCallable {
val value: String? = data[id]
if (value != null) {
data[id]
} else {
throw NotFoundException("dat with id $id does not exist")
}
}
}
class NotFoundException(message: String) : Exception(message)
}
First .materialize(), then .filter() on non-error events, then .dematerialize():
getStream(/* ... */)
.materialize()
.filter(notification -> { return !notification.isOnError(); })
.dematerialize()
I ended up mapping getValue() to Optional<String>, then calling onErrorResumeNext() on that and either returning Single.error() or Single.just(Optional.empty()). From there, the main stream could filter out the empty Optional.
private fun getStream(list: List<Int>): Single<List<String>> {
return Observable.fromIterable(list)
.flatMapSingle {
getValue(it)
.map {
Optional.of(it)
}
.onErrorResumeNext {
when (it) {
is NotFoundException -> Single.just(Optional.empty())
else -> Single.error(it)
}
}
}
.filter { it.isPresent }
.map { it.get() }
.toList()
}
Related
I currently face the problem of correctly closing resources that never leave their containing Either.
The relevant code looks something like this:
object SomeError
class MyRes : AutoCloseable { [...] }
fun createRes(): Either<SomeError, MyRes> { [...] }
fun extractData(res: MyRes): String { [...] }
fun theProblem(): Either<SomeError, String> {
return createRes()
.map { extractData(it) }
}
What is the most idiomatic way of closing the created MyRes? Closing it before that map prevents extractData from accessing it, and after the map I can't access it anymore via Either's operations. Closing it in extractData severely limits composability.
Currently I have an external List<AutoCloseable> that I iterate over after all the computations, but that can't be the intended way.
I am open to using Arrow Fx (e.g. Resource) if that helps, but I haven't found anything on how to combine Either and Resource in an elegant way.
It's possible to combine the either and Resource safely.
object SomeError
class MyRes : AutoCloseable { [...] }
fun createRes(): Resource<Either<SomeError, MyRes>> { [...] }
fun extractData(res: MyRes): String { [...] }
suspend fun solution(): Either<SomeError, String> = either {
createRes().use { either: Either<SomeError, MyRes> ->
val res = either.bind()
val string = extractData(res)
// call other Either code + `bind()` safely here
[...]
} // <-- MyRes will automatically close here
}
If in this code you encounter Either.Left and you call bind() on it the Resource will first close, because we jump outside of use, and then either will return the encountered Either.Left.
One possible solution I found was wrapping the block passed to map:
fun <B : AutoCloseable, C> andClose(f: (B) -> C): (B) -> C =
{ b: B -> b.use { f(b) } }
fun theProblemSlightlySolved(): Either<SomeError, String> {
return createRes()
.map(andClose { extractData(it) })
}
I want to invoke a function that will notify the admin about some information missing, but I do not want to subscribe to this Mono, because I will subscribe to it later. The problem is I have some log which is called inside doOnSuccess() and when I use subscribe() and then build a response where I zip listOfWords value, the same log is logged twice and I do not want a code to behave that way.
Is there any way to retrieve that value in checkCondition() in a way that will not invoke doOnSuccess() or should I use some other function in merge() that can replace doOnSuccess()?
Should I use subscribe() only once on given Mono or is it allowed to use it multiple times?
Thank you in advance!
The functions are called in the presented order.
Code where log is called:
private fun merge(list1: Mono<List<String>>, list2: Mono<List<String>>) =
Flux.merge(
list1.flatMapMany { Flux.fromIterable(it) },
list2.flatMapMany { Flux.fromIterable(it) }
)
.collectList()
.doOnSuccess { LOG.debug("List of words: $it") }
Code where subscribe is called:
private fun checkCondition(
listOfWords: Mono<List<String>>,
) {
listOfWords.subscribe {
it.forEach { word ->
if (someCondition(word)) {
alarmSystem.notify("Something is missing for word {0}")
}
}
}
}
Code where response is built:
private fun buildResponse(
map: Mono<Map<String, String>>,
list1: List<SomeObject>,
listOfWords: Mono<List<String>>
): Mono<List<Answer>> {
val response = Mono.zip(map, Mono.just(list1), listOfWords)
.map { tuple ->
run {
val tupleMap = tuple.t1
val list = tuple.t2
val words = tuple.t3
list
.filter { someCondition(words) }
.map { obj -> NewObject(x,y) }
}
}
I have a situation where I need to observe userIds then use those userIds to observe users. Either userIds or users could change at any time and I want to keep the emitted users up to date.
Here is an example of the sources of data I have:
data class User(val name: String)
fun observeBestUserIds(): Flow<List<String>> {
return flow {
emit(listOf("abc", "def"))
delay(500)
emit(listOf("123", "234"))
}
}
fun observeUserForId(userId: String): Flow<User> {
return flow {
emit(User("${userId}_name"))
delay(2000)
emit(User("${userId}_name_updated"))
}
}
In this scenario I want the emissions to be:
[User(abc_name), User(def_name)], then
[User(123_name), User(234_name)], then
[User(123_name_updated), User(234_name_updated)]
I think I can achieve this in RxJava like this:
observeBestUserIds.concatMapSingle { ids ->
Observable.fromIterable(ids)
.concatMap { id ->
observeUserForId(id)
}
.toList()
}
What function would I write to make a flow that emits that?
I believe you're looking for combine, which gives you an array that you can easily call toList() on:
observeBestUserIds().collectLatest { ids ->
combine(
ids.map { id -> observeUserForId(id) }
) {
it.toList()
}.collect {
println(it)
}
}
And here's the inner part with more explicit parameter names since you can't see the IDE's type hinting on Stack Overflow:
combine(
ids.map { id -> observeUserForId(id) }
) { arrayOfUsers: Array<User> ->
arrayOfUsers.toList()
}.collect { listOfUsers: List<User> ->
println(listOfUsers)
}
Output:
[User(name=abc_name), User(name=def_name)]
[User(name=123_name), User(name=234_name)]
[User(name=123_name_updated), User(name=234_name)]
[User(name=123_name_updated), User(name=234_name_updated)]
Live demo (note that in the demo, all the output appears at once, but this is a limitation of the demo site - the lines appear with the timing you'd expect when the code is run locally)
This avoids the (abc_name_updated, def_name_updated) discussed in the original question. However, there's still an intermediate emission with 123_name_updated and 234_name because the 123_name_updated is emitted first and it sends the combined version immediately because they're the latest from each flow.
However, this can be avoided by debouncing the emissions (on my machine, a timeout as small as 1ms works, but I did 20ms to be conservative):
observeBestUserIds().collectLatest { ids ->
combine(
ids.map { id -> observeUserForId(id) }
) {
it.toList()
}.debounce(timeoutMillis = 20).collect {
println(it)
}
}
which gets you the exact output you wanted:
[User(name=abc_name), User(name=def_name)]
[User(name=123_name), User(name=234_name)]
[User(name=123_name_updated), User(name=234_name_updated)]
Live demo
This is unfortunatly non trivial with the current state of kotlin Flow, there seem to be important operators missing. But please notice that you are not looking for rxJavas toList(). If you would try to to do it with toList and concatMap in rxjava you would have to wait till all observabes finish.
This is not what you want.
Unfortunately for you I think there is no way around a custom function.
It would have to aggregate all the results returned by observeUserForId for all the ids which you would pass to it. It would also not be a simple windowing function, since in reality it is conceivable that one observeUserForId already returned twice and another call still didn't finish. So checking whether you already have the same number of users as you passed ids into your aggregating functions isn't enought, you also have to group by user id.
I'll try to add code later today.
Edit: As promised here is my solution I took the liberty of augmenting the requirements slightly. So the flow will emit every time all userIds have values and an underlying user changes. I think this is more likely what you want since users probably don't change properties in lockstep.
Nevertheless if this is not what you want leave a comment.
import kotlinx.coroutines.delay
import kotlinx.coroutines.flow.*
import kotlinx.coroutines.runBlocking
data class User(val name: String)
fun observeBestUserIds(): Flow<List<String>> {
return flow {
emit(listOf("abc", "def"))
delay(500)
emit(listOf("123", "234"))
}
}
fun observeUserForId(userId: String): Flow<User> {
return flow {
emit(User("${userId}_name"))
delay(2000)
emit(User("${userId}_name_updated"))
}
}
inline fun <reified K, V> buildMap(keys: Set<K>, crossinline valueFunc: (K) -> Flow<V>): Flow<Map<K, V>> = flow {
val keysSize = keys.size
val valuesMap = HashMap<K, V>(keys.size)
flowOf(*keys.toTypedArray())
.flatMapMerge { key -> valueFunc(key).map {v -> Pair(key, v)} }
.collect { (key, value) ->
valuesMap[key] = value
if (valuesMap.keys.size == keysSize) {
emit(valuesMap.toMap())
}
}
}
fun observeUsersForIds(): Flow<List<User>> {
return observeBestUserIds().flatMapLatest { ids -> buildMap(ids.toSet(), ::observeUserForId as (String) -> Flow<User>) }
.map { m -> m.values.toList() }
}
fun main() = runBlocking {
observeUsersForIds()
.collect { user ->
println(user)
}
}
This will return
[User(name=def_name), User(name=abc_name)]
[User(name=123_name), User(name=234_name)]
[User(name=123_name_updated), User(name=234_name)]
[User(name=123_name_updated), User(name=234_name_updated)]
You can run the code online here
You can use flatMapConcat
val users = observeBestUserIds()
.flatMapConcat { ids ->
flowOf(*ids.toTypedArray())
.map { id ->
observeUserForId(id)
}
}
.flattenConcat()
.toList()
or
observeBestUserIds()
.flatMapConcat { ids ->
flowOf(*ids.toTypedArray())
.map { id ->
observeUserForId(id)
}
}
.flattenConcat()
.collect { user ->
}
How to change the parameters with retry() in kotlin and webflux ?
There is a productInfo function, the function parameter is a collection of product ids.
When I input a wrong id in the list collection ids, the upstream interface will only return the wrong id. And get failed.
What I want to achieve is when the upstream interface returns the wrong id. The product info can remove the wrong id and have a second try with the right ids.
I tried to use retry() but I don't know how to change the parameters in the second try.
fun productInfo(ids: List<Pair<String, String>>): Flux<ProductItem> {
return productWebClient
.get()
.uri("product/items/${ids.joinToString(";") { "${it.second},${it.first}" }}")
.retrieve()
.bodyToFlux(ProductItem::class.java)
.onErrorResume {
logger.error("Fetch products failed." + it.message)
Mono.empty()
}
}
What you want is not retry(). I've built a solution making minor assumptions here and there. You can refer to this solution and make changes according to your requirements. I've used recursion here (productInfo()). You can replace the recursion call with webclient call if the error occurs only once.
fun productInfo(ids: List<Pair<String, String>>): Flux<ProductItem> {
val idsString = ids.joinToString(";") { "${it.second},${it.first}" }
return webClient
.get()
.uri("product/items/${idsString}")
.exchange()
.flatMapMany { response ->
if (response.statusCode().isError) {
response.body { clientHttpResponse, _ ->
clientHttpResponse.body.cast(String::class.java).collectList()
.flatMapMany<ProductItem> { eids ->
val ids2 = ids.filter { eids.contains("${it.second},${it.first}") }
productInfo(ids2)
}
}
} else {
response.bodyToFlux(ProductItem::class.java)
}
}
}
Consider the following two classes:
class ObjectA(val objectBs: List<ObjectB>,
val otherFields: Any)
class ObjectB(val key: String,
val otherFields: Any)
The task is to find and return the first ObjectB with a certain key in a List of ObjectA.
Just achieving the goal is simple enough, but doing it nicely and efficiently seems rather tricky. I can't find anything like a "firstIn" or "findIn" function that would allow me to return another type than ObjectA when iterating on a list of ObjectA.
I have a few approaches, one of which looks pretty nice, but is very inefficient:
listOfA.mapNotNull {
it.objectBs.firstOrNull {
item -> item.key == wantedKey
}
}.firstOrNull()
The obvious inefficiency of this code is that it will not stop iterating through listOfA when it has found a match (and there can only be one match, just to be clear).
Approaches using filter or find have similar problems, requiring redundant iterations through at least one list of ObjectB.
Is there something in kotlins standard library that would cover such a use case?
If you want an elegant solution you can just do a flatMap like this:
val result: ObjectB? = listOfA.flatMap { it.objectBs }.firstOrNull { it.key == "myKey" }
If you want the efficiency you can do something like this:
val result: ObjectB? = objectAs.firstOrNull {
it.objectBs.map(ObjectB::key).contains("myKey")
}?.objectBs?.firstOrNull { it.key == "myKey" }
You can also wrap these in an Optional and put it in a function so the users of this operation can have a clean API:
fun List<ObjectA>.findFirstObjectB(key: String): Optional<ObjectB> {
return Optional.ofNullable(firstOrNull {
it.objectBs.map(ObjectB::key).contains(key)
}?.objectBs?.firstOrNull { it.key == key })
}
By converting all the nested elements to a flattened Sequence, they can be iterated lazily, and the overhead of unnecessary iteration is eliminated. This trick is done by combining asSequence and flatMap:
listOfA.asSequence().flatMap { it.objectBs.asSequence() }.find { it.key == wantedKey }
I wrote and ran the following code to ensure that it works as expected:
class PrintSequenceDelegate<out T>(private val wrappedSequence: Sequence<T>) : Sequence<T> by wrappedSequence {
override fun iterator(): Iterator<T> {
val wrappedIterator = wrappedSequence.iterator()
return object : Iterator<T> by wrappedIterator {
override fun next(): T =
wrappedIterator.next().also { println("Retrieving: $it") }
}
}
}
fun <T> Sequence<T>.toPrintDelegate() = PrintSequenceDelegate(this)
fun main() {
val listOfLists = List(3) { i -> List(3) { j -> "$i$j" } }
println("List of lists: $listOfLists")
val found = listOfLists.asSequence().toPrintDelegate().flatMap { it.asSequence().toPrintDelegate() }.find { it == "11" }
println(if (found != null) "Found: $found" else "Not found")
}
Output:
List of lists: [[00, 01, 02], [10, 11, 12], [20, 21, 22]]
Retrieving: [00, 01, 02]
Retrieving: 00
Retrieving: 01
Retrieving: 02
Retrieving: [10, 11, 12]
Retrieving: 10
Retrieving: 11
Found: 11
Thus we see that the elements (12) after the element found in the containing nested list are not iterated, neither are the following nested lists ([20, 21, 22]).
Nothing fancy, but it does the job efficiently:
fun findBWithKey(listOfA: List<ObjectA>, wantedKey: String): ObjectB? {
listOfA.forEach {
it.objectBs.forEach { item ->
if(item.key == wantedKey){
return item
}
}
}
return null
}
I also like to use map and first, but doing the given task efficiently gets unecessary hard using those extension functions.
A simple flatMap does the trick:
listOfA.flatMap { it.objectBs }.first { it.key == wantedKey }
This will basically give you an intermediate List with all of them combined so that you can easily query the first matching one.
I would look in to coroutines or sequences if performance is critical.
You can optimize your code slightly by using firstOrNull on listOfA as well:
listOfA.filterNotNull().firstOrNull { item ->
item.objectBs.firstOrNull { it.key == wantedKey } != null
}
I would do some performance testing to see if this code is causing any issues before making it overly complex.