I encountered a case where I have a nested Flux. I don't care about the individual results of the inner flux as it returns Unit (in Kotlin / Void in Java), but I want to know if the Flux aborted due to an error or not. I thought I could use the then function, as the doc states: Error signal is replayed in the resulting Mono<V>
My problem can be reduced to the minimum (Kotlin) unit test:
#Test
fun fluxTest() {
val flux = Flux.just("willFail", "willSucceed")
.flatMap { outer ->
// In my real world example the inner flux is created via Flux.fromIterable from a property of the
// outer`-object
Flux.just(1)
.flatMap { inner ->
// this simulates a Mono.fromSupplier that can throw exceptions
if (outer == "willFail") Mono.error<Unit>(RuntimeException("bam"))
else Mono.just(Unit)
}
// We don't care about the Flux as it returns Unit/Void
// All we want to know is, whether there was an error or not
.then(Mono.just(outer))
}
.onErrorContinue { error, item -> println("$item => $error") }
.collectList()
StepVerifier.create(flux)
.expectNextMatches { it.size == 1 }
.verifyComplete()
}
So we have 2 elements. In the inner Flux one of the elements will fail on processing and the other won't. I expect the error to propagate through the pipeline where it is catched and discarded in the onErrorContinue.
Therefore I'd expect 1 element in the resulting list, but I get the original 2. I have no clue why.
Now comes the fun part: In this particular test case, I can replace Flux.just(1) with Mono.just(1) (in my real world case this doesn't work ofc because the flux has more than 1 element) and suddenly my test passes:
#Test
fun fluxTest() {
val flux = Flux.just("willFail", "willSucceed")
.flatMap { outer ->
// In my real world example the inner flux is created via Flux.fromIterable from a property of the
// outer`-object
Mono.just(1)
.flatMap { inner ->
// this simulates a Mono.fromSupplier that can throw exceptions
if (outer == "willFail") Mono.error<Unit>(RuntimeException("bam"))
else Mono.just(Unit)
}
// We don't care about the Flux as it returns Unit/Void
// All we want to know is, whether there was an error or not
.then(Mono.just(outer))
}
.onErrorContinue { error, item -> println("$item => $error") }
.collectList()
StepVerifier.create(flux)
.expectNextMatches { it.size == 1 }
.verifyComplete()
}
So obviously there is a difference in Mono.then(Mono<T>) and in Flux.then(Mono<T>), but it shouldn't since the Javadoc is the same right?
Side note: Instead of Flux.then(Mono.just(outer)) I also tried Mono.defer but that is not changing anything.
Related
At 'urichecking2' log, I can see there is value. But in 'uriChecking' the uriList is null.
why the uriList.add not work??
private fun getPhotoList() {
val fileName = intent.getStringExtra("fileName")
Log.d("fileNameChecking", "$fileName")
val listRef = FirebaseStorage.getInstance().reference.child("image").child(fileName!!)
var tmpUrl:Uri = Uri.parse(fileName)
Log.d("firstTmpUri","$tmpUrl")
listRef.listAll()
.addOnSuccessListener { listResult ->
for (item in listResult.items) {
item.downloadUrl.addOnCompleteListener { task ->
if (task.isSuccessful) {
tmpUrl = task.result
Log.d("secondTmpUri","$tmpUrl")
Log.d("urichecking2","$task.result")
uriList.add(task.result)
} else {
}
}.addOnFailureListener {
// Uh-oh, an error occurred!
}
}
}
Log.d("thirdTmpUri","$tmpUrl")
Log.d("urichecking", "$uriList")
}
If I do this, the log is output in the order of first, third, and second, and the desired value is in second, but when third comes out, it returns to the value of first.
The listAll method (like most cloud APIs these days, including downloadUrl which you also use) is asynchronous, since it needs to make a call to the server - which may take time. This means the code executes in a different order than you may expect, which is easiest to see if you add some logging:
Log.d("Firebase","Before starting listAll")
listRef.listAll()
.addOnSuccessListener { listResult ->
Log.d("Firebase","Got listResult")
}
Log.d("Firebase","After starting listAll")
When you run this code it outputs:
Before starting listAll
After starting listAll
Got listResult
This is probably not the order you expected, but it perfectly explains why you can't see the list result. By the time your Log.d("urichecking", "$uriList") runs, none of the uriList.add(task.result) has been called yet.
The solution for this is always the same: any code that needs the list result, has to be inside the addOnCompleteListener callback, be called from there, or be otherwise synchronized.
So in its simplest way:
listRef.listAll()
.addOnSuccessListener { listResult ->
for (item in listResult.items) {
item.downloadUrl.addOnCompleteListener { task ->
if (task.isSuccessful) {
uriList.add(task.result)
Log.d("urichecking", "$uriList")
}
}
}
}
This is an incredibly common mistake to make if you're new to programming with asynchronous APIs, so I recommend checking out
Asynchronous programming techniques in the Kotlin language guide
How to get URL from Firebase Storage getDownloadURL
Can someone help me with logic of the firebase on success listener
Why does my function that calls an API or launches a coroutine return an empty or null value?
You know that Array and List only store the same data struction.
I run the Code A and get the Result A.
It seems that the Flow can emit both Int value and String value, why?
Code A
import kotlinx.coroutines.*
import kotlinx.coroutines.flow.*
suspend fun performRequest(request: Int): Int {
delay(1000) // imitate long-running asynchronous work
return request
}
fun main() = runBlocking<Unit> {
(1..3).asFlow() // a flow of requests
.transform { request ->
emit("Making request $request")
if (request >1) {
emit(performRequest(request))
}
}
.collect { response -> println(response) }
}
Result A
Making request 1
Making request 2
2
Making request 3
3
This is not a question of Flow but Java/Kotling generics and type safety.
The type this flow returns is Comperable<*>
val flow: Flow<Comparable<*>> = (1..3).asFlow() // a flow of requests
.transform { request ->
emit("Making request $request")
if (request > 1) {
emit(performRequest(request))
}
If you explicitly specify which value you want to return Flow you can restrict the types.
About generics you can refer here or check any document about generics in java/kotlin, type safety you can refer this question
Also when you are in doubt what your specified type is use alt + enter with Android Studio to see avaialble options and select Specify type explicitly.
Disregarding the nature of this request, you can have the functionality you want by making your flow emit instances of some algebraic data type that is basically a "sum" (from the type-theoretic POV) of your constituent types:
sealed interface Record
data class IntData(val get: Int) : Record
data class Metadata(val get: String) : Record
// somewhere later (flow is of type Flow<Record>)
fun main() = runBlocking<Unit> {
(1..3).asFlow() // a flow of requests
.transform { request ->
emit(Metadata("Making request $request"))
if (request > 1) {
emit(IntData(performRequest(request)))
}
// probably want to handle the `else` case too
}
.collect { response -> println(response) }
}
This would be a good solution since it's extendable (i.e. you can add the other cases later on if you need to).
In your specific case though, since you just want to debug the flow, you might not want to actually emit the "metadata" and just go for the tests of your code directly.
Let's consider this function
#Transactional
fun conditionalInsertEntity(dbEntity: DBEntity): Mono<DBEntity> {
return fetchObjectByPublicId(dbEntity.publicId)
.switchIfEmpty {
r2DatabaseClient.insert()
.into(DBEntity::class.java)
.using(Flux.just(dbEntity))
.fetch()
.one()
.map { it["entity_id"] as Long }
.flatMap { fetchObjectById(it) }
}
}
while running above function with following driver code I get duplicate entry errors if the list contains duplicates. Ideally it shouldn't give that error because the above function is already handling the case for duplicate inserts!!
val result = Flux.fromIterable(listOf(dbEntity1, dbEntity1, dbEntity2))
.flatMap { conditionalInsertEntity(it) }
.collectList()
.block()
Realized that this is an issue of using flatMap instead of concatMap.
ConcatMap collects the result from individual publishers sequentially unlike flatMap. (more here)
Because I used flatMap, multiple publishers thought that the entity isn't already available in the DB
Could you please explain what exactly happens in Flux/Mono returned by HttpClient.response() ? I thought value generated by http client will NOT be passed downstream until Mono completes but I see that tons of requests are generated which ends up with reactor.netty.internal.shaded.reactor.pool.PoolAcquirePendingLimitException: Pending acquire queue has reached its maximum size of 8 exception. It works as expected (items being processed one by one) if I replace call to testRequest() with Mono.fromCallable { }.
What am I missing ?
Test code:
import org.asynchttpclient.netty.util.ByteBufUtils
import reactor.core.publisher.Flux
import reactor.core.publisher.Mono
import reactor.netty.http.client.HttpClient
import reactor.netty.resources.ConnectionProvider
class Test {
private val client = HttpClient.create(ConnectionProvider.create("meh", 4))
fun main() {
Flux.fromIterable(0..99)
.flatMap { obj ->
println("Creating request for: $obj")
testRequest()
.doOnError { ex ->
println("Failed request for: $obj")
ex.printStackTrace()
}
.map { res ->
obj to res
}
}
.doOnNext { (obj, res) ->
println("Created request for: $obj ${res.length} characters")
}
.collectList().block()!!
}
fun testRequest(): Mono<String> {
return client.get()
.uri("https://projectreactor.io/docs/netty/release/reference/index.html#_connection_pool")
.responseContent()
.reduce(StringBuilder(), { sb, buf ->
val str= ByteBufUtils.byteBuf2String(Charsets.UTF_8, buf)
sb.append(str)
})
.map { it.toString() }
}
}
When you create the ConnectionProvider like this ConnectionProvider.create("meh", 4), this means connection pool with max connections 4 and max pending requests 8. See here more about this.
When you use flatMap this means Transform the elements emitted by this Flux asynchronously into Publishers, then flatten these inner publishers into a single Flux through merging, which allow them to interleave See here more about this.
So what happens is that you are trying to run all requests simultaneously.
So you have two options:
If you want to use flatMap then increase the number of the pending requests.
If you want to keep the number of the pending requests you may consider for example using concatMap instead of flatMap, which means Transform the elements emitted by this Flux asynchronously into Publishers, then flatten these inner publishers into a single Flux, sequentially and preserving order using concatenation. See more here about this.
Goal: I want to repeatedly call a Retrofit service (GET) that returns paged data, until I've exhausted its pages. Going from page 0 to page n.
First, I've looked at these two answers already. The first actually works, but I'm not overly fond of the recursive solution as it could lead to stack overflow. The second fails the moment you try to use a scheduler.
Here's a sample of the second:
Observable.range(0, 5/*Integer.MAX_VALUE*/) // generates page values
.subscribeOn(Schedulers.io()) // need this to prevent UI hanging
// gamesService uses Schedulers.io() by default
.flatMapSingle { page -> gamesService.getGames(page) }
.takeWhile { games -> games.isNotEmpty() } // games is a List<Game>
.subscribe(
{ games -> db.insertAll(games) },
{ Logger.e(TAG, it, "Error getting daily games: ${it.message}") }
)
What I expect this to do is stop the moment that gamesService.getGames(page) returns an empty list. Instead, it continues hitting the endpoint for an indeterminate number of times, with incrementing page values. I have experimented a bit in unit tests with Single.just(intVal) and determined that the problem appears to be the fact that my service is automatically subscribed on Schedulers.io(). This is how I define my Retrofit services:
private inline fun <reified T> createService(okClient: OkHttpClient): T {
val rxAdapter = RxJava2CallAdapterFactory.createWithScheduler(Schedulers.io())
val retrofit = Retrofit.Builder()
.baseUrl(config.apiEndpoint.endpoint())
.client(okClient)
.addCallAdapterFactory(rxAdapter)
.addConverterFactory(moshiConverterFactory())
.build()
return retrofit.create(T::class.java)
}
It's really not an option to not use createWithScheduler() here.
Here's another idea I tried:
val atomic = AtomicInteger(0)
Observable.generate<Int> { it.onNext(atomic.getAndIncrement()) }
.subscribeOn(Schedulers.io())
.flatMapSingle { page -> gamesService.getGames(page) }
.takeWhile { games -> games.isNotEmpty() }
.subscribe(
{ games -> dailyGamesDao.insertAll(games) },
{ Logger.e(TAG, it, "Error getting daily games: ${it.message}") }
)
This is another case where it worked as expected right up until I introduced a Scheduler. The generator generates way too many values, when I'm expecting it to stop when the takeWhile discovers an empty list.
I've also tried various kinds of concat (concatWith, concatMap, etc).
At this point, I'm really just looking for someone to help me correct the obvious (to them) and completely basic misunderstanding I clearly have with RxJava operators.
I have found a partial solution. (I may edit this answer later if and when I find my "final" solution.)
tl;dr I should convert my Singles to Observables and use the flatMap overload that takes a maxConcurrency parameter. For example:
Observable.range(0, SOME_SUFFICIENTLY_LARGE_NUMBER)
.subscribeOn(Schedulers.io())
.flatMap({ page -> gamesService.getGames(page).toObservable }, 1 /* maxConcurrency */)
.takeWhile { games -> games.isNotEmpty() }
.subscribe(
{ games -> dailyGamesDao.insertAll(games) },
{ Logger.e(TAG, it, "Error getting daily games: ${it.message}") }
)
That basically does it. By limiting the number of concurrent threads to 1, I now have the "one after the other" behavior I was seeking. The only thing I don't like about this, and I suppose it's a minor gripe, is that my base Observable.range() can still emit a lot of values -- way more than ever get used by the downstream Singles/Observables.
PS: One reason I couldn't find this solution earlier is I was using RxJava 2.1.9. When I pushed it to 2.1.14, I had access to the new overloads. Oh well.