I need to load some data from server page by page until all the data is loaded. The data is considered to be fully loaded if at some point I received fewer items than I've requested. This is the working solution that I have right now:
return Observable.fromCallable { 0 }
.delay(500, TimeUnit.MILLISECONDS)
.repeat()
.scan { previousPage, _ -> previousPage + 1}
.concatMap { doLongFetch(it) }
.takeUntil { it.size < 100 }
fun doLongFetch(page: Int): Observable<List<ListItem>>() {
//Here I do the loading
}
However, there's a problem with the source observable. As you can see, it emits new values every 500 milliseconds to provide some input for the scan function. The delay is required since otherwise, it would emit thousands of values in a very short period of time, which is not required at all. Ideally, I want to remove that delay completely and make sure that the source observable emits another value only after the downstream has handled the previous one (meaning that the data has been requested and processed).
Any ideas on how I can do that?
Related
I've a method X that's getting data from the server via pub sub. This method returns a flow. I've another method that subscribes to the flow by method X but only wants to take the first 3 values max from the flow if the data is distinct compared to previous data. I've written the following code
fun subscribeToData() : Flow<List<MyData>> {
....
//incoming data
emit(list)
}
fun getUptoFirst3Items() {
subscribeToData()
.take(ITEM_COUNT) // ITEM_COUNT is 3
.distinctUntilChange() //only proceed if the data is different from the previous top 3 items
.mapIndex {
//do transformation
}
.collect { transformedListOf3Elements ->
}
}
Problem:
In collect{} I'm not getting 3 elements but rather I'm getting all the data that's coming in the flow.
I'm not sure what's wrong here? Can someone help me?
You have a Flow<List<MyData>> here, which means every element of this flow is itself a list.
The take operator is applied on the flow, so you will take the 3 first lists of the flow. Each individual list is not limited, unless you use take on the list itself.
So the name transformedListOf3Elements is incorrect, because the list is of an unknown number of elements, unless you filter it somehow in the map.
#Joffrey answer already explained why you get the whole list returned and suggested you use take() on the list itself.
If you want to take just the first ITEM_COUNT elements from every list that is emitted/observed, then you have to map the result and only take ITEM_COUNT items from the list each time, instead of taking ITEM_COUNT items from the flow.
fun getUptoFirst3Items() {
subscribeToData()
.map {
// in Kotlin stdlib Iterable<T> has an extension method take(n: Int)
// that will return a List<T> containing the first n element from the iterable
it.take(ITEM_COUNT)
// alternatively you can also use subList, but the semantics are not the same,
// so check the subList documentation, before using it
it.subList(0, ITEM_COUNT)
}
.distinctUntilChange() //only proceed if the data is different from the previous top 3 items
.mapIndex {
//do transformation
}
.collect { transformedListOf3Elements ->
}
}
Implemented Caching by the following link:
https://blog.mindorks.com/implement-caching-in-android-using-rxjava-operators
fun getSavedAddressList(): Maybe<List<SavedAddress>?>? {
return Observable.concat(
getAddressListMemory(),
getAddressListDatabase(),
getAddressListNetwork()).firstElement()
}
fun getAddressListDatabase(): Observable<List<SavedAddress>?> {
return Observable.create(ObservableOnSubscribe { emitter: ObservableEmitter<List<SavedAddress>?> ->
val list: ArrayList<SavedAddress> = addressDao.getAddressList() as ArrayList<SavedAddress>
if (list.isNotEmpty()) {
emitter.onNext(list)
}
emitter.onComplete()
if (list.isNotEmpty())
getAddressListNetwork().subscribeOn(schedulerProvider.io())?.observeOn(schedulerProvider.io())?.subscribe()
})
}
items in the database are retrieving perfectly after storing into database
problem is network calling is not happening after getting a list from database
I want to get three data source sequentially one after another and store latest data in the database
First of all, you're leaking the getAddressListNetwork Disposable in there because you are trying to do too much inside the getAddressListDatabase.
I think what you want is this:
fun getSavedAddressList(): Observable<List<SavedAddress>> {
return Observable.concat(
getAddressListMemory(),
getAddressListDatabase(),
getAddressListNetwork()).distinctUntilChanged())
}
This will always try to fetch the addresses from the 3 sources, and only emitting if the data is different than the previous emission, meaning the data is "fresher".
To be honest with you, I think you need to have a look at the concept of "stale data" and "cache invalidation".
I have a never ending stream as a sequence.
What I am aiming for is to take a batch from the sequence both based on time and size.
What I mean is if my sequence has 2250 messages right now I want to send 3 batches ( 1000, 1000, 250).
Also if till the next 5 minute I still have not accumulated a 1000 messages I will send it anyway with whatever has accumulated so far.
sequence
.chunked(1000)
.map { chunk ->
// do something with chunk
}
What I was expecting to have is something like .chunked(1000, 300) which 300 is second for when I want to send every 5 minutes.
Thanks in advance
Kotlin Sequence is a synchronous concept and is not supposed to be used in any kind of time-limited fashion. If you ask the sequence for the next element then it blocks invoker thread until it produces the next element and there is no way to cancel it.
However, kotlinx.coroutines library introduces the concept of Channel which is a rough analogue of a sequence for an asynchronous world, where operation may take some time to complete and they don't block threads while doing so. You can read more in this guide.
It does not provide a ready-to-use chunked operator, but makes it straightforward to write one. You can use the following code:
import kotlinx.coroutines.experimental.channels.*
import kotlinx.coroutines.experimental.selects.*
fun <T> ReceiveChannel<T>.chunked(size: Int, time: Long) =
produce<List<T>>(onCompletion = consumes()) {
while (true) { // this loop goes over each chunk
val chunk = mutableListOf<T>() // current chunk
val ticker = ticker(time) // time-limit for this chunk
try {
whileSelect {
ticker.onReceive {
false // done with chunk when timer ticks, takes priority over received elements
}
this#chunked.onReceive {
chunk += it
chunk.size < size // continue whileSelect if chunk is not full
}
}
} catch (e: ClosedReceiveChannelException) {
return#produce // that is normal exception when the source channel is over -- just stop
} finally {
ticker.cancel() // release ticker (we don't need it anymore as we wait for the first tick only)
if (chunk.isNotEmpty()) send(chunk) // send non-empty chunk on exit from whileSelect
}
}
}
As you can see from this code, it embeds some non-trivial decisions on what to do in corner cases. What should we do if timer expires but current chunk is still empty? This code start new time interval and does not send the previous (empty) chunk. Do we finish current chunk on timeout after last element, measure time from the first element, or measure time from the beginning of chunk? This code does the later.
This code is completely sequential -- its logic is easy to follow in a step-by-step way (there is not concurrency inside the code). One can adjust it to any project-speicfic requirements.
Goal: I want to repeatedly call a Retrofit service (GET) that returns paged data, until I've exhausted its pages. Going from page 0 to page n.
First, I've looked at these two answers already. The first actually works, but I'm not overly fond of the recursive solution as it could lead to stack overflow. The second fails the moment you try to use a scheduler.
Here's a sample of the second:
Observable.range(0, 5/*Integer.MAX_VALUE*/) // generates page values
.subscribeOn(Schedulers.io()) // need this to prevent UI hanging
// gamesService uses Schedulers.io() by default
.flatMapSingle { page -> gamesService.getGames(page) }
.takeWhile { games -> games.isNotEmpty() } // games is a List<Game>
.subscribe(
{ games -> db.insertAll(games) },
{ Logger.e(TAG, it, "Error getting daily games: ${it.message}") }
)
What I expect this to do is stop the moment that gamesService.getGames(page) returns an empty list. Instead, it continues hitting the endpoint for an indeterminate number of times, with incrementing page values. I have experimented a bit in unit tests with Single.just(intVal) and determined that the problem appears to be the fact that my service is automatically subscribed on Schedulers.io(). This is how I define my Retrofit services:
private inline fun <reified T> createService(okClient: OkHttpClient): T {
val rxAdapter = RxJava2CallAdapterFactory.createWithScheduler(Schedulers.io())
val retrofit = Retrofit.Builder()
.baseUrl(config.apiEndpoint.endpoint())
.client(okClient)
.addCallAdapterFactory(rxAdapter)
.addConverterFactory(moshiConverterFactory())
.build()
return retrofit.create(T::class.java)
}
It's really not an option to not use createWithScheduler() here.
Here's another idea I tried:
val atomic = AtomicInteger(0)
Observable.generate<Int> { it.onNext(atomic.getAndIncrement()) }
.subscribeOn(Schedulers.io())
.flatMapSingle { page -> gamesService.getGames(page) }
.takeWhile { games -> games.isNotEmpty() }
.subscribe(
{ games -> dailyGamesDao.insertAll(games) },
{ Logger.e(TAG, it, "Error getting daily games: ${it.message}") }
)
This is another case where it worked as expected right up until I introduced a Scheduler. The generator generates way too many values, when I'm expecting it to stop when the takeWhile discovers an empty list.
I've also tried various kinds of concat (concatWith, concatMap, etc).
At this point, I'm really just looking for someone to help me correct the obvious (to them) and completely basic misunderstanding I clearly have with RxJava operators.
I have found a partial solution. (I may edit this answer later if and when I find my "final" solution.)
tl;dr I should convert my Singles to Observables and use the flatMap overload that takes a maxConcurrency parameter. For example:
Observable.range(0, SOME_SUFFICIENTLY_LARGE_NUMBER)
.subscribeOn(Schedulers.io())
.flatMap({ page -> gamesService.getGames(page).toObservable }, 1 /* maxConcurrency */)
.takeWhile { games -> games.isNotEmpty() }
.subscribe(
{ games -> dailyGamesDao.insertAll(games) },
{ Logger.e(TAG, it, "Error getting daily games: ${it.message}") }
)
That basically does it. By limiting the number of concurrent threads to 1, I now have the "one after the other" behavior I was seeking. The only thing I don't like about this, and I suppose it's a minor gripe, is that my base Observable.range() can still emit a lot of values -- way more than ever get used by the downstream Singles/Observables.
PS: One reason I couldn't find this solution earlier is I was using RxJava 2.1.9. When I pushed it to 2.1.14, I had access to the new overloads. Oh well.
I have an disk storage that returns an Object? from disk (Could be already saved or not) and an BehaviourSubject (This data comes from other class call, check code below):
Code is:
private val subject: Subject<Optional<Element>> = BehaviorSubject.create()
fun getElements(): Observable<List<Element>> =
Observable.concat(Observable.just(storage.getElement()), subject)
.filter({ it.isPresent })
.take(1)
.flatMapSingle {
Observable.just(it.get())
.flatMapIterable { it.categories }
.toList()
}
fun updateSubject(response: Response) {
storage.save(response.element) //Save element in storage
subject.onNext(response.element.toOptional())
}
My problem is, in other class I do
getElements().subscribe(onElements(), onError());
First time, when storage has null it does nothing, even I've got a breakpoint in subject.onNext(response.element.toOptional()), hoping that onNext will trigger a stream for getElements, but nothing happens.
Second time, when I've already saved in storage the received element (So, storage.getElement() returns something) it works fine.
My functional description is:
Get element from both cache and subject, take first that arrives, and return it (First time it will be who the comes subject one), next time, i'm hoping that first one will be the storage one.
I am assuming that storage is some sort of persistence object so it so storage.getElement() might return a valid object at the time you create the subject?
If that is the case, then I think you should check to see if you have a stored object before you create the subject, if so use BehaviorSubject.createDefalut(storedObject) if it does exist or BehaviorSubject.create() if not.
Then inside your getElements() function I think you can just use subject.filter() ....
Your null optional elements are being filtered out by your .filter { it.isPresent } call so nothing gets emitted downstream.