How to call 2 fluxes in parallel and manipulate on them afterwards - spring-webflux

So I'm new to reactive programming and writing my first program using spring web-flux and I want to call 2 resources in parallel but I couldn't find how.
I've already Implemented the following methods
public Flux<Date> getDatesToExclude(); //fetches dates from external api
public Flux<UserDates> getAvailableUserDates(); //fetches dates from a json file
UserDates DTO
public class UserDates{
private String user;
private List<Date> dates;
//* getters setters *//
}
and I want to implement the following which should
request getDatesToExclude() and getAvailableUserDates() in parallel.
filter getAvailableUserDates() to the specific user.
filter out the dates to exclude.
public Flux<UserDates> getAvailableUserDates(String user);
I tried to chain the methods using zipWith but found it would work on the items 1 by 1 which doesn't seem useful in this case.
Do I have to use completableFuture in this case?

Just use collectList() and also make sure your different flux executions will be executed on different threads. Below is an example that you can easily adapt to your code.
Mono<List<String>> words = Flux.just("sun", "sky", "beach", "water", "sun", "tree")
.delayElements(Duration.ofMillis(100)) // just to slow down emitting, for testing
.filter(user -> user.equals("sun")) // we want to filter only "sun" words
.collectList() // collect all the filtered items to a List
.subscribeOn(Schedulers.parallel()); // when subscription is made, the process with start on parallel scheduler
Mono<List<Integer>> values = Flux.just(1, 2, 6, 1, 20, 31, 4, 1, 3, 21, 44, 67, 1, 90, 87)
.delayElements(Duration.ofMillis(200)) // just to slow down emitting, for testing
.filter(value -> value.equals(1)) // we want to filter only values equal to 1
.collectList() // collect all the filtered items to a List
.subscribeOn(Schedulers.parallel()); // when subscription is made, the process with start on parallel scheduler
Mono.zip(words, values)
.doOnNext(tuple -> {
// do whatever you want with
// tuple.getT1() - list of words results
// tuple.getT2() - list of values results
// I use doOnNext() just for print something, but you should use the operator that depends on your purposes
System.out.println(String.format("Words list size: %s, values list size: %s",
tuple.getT1().size(), tuple.getT2().size())
);
})
.subscribe();
Thread.sleep(10000L);
// just for testing purposes since all the processes after subscribe()
// happen on other threads

Related

MutableList of MutableLists in Kotlin: adding element error

Why I'm getting "java.lang.IndexOutOfBoundsException: Index 0 out of bounds for length 0" while running next code??? :
val totalList = mutableListOf<MutableList<Int>>()
fun main() {
for (i in 0..15) {
for (j in 0..10) {
*some operations and calculations with **var element of type Int***
totalList[i].add(element)
}
}
}
I was thinking that in such case while iterating through 'j' it should add elements to mutableList[i], after this it should start adding elements to mutableList[i + 1] etc.... But instead I am recieving IndexOutOfBoundsException....
val totalList = mutableListOf<MutableList<Int>>()
All this does is create one list which is going to contain MutableList<Int> items. Right now, there's nothing in it (you've supplied no initial elements in the parentheses).
Skip forward a bit, and you do this:
totalList[0].add(element)
You're trying to get the first element of that empty list and add to it. But there is no first element (index 0) because the list is empty (length 0). That's what the error is telling you.
There's lots of ways to handle this - one thing you could do is create your lists up-front:
// create the 16 list items you want to access in the loop
// (the number is the item count, the lambda generates each item)
val totalList = MutableList(16) { mutableListOf<Int>() }
// then refer to that list's properties in your loop (no hardcoded 0..15)
for (i in totalList.indices) {
...
// guaranteed to exist since i is generated from the list's indices
totalList[i].add(element)
}
Or you could do it the way you are now, only using getOrElse to generate the empty list on-demand, when you try to get it but it doesn't exist:
for (i in 0..15) {
for (j in 0..10) {
// if the element at i doesn't exist, create a list instead, but also
// add it to the main list (see below)
totalList.getOrElse(i) {
mutableListOf<Int>().also { totalList.add(it) }
}.add(element)
}
}
Personally I don't really like this, you're using explicit indices but you're adding new list items to the end of the main list. That implicity requires that you're iterating over the list items in order - which you are here, but there's nothing enforcing that. If the order ever changed, it would break.
I'd prefer the first approach - create your structure of lists in advance, then iterate over those and fill them as necessary. Or you might want to consider arrays instead, since you have a fixed collection size you're "completing" by adding items to specific indices
Another approach (that I mentioned in the comments) is to create each list as a whole, complete thing, and then add that to your main list. This is generally how you do things in Kotlin - the standard library contains a lot of functional tools to allow you to chain operations together, transform things, and create immutable collections (which are safer and more explicit about whether they're meant to be changed or they're a fixed set of data).
for (i in 0..15) {
// map transforms each element of the range (each number) to an item,
// resulting in a list of items
val items = (0..10).map { j ->
// do whatever you're doing
// the last expression in the lambda is its resulting value,
// i.e. the item that ends up in the list
element
}
// now you have a complete list of items, add them to totalList
totalList.add(items)
}
(Or you could create the list directly with List(11) { j -> ... } but this is a more general example of transforming a bunch of things to a bunch of other things)
That example there is kinda half and half - you still have the imperative for loop going on as well. Writing it all using the same approach, you can get:
val totalList = (0..15).map { i ->
(0..10).map { j ->
// do stuff
element
}
}
I'd probably prefer the List(count) { i -> ... } approach for this, it's a better fit (this is a general example). That would also be better since you could use MutableList instead of List, if you really need them to be mutable (with the maps you could just chain .toMutableList() after the mapping function, as another step in the chain). Generally in Kotlin, collections are immutable by default, and this kind of approach is how you build them up without having to create a mutable list etc. and add items to it yourself

Flow.take(ITEM_COUNT) returning all the elements rather then specified amount of elements

I've a method X that's getting data from the server via pub sub. This method returns a flow. I've another method that subscribes to the flow by method X but only wants to take the first 3 values max from the flow if the data is distinct compared to previous data. I've written the following code
fun subscribeToData() : Flow<List<MyData>> {
....
//incoming data
emit(list)
}
fun getUptoFirst3Items() {
subscribeToData()
.take(ITEM_COUNT) // ITEM_COUNT is 3
.distinctUntilChange() //only proceed if the data is different from the previous top 3 items
.mapIndex {
//do transformation
}
.collect { transformedListOf3Elements ->
}
}
Problem:
In collect{} I'm not getting 3 elements but rather I'm getting all the data that's coming in the flow.
I'm not sure what's wrong here? Can someone help me?
You have a Flow<List<MyData>> here, which means every element of this flow is itself a list.
The take operator is applied on the flow, so you will take the 3 first lists of the flow. Each individual list is not limited, unless you use take on the list itself.
So the name transformedListOf3Elements is incorrect, because the list is of an unknown number of elements, unless you filter it somehow in the map.
#Joffrey answer already explained why you get the whole list returned and suggested you use take() on the list itself.
If you want to take just the first ITEM_COUNT elements from every list that is emitted/observed, then you have to map the result and only take ITEM_COUNT items from the list each time, instead of taking ITEM_COUNT items from the flow.
fun getUptoFirst3Items() {
subscribeToData()
.map {
// in Kotlin stdlib Iterable<T> has an extension method take(n: Int)
// that will return a List<T> containing the first n element from the iterable
it.take(ITEM_COUNT)
// alternatively you can also use subList, but the semantics are not the same,
// so check the subList documentation, before using it
it.subList(0, ITEM_COUNT)
}
.distinctUntilChange() //only proceed if the data is different from the previous top 3 items
.mapIndex {
//do transformation
}
.collect { transformedListOf3Elements ->
}
}

Why can't I use map on Kotlins Regex Result sequence

I worked with Kotlin's Regex API to get occurences of some regular expression. I wanted to convert the finding directly into another object so I intuitively used map() on the result sequence.
I was very surprised that the map function is never called but forEach is working. This example should make it clear:
val regex = "a.".toRegex()
val txt = "abacad"
var counter = 0
regex.findAll(txt).forEach { counter++ }
println(counter) // 3
regex.findAll(txt).map { counter++ }
println(counter) // still 3 since map is not called
regex.findAll(txt).forEach { counter++ }
println(counter) // 6
My question is why? Did I oversee it in the documentation?
(tested on Kotlin 1.5.30)
findAll() returns a Sequence<MatchResult>. Operations on Sequence are classified either as intermediate or terminal. The documentation for the functions declares which type they are. map and onEach are intermediate. Their action is deferred until a terminal operation is made. forEach is terminal.
Manipulating a Sequence with map returns a new Sequence that will perform the mapping function only when it is actually iterated, such as by a call to forEach or using it in a for loop.
This is the purpose of Sequence, to defer mutating functional calls. It can reduce allocations of intermediate Lists, or in some cases avoid applying the mutations on every single item, such as if the terminal call in the chain is a find() call.

Repeat request multiple times with different params using RxJava

I need to load some data from server page by page until all the data is loaded. The data is considered to be fully loaded if at some point I received fewer items than I've requested. This is the working solution that I have right now:
return Observable.fromCallable { 0 }
.delay(500, TimeUnit.MILLISECONDS)
.repeat()
.scan { previousPage, _ -> previousPage + 1}
.concatMap { doLongFetch(it) }
.takeUntil { it.size < 100 }
fun doLongFetch(page: Int): Observable<List<ListItem>>() {
//Here I do the loading
}
However, there's a problem with the source observable. As you can see, it emits new values every 500 milliseconds to provide some input for the scan function. The delay is required since otherwise, it would emit thousands of values in a very short period of time, which is not required at all. Ideally, I want to remove that delay completely and make sure that the source observable emits another value only after the downstream has handled the previous one (meaning that the data has been requested and processed).
Any ideas on how I can do that?

Spring webflux difference between block, flatmap and subscribe

I have an api which needs to call 3 other apis, the second and third api calls rely on the result of the first.
I'm slightly confused about the best way to do this and the difference between using block, subscribe and flatmap. All 3 of these methods work for me but I am not sure which one is the best one to use.
This is what I currently have:
webClient1.getApi1(request.getId())
.subscribe(api1Response -> {
if (api1Response.hasData()) {
Mono<ApiTwoResponse> monoTwo = webClient2
.post()
.syncBody(...)
.bodyToMono(ApiTwoResponse.class)
monoTwo.subscribe(two -> log.info(two));
Mono<ApiThreeResponse> monoThree = webClient3
.put()
.syncBody(...)
.bodyToMono(ApiThreeResponse.class)
monoThree.subscribe(three -> log.info(three));
}
});
I've also tried block although this seems to be discouraged:
Api1Response response = webClient1.getApi1(request.getId()).block()
and i also tried flatmap although this forces you to return something:
webClient1.getApi1(request.getId())
.flatmap(api1Response -> {
...
return Mono.empty();
});
Any help and feedback on the above code is appreciated.
block operation, stops and waits essentially. It would be the equivalent to Future.get() in java. It defeats the purpose of non-blocking code.
Flatmap flattens a sequence of sequence into a single sequence, so a List {List{?}} will turn into a list{Object}.
subscribe essentially starts to listen, and can perform actions. Usually nothing happens until subscribe.
But for your use case, you can use filter here is an example,
Which looks filters over the {true, false} items, then for each filter that is true,
I zip the results of two mono's together, then subscribe with an action
Flux<Boolean> bool = Flux.just(true, false);
Mono<Integer> mono1 = Mono.just(1);
Mono<String> mono2 = Mono.just("string");
bool.filter(b -> b)
.flatMap(b -> Mono.zip(mono1, mono2))
.subscribe(tuple -> System.out.println(tuple.getT1() + ", " + tuple.getT2()));