Continue zip(), if one source is completed - spring-webflux

Question

I have some troubles with .zip() operator.
Let me simplify my problem on a small example.
Flux<Integer> flux1 = Flux.just(9, 8, 3, -2);
Flux<Integer> flux2 = Flux.just(7);
Flux<Integer> flux3 = Flux.just(6, 5, 4, -4);
List<Flux<Integer>> list1 = Arrays.asList(flux1, flux2, flux3);
TreeSet<Integer> set = new TreeSet<>(Comparator.reverseOrder());
Set<Integer> list = Flux.zip(list1, objects -> {
boolean setChanged = false;
for (Object o : objects) {
Integer i = (Integer) o;
if (set.size() < 5 || i > set.last()) {
setChanged = true;
set.add(i);
if (set.size() > 5) {
set.pollLast();
}
}
}
return setChanged;
}).takeWhile(val -> val)
.then(Mono.just(set))
.block();
System.out.println(list);
Here I have 3 different sources(they are sorted descending by default, and the number of them could be much bigger), and I want to get from them a collection of 5 elements sorted descending. Unfortunately, I can't just use concat() or merge() operators, because sources in a real life can be really big, but I need only small amount of elements.
I am expecting [9, 8, 7, 6, 5] here, but one of the sources is completed after first iteration of zipping.
Could you please suggest how I can get around with this problem?

Answer 1

You can try the reduce operation
#Test
void test() {
Flux<Integer> flux1 = Flux.just(9, 8, 3, -2);
Flux<Integer> flux2 = Flux.just(7, 0, -2, 4,3,2,2,1);
Flux<Integer> flux3 = Flux.just(6, 5, 4, -4);
var k = 5;
List<Flux<Integer>> publishers = List.of(flux1, flux2, flux3);
var flux = Flux.merge(publishers)
.log()
.limitRate(2)
.buffer(2)
.reduce((l1, l2) -> {
System.out.println(l1);
System.out.println(l2);
return Stream.concat(
l1.stream(),
l2.stream()
).sorted(Comparator.reverseOrder())
.limit(k)
.collect(Collectors.toList());
})
.log();
StepVerifier.create(flux)
.expectNext(List.of(9,8,7,6,5))
.expectComplete()
.verify();
}
You can fetch data in chunks and compare them to find the top K elements.
In a sequential case it will fetch a new batch, compare it to the current top k result and return a new topk like in the example above (PriorityQueue may work better for sorting if k is big).
If you're using parallel schedulers and batches are fetched in parallel, then it can compare them with each other independently that should be a bit faster.
Also you have full control over the fetched data via rateLimit, buffer, delayElements, etc