Tell Flux to emit next item after async processing - spring-webflux

Question

I'm using Project Reactor with Webflux to try to read data from a message queue, then process it in chunks (eg, five at a time) and make a request to an API with each chunk. The API does not work well with high throughput, so I need to have control over how many requests are sent concurrently.
Basically, I'd like to have a WebClient call finish, then be able to tell the Flux that we're ready to process more.
I was using this code to try to emulate the desired functionality, and I'm getting results that I don't understand:
fun main() {
val subscriber = CustomSubscriber()
Flux.create<Int> { sink ->
sink.onRequest {
sink.next(1)
}
}
.doOnNext {
println("hit first next with $it")
}
.delayElements(Duration.ofSeconds(1)) // Mock WebClient call
.doOnNext {
println("before request")
subscriber.request(1)
println("after request")
}
.subscribeWith(subscriber)
Thread.sleep(10000)
}
class CustomSubscriber : BaseSubscriber<Int>() {
override fun hookOnSubscribe(subscription: Subscription) {
subscription.request(1)
}
}
The output of this code is
hit first next with 1
before request
after request
What I was hoping for is this:
hit first next with 1 // one second passes
before request
after request
hit first next with 1 // one second passes
before request
after request
hit first next with 1 // one second passes
before request
after request
hit first next with 1 // one second passes
before request
after request
(Infinite loop)
So the request method is called, but the number is never emitted.
Oddly, when I call request in a separate Flux, I'm getting the desired behavior:
fun main() {
val subscriber = CustomSubscriber()
Flux.create<Int> { sink ->
sink.onRequest {
sink.next(1)
}
}
.doOnNext {
println("hit first next with $it")
}
.subscribeWith(subscriber)
Flux.range(0, 5)
.delayElements(Duration.ofSeconds(3))
.doOnNext { subscriber.request(1) }
.subscribe()
Thread.sleep(10000)
}
class CustomSubscriber : BaseSubscriber<Int>() {
override fun hookOnSubscribe(subscription: Subscription) {
subscription.request(1)
}
}
So it seems like there is an issue with calling the request method in the doOnNext method of the original Flux?
I'm not married to the idea of using a FluxSink, that just seemed like a way to have more explicit control of the data emission.

Answer 1

I think what you are looking for is custom subscriber, which consumes data at its own pace based on some logic. Something like this.
Flux.range(0, 14)
.subscribeWith(object : Subscriber<Int> {
private var count = 0
lateinit var subscription: Subscription
override fun onSubscribe(s: Subscription) {
subscription = s
s.request(2)
}
override fun onNext(parameter: Int) {
println("Before request")
// ----- some processing
println("After request")
count++
if (count >= 2) {
println("Requesting more......")
count = 0
subscription.request(2)
}
}
override fun onError(t: Throwable) {}
override fun onComplete() {
println("Done")
}
})