I have a situation where i am using flatmap to transform a sequencial third party webservices calls (first get1 and second get2) to prepare a flux of responses (get2) to process with another method in subscribe.
Flux<String> max = Flux.fromIterable(markets)
.flatMap(market -> apiService.get1(market, value, entry))
.flatMap(response -> {
String maxValue = mapper.readTree(response).get("MaxValue").asText();
return apiService.get2(maxValue);
});
this works but when i subscribe the stream flux i am interest in send the market value that used on first flatmap to relay as a parameter method of processMaxValues()
max.log().subscribe(response -> processMaxValues(response, market));
What is the best way to implment that ?
Related
Every time I think I understand Webflux and project reactor, I find out I have no idea.
So I making some API calls... I want to call 1 first ... Get information back use that information, to make subsequent calls.
so I do this like so
public Mono<ResponseObject> createAggregatedRecords(RecordToPersist recordToPersist){
return randomApiClient.createRecord(recordToPersist)
.flatMap(result -> {
return Mono.zip(
webClientInstance.createOtherRecord1(result.getChildRecord1()),
webClientInstance2.createOtherRecord2(result.getChildRecord2()),
webClientInstance3.createOtherRecord3(result.getChildRecord3()))
.map(tupple -> {
ResponseObject respObj = new ResponseObject();
respObj.setChildResult1(tupple.getT1());
respObj.setChildResult2(tupple.getT2());
respObj.setChildResult3(tupple.getT3());
return respObj;
}
}).doOnSuccess(res -> log.info("This throws an error: {}", res.getChildResult1.getFirstField()))
}
Now, for some reason, I am returning a null object with this very code to my Controller and I am not printing out the object in Json.
I suspect it is because I am nesting the Mono.zip inside the flatmap, and am not returning the results back correctly. I am making all of those API calls though as my End-to-End integration tests are succeeding.
Now I thought that I would return that response object from the .map function from the Mono.zip chain and then return that to the flatMap call in the chain. If I put observers on the chain like a doOnSuccess and print out response object fields I get a null pointer ... Not sure what I am missing
Is this a good pattern to achieve that goal? Or should I try a different path?
Why can I not get the response Object to return?
I'm very new to reactive programming.
My project based on Spring WebFlux.
Here is my merge-method:
//fn1 and fn2 are JS functions with one int param, count - I execute them with param from 0 to count.
public Flux<String> generateUnordered(String fn1, String fn2, int count) {
Flux<String> fn1Flux = generateFunctionResultFlux(fn1, count, FUNCTION_1)
.map(fnResult->String.format(UNORDERED_OUTPUT,fnResult[0],fnResult[1],fnResult[2],fnResult[3]));
Flux<String> fn2Flux = generateFunctionResultFlux(fn2, count, FUNCTION_2)
.map(fnResult->String.format(UNORDERED_OUTPUT,fnResult[0],fnResult[1],fnResult[2],fnResult[3]));
return Flux.merge(fn1Flux, fn2Flux)
.delayElements(Duration.ofMillis(DELAY));
}
when I execute it with JS functions as below and count=2
function fn1(number) {return number +100;}
function fn2(number) {return number +200;}
I get something like that :
0,FUNCTION_2,200.0,0
0,FUNCTION_1,200.0,0
But there is clearly seen that I somehow get fn2 result in fn1 row!
What am I doeing wrong and how can I fix it?
GitHub link on project:WebFlux project
GitHub link on class: FluxGenerator class
With merge operator sources are subscribed eagerly. So I guess first flux will emit as soon as subscribed and the second one will be. Because they emit immediately. If you want to see interleaved flux you'll need to delay your emmision.
Flux<Integer> fluxOfIntegers = Flux.merge(
evenNumbers.delayElements(Duration.ofMillis(500L)),
oddNumbers.delayElements(Duration.ofMillis(300L)));
StepVerifier.create(fluxOfIntegers)
.expectNext(1)
.expectNext(2)
.expectNext(3)
.expectNext(5)
.expectNext(4)
.expectComplete()
.verify();
If you want more examples of operators you can check this
The problem was in other part of programm.
To calculate JS functions I used class field ScriptEngine that is NOT thread-safe.
Making ScriptEngine local field solved my problem.
I'm trying to execute a list requests using WebClient, then filter them finding the first one that succeed (if any) and return that. Or fall back to a default response if non succeeded.
The problem I'm facing is that when I call .collectList() on a Flux<ServerResponse>, the list is always empty. I would have expected the list to contain N number of ServerResponse based on the number of requests I issued earlier.
public Mono<ServerResponse> retry(ServerRequest request) {
return Flux.fromIterable(request.headers().header(SEQUENCE_HEADER_NAME))
.map(URI::create)
// Build a "list" of responses
.flatMap(uri -> webClientBuilder.baseUrl(uri.toString()).build()
.method(Objects.requireNonNull(request.method()))
.headers(headers -> request.headers().asHttpHeaders().forEach((key, values) -> {
if (!SEQUENCE_HEADER_NAME.equals(key)) {
headers.addAll(key, values);
}
}))
.body(BodyInserters.fromDataBuffers(request.body(BodyExtractors.toDataBuffers())))
.exchange()
.flatMap(clientResponse -> ServerResponse.status(clientResponse.statusCode())
.headers(headers -> headers.addAll(clientResponse.headers().asHttpHeaders()))
.body(BodyInserters.fromDataBuffers(clientResponse.body(BodyExtractors.toDataBuffers()))))
)
// "Wait" for all of them to complete so we can filter
.collectList()
.flatMap(clientResponses -> {
List<ServerResponse> filteredResponses = clientResponses.stream()
.filter(response -> response.statusCode().is2xxSuccessful())
.collect(Collectors.toList());
if (filteredResponses.isEmpty()) {
log.error("No request succeeded; defaulting to {}", HttpStatus.BAD_REQUEST.toString());
return ServerResponse.badRequest().build();
}
if (filteredResponses.size() > 1) {
log.error("Multiple requests succeeded; defaulting to {}", HttpStatus.BAD_REQUEST.toString());
return ServerResponse.badRequest().build();
}
return Mono.just(filteredResponses.get(0));
});
}
Any ideas why .collectList() always returns an empty list?
Well, it seems to me you have a confused requirement in that you want the First Mono that responds but you are trying to put that functionality into a Flux which is meant to process all items in the flow efficiently. Mono in Webflux is meant to create a flow that will perform a series of transformations on the item in the flow efficiently. Nothing in your requirement of testing a bunch of URIs for the first one that succeeds is what WebFlux is good for so I have to question why try to force that into the framework.
You might argue that a Flux is giving you better asynchronous processing but I don't think that's the case when it is a bunch of WebClient calls. WebClient is still HTTP under the hood and so each item in the flow stops and starts around WebClient. If you want to do HTTP asynchronously you should use a ThreadPool and Callable.
I am trying to call external service in a micro-service application to get all responses in parallel and combine them before starting the other computation. I know i can use block() call on each Mono object but that will defeat the purpose of using reactive api. is it possible to fire up all requests in parallel and combine them at one point.
Sample code is as below. In this case "Done" prints before actual response comes up. I also know that subscribe call is non blocking.
I want "Done" to be printed after all responses has been collected, so need some kind of blocking. however do not want to block each and every request
final List<Mono<String>> responseOne = new ArrayList<>();
IntStream.range(0, 10).forEach(i -> {
Mono<String> responseMono =
WebClient.create("https://jsonplaceholder.typicode.com/posts")
.post()
.retrieve()
.bodyToMono(String.class)
;
System.out.println("create mono response lazy initialization");
responseOne.add(responseMono);
});
Flux.merge(responseOne).collectList().subscribe( res -> {
System.out.println(res);
});
System.out.println("Done");
Based on the suggestion, I came up with this which seems to work for me.
StopWatch watch = new StopWatch();
watch.start();
final List<Mono<String>> responseOne = new ArrayList<>();
IntStream.range(0, 10).forEach(i -> {
Mono<String> responseMono =
WebClient.create("https://jsonplaceholder.typicode.com/posts")
.post()
.retrieve()
.bodyToMono(String.class);
System.out.println("create mono response lazy initialization");
responseOne.add(responseMono);
});
CompletableFuture<List<String>> futureCount = new CompletableFuture<>();
List<String> res = new ArrayList<>();
Mono.zip(responseOne, Arrays::asList)
.flatMapIterable(objects -> objects) // make flux of objects
.doOnComplete(() -> {
futureCount.complete(res);
}) // will be printed on completion of the flux created above
.subscribe(responseString -> {
res.add((String) responseString);
}
);
watch.stop();
List<String> response = futureCount.get();
System.out.println(response);
// do rest of the computation
System.out.println(watch.getLastTaskTimeMillis());
If you want your calls to be parallel it is a good idea to use Mono.zip
Now, you want Done to be printed after the collection of all the responses
So, you can modify your code as below
final List<Mono<String>> responseMonos = IntStream.range(0, 10).mapToObj(
index -> WebClient.create("https://jsonplaceholder.typicode.com/posts").post().retrieve()
.bodyToMono(String.class)).collect(Collectors.toList()); // create iterable of mono of network calls
Mono.zip(responseMonos, Arrays::asList) // make parallel network calls and collect it to a list
.flatMapIterable(objects -> objects) // make flux of objects
.doOnComplete(() -> System.out.println("Done")) // will be printed on completion of the flux created above
.subscribe(responseString -> System.out.println("responseString = " + responseString)); // subscribe and start emitting values from flux
It's also not a good idea to call subscribe or block explicitly in your reactive code.
is it possible to fire up all requests in parallel and combine them at one point.
That's exactly what your code is doing already. If you don't believe me, stick .delayElement(Duration.ofSeconds(2)) after your bodyToMono() call. You'll see that your list prints out after just over 2 seconds, rather than 20 (which is what it would be if executing sequentially 10 times.)
The combining part is happening in your Flux.merge().collectList() call.
In this case "Done" prints before actual response comes up.
That's to be expected, as your last System.out.println() call is executing outside of the reactive callback chain. If you want "Done" to print after your list is printed (which you've confusingly given the variable name s in the consumer passed to your subscribe() call) then you'll need to put it inside that consumer, not outside it.
If you're interfacing with an imperative API, and you therefore need to block on the list, you can just do:
List<String> list = Flux.merge(responseOne).collectList().block();
...which will still execute the calls in parallel (so still gain you some advantage), but then block until all of them are complete and combined into a list. (If you're just using reactor for this type of usage however, it's debatable if it's worthwhile.)
I have an api which needs to call 3 other apis, the second and third api calls rely on the result of the first.
I'm slightly confused about the best way to do this and the difference between using block, subscribe and flatmap. All 3 of these methods work for me but I am not sure which one is the best one to use.
This is what I currently have:
webClient1.getApi1(request.getId())
.subscribe(api1Response -> {
if (api1Response.hasData()) {
Mono<ApiTwoResponse> monoTwo = webClient2
.post()
.syncBody(...)
.bodyToMono(ApiTwoResponse.class)
monoTwo.subscribe(two -> log.info(two));
Mono<ApiThreeResponse> monoThree = webClient3
.put()
.syncBody(...)
.bodyToMono(ApiThreeResponse.class)
monoThree.subscribe(three -> log.info(three));
}
});
I've also tried block although this seems to be discouraged:
Api1Response response = webClient1.getApi1(request.getId()).block()
and i also tried flatmap although this forces you to return something:
webClient1.getApi1(request.getId())
.flatmap(api1Response -> {
...
return Mono.empty();
});
Any help and feedback on the above code is appreciated.
block operation, stops and waits essentially. It would be the equivalent to Future.get() in java. It defeats the purpose of non-blocking code.
Flatmap flattens a sequence of sequence into a single sequence, so a List {List{?}} will turn into a list{Object}.
subscribe essentially starts to listen, and can perform actions. Usually nothing happens until subscribe.
But for your use case, you can use filter here is an example,
Which looks filters over the {true, false} items, then for each filter that is true,
I zip the results of two mono's together, then subscribe with an action
Flux<Boolean> bool = Flux.just(true, false);
Mono<Integer> mono1 = Mono.just(1);
Mono<String> mono2 = Mono.just("string");
bool.filter(b -> b)
.flatMap(b -> Mono.zip(mono1, mono2))
.subscribe(tuple -> System.out.println(tuple.getT1() + ", " + tuple.getT2()));