I have an api which needs to call 3 other apis, the second and third api calls rely on the result of the first.
I'm slightly confused about the best way to do this and the difference between using block, subscribe and flatmap. All 3 of these methods work for me but I am not sure which one is the best one to use.
This is what I currently have:
webClient1.getApi1(request.getId())
.subscribe(api1Response -> {
if (api1Response.hasData()) {
Mono<ApiTwoResponse> monoTwo = webClient2
.post()
.syncBody(...)
.bodyToMono(ApiTwoResponse.class)
monoTwo.subscribe(two -> log.info(two));
Mono<ApiThreeResponse> monoThree = webClient3
.put()
.syncBody(...)
.bodyToMono(ApiThreeResponse.class)
monoThree.subscribe(three -> log.info(three));
}
});
I've also tried block although this seems to be discouraged:
Api1Response response = webClient1.getApi1(request.getId()).block()
and i also tried flatmap although this forces you to return something:
webClient1.getApi1(request.getId())
.flatmap(api1Response -> {
...
return Mono.empty();
});
Any help and feedback on the above code is appreciated.
block operation, stops and waits essentially. It would be the equivalent to Future.get() in java. It defeats the purpose of non-blocking code.
Flatmap flattens a sequence of sequence into a single sequence, so a List {List{?}} will turn into a list{Object}.
subscribe essentially starts to listen, and can perform actions. Usually nothing happens until subscribe.
But for your use case, you can use filter here is an example,
Which looks filters over the {true, false} items, then for each filter that is true,
I zip the results of two mono's together, then subscribe with an action
Flux<Boolean> bool = Flux.just(true, false);
Mono<Integer> mono1 = Mono.just(1);
Mono<String> mono2 = Mono.just("string");
bool.filter(b -> b)
.flatMap(b -> Mono.zip(mono1, mono2))
.subscribe(tuple -> System.out.println(tuple.getT1() + ", " + tuple.getT2()));
Related
I need to iterate 100's of ids in parallel and collect the result in list. I am trying to do it in following way
val context = newFixedThreadPoolContext(5, "custom pool")
val list = mutableListOf<String>()
ids.map {
val result:Deferred<String> = async(context) {
getResult(it)
}
//list.add(result.await()
}.mapNotNull(result -> list.add(result.await())
I am getting error at
mapNotNull(result -> list.add(result.await())
as await method is not available. Why await is not applicable at this place? Instead commented line
//list.add(result.await()
is working fine.
What is the best way to run this block in parallel using coroutine with custom thread pool?
Generally, you go in the right direction: you need to create a list of Deferred and then await() on them.
If this is exactly the code you are using then you did not return anything from your first map { } block, so you don't get a List<Deferred> as you expect, but List<Unit> (list of nothing). Just remove val result:Deferred<String> = - this way you won't assign result to a variable, but return it from the lambda. Also, there are two syntactic errors in the last line: you used () instead of {} and there is a missing closing parenthesis.
After these changes I believe your code will work, but still, it is pretty weird. You seem to mix two distinct approaches to transform a collection into another. One is using higher-order functions like map() and another is using a loop and adding to a list. You use both of them at the same time. I think the following code should do exactly what you need (thanks #Joffrey for improving it):
val list = ids.map {
async(context) {
getResult(it)
}
}.awaitAll().filterNotNull()
I'm actually confused on assembly time and subscription time. I know mono's are lazy and does not get executed until its subscribed. Below is a method.
public Mono<UserbaseEntityResponse> getUserbaseDetailsForEntityId(String id) {
GroupRequest request = ImmutableGroupRequest
.builder()
.cloudId(id)
.build();
Mono<List<GroupResponse>> response = ussClient.getGroups(request);
List<UserbaseEntityResponse.GroupPrincipal> groups = new CopyOnWriteArrayList<>();
response.flatMapIterable(elem -> elem)
.toIterable().iterator().forEachRemaining(
groupResponse -> {
groupResponse.getResources().iterator().forEachRemaining(
resource -> {
groups.add(ImmutableGroupPrincipal
.builder()
.groupId(resource.getId())
.name(resource.getDisplayName())
.addAllUsers(convertMemebersToUsers(resource))
.build());
}
);
}
);
log.debug("Response Object - " + groups.toString());
ImmutableUserbaseEntityResponse res = ImmutableUserbaseEntityResponse
.builder()
.userbaseId(id)
.addAllGroups(groups)
.build();
Flux<UserbaseEntityResponse.GroupPrincipal> f = Flux.fromIterable(res.getGroups())
.parallel()
.runOn(Schedulers.parallel())
.doOnNext(groupPrincipal -> getResourcesForGroup((ImmutableGroupPrincipal)groupPrincipal, res.getUserbaseId()))
.sequential();
return Mono.just(res);
}
This gets executed Mono<List<GroupResponse>> response = ussClient.getGroups(request); without calling subscribe, however below will not get executed unless I call subscribe on that.
Flux<UserbaseEntityResponse.GroupPrincipal> f = Flux.fromIterable(res.getGroups())
.parallel()
.runOn(Schedulers.parallel())
.doOnNext(groupPrincipal -> getResourcesForGroup((ImmutableGroupPrincipal)groupPrincipal, res.getUserbaseId()))
.sequential();
Can I get some more input on assembly time vs subscription?
"Nothing happens until you subscribe" isn't quite true in all cases. There's three scenarios in which a publisher (Mono or Flux) will be executed:
You subscribe;
You block;
The publisher is "hot".
Note that the above scenarios all apply to an entire reactive chain - i.e. if I subscribe to a publisher, everything upstream (dependent on that publisher) also executes. That's why frameworks can, and should call subscribe when they need to, causing the reactive chain defined in a controller to execute.
In your case it's actually the second of these - you're blocking, which is essentially a "subscribe and wait for the result(s)". Usually the methods that block are clearly labelled, but again that's not always the case - in your case it's the toIterable() method on Flux doing the blocking:
Transform this Flux into a lazy Iterable blocking on Iterator.next() calls.
But ah, you say, I'm not calling Iterator.next() - what gives?!
Well, implicitly you are by calling forEachRemaining():
The default implementation behaves as if:
while (hasNext())
action.accept(next());
...and as per the above rule, since ussClient.getGroups(request) is upstream of this blocking call, it gets executed.
I'm trying to execute a list requests using WebClient, then filter them finding the first one that succeed (if any) and return that. Or fall back to a default response if non succeeded.
The problem I'm facing is that when I call .collectList() on a Flux<ServerResponse>, the list is always empty. I would have expected the list to contain N number of ServerResponse based on the number of requests I issued earlier.
public Mono<ServerResponse> retry(ServerRequest request) {
return Flux.fromIterable(request.headers().header(SEQUENCE_HEADER_NAME))
.map(URI::create)
// Build a "list" of responses
.flatMap(uri -> webClientBuilder.baseUrl(uri.toString()).build()
.method(Objects.requireNonNull(request.method()))
.headers(headers -> request.headers().asHttpHeaders().forEach((key, values) -> {
if (!SEQUENCE_HEADER_NAME.equals(key)) {
headers.addAll(key, values);
}
}))
.body(BodyInserters.fromDataBuffers(request.body(BodyExtractors.toDataBuffers())))
.exchange()
.flatMap(clientResponse -> ServerResponse.status(clientResponse.statusCode())
.headers(headers -> headers.addAll(clientResponse.headers().asHttpHeaders()))
.body(BodyInserters.fromDataBuffers(clientResponse.body(BodyExtractors.toDataBuffers()))))
)
// "Wait" for all of them to complete so we can filter
.collectList()
.flatMap(clientResponses -> {
List<ServerResponse> filteredResponses = clientResponses.stream()
.filter(response -> response.statusCode().is2xxSuccessful())
.collect(Collectors.toList());
if (filteredResponses.isEmpty()) {
log.error("No request succeeded; defaulting to {}", HttpStatus.BAD_REQUEST.toString());
return ServerResponse.badRequest().build();
}
if (filteredResponses.size() > 1) {
log.error("Multiple requests succeeded; defaulting to {}", HttpStatus.BAD_REQUEST.toString());
return ServerResponse.badRequest().build();
}
return Mono.just(filteredResponses.get(0));
});
}
Any ideas why .collectList() always returns an empty list?
Well, it seems to me you have a confused requirement in that you want the First Mono that responds but you are trying to put that functionality into a Flux which is meant to process all items in the flow efficiently. Mono in Webflux is meant to create a flow that will perform a series of transformations on the item in the flow efficiently. Nothing in your requirement of testing a bunch of URIs for the first one that succeeds is what WebFlux is good for so I have to question why try to force that into the framework.
You might argue that a Flux is giving you better asynchronous processing but I don't think that's the case when it is a bunch of WebClient calls. WebClient is still HTTP under the hood and so each item in the flow stops and starts around WebClient. If you want to do HTTP asynchronously you should use a ThreadPool and Callable.
I am trying to call external service in a micro-service application to get all responses in parallel and combine them before starting the other computation. I know i can use block() call on each Mono object but that will defeat the purpose of using reactive api. is it possible to fire up all requests in parallel and combine them at one point.
Sample code is as below. In this case "Done" prints before actual response comes up. I also know that subscribe call is non blocking.
I want "Done" to be printed after all responses has been collected, so need some kind of blocking. however do not want to block each and every request
final List<Mono<String>> responseOne = new ArrayList<>();
IntStream.range(0, 10).forEach(i -> {
Mono<String> responseMono =
WebClient.create("https://jsonplaceholder.typicode.com/posts")
.post()
.retrieve()
.bodyToMono(String.class)
;
System.out.println("create mono response lazy initialization");
responseOne.add(responseMono);
});
Flux.merge(responseOne).collectList().subscribe( res -> {
System.out.println(res);
});
System.out.println("Done");
Based on the suggestion, I came up with this which seems to work for me.
StopWatch watch = new StopWatch();
watch.start();
final List<Mono<String>> responseOne = new ArrayList<>();
IntStream.range(0, 10).forEach(i -> {
Mono<String> responseMono =
WebClient.create("https://jsonplaceholder.typicode.com/posts")
.post()
.retrieve()
.bodyToMono(String.class);
System.out.println("create mono response lazy initialization");
responseOne.add(responseMono);
});
CompletableFuture<List<String>> futureCount = new CompletableFuture<>();
List<String> res = new ArrayList<>();
Mono.zip(responseOne, Arrays::asList)
.flatMapIterable(objects -> objects) // make flux of objects
.doOnComplete(() -> {
futureCount.complete(res);
}) // will be printed on completion of the flux created above
.subscribe(responseString -> {
res.add((String) responseString);
}
);
watch.stop();
List<String> response = futureCount.get();
System.out.println(response);
// do rest of the computation
System.out.println(watch.getLastTaskTimeMillis());
If you want your calls to be parallel it is a good idea to use Mono.zip
Now, you want Done to be printed after the collection of all the responses
So, you can modify your code as below
final List<Mono<String>> responseMonos = IntStream.range(0, 10).mapToObj(
index -> WebClient.create("https://jsonplaceholder.typicode.com/posts").post().retrieve()
.bodyToMono(String.class)).collect(Collectors.toList()); // create iterable of mono of network calls
Mono.zip(responseMonos, Arrays::asList) // make parallel network calls and collect it to a list
.flatMapIterable(objects -> objects) // make flux of objects
.doOnComplete(() -> System.out.println("Done")) // will be printed on completion of the flux created above
.subscribe(responseString -> System.out.println("responseString = " + responseString)); // subscribe and start emitting values from flux
It's also not a good idea to call subscribe or block explicitly in your reactive code.
is it possible to fire up all requests in parallel and combine them at one point.
That's exactly what your code is doing already. If you don't believe me, stick .delayElement(Duration.ofSeconds(2)) after your bodyToMono() call. You'll see that your list prints out after just over 2 seconds, rather than 20 (which is what it would be if executing sequentially 10 times.)
The combining part is happening in your Flux.merge().collectList() call.
In this case "Done" prints before actual response comes up.
That's to be expected, as your last System.out.println() call is executing outside of the reactive callback chain. If you want "Done" to print after your list is printed (which you've confusingly given the variable name s in the consumer passed to your subscribe() call) then you'll need to put it inside that consumer, not outside it.
If you're interfacing with an imperative API, and you therefore need to block on the list, you can just do:
List<String> list = Flux.merge(responseOne).collectList().block();
...which will still execute the calls in parallel (so still gain you some advantage), but then block until all of them are complete and combined into a list. (If you're just using reactor for this type of usage however, it's debatable if it's worthwhile.)
I'm using Kotlin and Arrow and the WebClient from spring-webflux. What I'd like to do is to transform a Mono instance to an Either.
The Either instance is created by calling Either.right(..) when the response of the WebClient is successful or Either.left(..) when WebClient returns an error.
What I'm looking for is a method in Mono similar to Either.fold(..) where I can map over the successful and erroneous result and return a different type than a Mono. Something like this (pseudo-code which doesn't work):
val either : Either<Throwable, ClientResponse> =
webClient().post().exchange()
.fold({ throwable -> Either.left(throwable) },
{ response -> Either.right(response)})
How should one go about?
There is no fold method on Mono but you can achieve the same using two methods: map and onErrorResume. It would go something like this:
val either : Either<Throwable, ClientResponse> =
webClient().post()
.exchange()
.map { Either.right(it) }
.onErrorResume { Either.left(it).toMono() }
I'm not really familiar with that Arrow library nor the typical use case for it, so I'll use Java snippets to make my point here.
First I'd like first to point that this type seems to be blocking and not lazy (unlike Mono). Translating a Mono to that type means that you'll make your code blocking and that you shouldn't do that, for example, in the middle of a Controller handler or you will block your whole server.
This is more or less the equivalent of this:
Mono<ClientResponse> response = webClient.get().uri("/").exchange();
// blocking; return the response or throws an exception
ClientResponse blockingResponse = response.block();
That being said, I think you should be able to convert a Mono to that type by either calling block() on it and a try/catch block around it, or turning it first into a CompletableFuture first, like:
Mono<ClientResponse> response = webClient.get().uri("/").exchange();
Either<Throwable, ClientResponse> either = response
.toFuture()
.handle((resp, t) -> Either.fold(t, resp))
.get();
There might be better ways to do that (especially with inline functions), but they all should involve blocking on the Mono in the first place.