I have a main method whose return type WebClient. In this method I get a Mono object and using subscribe I'm trying to call another method which returns webclient object. Now within subscribe, I have webclient object which I want to return. I'm blocked here as I'm not sure how to return the object and where to put the return keyword.
Main method:-
public WebClient getWebClientWithAuthorization(String t) {
-----
----
Mono<AccessToken> accessToken = authenticationProvider.getUserAccessToken(serviceConnectionDetails, queryParams);
Disposable disposable = accessToken.subscribe(
value -> getWebClientBuilder(t, value.getAccessToken()),
error -> error.printStackTrace(),
() -> System.out.println("completed without a value")
);
}
Below getWebClientBuilder method returns webclient object:-
private WebClient getWebClientBuilder(String tenantDomain, String accessToken) {
//TODO Logic for Saving the Token using Caching/Redis mechanism will be taken up from here and implemented in future Sprints
logger.info("Bearer token received: "+ CommerceConnectorConstants.REQUEST_HEADER_AUTHORIZATION_BEARER +" "+ accessToken);
if (null != proxyHost) {
return utilsbuilder.baseUrl(tenantDomain).filter(oauth2Credentials(accessToken)).clientConnector(getHttpConnector()).build();
} else {
return utilsbuilder
.baseUrl(tenantDomain)
.filter(oauth2Credentials(accessToken))
.build();
}
}
Now in getWebClientWithAuthorization method, where to put the return keyword inside subscribe or outside subscribe.
Think "Reactive" end to end
In my opinion, what is the most important when star building application using Reactive Programming is treating any call as asynchronous hence providing end to end asynchronous and non-blocking communication.
Thus, what I suggest you is providing instead of synchronous type a Mono<WebClient> in the following way:
public Mono<WebClient> getWebClientWithAuthorization(String t) {
-----
----
Mono<AccessToken> accessToken = authenticationProvider.getUserAccessToken(serviceConnectionDetails, queryParams);
return accessToken.map(value -> getWebClientBuilder(t, value.getAccessToken()))
.doOnError(error -> error.printStackTrace())
.doOnComplete(() -> System.out.println("completed without a value"))
);
}
So, now you may easily map value to the WebClient's instance and send it to the downstream. In turn, your downstream may react to that value and transform WebClient to the execution of HTTP call as it is shown in the following example:
getWebClientWithAuthorization("some magic string here")
.flatMapMany(webClient -> webClient.get()
.uri("some uri here")
.retrieve()
.bodyToFlux(MessageResponse.class))
// operate with downstream items somehow
// .map
// .filter
// .etc
// and return Mono or Flux again to just continue the flow
And remember, just continue the flow and everywhere specify reactive types if async communication is supposed. There is no sense to subscribe to the source until you met some network boundary or some logical end of the stream where you do not have to return something back.
Related
I have a question about handling deserialization exceptions in Spring Cloud Stream while processing batches (i.e. batch-mode: true).
Per the documentation here, https://docs.spring.io/spring-kafka/docs/2.5.12.RELEASE/reference/html/#error-handling-deserializer, (looking at the implementation of FailedFooProvider), it looks like this function should return a subclass of the original message.
Is the intent here that a list of both Foo's and BadFoo's will end up at the original #StreamListener method, and then it will be up to the code (i.e. me) to sort them out and handle separately? I suspect this is the case, as I've read that the automated DLQ sending isn't desirable for batch error handling, as it would resubmit the whole batch.
And if this is the case, what if there is more than one message type received by the app via different #StreamListener's, say Foo's and Bar's. What type should the value function return in that case? Below is the pseudo code to illustrate the second question?
#StreamListener
public void readFoos(List<Foo> foos) {
List<> badFoos = foos.stream()
.filter(f -> f instanceof BadFoo)
.map(f -> (BadFoo) f)
.collect(Collectors.toList());
// logic
}
#StreamListener
public void readBars(List<Bar> bars) {
// logic
}
// Updated to return Object and let apply() determine subclass
public class FailedFooProvider implements Function<FailedDeserializationInfo, Object> {
#Override
public Object apply(FailedDeserializationInfo info) {
if (info.getTopics().equals("foo-topic") {
return new BadFoo(info);
}
else if (info.getTopics().equals("bar-topic") {
return new BadBar(info);
}
}
}
Yes, the list will contain the function result for failed deserializations; the application needs to handle them.
The function needs to return the same type that would have been returned by a successful deserialization.
You can't use conditions with batch listeners. If the list has a mixture of Foos and Bars, they all go to the same listener.
I have a scenario where I have to fetch some string asynchronously. I would like to create a method where I can listen to when this value is fetched successfully and then provided to the listener.
Now, this can be done easily via many ways including a callback listener or a lambda.
But what do I use so that all subsequent calls to this method, also provide the string back - without having to fetch it again, as it has already been fetched once. So a solution where the listener is still attached but is provided the value right away since it is available.
I know how to do this via old fashioned callback listeners, where the value is stored and then for subsequent calls it can be returned right away via the callback.
But is there a more compact/sophisticated way to do it, let's say via Rx?
Thanks.
I think you can just use cache() operator. It will be something like this:
val value: Single<String> by lazy {
// emulation of your callback
Single.create<String> { emitter ->
Thread.sleep(1000)
println("do some work")
emitter.onSuccess("test")
}.cache()
}
fun main() {
value.subscribe { str -> println(str) }
value.subscribe { str -> println(str) }
value.subscribe { str -> println(str) }
Thread.sleep(2000)
// output:
// do some work
// test
// test
// test
}
I am using spring-webflux 5.1.7-RELEASE. I want to trigger the callback for webclient just before the request is sent. The code for this is:
ExchangeFilterFunction logRequest (SomeLogObject someLogObject) {
return ExchangeFilterFunction.ofRequestProcessor(clientRequest -> {
/**
business logic for callback goes here
*/
return Mono.just(clientRequest);
});
}
//code for plugging in logRequest callback (at some othe place)
WebClient webClient = WebClient
.builder()
.clientConnector(new ReactorClientHttpConnector(HttpClient.from(tcpClient)))
.baseUrl(baseURL)
.filters(exchangeFilterFunctions -> exchangeFilterFunctions.add(logRequest(someLogObject)))
.build();
webClient
.get()
.uri(uriBuilder -> uriBuilder.path("some_uri_path").queryParams(queryParam).build())
.header("some_header_key", "some_header_value")
.retrieve().bodyToMono(String.class);
Here the logRequest is triggered in beginning itself (much before the request is triggered). As per my debugging I found that it gets triggered when retrieve() is called.
Is there a way to ensure that logRequest gets triggered immediately before the request is sent and NOT when the mono is created?
Thanks in advance
I think you are looking for the following change:
ExchangeFilterFunction logRequest (SomeLogObject someLogObject){
return ExchangeFilterFunction.ofRequestProcessor(clientRequest ->
Mono.defer(() -> {
/**
business logic for callback goes here
*/
return Mono.just(clientRequest);
})
);
}
Mono.defer() will delay code execution until the real request.
I am trying to call external service in a micro-service application to get all responses in parallel and combine them before starting the other computation. I know i can use block() call on each Mono object but that will defeat the purpose of using reactive api. is it possible to fire up all requests in parallel and combine them at one point.
Sample code is as below. In this case "Done" prints before actual response comes up. I also know that subscribe call is non blocking.
I want "Done" to be printed after all responses has been collected, so need some kind of blocking. however do not want to block each and every request
final List<Mono<String>> responseOne = new ArrayList<>();
IntStream.range(0, 10).forEach(i -> {
Mono<String> responseMono =
WebClient.create("https://jsonplaceholder.typicode.com/posts")
.post()
.retrieve()
.bodyToMono(String.class)
;
System.out.println("create mono response lazy initialization");
responseOne.add(responseMono);
});
Flux.merge(responseOne).collectList().subscribe( res -> {
System.out.println(res);
});
System.out.println("Done");
Based on the suggestion, I came up with this which seems to work for me.
StopWatch watch = new StopWatch();
watch.start();
final List<Mono<String>> responseOne = new ArrayList<>();
IntStream.range(0, 10).forEach(i -> {
Mono<String> responseMono =
WebClient.create("https://jsonplaceholder.typicode.com/posts")
.post()
.retrieve()
.bodyToMono(String.class);
System.out.println("create mono response lazy initialization");
responseOne.add(responseMono);
});
CompletableFuture<List<String>> futureCount = new CompletableFuture<>();
List<String> res = new ArrayList<>();
Mono.zip(responseOne, Arrays::asList)
.flatMapIterable(objects -> objects) // make flux of objects
.doOnComplete(() -> {
futureCount.complete(res);
}) // will be printed on completion of the flux created above
.subscribe(responseString -> {
res.add((String) responseString);
}
);
watch.stop();
List<String> response = futureCount.get();
System.out.println(response);
// do rest of the computation
System.out.println(watch.getLastTaskTimeMillis());
If you want your calls to be parallel it is a good idea to use Mono.zip
Now, you want Done to be printed after the collection of all the responses
So, you can modify your code as below
final List<Mono<String>> responseMonos = IntStream.range(0, 10).mapToObj(
index -> WebClient.create("https://jsonplaceholder.typicode.com/posts").post().retrieve()
.bodyToMono(String.class)).collect(Collectors.toList()); // create iterable of mono of network calls
Mono.zip(responseMonos, Arrays::asList) // make parallel network calls and collect it to a list
.flatMapIterable(objects -> objects) // make flux of objects
.doOnComplete(() -> System.out.println("Done")) // will be printed on completion of the flux created above
.subscribe(responseString -> System.out.println("responseString = " + responseString)); // subscribe and start emitting values from flux
It's also not a good idea to call subscribe or block explicitly in your reactive code.
is it possible to fire up all requests in parallel and combine them at one point.
That's exactly what your code is doing already. If you don't believe me, stick .delayElement(Duration.ofSeconds(2)) after your bodyToMono() call. You'll see that your list prints out after just over 2 seconds, rather than 20 (which is what it would be if executing sequentially 10 times.)
The combining part is happening in your Flux.merge().collectList() call.
In this case "Done" prints before actual response comes up.
That's to be expected, as your last System.out.println() call is executing outside of the reactive callback chain. If you want "Done" to print after your list is printed (which you've confusingly given the variable name s in the consumer passed to your subscribe() call) then you'll need to put it inside that consumer, not outside it.
If you're interfacing with an imperative API, and you therefore need to block on the list, you can just do:
List<String> list = Flux.merge(responseOne).collectList().block();
...which will still execute the calls in parallel (so still gain you some advantage), but then block until all of them are complete and combined into a list. (If you're just using reactor for this type of usage however, it's debatable if it's worthwhile.)
On the backend im doing:
#PostMapping(path = "/products", consumes = MediaType.APPLICATION_STREAM_JSON_VALUE)
public void saveProducts(#Valid #RequestBody Flux<Product> products) {
products.subscribe(product -> log.info("product: " + product.toString()));
}
And on the frontend im calling this using:
this.targetWebClient
.post()
.uri(productUri)
.accept(MediaType.APPLICATION_STREAM_JSON)
.contentType(MediaType.APPLICATION_STREAM_JSON)
.body(this.sourceWebClient
.get()
.uri(uriBuilder -> uriBuilder.path(this.sourceEndpoint + "/id")
.queryParam("date", date)
.build())
.accept(MediaType.APPLICATION_STREAM_JSON)
.retrieve()
.bodyToFlux(Product.class), Product.class)
.exchange()
.subscribe();
What happens now is that I have 472 products which need to get saved but only one of them is actually saving. The stream closes after the first and I cant find out why.
If I do:
...
.retrieve()
.bodyToMono(Void.class);
instead, the request isnt even arriving at the backend.
I also tried fix amount of elements:
.body(Flux.just(new Product("123"), new Product("321")...
And with that also only the first arrived.
EDIT
I changed the code:
#PostMapping(path = "/products", consumes =
MediaType.APPLICATION_STREAM_JSON_VALUE)
public Mono<Void> saveProducts(#Valid #RequestBody Flux<Product> products) {
products.subscribe(product -> this.service.saveProduct(product));
return Mono.empty();
}
and:
this.targetWebClient
.post()
.uri(productUri)
.accept(MediaType.APPLICATION_STREAM_JSON)
.contentType(MediaType.APPLICATION_STREAM_JSON)
.body(this.sourceWebClient
.get()
.uri(uriBuilder -> uriBuilder.path(this.sourceEndpoint + "/id")
.queryParam("date", date)
.build())
.accept(MediaType.APPLICATION_STREAM_JSON)
.retrieve()
.bodyToFlux(Product.class), Product.class)
.exchange()
.block();
That led to the behaviour that one product was saved twice (because the backend endpoint was called twice) but again only just one item. And also we got an error on the frontend side:
IOException: Connection reset by peer
Same for:
...
.retrieve()
.bodyToMono(Void.class)
.subscribe();
Doing the following:
this.targetWebClient
.post()
.uri(productUri)
.accept(MediaType.APPLICATION_STREAM_JSON)
.contentType(MediaType.APPLICATION_STREAM_JSON)
.body(this.sourceWebClient
.get()
.uri(uriBuilder -> uriBuilder.path(this.sourceEndpoint + "/id")
.queryParam("date", date)
.build())
.accept(MediaType.APPLICATION_STREAM_JSON)
.retrieve()
.bodyToFlux(Product.class), Product.class)
.retrieve();
Leads to the behaviour that the backend again isnt called at all.
The Reactor documentation does say that nothing happens until you subscribe, but it doesn't mean you should subscribe in your Spring WebFlux code.
Here are a few rules you should follow in Spring WebFlux:
If you need to do something in a reactive fashion, the return type of your method should be Mono or Flux
Within a method returning a reactive typoe, you should never call block or subscribe, toIterable, or any other method that doesn't return a reactive type itself
You should never do I/O-related in side-effects DoOnXYZ operators, as they're not meant for that and this will cause issues at runtime
In your case, your backend should use a reactive repository to save your data and should look like:
#PostMapping(path = "/products", consumes = MediaType.APPLICATION_STREAM_JSON_VALUE)
public Mono<Void> saveProducts(#Valid #RequestBody Flux<Product> products) {
return productRepository.saveAll(products).then();
}
In this case, the Mono<Void> return type means that your controller won't return anything as a response body but will signal still when it's done processing the request. This might explain why you're seeing that behavior - by the time the controller is done processing the request, all products are not saved in the database.
Also, remember the rules noted above. Depending on where your targetWebClient is used, calling .subscribe(); on it might not be the solution. If it's a test method that returns void, you might want to call block on it and get the result to test assertions on it. If this is a component method, then you should probably return a Publisher type as a return value.
EDIT:
#PostMapping(path = "/products", consumes =
MediaType.APPLICATION_STREAM_JSON_VALUE)
public Mono<Void> saveProducts(#Valid #RequestBody Flux<Product> products) {
products.subscribe(product -> this.service.saveProduct(product));
return Mono.empty();
}
Doing this isn't right:
calling subscribe decouples the processing of the request/response from that saveProduct operation. It's like starting that processing in a different executor.
returning Mono.empty() signals Spring WebFlux that you're done right away with the request processing. So Spring WebFlux will close and clean the request/response resources; but your saveProduct process is still running and won't be able to read from the request since Spring WebFlux closed and cleaned it.
As suggested in the comments, you can wrap blocking operations with Reactor (even though it's not advised and you may encounter performance issues) and make sure that you're connecting all the operations in a single reactive pipeline.