This is the content type of the request
I use spring boot and this code to redirect the stream from the web server to the browser.
public Flux<String> getDataStream(String configType, String configId, String user){
Flux<String> response = null;
Optional<String> url = getLogURL(configId);
if(url.isPresent()){
logger.info("#WebClient initiation");
WebClient webClient = WebClient.create();
response = webClient.get()
.uri(url.get())
.accept(MediaType.ALL)
.retrieve()
.bodyToFlux(String.class);
}
logger.info("#WebClient response");
return response;
}
After making the request, I can see #WebClient initiation and #WebClient response in a log file.
This is the client-side JavaScript code statement, that hits the above code getDataStream()
let eventSource = new EventSource(STREAM_URL);
But it doesn't get any response from the server. I couldn't figure out what I am missing. The URL that getDataStream()->WebClient hits, confirmed to produces output.
There are several options to stream data from the server using WebFlux:
Server-sent events pushing individual events (media type: text/event-stream)
Streaming events separated by newlines (media type: application/x-ndjson)
If you need plain text content - use text/event-stream. Check WebFlux async responses from controllers for a complete example.
To see response results you need to log it as a part of the reactive flow. Also, there is no null in reactive and you need to return Flux.empty() instead.
#GetMapping(produces = TEXT_EVENT_STREAM_VALUE)
public Flux<String> getDataStream(String configType, String configId, String user){
Optional<String> url = getLogURL(configId);
if (url.isEmpty()) {
return Flux.empty();
}
log.info("#WebClient initiation");
WebClient webClient = WebClient.create();
return webClient.get()
.uri(url.get())
.accept(MediaType.ALL)
.retrieve()
.bodyToFlux(String.class)
.doOnSubscribe(s -> log.info("#WebClient request"))
.doOnNext(rec -> log.info("#WebClient response record: {}", rec));
}
Related
Hi I am trying with Spring WebFlux and reactor.
What I want to achieve is to aggregate some requests into one call to a third party API. And then return the results. My code looks like this:
#RestController
#RequiredArgsConstructor
#RequestMapping(value = "/aggregate", produces = MediaType.APPLICATION_JSON_VALUE)
public class AggregationController {
private final WebClient externalApiClient;
//sink that stores the query params
private Sinks.Many<String> sink = Sinks.many().multicast().onBackpressureBuffer();
#GetMapping
public Mono<Mono<Map<String, BigDecimal>>> aggregate(#RequestParam(value = "pricing", required = false) List<String> countryCodes) {
//store the query params in the sink
countryCodes
.forEach(sink::tryEmitNext);
return Mono.from(
sink.asFlux()
.distinct()
.bufferTimeout(5, Duration.ofSeconds(5))
)
.map(countries -> String.join(",", countries))
.map(concatenatedCountries -> invokeExternalPricingApi(concatenatedCountries))
.doOnSuccess(s -> sink.emitComplete((signalType, emitResult) -> emitResult.isSuccess()));
}
private Mono<Map<String, BigDecimal>> invokeExternalPricingApi(String countryCodes) {
return externalApiClient.get().uri(uriBuilder -> uriBuilder
.path("/pricing")
.queryParam("q", countryCodes)
.build())
.accept(MediaType.APPLICATION_JSON)
.retrieve()
.bodyToMono(new ParameterizedTypeReference<>() {
});
}
}
I have the Sinks.Many<String> sink to accumulate the query params from the callers of the API. When there are 5 items or 5 seconds expired, I want to call the third party API. The issue is that when I do two requests like:
GET localhost:8081/aggregate?pricing=CH,EU
GET localhost:8081/aggregate?pricing=AU,NZ,PT
On one request I get the response and on the other I get null. Also, after the first interaction the system stops working as if the sink was broken.
What am I missing here?
Before the deprecation of WebClient.exchange method I used to get ClientResponse body as Flux<DataBuffer> and manipulated it.
In Spring 5.3 the exchange() method is deprecated and I would like to change the implementation as recommended:
#deprecated since 5.3 due to the possibility to leak memory and/or
connections; please,
use {#link #exchangeToMono(Function)}, {#link #exchangeToFlux(Function)};
consider also using {#link #retrieve()} ...
Tried to do the same call in the lambda passed to exchangeToMono, but clientResponse.bodyToFlux(DataBuffer::class.java) always return an empty flux; other experiments (i.e. getting body as mono string) also could not help to get the body.
What is the standard way to get the ClientResponse body in Spring 5.3 ?
I am looking for a low-level body representation: something like "data buffer", "byte array" or "input stream"; to avoid any kind of parsing/deserialisation.
Before Spring 5.3:
webClient
.method(GET)
.uri("http://somewhere.com")
.exchange()
.flatMap { clientResponse ->
val bodyRaw: Flux<DataBuffer> = clientResponse.bodyToFlux(DataBuffer::class.java)
// ^ body as expected
// other operations
}
After Spring 5.3
webClient
.method(GET)
.uri("http://somewhere.com")
.exchangeToMono { clientResponse ->
val bodyRaw: Flux<DataBuffer> = clientResponse.bodyToFlux(DataBuffer::class.java)
// ^ always empty flux
// other operations
}
The new exchangeToMono and exchangeToFlux methods expect the body to be decoded inside the callback. Check this GitHub issue for details.
Looking at your example, probably you could use retrieve, that is safer alternative, with bodyToFlux
webClient
.method(GET)
.uri("http://somewhere.com")
.retrieve()
.bodyToFlux(DataBuffer.class)
or toEntityFlux if you need access to response details like headers and status
webClient
.method(GET)
.uri("http://somewhere.com")
.retrieve()
.toEntityFlux(DataBuffer.class)
Handling errors
Option 1. Use onErrorResume and handle WebClientResponseException
webClient
.method(GET)
.uri("http://somewhere.com")
.retrieve()
.bodyToFlux(DataBuffer.class)
.onErrorResume(WebClientResponseException.class, ex -> {
if (ex.getStatusCode().equals(HttpStatus.NOT_FOUND)) {
// ignore 404 and return empty
return Mono.empty();
}
return Mono.error(ex);
});
Option 2. Use onStatus convenience method to get access to response.
webClient
.method(GET)
.uri("http://somewhere.com")
.retrieve()
.onStatus(status -> status.equals(HttpStatus.NOT_FOUND), res -> {
// ignore 404 and return empty
return Mono.empty();
})
.bodyToFlux(DataBuffer.class)
Both methods could be used to deserialize error response Getting the response body in error case with Spring WebClient
I'd like to create a Spring WebClient that ignores a specific HTTP error. From the documentation of WebClient.retrieve():
By default, 4xx and 5xx responses result in a WebClientResponseException. To customize error handling, use ResponseSpec.onStatus(Predicate, Function) handlers.
I want all calls through a WebClient instance to ignore the specific HTTP error. That is why onStatus() is of no use to me (it has to be set per response).
The best I could come up with is this:
WebClient webClient = WebClient.builder().filter((request, next) -> {
Mono<ClientResponse> response = next.exchange(request);
response = response.onErrorResume(WebClientResponseException.class, ex -> {
return ex.getRawStatusCode() == 418 ? Mono.empty() : Mono.error(ex);
});
return response;
}).build();
URI uri = UriComponentsBuilder.fromUriString("https://httpstat.us/418").build().toUri();
webClient.get().uri(uri).retrieve().toBodilessEntity().block();
but it does throw the exception instead of ignoring it (the lambda passed to onErrorResume() is never called).
Edited: fixed the mistake pointed out by the first answer.
After extensive debugging of spring-webflux 5.3.4 and with the help of some ideas by Martin Tarjányi, I've come to this as the only possible "solution":
WebClient webClient = WebClient.builder().filter((request, next) -> {
return next.exchange(request).flatMap(res -> {
if (res.rawStatusCode() == HttpStatus.I_AM_A_TEAPOT.value()) {
res = res.mutate().rawStatusCode(299).build();
}
return Mono.just(res);
});
}).build();
URI uri = UriComponentsBuilder.fromUriString("https://httpstat.us/418").build().toUri();
String body = webClient.get().uri(uri).retrieve().toEntity(String.class).block().getBody();
The background: I am migrating some code from RestTemplate to WebClient. The old code looks like this:
RestTemplate restTemplate = ...;
restTemplate.setErrorHandler(new DefaultResponseErrorHandler() {
#Override
public void handleError(ClientHttpResponse response) throws IOException {
if (response.getRawStatusCode() == HttpStatus.I_AM_A_TEAPOT.value()) {
return;
}
super.handleError(response);
}
});
URI uri = UriComponentsBuilder.fromUriString("https://httpstat.us/418").build().toUri();
String body = restTemplate.getForEntity(uri, String.class).getBody();
I believe it is a straightforward and common case.
WebClient is not yet a 100% replacement for RestTemplate.
UPDATE: Turns out this answer doesn't address the core problem of filtering out a specific status code, just addresses a general coding pattern.
The reason onErrorResume lambda is not called is that response.onErrorResume creates a brand new Mono and your code does not use the result (i.e. it's not assigned to the response variable), so in the end a Mono without the onErrorResume operator is returned.
Using Project Reactor it's usually a good practice to avoid declaring local Mono and Flux variables and use a single chain instead. This helps to avoid similar subtle bugs.
WebClient webClient = WebClient.builder()
.filter((request, next) -> next.exchange(request)
.onErrorResume(WebClientResponseException.class, ex -> ex.getRawStatusCode() == 418 ? Mono.empty() : Mono.error(ex)))
.build();
I am trying to call a twitter endpoint that gives you a constant streams of json results back to the client
https://documenter.getpostman.com/view/9956214/T1LMiT5U#977c147d-0462-4553-adfa-d7a1fe59c3ec
I try to make a call to the endpoint like this
val url = "https://api.twitter.com/2/tweets/search/stream"
_streamChannel = _client.get<ByteReadChannel>(token, url) //Stops here
val byteBufferSize = 1024
val byteBuffer = ByteArray(byteBufferSize)
_streamChannel?.let {
while (_streamChannel!!.availableForRead > 0) {
_streamChannel!!.readAvailable(byteBuffer, 0, byteBufferSize)
val s = String(byteBuffer)
parseStreamResponseString(s).forEach {
emit(Response.Success(it))
}
}
}
my client.get code is this
suspend inline fun <reified T> get(authKey: String, url: String): T? {
val response = _client.get<HttpResponse>(url) {
header("Authorization", "Bearer $authKey")
}
when (response.status.value) {
in 300..399 -> throw RedirectResponseException(response)
in 400..499 -> throw ClientRequestException(response)
in 500..599 -> throw ServerResponseException(response)
}
if (response.status.value >= 600) {
throw ResponseException(response)
}
return response.receive<T>()
}
When I make the request it just sits there in what I am assuming is waiting for the full response to be returned before giving it to me.
Edit
I also tried using scoped streaming but it just sits at the line readAvailable I know there are messages coming through because when I run the request via cURL I am constantly getting data
_client.get<HttpStatement> {
header("Authorization", "Bearer $authKey")
url(urlString)
contentType(ContentType.Application.Json)
method = HttpMethod.Get
}.execute {
val streamChannel = it.receive<ByteReadChannel>()
val byteBufferSize = 1024
val byteBuffer = ByteArray(byteBufferSize)
streamChannel.readAvailable(byteBuffer, 0, byteBufferSize) // Stops here
val s = String(byteBuffer)
}
How do I process a constant stream of json data using Ktor?
As far as I am aware, the Ktor client does note expose access to the IO buffer of the request in the way that twitter's streaming API requires.
From the twitter documentation here:
Some HTTP client libraries only return the response body after the connection has been closed by the server. These clients will not work for accessing the Streaming API. You must use an HTTP client that will return response data incrementally. Most robust HTTP client libraries will provide this functionality. The Apache HttpClient will handle this use case, for example.
What you are doing is telling Ktor that the thing you are getting is a ByteReadChannel, and so, once the request closes (which will never happen with this twitter endpoint) the Ktor client would attempt to use whatever plugin (json for example) you were using to parse that data into a ByteReadChannel. It would also not be able to do that, because the data you are getting from twitter is not a ByteReadChannel, it is a new line seperated list of json objects.
I have spring boot controller
#PostMapping(path = ["/download"])
fun getFile(#RequestBody myObjectRq: myObjectRq, httpServletResponse: HttpServletResponse): CompletableFuture<HttpServletResponse> {
return GlobalScope.async {
val response = webService.getFile(myObjectRq)
response?.let {
httpServletResponse.setHeader("Content-Type", response.headers.get("Content-Type"))
httpServletResponse.setHeader("Content-Disposition", response.headers.get("Content-Disposition"))
httpServletResponse.writer.write(String(response.content.toByteArray()))
httpServletResponse.writer.flush()
httpServletResponse.status = response.status.value
}
httpServletResponse
}.asCompletableFuture()
}
in which I use service which in turn uses ktor client to send post request to external server which should respond sending csv file. csv file content depends on values I send in myObjectRq.
Service:
suspend fun getFile(myObjectRq: myObjectRq): HttpResponse {
val response = ktorClient.post<HttpResponse> {
accept(ContentType.Application.OctetStream)
url(externalWebServerUrl)
body = myObjectRq
contentType(ContentType.Application.Json)
}
log.info(String(response.content.toByteArray()))
response
}
Headers in response are properly set, also log.info(String(response.content.toByteArray())) in the method prints out the content of received file, but I can't set it as a body of HttpServletResponse. I keep getting org.springframework.web.HttpMediaTypeNotAcceptableException: Could not find acceptable representation.
Also I get Inappropriate blocking method call for httpServletResponse.writer which kind of breaks async qualities of ktor client.
What do I do wrong? How should I solve it?
So, I think SpringBoot is confused with your return type. It is trying to find a way to serialize your return CompletableFuture<HttpServletResponse> into the body of the HTTP response but failing. I believe you can achieve the same result by changing your implementation as follows:
#PostMapping(path = ["/download"])
fun getFile(#RequestBody myObjectRq: myObjectRq, httpServletResponse: HttpServletResponse): CompletableFuture<Void> {
return GlobalScope.async {
val response = webService.getFile(myObjectRq)
response?.let {
httpServletResponse.setHeader("Content-Type", response.headers.get("Content-Type"))
httpServletResponse.setHeader("Content-Disposition", response.headers.get("Content-Disposition"))
httpServletResponse.writer.write(String(response.content.toByteArray()))
httpServletResponse.writer.flush()
httpServletResponse.status = response.status.value
}
null
}.asCompletableFuture()
}
I actually managed to solve this using CompletableFuture<ResponseEntity<ByteArray>> as return type and setting body of the response this way:
ResponseEntity.ok().body(response.content.toByteArray())
This also removed Inappropriate blocking method call warnings.