Aggregating requests in WebFlux - spring-webflux

Hi I am trying with Spring WebFlux and reactor.
What I want to achieve is to aggregate some requests into one call to a third party API. And then return the results. My code looks like this:
#RestController
#RequiredArgsConstructor
#RequestMapping(value = "/aggregate", produces = MediaType.APPLICATION_JSON_VALUE)
public class AggregationController {
private final WebClient externalApiClient;
//sink that stores the query params
private Sinks.Many<String> sink = Sinks.many().multicast().onBackpressureBuffer();
#GetMapping
public Mono<Mono<Map<String, BigDecimal>>> aggregate(#RequestParam(value = "pricing", required = false) List<String> countryCodes) {
//store the query params in the sink
countryCodes
.forEach(sink::tryEmitNext);
return Mono.from(
sink.asFlux()
.distinct()
.bufferTimeout(5, Duration.ofSeconds(5))
)
.map(countries -> String.join(",", countries))
.map(concatenatedCountries -> invokeExternalPricingApi(concatenatedCountries))
.doOnSuccess(s -> sink.emitComplete((signalType, emitResult) -> emitResult.isSuccess()));
}
private Mono<Map<String, BigDecimal>> invokeExternalPricingApi(String countryCodes) {
return externalApiClient.get().uri(uriBuilder -> uriBuilder
.path("/pricing")
.queryParam("q", countryCodes)
.build())
.accept(MediaType.APPLICATION_JSON)
.retrieve()
.bodyToMono(new ParameterizedTypeReference<>() {
});
}
}
I have the Sinks.Many<String> sink to accumulate the query params from the callers of the API. When there are 5 items or 5 seconds expired, I want to call the third party API. The issue is that when I do two requests like:
GET localhost:8081/aggregate?pricing=CH,EU
GET localhost:8081/aggregate?pricing=AU,NZ,PT
On one request I get the response and on the other I get null. Also, after the first interaction the system stops working as if the sink was broken.
What am I missing here?

Related

Spring WebClient is not emitting any data from request

This is the content type of the request
I use spring boot and this code to redirect the stream from the web server to the browser.
public Flux<String> getDataStream(String configType, String configId, String user){
Flux<String> response = null;
Optional<String> url = getLogURL(configId);
if(url.isPresent()){
logger.info("#WebClient initiation");
WebClient webClient = WebClient.create();
response = webClient.get()
.uri(url.get())
.accept(MediaType.ALL)
.retrieve()
.bodyToFlux(String.class);
}
logger.info("#WebClient response");
return response;
}
After making the request, I can see #WebClient initiation and #WebClient response in a log file.
This is the client-side JavaScript code statement, that hits the above code getDataStream()
let eventSource = new EventSource(STREAM_URL);
But it doesn't get any response from the server. I couldn't figure out what I am missing. The URL that getDataStream()->WebClient hits, confirmed to produces output.
There are several options to stream data from the server using WebFlux:
Server-sent events pushing individual events (media type: text/event-stream)
Streaming events separated by newlines (media type: application/x-ndjson)
If you need plain text content - use text/event-stream. Check WebFlux async responses from controllers for a complete example.
To see response results you need to log it as a part of the reactive flow. Also, there is no null in reactive and you need to return Flux.empty() instead.
#GetMapping(produces = TEXT_EVENT_STREAM_VALUE)
public Flux<String> getDataStream(String configType, String configId, String user){
Optional<String> url = getLogURL(configId);
if (url.isEmpty()) {
return Flux.empty();
}
log.info("#WebClient initiation");
WebClient webClient = WebClient.create();
return webClient.get()
.uri(url.get())
.accept(MediaType.ALL)
.retrieve()
.bodyToFlux(String.class)
.doOnSubscribe(s -> log.info("#WebClient request"))
.doOnNext(rec -> log.info("#WebClient response record: {}", rec));
}

Is it possible to pass Flux to the body of webClient POST?

I have an endpoint like this :
#PostMapping("/products")
Flux<Product> getProducts(#RequestBody Flux<String> ids) {
return Flux...
}
On my client side, I want to consume this endpoint but not sure how to pass a Flux of String in the body (I don't want to make it a list)
Flux<Product> getProducts(Flux<String> ids) {
return webClient.post().uri("/products")
.body(/* .. how should I do here? ..*/)
.retrieve()
.bodyToFlux(Product.class);
}
You can, in fact, pass in a Flux into the .body() method on the WebClient
Flux<Person> personFlux = ... ;
Mono<Void> result = client.post()
.uri("/persons/{id}", id)
.contentType(MediaType.APPLICATION_STREAM_JSON)
.body(personFlux, Person.class)
.retrieve()
.bodyToMono(Void.class);
Example taken from Spring Reference Docs
The variation of the body() method you'd want to use is:
<T,P extends org.reactivestreams.Publisher<T>> WebClient.RequestHeadersSpec<?> body(P publisher,
Class<T> elementClass)
Relevant JavaDoc for this method

how to reverse the measurement data using MeasurementFilter in Java SDK for Cumulocity Api?

I am using below code to get the latest measurement API details for specific device but its not returning the data in descending order:
import com.cumulocity.sdk.client.measurement.MeasurementFilter;
import com.cumulocity.sdk.client.Platform;
import com.cumulocity.rest.representation.measurement.MeasurementRepresentation;
#Autowired
private Platform platform;
MeasurementFilter filter = new MeasurementFilter().byType("type").bySource("deviceId").byDate(fromDate,dateTo);
Iterable<MeasurementRepresentation> mRep = platform.getMeasurementApi().getMeasurementsByFilter(filter).get().elements(1);
List<MeasurementRepresentation> mRepList = StreamSupport.stream(mRep.spliterator(), false).collect(Collectors.toList());
...
MeasurementFilter api
we can get the latest data using 'revert=true' in Http REST url call..
../measurement/measurements?source={deviceId}&type={type}&dateTo=xxx&dateFrom=xxx&revert=true
How we can use 'revert=true' or other way to get measurement details in order using Cumulocity Java SDK? appreciate your help here.
The SDK currently has no out-of-the-box QueryParam for revert parameter so you have to create it yourself:
import com.cumulocity.sdk.client.Param;
public class RevertParam implements Param {
#Override
public String getName() {
return "revert";
}
}
And then you can combine it with your query. Therefore you to include your Query Param when you use the get() on the MeasurementCollection. You are currently not passing anything but you can pass pageSize and an arbitrary number of QueryParam.
private Iterable<MeasurementRepresentation> getMeasurementByFilterAndQuery(int pageSize, MeasurementFilter filter, QueryParam... queryParam) {
MeasurementCollection collection = measurementApi.getMeasurementByFilter(filter);
Iterable<MeasurementRepresentation> iterable = collection.get(pageSize, queryParam).allPages();
return iterable;
}
private Optional<MeasurementRepresentation> getLastMeasurement(GId source) {
QueryParam revertQueryParam = new QueryParam(new RevertParam(), "true");
MeasurementFilter filter = new MeasurementFilter()
.bySource(source)
.byFromDate(new DateTime(0).toDate());
Iterable<MeasurementRepresentation> iterable = measurementRepository.getMeasurementByFilterAndQuery(1, filter, revertQueryParam);
if (iterable.iterator().hasNext()) {
return Optional.of(iterable.iterator().next());
} else {
return Optional.absent();
}
}
Extending your code it could look like this:
QueryParam revertQueryParam = new QueryParam(new RevertParam(), "true");
MeasurementFilter filter = new MeasurementFilter().byType("type").bySource("deviceId").byDate(fromDate,dateTo);
Iterable<MeasurementRepresentation> mRep = platform.getMeasurementApi().getMeasurementsByFilter(filter).get(1, revertQueryParam);
List<MeasurementRepresentation> mRepList = StreamSupport.stream(mRep.spliterator(), false).collect(Collectors.toList());
What you did with elements is not incorrect but it is not limiting the API call to just return one value. It would query with defaultPageSize (=5) and then on Iterable level limit it to only return one. The elements() function is more for usage when you need more elements than the maxPageSize (=2000). Then it will handle automatic requesting for additional pages and you can just loop through the Iterable.

Only applying a Spring Cloud Gateway filter on some responses

I have a Spring Cloud Gateway service sitting in front of a number of backend services but currently it does not log very much. I wanted to log responses from backend services whenever they return unexpected response status codes but I've hit on the following problem.
I can log the response body for requests using a modifyResponseBody filter and a RewriteFunction like so:
.route("test") { r ->
r.path("/httpbin/**")
.filters { f ->
f.modifyResponseBody(String::class.java, String::class.java){ exchange, string ->
if(e.response.statusCode.is2xxSuccessful)println("Server error")
Mono.just(string)
}
}.uri("http://httpbin.org")
}
My issue with this method is that I'm parsing the response ByteArray to a String on every response, with the overhead that implies, even though I'm just using the body string on a small subset of those responses.
I've tried instead to implement a custom filter like so:
.route("test2") {r ->
r.path("/httpbin2/**")
.filters { f ->
f.filter(LogUnexpectedResponseFilter()
.apply(LogUnexpectedResponseFilter.Config(listOf(HttpStatus.OK))))
}.uri("http://httpbin.org")
}
class LogUnexpectedResponseFilter : AbstractGatewayFilterFactory<LogUnexpectedResponseFilter.Config>() {
val logger = LoggerFactory.getLogger(this::class.java)
override fun apply(config: Config?): GatewayFilter {
return GatewayFilter { exchange, chain ->
logger.info("running custom filter")
if (config?.errors!!.contains(exchange.response.statusCode)){
return#GatewayFilter ModifyResponseBodyGatewayFilterFactory().apply(ModifyResponseBodyGatewayFilterFactory.Config()
.setInClass(String::class.java)
.setOutClass(String::class.java)
.setRewriteFunction(String::class.java, String::class.java) { _, body ->
logger.error(body)
Mono.just(body)
}).filter(exchange, chain)
} else {
chain.filter(exchange)
}
}
}
data class Config(val errors: List<HttpStatus>)
}
What this is supposed to do is simply let the request pass through on most requests but apply the log filter on those that I have configured it to (although in this example I'm having it log on 200 status responses).
What I'm seeing when I debug is that it correctly applies the right filter but the RewriteFunction inside it isn't being run at all. What am I missing?

Streaming objects from S3 Object using Spring Aws Integration

I am working on a usecase where I am supposed to poll S3 -> read the stream for the content -> do some processing and upload it to another bucket rather than writing the file in my server.
I know I can achieve it using S3StreamingMessageSource in Spring aws integration but the problem I am facing is that I do not know on how to process the message stream received by polling
public class S3PollerConfigurationUsingStreaming {
#Value("${amazonProperties.bucketName}")
private String bucketName;
#Value("${amazonProperties.newBucket}")
private String newBucket;
#Autowired
private AmazonClientService amazonClient;
#Bean
#InboundChannelAdapter(value = "s3Channel", poller = #Poller(fixedDelay = "100"))
public MessageSource<InputStream> s3InboundStreamingMessageSource() {
S3StreamingMessageSource messageSource = new S3StreamingMessageSource(template());
messageSource.setRemoteDirectory(bucketName);
messageSource.setFilter(new S3PersistentAcceptOnceFileListFilter(new SimpleMetadataStore(),
"streaming"));
return messageSource;
}
#Bean
#Transformer(inputChannel = "s3Channel", outputChannel = "data")
public org.springframework.integration.transformer.Transformer transformer() {
return new StreamTransformer();
}
#Bean
public S3RemoteFileTemplate template() {
return new S3RemoteFileTemplate(new S3SessionFactory(amazonClient.getS3Client()));
}
#Bean
public PollableChannel s3Channel() {
return new QueueChannel();
}
#Bean
IntegrationFlow fileStreamingFlow() {
return IntegrationFlows
.from(s3InboundStreamingMessageSource(),
e -> e.poller(p -> p.fixedDelay(30, TimeUnit.SECONDS)))
.handle(streamFile())
.get();
}
}
Can someone please help me with the code to process the stream ?
Not sure what is your problem, but I see that you have a mix of concerns. If you use messaging annotations (see #InboundChannelAdapter in your config), what is the point to use the same s3InboundStreamingMessageSource in the IntegrationFlow definition?
Anyway it looks like you have already explored for yourself a StreamTransformer. This one has a charset property to convert your InputStreamfrom the remote S3 resource to the String. Otherwise it returns a byte[]. Everything else is up to you what and how to do with this converted content.
Also I don't see reason to have an s3Channel as a QueueChannel, since the start of your flow is pollable anyway by the #InboundChannelAdapter.
From big height I would say we have more questions to you, than vise versa...
UPDATE
Not clear what is your idea for InputStream processing, but that is really a fact that after S3StreamingMessageSource you are going to have exactly InputStream as a payload in the next handler.
Also not sure what is your streamFile(), but it must really expect InputStream as an input from the payload of the request message.
You also can use the mentioned StreamTransformer over there:
#Bean
IntegrationFlow fileStreamingFlow() {
return IntegrationFlows
.from(s3InboundStreamingMessageSource(),
e -> e.poller(p -> p.fixedDelay(30, TimeUnit.SECONDS)))
.transform(Transformers.fromStream("UTF-8"))
.get();
}
And the next .handle() will be ready for String as a payload.