I am using spring hexagonal architecture (port and adapter) as my application need to read the stream of data from the source topic, process/transforms the data, and send it to destination topic.
My application need to do the following actions.
Read the data (which will have the call back url)
Make an http call with the url in the incoming data (using webclient)
Get the a actual data and it needs to be transformed into another format.
Send the transformed data to the outgoing topic.
Here is my code,
public Flux<TargeData> getData(Flux<Message<EventInput>> message)
{
return message
.flatMap(it -> {
Event event = objectMapper.convertValue(it.getPayload(), Event.class);
String eventType = event.getHeader().getEventType();
String callBackURL = "";
if (DISTRIBUTOR.equals(eventType)) {
callBackURL = event.getHeader().getCallbackEnpoint();
WebClient client = WebClient.create();
Flux<NodeInput> nodeInputFlux = client.get()
.uri(callBackURL)
.headers(httpHeaders -> {
httpHeaders.setContentType(MediaType.APPLICATION_JSON);
List<MediaType> acceptTypes = new ArrayList<>();
acceptTypes.add(MediaType.APPLICATION_JSON);
httpHeaders.setAccept(acceptTypes);
})
.exchangeToFlux(response -> {
if (response.statusCode()
.equals(HttpStatus.OK)) {
System.out.println("Response is OK");
return response.bodyToFlux(NodeInput.class);
}
return Flux.empty();
});
nodeInputFlux.subscribe( nodeInput -> {
SourceData source = objectMapper.convertValue(nodeInput, SourceData.class);
// return Flux.fromIterable(this.TransformImpl.transform(source));
});
}
return Flux.empty();
});
}
The commented line in the above code is giving the compilation as subscribe method does not allow return types.
I need a solution "without using block" here.
Please help me here, Thanks in advance.
I think i understood the logic. What do you may want is this:
public Flux<TargeData> getData(Flux<Message<EventInput>> message) {
return message
.flatMap(it -> {
// 1. marshall and unmarshall operations are CPU expensive and could harm event loop
return Mono.fromCallable(() -> objectMapper.convertValue(it.getPayload(), Event.class))
.subscribeOn(Schedulers.parallel());
})
.filter(event -> {
// 2. Moving the if-statement yours to a filter - same behavior
String eventType = event.getHeader().getEventType();
return DISTRIBUTOR.equals(eventType);
})
// Here is the trick 1 - your request below return Flux of SourceData the we will flatten
// into a single Flux<SourceData> instead of Flux<List<SourceData>> with flatMapMany
.flatMap(event -> {
// This WebClient should not be created here. Should be a singleton injected on your class
WebClient client = WebClient.create();
return client.get()
.uri(event.getHeader().getCallbackEnpoint())
.accept(MediaType.APPLICATION_JSON)
.exchangeToFlux(response -> {
if (response.statusCode().equals(HttpStatus.OK)) {
System.out.println("Response is OK");
return response.bodyToFlux(SourceData.class);
}
return Flux.empty();
});
})
// Here is the trick 2 - supposing that transform return a Iterable of TargetData, then you should do this and will have Flux<TargetData>
// and flatten instead of Flux<List<TargetData>>
.flatMapIterable(source -> this.TransformImpl.transform(source));
}
I have this server endpoint using spring-webflux and I would like to return ServerResponse.badRequest() when the serverRequest receives a wrong parameter. The request curl -s "http://localhost:8080/4?a=5&b=3"; echo for instance, contains the right parameters. But the request curl -s "http://localhost:8080/one?a=5&b=3"; echo contains a string instead of an Integer. Then the conversion new BidRequest(Integer.parseInt(tuple2.getT1()), tuple2.getT2().toSingleValueMap()) will throw an error.
I was doing .onErrorReturn(new BidRequest(0, null)) but now I want to implement some operation that return ServerResponse.badRequest(). So I added in the end .onErrorResume(error -> ServerResponse.badRequest().build()) in the end, but It is not working. I also added on the place of the code .onErrorReturn() and it does not compile.
public Mono<ServerResponse> bidRequest(ServerRequest serverRequest) {
var adId = serverRequest.pathVariable("id");
var attributes = serverRequest.queryParams();
log.info("received bid request with adID: {} attributes: {}", adId, attributes);
return Mono.just(Tuples.of(adId, attributes))
.map(tuple2 -> new BidRequest(Integer.parseInt(tuple2.getT1()), tuple2.getT2().toSingleValueMap()))
// I WANT TO REPLACE IT FOR A BAD REQUEST
// .onErrorReturn(new BidRequest(0, null))
.flatMap(bidRequest -> {
return Flux.fromStream(bidderService.bidResponseStream(bidRequest))
.flatMap(this::gatherResponses)
.reduce((bidResp1, bidResp2) -> {
if (bidResp1.getBid() > bidResp2.getBid()) return bidResp1;
else return bidResp2;
});
})
.map(bid -> {
var price = bid.getContent().replace("$price$", bid.getBid().toString());
bid.setContent(price);
return bid;
})
.flatMap(winner -> {
return ServerResponse.ok()
.contentType(MediaType.APPLICATION_JSON)
.body(BodyInserters.fromValue(winner.getContent()));
})
.switchIfEmpty(ServerResponse.notFound().build())
// THIS DOES NOT RETURN ANY BAD REQUEST
.onErrorResume(error -> ServerResponse.badRequest().build());
}
I solved based on this answer using flatmap and returning a Mono.just() or a Mono.error(new ResponseStatusException(HttpStatus.BAD_REQUEST));.
return Mono
.just(Tuples.of(adId, attributes))
.flatMap(tuple2 -> {
if (validate(tuple2)) {
log.info("request parameters valid: {}", tuple2);
return Mono.just(new BidRequest(Integer.parseInt(tuple2.getT1()), tuple2.getT2().toSingleValueMap()));
} else {
log.error("request parameters invalid: {}", tuple2);
return Mono.error(new ResponseStatusException(HttpStatus.BAD_REQUEST));
}
})
.flatMap(....
private boolean validate(Tuple2<String, MultiValueMap<String, String>> tuple2) {
return GenericValidator.isInteger(tuple2.getT1());
}
I have a generic screen that subscribes to an RxJava2 flowable that returns a List. It then displays the content in the list.
I have a use case now though where I need to collect data from multiple endpoints, and emit data once some complete, and then emit data again once the remaining ones complete.
I'm doing this using Flowable.create() but I've seen a lot of posts saying that there's usually a better and safer way to do so than using create? I seem to believe that is the case since I need to subscribe to an observable within the observable which ideally I wouldn't want to do?
Because I subscribe within, I know the emitter can become cancelled within the observable while other network calls are completing so I've added checks to ensure it doesn't throw an error after its disposed which do work (at least in testing...) [I also just remembered I have the code available to dispose of the inner subscription if I kept it like this, when the outer is disposed]
The first 2 calls may be incredibly fast (or instant) which is why i want to emit the first result right away, and then the following 4 network calls which rely on that data may take time to process.
It looks roughly like this right now...
return Flowable.create<List<Object>>({ activeEmitter ->
Single.zip(
single1(),
single2(),
BiFunction { single1Result: Object, single2result: Object ->
if (single1result.something || single2Result.somethingElse) {
activeEmitter.onNext(function(single1result, single2result) //returns list
}
Single.zip(
single3(single1result),
single4(single2result),
single5(single1result),
single6(single2result),
Function4 { single3Result: Object,
single4Result: Object,
single5Result: Object,
single6Result: Object ->
ObjectHolder(single1Result, single2Result, single3Result, single4Result, single5Result, single6Result)
}
)
}
).flatMap { objectHolder ->
objects.flatMap { objectHolder ->
Single.just(parseObjects(objectHolder))
}
}.subscribeBy(
onError = { error ->
if (!activeEmitter.isCancelled) {
activeEmitter.onError(error)
}
},
onSuccess = { results ->
if (!activeEmitter.isCancelled) {
activeEmitter.onNext(results)
activeEmitter.onComplete()
}
}
)
}, BackpressureStrategy.BUFFER)
I can't figure out another way to return a Flowable that emits the results of multiple different network calls without doing it like this?
Is there a different/better way I can't find?
I worked this out given ctranxuan response. Posting so he can tweak/optimize and then I accept his answer
return Single.zip(single1(), single2(),
BiFunction { single1result: Object, single2result: Object ->
Pair(single1result, single2result)
}
).toFlowable()
.flatMap { single1AndSingle2 ->
if (isFirstLoad) {
createItemOrNull(single1AndSingle2.first, single1AndSingle2.second)?.let { result ->
Single.just(listOf(result)).mergeWith(proceedWithFinalNetworkCalls(single1AndSingle2))
}.orElse {
proceedWithFinalNetworkCalls(single1AndSingle2).toFlowable()
}
} else {
proceedWithFinalNetworkCalls(single1AndSingle2).toFlowable()
}
}.doOnComplete {
isFirstLoad = false
}
fun proceedWithFinalNetworkCalls(): Flowable<List> {
return Single.zip(
single3(single1result),
single4(single2result),
single5(single1result),
single6(single2result),
Function4 { single3Result: Object,
single4Result: Object,
single5Result: Object,
single6Result: Object ->
ObjectHolder(single1Result, single2Result, single3Result, single4Result, single5Result, single6Result)
}
)
Sorry, it's in Java but from what I've understood, something like that may be a possible solution?
public static void main(String[] args) {
final Single<String> single1 = single1().cache();
single1.map(List::of)
.mergeWith(single1.zipWith(single2(), Map::entry)
.flatMap(entry -> Single.zip(
single3(entry.getKey()),
single4(entry.getValue()),
single5(entry.getKey()),
single6(entry.getValue()),
(el3, el4, el5, el6) -> objectHolder(entry.getKey(), entry.getValue(), el3, el4, el5, el6))))
.subscribe(System.out::println,
System.err::println);
Flowable.timer(1, MINUTES) // Just to block the main thread for a while
.blockingSubscribe();
}
private static List<String> objectHolder(final String el1,
final String el2,
final String el3,
final String el4,
final String el5,
final String el6) {
return List.of(el1, el2, el3, el4, el5, el6);
}
static Single<String> single1() {
return Single.just("s1");
}
static Single<String> single2() {
return Single.just("s2");
}
static Single<String> single3(String value) {
return single("s3", value);
}
static Single<String> single4(String value) {
return single("s4", value);
}
static Single<String> single5(String value) {
return single("s5", value);
}
static Single<String> single6(String value) {
return single("s6", value);
}
static Single<String> single(String value1, String value2) {
return Single.just(value1).map(l -> l + "_" + value2);
}
This outputs:
[s1]
[s1, s2, s3_s1, s4_s2, s5_s1, s6_s2]
I am using ParallelFlux for running many task.
but when i am receiving webClient response using bodyToFlux method its merging all the output response instead of getting one by one.
i want the output should be one by one not single string, is there any other method instead of bodyToFLux need to use.
request method:
Flux<String> responsePost = webClient.build()
//removed get,url and retrieve here
.bodyToFlux(String.class);
responsePost.subscribe(s -> {
//display response
});
response method:
public ParallelFlux<String> convertListToMap() {
//created list of string str
return Flux.fromIterable(str)
.parallel(3)
.runOn(Schedulers.parallel())
.map( s -> {
//some logic here
});
}
output:
parallel fulx reponse: springwebfluxparellelExample
Here instead of returning ParallelFlux create an
ResponseBody class with variable you want to return, assign your response to
variable inside ResponseBody and return it to caller of API function.
public ParallelFlux<ResponseBody> convertListToMap() {
//created list of string str
return Flux.fromIterable(str)
.parallel(3)
.runOn(Schedulers.parallel())
.map( s -> {
//some logic here
});
}
Flux<String> responsePost = webClient.build()
//removed get,url and retrieve here
.bodyToFlux(ResponseBody.class);
Problem statement: I have a POST request to a booking-service API that gets a BookingRecord. I map it to extract the values so as to call another fare-service API using WebClient. I receive a Mono<Fare> from that call. I need to check whether the value of getFare() method of BookingRecord type is same as the getFare() of the Fare type returned by the WebClient. If not, I need to raise and exception, and pass it on to the caller. Here caller is another Microservice, ui-service calling the booking-service API (so how should I deal with this, pass the error back or else what is the best thing to do?) or else I will save the new BookingRecord and return the id of that record to the caller. What is the best flow sequence for this? I tried my best without much success and am pasting the code here.
public HandlerFunction<ServerResponse> book = request ->
{
request.bodyToMono(BookingRecord.class)
.map(br ->
{
this.webClient.get()
.uri("/fares/get/{flightNumber}/{flightDate}",
br.getFlightNumber(),
br.getFlightDate())
.retrieve()
.bodyToMono(Fare.class)
.map(f ->
{
if (!f.getFare()
.equals(br.getFare()))
{
throw new RuntimeException("Fare is tampered");
}
else
{
id = bookingRepository.save(br).getId();
}
return id;
})
.subscribe();
return id;
});
return ServerResponse.ok()
.body(BodyInserters.fromObject(id));
};
After much tweaking, this is what I did. Hope its the right thing to do. 1. I raise a 500 Http error from the fare-service itself instead of checking in the booking-service.
public HandlerFunction<ServerResponse> getFare = request ->
{
String flightNumber = request.pathVariable("flightNumber");
String flightDate = request.pathVariable("flightDate");
String fare = request.pathVariable("fare");
Mono<ServerResponse> notFound = ServerResponse.notFound()
.build();
return Mono
.justOrEmpty(faresRepository.getFareByFlightNumberAndFlightDateAndFare(flightNumber,
flightDate,
fare))
.flatMap(f -> ServerResponse.ok()
.contentType(APPLICATION_JSON)
.body(fromObject(f)))
.switchIfEmpty(notFound);
};
Handled the exception like so in booking-service using onStatus() method
public HandlerFunction<ServerResponse> book = request ->
{
logger.info("Inside Book function");
return request.bodyToMono(BookingRecord.class)
.flatMap(br ->
{
logger.info("Calling fare-service");
return this.webClient.get()
.uri("/fares/get/{flightNumber}/{flightDate}/{fare}",
br.getFlightNumber(),
br.getFlightDate(),
br.getFare())
.retrieve()
.onStatus(HttpStatus::isError,
x -> Mono
.error(new RuntimeException("Fare has been tampered with!!")))
.bodyToMono(Fare.class);
})
.map(fare ->
{
logger.info("Saving a BookingRecord");
BookingRecord br = new BookingRecord();
br.setFlightNumber(fare.getFlightNumber());
br.setFlightDate(fare.getFlightDate());
br.setFare(fare.getFare());
long id = bookingRepository.save(br)
.getId();
return id;
})
.flatMap(id -> ServerResponse.ok()
.body(BodyInserters.fromObject(id)));
};
This way, I get an exception for fare tampering or get the id for a successful db save.