Multiple POST requests with RxJava - kotlin

I have a list of Books.
open class Book(
#PrimaryKey
var id: String? = null,
var title: String? = null,
var author: String? = null
): RealmObject()
While for-looping, I filter some books and create an Observable with filtered ones. I add each observable to an array.
val listInserts = ArrayList<Observable<Book>>()
for loop(..) {
if (localBook condition) {
val postObservable = networkApiAdapter.insert(localBook)
listInserts.add(postObservable)
}
}
I merge (or concat) the observables hoping for some sequential POST requests.
Observable.concat(listInserts)
.subscribeOn(Schedulers.io())
.observeOn(AndroidSchedulers.mainThread())
.subscribe(
{ s ->
println(s)
},
{ err ->
println(err)
},
{ println("onComplete") }
)
Always only one POST request is received at my MongoDb+Flask server and I also get this error:
com.google.gson.JsonSyntaxException: java.lang.IllegalStateException: Expected BEGIN_OBJECT but was NUMBER at line 1 column 4 path $
networkApiAdapter functions, Retrofit:
class NetworkAPIAdapter private constructor() {
fun insert(dto: Book): Observable<Book> {
println(dto.toString())
return bookService.insert(dto.title!!, dto.author!!)
}
interface BooksService {
#FormUrlEncoded
#POST(URL_ORDERS_ALL)
fun insert(
#Field("title") title: String,
#Field("author") author: String
): Observable<Book>
}
}
Any help would be welcomed. I don't know how to do multiples requests. I have tried many solutions with zip, repeatUntil, flatMap, but none worked.
After solving this, I have to delete all local books and do a GET request. All somehow working using RxJava.

Your BooksService.insert is defined as a POST request that expects a response from server in form of a Book JSON string (i.e. Observable<Book>). I think that is not what your server returns after the POST request goes through. Check what your POST request returns, I think it returns some Int (probably a response status) instead of a Book, which would explain your error:
com.google.gson.JsonSyntaxException: java.lang.IllegalStateException: Expected BEGIN_OBJECT but was NUMBER at line 1 column 4 path $
If that is the case, your concatenated Observable will terminate as soon as the error is thrown, i.e. right after your first POST request, therefore the remaining requests will not be executed.
If my assumptions were right, change your service to:
interface BooksService {
#FormUrlEncoded
#POST(URL_ORDERS_ALL)
fun insert(
#Field("title") title: String,
#Field("author") author: String
): Observable<Int>
}
Also, if you want your other requests to execute even if the previous throws an error, do:
val postObservable = networkApiAdapter.insert(localBook).onErrorReturn(_ -> SOME_INT_FLAG)
UPDATE
To execute your required logic after all the POST requests have been sent, try:
Observable.concat(listInserts)
// complete this observable after all POST requests have been sent
// not that it does not necesarily mean that all responses were 200, you should implement yourself what happens to items that were not successfully `POST`ed
.take(listInserts.size)
.doOnComplete {
// called when this observable completes, i.e. when all POST requests have been sent
executeFinalActions()
}
.subscribeOn(Schedulers.io())
.observeOn(AndroidSchedulers.mainThread())
.subscribe(
{ s->
println(s)
},
{ err ->
println(err)
},
{ println("onComplete") }
)
And implement the executeFinalActions() function:
private fun executeFinalActions() {
realm.executeTransaction(Realm::deleteAll)
networkApiAdapter.fetchAll()
.subscribeOn(Schedulers.io())
.observeOn(AndroidSchedulers.mainThread())
.subscribe(
{ books ->
// todo: store books
},
{ err ->
println(err)
}
)
}
PS: you may want to have a look what Observable.take() does: http://reactivex.io/documentation/operators/take.html

Related

How to invoke function based on condition of iterated value of Mono<List<String>> without using subscribe()?

I want to invoke a function that will notify the admin about some information missing, but I do not want to subscribe to this Mono, because I will subscribe to it later. The problem is I have some log which is called inside doOnSuccess() and when I use subscribe() and then build a response where I zip listOfWords value, the same log is logged twice and I do not want a code to behave that way.
Is there any way to retrieve that value in checkCondition() in a way that will not invoke doOnSuccess() or should I use some other function in merge() that can replace doOnSuccess()?
Should I use subscribe() only once on given Mono or is it allowed to use it multiple times?
Thank you in advance!
The functions are called in the presented order.
Code where log is called:
private fun merge(list1: Mono<List<String>>, list2: Mono<List<String>>) =
Flux.merge(
list1.flatMapMany { Flux.fromIterable(it) },
list2.flatMapMany { Flux.fromIterable(it) }
)
.collectList()
.doOnSuccess { LOG.debug("List of words: $it") }
Code where subscribe is called:
private fun checkCondition(
listOfWords: Mono<List<String>>,
) {
listOfWords.subscribe {
it.forEach { word ->
if (someCondition(word)) {
alarmSystem.notify("Something is missing for word {0}")
}
}
}
}
Code where response is built:
private fun buildResponse(
map: Mono<Map<String, String>>,
list1: List<SomeObject>,
listOfWords: Mono<List<String>>
): Mono<List<Answer>> {
val response = Mono.zip(map, Mono.just(list1), listOfWords)
.map { tuple ->
run {
val tupleMap = tuple.t1
val list = tuple.t2
val words = tuple.t3
list
.filter { someCondition(words) }
.map { obj -> NewObject(x,y) }
}
}

Spring Cloud Gateway: Post Filter Web Client Request

We are using Spring Cloud Gateway in order to route requests to multiple underlying services. The calls to these underlying services will be sequential and potentially feed into one another (response from one being used in the request for the next). We have a working solution for when we need to make those requests sequentially BEFORE the main request, but after the main request we are having problems with feeding the response of one proxy request into the request of the next.
The way we have planned on feeding the response from one request to the next is by making the request using a WebClient in the GatewayFilter and storing the response string in the exchange's attribute store. Then during the next proxy request we supply an attribute name to optionally pull the request body from. This works well when using "pre" filters, because the first proxy request is built, executed and response cached before the second request is built and executed, so the chain of attributes works as expected. The problem comes when working with "post" filters. In the post proxy, the web client requests are all built before the subsequent request has finished. So the attribute store never has the response from the previous request, meaning the next request doesn't work as intended because it doesn't have a valid request body.
My understanding was that calling chain.filter(exchange).then(Mono.fromRunnable{ ... }) would cause the .then logic to execute only after the prior filters had fully completed. This does not seem to be the case. In other filter types like logging, response manipulation, etc the post filters execute in the correct order, but when creating a WebClient they don't seem to.
Does anyone have any ideas on how this desired behavior might be achievable?
Pre-Proxy Filter Code(Working):
class PreProxyGatewayFilterFactory: AbstractGatewayFilterFactory<PreProxyGatewayFilterFactory.Params>(Params::class.java) {
override fun apply(params: Params): GatewayFilter {
return OrderedGatewayFilter(
{ exchange, chain ->
ServerWebExchangeUtils.cacheRequestBody(exchange){
val cachedExchange = exchange.mutate().request(it).build()
executeRequest(cachedExchange, params)
.map { response ->
val body = response.body.toString()
cacheResponse(
response.body.toString(),
params.cachedResponseBodyAttributeName,
cachedExchange
)
}
.flatMap(chain::filter)
}
}, params.order)
}
private fun cacheResponse(response: String, attributeName: String?, exchange: ServerWebExchange): ServerWebExchange{
if(!attributeName.isNullOrBlank()){
exchange.attributes[attributeName] = response
}
return exchange
}
private fun executeRequest(exchange: ServerWebExchange, params: Params): Mono<ResponseEntity<String>>{
val request = when(exchange.request.method){
HttpMethod.PUT -> WebClient.create().put().uri(params.proxyPath).body(createProxyRequestBody(exchange, params.cachedRequestBodyAttributeName))
HttpMethod.POST -> WebClient.create().post().uri(params.proxyPath).body(createProxyRequestBody(exchange, params.cachedRequestBodyAttributeName))
HttpMethod.GET -> WebClient.create().get().uri(params.proxyPath)
HttpMethod.DELETE -> WebClient.create().delete().uri(params.proxyPath)
else -> throw Exception("Invalid request method passed in to the proxy filter")
}
return request.headers { headers ->
headers.addAll(exchange.request.headers)
headers.remove(CONTENT_LENGTH)
}
.exchange()
.flatMap{ response ->
response.toEntity(String::class.java)
}
}
private fun createProxyRequestBody(exchange: ServerWebExchange, attributeName: String?): BodyInserter<out Flux<out Any>, ReactiveHttpOutputMessage> {
val cachedBody = attributeName?.let { attrName ->
exchange.getAttributeOrDefault<String>(attrName, "null")
} ?: "null"
return if(cachedBody != "null"){
BodyInserters.fromPublisher(Flux.just(cachedBody), String::class.java)
} else {
BodyInserters.fromDataBuffers(exchange.request.body)
}
}
data class Params(
val proxyPath: String = "",
val cachedRequestBodyAttributeName: String? = null,
val cachedResponseBodyAttributeName: String? = null,
val order: Int = 0
)
}
Post-Proxy Filter Code (Not Working)
class PostProxyGatewayFilterFactory: AbstractGatewayFilterFactory<PostProxyGatewayFilterFactory.Params>(Params::class.java) {
override fun apply(params: Params): GatewayFilter {
return OrderedGatewayFilter(
{ exchange, chain ->
ServerWebExchangeUtils.cacheRequestBody(exchange){
val cachedExchange = exchange.mutate().request(it).build()
//Currently using a cached body does not work in post proxy
chain.filter(cachedExchange).then( Mono.fromRunnable{
executeRequest(cachedExchange, params)
.map { response ->
cacheResponse(
response.body.toString(),
params.cachedResponseBodyAttributeName,
cachedExchange
)
}
.flatMap {
Mono.empty<Void>()
}
})
}
}, params.order)
}
private fun cacheResponse(response: String, attributeName: String?, exchange: ServerWebExchange): ServerWebExchange{
if(!attributeName.isNullOrBlank()){
exchange.attributes[attributeName] = response
}
return exchange
}
private fun executeRequest(exchange: ServerWebExchange, params: Params): Mono<ResponseEntity<String>>{
val request = when(exchange.request.method){
HttpMethod.PUT -> WebClient.create().put().uri(params.proxyPath).body(createProxyRequestBody(exchange, params.cachedRequestBodyAttributeName))
HttpMethod.POST -> WebClient.create().post().uri(params.proxyPath).body(createProxyRequestBody(exchange, params.cachedRequestBodyAttributeName))
HttpMethod.GET -> WebClient.create().get().uri(params.proxyPath)
HttpMethod.DELETE -> WebClient.create().delete().uri(params.proxyPath)
else -> throw Exception("Invalid request method passed in to the proxy filter")
}
return request.headers { headers ->
headers.addAll(exchange.request.headers)
headers.remove(CONTENT_LENGTH)
}
.exchange()
.flatMap{ response ->
response.toEntity(String::class.java)
}
}
private fun createProxyRequestBody(exchange: ServerWebExchange, attributeName: String?): BodyInserter<out Flux<out Any>, ReactiveHttpOutputMessage> {
val cachedBody = attributeName?.let { attrName ->
exchange.getAttributeOrDefault<String>(attrName, "null")
} ?: "null"
return if(cachedBody != "null"){
BodyInserters.fromPublisher(Flux.just(cachedBody), String::class.java)
} else {
BodyInserters.fromDataBuffers(exchange.request.body)
}
}
data class Params(
val proxyPath: String = "",
val cachedRequestBodyAttributeName: String? = null,
val cachedResponseBodyAttributeName: String? = null,
val order: Int = 0
)
}
Was finally able to get to a working solution for the post filter proxy pulling it's request body from the attributes. It was a relatively straightforward fix that I just couldn't find the answer to. Instead of using chain.filter(exchange).then(Mono.fromRunnable { ...execute proxy request...}) I just needed to use chain.filter(exchange).then(Mono.defer { ...execute proxy request...}).

How to add condition to method "retry" in kotlin and webflux when api return error data?

How to change the parameters with retry() in kotlin and webflux ?
There is a productInfo function, the function parameter is a collection of product ids.
When I input a wrong id in the list collection ids, the upstream interface will only return the wrong id. And get failed.
What I want to achieve is when the upstream interface returns the wrong id. The product info can remove the wrong id and have a second try with the right ids.
I tried to use retry() but I don't know how to change the parameters in the second try.
fun productInfo(ids: List<Pair<String, String>>): Flux<ProductItem> {
return productWebClient
.get()
.uri("product/items/${ids.joinToString(";") { "${it.second},${it.first}" }}")
.retrieve()
.bodyToFlux(ProductItem::class.java)
.onErrorResume {
logger.error("Fetch products failed." + it.message)
Mono.empty()
}
}
What you want is not retry(). I've built a solution making minor assumptions here and there. You can refer to this solution and make changes according to your requirements. I've used recursion here (productInfo()). You can replace the recursion call with webclient call if the error occurs only once.
fun productInfo(ids: List<Pair<String, String>>): Flux<ProductItem> {
val idsString = ids.joinToString(";") { "${it.second},${it.first}" }
return webClient
.get()
.uri("product/items/${idsString}")
.exchange()
.flatMapMany { response ->
if (response.statusCode().isError) {
response.body { clientHttpResponse, _ ->
clientHttpResponse.body.cast(String::class.java).collectList()
.flatMapMany<ProductItem> { eids ->
val ids2 = ids.filter { eids.contains("${it.second},${it.first}") }
productInfo(ids2)
}
}
} else {
response.bodyToFlux(ProductItem::class.java)
}
}
}

Vert.x Reactive Kafka client: chaining not working when writing?

I am using io.vertx.reactivex.kafka.client.producer.KafkaProducer client. The client has a
rxWrite function which returns Single<RecordMetadata>. However I need to log error if any, during write operation. It apparently is not getting executed.
I have written following working example.
test(): Function to test the chaining and logging
fun test(): Single<Int> {
val data = Single.just(ArrayList<String>().apply {
add("Hello")
add("World")
})
data.flattenAsObservable<String> { list -> list }
.flatMap { advertiser ->
//does not work with writeKafka
writeError(advertiser).toObservable().doOnError({ println("Error $data") })
}
.subscribe({ record -> println(record) }, { e -> println("Error2 $e") })
return data.map { it.size }
}
writeKafka: Writes the given given string into Kafka and returns Single
fun writeKafka(param: String): Single<RecordMetadata> {
//null topic to produce IllegalArgumentException()
val record = KafkaProducerRecord.create(null, UUID.randomUUID().toString(), param)
return kafkaProducer.rxWrite(record)
}
writeError: Always return a single with error of same type
fun writeError(param: String): Single<RecordMetadata> {
return Single.error<RecordMetadata>(IllegalArgumentException())
}
So when I call writeKafka It only prints Error2 but if I use writeError it prints both Error and Error2. Looks like the single returned by writeKafka is still waiting for result, but then why even Error2 is printed?
I am pretty newbie in RxJava2, could somebody point out any error in that?
It is important to read and post the stacktrace of errors so that the problem can be isolated.
In this case, looks like you get the IllegalArgumentException from create and you don't get any Single because the relevant Kafka class throws it. return kafkaProducer.rxWrite(record) never executes at all and you practically crash the flatMap. doOnError never gets into play hence only the "Error2" is printed.

RxAndroid - Handle Errors with Zip operator

I'm trying to find a way to execute requests in parallel and handle them when every observable finishes. Despite everything is working when all observables gives a response, I not seeing a way to handle each all errors when everything is finished.
This is a sample of zip operator, which basically executes 2 requests in parallel:
Observable.zip(
getObservable1()
.onErrorResumeNext { errorThrowable: Throwable ->
Observable.error(ErrorEntity(Type.ONE, errorThrowable))
}.subscribeOn(Schedulers.io()),
getObservable2()
.onErrorResumeNext { errorThrowable: Throwable ->
Observable.error(ErrorEntity(Type.TWO, errorThrowable))
}.subscribeOn(Schedulers.io()),
BiFunction { value1: String, value2: String ->
return#BiFunction value1 + value2
})
//execute requests should be on io() thread
.subscribeOn(Schedulers.io())
//there are other tasks inside subscriber that need io() thread
.observeOn(AndroidSchedulers.mainThread())
.subscribe(
{ result ->
Snackbar.make(view, "Replace with your own action " + result, Snackbar.LENGTH_LONG)
.setAction("Action", null).show()
},
{ error ->
Log.d("TAG", "Error is : " + (error as ErrorEntity).error.message)
}
)
private fun getObservable1(): Observable<String> {
return Observable.defer {
throw Throwable("Error 1")
}
}
private fun getObservable2(): Observable<String> {
return Observable.defer {
throw Throwable("Error 2")
}
}
Problem with this approach is that there is no mechanism to join each error like BiFunction do for the success case. Therefore, the zip operator will only trigger the first error and will ignore the others.
Output:
D/TAG: Error is : Error 1
Is there any way to retrieve all errors only after every observable inside zip completed or gave an error?
My main goal is to see which requests gave an error and execute only those after a dialog appears to the user asking him if he wants to retry the failed requests.
You can model your observables using data classes. E.g.
sealed class Response {
data class Success(val data: String) : Response()
data class Error(val t: Throwable) : Response()
}
then you can map your observables to Response like this:
val first: Observable<Response> = observable1
.map<Response> { Response.Success(it) }
.onErrorReturn { Response.Error(it) }
val second: Observable<Response> = observable2
.map<Response> { Response.Success(it) }
.onErrorReturn { Response.Error(it) }
and you can combine them:
Observable.zip(
first,
second,
BiFunction { t1: Response, t2: Response -> Pair(t1, t2) }
).subscribe({println(it)})
this prints:
(Error(t=java.lang.Exception: Error 1), Error(t=java.lang.Exception:
Error 2))
Also take a look at this article.