I'm working with an API that only displays an event's venue id in the response when performing an event search API call. I'm looking to see if there's a way to make a Spring WebClient request to fetch the venue information as Flux items are being emitted.
val events = eventService.fetchEventsByLocation(lat,lon,radius)
.flatMapIterable { eventResponse -> EventTransformer.map(eventResponse)}
.doOnNext { transformedEvent -> this.repository.save(transformedEvent) }
fun fetchEventsByLocation(lat:Double?,lon:Double?,radius:Double?): Mono<EventResponse> {
val builder = UriComponentsBuilder.fromPath(SEARCH_EVENTS)
.queryParam("categories", "103")
.queryParam("location.within", 20.toString() + "mi")
.queryParam("location.latitude", lat)
.queryParam("location.longitude", lon)
.queryParam("token",apiKey)
return this.webClient.get()
.uri(builder.toUriString())
.accept(MediaType.APPLICATION_JSON_UTF8)
.exchange()
.flatMap { response -> response.bodyToMono(String::class.java) }
.map { response -> transform(response) }
}
fun fetchEventVenue(id:String?): Mono<Venue> {
val builder = UriComponentsBuilder.fromPath(VENUES + id)
.queryParam("token",apiKey)
return this.webClient.get()
.uri(builder.toUriString())
.accept(MediaType.APPLICATION_JSON_UTF8)
.exchange()
.flatMap { response -> response.bodyToMono(Venue::class.java) }
}
doOnNext is an intermediate operation. Without a terminal operation (such as subscribe), the stream isn't consumed.
Related
I build a web server with Ktor, and want to cache API method result. But I don't know how to get the response body from call.response. Code like below
fun Application.module(){
// before method was called
intercept(ApplicationCallPipeline.Features) {
val cache = redis.get("cache")
if(cache) {
call.respond(cache)
}
return #intercept finish()
}
// after method was called
intercept(ApplicationCallPipeline.Fallback) {
// TODO, how to get call.response.body
redis.set("cache", call.response.body)
}
}
If I can't get the response body, Any other solution to cache the result of an API method in Ktor?
Inside an interceptor, for the ApplicationCallPipeline.Features phase you can add a phase just before the ApplicationSendPipeline.Engine and intercept it to retreive a response body (content):
val phase = PipelinePhase("phase")
call.response.pipeline.insertPhaseBefore(ApplicationSendPipeline.Engine, phase)
call.response.pipeline.intercept(phase) { response ->
val content: ByteReadChannel = when (response) {
is OutgoingContent.ByteArrayContent -> ByteReadChannel(response.bytes())
is OutgoingContent.NoContent -> ByteReadChannel.Empty
is OutgoingContent.ReadChannelContent -> response.readFrom()
is OutgoingContent.WriteChannelContent -> GlobalScope.writer(coroutineContext, autoFlush = true) {
response.writeTo(channel)
}.channel
else -> error("")
}
// Do something with content
}
I'm using Project Reactor with Webflux to try to read data from a message queue, then process it in chunks (eg, five at a time) and make a request to an API with each chunk. The API does not work well with high throughput, so I need to have control over how many requests are sent concurrently.
Basically, I'd like to have a WebClient call finish, then be able to tell the Flux that we're ready to process more.
I was using this code to try to emulate the desired functionality, and I'm getting results that I don't understand:
fun main() {
val subscriber = CustomSubscriber()
Flux.create<Int> { sink ->
sink.onRequest {
sink.next(1)
}
}
.doOnNext {
println("hit first next with $it")
}
.delayElements(Duration.ofSeconds(1)) // Mock WebClient call
.doOnNext {
println("before request")
subscriber.request(1)
println("after request")
}
.subscribeWith(subscriber)
Thread.sleep(10000)
}
class CustomSubscriber : BaseSubscriber<Int>() {
override fun hookOnSubscribe(subscription: Subscription) {
subscription.request(1)
}
}
The output of this code is
hit first next with 1
before request
after request
What I was hoping for is this:
hit first next with 1 // one second passes
before request
after request
hit first next with 1 // one second passes
before request
after request
hit first next with 1 // one second passes
before request
after request
hit first next with 1 // one second passes
before request
after request
(Infinite loop)
So the request method is called, but the number is never emitted.
Oddly, when I call request in a separate Flux, I'm getting the desired behavior:
fun main() {
val subscriber = CustomSubscriber()
Flux.create<Int> { sink ->
sink.onRequest {
sink.next(1)
}
}
.doOnNext {
println("hit first next with $it")
}
.subscribeWith(subscriber)
Flux.range(0, 5)
.delayElements(Duration.ofSeconds(3))
.doOnNext { subscriber.request(1) }
.subscribe()
Thread.sleep(10000)
}
class CustomSubscriber : BaseSubscriber<Int>() {
override fun hookOnSubscribe(subscription: Subscription) {
subscription.request(1)
}
}
So it seems like there is an issue with calling the request method in the doOnNext method of the original Flux?
I'm not married to the idea of using a FluxSink, that just seemed like a way to have more explicit control of the data emission.
I think what you are looking for is custom subscriber, which consumes data at its own pace based on some logic. Something like this.
Flux.range(0, 14)
.subscribeWith(object : Subscriber<Int> {
private var count = 0
lateinit var subscription: Subscription
override fun onSubscribe(s: Subscription) {
subscription = s
s.request(2)
}
override fun onNext(parameter: Int) {
println("Before request")
// ----- some processing
println("After request")
count++
if (count >= 2) {
println("Requesting more......")
count = 0
subscription.request(2)
}
}
override fun onError(t: Throwable) {}
override fun onComplete() {
println("Done")
}
})
We are using Spring Cloud Gateway in order to route requests to multiple underlying services. The calls to these underlying services will be sequential and potentially feed into one another (response from one being used in the request for the next). We have a working solution for when we need to make those requests sequentially BEFORE the main request, but after the main request we are having problems with feeding the response of one proxy request into the request of the next.
The way we have planned on feeding the response from one request to the next is by making the request using a WebClient in the GatewayFilter and storing the response string in the exchange's attribute store. Then during the next proxy request we supply an attribute name to optionally pull the request body from. This works well when using "pre" filters, because the first proxy request is built, executed and response cached before the second request is built and executed, so the chain of attributes works as expected. The problem comes when working with "post" filters. In the post proxy, the web client requests are all built before the subsequent request has finished. So the attribute store never has the response from the previous request, meaning the next request doesn't work as intended because it doesn't have a valid request body.
My understanding was that calling chain.filter(exchange).then(Mono.fromRunnable{ ... }) would cause the .then logic to execute only after the prior filters had fully completed. This does not seem to be the case. In other filter types like logging, response manipulation, etc the post filters execute in the correct order, but when creating a WebClient they don't seem to.
Does anyone have any ideas on how this desired behavior might be achievable?
Pre-Proxy Filter Code(Working):
class PreProxyGatewayFilterFactory: AbstractGatewayFilterFactory<PreProxyGatewayFilterFactory.Params>(Params::class.java) {
override fun apply(params: Params): GatewayFilter {
return OrderedGatewayFilter(
{ exchange, chain ->
ServerWebExchangeUtils.cacheRequestBody(exchange){
val cachedExchange = exchange.mutate().request(it).build()
executeRequest(cachedExchange, params)
.map { response ->
val body = response.body.toString()
cacheResponse(
response.body.toString(),
params.cachedResponseBodyAttributeName,
cachedExchange
)
}
.flatMap(chain::filter)
}
}, params.order)
}
private fun cacheResponse(response: String, attributeName: String?, exchange: ServerWebExchange): ServerWebExchange{
if(!attributeName.isNullOrBlank()){
exchange.attributes[attributeName] = response
}
return exchange
}
private fun executeRequest(exchange: ServerWebExchange, params: Params): Mono<ResponseEntity<String>>{
val request = when(exchange.request.method){
HttpMethod.PUT -> WebClient.create().put().uri(params.proxyPath).body(createProxyRequestBody(exchange, params.cachedRequestBodyAttributeName))
HttpMethod.POST -> WebClient.create().post().uri(params.proxyPath).body(createProxyRequestBody(exchange, params.cachedRequestBodyAttributeName))
HttpMethod.GET -> WebClient.create().get().uri(params.proxyPath)
HttpMethod.DELETE -> WebClient.create().delete().uri(params.proxyPath)
else -> throw Exception("Invalid request method passed in to the proxy filter")
}
return request.headers { headers ->
headers.addAll(exchange.request.headers)
headers.remove(CONTENT_LENGTH)
}
.exchange()
.flatMap{ response ->
response.toEntity(String::class.java)
}
}
private fun createProxyRequestBody(exchange: ServerWebExchange, attributeName: String?): BodyInserter<out Flux<out Any>, ReactiveHttpOutputMessage> {
val cachedBody = attributeName?.let { attrName ->
exchange.getAttributeOrDefault<String>(attrName, "null")
} ?: "null"
return if(cachedBody != "null"){
BodyInserters.fromPublisher(Flux.just(cachedBody), String::class.java)
} else {
BodyInserters.fromDataBuffers(exchange.request.body)
}
}
data class Params(
val proxyPath: String = "",
val cachedRequestBodyAttributeName: String? = null,
val cachedResponseBodyAttributeName: String? = null,
val order: Int = 0
)
}
Post-Proxy Filter Code (Not Working)
class PostProxyGatewayFilterFactory: AbstractGatewayFilterFactory<PostProxyGatewayFilterFactory.Params>(Params::class.java) {
override fun apply(params: Params): GatewayFilter {
return OrderedGatewayFilter(
{ exchange, chain ->
ServerWebExchangeUtils.cacheRequestBody(exchange){
val cachedExchange = exchange.mutate().request(it).build()
//Currently using a cached body does not work in post proxy
chain.filter(cachedExchange).then( Mono.fromRunnable{
executeRequest(cachedExchange, params)
.map { response ->
cacheResponse(
response.body.toString(),
params.cachedResponseBodyAttributeName,
cachedExchange
)
}
.flatMap {
Mono.empty<Void>()
}
})
}
}, params.order)
}
private fun cacheResponse(response: String, attributeName: String?, exchange: ServerWebExchange): ServerWebExchange{
if(!attributeName.isNullOrBlank()){
exchange.attributes[attributeName] = response
}
return exchange
}
private fun executeRequest(exchange: ServerWebExchange, params: Params): Mono<ResponseEntity<String>>{
val request = when(exchange.request.method){
HttpMethod.PUT -> WebClient.create().put().uri(params.proxyPath).body(createProxyRequestBody(exchange, params.cachedRequestBodyAttributeName))
HttpMethod.POST -> WebClient.create().post().uri(params.proxyPath).body(createProxyRequestBody(exchange, params.cachedRequestBodyAttributeName))
HttpMethod.GET -> WebClient.create().get().uri(params.proxyPath)
HttpMethod.DELETE -> WebClient.create().delete().uri(params.proxyPath)
else -> throw Exception("Invalid request method passed in to the proxy filter")
}
return request.headers { headers ->
headers.addAll(exchange.request.headers)
headers.remove(CONTENT_LENGTH)
}
.exchange()
.flatMap{ response ->
response.toEntity(String::class.java)
}
}
private fun createProxyRequestBody(exchange: ServerWebExchange, attributeName: String?): BodyInserter<out Flux<out Any>, ReactiveHttpOutputMessage> {
val cachedBody = attributeName?.let { attrName ->
exchange.getAttributeOrDefault<String>(attrName, "null")
} ?: "null"
return if(cachedBody != "null"){
BodyInserters.fromPublisher(Flux.just(cachedBody), String::class.java)
} else {
BodyInserters.fromDataBuffers(exchange.request.body)
}
}
data class Params(
val proxyPath: String = "",
val cachedRequestBodyAttributeName: String? = null,
val cachedResponseBodyAttributeName: String? = null,
val order: Int = 0
)
}
Was finally able to get to a working solution for the post filter proxy pulling it's request body from the attributes. It was a relatively straightforward fix that I just couldn't find the answer to. Instead of using chain.filter(exchange).then(Mono.fromRunnable { ...execute proxy request...}) I just needed to use chain.filter(exchange).then(Mono.defer { ...execute proxy request...}).
I have a POST Gateway filter that I want to modify the response body with the response of a separate webclient request within the gateway filter. I am able to get as far as sending the WebClient().create().post().exchange() within my custom filter. I can see this in my logs
onStateChange(POST{uri=/test, connection=PooledConnection{channel=[id: 0x38eb2b4f, L:/127.0.0.1:51643 - R:localhost/127.0.0.1:9000]}}, [request_prepared])
and
onStateChange(POST{uri=/test, connection=PooledConnection{channel=[id: 0x38eb2b4f, L:/127.0.0.1:51643 - R:localhost/127.0.0.1:9000]}}, [request_sent])
the connection hangs here and doesn't complete.
here is my custom filter code
class PostProxyGatewayFilterFactory : AbstractGatewayFilterFactory<PostProxyGatewayFilterFactory.Params>(Params::class.java) {
override fun apply(params: Params): GatewayFilter {
val cachedBody = StringBuilder()
return GatewayFilter { exchange: ServerWebExchange, chain: GatewayFilterChain ->
chain.filter(exchange).then(
executeRequest(cachedBody,exchange, params)
.map {
val mr = ResponseHandler(exchange)
mr.mutateResponse(it.body.toString())
}.flatMap{
it
}
) }
}
data class Params(
val urlPath: String = "",
)
private fun cache(cachedBody: StringBuilder, buffer: DataBuffer) {
cachedBody.append(Charsets.UTF_8.decode(buffer.asByteBuffer())
.toString())
}
private fun executeRequest(cachedBody: StringBuilder, exchange: ServerWebExchange, params: PostProxyGatewayFilterFactory.Params): Mono<ResponseEntity<JsonNode>>{
val request = when(exchange.request.method){
HttpMethod.PUT -> WebClient.create().put().uri(params.urlPath).body(BodyInserters.fromDataBuffers(exchange.request.body.doOnNext{ cache(cachedBody, it)}))
HttpMethod.POST -> WebClient.create().post().uri(params.urlPath).body(BodyInserters.fromDataBuffers(exchange.request.body.doOnNext{ cache(cachedBody, it)}))
HttpMethod.GET -> WebClient.create().get().uri(params.urlPath)
HttpMethod.DELETE -> WebClient.create().delete().uri(params.urlPath)
else -> throw Exception("Invalid request method passed in to the proxy filter")
}
return request.headers { it.addAll(exchange.request.headers) }
.exchange()
.flatMap{
it.toEntity(JsonNode::class.java)
}
}
}
here is my ResponseHandler class
class ResponseHandler(val delegate: ServerWebExchange) {
fun mutateResponse(body: String): Mono<Void> {
val bytes: ByteArray = body.toByteArray(StandardCharsets.UTF_8)
val buffer: DataBuffer = delegate.response.bufferFactory().wrap(bytes)
return delegate.response.writeWith(Flux.just(buffer))
}
}
here is application.yml
- id: proxy
uri: http://${HOST:localhost}:${PORT:9000}
predicates:
- Path=/proxy/**
filters:
- RewritePath=/test(?<segment>/?.*), $\{segment}
- name: PostProxy
args:
proxyBasePath: http://localhost:9000/thisSecond
So the idea is to send request to localhost:9001/test/thisFirst (proxied to localhost:9000/thisFirst which does happen successfully),get this response back and do nothing with this response, make WebClient request to localhost:9000/thisSecond via executeRequest() ,return this response, and then use that response as the new exchange.response body. I am also not sure if ResponseHandler is correct as executeRequest() never finishes. This would be part 2 of the question once I can resolve why executeRequest() never finishes.
I'm studying about RxJava with Retrofit, and I'm trying to combine two requests. But its not making a request to getToken api. It's a simple code just for study case
This is what I have now, what am I doing wrong?
apiManager.getToken(body)
.subscribeOn(Schedulers.io())
.map { people -> saveUser(people) }
.doOnNext { car -> Log.d("car",car.toString()) }
.flatMap { car -> Observable.from(car!!.items) }
.flatMap { carId -> val header = HashMap<String, String>()
header.put("Authorization", "Bearer " + user!!.authorization)
apiManager.getCarItens(header, carId.id!!) }
.doOnCompleted { showUser(user) }
.subscribeOn(AndroidSchedulers.mainThread())
You are just defining your Observable, but you are not subscribing to it, so the stream won't get data at all:
apiManager.getToken(body)
.subscribeOn(Schedulers.io())
.map { people -> saveUser(people) }
.doOnNext { car -> Log.d("car", car.toString()) }
.flatMap { car -> Observable.from(car!!.items) }
.flatMap { carId ->
val header = HashMap<String, String>()
header.put("Authorization", "Bearer " + user!!.authorization)
apiManager.getCarItens(header, carId.id!!)
}
.doOnCompleted { showUser(user) }
.subscribeOn(AndroidSchedulers.mainThread())
.subscribe() // This is what you were missing!
I'd recommend you cleanup some things, though:
Avoid using !!: it will cause an Exception if the objects are null, which will bubble up, given that you are not handling them (you could, as an onError parameter to subscribe
Instead of doOnCompleted, use onCompleted parameter to subscribe, which makes it be next to the error handling, and it's easier to read