I need to add a header to requests based on the body. How can I add interceptor for client requests? I have tried
httpClient.requestPipeline.intercept(HttpRequestPipeline.Send) { context: PipelineContext<Any, HttpRequestBuilder> ->
context.proceed()
}
But it wasn't called
You can use the State phase of the HttpSendPipeline to modify request:
val client = HttpClient(Apache)
client.sendPipeline.intercept(HttpSendPipeline.State) {
context.headers.append("myHeader", context.body.toString())
}
val r = client.get<String>("http://httpbin.org/get")
println(r)
Related
I have a KTOR backend which serves as a broker between the frontend-client and an external REST API. I want to make KTOR fetch the chunks of a file from the REST API, and as it receives these chunks KTOR should pass them on to the client, without having to temporarily store the entire file. The file can be very large, which is why the only option is to stream it.
I have made this simple illustration to show what I want to achieve:
I have something like this in my code so far, but it doesn't seem to work correctly:
get("/file") {
val uri = "/rest-api"
downloadFileClient.prepareGet(uri).execute {response ->
call.respondOutputStream(ContentType.Application.Pdf, HttpStatusCode.OK, producer = {response.bodyAsChannel()})
}
}
You can respond with an object of the OutgoingContent.ReadChannelContent class which can use client's response as a source:
get("/file") {
val uri = "/rest-api"
downloadFileClient.prepareGet(uri).execute { response ->
val channel = response.bodyAsChannel()
call.respond(object : OutgoingContent.ReadChannelContent() {
override fun readFrom(): ByteReadChannel = channel
override val status: HttpStatusCode = HttpStatusCode.OK
override val contentType: ContentType = ContentType.Application.Pdf
})
}
}
I am trying to call a twitter endpoint that gives you a constant streams of json results back to the client
https://documenter.getpostman.com/view/9956214/T1LMiT5U#977c147d-0462-4553-adfa-d7a1fe59c3ec
I try to make a call to the endpoint like this
val url = "https://api.twitter.com/2/tweets/search/stream"
_streamChannel = _client.get<ByteReadChannel>(token, url) //Stops here
val byteBufferSize = 1024
val byteBuffer = ByteArray(byteBufferSize)
_streamChannel?.let {
while (_streamChannel!!.availableForRead > 0) {
_streamChannel!!.readAvailable(byteBuffer, 0, byteBufferSize)
val s = String(byteBuffer)
parseStreamResponseString(s).forEach {
emit(Response.Success(it))
}
}
}
my client.get code is this
suspend inline fun <reified T> get(authKey: String, url: String): T? {
val response = _client.get<HttpResponse>(url) {
header("Authorization", "Bearer $authKey")
}
when (response.status.value) {
in 300..399 -> throw RedirectResponseException(response)
in 400..499 -> throw ClientRequestException(response)
in 500..599 -> throw ServerResponseException(response)
}
if (response.status.value >= 600) {
throw ResponseException(response)
}
return response.receive<T>()
}
When I make the request it just sits there in what I am assuming is waiting for the full response to be returned before giving it to me.
Edit
I also tried using scoped streaming but it just sits at the line readAvailable I know there are messages coming through because when I run the request via cURL I am constantly getting data
_client.get<HttpStatement> {
header("Authorization", "Bearer $authKey")
url(urlString)
contentType(ContentType.Application.Json)
method = HttpMethod.Get
}.execute {
val streamChannel = it.receive<ByteReadChannel>()
val byteBufferSize = 1024
val byteBuffer = ByteArray(byteBufferSize)
streamChannel.readAvailable(byteBuffer, 0, byteBufferSize) // Stops here
val s = String(byteBuffer)
}
How do I process a constant stream of json data using Ktor?
As far as I am aware, the Ktor client does note expose access to the IO buffer of the request in the way that twitter's streaming API requires.
From the twitter documentation here:
Some HTTP client libraries only return the response body after the connection has been closed by the server. These clients will not work for accessing the Streaming API. You must use an HTTP client that will return response data incrementally. Most robust HTTP client libraries will provide this functionality. The Apache HttpClient will handle this use case, for example.
What you are doing is telling Ktor that the thing you are getting is a ByteReadChannel, and so, once the request closes (which will never happen with this twitter endpoint) the Ktor client would attempt to use whatever plugin (json for example) you were using to parse that data into a ByteReadChannel. It would also not be able to do that, because the data you are getting from twitter is not a ByteReadChannel, it is a new line seperated list of json objects.
I use IDEA to generate a template and notice that runBlocking in Application.module like:
runBlocking {
// Sample for making a HTTP Client request
val message = client.post<JsonSampleClass> {
url("http://127.0.0.1:8080/path/to/endpoint")
contentType(ContentType.Application.Json)
body = JsonSampleClass(hello = "world")
}
}
But when I write like that to send a Post request to another server (such as a server to get weather), I got:
java.io.IOException: Broken pipe
I don't know that if I write it in a wrong way or just in a wrong place.
for sure, the date class is worth JsonSampleClass, you need to change this class as in the overgrowth response or use HttpResponse.
Example:
runBlocking {
// Sample for making a HTTP Client request
val message = client.post<HttpResponse> { // or your data class
url("url")
contentType(ContentType.Application.Json)
body = your data class
}
}
I have a Spring Cloud Gateway service sitting in front of a number of backend services but currently it does not log very much. I wanted to log responses from backend services whenever they return unexpected response status codes but I've hit on the following problem.
I can log the response body for requests using a modifyResponseBody filter and a RewriteFunction like so:
.route("test") { r ->
r.path("/httpbin/**")
.filters { f ->
f.modifyResponseBody(String::class.java, String::class.java){ exchange, string ->
if(e.response.statusCode.is2xxSuccessful)println("Server error")
Mono.just(string)
}
}.uri("http://httpbin.org")
}
My issue with this method is that I'm parsing the response ByteArray to a String on every response, with the overhead that implies, even though I'm just using the body string on a small subset of those responses.
I've tried instead to implement a custom filter like so:
.route("test2") {r ->
r.path("/httpbin2/**")
.filters { f ->
f.filter(LogUnexpectedResponseFilter()
.apply(LogUnexpectedResponseFilter.Config(listOf(HttpStatus.OK))))
}.uri("http://httpbin.org")
}
class LogUnexpectedResponseFilter : AbstractGatewayFilterFactory<LogUnexpectedResponseFilter.Config>() {
val logger = LoggerFactory.getLogger(this::class.java)
override fun apply(config: Config?): GatewayFilter {
return GatewayFilter { exchange, chain ->
logger.info("running custom filter")
if (config?.errors!!.contains(exchange.response.statusCode)){
return#GatewayFilter ModifyResponseBodyGatewayFilterFactory().apply(ModifyResponseBodyGatewayFilterFactory.Config()
.setInClass(String::class.java)
.setOutClass(String::class.java)
.setRewriteFunction(String::class.java, String::class.java) { _, body ->
logger.error(body)
Mono.just(body)
}).filter(exchange, chain)
} else {
chain.filter(exchange)
}
}
}
data class Config(val errors: List<HttpStatus>)
}
What this is supposed to do is simply let the request pass through on most requests but apply the log filter on those that I have configured it to (although in this example I'm having it log on 200 status responses).
What I'm seeing when I debug is that it correctly applies the right filter but the RewriteFunction inside it isn't being run at all. What am I missing?
I have spring boot controller
#PostMapping(path = ["/download"])
fun getFile(#RequestBody myObjectRq: myObjectRq, httpServletResponse: HttpServletResponse): CompletableFuture<HttpServletResponse> {
return GlobalScope.async {
val response = webService.getFile(myObjectRq)
response?.let {
httpServletResponse.setHeader("Content-Type", response.headers.get("Content-Type"))
httpServletResponse.setHeader("Content-Disposition", response.headers.get("Content-Disposition"))
httpServletResponse.writer.write(String(response.content.toByteArray()))
httpServletResponse.writer.flush()
httpServletResponse.status = response.status.value
}
httpServletResponse
}.asCompletableFuture()
}
in which I use service which in turn uses ktor client to send post request to external server which should respond sending csv file. csv file content depends on values I send in myObjectRq.
Service:
suspend fun getFile(myObjectRq: myObjectRq): HttpResponse {
val response = ktorClient.post<HttpResponse> {
accept(ContentType.Application.OctetStream)
url(externalWebServerUrl)
body = myObjectRq
contentType(ContentType.Application.Json)
}
log.info(String(response.content.toByteArray()))
response
}
Headers in response are properly set, also log.info(String(response.content.toByteArray())) in the method prints out the content of received file, but I can't set it as a body of HttpServletResponse. I keep getting org.springframework.web.HttpMediaTypeNotAcceptableException: Could not find acceptable representation.
Also I get Inappropriate blocking method call for httpServletResponse.writer which kind of breaks async qualities of ktor client.
What do I do wrong? How should I solve it?
So, I think SpringBoot is confused with your return type. It is trying to find a way to serialize your return CompletableFuture<HttpServletResponse> into the body of the HTTP response but failing. I believe you can achieve the same result by changing your implementation as follows:
#PostMapping(path = ["/download"])
fun getFile(#RequestBody myObjectRq: myObjectRq, httpServletResponse: HttpServletResponse): CompletableFuture<Void> {
return GlobalScope.async {
val response = webService.getFile(myObjectRq)
response?.let {
httpServletResponse.setHeader("Content-Type", response.headers.get("Content-Type"))
httpServletResponse.setHeader("Content-Disposition", response.headers.get("Content-Disposition"))
httpServletResponse.writer.write(String(response.content.toByteArray()))
httpServletResponse.writer.flush()
httpServletResponse.status = response.status.value
}
null
}.asCompletableFuture()
}
I actually managed to solve this using CompletableFuture<ResponseEntity<ByteArray>> as return type and setting body of the response this way:
ResponseEntity.ok().body(response.content.toByteArray())
This also removed Inappropriate blocking method call warnings.