How do I monitor request progress in Ktor http client?
For example: I have request like this:
val response = HttpClient().get<String>("https://stackoverflow.com/")
and I want to monitor request progress with progress bar like this:
fun progress(downloaded: Long, contentLength: Long) {
// Update progress bar or whatever
}
How do I set progress() to be called by HttpClient?
edit: This is Kotlin Multiplatform project. Relevant dependencies are:
implementation 'io.ktor:ktor-client-core:1.2.5'
implementation 'io.ktor:ktor-client-cio:1.2.5'
Starting with Ktor 1.6.0, you can react to download progress change using the onDownload extension function exposed by HttpRequestBuilder:
val channel = get<ByteReadChannel>("https://ktor.io/") {
onDownload { bytesSentTotal, contentLength ->
println("Received $bytesSentTotal bytes from $contentLength")
}
}
There is also the onUpload function that can be used to display upload progress:
onUpload { bytesSentTotal, contentLength ->
println("Sent $bytesSentTotal bytes from $contentLength")
}
Here are runnable examples from the Ktor documentation:
download-file
upload-file
How to emit the download progress into a Flow?
I want to observe the download progress by a Flow, so I write a function like this:
suspend fun downloadFile(file: File, url: String): Flow<Int>{
val client = HttpClient(Android)
return flow{
val httpResponse: HttpResponse = client.get(url) {
onDownload { bytesSentTotal, contentLength ->
val progress = (bytesSentTotal * 100f / contentLength).roundToInt()
emit(progress)
}
}
val responseBody: ByteArray = httpResponse.receive()
file.writeBytes(responseBody)
}
}
but the onDownload will be called only once. If I remove the emit(progress) it will work.
#andrey-aksyonov
Related
emphasized textI am trying to use Kotlin Flow to process some data asynchronously and in parallel, and stream the responses to the client as they occur, as opposed to waiting until all the jobs are complete.
After unsuccessfully trying to just send the flow itself to the response, like this: call.respond(HttpStatusCode.OK, flow.toList())
... I tinkered for hours trying to figure it out, and came up with the following. Is this correct? It seems there should be a more idiomatic way of sending a Flow<MyData> as a response, like one can with a Flux<MyData> in Spring Boot.
Also, it seems that using the below method does not cancel the Flow when the HTTP request is cancelled, so how would one cancel it in Ktor?
data class MyData(val number: Int)
class MyService {
fun updateAllJobs(): Flow<MyData> =
flow {
buildList { repeat(10) { add(MyData(Random.nextInt())) } }
// Docs recommend using `onEach` to "delay" elements.
// However, if I delay here instead of in `map`, all elements are held
// and emitted at once at the very end of the cumulative delay.
// .onEach { delay(500) }
.map {
// I want to emit elements in a "stream" as each is computed.
delay(500)
emit(it)
}
}
}
fun Route.jobRouter() {
val service: MyService by inject() // injected with Koin
put("/jobs") {
val flow = service.updateAllJobs()
// Just using the default Jackson mapper for this example.
val mapper = jsonMapper { }
// `respondOutputStream` seems to be the only way to send a Flow as a stream.
call.respondOutputStream(ContentType.Application.Json, HttpStatusCode.OK) {
flow.collect {
println(it)
// The data does not stream without the newline and `flush()` call.
write((mapper.writeValueAsString(it) + "\n").toByteArray())
flush()
}
}
}
}
The best solution I was able to find (although I don't like it) is to use respondBytesWriter to write data to a response body channel. In the handler, a new job to collect the flow is launched to be able to cancel it if the channel is closed for writing (HTTP request is canceled):
fun Route.jobRouter(service: MyService) {
put("/jobs") {
val flow = service.updateAllJobs()
val mapper = jsonMapper {}
call.respondBytesWriter(contentType = ContentType.Application.Json) {
val job = launch {
flow.collect {
println(it)
try {
writeStringUtf8(mapper.writeValueAsString(it))
flush()
} catch (_: ChannelWriteException) {
cancel()
}
}
}
job.join()
}
}
}
I make a request to the server, but there is no body in the response.
Accordingly, the return value type of response is Unit.
suspend fun foo(
url: String,
id: Long
) {
val requestUrl = "$url/Subscriptions?id=${id}"
val response = httpApiClient.delete<Unit>(requestUrl) {
headers {
append(HttpHeaders.Authorization, createRequestToken(token))
}
}
return response
}
How in this case to receive the code of the executed request?
HttpResponseValidator {
validateResponse { response ->
TODO()
}
}
using a similar construction and throwing an error, for example, is not an option, since one http client is used for several requests, and making a new http client for one request is strange. is there any other way out?
You can specify the HttpResponse type as a type argument instead of Unit to get an object that allows you to access the status property (HTTP status code), headers, to receive the body of a response, etc. Here is an example:
import io.ktor.client.HttpClient
import io.ktor.client.engine.apache.*
import io.ktor.client.request.*
import io.ktor.client.statement.*
suspend fun main() {
val client = HttpClient(Apache)
val response = client.get<HttpResponse>("https://httpbin.org/get")
// the response body isn't received yet
println(response.status)
}
I build a web server with Ktor, and want to cache API method result. But I don't know how to get the response body from call.response. Code like below
fun Application.module(){
// before method was called
intercept(ApplicationCallPipeline.Features) {
val cache = redis.get("cache")
if(cache) {
call.respond(cache)
}
return #intercept finish()
}
// after method was called
intercept(ApplicationCallPipeline.Fallback) {
// TODO, how to get call.response.body
redis.set("cache", call.response.body)
}
}
If I can't get the response body, Any other solution to cache the result of an API method in Ktor?
Inside an interceptor, for the ApplicationCallPipeline.Features phase you can add a phase just before the ApplicationSendPipeline.Engine and intercept it to retreive a response body (content):
val phase = PipelinePhase("phase")
call.response.pipeline.insertPhaseBefore(ApplicationSendPipeline.Engine, phase)
call.response.pipeline.intercept(phase) { response ->
val content: ByteReadChannel = when (response) {
is OutgoingContent.ByteArrayContent -> ByteReadChannel(response.bytes())
is OutgoingContent.NoContent -> ByteReadChannel.Empty
is OutgoingContent.ReadChannelContent -> response.readFrom()
is OutgoingContent.WriteChannelContent -> GlobalScope.writer(coroutineContext, autoFlush = true) {
response.writeTo(channel)
}.channel
else -> error("")
}
// Do something with content
}
I'm trying to make a proxy server by ktor, it proxy a http-flv infinite stream, after client close, it should do something to release resource. like
client.get<HttpStatement>(url.value.url).execute { response ->
val channel: ByteReadChannel = response.receive()
while (!channel.isClosedForRead) {
logger.info("status ${call.request.receiveChannel().isClosedForRead}")
val packet = channel.readRemaining(DEFAULT_BUFFER_SIZE.toLong())
logger.info("flushing...")
withContext(Dispatchers.IO) {
write(packet.readBytes())
flush()
}
}
// do something to clean resource
}
try to use router event and flush writer, didn't work
I got something like this:
private val client = HttpClient {
install(JsonFeature) {
serializer = GsonSerializer()
}
install(ExpectSuccess)
}
and make request like
private fun HttpRequestBuilder.apiUrl(path: String, userId: String? = null) {
header(HttpHeaders.CacheControl, "no-cache")
url {
takeFrom(endPoint)
encodedPath = path
}
}
but I need to check request and response body, is there any way to do it? in console/in file?
You can achieve this with the Logging feature.
First add the dependency:
implementation "io.ktor:ktor-client-logging-native:$ktor_version"
Then install the feature:
private val client = HttpClient {
install(Logging) {
logger = Logger.DEFAULT
level = LogLevel.ALL
}
}
Bonus:
If you need to have multiple HttpClient instances throughout your application and you want to reuse some of the configuration, then you can create an extension function and add the common logic in there. For example:
fun HttpClientConfig<*>.default() {
install(Logging) {
logger = Logger.DEFAULT
level = LogLevel.ALL
}
// Add all the common configuration here.
}
And then initialize your HttpClient like this:
private val client = HttpClient {
default()
}
I ran into this as well. I switched to using the Ktor OkHttp client as I'm familiar with the logging mechanism there.
Update your pom.xml or gradle.build to include that client (copy/paste from the Ktor site) and also add the OkHttp Logging Interceptor (again, copy/paste from that site). Current version is 3.12.0.
Now configure the client with
val client = HttpClient(OkHttp) {
engine {
val loggingInterceptor = HttpLoggingInterceptor()
loggingInterceptor.level = Level.BODY
addInterceptor(loggingInterceptor)
}
}
Regardless of which client you use or framework you are on, you can implement your own logger like so:
private val client = HttpClient {
// Other configurations...
install(Logging) {
logger = CustomHttpLogger()
level = LogLevel.BODY
}
}
Where CustomHttpLogger is any class that implements the ktor Logger interface, like so:
import io.ktor.client.features.logging.Logger
class CustomHttpLogger(): Logger {
override fun log(message: String) {
Log.d("loggerTag", message) // Or whatever logging system you want here
}
}
You can read more about the Logger interface in the documentation here or in the source code here
It looks like we should handle the response in HttpReceivePipeline. We could clone the origin response and use it for logging purpose:
scope.receivePipeline.intercept(HttpReceivePipeline.Before) { response ->
val (loggingContent, responseContent) = response.content.split(scope)
launch {
val callForLog = DelegatedCall(loggingContent, context, scope, shouldClose = false)
....
}
...
}
The example implementation could be found here: https://github.com/ktorio/ktor/blob/00369bf3e41e91d366279fce57b8f4c97f927fd4/ktor-client/ktor-client-core/src/io/ktor/client/features/observer/ResponseObserver.kt
and would be available in next minor release as a client feature.
btw: we could implement the same scheme for the request.
A custom structured log can be created with the HttpSend plugin
Ktor 2.x:
client.plugin(HttpSend).intercept { request ->
val call = execute(request)
val response = call.response
val durationMillis = response.responseTime.timestamp - response.requestTime.timestamp
Log.i("NETWORK", "[${response.status.value}] ${request.url.build()} ($durationMillis ms)")
call
}
Ktor 1.x:
client.config {
install(HttpSend) {
intercept { call, _ ->
val request = call.request
val response = call.response
val durationMillis = response.responseTime.timestamp - response.requestTime.timestamp
Log.i("NETWORK", "[${response.status.value}] ${request.url} ($durationMillis ms)")
call
}
}
}
Check out Kotlin Logging, https://github.com/MicroUtils/kotlin-logging it isused by a lot of open source frameworks and takes care of all the prety printing.
You can use it simply like this:
private val logger = KotlinLogging.logger { }
logger.info { "MYLOGGER INFO" }
logger.warn { "MYLOGGER WARNING" }
logger.error { "MYLOGGER ERROR" }
This will print the messages on the console.