Ktor Server/Application request/response body logging - kotlin

Is there any way to log the request and response body from the ktor server communication?
The buildin CallLogging feature only logs the metadata of a call. I tried writing my own logging feature like in this example: https://github.com/Koriit/ktor-logging/blob/master/src/main/kotlin/korrit/kotlin/ktor/features/logging/Logging.kt
class Logging(private val logger: Logger) {
class Configuration {
var logger: Logger = LoggerFactory.getLogger(Logging::class.java)
}
private suspend fun logRequest(call: ApplicationCall) {
logger.info(StringBuilder().apply {
appendLine("Received request:")
val requestURI = call.request.path()
appendLine(call.request.origin.run { "${method.value} $scheme://$host:$port$requestURI $version" })
call.request.headers.forEach { header, values ->
appendLine("$header: ${values.firstOrNull()}")
}
try {
appendLine()
appendLine(String(call.receive<ByteArray>()))
} catch (e: RequestAlreadyConsumedException) {
logger.error("Logging payloads requires DoubleReceive feature to be installed with receiveEntireContent=true", e)
}
}.toString())
}
private suspend fun logResponse(call: ApplicationCall, subject: Any) {
logger.info(StringBuilder().apply {
appendLine("Sent response:")
appendLine("${call.request.httpVersion} ${call.response.status()}")
call.response.headers.allValues().forEach { header, values ->
appendLine("$header: ${values.firstOrNull()}")
}
when (subject) {
is TextContent -> appendLine(subject.text)
is OutputStreamContent -> appendLine() // ToDo: How to get response body??
else -> appendLine("unknown body type")
}
}.toString())
}
/**
* Feature installation.
*/
fun install(pipeline: Application) {
pipeline.intercept(ApplicationCallPipeline.Monitoring) {
logRequest(call)
proceedWith(subject)
}
pipeline.sendPipeline.addPhase(responseLoggingPhase)
pipeline.sendPipeline.intercept(responseLoggingPhase) {
logResponse(call, subject)
}
}
companion object Feature : ApplicationFeature<Application, Configuration, Logging> {
override val key = AttributeKey<Logging>("Logging Feature")
val responseLoggingPhase = PipelinePhase("ResponseLogging")
override fun install(pipeline: Application, configure: Configuration.() -> Unit): Logging {
val configuration = Configuration().apply(configure)
return Logging(configuration.logger).apply { install(pipeline) }
}
}
}
It works fine for logging the request body using the DoubleReceive plugin. And if the response is plain text i can log the response as the subject in the sendPipeline interception will be of type TextContent or like in the example ByteArrayContent.
But in my case i am responding a data class instance with Jackson ContentNegotiation. In this case the subject is of type OutputStreamContent and i see no options to geht the serialized body from it.
Any idea how to log the serialized response json in my logging feature? Or maybe there is another option using the ktor server? I mean i could serialize my object manually and respond plain text, but thats an ugly way to do it.

I'm not shure about if this is the best way to do it, but here it is:
public fun ApplicationResponse.toLogString(subject: Any): String = when(subject) {
is TextContent -> subject.text
is OutputStreamContent -> {
val channel = ByteChannel(true)
runBlocking {
(subject as OutputStreamContent).writeTo(channel)
val buffer = StringBuilder()
while (!channel.isClosedForRead) {
channel.readUTF8LineTo(buffer)
}
buffer.toString()
}
}
else -> String()
}

Related

Log request id with SLF4J MDC in Spring webflux with coroutines

I am trying to log the request id and other properties like request path from requests to the Spring webflux endpoint with RouterFunctions. The issue is when I put in the MDC information it gets lost at some point for example when making a database request with R2DBC. I was wondering what a possible solution was since none of the solutions I was able to find worked. Below is some of the code that I am using for my project:
RouterFunction code:
fun routes(
testResource: TestResource,
errorResource: ErrorResource
): RouterFunction<ServerResponse> = coRouter {
filter(::filterMDC)
filter(::filterException)
"api".nest {
"/v1/test/{test}".nest {
GET("", testResource::test)
}
}
}
filterMDC code:
suspend fun filterMDC(request: ServerRequest, next: suspend (ServerRequest) -> ServerResponse): ServerResponse {
MDC.put(
REQUEST_ID_MDC_PARAM,
request.headers().firstHeader(REQUEST_ID_HEADER_NAME) ?: UUID.randomUUID().toString()
)
....
MDC.put(
RESOURCE_MDC_PARAM,
"${request.methodName()} ${request.uri()}"
)
return next(request)
}
testResource test code:
suspend fun test(
serverRequest: ServerRequest
): ServerResponse {
logger.info(
"foo" to "bar"
) { "foobar" }
val user = withContext(MDCContext()) {
testRepository.getUser()
}
return ServerResponse.ok().bodyValueAndAwait(user.let(::toApiResponse))
}
TestRepository code:
class TestRepository(
private val r2dbcEntityTemplate: R2dbcEntityTemplate
) {
suspend fun getUser(): User {
return r2dbcEntityTemplate
.select<User>()
.from("users")
.awaitOne()
}
}
I have tried to use MDCContext from SLF4J for coroutines but that didn't solve the issue with the MDC context being lost. Is there a good way to solve this?

rxjava, how to inspect the result of a Single

using kotlin, having code
fun fetchRemoteDataApi(): Single<RemoteDataResponse> = networkApi.getData()
// it is just a retrofit
#GET(".../api/getData")
fun getData() : Single<RemoteDataResponse>
fun mergeApiWithDb(): Completable = fetchRemoteDataApi()
.zipWith(localDao.getAll())
.flatMapCompletable { (remoteData, localData) ->
doMerge(remoteData, localData) //<== return a Completable
}
the code flow:
val mergeApiDbCall = mergeApiWithDb().onErrorComplete().cache() //<=== would like do some inspection at this level
PublishSubject.create<Unit>().toFlowable(BackpressureStrategy.LATEST)
.compose(Transformers.flowableIO())
.switchMap {
//merge DB with api, or local default value first then listen to DB change
mergeApiDbCall.andThen(listAllTopics())
.concatMapSingle { topics -> remoteTopicUsers.map { topics to it } }
}
.flatMapCompletable { (topics, user) ->
// do something return Completable
}
.subscribe({
...
}, { throwable ->
...
})
and when making the call
val mergeApiDbCall = mergeApiWithDb().onErrorComplete().cache()
the question is if would like to inspect on the Singles<RemoteDataResponse> returned from fetchRemoteDataApi() (i.e. using Log.i(...) to printout the content of RemoteDataResponse, etc.), either in got error or success case, how to do it?
/// the functions
fun listAllTopics(): Flowable<List<String>> = localRepoDao.getAllTopics()
// which a DAO:
#Query("SELECT topic FROM RemoteDataTable WHERE read = 1")
fun getAllTopics(): Flowable<List<String>>
///
private val remoteTopicUsers: Single<List<User>>
get() {
return Single.create {
networkApi.getTopicUsers(object : ICallback.IGetTopicUsersCallback {
override fun onSuccess(result: List<User>) = it.onSuccess(result)
override fun onError(errorCode: Int, errorMsg: String?) = it.onError(Exception(errorCode, errorMsg))
})
}
}
You cannot extract information about elements from the Completable. Though you can use doOnComplete() on Completable, it will not provide you any information about the element.
You can inspect elements if you call doOnSuccess() on your Single, so you need to incorporate this call earlier in your code. To inspect errors you can use doOnError() on both Completable or Single.

How can I override logRequest/logResponse to log custom message in Ktor client logging?

Currently, the ktor client logging implementation is as below, and it works as intended but not what I wanted to have.
public class Logging(
public val logger: Logger,
public var level: LogLevel,
public var filters: List<(HttpRequestBuilder) -> Boolean> = emptyList()
)
....
private suspend fun logRequest(request: HttpRequestBuilder): OutgoingContent? {
if (level.info) {
logger.log("REQUEST: ${Url(request.url)}")
logger.log("METHOD: ${request.method}")
}
val content = request.body as OutgoingContent
if (level.headers) {
logger.log("COMMON HEADERS")
logHeaders(request.headers.entries())
logger.log("CONTENT HEADERS")
logHeaders(content.headers.entries())
}
return if (level.body) {
logRequestBody(content)
} else null
}
Above creates a nightmare while looking at the logs because it's logging in each line. Since I'm a beginner in Kotlin and Ktor, I'd love to know the way to change the behaviour of this. Since in Kotlin, all classes are final unless opened specifically, I don't know how to approach on modifying the logRequest function behaviour. What I ideally wanted to achieve is something like below for an example.
....
private suspend fun logRequest(request: HttpRequestBuilder): OutgoingContent? {
...
if (level.body) {
val content = request.body as OutgoingContent
return logger.log(value("url", Url(request.url)),
value("method", request.method),
value("body", content))
}
Any help would be appreciative
No way to actually override a private method in a non-open class, but if you just want your logging to work differently, you're better off with a custom interceptor of the same stage in the pipeline:
val client = HttpClient(CIO) {
install("RequestLogging") {
sendPipeline.intercept(HttpSendPipeline.Monitoring) {
logger.info(
"Request: {} {} {} {}",
context.method,
Url(context.url),
context.headers.entries(),
context.body
)
}
}
}
runBlocking {
client.get<String>("https://google.com")
}
This will produce the logging you want. Of course, to properly log POST you will need to do some extra work.
Maybe this will be useful for someone:
HttpClient() {
install("RequestLogging") {
responsePipeline.intercept(HttpResponsePipeline.After) {
val request = context.request
val response = context.response
kermit.d(tag = "Network") {
"${request.method} ${request.url} ${response.status}"
}
GlobalScope.launch(Dispatchers.Unconfined) {
val responseBody =
response.content.tryReadText(response.contentType()?.charset() ?: Charsets.UTF_8)
?: "[response body omitted]"
kermit.d(tag = "Network") {
"${request.method} ${request.url} ${response.status}\nBODY START" +
"\n$responseBody" +
"\nBODY END"
}
}
}
}
}
You also need to add a method from the Ktor Logger.kt class to your calss with HttpClient:
internal suspend inline fun ByteReadChannel.tryReadText(charset: Charset): String? = try {
readRemaining().readText(charset = charset)
} catch (cause: Throwable) {
null
}

How to be notified when all futures in Future.compose chain succeeded?

My application (typical REST server that calls other REST services internally) has two main classes to perform the bootstrapping procedure.
There is the Application.kt class that is supposed to configure the vertx instance itself and to register certain modules (jackson kotlin integration for example):
class Application(
private val profileSetting: String? = System.getenv("ACTIVE_PROFILES"),
private val logger: Logger = LoggerFactory.getLogger(Application::class.java)!!
) {
fun bootstrap() {
val profiles = activeProfiles()
val meterRegistry = configureMeters()
val vertx = bootstrapVertx(meterRegistry)
vertx.deployVerticle(ApplicationBootstrapVerticle(profiles)) { startup ->
if (startup.succeeded()) {
logger.info("Application startup finished")
} else {
logger.error("Application startup failed", startup.cause())
vertx.close()
}
}
}
}
In addition there is a ApplicationBootstrapVerticle.kt class that is supposed to deploy the different verticles in a defined order. Some of them in sequence, some of them in parallel:
class ApplicationBootstrapVerticle(
private val profiles: List<String>,
private val logger: Logger = LoggerFactory.getLogger(ApplicationBootstrapVerticle::class.java)
) : AbstractVerticle() {
override fun start(startFuture: Future<Void>) {
initializeApplicationConfig().compose {
logger.info("Application configuration initialized")
initializeScheduledJobs()
}.compose {
logger.info("Scheduled jobs initialized")
initializeRestEndpoints()
}.compose {
logger.info("Http server started")
startFuture
}.setHandler { ar ->
if (ar.succeeded()) {
startFuture.complete()
} else {
startFuture.fail(ar.cause())
}
}
}
private fun initializeApplicationConfig(): Future<String> {
return Future.future<String>().also {
vertx.deployVerticle(
ApplicationConfigVerticle(profiles),
it.completer()
)
}
}
private fun initializeScheduledJobs(): CompositeFuture {
val stationsJob = Future.future<String>()
val capabilitiesJob = Future.future<String>()
return CompositeFuture.all(stationsJob, capabilitiesJob).also {
vertx.deployVerticle(
StationQualitiesVerticle(),
stationsJob.completer()
)
vertx.deployVerticle(
VideoCapabilitiesVerticle(),
capabilitiesJob.completer()
)
}
}
private fun initializeRestEndpoints(): Future<String> {
return Future.future<String>().also {
vertx.deployVerticle(
RestEndpointVerticle(dispatcherFactory = RouteDispatcherFactory(vertx)),
it.completer()
)
}
}
}
I am not sure if this is the supposed way to bootstrap an application, if there is any. More important though, I am not sure if I understand the Future.compose mechanics correctly.
The application starts up successfully and I see all desired log messages except the
Application startup finished
message. Also the following code is never called in case of successs:
}.setHandler { ar ->
if (ar.succeeded()) {
startFuture.complete()
} else {
startFuture.fail(ar.cause())
}
}
In case of an failure though, for example when my application configuration files (yaml) cannot be parsed because there is an unknown field in the destination entity, the log message
Application startup failed
appears in the logs and also the code above is invoked.
I am curious what is wrong with my composed futures chain. I thought that the handler would be called after the previous futures succeeded or one of them failed but I think it's only called in case of success.
Update
I suppose that an invocation of startFuture.complete() was missing. By adapting the start method, it finally worked:
override fun start(startFuture: Future<Void>) {
initializeApplicationConfig().compose {
logger.info("Application configuration initialized")
initializeScheduledJobs()
}.compose {
logger.info("Scheduled jobs initialized")
initializeRestEndpoints()
}.compose {
logger.info("Http server started")
startFuture.complete()
startFuture
}.setHandler(
startFuture.completer()
)
}
I am not sure though, if this is the supposed way to handle this future chain.
The solution that worked for me looks like this:
override fun start(startFuture: Future<Void>) {
initializeApplicationConfig().compose {
logger.info("Application configuration initialized")
initializeScheduledJobs()
}.compose {
logger.info("Scheduled jobs initialized")
initializeRestEndpoints()
}.setHandler { ar ->
if(ar.succeeded()) {
logger.info("Http server started")
startFuture.complete()
} else {
startFuture.fail(ar.cause())
}
}
}

How to log requests in ktor http client?

I got something like this:
private val client = HttpClient {
install(JsonFeature) {
serializer = GsonSerializer()
}
install(ExpectSuccess)
}
and make request like
private fun HttpRequestBuilder.apiUrl(path: String, userId: String? = null) {
header(HttpHeaders.CacheControl, "no-cache")
url {
takeFrom(endPoint)
encodedPath = path
}
}
but I need to check request and response body, is there any way to do it? in console/in file?
You can achieve this with the Logging feature.
First add the dependency:
implementation "io.ktor:ktor-client-logging-native:$ktor_version"
Then install the feature:
private val client = HttpClient {
install(Logging) {
logger = Logger.DEFAULT
level = LogLevel.ALL
}
}
Bonus:
If you need to have multiple HttpClient instances throughout your application and you want to reuse some of the configuration, then you can create an extension function and add the common logic in there. For example:
fun HttpClientConfig<*>.default() {
install(Logging) {
logger = Logger.DEFAULT
level = LogLevel.ALL
}
// Add all the common configuration here.
}
And then initialize your HttpClient like this:
private val client = HttpClient {
default()
}
I ran into this as well. I switched to using the Ktor OkHttp client as I'm familiar with the logging mechanism there.
Update your pom.xml or gradle.build to include that client (copy/paste from the Ktor site) and also add the OkHttp Logging Interceptor (again, copy/paste from that site). Current version is 3.12.0.
Now configure the client with
val client = HttpClient(OkHttp) {
engine {
val loggingInterceptor = HttpLoggingInterceptor()
loggingInterceptor.level = Level.BODY
addInterceptor(loggingInterceptor)
}
}
Regardless of which client you use or framework you are on, you can implement your own logger like so:
private val client = HttpClient {
// Other configurations...
install(Logging) {
logger = CustomHttpLogger()
level = LogLevel.BODY
}
}
Where CustomHttpLogger is any class that implements the ktor Logger interface, like so:
import io.ktor.client.features.logging.Logger
class CustomHttpLogger(): Logger {
override fun log(message: String) {
Log.d("loggerTag", message) // Or whatever logging system you want here
}
}
You can read more about the Logger interface in the documentation here or in the source code here
It looks like we should handle the response in HttpReceivePipeline. We could clone the origin response and use it for logging purpose:
scope.receivePipeline.intercept(HttpReceivePipeline.Before) { response ->
val (loggingContent, responseContent) = response.content.split(scope)
launch {
val callForLog = DelegatedCall(loggingContent, context, scope, shouldClose = false)
....
}
...
}
The example implementation could be found here: https://github.com/ktorio/ktor/blob/00369bf3e41e91d366279fce57b8f4c97f927fd4/ktor-client/ktor-client-core/src/io/ktor/client/features/observer/ResponseObserver.kt
and would be available in next minor release as a client feature.
btw: we could implement the same scheme for the request.
A custom structured log can be created with the HttpSend plugin
Ktor 2.x:
client.plugin(HttpSend).intercept { request ->
val call = execute(request)
val response = call.response
val durationMillis = response.responseTime.timestamp - response.requestTime.timestamp
Log.i("NETWORK", "[${response.status.value}] ${request.url.build()} ($durationMillis ms)")
call
}
Ktor 1.x:
client.config {
install(HttpSend) {
intercept { call, _ ->
val request = call.request
val response = call.response
val durationMillis = response.responseTime.timestamp - response.requestTime.timestamp
Log.i("NETWORK", "[${response.status.value}] ${request.url} ($durationMillis ms)")
call
}
}
}
Check out Kotlin Logging, https://github.com/MicroUtils/kotlin-logging it isused by a lot of open source frameworks and takes care of all the prety printing.
You can use it simply like this:
private val logger = KotlinLogging.logger { }
logger.info { "MYLOGGER INFO" }
logger.warn { "MYLOGGER WARNING" }
logger.error { "MYLOGGER ERROR" }
This will print the messages on the console.