How to log requests in ktor http client? - kotlin

I got something like this:
private val client = HttpClient {
install(JsonFeature) {
serializer = GsonSerializer()
}
install(ExpectSuccess)
}
and make request like
private fun HttpRequestBuilder.apiUrl(path: String, userId: String? = null) {
header(HttpHeaders.CacheControl, "no-cache")
url {
takeFrom(endPoint)
encodedPath = path
}
}
but I need to check request and response body, is there any way to do it? in console/in file?

You can achieve this with the Logging feature.
First add the dependency:
implementation "io.ktor:ktor-client-logging-native:$ktor_version"
Then install the feature:
private val client = HttpClient {
install(Logging) {
logger = Logger.DEFAULT
level = LogLevel.ALL
}
}
Bonus:
If you need to have multiple HttpClient instances throughout your application and you want to reuse some of the configuration, then you can create an extension function and add the common logic in there. For example:
fun HttpClientConfig<*>.default() {
install(Logging) {
logger = Logger.DEFAULT
level = LogLevel.ALL
}
// Add all the common configuration here.
}
And then initialize your HttpClient like this:
private val client = HttpClient {
default()
}

I ran into this as well. I switched to using the Ktor OkHttp client as I'm familiar with the logging mechanism there.
Update your pom.xml or gradle.build to include that client (copy/paste from the Ktor site) and also add the OkHttp Logging Interceptor (again, copy/paste from that site). Current version is 3.12.0.
Now configure the client with
val client = HttpClient(OkHttp) {
engine {
val loggingInterceptor = HttpLoggingInterceptor()
loggingInterceptor.level = Level.BODY
addInterceptor(loggingInterceptor)
}
}

Regardless of which client you use or framework you are on, you can implement your own logger like so:
private val client = HttpClient {
// Other configurations...
install(Logging) {
logger = CustomHttpLogger()
level = LogLevel.BODY
}
}
Where CustomHttpLogger is any class that implements the ktor Logger interface, like so:
import io.ktor.client.features.logging.Logger
class CustomHttpLogger(): Logger {
override fun log(message: String) {
Log.d("loggerTag", message) // Or whatever logging system you want here
}
}
You can read more about the Logger interface in the documentation here or in the source code here

It looks like we should handle the response in HttpReceivePipeline. We could clone the origin response and use it for logging purpose:
scope.receivePipeline.intercept(HttpReceivePipeline.Before) { response ->
val (loggingContent, responseContent) = response.content.split(scope)
launch {
val callForLog = DelegatedCall(loggingContent, context, scope, shouldClose = false)
....
}
...
}
The example implementation could be found here: https://github.com/ktorio/ktor/blob/00369bf3e41e91d366279fce57b8f4c97f927fd4/ktor-client/ktor-client-core/src/io/ktor/client/features/observer/ResponseObserver.kt
and would be available in next minor release as a client feature.
btw: we could implement the same scheme for the request.

A custom structured log can be created with the HttpSend plugin
Ktor 2.x:
client.plugin(HttpSend).intercept { request ->
val call = execute(request)
val response = call.response
val durationMillis = response.responseTime.timestamp - response.requestTime.timestamp
Log.i("NETWORK", "[${response.status.value}] ${request.url.build()} ($durationMillis ms)")
call
}
Ktor 1.x:
client.config {
install(HttpSend) {
intercept { call, _ ->
val request = call.request
val response = call.response
val durationMillis = response.responseTime.timestamp - response.requestTime.timestamp
Log.i("NETWORK", "[${response.status.value}] ${request.url} ($durationMillis ms)")
call
}
}
}

Check out Kotlin Logging, https://github.com/MicroUtils/kotlin-logging it isused by a lot of open source frameworks and takes care of all the prety printing.
You can use it simply like this:
private val logger = KotlinLogging.logger { }
logger.info { "MYLOGGER INFO" }
logger.warn { "MYLOGGER WARNING" }
logger.error { "MYLOGGER ERROR" }
This will print the messages on the console.

Related

Is there any way to drop request inside plugin

Now I'm developing server application with ktor 2(2.0.0-eap-256).
What I want to do is, according to header or other information, Reject or set adequate http status to response and do not let request go into service logic.
Below is What I tried.
val testPlugin = createApplication("testPlugin") {
onCall {
if (call.request.headers["auth"] == null) {
call.respond(HttpStatusCode.BadRequest)
return#onCall
}
}
}
fun Application.testRouting() {
routing {
get("/") { call.respond("hello") }
}
}
fun Application.applyPlugin() {
install(testPlugin)
}
But request goes into service logic defined by routing(with response which has HttpStatusCode.BadRequest). Is there any idea?
And also, I want to ask my understand about onCall/onCallReceive/onCallRespond is right
onCall is invoked first, when request come.
then, onCallReceive is invoked to handle request data such as file, body, etc
after all service logic, onCallRespond is invoked.
Edit
About the last question, it is solved. onCallReceive is called when I invoke call.receive() to get request content
Edit
Add routing code
Edit
So, I edit plugin like this.
val testPlugin = createApplication(
name = "testPlugin",
createConfiguration = { TestPluginConfig() }
) {
pluginConfig.apply {
pipeline!!.intercept(ApplicationCallPipeline.Plugins){
if (call.request.headers["auth"] == null) {
call.respond(HttpStatusCode.BadRequest)
finish()
}
}
}
}
data class TestPluginConfig(
var pipeline: Application? = null // io.ktor.sever.Application
)
fun Application.testRouting() {
routing {
get("/") { call.respond("hello") }
}
}
fun Application.applyPlugin() {
val pipeline = this // io.ktor.sever.Application
install(testPlugin) { pipeline = pipeline }
}
It works just as I wanted
very thanks to Aleksei Tirman
According to Rustam Siniukov on the kotlin slack here it's enough to use call.respond in the plugin.
My tests confirmed this.

Micronaut-Core: How to create dynamic endpoints

Simple question. Is it possible to create endpoints without #Endpoint?
I want to create rather dynamic endpoints by a file and depending on the content to its context.
Thanks!
Update about my idea. I would to create something like a plugin system, to make my application more extensible for maintenance and future features.
It is worth to be mentioned I am using Micronaut with Kotlin. Right now I've got fixed defined Endpoints, which matches my command scripts.
My description files will be under /src/main/resources
I've got following example description file how it might look like.
ENDPOINT: GET /myapi/customendpoint/version
COMMAND: """
#!/usr/bin/env bash
# This will be executed via SSH and streamed to stdout for further handling
echo "1.0.0"
"""
# This is a template JSON which will generate a JSON as production on the endpoint
OUTPUT: """
{
"version": "Server version: $RESULT"
}
"""
How I would like to make it work with the application.
import io.micronaut.docs.context.events.SampleEvent
import io.micronaut.context.event.StartupEvent
import io.micronaut.context.event.ShutdownEvent
import io.micronaut.runtime.event.annotation.EventListener
#Singleton
class SampleEventListener {
/*var invocationCounter = 0
#EventListener
internal fun onSampleEvent(event: SampleEvent) {
invocationCounter++
}*/
#EventListener
internal fun onStartupEvent(event: StartupEvent) {
// 1. I read all my description files
// 2. Parse them (for what I created a parser)
// 3. Now the tricky part, how to add those information to Micronaut Runtime
val do = MyDescription() // After I parsed
// Would be awesome if it is that simple! :)
Micronaut.addEndpoint(
do.getEndpoint(), do.getHttpOption(),
MyCustomRequestHandler(do.getCommand()) // Maybe there is a base class for inheritance?
)
}
#EventListener
internal fun onShutdownEvent(event: ShutdownEvent) {
// shutdown logic here
}
}
You can create a custom RouteBuilder that will register your custom endpoints at runtime:
#Singleton
class CustomRouteBuilder extends DefaultRouteBuilder {
#PostConstruct
fun initRoutes() {
val do = MyDescription();
val method = do.getMethod();
val routeUri = do.getEndpoint();
val routeHandle = MethodExecutionHandle<Object, Object>() {
// implement the 'MethodExecutionHandle' in a suitable manner to invoke the 'do.getCommand()'
};
buildRoute(HttpMethod.parse(method), routeUri, routeHandle);
}
}
Note that while this would still feasible, it would be better to consider another extension path as the solution defeats the whole Micronaut philosophy of being an AOT compilation framework.
It was actually pretty easy. The solution for me was to implement a HttpServerFilter.
#Filter("/api/sws/custom/**")
class SwsRouteFilter(
private val swsService: SwsService
): HttpServerFilter {
override fun doFilter(request: HttpRequest<*>?, chain: ServerFilterChain?): Publisher<MutableHttpResponse<*>> {
return Flux.from(Mono.fromCallable {
runBlocking {
swsService.execute(request)
}
}.subscribeOn(Schedulers.boundedElastic()).flux())
}
}
And the service can process with the HttpRequest object:
suspend fun execute(request: HttpRequest<*>?): MutableHttpResponse<Feedback> {
val path = request!!.path.split("/api/sws/custom")[1]
val httpMethod = request.method
val parameters: Map<String, List<String>> = request.parameters.asMap()
// TODO: Handle request body
// and do your stuff ...
}

Ktor Server/Application request/response body logging

Is there any way to log the request and response body from the ktor server communication?
The buildin CallLogging feature only logs the metadata of a call. I tried writing my own logging feature like in this example: https://github.com/Koriit/ktor-logging/blob/master/src/main/kotlin/korrit/kotlin/ktor/features/logging/Logging.kt
class Logging(private val logger: Logger) {
class Configuration {
var logger: Logger = LoggerFactory.getLogger(Logging::class.java)
}
private suspend fun logRequest(call: ApplicationCall) {
logger.info(StringBuilder().apply {
appendLine("Received request:")
val requestURI = call.request.path()
appendLine(call.request.origin.run { "${method.value} $scheme://$host:$port$requestURI $version" })
call.request.headers.forEach { header, values ->
appendLine("$header: ${values.firstOrNull()}")
}
try {
appendLine()
appendLine(String(call.receive<ByteArray>()))
} catch (e: RequestAlreadyConsumedException) {
logger.error("Logging payloads requires DoubleReceive feature to be installed with receiveEntireContent=true", e)
}
}.toString())
}
private suspend fun logResponse(call: ApplicationCall, subject: Any) {
logger.info(StringBuilder().apply {
appendLine("Sent response:")
appendLine("${call.request.httpVersion} ${call.response.status()}")
call.response.headers.allValues().forEach { header, values ->
appendLine("$header: ${values.firstOrNull()}")
}
when (subject) {
is TextContent -> appendLine(subject.text)
is OutputStreamContent -> appendLine() // ToDo: How to get response body??
else -> appendLine("unknown body type")
}
}.toString())
}
/**
* Feature installation.
*/
fun install(pipeline: Application) {
pipeline.intercept(ApplicationCallPipeline.Monitoring) {
logRequest(call)
proceedWith(subject)
}
pipeline.sendPipeline.addPhase(responseLoggingPhase)
pipeline.sendPipeline.intercept(responseLoggingPhase) {
logResponse(call, subject)
}
}
companion object Feature : ApplicationFeature<Application, Configuration, Logging> {
override val key = AttributeKey<Logging>("Logging Feature")
val responseLoggingPhase = PipelinePhase("ResponseLogging")
override fun install(pipeline: Application, configure: Configuration.() -> Unit): Logging {
val configuration = Configuration().apply(configure)
return Logging(configuration.logger).apply { install(pipeline) }
}
}
}
It works fine for logging the request body using the DoubleReceive plugin. And if the response is plain text i can log the response as the subject in the sendPipeline interception will be of type TextContent or like in the example ByteArrayContent.
But in my case i am responding a data class instance with Jackson ContentNegotiation. In this case the subject is of type OutputStreamContent and i see no options to geht the serialized body from it.
Any idea how to log the serialized response json in my logging feature? Or maybe there is another option using the ktor server? I mean i could serialize my object manually and respond plain text, but thats an ugly way to do it.
I'm not shure about if this is the best way to do it, but here it is:
public fun ApplicationResponse.toLogString(subject: Any): String = when(subject) {
is TextContent -> subject.text
is OutputStreamContent -> {
val channel = ByteChannel(true)
runBlocking {
(subject as OutputStreamContent).writeTo(channel)
val buffer = StringBuilder()
while (!channel.isClosedForRead) {
channel.readUTF8LineTo(buffer)
}
buffer.toString()
}
}
else -> String()
}

How can I override logRequest/logResponse to log custom message in Ktor client logging?

Currently, the ktor client logging implementation is as below, and it works as intended but not what I wanted to have.
public class Logging(
public val logger: Logger,
public var level: LogLevel,
public var filters: List<(HttpRequestBuilder) -> Boolean> = emptyList()
)
....
private suspend fun logRequest(request: HttpRequestBuilder): OutgoingContent? {
if (level.info) {
logger.log("REQUEST: ${Url(request.url)}")
logger.log("METHOD: ${request.method}")
}
val content = request.body as OutgoingContent
if (level.headers) {
logger.log("COMMON HEADERS")
logHeaders(request.headers.entries())
logger.log("CONTENT HEADERS")
logHeaders(content.headers.entries())
}
return if (level.body) {
logRequestBody(content)
} else null
}
Above creates a nightmare while looking at the logs because it's logging in each line. Since I'm a beginner in Kotlin and Ktor, I'd love to know the way to change the behaviour of this. Since in Kotlin, all classes are final unless opened specifically, I don't know how to approach on modifying the logRequest function behaviour. What I ideally wanted to achieve is something like below for an example.
....
private suspend fun logRequest(request: HttpRequestBuilder): OutgoingContent? {
...
if (level.body) {
val content = request.body as OutgoingContent
return logger.log(value("url", Url(request.url)),
value("method", request.method),
value("body", content))
}
Any help would be appreciative
No way to actually override a private method in a non-open class, but if you just want your logging to work differently, you're better off with a custom interceptor of the same stage in the pipeline:
val client = HttpClient(CIO) {
install("RequestLogging") {
sendPipeline.intercept(HttpSendPipeline.Monitoring) {
logger.info(
"Request: {} {} {} {}",
context.method,
Url(context.url),
context.headers.entries(),
context.body
)
}
}
}
runBlocking {
client.get<String>("https://google.com")
}
This will produce the logging you want. Of course, to properly log POST you will need to do some extra work.
Maybe this will be useful for someone:
HttpClient() {
install("RequestLogging") {
responsePipeline.intercept(HttpResponsePipeline.After) {
val request = context.request
val response = context.response
kermit.d(tag = "Network") {
"${request.method} ${request.url} ${response.status}"
}
GlobalScope.launch(Dispatchers.Unconfined) {
val responseBody =
response.content.tryReadText(response.contentType()?.charset() ?: Charsets.UTF_8)
?: "[response body omitted]"
kermit.d(tag = "Network") {
"${request.method} ${request.url} ${response.status}\nBODY START" +
"\n$responseBody" +
"\nBODY END"
}
}
}
}
}
You also need to add a method from the Ktor Logger.kt class to your calss with HttpClient:
internal suspend inline fun ByteReadChannel.tryReadText(charset: Charset): String? = try {
readRemaining().readText(charset = charset)
} catch (cause: Throwable) {
null
}

Is there a way to specify webSessionManager when using WebTestClientAutoConfiguration?

I am using Webflux in Spring Boot 2.0.3.RELEASE to create REST API. With that implementation, I customize and use the webSessionManager as below.
#EnableWebFluxSecurity
#Configuration
class SecurityConfiguration {
#Bean
fun webSessionManager(): WebSessionManager {
return DefaultWebSessionManager().apply {
sessionIdResolver = HeaderWebSessionIdResolver().apply {
headerName = "X-Sample"
}
sessionStore = InMemoryWebSessionStore()
}
}
// ...
}
And in order to test the REST API, I created a test code as follows. (addUser and signin are extension functions.)
#RunWith(SpringRunner::class)
#SpringBootTest
#AutoConfigureWebTestClient
#FixMethodOrder(MethodSorters.NAME_ASCENDING)
class UserTests {
#Autowired
private lateinit var client: WebTestClient
#Test
fun testGetUserInfo() {
client.addUser(defaultUser)
val sessionKey = client.signin(defaultUser)
client.get().uri(userPath)
.header("X-Sample", sessionKey)
.exchange()
.expectStatus().isOk
.expectBody()
.jsonInStrict("""
{
"user": {
"mail_address": "user#example.com"
}
}
""".trimIndent())
}
// ...
}
The test failed. It is refused by authorization. However, if I start the server and run it from curl it will succeed in the authorization.
After investigating the cause, it turned out that org.springframework.test.web.reactive.server.AbstractMockServerSpec set webSessionManager to DefaultWebSessionManager. Default is used, not the webSessionManager I customized. For this reason, it could not get the session ID.
AbstractMockServerSpec.java#L41
AbstractMockServerSpec.java#L72-L78
How can I change the webSessionManager of AbstractMockServerSpec?
Also, I think that it is better to have the following implementation, what do you think?
abstract class AbstractMockServerSpec<B extends WebTestClient.MockServerSpec<B>>
implements WebTestClient.MockServerSpec<B> {
// ...
private WebSessionManager sessionManager = null;
// ...
#Override
public WebTestClient.Builder configureClient() {
WebHttpHandlerBuilder builder = initHttpHandlerBuilder();
builder.filters(theFilters -> theFilters.addAll(0, this.filters));
if (this.sessionManager != null) {
builder.sessionManager(this.sessionManager);
}
this.configurers.forEach(configurer -> configurer.beforeServerCreated(builder));
return new DefaultWebTestClientBuilder(builder);
}
// ...
}
Spring Framework's AbstractMockServerSpec is providing a method to customize the WebSessionManager already.
Thanks for opening SPR-17094, this problem will be solved with that ticket - the AbstractMockServerSpec is already looking into the application context for infrastructure bits, this should check for a WebSessionManager as well.