I try to write a kotlin multiplatform library (android and ios) that uses ktor. Thereby I experience some issues with kotlins coroutines:
When writing tests I always get kotlinx.coroutines.JobCancellationException: Parent job is Completed; job=JobImpl{Completed}#... exception.
I use ktors mock engine for my tests:
client = HttpClient(MockEngine)
{
engine
{
addHandler
{ request ->
// Create response object
}
}
}
A sample method (commonMain module) using ktor. All methods in my library are written in a similar way. The exception occures if client.get is called.
suspend fun getData(): Either<Exception, String> = coroutineScope
{
// Exception occurs in this line:
val response: HttpResponse = client.get { url("https://www.google.com") }
return if (response.status == HttpStatusCode.OK)
{
(response.readText() as T).right()
}
else
{
Exception("Error").left()
}
}
A sample unit test (commonTest module) for the above method. The assertTrue statement is never called since the exception is thrown before.
#Test
fun getDataTest() = runTest
{
val result = getData()
assertTrue(result.isRight())
}
Actual implementation of runTest in androidTest and iosTest modules.
actual fun<T> runTest(block: suspend () -> T) { runBlocking { block() } }
I thought when I use coroutineScope, it waits until all child coroutines are done. What am I doing wrong and how can I fix this exception?
you can't cache HttpClient of CIO in client variable and reuse, It would be best if change the following code in your implementation.
val client:HttpClient get() = HttpClient(MockEngine) {
engine {
addHandler { request ->
// Create response object
}
}
}
The library must be updated, this glitch is in the fix report here: https://newreleases.io/project/github/ktorio/ktor/release/1.6.1
The problem is that you cannot use the same instance of the HttpClient. My ej:
HttpClient(CIO) {
install(JsonFeature) {
serializer = GsonSerializer()
}
}.use { client ->
return#use client.request("URL") {
method = HttpMethod.Get
}
}
Related
I am trying to retrieve the base url from my proto datastore to be used to initialize my ktor client instance I know how to get the data from the datastore but I don't know how to block execution until that value is received so the client can be initialized with the base url
So my ktor client service asks for a NetworkURLS class which has a method to return the base url
Here is my property to retrieve terminalDetails from my proto datastore
val getTerminalDetails: Flow<TerminalDetails> = cxt.terminalDetails.data
.catch { e ->
if (e is IOException) {
Log.d("Error", e.message.toString())
emit(TerminalDetails.getDefaultInstance())
} else {
throw e
}
}
Normally when I want to get the values I would do something like this
private fun getTerminalDetailsFromStore() {
try {
viewModelScope.launch(Dispatchers.IO) {
localRepository.getTerminalDetails.collect {
_terminalDetails.value = it
}
}
} catch(e: Exception) {
Log.d("AdminSettingsViewModel Error", e.message.toString()) // TODO: Handle Error Properly
}
}
but in my current case what I am looking to do is return terminalDetails.backendHost from a function and that where the issue comes in I know I need to use a coroutine scope to retrieve the value so I don't need to suspend the function but how to a prevent the function returning until the coroutine scope has finished?
I have tried using async and runBlocking but async doesn't work the way I would think it would and runBlocking hangs the entire app
fun backendURL(): String = runBlocking {
var url: String = "localhost"
val job = CoroutineScope(Dispatchers.IO).async {
repo.getTerminalDetails.collect {
it.backendHost
}
}
url
}
Can anyone give me some assistance on getting this to work?
EDIT: Here is my temporary solution, I do not intend on keeping it this way, The issue with runBlocking{} turned out to be the Flow<T> does not finish so runBlocking{} continues to block the app.
fun backendURL(): String {
val details = MutableStateFlow<TerminalDetails>(TerminalDetails.getDefaultInstance())
val job = CoroutineScope(Dispatchers.IO).launch {
repo.getTerminalDetails.collect {
details.value = it
}
}
runBlocking {
delay(250L)
}
return details.value.backendHost
}
EDIT 2: I fully fixed my issue. I created a method with the same name as my val (personal decision) which utilizes runBlocking{} and Flow<T>.first() to block while the value is retrieve. The reason I did not replace my val with the function is there are places where I need the information as well where I can utilize coroutines properly where I am not initializing components on my app
val getTerminalDetails: Flow<TerminalDetails> = cxt.terminalDetails.data
.catch { e ->
if (e is IOException) {
Log.d("Error", e.message.toString())
emit(TerminalDetails.getDefaultInstance())
} else {
throw e
}
}
fun getTerminalDetails(): TerminalDetails = runBlocking {
cxt.terminalDetails.data.first()
}
I'm trying to execute a db transaction with the vertx reactive sql client in a coroutine.
Somehow I can't figure out how I can convert the CompletableFuture to the desired io.vertx.core.Future type. Are there any helper methods or extensions to do this easily ?
val client : PgPool
...
suspend fun someServiceFunction () {
coroutineScope {
client.withTransaction { connection ->
val completableFuture = async {
repository.save(connection, requestDTO) //This is a suspend function
}.asCompletableFuture()
//Return type has to be a io.vertx.core.Future
//How can I transform the completableFuture to it ?
}
}
}
Thank you for your help !
Vert.x Future has a conversion method:
future = Future.fromCompletionStage(completionStage, vertxContext)
I adapted this from the code for asCompletableFuture() to use as an alternative. Disclaimer: I don't use Vert.x and I didn't test this.
fun <T> Deferred<T>.asVertxFuture(): Future<T> {
val promise = Promise.promise<T>()
invokeOnCompletion {
try {
promise.complete(getCompleted())
} catch (t: Throwable) {
promise.fail(t)
}
}
return promise.future()
.onComplete { result ->
cancel(result.cause()?.let {
it as? CancellationException ?: CancellationException("Future was completed exceptionally", it)
})
}
}
I wonder if mixing coroutines in with Vert.x could hurt performance because you're not using the Vert.x thread pools. Maybe you can create a Dispatchers.Vertx that borrows its thread pools.
I have a suspend function
private suspend fun getResponse(record: String): HashMap<String, String> {}
When I call it in my main function I'm doing this, but the type of response is Job, not HashMap, how can I get the correct return type?
override fun handleRequest(event: SQSEvent?, context: Context?): Void? {
event?.records?.forEach {
try {
val response: Job = GlobalScope.launch {
getResponse(it.body)
}
} catch (ex: Exception) {
logger.error("error message")
}
}
return null
}
Given your answers in the comments, it looks like you're not looking for concurrency here. The best course of action would then be to just make getRequest() a regular function instead of a suspend one.
Assuming you can't change this, you need to call a suspend function from a regular one. To do so, you have several options depending on your use case:
block the current thread while you do your async stuff
make handleRequest a suspend function
make handleRequest take a CoroutineScope to start coroutines with some lifecycle controlled externally, but that means handleRequest will return immediately and the caller has to deal with the running coroutines (please don't use GlobalScope for this, it's a delicate API)
Option 2 and 3 are provided for completeness, but most likely in your context these won't work for you. So you have to block the current thread while handleRequest is running, and you can do that using runBlocking:
override fun handleRequest(event: SQSEvent?, context: Context?): Void? {
runBlocking {
// do your stuff
}
return null
}
Now what to do inside runBlocking depends on what you want to achieve.
if you want to process elements sequentially, simply call getResponse directly inside the loop:
override fun handleRequest(event: SQSEvent?, context: Context?): Void? {
runBlocking {
event?.records?.forEach {
try {
val response = getResponse(it.body)
// do something with the response
} catch (ex: Exception) {
logger.error("error message")
}
}
}
return null
}
If you want to process elements concurrently, but independently, you can use launch and put both getResponse() and the code using the response inside the launch:
override fun handleRequest(event: SQSEvent?, context: Context?): Void? {
runBlocking {
event?.records?.forEach {
launch { // coroutine scope provided by runBlocking
try {
val response = getResponse(it.body)
// do something with the response
} catch (ex: Exception) {
logger.error("error message")
}
}
}
}
return null
}
If you want to get the responses concurrently, but process all responses only when they're all done, you can use map + async:
override fun handleRequest(event: SQSEvent?, context: Context?): Void? {
runBlocking {
val responses = event?.records?.mapNotNull {
async { // coroutine scope provided by runBlocking
try {
getResponse(it.body)
} catch (ex: Exception) {
logger.error("error message")
null // if you want to still handle other responses
// you could also throw an exception otherwise
}
}
}.map { it.await() }
// do something with all responses
}
return null
}
You can use GlobalScope.async() instead of launch() - it returns Deferred, which is a future/promise object. You can then call await() on it to get a result of getResponse().
Just make sure not to do something like: async().await() - it wouldn't make any sense, because it would still run synchronously. If you need to run getResponse() on all event.records in parallel, then you can first go in loop and collect all deffered objects and then await on all of them.
Currently, the ktor client logging implementation is as below, and it works as intended but not what I wanted to have.
public class Logging(
public val logger: Logger,
public var level: LogLevel,
public var filters: List<(HttpRequestBuilder) -> Boolean> = emptyList()
)
....
private suspend fun logRequest(request: HttpRequestBuilder): OutgoingContent? {
if (level.info) {
logger.log("REQUEST: ${Url(request.url)}")
logger.log("METHOD: ${request.method}")
}
val content = request.body as OutgoingContent
if (level.headers) {
logger.log("COMMON HEADERS")
logHeaders(request.headers.entries())
logger.log("CONTENT HEADERS")
logHeaders(content.headers.entries())
}
return if (level.body) {
logRequestBody(content)
} else null
}
Above creates a nightmare while looking at the logs because it's logging in each line. Since I'm a beginner in Kotlin and Ktor, I'd love to know the way to change the behaviour of this. Since in Kotlin, all classes are final unless opened specifically, I don't know how to approach on modifying the logRequest function behaviour. What I ideally wanted to achieve is something like below for an example.
....
private suspend fun logRequest(request: HttpRequestBuilder): OutgoingContent? {
...
if (level.body) {
val content = request.body as OutgoingContent
return logger.log(value("url", Url(request.url)),
value("method", request.method),
value("body", content))
}
Any help would be appreciative
No way to actually override a private method in a non-open class, but if you just want your logging to work differently, you're better off with a custom interceptor of the same stage in the pipeline:
val client = HttpClient(CIO) {
install("RequestLogging") {
sendPipeline.intercept(HttpSendPipeline.Monitoring) {
logger.info(
"Request: {} {} {} {}",
context.method,
Url(context.url),
context.headers.entries(),
context.body
)
}
}
}
runBlocking {
client.get<String>("https://google.com")
}
This will produce the logging you want. Of course, to properly log POST you will need to do some extra work.
Maybe this will be useful for someone:
HttpClient() {
install("RequestLogging") {
responsePipeline.intercept(HttpResponsePipeline.After) {
val request = context.request
val response = context.response
kermit.d(tag = "Network") {
"${request.method} ${request.url} ${response.status}"
}
GlobalScope.launch(Dispatchers.Unconfined) {
val responseBody =
response.content.tryReadText(response.contentType()?.charset() ?: Charsets.UTF_8)
?: "[response body omitted]"
kermit.d(tag = "Network") {
"${request.method} ${request.url} ${response.status}\nBODY START" +
"\n$responseBody" +
"\nBODY END"
}
}
}
}
}
You also need to add a method from the Ktor Logger.kt class to your calss with HttpClient:
internal suspend inline fun ByteReadChannel.tryReadText(charset: Charset): String? = try {
readRemaining().readText(charset = charset)
} catch (cause: Throwable) {
null
}
Is there a thread-safe method in Ktor where it is possible to statically access the current ApplicationCall? I am trying to get the following simple example to work;
object Main {
fun start() {
val server = embeddedServer(Jetty, 8081) {
intercept(ApplicationCallPipeline.Call) {
// START: this will be more dynamic in the future, we don't want to pass ApplicationCall
Addon.processRequest()
// END: this will be more dynamic in the future, we don't want to pass ApplicationCall
call.respondText(output, ContentType.Text.Html, HttpStatusCode.OK)
return#intercept finish()
}
}
server.start(wait = true)
}
}
fun main(args: Array<String>) {
Main.start();
}
object Addon {
fun processRequest() {
val call = RequestUtils.getCurrentApplicationCall()
// processing of call.request.queryParameters
// ...
}
}
object RequestUtils {
fun getCurrentApplicationCall(): ApplicationCall {
// Here is where I am getting lost..
return null
}
}
I would like to be able to get the ApplicationCall for the current context to be available statically from the RequestUtils so that I can access information about the request anywhere. This of course needs to scale to be able to handle multiple requests at the same time.
I have done some experiments with dependency inject and ThreadLocal, but to no success.
Well, the application call is passed to a coroutine, so it's really dangerous to try and get it "statically", because all requests are treated in a concurrent context.
Kotlin official documentation talks about Thread-local in the context of coroutine executions. It uses the concept of CoroutineContext to restore Thread-Local values in specific/custom coroutine context.
However, if you are able to design a fully asynchronous API, you will be able to bypass thread-locals by directly creating a custom CoroutineContext, embedding the request call.
EDIT: I've updated my example code to test 2 flavors:
async endpoint: Solution fully based on Coroutine contexts and suspend functions
blocking endpoint: Uses a thread-local to store application call, as referred in kotlin doc.
import io.ktor.server.engine.embeddedServer
import io.ktor.server.jetty.Jetty
import io.ktor.application.*
import io.ktor.http.ContentType
import io.ktor.http.HttpStatusCode
import io.ktor.response.respondText
import io.ktor.routing.get
import io.ktor.routing.routing
import kotlinx.coroutines.asContextElement
import kotlinx.coroutines.launch
import kotlin.coroutines.AbstractCoroutineContextElement
import kotlin.coroutines.CoroutineContext
import kotlin.coroutines.coroutineContext
/**
* Thread local in which you'll inject application call.
*/
private val localCall : ThreadLocal<ApplicationCall> = ThreadLocal();
object Main {
fun start() {
val server = embeddedServer(Jetty, 8081) {
routing {
// Solution requiring full coroutine/ supendable execution.
get("/async") {
// Ktor will launch this block of code in a coroutine, so you can create a subroutine with
// an overloaded context providing needed information.
launch(coroutineContext + ApplicationCallContext(call)) {
PrintQuery.processAsync()
}
}
// Solution based on Thread-Local, not requiring suspending functions
get("/blocking") {
launch (coroutineContext + localCall.asContextElement(value = call)) {
PrintQuery.processBlocking()
}
}
}
intercept(ApplicationCallPipeline.ApplicationPhase.Call) {
call.respondText("Hé ho", ContentType.Text.Plain, HttpStatusCode.OK)
}
}
server.start(wait = true)
}
}
fun main() {
Main.start();
}
interface AsyncAddon {
/**
* Asynchronicity propagates in order to properly access coroutine execution information
*/
suspend fun processAsync();
}
interface BlockingAddon {
fun processBlocking();
}
object PrintQuery : AsyncAddon, BlockingAddon {
override suspend fun processAsync() = processRequest("async", fetchCurrentCallFromCoroutineContext())
override fun processBlocking() = processRequest("blocking", fetchCurrentCallFromThreadLocal())
private fun processRequest(prefix : String, call : ApplicationCall?) {
println("$prefix -> Query parameter: ${call?.parameters?.get("q") ?: "NONE"}")
}
}
/**
* Custom coroutine context allow to provide information about request execution.
*/
private class ApplicationCallContext(val call : ApplicationCall) : AbstractCoroutineContextElement(Key) {
companion object Key : CoroutineContext.Key<ApplicationCallContext>
}
/**
* This is your RequestUtils rewritten as a first-order function. It defines as asynchronous.
* If not, you won't be able to access coroutineContext.
*/
suspend fun fetchCurrentCallFromCoroutineContext(): ApplicationCall? {
// Here is where I am getting lost..
return coroutineContext.get(ApplicationCallContext.Key)?.call
}
fun fetchCurrentCallFromThreadLocal() : ApplicationCall? {
return localCall.get()
}
You can test it in your navigator:
http://localhost:8081/blocking?q=test1
http://localhost:8081/blocking?q=test2
http://localhost:8081/async?q=test3
server log output:
blocking -> Query parameter: test1
blocking -> Query parameter: test2
async -> Query parameter: test3
The key mechanism you want to use for this is the CoroutineContext. This is the place that you can set key value pairs to be used in any child coroutine or suspending function call.
I will try to lay out an example.
First, let us define a CoroutineContextElement that will let us add an ApplicationCall to the CoroutineContext.
class ApplicationCallElement(var call: ApplicationCall?) : AbstractCoroutineContextElement(ApplicationCallElement) {
companion object Key : CoroutineContext.Key<ApplicationCallElement>
}
Now we can define some helpers that will add the ApplicationCall on one of our routes. (This could be done as some sort of Ktor plugin that listens to the pipeline, but I don't want to add to much noise here).
suspend fun PipelineContext<Unit, ApplicationCall>.withCall(
bodyOfCall: suspend PipelineContext<Unit, ApplicationCall>.() -> Unit
) {
val pipeline = this
val appCallContext = buildAppCallContext(this.call)
withContext(appCallContext) {
pipeline.bodyOfCall()
}
}
internal suspend fun buildAppCallContext(call: ApplicationCall): CoroutineContext {
var context = coroutineContext
val callElement = ApplicationCallElement(call)
context = context.plus(callElement)
return context
}
And then we can use it all together like in this test case below where we are able to get the call from a nested suspending function:
suspend fun getSomethingFromCall(): String {
val call = coroutineContext[ApplicationCallElement.Key]?.call ?: throw Exception("Element not set")
return call.parameters["key"] ?: throw Exception("Parameter not set")
}
fun Application.myApp() {
routing {
route("/foo") {
get {
withCall {
call.respondText(getSomethingFromCall())
}
}
}
}
}
class ApplicationCallTest {
#Test
fun `we can get the application call in a nested function`() {
withTestApplication({ myApp() }) {
with(handleRequest(HttpMethod.Get, "/foo?key=bar")) {
assertEquals(HttpStatusCode.OK, response.status())
assertEquals("bar", response.content)
}
}
}
}