I am trying to schedule a task in my Ktor application, however I have not been able to find anything online about how to do this. Does anyone have any recommendations or been able to do this before?
Ktor doesn't have built-in scheduler, so you'd have to implement your own
I've written small class using Java's Executors for this task for myself, you might find it useful
class Scheduler(private val task: Runnable) {
private val executor = Executors.newScheduledThreadPool(1)!!
fun scheduleExecution(every: Every) {
val taskWrapper = Runnable {
task.run()
}
executor.scheduleWithFixedDelay(taskWrapper, every.n, every.n, every.unit)
}
fun stop() {
executor.shutdown()
try {
executor.awaitTermination(1, TimeUnit.HOURS)
} catch (e: InterruptedException) {
}
}
}
data class Every(val n: Long, val unit: TimeUnit)
Related
emphasized textI am trying to use Kotlin Flow to process some data asynchronously and in parallel, and stream the responses to the client as they occur, as opposed to waiting until all the jobs are complete.
After unsuccessfully trying to just send the flow itself to the response, like this: call.respond(HttpStatusCode.OK, flow.toList())
... I tinkered for hours trying to figure it out, and came up with the following. Is this correct? It seems there should be a more idiomatic way of sending a Flow<MyData> as a response, like one can with a Flux<MyData> in Spring Boot.
Also, it seems that using the below method does not cancel the Flow when the HTTP request is cancelled, so how would one cancel it in Ktor?
data class MyData(val number: Int)
class MyService {
fun updateAllJobs(): Flow<MyData> =
flow {
buildList { repeat(10) { add(MyData(Random.nextInt())) } }
// Docs recommend using `onEach` to "delay" elements.
// However, if I delay here instead of in `map`, all elements are held
// and emitted at once at the very end of the cumulative delay.
// .onEach { delay(500) }
.map {
// I want to emit elements in a "stream" as each is computed.
delay(500)
emit(it)
}
}
}
fun Route.jobRouter() {
val service: MyService by inject() // injected with Koin
put("/jobs") {
val flow = service.updateAllJobs()
// Just using the default Jackson mapper for this example.
val mapper = jsonMapper { }
// `respondOutputStream` seems to be the only way to send a Flow as a stream.
call.respondOutputStream(ContentType.Application.Json, HttpStatusCode.OK) {
flow.collect {
println(it)
// The data does not stream without the newline and `flush()` call.
write((mapper.writeValueAsString(it) + "\n").toByteArray())
flush()
}
}
}
}
The best solution I was able to find (although I don't like it) is to use respondBytesWriter to write data to a response body channel. In the handler, a new job to collect the flow is launched to be able to cancel it if the channel is closed for writing (HTTP request is canceled):
fun Route.jobRouter(service: MyService) {
put("/jobs") {
val flow = service.updateAllJobs()
val mapper = jsonMapper {}
call.respondBytesWriter(contentType = ContentType.Application.Json) {
val job = launch {
flow.collect {
println(it)
try {
writeStringUtf8(mapper.writeValueAsString(it))
flush()
} catch (_: ChannelWriteException) {
cancel()
}
}
}
job.join()
}
}
}
Recently I've been trying to familiarize myself with Kotlin some more, so I decided to write a webscraper utilizing coroutines. What I want to accomplish is pull each page, harvest it for links and contents or posts, then feed the links back to the process, until there is nowhere left to go. As of now it has some obvious shortcomings, such no delay enforced between calls or saving addresses and only visiting new ones. But the questions I have are regarding coroutines, here.
Consider the following class,. I've added some toy classes to simulate how it is intended to work, which I won't detail, but you can imagine how they work.
class Scraper(
private val client: Client = ToyClient(delayMillis = 1000, alwaysFindBody = "Test body"),
private val extraction: Extraction = ToyExtraction(
alwaysFindLinks = listOf("https://google.com"),
alwaysFindPosts = listOf("Test post")
),
private val repository: Repository = ToyRepository()
) {
// I could manage my own coroutine scope's lifecycle, but how would I go about this?
// private val scope = CoroutineScope(Dispatchers.Default + SupervisorJob())
private val seed = "https://google.com"
private val log = KotlinLogging.logger {}
fun start() = runBlocking {
log.info { "Scraping started!" }
scrape(seed).join()
log.info { "Scraping finished!" }
}
private fun CoroutineScope.scrape(address: String): Job = launch(Dispatchers.Default) {
log.info { "A scraping coroutine has started" }
val page = request(address)
val contents = extract(page)
save(contents)
contents.links.forEach { scrape(it) }
// Job would not progress here after submitting new jobs, only after each children have been completed
// log.info { "A scraping coroutine has finished" }
}
private suspend fun request(address: String): Page {
log.info { "Getting page: $address" }
return client.get(address)
}
private suspend fun extract(page: Page): PageContents {
log.info { "Extracting page: ${page.address}" }
return extraction.extract(page)
}
private suspend fun save(contents: PageContents) {
log.info { "Processing contents of: $contents" }
repository.save(contents.posts)
}
}
The main recursive operation is CoroutineScope.scrape() which launches a job, which itself can launch children jobs as well and so on.
My main questions are:
If I were to manage the scope myself as a property, how could I do that and achieve the same behavior? That is, I would wait for all dynamically spawned jobs to complete as well, return when all are finished.
I wrote my webclient's function using a 3rd party library as such:
fun suspend get(address: String): Page { ... }
Am I fine just marking this method as suspend to get all benefits from this in terms of coroutines?
Thanks in advance!
You don't even need a scope for that, launch a top-level job and use job.join() to await until it and all its children are done. If you want to block while waiting for that to happen, then you are already doing it right by using runBlocking.
No, marking a function as suspend doesn't affect its blocking behavior. It only allows the function to suspend itself, which must be explicit either in your code or the code you're calling into.
I try to write a kotlin multiplatform library (android and ios) that uses ktor. Thereby I experience some issues with kotlins coroutines:
When writing tests I always get kotlinx.coroutines.JobCancellationException: Parent job is Completed; job=JobImpl{Completed}#... exception.
I use ktors mock engine for my tests:
client = HttpClient(MockEngine)
{
engine
{
addHandler
{ request ->
// Create response object
}
}
}
A sample method (commonMain module) using ktor. All methods in my library are written in a similar way. The exception occures if client.get is called.
suspend fun getData(): Either<Exception, String> = coroutineScope
{
// Exception occurs in this line:
val response: HttpResponse = client.get { url("https://www.google.com") }
return if (response.status == HttpStatusCode.OK)
{
(response.readText() as T).right()
}
else
{
Exception("Error").left()
}
}
A sample unit test (commonTest module) for the above method. The assertTrue statement is never called since the exception is thrown before.
#Test
fun getDataTest() = runTest
{
val result = getData()
assertTrue(result.isRight())
}
Actual implementation of runTest in androidTest and iosTest modules.
actual fun<T> runTest(block: suspend () -> T) { runBlocking { block() } }
I thought when I use coroutineScope, it waits until all child coroutines are done. What am I doing wrong and how can I fix this exception?
you can't cache HttpClient of CIO in client variable and reuse, It would be best if change the following code in your implementation.
val client:HttpClient get() = HttpClient(MockEngine) {
engine {
addHandler { request ->
// Create response object
}
}
}
The library must be updated, this glitch is in the fix report here: https://newreleases.io/project/github/ktorio/ktor/release/1.6.1
The problem is that you cannot use the same instance of the HttpClient. My ej:
HttpClient(CIO) {
install(JsonFeature) {
serializer = GsonSerializer()
}
}.use { client ->
return#use client.request("URL") {
method = HttpMethod.Get
}
}
I have a very simple Kotlin program like
fun main() {
val scope = CoroutineScope(Dispatchers.Default)
val job = scope.launch() { // I only check if this Job isActive later
withTimeout(2000) {
terminate(task)
}
}
}
private suspend fun terminate(task: Task): Nothing = suspendCoroutine {
throw IllegalAccessError("Task ${task.name} should honor timeouts!")
}
When terminate() is called I want my program to blow. I don't want to recover. However, I can't only see
Exception in thread "DefaultDispatcher-worker-2"
abc.xyz.mainKt$terminate$$inlined$suspendCoroutine$lambda$1: Task Robot should honor timeouts!
// More stacktrace ...
in logs, since Coroutines is "swallowing" this Exception.
Therefore, my question is : how would be a guaranteed way to blow my program when a timeout happens, with a design driven by Kotlin Coroutines?
How about this?
fun main() = runBlocking {
withTimeout(2000) {
terminate(task)
}
}
I use webflux with netty and jdbc, so I wrap blocking jdbc operation the next way:
static <T> Mono<T> fromOne(Callable<T> blockingOperation) {
return Mono.fromCallable(blockingOperation)
.subscribeOn(jdbcScheduler)
.publishOn(Schedulers.parallel());
}
Blocking operation will be processed by the jdbcScheduler, and I want the other pipeline will be proccesed by webflux event-loop scheduler.
How to get webflux event-loop scheduler?
I will strongly advise to revisit the technology options. If you are going to use jdbc, which is still blocking, then you should not use webflux. This is because webflux will shine in a non-blocking stack but coupled with Jdbc it will act as a bottleneck. The performance will actually go down.
I agree with #Vikram Rawat use jdbc is very dangerous mainly because jdbc is a bocking IO api and use an event loop reactive model is very dangerous because basically it is very easy block all the server.
However, even if it is an experimental effort I suggest you to stay tuned to R2DBC project that is able to leverage a no blocking api for sql I used it for a spike and it is very elegant.
I can provide you an example taken from a my home project on github based on sprign boot 2.1 and kotlin:
web layer
#Configuration
class ReservationRoutesConfig {
#Bean
fun reservationRoutes(#Value("\${baseServer:http://localhost:8080}") baseServer: String,
reservationRepository: ReservationRepository) =
router {
POST("/reservation") {
it.bodyToMono(ReservationRepresentation::class.java)
.flatMap { Mono.just(ReservationRepresentation.toDomain(reservationRepresentation = it)) }
.flatMap { reservationRepository.save(it).toMono() }
.flatMap { ServerResponse.created(URI("$baseServer/reservation/${it.reservationId}")).build() }
}
GET("/reservation/{reservationId}") {
reservationRepository.findOne(it.pathVariable("reservationId")).toMono()
.flatMap { Mono.just(ReservationRepresentation.toRepresentation(it)) }
.flatMap { ok().body(BodyInserters.fromObject(it)) }
}
DELETE("/reservation/{reservationId}") {
reservationRepository.delete(it.pathVariable("reservationId")).toMono()
.then(noContent().build())
}
}
}
repository layer:
class ReactiveReservationRepository(private val databaseClient: TransactionalDatabaseClient,
private val customerRepository: CustomerRepository) : ReservationRepository {
override fun findOne(reservationId: String): Publisher<Reservation> =
databaseClient.inTransaction {
customerRepository.find(reservationId).toMono()
.flatMap { customer ->
it.execute().sql("SELECT * FROM reservation WHERE reservation_id=$1")
.bind("$1", reservationId)
.exchange()
.flatMap { sqlRowMap ->
sqlRowMap.extract { t, u ->
Reservation(t.get("reservation_id", String::class.java)!!,
t.get("restaurant_name", String::class.java)!!,
customer, t.get("date", LocalDateTime::class.java)!!)
}.one()
}
}
}
override fun save(reservation: Reservation): Publisher<Reservation> =
databaseClient.inTransaction {
customerRepository.save(reservation.reservationId, reservation.customer).toMono()
.then(it.execute().sql("INSERT INTO reservation (reservation_id, restaurant_name, date) VALUES ($1, $2, $3)")
.bind("$1", reservation.reservationId)
.bind("$2", reservation.restaurantName)
.bind("$3", reservation.date)
.fetch().rowsUpdated())
}.then(Mono.just(reservation))
override fun delete(reservationId: String): Publisher<Void> =
databaseClient.inTransaction {
customerRepository.delete(reservationId).toMono()
.then(it.execute().sql("DELETE FROM reservation WHERE reservation_id = $1")
.bind("$1", reservationId)
.fetch().rowsUpdated())
}.then(Mono.empty())
}
I hope that can help you