Run a long running task in Kotlin (Ktor backend) - kotlin

In my Ktor backend, I have a simple get API request. It will make some database update and finally return a message. But, before returning the response, the API will initiate a long-running task (some database operation and file writing tasks) which might take some time to finish. The long-running task should continue its job even if the API returns its response.
This is my get API request
routing {
get("/testAPI") {
val x= queryResolver()
println(" return from API")
call.respondText("Hello ..")
}
}
Below are my handling functions. queryResolver() is called from the API and then it will call myService() which will call db_update1(), db_update2() and finally starts the long_running_task() .
suspend fun queryResolver(): String{
println(" queryResolver- start -${Thread.currentThread().name}")
delay(2000)
val x = myService()
println(" queryResolver - end")
return "..return from queryResolver "
}
suspend fun myService(): String{
println(" service - start -${Thread.currentThread().name}")
delay(2000)
val x = db_update1()
val y = db_update2()
GlobalScope.launch() {
println(".. launch start -${Thread.currentThread().name} ")
long_running_task()
println(".. launch - end ")
}
println(" service - end")
return "..return from service -> "
}
suspend fun db_update1(): String{
println(" db_update1 - start -${Thread.currentThread().name}")
delay(2000)
println(" db_update1 - end")
return "..return from db_update1 -> "
}
suspend fun db_update2(): String{
println(" db_update2 - start -${Thread.currentThread().name}")
delay(3000)
println(" db_update2 - end")
return "..return from db_update2 -> "
}
suspend fun long_running_task(): String{
println("### long_running_task - start -${Thread.currentThread().name}")
delay(10000)
println("### long_running_task - end")
return "..return from long_running_task -> "
}
This code gives the below output which is desired output.
Start API
queryResolver- start -eventLoopGroupProxy-4-2
service - start -eventLoopGroupProxy-4-2
db_update1 - start -eventLoopGroupProxy-4-2
db_update1 - end
db_update2 - start -eventLoopGroupProxy-4-2
db_update2 - end
service - end
queryResolver - end
return from API
.. launch start -DefaultDispatcher-worker-1
### long_running_task - start -DefaultDispatcher-worker-1
### long_running_task - end
.. launch - end
But, using the global scope is not a good idea in this case.
Now, what is the best way to keep alive(to allow it to finish its task) the long_running_task() even if the API response returns?
If I want to reuse the scope and the context of the existing scope and context how can I do that?

Related

How to cancel a job in Kotlin using job id passed to an API?

I want to build an API in Kotlin that accepts a job id and cancels the job if it is running. I know simple cancellation in Kotlin works like this:
val job = launch(Dispatchers.Default) {
for (i in 0..1000) {
delay(50)
println("$i..")
}
println("Job is completed")
}
delay(500)
println("Cancelling")
job.cancel()
job.join()
println("Cancelled and done")
But what I want to do is
Have a StartJob API which starts a job and returns back the job id associated with this job. This id can be numeric or string.
Build a JobCancellation API which takes a job id as input and cancels that job if it is running. There can be multiple jobs with unique job ids running at the same time.
How do I implement this?
Barebones ID generator:
internal class IdProvider {
private var previous = AtomicLong(-1L)
fun next() = previous.addAndGet(1)
}
Then you can create a map of Job IDs to Jobs in your API:
class JobRegistry {
private val idProvider = IdProvider()
private val map = ConcurrentHashMap<Long, Job>()
fun launch(
scope: CoroutineScope,
context: CoroutineContext = EmptyCoroutineContext,
start: CoroutineStart = CoroutineStart.DEFAULT,
block: suspend CoroutineScope.() -> Unit
): Long {
val id = idProvider.next()
map[id] = scope.launch(context, start, block)
.also { it.invokeOnCompletion { map.remove(id) } }
return id
}
fun cancel(jobId: Long) {
map[jobId]?.cancel()
}
}

How to gracefully access the results of asynchronous functions in Kotlin Coroutine with Mutex synchronously?

I want to process some data in an IO thread and need to access the final data after the processing is complete. This is my Process sample code:
private class Process {
private val scope = CoroutineScope(Dispatchers.IO)
private val value: AtomicInteger = AtomicInteger(0)
private val mutex = Mutex()
/**
* Do background task to change value.
*/
fun doIt() {
scope.launch {
println("SCOPE - start at ${System.currentTimeMillis()}")
mutex.withLock {
println("Process - doIt start delay ${System.currentTimeMillis()}")
delay(10)
value.incrementAndGet()
println("Process - value increase to $value")
}
println("SCOPE - end at ${System.currentTimeMillis()}")
}
}
/**
* Get value sync.
*/
fun getValue(): Int {
return runBlocking(scope.coroutineContext) {
mutex.withLock {
println("Process - getValue ${System.currentTimeMillis()}")
value.get()
}
}
}
/**
* Reset value.
*/
fun reset() {
runBlocking(scope.coroutineContext) {
mutex.withLock {
value.set(0)
println("Process - value reset to ${value.get()}")
}
}
}
}
And the Process usage:
fun main() {
val process = Process()
repeat(100) {
println("-----------------------------")
println("Start - ${System.currentTimeMillis()}")
process.reset()
process.doIt()
val result = run {
val count = process.getValue()
if (count == 1) {
println("Count - $count ${System.currentTimeMillis()}")
true
} else {
println("Mutex failed in $it with $count")
false
}
}
if (!result) return#main
}
}
My expected result log is 100 snippets like this:
-----------------------------
Start - 1645717971970
Process - value reset to 0
SCOPE - start at 1645717972011
Process - doIt start delay 1645717972011
Process - value increase to 1
SCOPE - end at 1645717972034
Process - getValue 1645717972034
Count - 1 1645717972035
But the test is always interrupted when it reaches the third or fourth loop:
-----------------------------
Start - 1645717971970
Process - value reset to 0
SCOPE - start at 1645717972011
Process - doIt start delay 1645717972011
Process - value increase to 1
SCOPE - end at 1645717972034
Process - getValue 1645717972034
Count - 1 1645717972035
-----------------------------
Start - 1645717972035
Process - value reset to 0
SCOPE - start at 1645717972036
Process - doIt start delay 1645717972036
Process - value increase to 1
SCOPE - end at 1645717972049
Process - getValue 1645717972049
Count - 1 1645717972050
-----------------------------
Start - 1645717972050
Process - value reset to 0
SCOPE - start at 1645717972050
Process - getValue 1645717972051
Process - doIt start delay 1645717972051
Mutex failed in 2 with 0
Can anyone tell me how I should synchronize to get the results of asynchronous code that has already started executing under this case?
Have you considered using a Deferred result in place of your AtomicInteger?
var result: Deferred<Int> = CompletableDeferred(0)
fun doIt() {
result = async { /* get new value */ }
}
fun getValue() = runBlocking { result.await() }
fun reset() {
result = CompletableDeferred(0)
}
You might need to introduce some more complexity if you need to handle cancellation of the previous job each time doIt is called.

Will the collect of the Flow block to execute?

I run the Code A and get the Result A. In my mind, it should be the Result B.
It seems that flow.collect { value -> println(value) } block to execute.
Will the collect of the Flow block to execute ?
Code A
fun simple(): Flow<Int> = flow {
println("Flow started")
for (i in 1..3) {
delay(300)
emit(i)
}
}
fun main() = runBlocking<Unit> {
println("Calling simple function...")
val flow = simple()
println("Calling collect...")
flow.collect { value -> println(value) } //Block?
println("Calling collect again...")
}
Result A
Calling simple function...
Calling collect...
Flow started
1
2
3
Calling collect again...
Result B
Calling simple function...
Calling collect...
Flow started
Calling collect again...
1
2
3
BTW, I run Code 1 and get the result 1 as I expected.
Code 1
fun simple(): Flow<Int> = flow {
for (i in 1..3) {
delay(100)
emit(i)
}
}
fun main() = runBlocking<Unit> {
launch {
for (k in 1..3) {
println("I'm not blocked $k")
delay(100)
}
}
simple().collect { value -> println(value) }
}
Result 1
I'm not blocked 1
1
I'm not blocked 2
2
I'm not blocked 3
3
Suspend functions do not block, but they are synchronous, meaning the execution of the code in the coroutine waits for the suspend function to return before continuing. The difference between suspend function calls and blocking function calls is that the thread is released to be used for other tasks while the coroutine is waiting for the suspend function to return.
collect is a suspend function that internally calls its lambda repeatedly and synchronously (suspending, not blocking) and doesn't return until the Flow is completed.
launch is an asynchronous function that starts a coroutine. It returns immediately without waiting for its coroutine to complete, which is why Code 1 behaved as you expected.

Tell Flux to emit next item after async processing

I'm using Project Reactor with Webflux to try to read data from a message queue, then process it in chunks (eg, five at a time) and make a request to an API with each chunk. The API does not work well with high throughput, so I need to have control over how many requests are sent concurrently.
Basically, I'd like to have a WebClient call finish, then be able to tell the Flux that we're ready to process more.
I was using this code to try to emulate the desired functionality, and I'm getting results that I don't understand:
fun main() {
val subscriber = CustomSubscriber()
Flux.create<Int> { sink ->
sink.onRequest {
sink.next(1)
}
}
.doOnNext {
println("hit first next with $it")
}
.delayElements(Duration.ofSeconds(1)) // Mock WebClient call
.doOnNext {
println("before request")
subscriber.request(1)
println("after request")
}
.subscribeWith(subscriber)
Thread.sleep(10000)
}
class CustomSubscriber : BaseSubscriber<Int>() {
override fun hookOnSubscribe(subscription: Subscription) {
subscription.request(1)
}
}
The output of this code is
hit first next with 1
before request
after request
What I was hoping for is this:
hit first next with 1 // one second passes
before request
after request
hit first next with 1 // one second passes
before request
after request
hit first next with 1 // one second passes
before request
after request
hit first next with 1 // one second passes
before request
after request
(Infinite loop)
So the request method is called, but the number is never emitted.
Oddly, when I call request in a separate Flux, I'm getting the desired behavior:
fun main() {
val subscriber = CustomSubscriber()
Flux.create<Int> { sink ->
sink.onRequest {
sink.next(1)
}
}
.doOnNext {
println("hit first next with $it")
}
.subscribeWith(subscriber)
Flux.range(0, 5)
.delayElements(Duration.ofSeconds(3))
.doOnNext { subscriber.request(1) }
.subscribe()
Thread.sleep(10000)
}
class CustomSubscriber : BaseSubscriber<Int>() {
override fun hookOnSubscribe(subscription: Subscription) {
subscription.request(1)
}
}
So it seems like there is an issue with calling the request method in the doOnNext method of the original Flux?
I'm not married to the idea of using a FluxSink, that just seemed like a way to have more explicit control of the data emission.
I think what you are looking for is custom subscriber, which consumes data at its own pace based on some logic. Something like this.
Flux.range(0, 14)
.subscribeWith(object : Subscriber<Int> {
private var count = 0
lateinit var subscription: Subscription
override fun onSubscribe(s: Subscription) {
subscription = s
s.request(2)
}
override fun onNext(parameter: Int) {
println("Before request")
// ----- some processing
println("After request")
count++
if (count >= 2) {
println("Requesting more......")
count = 0
subscription.request(2)
}
}
override fun onError(t: Throwable) {}
override fun onComplete() {
println("Done")
}
})

Coroutines how to wait for the data and then continue process

I'm learning coroutines with kotlin, and I have problem how process can wait until process 1 finished then it continue to process 2, from my sample below I have object Network which access API server using getNews(it's running well and get the data)
I called this getNews from refreshNews using asynch - await, with the purpose it wait for the data then it continue running, but "The program Not Wait", it just running process 2 then process 1 finish, so I cannot capture data from API in refresh news
// process 1 - calling api this running well can get the data see process 2
object Network {
var status : NewsApiStatus = NewsApiStatus.LOADING
private var viewModelJob = Job()
private val coroutineScope = CoroutineScope(viewModelJob + Dispatchers.Main)
fun getNews(filter: String, page: Int =1) : newsData? {
var allNews : newsData? = null
coroutineScope.launch {
RetrofitClient.instance.getAllNews(filter, page).enqueue(object: Callback<newsData>{
override fun onFailure(call: Call<newsData>, t: Throwable) {
status = NewsApiStatus.ERROR
}
override fun onResponse(
call: Call<newsData>,
response: Response<newsData>
) {
status = NewsApiStatus.DONE
var listResult = response.body()
if (listResult != null) {
if (listResult.data.isNotEmpty()) {
allNews = listResult
Timber.tag(TAG).i( "process 1 total allNews = ${allNews!!.data.size}")
}
}
}
})
}
return(allNews)
}
}
// process 2 - calling process 1 with runBlocking
fun refreshNews() = runBlocking{
val newsData = async {
Network.getNews("")
}
Timber.tag(TAG).i("proses 2 ${newsData.await()?.data?.size}")
// here I want newsData to wait until it has data
}
// this main program that call process 2
class NewsListViewModel(application: Application) : AndroidViewModel(application) {
init {
refreshNews()
}
}
launch returns a reference to the started job. You can use it to wait for the job to finish by calling join():
val job = GlobalScope.launch { // launch a new coroutine and keep a reference to its Job
// ...
}
runBlocking {
job.join() // wait until child coroutine completes
}
Currently, your getNews() launches a coroutine and immediately returns. allNews isn't initialised at that point yet.
You need to either call job.join() inside getNews() (would make it blocking), or use async inside getNews() and return its result if you want to keep it asynchronous (you'd need to take the result differently from your http client as you won't be able to initialise the variable declared outside).
It's worth to go through the official coroutine docs:
https://kotlinlang.org/docs/reference/coroutines/basics.html
https://kotlinlang.org/docs/tutorials/coroutines/coroutines-basic-jvm.html