So let's say I have a bunch of couroutines running that interact with some webservice and since I don't want to spam it I wanna limit the requests to 1 request every x seconds max. For that I could use some code like this:
fun CoroutineScope.rateLimiter(tokens: SendChannel<Unit>, rate: Int) = launch {
var lastToken = System.currentTimeMillis()
while (isActive) {
val currentTime = System.currentTimeMillis()
if (currentTime - lastToken < rate) {
delay(currentTime - lastToken)
}
tokens.send(Unit)
}
}
fun CoroutineScope.request(tokens: ReceiveChannel<Unit>) = launch {
for (token in tokens) {
//Do Web request
}
}
1.) Is this way to do that efficient?
2.) This isn't expandable to say limit something to x bytes/second or something where I would need to request x tokens out of a Token Bucket, what would be the best way to implement something like that with coroutines?
If you wanna skip having dependency on jobs and channels that might create more permits than are being consumed, and then having a stampeeding herd once some process starts taking permits, maybe this is the solution for you.
(Some jvm-style in here, but replaceable for Multi-platform)
import kotlin.math.max
import kotlinx.coroutines.delay
import kotlinx.coroutines.sync.Mutex
import kotlinx.coroutines.sync.withLock
class SimpleRateLimiter(eventsPerSecond: Double) {
private val mutex = Mutex()
#Volatile
private var next: Long = Long.MIN_VALUE
private val delayNanos: Long = (1_000_000_000L / eventsPerSecond).toLong()
/**
* Suspend the current coroutine until it's calculated time of exit
* from the rate limiter
*/
suspend fun acquire() {
val now: Long = System.nanoTime()
val until = mutex.withLock {
max(next, now).also {
next = it + delayNanos
}
}
if (until != now) {
delay((until - now) / 1_000_000)
}
}
}
It ofc comes with other tradeoffs.
Behavior when nanoTime is nearing Long.MAX_VALUE is most definitely corrupted.
No impl for maxDelay/timeout
No way of grabbing multiple termits
No tryAquire implementation
If you want an IntervalLimiter that allows X requests every Y seconds, and then throws exceptions, there is the RateLimiter in Resilience4J
Or if you want something that is a lot more fully featured, I'm working on a PR to create both a RateLimiter and an IntervalLimiter in coroutines core project.
Related
There's applications I'd like to write where I'd like some things to occur on a schedule.
Polling a URL for updates every few minutes seems to be a fairly common use case. In this particular case, though, I'm just trying to implement a clock.
This works:
#Composable
fun App() {
var ticks by remember { mutableStateOf(0) }
// Not 100% happy about this unused variable either
val timer = remember {
Timer().apply {
val task = object : TimerTask() {
override fun run() {
ticks++
}
}
scheduleAtFixedRate(task, 1000L, 1000L)
}
}
MaterialTheme {
Text(
// A real application would format this number properly,
// but that's a different question
text = "$ticks"
)
}
}
But I had to import java.util.Timer, so it's not going to be portable.
Jetpack Compose can do animation, so it surely has its own timer somewhere, implying there should be some portable way to do this as well, but I can't seem to figure it out.
Is there a cross-platform way to get a timer for this purpose?
In Compose you can use LaunchedEffect - it's a side effect which is run on a coroutine scope, so you can use delay inside, like this:
var ticks by remember { mutableStateOf(0) }
LaunchedEffect(Unit) {
while(true) {
delay(1.seconds)
ticks++
}
}
I just wanted to share an alternative I have experimented with in case someone else thinks of it and encounters the same issues I have. Here is the naive implementation:
#Composable
fun Countdown(targetTime: Long, content: #Composable (remainingTime: Long) -> Unit) {
var remainingTime by remember(targetTime) {
mutableStateOf(targetTime - System.currentTimeMillis())
}
content.invoke(remainingTime)
LaunchedEffect(remainingTime) {
delay(1_000L)
remainingTime = targetTime - System.currentTimeMillis()
}
}
Assuming you want a precission of up to a second, this snippet will cause the LaunchedEffect to update a second after remainingTime is updated, updating remainingTime in relation with the current time in milliseconds. This basically creates a loop. Wrapping this logic in a #Composable is good as it prevents the excessive re-composition this would cause if you embedded your LaunchedEffect in a large component tree.
This works, but there is a catch: you will eventually notice your timer is skipping seconds. This happens because there will be some extra delay between the assignment of a new value to the remainingTime variable and the re-execution of LaunchedEffect which will essentially mean there is more than a second between updates.
Here is an improved implementation of the above:
#Composable
fun Countdown(targetTime: Long, content: #Composable (remainingTime: Long) -> Unit) {
var remainingTime by remember(targetTime) {
mutableStateOf(targetTime - System.currentTimeMillis())
}
content.invoke(remainingTime)
LaunchedEffect(remainingTime) {
val diff = remainingTime - (targetTime - System.currentTimeMillis())
delay(1_000L - diff)
remainingTime = targetTime - System.currentTimeMillis()
}
}
We simply subtract the time it took for LaunchedEffect to re-execute from the intended delay time. This will avoid your timer skipping seconds.
The extra delay should not be an issue for the implementation in the accepted answer. The only advantage of this approach I noticed is that the loop will stop running once we navigate away from the screen. In my tests, the while loop with a true condition kept running when navigating to another Activity.
#Composable
fun TimerTicks(
initTick: Long = 1_000L,
interval: Long = 1_000L,
content: #Composable (tickTime: Long) -> Unit
) {
var ticks by remember(initTick) {
mutableStateOf(initTick)
}
content.invoke(ticks)
LaunchedEffect(ticks) {
val diff = ticks + interval
delay(interval)
ticks = diff
}
}
I feel like I'm trying to reinvent the wheel, but I can't find what I should be using instead. Also, my wheel looks square-ish.
I want to get the result of some long-running operation (download/async API/compute/etc) in a thread-safe, cached way, that is:
if the long-running op has already completed, just return the cached value
if the long-running op is running, just wait for it to complete
otherwise start the long-running op and wait
I'd like all of the above waiting to be done in non-blocking suspend funcs. Am I missing some obvious doodad in the standard toolset?
I'd also like the cached value to be resettable (invalidate the result, so that the next get() starts over).
Just launching a coroutine will create a bunch of parallel long-running ops if called multiple times before it's done (fails case #2).
Both Flow and Deferred can be used to do that, I think, but both need a bunch of logic around them (like deciding when to start the op vs when to just wait).
So far I found this relatively simple way:
class CachedComputation<T>(private val compute: suspend () -> T) {
private var cache = GlobalScope.async(Dispatchers.Unconfined, start = CoroutineStart.LAZY) { compute() }
suspend fun get(): T {
return cache.await()
}
fun reset() {
cache = GlobalScope.async(Dispatchers.Unconfined, start = CoroutineStart.LAZY) { compute() }
}
}
It might need some synchronisation between get() and reset(). But the main problem with that is the usage of GlobalScope. Passing a Scope to get() is ugly and still starts a new coroutine, even though it's already a suspend func. I'd rather have it confined to the context of the caller.
I can solve it with a more complicated class:
class CachedComputation<T>(private val compute: suspend () -> T) {
private var future: CompletableDeferred<T>? = null
/**
* Return the result of computation once it's available.
* If the result is ready, return it immediately.
* If the result is being computed, wait for it to finish and return.
* If the computation is not running, start it and wait for result.
*/
suspend fun get(): T {
val (needToCompute, f) = newOrCurrentFuture()
return if (needToCompute) {
compute().also { f.complete(it) }
} else {
f.await()
}
}
/**
* If the computation hasn't started yet, create, save and return a new deferred result
* and indicate the need to start the computation.
* Otherwise return the current deferred result that can be awaited on, and indicate
* there's no need to start a new computation.
*/
#Synchronized
private fun newOrCurrentFuture(): Pair<Boolean, CompletableDeferred<T>> {
val currentFuture = future
return if (currentFuture != null)
Pair(false, currentFuture)
else
Pair(true, newFuture())
}
/**
* Create a new deferred result and save it in the class field
*/
private fun newFuture(): CompletableDeferred<T> {
return CompletableDeferred<T>()
.also { future = it }
}
#Synchronized
fun reset() {
future = null
}
}
But it seems unnecessarily complicated.
I have a batch processor that I want to refactor to be expressed a 1-to-1 fashion based on input to increase readability, and for further optimization later on. The issue is that there is a service that should be called in batches to reduce HTTP overhead, so mixing the 1-to-1 code with the batch code is a bit tricky, and we may not want to call the service with every input. Results can be sent out eagerly one-by-one, but order must be maintained, so something like a flow doesn't seem to work.
So, ideally the batch processor would look something like this:
class Processor<A, B> {
val service: Service<A, B>
val scope: CoroutineScope
fun processBatch(input: List<A>) {
input.map {
Pair(it, scope.async { service.call(it) })
}.map {
(a, b) ->
runBlocking { b.await().let { /** handle result, do something with a if result is null, etc **/ } }
}
}
}
The desire is to perform all of the service logic in such a way that it is executing in the background, automatically splitting the inputs for the service into batches, executing them asynchronously, and somehow mapping the result of the batch call into the suspended call.
Here is a hacky implementation:
class Service<A, B> {
val inputContainer: MutableList<A>
val outputs: MutableList<B>
val runCalled = AtomicBoolean(false)
val batchSize: Int
suspended fun call(input: A): B? {
// some prefiltering logic that returns a null early
val index = inputContainer.size
inputContainer.add(a) // add to overall list for later batching
return suspend {
run()
outputs[index]
}
}
fun run() {
val batchOutputs = mutableListOf<Deferred<List<B?>>>()
if (!runCalled.getAndSet(true)) {
inputs.chunked(batchSize).forEach {
batchOutputs.add(scope.async { batchCall(it) })
}
runBlocking {
batchOutputs.map {
val res = result.await()
outputs.addAll(res)
}
}
}
}
suspended fun batchCall(input: List<A>): List<B?> {
// batch API call, etc
}
}
Something like this could work but there are several concerns:
All API calls go out at once. Ideally this would be batching and executing in the background while other inputs are being scheduled, but this is not .
Processing of the service result for the first input cannot resume until all results have been returned. Ideally we could process the result if the service call has returned, while other results continue to be performed in the background.
Containers of intermediate results seem hacky and prone to bugs. Cleanup logic is also needed, which introduces more hacky bits into the rest of the code
I can think of several optimizations to the address 1 and 2, but I imagine concerns related to 3 would be worse. This seems like a fairly common call pattern and I would expect there to be a library or much simpler design pattern to accomplish this, but I haven't been able to find anything. Any guidance is appreciated.
You're on the right track by using Deferred. The solution I would use is:
When the caller makes a request, create a CompletableDeferred
Using a channel, pass this CompletableDeferred to the service for later completion
Have the caller suspend until the service completes the CompletableDeferred
It might look something like this:
val requestChannel = Channel<Pair<Request, CompletableDeferred<Result>>()
suspend fun doRequest(request: Request): Result {
val result = CompletableDeferred<Result>()
requestChannel.send(Pair(request, result))
return result.await()
}
fun run() = scope.launch {
while(isActive) {
val (requests, deferreds) = getBatch(batchSize).unzip()
val results = batchCall(requests)
(results zip deferreds).forEach { (result, deferred) ->
deferred.complete(result)
}
}
}
suspend fun getBatch(batchSize: Int) = buildList {
repeat(batchSize) {
add(requestChannel.receive())
}
}
I have a function generating a very long sequence of work items. Generating these items is fast, but there are too many in total to store a list of them in memory. Processing the items produces no results, just side effects.
I would like to process these items across multiple threads. One solution is to have a thread read from the generator and write to a concurrent bounded queue, and a number of executor threads polling for work from the bounded queue, but this is a lot of things to set up.
Is there anything in the standard library that would help me do that?
I had initially tried
items.map { async(executor) process(it) }.forEach { it.await() }
But, as pointed out in how to implement parallel mapping for sequences in kotlin, this doesn't work for reasons that are obvious in retrospect.
Is there a quick way to do this (possibly with an external library), or is manually setting up a bounded queue in the middle my best option?
You can look at coroutines combined with channels.
If all work items can be emmited on demand with producer channel. Then it's possible to await for each items and process it with a pool of threads.
An example :
sealed class Stream {
object End: Stream()
class Item(val data: Long): Stream()
}
val produceCtx = newSingleThreadContext("producer")
// A dummy producer that send one million Longs on its own thread
val producer = CoroutineScope(produceCtx).produce {
for (i in (0 until 1000000L)) send(Stream.Item(i))
send(Stream.End)
}
val workCtx = newFixedThreadPoolContext(4, "work")
val workers = Channel<Unit>(4)
repeat(4) { workers.offer(Unit) }
for(_nothing in workers) { // launch 4 times then wait for a task to finish
launch(workCtx) {
when (val item = producer.receive()) {
Stream.End -> workers.close()
is Stream.Item -> {
workFunction(item.data) // Actual work here
workers.offer(Unit) // Notify to launch a new task
}
}
}
}
Your magic word would be .asSequence():
items
.asSequence() // Creates lazy executable sequence
.forEach { launch { executor.process(it) } } // If you don't need the value aftrwards, use 'launch', a.k.a. "fire and forget"
but there are too many in total to store a list of them in memory
Then don't map to list and don't collect the values, no matter if you work with Kotlin or Java.
As long as you are on the JVM, you can write yourself an extension function, that works the sequence in chunks and spawns futures for all entries in a chunk. Something like this:
#Suppress("UNCHECKED_CAST")
fun <T, R> Sequence<T>.mapParallel(action: (value: T) -> R?): Sequence<R?> {
val numThreads = Runtime.getRuntime().availableProcessors() - 1
return this
.chunked(numThreads)
.map { chunk ->
val threadPool = Executors.newFixedThreadPool(numThreads)
try {
return#map chunk
.map {
// CAUTION -> needs to be written like this
// otherwise the submit(Runnable) overload is called
// which always returns an empty Future!!!
val callable: () -> R? = { action(it) }
threadPool.submit(callable)
}
} finally {
threadPool.shutdown()
}
}
.flatten()
.map { future -> future.get() }
}
You can then just use it like:
items
.mapParallel { /* process an item */ }
.forEach { /* handle the result */ }
As long as workload per item is similar, this gives a good parallel processing.
I'm making thousands of HTTP requests using async/await and would like to have a progress indicator. I've added one in a naive way, but noticed that the counter value never reaches the total when all requests are done. So I've created a simple test and, sure enough, it doesn't work as expected:
fun main(args: Array<String>) {
var i = 0
val range = (1..100000)
range.map {
launch {
++i
}
}
println("$i ${range.count()}")
}
The output is something like this, where the first number always changes:
98800 100000
I'm probably missing some important detail about concurrency/synchronization in JVM/Kotlin, but don't know where to start. Any tips?
UPDATE: I ended up using channels as Marko suggested:
/**
* Asynchronously fetches stats for all symbols and sends a total number of requests
* to the `counter` channel each time a request completes. For example:
*
* val counterActor = actor<Int>(UI) {
* var counter = 0
* for (total in channel) {
* progressLabel.text = "${++counter} / $total"
* }
* }
*/
suspend fun getAssetStatsWithProgress(counter: SendChannel<Int>): Map<String, AssetStats> {
val symbolMap = getSymbols()?.let { it.map { it.symbol to it }.toMap() } ?: emptyMap()
val total = symbolMap.size
return symbolMap.map { async { getAssetStats(it.key) } }
.mapNotNull { it.await().also { counter.send(total) } }
.map { it.symbol to it }
.toMap()
}
The explanation what exactly makes your wrong approach fail is secondary: the primary thing is fixing the approach.
Instead of async-await or launch, for this communication pattern you should instead have an actor to which all the HTTP jobs send their status. This will automatically handle all your concurrency issues.
Here's some sample code, taken from the link you provided in the comment and adapted to your use case. Instead of some third party asking it for the counter value and updating the GUI with it, the actor runs in the UI context and updates the GUI itself:
import kotlinx.coroutines.experimental.*
import kotlinx.coroutines.experimental.channels.*
import kotlin.system.*
import kotlin.coroutines.experimental.*
object IncCounter
fun counterActor() = actor<IncCounter>(UI) {
var counter = 0
for (msg in channel) {
updateView(++counter)
}
}
fun main(args: Array<String>) = runBlocking {
val counter = counterActor()
massiveRun(CommonPool) {
counter.send(IncCounter)
}
counter.close()
println("View state: $viewState")
}
// Everything below is mock code that supports the example
// code above:
val UI = newSingleThreadContext("UI")
fun updateView(newVal: Int) {
viewState = newVal
}
var viewState = 0
suspend fun massiveRun(context: CoroutineContext, action: suspend () -> Unit) {
val numCoroutines = 1000
val repeatActionCount = 1000
val time = measureTimeMillis {
val jobs = List(numCoroutines) {
launch(context) {
repeat(repeatActionCount) { action() }
}
}
jobs.forEach { it.join() }
}
println("Completed ${numCoroutines * repeatActionCount} actions in $time ms")
}
Running it prints
Completed 1000000 actions in 2189 ms
View state: 1000000
You're losing writes because i++ is not an atomic operation - the value has to be read, incremented, and then written back - and you have multiple threads reading and writing i at the same time. (If you don't provide launch with a context, it uses a threadpool by default.)
You're losing 1 from your count every time two threads read the same value as they will then both write that value plus one.
Synchronizing in some way, for example by using an AtomicInteger solves this:
fun main(args: Array<String>) {
val i = AtomicInteger(0)
val range = (1..100000)
range.map {
launch {
i.incrementAndGet()
}
}
println("$i ${range.count()}") // 100000 100000
}
There's also no guarantee that these background threads will be done with their work by the time you print the result and your program ends - you can test it easily by adding just a very small delay inside launch, a couple milliseconds. With that, it's a good idea to wrap this all in a runBlocking call which will keep the main thread alive and then wait for the coroutines to all finish:
fun main(args: Array<String>) = runBlocking {
val i = AtomicInteger(0)
val range = (1..100000)
val jobs: List<Job> = range.map {
launch {
i.incrementAndGet()
}
}
jobs.forEach { it.join() }
println("$i ${range.count()}") // 100000 100000
}
Have you read Coroutines basics? There's exact same problem as yours:
val c = AtomicInteger()
for (i in 1..1_000_000)
launch {
c.addAndGet(i)
}
println(c.get())
This example completes in less than a second for me, but it prints some arbitrary number, because some coroutines don't finish before main() prints the result.
Because launch is not blocking, there's no guarantee all of coroutines will finish before println. You need to use async, store the Deferred objects and await for them to finish.