I want to use Kotlin coroutines in my reactive sql client transactions.
For simplicity, I was going to use the provided helper function io.vertx.mutiny.sqlclient.Pool#withTransaction mentioned in the docs here. Since the passed function is not a coroutine suspend function, I'm getting an error like Suspension functions can be called only within coroutine body when Im trying to compile a code like the following
val client : PgPool
...
suspend fun someServiceFunction () {
client.withTransaction { connection ->
repository.save(connection, entity).awaitSuspending() //This is not working
...
}
}
The function header for the withTransaction looks like this
#CheckReturnValue
public <T> Uni<T> withTransaction(Function<SqlConnection, Uni<T>> function)
I'm asking myself if there is still a way to use this with kotlin coroutines, since I'm pretty new to them.
Thank you for any help !
I'm not familiar with Mutiny nor Quarkus, but it seems there is a way to convert from Deferred<T> to Uni<T> in mutiny-kotlin, which you seem to be using.
You could therefore create your own suspending version of withTransaction like this:
import io.vertx.mutiny.sqlclient.SqlConnection
import io.vertx.mutiny.sqlclient.Pool
#OptIn(ExperimentalCoroutinesApi::class)
suspend fun <T> Pool.withTransaction(block: suspend (SqlConnection) -> T): T = coroutineScope {
client.withTransaction { connection ->
async { block(connection) }.asUni()
}.awaitSuspending()
}
And then use it:
suspend fun someServiceFunction() {
client.withTransaction { connection ->
repository.save(connection, entity).awaitSuspending()
// potentially other suspending stuff here, without the need to combine()
}
}
But that begs the question, why use the Mutiny variant of Vertx things, if in the end you want to use Kotlin coroutines? I think by default Vertx works with Java futures which are also integrated with coroutines.
Related
Sometimes we want to have a suspend function that calls other suspend functions inside of a transaction:
suspend fun mySuspendFunction(jooqContext: DSLContext) {
jooqContext.transaction { config ->
val transactionContext: DSLContext = config.dsl()
// ... some code that uses transactionContext ...
anotherSuspendFunction(transactionContext)
// ... more code that uses transactionContext ...
}
}
However, this won't compile, because jOOQ's DSLContext.transaction takes a non-suspending function as its parameter, and so the body of the transaction cannot call a suspend function directly. The above will fail to compile with an error like:
Suspension functions can be called only within coroutine body
One workaround is to use runBlocking around the suspend calls:
suspend fun mySuspendFunction(jooqContext: DSLContext) {
jooqContext.transaction { config ->
val transactionContext: DSLContext = config.dsl()
// ... some code that uses transactionContext ...
runBlocking {
anotherSuspendFunction(transactionContext)
}
// ... more code that uses transactionContext ...
}
}
However the documentation for runBlocking advises against using it in this way ("designed to bridge regular blocking code to libraries that are written in suspending style, to be used in main functions and in tests").
That's just a hint of the real problem, which is that because DSLContext.transaction is calling a non-suspend function for the transaction body, the coroutine's thread is blocking (ie: cannot suspend) for the duration of the transaction.
I thought about making a suspend-friendly transaction wrapper for Kotlin,
but jOOQ does not yet have a procedural transaction API. Also, DSLContext.connection blocks on the supplied function so I don't see a way to get a hold of the single Connection for the duration of the transaction without blocking.
Is there a way to use a jOOQ transaction in a Kotlin coroutine that does not block for the duration of the transaction, and which allows suspend functions to be called from within the transaction body?
jOOQ 3.16
jOOQ 3.16 does not yet have a reactive transaction API. This will be addressed with jOOQ 3.17, soon: https://github.com/jOOQ/jOOQ/issues/13502
jOOQ 3.17 bridging between reactive and coroutines manually
With jOOQ 3.17, you can write:
suspend fun mySuspendFunction(jooqContext: DSLContext): Any {
return jooqContext.transactionPublisher { config ->
// Turn the suspension result into a Mono, which implements the reactive
// streams Publisher<T> SPI, which jOOQ expects as a result from a
// TransactionalPublishable
mono {
anotherSuspendFunction(config)
}
}
// Turn the Publisher<T> that is returned from transactionPublisher() back
// into a suspension result
.awaitFirst()
}
This is assuming, you're using the recommended reactive streams / coroutines bridges:
<dependency>
<groupId>org.jetbrains.kotlinx</groupId>
<artifactId>kotlinx-coroutines-core</artifactId>
<version>${kotlinx.coroutines.version}</version>
</dependency>
<dependency>
<groupId>org.jetbrains.kotlinx</groupId>
<artifactId>kotlinx-coroutines-reactor</artifactId>
<version>${kotlinx.coroutines.version}</version>
</dependency>
jOOQ 3.17, alternatively, using jooq-kotlin-coroutines
With https://github.com/jOOQ/jOOQ/issues/9335, there will be a new module jooq-kotlin-coroutines, which you can pull in to bridge between the worlds for you, e.g.
suspend fun mySuspendFunction(jooqContext: DSLContext) {
jooqContext.transactionCoroutine { config ->
anotherSuspendFunction(config)
}
}
This just removes the glue code for you
jOOQ transaction APIs are just convenience. You don't need them
Notice that you don't need jOOQ's transaction API, nor do you need it to be "procedural." All of this API is just convenience on top of JDBC or R2DBC. You can always access your R2DBC Connection directly, and invoke its transaction API directly to achieve non-blocking transactional behaviour. It's just a bit more glue code.
I will write about the way I know how to do it without using the API provided by jOOQ.
Here is how to make the Transactional annotation work with an R2DBC connection.
https://github.com/jOOQ/jOOQ/issues/12218#issuecomment-1213929029
You can also use TransactionalOperator to perform transactions manually if you use jOOQ with a connection that Spring manages transactions in the way described above.
import kotlinx.coroutines.CoroutineScope
import kotlinx.coroutines.reactor.awaitSingle
import kotlinx.coroutines.reactor.mono
import org.springframework.stereotype.Component
import org.springframework.transaction.reactive.TransactionalOperator
#Component
class TransactionOperation(private val transactionalOperator: TransactionalOperator) {
suspend operator fun <T> invoke(block: suspend CoroutineScope.() -> T): T = transactionalOperator
.transactional(mono(block = block))
.awaitSingle()
}
#Component
class Foo(private val transactionOperation: TransactionOperation) {
suspend fun saveByJooq(entity: Any): Any { TODO("Saving process here.") }
suspend fun call() {
// call saving process with transaction.
transactionOperation { saveByJooq("foo") }
}
}
I use kotlin multiplatform which prohibits using runBlocking in common code since it is not supported by JS implementation.
My goal is to be able to call suspend functions from my non-suspend function like in the example below. Also I do not care about JS because I will use only JVM, Android, iOS targets
fun main() {
runBlocking {
doSomething()
}
}
suspend fun doSomething() {
}
One solution I can think about is to create expected and actual classes and make runBlocking call separately on each platform actual class, but I want to avoid this as it will cause some code duplication.
runBlocking {
doSomething()
}
Are there any better solution how to bridge blocking and non-blocking code together in common module?
In the common code you can use:
CoroutineScope(Dispatchers.Default).launch {
}
or
MainScope().launch {
}
Depending on the scope you need.
And don't forget to use -native-mt version of coroutines if you're targeting iOS, more info here
This won't block you current thread, as runBlocking does, so if you really need this functionality, you indeed had to implement it with expect/actual, but I have not faced a similar need.
Suppose I have a blocking function because of some third party library. Something along these lines:
fun useTheLibrary(arg: String): String {
val result = BlockingLibrary.doSomething(arg)
return result
}
Invocations to BlockingLibrary.doSomething should run on a separate ThreadPoolExecutor.
What's the proper way (assuming there is a way) of achieving this with kotlin?
Note: I've read this thread but seems pretty outdated
If the blocking code is blocking because of CPU use, you should use Dispatchers.Default. If it is network- or disk-bound, use Dispatchers.IO. You can make this into a suspending function and wrap the blocking call in withContext to allow this function to properly suspend when called from a coroutine:
suspend fun useTheLibrary(arg: String): String = withContext(Dispatchers.Default) {
BlockingLibrary.doSomething(arg)
}
If you need to use a specific ThreadPoolExecutor because of API requirements, you can use asCoroutineDispatcher().
val myDispatcher = myExecutor.asCoroutineDispatcher()
//...
suspend fun useTheLibrary(arg: String): String = withContext(myDispatcher) {
BlockingLibrary.doSomething(arg)
}
If your library contains a callback-based way to run the blocking code, you can convert it into a suspend function using suspendCoroutine() or suspendCancellableCoroutine(). In this case, you don't need to worry about executors or dispatchers, because it's handled by the library's own thread pool. Here's an example in the Retrofit library, where they convert their own callback-based API into a suspend function.
Lets say I have these Kotlin suspend functions doing various operations:
suspend fun operation1(){ //some code }
suspend fun operation2(){ //some code }
suspend fun operation3(){ //some code }
suspend fun operation4(){ //some code }
suspend fun operation5(){ //some code }
I have a function which gets called by an external library (which we can't modify), where I make calls to these functions (of course its logic is more complex in practice). Let's assume the library can make any number of calls in a short time.
fun calledByLibrary(someParam: String){
GlobalScope.launch{
when(someParam){
"someValue1" -> operation1()
"someValue2" -> operation2()
"someValue3" -> operation3()
"someValue4" -> operation4()
"someValue5" -> operation5()
}
}
}
Now, we had a bug, and figured that operation3 and operation4 can not be suspended to run in 'parallel'. They even need to be executed in the same order as called by the external library. For all the other combinations it's ok to do that, so we would like to keep them as suspend to support this.
What's the best way to solve this?
One simple, and easy way would be to use a Mutex in operation3 and operation4 to ensure they don't run in the same time, like this:
val mutex = Mutex()
suspend fun operation3(){ mutex.withLock(){/*some code*/} }
suspend fun operation4(){ mutex.withLock(){/*some code*/} }
But this does not guarantee that they will be executed in the order they have been requested by the library.
Also tried searching for running coroutines in a queue, but surprisingly did not find anything trivial. (Current solution is using a single threaded dispatcher and runBlocking, which is far from ideal)
After tying a lot of things, seems like an actor can be a solution, as a lot of people mentioned using channels, which is a part of the actors. But implementing an actor seemed a complex solution, for such a simple issue, so I tried to avoid it and find a more simple one.
Turned out that the idea with the mutex can be easily fixed to also ensure the order of execution, by using the same thread in the coroutine.
The operation functions already had the logic to use the proper thread when needed (if they did not it would be a trivial thing to add). So the only change needed is to launch the coroutine in the callback on the same thread always. This can be done easily by using Dispachers.Unconfined, or if you want to avoid that, you can create your own single threaded dispacher too.
So, the of course highly simplified solution, would look like this:
fun calledByLibrary(someParam: String){
//You can use any single threaded dispatcher here
//if you want to avoid the unconfined
GlobalScope.launch(Dispachers.Unconfined){
when(someParam){
"someValue1" -> operation1()
"someValue2" -> operation2()
"someValue3" -> operation3()
"someValue4" -> operation4()
"someValue5" -> operation5()
}
}
}
val mutex = Mutex()
suspend fun operation1(){ launch{/*some code*/} }
suspend fun operation2(){ launch{/*some code*/} }
suspend fun operation3(){ mutex.withLock(){launch{/*some code*/}} }
suspend fun operation4(){ mutex.withLock(){launch{/*some code*/}} }
suspend fun operation5(){ launch{/*some code*/} }
You can also do something like this from within a coroutine scope. The await() makes sure to wait for the results before continueing.
val resultOne = async { repository.getSomethingFromApi() }
val resultTwo = async { repository.getSomethingElseFromApi() }
val resultOneFromApi = resultOne.await()
val resultTwoFromApi = resultTwo.await()
I'm trying to poll a paginated API and provide new items to the user as they appear.
fun connect(): Sequence<T> = buildSequence {
while (true) {
// result is a List<T>
val result = dataSource.getFirstPage()
yieldAll(/* the new data in `result` */)
// Block the thread for a little bit
}
}
Here's the sample usage:
for (item in connect()) {
// do something as each item is made available
}
My first thought was to use the delay function, but I get this message:
Restricted suspended functions can only invoke member or extension suspending functions on their restricted coroutine scope
This is the signature for buildSequence:
public fun <T> buildSequence(builderAction: suspend SequenceBuilder<T>.() -> Unit): Sequence<T>
I think this message means that I can only use the suspend functions in SequenceBuilder: yield and yieldAll and that using arbitrary suspend function calls aren't allowed.
Right now I'm using this to block the sequence building by one second after every time the API is polled:
val resumeTime = System.nanoTime() + TimeUnit.SECONDS.toNanos(1)
while (resumeTime > System.nanoTime()) {
// do nothing
}
This works, but it really doesn't seem like a good solution. Has anybody encountered this issue before?
Why does it not work? Some research
When we look at buildSequence, we can see that it takes an builderAction: suspend SequenceBuilder<T>.() -> Unit as its argument. As a client of that method, you'll be able to hand on a suspend lambda that has SequenceBuilder as its receiver (read about lambda with receiver here).
The SequenceBuilder itself is annotated with RestrictSuspension:
#RestrictsSuspension
#SinceKotlin("1.1")
public abstract class SequenceBuilder<in T> ...
The annotation is defined and commented like this:
/**
* Classes and interfaces marked with this annotation are restricted
* when used as receivers for extension `suspend` functions.
* These `suspend` extensions can only invoke other member or extension
* `suspend` functions on this particular receiver only
* and are restricted from calling arbitrary suspension functions.
*/
#SinceKotlin("1.1") #Target(AnnotationTarget.CLASS) #Retention(AnnotationRetention.BINARY)
public annotation class RestrictsSuspension
As the RestrictSuspension documentation tells, in the case of buildSequence, you can pass a lambda with SequenceBuilder as its receiver but with restricted possibilities since you'll only be able to call "other member or extension suspend functions on this particular receiver". That means, the block passed to buildSequence may call any method defined on SequenceBuilder (like yield, yieldAll). Since, on the other hand, the block is "restricted from calling arbitrary suspension functions", using delay does not work. The resulting compiler error verifies it:
Restricted suspended functions can only invoke member or extension suspending functions on their restricted coroutine scope.
Ultimately, you need to be aware that the buildSequence creates a coroutine that is an example of a synchronous coroutine. In your example, the sequence code will be executed in the same thread that consumes the sequence by calling connect().
How to delay the sequence?
As we learned, The buildSequence creates a synchronous sequence. It's fine to use regular Thread blocking here:
fun connect(): Sequence<T> = buildSequence {
while (true) {
val result = dataSource.getFirstPage()
yieldAll(result)
Thread.sleep(1000)
}
}
But, do you really want an entire thread to be blocked? Alternatively, you can implement asynchronous sequences as described here. As a result, using delay and other suspending functions will be valid.
Just for an alternate solution...
If what you're really trying to do is asynchronously produce elements, you can use Flows which are basically asynchronous sequences.
Here is a quick table:
Sync
Async
Single
Normal valuefun example(): String
suspendingsuspend fun example(): Stringorfun example(): Deferred<String>
Many
Sequencefun example(): Sequence<String>
Flowfun example(): Flow<String>
You can convert your Sequence<T> to a Flow<T> by replacing the sequence { ... } builder with the flow { ... } builder and then replace yield/yieldAll with emit/emitAll:
fun example(): Flow<String> = flow {
(1..5).forEach { getString().let { emit(it) } }
}
suspend fun getString(): String = { ... }
So, for your example:
fun connect(): Flow<T> = flow {
while (true) {
// Call suspend function to get data from dataSource
val result: List<T> = dataSource.getFirstPage()
emitAll(result)
// _Suspend_ for a little bit
delay(1000)
}
}