I want to observe the download progress by a Flow,
so I wrote a function like this:
suspend fun downloadFile(file: File, url: String): Flow<Int>{
val client = HttpClient(Android)
return flow{
val httpResponse: HttpResponse = client.get(url) {
onDownload { bytesSentTotal, contentLength ->
val progress = (bytesSentTotal * 100f / contentLength).roundToInt()
emit(progress)
}
}
val responseBody: ByteArray = httpResponse.receive()
file.writeBytes(responseBody)
}
}
but the onDownload will be called only once, and the file will not be downloaded. If I remove the emit(progress) it will work.
io.ktor:ktor-client-android:1.6.7
Use callbackFlow instead of flow. A regular flow can't launch background code, and can only emit values from code inside the flow itself. Meanwhile, a callback flow can launch other work in the background, and then receive callbacks from it.
Related
The problem is very simple, but I can't really seem to wrap my head around it. I'm launching a non-blocking thread in the IO scope in order to read from a file. However, I can't get the result in time before I return from the method - it always returns the initial empty value "". What am I missing here?
private fun getFileContents(): String {
var result = ""
val fileName = getFilename()
val job = CoroutineScope(Dispatchers.IO).launch {
kotlin.runCatching {
val file = getFile(fileName)
file.openFileInput().use { inputStream ->
result = String(inputStream.readBytes(), Charsets.UTF_8)
}
}
}
return result
}
Coroutines are launched asynchronously. Your non-suspending function cannot wait for the result without blocking. For more information about why asynchronous code results in your function returning with the default result, read the answers here.
getFileContents() has to be a suspend function to be able to return something without blocking, in which case you don't need to launch a coroutine either. But then whatever calls this function must be in a suspend function or coroutine.
private suspend fun getFileContents(): String = withContext(Dispatchers.IO) {
val fileName = getFilename()
kotlin.runCatching {
val file = getFile(fileName)
file.openFileInput().use { inputStream ->
result = String(inputStream.readBytes(), Charsets.UTF_8)
}
}.getOrDefault("")
}
There are two "worlds" of code: either you are in a suspending/coroutine context or you are not. When you are in a function that is not a suspend function, you can only return results that can be computed immediately, or you can block until the result is ready.
Generally, if you're using coroutines, you launch a coroutine at some high level in your code, and then you are free to use suspend functions everywhere because almost all of your code is initially triggered by a coroutine. By "high level", I mean you launch the coroutine when a UI screen appears or a UI button is pressed, for example.
Basically, your coroutine launches are usually in UI listeners and UI event functions, not in lower-level code like the function in your question. The coroutine calls a suspend function, which can call other suspend functions, so you don't need to launch more coroutines to perform your various sequential tasks.
The alternate solution is to return a Deferred with the result, like this:
private fun getFileContents(): Deferred<String> {
val fileName = getFilename()
return CoroutineScope(Dispatchers.IO).async {
kotlin.runCatching {
val file = getFile(fileName)
file.openFileInput().use { inputStream ->
result = String(inputStream.readBytes(), Charsets.UTF_8)
}
}.getOrDefault("")
}
}
But to unpack the result, you will need to call await() on the Deferred instance inside a coroutine somewhere.
I am trying call
override suspend fun getLoginResponse(loginRequest: LoginRequest) = flow {
emit(ApiResult.Loading)
networkCall {
loginService.postLoginResponse(loginRequest)
}.let { apiResult->
apiResult.isSuccessAndNotNull().letOnTrueOnSuspend {
(apiResult.getResult() as? LoginResponse)?.let {
emit(ApiResult.Success(it))
Timber.d(it.toString())
} ?: run { emit(ApiResult.Error(TypeCastException("unknown error.")))
Timber.d(TypeCastException("unknown error."))}
}
}
}.flowOn(Dispatchers.IO)
from my viewModel like this :
private fun loginResponse(email: String, password: String, device: String){
viewModelScope.launch {
try {
var loginRequest = LoginRequest(email, password, device)
loginResponseFromServer = loginRepository.getLoginResponse(loginRequest)
.asLiveData(viewModelScope.coroutineContext+Dispatchers.Default)
Timber.d(loginResponseFromServer.toString())
}
catch (e: NetworkErrorException){
validationError.value = "Network communication error!"
}
}
}
When I debug or run the code getLoginResponse not even calling. Is there anything I am missing?
First of all, getLoginResponse doesn't need to be a suspend function since it just returns a cold Flow. If you remove the suspend modifier, you won't need a coroutine to call it or convert it to LiveData.
Second, a LiveData that is built with .asLiveData() doesn't begin to collect the Flow (remains cold) until it first becomes active. This is in the docs for the function. It becomes active when it receives its first observer, but your code has not begun to observe it, which is why the code in your flow block is never called.
You also don't need to specify a different dispatcher for your LiveData. It doesn't matter which dispatcher you're collecting in since collecting it isn't blocking code.
However, LiveData isn't something that should be collected within a ViewModel. It's for UI to interact. The LiveData should be observed from the Fragment.
You need to move your catching of the network exception into your flow builder. The exception will not be thrown at the time of creating the Flow or LiveData, but rather at the time the request is being made (in the Flow's execution).
I'm not sure exactly how to rewrite your flow builder to properly catch because it has functions I haven't seen. Just a tip, but chaining together lots of scope functions into one statement makes code hard to read and reason about.
So to do this as LiveData, you can change your code as follows:
private fun loginResponse(email: String, password: String, device: String): LiveData<LoginResponse> {
val loginRequest = LoginRequest(email, password, device)
return loginRepository.getLoginResponse(loginRequest)
.asLiveData()
}
And then observe it in your Fragment.
However
LiveData and Flow don't really fit this use case, because you want to make a single request and get a single response. Your repository should just expose a suspend function that returns the response. Then your ViewModel can have a suspend function that just passes through the response by calling the repository's suspend function.
I would like to use a Flow as a return type for all functions in my repository. For ex:
suspend fun create(item:T): Flow<Result<T>>
This function should call 2 data sources: remote(to save data on the server) and local(to save returned data from the server locally). The question is how I can implement this scenario:
try to save data with RemoteDataSource
if 1. fails - try it N times with M timeout
if data has finally returned from the server - same them locally with LocalDataSource
return flow with locally saved data
RemoteDataSource and LocalDataSource both have fun create with the same signature:
suspend fun create(item:T): Flow<Result<T>>
So they both return flow of data. If you have any ideas about how to implement it, I will be grateful.
------ Update #1 ------
a part of a possible solution:
suspend fun create(item:T): Flow<T> {
// save item remotely
return remoteDataSource.create(item)
// todo: call retry if fails
// save to local a merge two flows in one
.flatMapConcat { remoteData ->
localDataSource.create(remoteData)
}
.map {
// other mapping
}
}
Is it a working idea?
I think you have the right idea but you are trying to do everything at once.
What I found works best (and easily) is to have:
an exposed flow of data coming from your local datasource (easy with Room)
one or more exposed suspend functions like create or refresh that operate on the remote data source and save to the local one (if there is no error)
For ex I have a repository that fetches vehicles in my project (the isCurrent info is only local and isLeft/isRight is because I use Either but any error handling applies):
class VehicleRepositoryImpl(
private val localDataSource: LocalVehiclesDataSource,
private val remoteDataSource: RemoteVehiclesDataSource
) : VehicleRepository {
override val vehiclesFlow = localDataSource.vehicleListFlow
override val currentVehicleFlow = localDataSource.currentVehicleFLow
override suspend fun refresh() {
remoteDataSource.getVehicles()
.fold(
ifLeft = { /* handle errors, retry, ... */ },
ifRight = { reset(it) }
)
}
private suspend fun reset(vehicles: List<VehicleEntity>) {
val current = currentVehicleFlow.first()
localDataSource.reset(vehicles)
if (current != null) localDataSource.setCurrentVehicle(current)
}
override suspend fun setCurrentVehicle(vehicle: VehicleEntity) =
localDataSource.setCurrentVehicle(vehicle)
override suspend fun clear() = localDataSource.clear()
}
Hope this helps and you can adapt it to your case :)
Is there any kotlin idiomatic way to read a file content's asynchronously? I couldn't find anything in documentation.
A least as of Java 7 (which is where Android is stuck), there isn't any API that would tap into the low-level async file IO support (like io_uring). There is a class called AsynchronousFileChannel, but, as its docs state,
An AsynchronousFileChannel is associated with a thread pool to which tasks are submitted to handle I/O events and dispatch to completion handlers that consume the results of I/O operations on the channel.
That makes it no better than the following, bog-standard Kotlin idiom:
launch {
val contents = withContext(Dispatchers.IO) {
FileInputStream("filename.txt").use { it.readBytes() }
}
processContents(contents)
}
go_on_with_other_stuff_while_file_is_loading()
This uses Kotlin's own dedicated IO thread pool and unblocks the UI thread. If you're on Android, that is your actual concern, anyway.
Java NIO Asynchronous Channel is the tool you want.
Check out this AsynchronousFileChannel.aRead extension function from coroutine example:
suspend fun AsynchronousFileChannel.aRead(buf: ByteBuffer): Int =
suspendCoroutine { cont ->
read(buf, 0L, Unit, object : CompletionHandler<Int, Unit> {
override fun completed(bytesRead: Int, attachment: Unit) {
cont.resume(bytesRead)
}
override fun failed(exception: Throwable, attachment: Unit) {
cont.resumeWithException(exception)
}
})
}
You just open an AsynchronousFileChannel then call this aRead() in a coroutine,
val channel = AsynchronousFileChannel.open(Paths.get(fileName))
try {
val buf = ByteBuffer.allocate(4096)
val bytesRead = channel.aRead(buf)
} finally {
channel.close()
}
It's an essential function, don't know why it is not part of coroutine-core lib.
javasync/RxIo uses Java NIO Asynchronous Channel to provide a non-blocking API to read and write a file content's asynchronously, including kotlin idiomatic way. Next you have two examples: one reading/writing in bulk through coroutines, and other iterating lines through an asynchronous Kotlin Flow:
suspend fun copyNio(from: String, to: String) {
val data = Path(from).readText() // suspension point
Path(to).writeText(data) // suspension point
}
fun printLinesFrom(filename: String) {
Path(filename)
.lines() // Flow<String>
.onEach(::println)
.collect() // block if you want to wait for completion
}
Disclaimer I am the author and main contributor of javasync/RxIo
I need to upload many files to S3, it would take hours to complete that job sequentially. That's exactly what Kotlin's new coroutines excels in, so I wanted to give them a first try instead of fiddling around again with some Thread-based execution service.
Here is my (simplified) code:
fun upload(superTiles: Map<Int, Map<Int, SuperTile>>) = runBlocking {
val s3 = AmazonS3ClientBuilder.standard().withRegion("eu-west-1").build()
for ((x, ys) in superTiles) {
val jobs = mutableListOf<Deferred<Any>>()
for ((y, superTile) in ys) {
val job = async(CommonPool) {
uploadTile(s3, x, y, superTile)
}
jobs.add(job)
}
jobs.map { it.await() }
}
}
suspend fun uploadTile(s3: AmazonS3, x: Int, y: Int, superTile: SuperTile) {
val json: String = "{}"
val key = "$s3Prefix/x4/$z/$x/$y.json"
s3.putObject(PutObjectRequest("my_bucket", ByteArrayInputStream(json.toByteArray()), metadata))
}
The problem: the code is still very slow and logging reveals that requests are still executed sequentially: a job is finished before the next one is created. Only in very few cases (1 out of 10) I see jobs running concurrently.
Why does the code not run much faster / concurrently? What can I do about it?
Kotlin coroutines excel when you work with asynchronous API, while AmazonS3.putObject API that you are using is an old-school blocking, synchronous API, so you get only as many concurrent uploads as the number of threads in the CommonPool that you are using. There is no value in marking your uploadTile function with suspend modified, because it does not use any suspending functions in its body.
The first step in getting more throughput in your upload task is to start using asynchronous API for that. I'd suggest to look at Amazon S3 TransferManager for that purse. See if that gets your problem solved first.
Kotlin coroutines are designed to help you to combine your async APIs into a easy-to-use logical workflows. For example, it is straightforward to adapt asynchronous API of TransferManager for use with coroutines by writing the following extension function:
suspend fun Upload.await(): UploadResult = suspendCancellableCoroutine { cont ->
addProgressListener {
if (isDone) {
// we know it should not actually wait when done
try { cont.resume(waitForUploadResult()) }
catch (e: Throwable) { cont.resumeWithException(e) }
}
}
cont.invokeOnCompletion { abort() }
}
This extension enables you to write very fluent code that works with TransferManager and you can rewrite your uploadTile function to work with TransferManager instead of working with blocking AmazonS3 interface:
suspend fun uploadTile(tm: TransferManager, x: Int, y: Int, superTile: SuperTile) {
val json: String = "{}"
val key = "$s3Prefix/x4/$z/$x/$y.json"
tm.upload(PutObjectRequest("my_bucket", ByteArrayInputStream(json.toByteArray()), metadata))
.await()
}
Notice, how this new version of uploadTile uses a suspending function await that was defined above.