android Camera2 take frames and convert to bitmap - kotlin

I'm trying to convert a frame format to bitmap but always receiving null. Can someone help me with this issue?
I am using camera2 in kotlin
private val imageListener = ImageReader.OnImageAvailableListener { reader ->
val image = reader?.acquireLatestImage()
try {
val imageBytes = image?.planes?.first()?.buffer?.toByteArray()
val imageBase64 = Base64.encodeToString(imageBytes, Base64.DEFAULT)
Log.d(TAG, imageBase64)}
catch (e: Exception) {
e.printStackTrace()
}
image?.close()
}
}
I'm trying to change it to base46, but the base64 doesn't decode. When I try to decode it online using a PNG or JPG, I always get an image with a 400x400 transparent color.

I'm assuming the capture request was configured to encode the output to a JPEG?
If so you'll want to read the bytes to a byte array and then convert the jpeg to a Bitmap.
val buffer = image?.?planes[0]?.buffer ?: return
val jpegData = ByteArray(buffer.remaining()).apply { buffer.get(this) }
val bitmap = BitmapFactory.decodeByteArray(jpegData, 0, jpegData.size)
Have you considered CameraX? It'll simplify capturing images and will manage a lot of quirks. https://developer.android.com/training/camerax/take-photo
Here's a quick overview of something you could try:
class CameraActivity : AppCompatActivity() {
...
val cameraController = LifecycleCameraController(this)
cameraController.bindToLifecycle(this as LifecycleOwner)
// Attach the CameraController to PreviewView
val previewView = findViewById(R.id.preview_view)
previewView.setController(cameraController)
// Take a picture
cameraController.takePicture(
ContextCompat.getMainExecutor(this), object : ImageCapture.OnImageCapturedCallback {
override fun onCaptureSuccess(image: ImageProxy) {
val buffer = image.planes[0].buffer
val jpegData = ByteArray(buffer.remaining()).apply { buffer.get(this) }
val bitmap BitmapFactory.decodeByteArray(jpegData, 0, jpegData.size)
// show bitmap
}
override fun onError(exception: ImageCaptureException) {
Log.e(TAG, "Error: $exception")
}
})
}

Related

MutableLiveData, LiveData List empty or add item first position

I don't quite understand the behavior of MutableLiveData or I can't find the proper documentation.
But I need help.
I need to add an element to the first position of the array and tried with "plus" but it doesn't work properly. Or is there a way to add multiple elements of an array in the first position?
_series.value = _series.value?.plus(it) // ??? 11111111
I can't figure out how to delete all the elements of the array, I did it with
_series.value = null // ??? 2222222
_series.value = ArrayList() /// ??? 2222222
but I don't think it is the best way to do it.
class SerieViewModel(val database: SerieDatabaseDao) : ViewModel() {
private val _series = MutableLiveData<List<Serie>?>()
val series: LiveData<List<Serie>?>
get() = _series
fun fetchSeries(code: String) {
viewModelScope.launch {
try {
val response = CodesApi.retrofitService.getSeries(code)
val series = response.body()?.data
series?.forEach {
_series.value = _series.value?.plus(it) // ??? 11111111
}
}
} catch (e: Exception) {
Log.d(TAG, "fetchSeries: ${e.message}")
}
}
}
fun clearDb() = viewModelScope.launch {
database.clear()
_series.value = null // ??? 2222222
_series.value = ArrayList<Serie>() /// ??? 2222222
}
thanks
You are storing a List<Serie?> which is fine thats what you should do, but you can't add elements to List, you need to convert List to ArrayList and add elements to the ArrayList than update the MutableLiveData, follow the code below.
to clear all the elements from the List you can set the value to emptyList(), follow the code below.
Note: Take a look about the difference between List, ArrayList, Array in kotlin they are not the same
class SerieViewModel(val database: SerieDatabaseDao) : ViewModel() {
private val _series = MutableLiveData<List<Serie>?>()
val series: LiveData<List<Serie>?>
get() = _series
fun fetchSeries(code: String) {
viewModelScope.launch {
try {
val response = CodesApi.retrofitService.getSeries(code)
val series = response.body()?.data
val liveDataSeries =
if (_series.value == null) emptyList<Serie>() // set the array list to empty if _series.value is null
else ArrayList(_series.value) // else: add _series.value elements to the array list
series?.forEach {
liveDataSeries.add(it) // add each serie to the array list
}
_series.value = liveDataSeries // update _series.value
} catch (e: Exception) {
Log.d(TAG, "fetchSeries: ${e.message}")
}
}
}
fun clearDb() = viewModelScope.launch {
database.clear()
_series.value = emptyList()
}
}

How do you fling a ScrollState in Jetpack Compose?

For reasons that have to do with Jetpack Compose input modifiers consuming all MotionEvents, I find myself writing my own scroll routine for a Composable, of which I have access to the ScrollState.
I have figured out everything I need except flinging. I can't see how to apply performFling(initialVelocity) on a ScrollState. All I can find in the docs are ScrollState.scrollTo and ScrollState.scrollBy, which aren't so useful with flings, since the scroll destination or size is unknown.
I also can't find a ScrollState listener, similar to onScrollStateChanged(state: Int) in the old Android world, that fires when scrolling state changes.
Here is what I have in case somebody can point me in the right direction:
var lastY: Float? = null
var velocityTracker: VelocityTracker? = null
fun scroll(event: MotionEvent) {
when (event.action) {
MotionEvent.ACTION_DOWN -> {
velocityTracker = VelocityTracker.obtain()
lastY = event.y
}
MotionEvent.ACTION_UP -> {
lastY = null
velocityTracker?.let {
it.computeCurrentVelocity(1000)
val initialVelocity = it.yVelocity
velocityTracker?.recycle()
coroutineScope.launch {
???? scrollState.PERFORMFLING?(initialVelocity) ????
AND THEN WHEN THE FLING IS FINISHED viewModel.scrollOffset = scrollState.value
}
}
}
else -> {
velocityTracker?.addMovement(event)
lastY?.let {
val scrollAmount = it - event.y
lastY = event.y
coroutineScope.launch {
scrollState.scrollBy(scrollAmount)
viewModel.scrollOffset = scrollState.value
}
}
}
}
}
You could try using a nestedScroll:
val nestedScrollConnection = remember {
object : NestedScrollConnection {
override suspend fun onPostFling(consumed: Velocity, available: Velocity): Velocity {
return super.onPostFling(consumed, available)
}
override suspend fun onPreFling(available: Velocity): Velocity {
return super.onPreFling(available)
}
}
}
Column(modifier = Modifier
.verticalScroll(rememberScrollState())
.nestedScroll(nestedScrollConnection)) {
}
If this doesn't work, just search
cs.android.com
for theNestedScrollConnection and it should give you a hint on how to handle flinging in Compose. Maybe NestedScrollConnection is all you need since it provides support for scrolling as well. You probably can ditch your code and just use NestedScrollConnection. To see NestedScrollConnection in action, check out the demo:
https://github.com/JohannBlake/Jetmagic

Kotlin - Assign value to variable or break and return function

I would like to assign new value to my variable but if it is null I would like to break the flow, print out some logs and then call to return.
My variable's type is Image and it cannot be null.
Below is my solution.
Basic code I started:
val img = reader.acquireLatestImage() ?: return
Code I finally created:
val img : Image = (reader.acquireLatestImage() ?: (Log.d(TAG, "Image is null!").also { return })) as Image
Is it the best way to do it?
Is it possible to create extension for that?
It turns out the compiler is not smart enough to deal with also here but if you used let it would work without the cast:
val img : Image = reader.acquireLatestImage() ?: Log.d(TAG, "Image is null!").let { return }
This is because the type of Log.d(...) is Unit, so the type of Log.d(...).also { ... } is also Unit in general. The compiler doesn't see that the non-local return in this specific case will prevent the whole expression Log.d(...).also { ... } from resulting in a value (in practice it should have the type Nothing, but the compiler doesn't see it and still gives it the type Unit).
let, on the contrary, returns the value of the lambda expression, which in this case is really of type Nothing and therefore is correctly understood by the compiler.
Another option would be to reverse your null-handling section:
val img = reader.acquireLatestImage() ?: return Unit.also { Log.d(TAG, "Image is null!!") }
Is it the best way to do it?
In this form, I'm not convinced even by my own versions, which are probably not the most maintainable way of doing it. I would personally favor a plain if statement here for the log and return. The smart casting of Kotlin with flow typing will allow you to use your img variable with type Image after the check, even though it starts out with type Image?:
val img = reader.acquireLatestImage() // this is of type "Image?"
if (img == null) {
Log.d(TAG, "Image is null!")
return
}
// now img is smart-cast to the type Image (non-nullable)
Is it possible to create extension for that?
You won't be able to encapsulate the non-local return inside an extension function, but you could create an extension function for the logging, and then use a plain elvis operator:
fun YourReader.acquireLatestImageOrLog(): Image? {
val img = acquireLatestImage()
if (img == null) {
Log.d(TAG, "Image is null")
}
return img
}
val img = reader.acquireLatestImageOrLog() ?: return
You could even go more general:
fun <T> T?.orLog(message: String): T? {
if (this == null) {
Log.d(TAG, message)
}
return this
}
val img = reader.acquireLatestImage().orLog("Image is null!") ?: return
Here's one example using run:
val img: Image = reader.acquireLatestImage() ?: run {
Log.d(TAG, "Image is null!")
// You can place also more code in here -->
// ...
// <--
return
}
Or if you simply want it in oneliner, you can place ; after Log.d(...):
val img: Image = reader.acquireLatestImage() ?: run { Log.d(TAG, "Image is null!"); return }

Kotlin flows as a message queue between coroutines

I'm attempting to use Kotlin's Flow class as a message queue to transfer data from a producer (a camera) to a set of workers (image analyzers) running on separate coroutines.
The producer in my case is a camera, and will run substantially faster than the workers. Back pressure should be handled by dropping data so that the image analyzers are always operating on the latest images from the camera.
When using channels, this solution works, but seems messy and does not provide an easy way for me to translate the data between the camera and the analyzers (like flow.map).
class ImageAnalyzer<Result> {
fun analyze(image: Bitmap): Result {
// perform some work on the image and return a Result. This can take a long time.
}
}
class CameraAdapter {
private val imageChannel = Channel<Bitmap>(capacity = Channel.RENDEZVOUS)
private val imageReceiveMutex = Mutex()
// additional code to make this camera work and listen to lifecycle events of the enclosing activity.
protected fun sendImageToStream(image: CameraOutput) {
// use channel.offer to ensure the latest images are processed
runBlocking { imageChannel.offer(image) }
}
#OnLifecycleEvent(Lifecycle.Event.ON_DESTROY)
fun onDestroy() {
runBlocking { imageChannel.close() }
}
/**
* Get the stream of images from the camera.
*/
fun getImageStream(): ReceiveChannel<Bitmap> = imageChannel
}
class ImageProcessor<Result>(workers: List<ImageAnalyzer<Result>>) {
private val analysisResults = Channel<Result>(capacity = Channel.RENDEZVOUS)
private val cancelMutex = Mutex()
var finished = false // this can be set elsewhere when enough images have been analyzed
fun subscribeTo(channel: ReceiveChannel<Bitmap>, processingCoroutineScope: CoroutineScope) {
// omit some checks to make sure this is not already subscribed
processingCoroutineScope.launch {
val workerScope = this
workers.forEachIndexed { index, worker ->
launch(Dispatchers.Default) {
startWorker(channel, workerScope, index, worker)
}
}
}
}
private suspend fun startWorker(
channel: ReceiveChannel<Bitmap>,
workerScope: CoroutineScope,
workerId: Int,
worker: ImageAnalyzer
) {
for (bitmap in channel) {
analysisResults.send(worker.analyze(bitmap))
cancelMutex.withLock {
if (finished && workerScope.isActive) {
workerScope.cancel()
}
}
}
}
}
class ExampleApplication : CoroutineScope {
private val cameraAdapter: CameraAdapter = ...
private val imageProcessor: ImageProcessor<Result> = ...
fun analyzeCameraStream() {
imageProcessor.subscribeTo(cameraAdapter.getImageStream())
}
}
What's the proper way to do this? I would like to use a ChannelFlow instead of a Channel to pass data between the camera and the ImageProcessor. This would allow me to call flow.map to add metadata to the images before they're sent to the analyzers. However, when doing so, each ImageAnalyzer gets a copy of the same image instead of processing different images in parallel. Is it possible to use a Flow as a message queue rather than a broadcaster?
I got this working with flows! It was important to keep the flows backed by a channel throughout this sequence so that each worker would pick up unique images to operate on. I've confirmed this functionality through unit tests.
Here's my updated code for posterity:
class ImageAnalyzer<Result> {
fun analyze(image: Bitmap): Result {
// perform some work on the image and return a Result. This can take a long time.
}
}
class CameraAdapter {
private val imageStream = Channel<Bitmap>(capacity = Channel.RENDEZVOUS)
private val imageReceiveMutex = Mutex()
// additional code to make this camera work and listen to lifecycle events of the enclosing activity.
protected fun sendImageToStream(image: CameraOutput) {
// use channel.offer to enforce the drop back pressure strategy
runBlocking { imageChannel.offer(image) }
}
#OnLifecycleEvent(Lifecycle.Event.ON_DESTROY)
fun onDestroy() {
runBlocking { imageChannel.close() }
}
/**
* Get the stream of images from the camera.
*/
fun getImageStream(): Flow<Bitmap> = imageChannel.receiveAsFlow()
}
class ImageProcessor<Result>(workers: List<ImageAnalyzer<Result>>) {
private val analysisResults = Channel<Result>(capacity = Channel.RENDEZVOUS)
private val cancelMutex = Mutex()
var finished = false // this can be set elsewhere when enough images have been analyzed
fun subscribeTo(flow: Flow<Bitmap>, processingCoroutineScope: CoroutineScope): Job {
// omit some checks to make sure this is not already subscribed
return processingCoroutineScope.launch {
val workerScope = this
workers.forEachIndexed { index, worker ->
launch(Dispatchers.Default) {
startWorker(flow, workerScope, index, worker)
}
}
}
}
private suspend fun startWorker(
flow: Flow<Bitmap>,
workerScope: CoroutineScope,
workerId: Int,
worker: ImageAnalyzer
) {
while (workerScope.isActive) {
flow.collect { bitmap ->
analysisResults.send(worker.analyze(bitmap))
cancelMutex.withLock {
if (finished && workerScope.isActive) {
workerScope.cancel()
}
}
}
}
}
fun getAnalysisResults(): Flow<Result> = analysisResults.receiveAsFlow()
}
class ExampleApplication : CoroutineScope {
private val cameraAdapter: CameraAdapter = ...
private val imageProcessor: ImageProcessor<Result> = ...
fun analyzeCameraStream() {
imageProcessor.subscribeTo(cameraAdapter.getImageStream())
}
}
It appears that, so long as the flow is backed by a channel, the subscribers will each get a unique image.

Kotlin Coroutines Run blocking not working as expected and Cannot block execution of for loops

In my app i'm executing two for loops however those for loops need to scheduled in a order here is the use case:
There are two for loops :
1- ImageStickerslist
2-TextStickerslist
What i want to do is after imagestickerslist if properly finised only then textstickerslist will be executed.
Here imagesticker list consists of url path which is used to load images from glide however if those images are of high resolution it eventually makes the thread continue even if the image is not yet loaded from url. To solve this tried adding blocking calls to glide on ready and full method but it won't prove to be of any help. I'm very confused how blocking calls work any help into this will be really appreciated.
Here's my code where for loops are executed:
runBlocking {
launch {
imagestickers.forEach {
runBlocking {
var image = it.path
var x = it.x
var y = it.y
image!!.log()
setimagestickers(image!!, x!!, y!!, it.width!!, it.height!!)
}
}
}.join()
textstickers.forEach {
runBlocking {
var text = it.text.toString()
var color = it.color
var font = it.font
var size = it.size
var x = it.x
var y = it.y
setTextSticker(text, Color.parseColor(color), size!!, x!!, y!!)
}
}
}
Here are my two methods where main computation is taking place:
fun setimagestickers(path:String,x:Int,y:Int,w:Int,h:Int){
Glide.with(this#NewStickerActivity).asBitmap().timeout(6000000).load(path).into(object : CustomTarget<Bitmap>() {
override fun onLoadCleared(placeholder: Drawable?) {
}
override fun onResourceReady(resource: Bitmap, transition: Transition<in Bitmap>?) {
var size: ViewGroup.LayoutParams
var bmp1 = resource
size = UiHelper.getHeightWidth(this#NewStickerActivity, (w).toInt(), (h).toInt())
var resizedBitmap = Bitmap.createScaledBitmap(bmp1, size.width, size.height, false)
var drawable = BitmapDrawable(resources, resizedBitmap)
var dsImageSticker = DrawableSticker(drawable)
dsImageSticker.setTag("ImageSticker")
var pm: List<Int>
if (density > 3.0) {
pm = UiHelper.getmargins(this#NewStickerActivity, (x).toInt(), (y).toInt())
} else {
pm = UiHelper.getmargins(this#NewStickerActivity, (x).toInt(), (y).toInt())
}
Log.i("Hmmm:", pm.get(0).toFloat().toString() + "::" + pm.get(1).toFloat().toString())
stickerView.addStickerAndSetMatrix1(
dsImageSticker,
pm.get(0).toFloat(),
pm.get(1).toFloat()
)
}
})
}
fun setTextSticker(text: String, color: Int,size: Int, x: Int, y: Int){
val bmp1: Bitmap
val drawable: Drawable
var l: List<Int>
if (density > 3.0) {
l = UiHelper.getmargins(this#NewStickerActivity, (x).toInt(), (y * 1.07).toInt())
} else {
l = UiHelper.getmargins(this#NewStickerActivity, x.toInt(), y.toInt())
}
//var tf = Typeface.createFromFile(assets,"fonts/"+path)
var tf = Typeface.createFromAsset(assets, "fonts/Myriad Pro Bold SemiExtended.ttf")
bmp1 = createBitmapFromLayoutWithText(this#NewStickerActivity, size, text, color, 0f, tf, 0f, 0f, color, Gravity.LEFT)
drawable = BitmapDrawable(resources, bmp1)
var dsTextSticker = DrawableSticker(drawable)
dsTextSticker.setTag("textSticker")
Log.i("Hmmm:", l.get(0).toFloat().toString() + "::" + l.get(1).toFloat().toString())
/*if (rotate) {
stic.addStickerAndSetrotate(
dsTextSticker, rotationdegress,
l.get(0).toFloat(),
l.get(1).toFloat()
)
} else {*/
stickerView.addStickerAndSetMatrix1(
dsTextSticker,
l.get(0).toFloat(),
l.get(1).toFloat())
}
UPDATE:
I got this working without coroutines by incrementing and fetching images in a sequence:
Firstly i took a Int and then kept incrementing until it reached list size here is my code:
Firstly i did this :
var i = 0
setimagestickers(imagestickers.get(i).path!!, imagestickers.get(i).x!!, imagestickers.get(i).y!!, imagestickers.get(i).width!!, imagestickers.get(i).height!!)
After that inside on resource ready did the trick!!
Glide.with(this#NewStickerActivity).asBitmap().timeout(6000000).load(path).into(object : CustomTarget<Bitmap>() {
override fun onLoadCleared(placeholder: Drawable?) {
}
override fun onResourceReady(resource: Bitmap, transition: Transition<in Bitmap>?) {
var size: ViewGroup.LayoutParams
var bmp1 = resource
size = UiHelper.getHeightWidth(this#NewStickerActivity, (w).toInt(), (h).toInt())
var resizedBitmap = Bitmap.createScaledBitmap(bmp1, size.width, size.height, false)
var drawable = BitmapDrawable(resources, resizedBitmap)
var dsImageSticker = DrawableSticker(drawable)
dsImageSticker.setTag("ImageSticker")
var pm: List<Int>
if (density > 3.0) {
pm = UiHelper.getmargins(this#NewStickerActivity, (x).toInt(), (y).toInt())
} else {
pm = UiHelper.getmargins(this#NewStickerActivity, (x).toInt(), (y).toInt())
}
Log.i("Hmmm:", pm.get(0).toFloat().toString() + "::" + pm.get(1).toFloat().toString())
stickerView.addStickerAndSetMatrix1(
dsImageSticker,
pm.get(0).toFloat(),
pm.get(1).toFloat()
)
i++
if(i < imagestickers.size){
setimagestickers(imagestickers.get(i).path!!, imagestickers.get(i).x!!, imagestickers.get(i).y!!, imagestickers.get(i).width!!, imagestickers.get(i).height!!)
}
else{
if(textstickers.isNullOrEmpty()){
loader!!.hide()
}
else {
setTextSticker(textstickers.get(j).text!!, Color.parseColor(textstickers.get(j).color), textstickers.get(j).size!!, textstickers.get(j).x!!, textstickers.get(j).y!!)
}
}
}
})
However i'm still wondering how can i solve it with coroutines rather than this approach!!!
Basically, you're making async Glide calls without suspending the coroutine until they are done. The first thing you must change is introduce a suspend fun getImageSticker():
suspend fun getSticker(path: String): Bitmap =
suspendCancellableCoroutine { continuation -> Glide
.with(this#NewStickerActivity)
.asBitmap()
.timeout(6000000)
.load(path)
.into(object : CustomTarget<Bitmap>() {
override fun onResourceReady(resource: Bitmap, x: Transition<in Bitmap>?) {
continuation.resume(resource)
}
})
}
Note that this returns a Bitmap. Now you can just call it as if it was a regular, blocking function, and use its result from the caller side:
suspend fun setimagestickers(path: String, x: Int, y: Int, w: Int, h: Int) {
val resource = getSticker(path)
var size: ViewGroup.LayoutParams
var bmp1 = resource
size = UiHelper.getHeightWidth(this#NewStickerActivity, (w).toInt(), (h).toInt())
var resizedBitmap = Bitmap.createScaledBitmap(bmp1, size.width, size.height, false)
}
Now, if you want to parallelize setimagestickers calls, launch them in a loop:
launch {
imagestickers.map {
launch {
var image = it.path
var x = it.x
var y = it.y
image!!.log()
setimagestickers(image!!, x!!, y!!, it.width!!, it.height!!)
}
}.joinAll()
textstickers.forEach {
...
}
}
None of your code will benefit from runBlocking or the coroutines you launch.
If you are aiming to simply process each of your Image type objects in paralellel due to the long running nature of each task, then you can do something like this:
fun main() {
runBlocking {
imagestickers.map {
launch {
var image = it.path
var x = it.x
var y = it.y
image!!.log()
setimagestickers(image!!, x!!, y!!, it.width!!, it.height!!)
}
}.joinAll()
textstickers.map {
launch {
var text = it.text.toString()
var color = it.color
var font = it.font
var size = it.size
var x = it.x
var y = it.y
setTextSticker(text, Color.parseColor(color), size!!, x!!, y!!)
}
}.joinAll()
}
}
Why I removed some of the stuff you were doing:
The nested runBlocking calls you had did nothing. `runBlocking means "block until whatever is inside this block is finished". Your for loops were not async, and would run until they were done anyway.
The first coroutine launch you had did nothing as well. You launched it, and then immediately called join. This means that you launch a coroutine, but then immediately wait until it is finished before proceeding.
What this does:
Rather than loops, we map each image object to a list of jobs for concurrently running coroutines.
We then wait for every job to finish in each list before proceeding to the next list.
Please Note: It does not look like any of your code benefits from having non-blocking IO, in which case, this parralelism will be limited to the amount of threads you have. That fetching of the image will block the thread it is on (regardless of using coroutines) unless you use a non-blocking library that properly yields while it fetches your image.