How to access camera frames in flutter quickly - camera

Question

I would like to implement near real-time OCR on the camera feed of my flutter app. To do this I would like to access the camera data in a speedy manner.
As far as I can tell I have two options, and have hit roadblocks with both:
Take a screenshot of the CameraPreview by putting a RepaintBoundary around it and creating a RenderRepaintBoundary, and calling boundary.toImage(). The problem with this method is that the .toImage method only seems to capture the painted widgets in the boundary and not the data from the camera preview. Simmilar to the issue described here: https://github.com/flutter/flutter/issues/17687
Capture an image with controller.takePicture(filePath) from Camera 0.2.1, similar to the example docs. The problem here is that it takes super long before the image becomes available (2-3 seconds). I guess that this is because the file is saved to the disc on capture and then needs to be read from the file again.
Is there any way that one can directly access the picture information after capture, to do things like pre-process and OCR?

Answer 1

For "near real-time OCR", you need CameraController#startImageStream
example code
import 'package:camera/camera.dart';
import 'package:flutter/foundation.dart';
import 'package:flutter/material.dart';
void main() => runApp(MaterialApp(home: _MyHomePage()));
class _MyHomePage extends StatefulWidget {
#override
_MyHomePageState createState() => _MyHomePageState();
}
class _MyHomePageState extends State<_MyHomePage> {
dynamic _scanResults;
CameraController _camera;
bool _isDetecting = false;
CameraLensDirection _direction = CameraLensDirection.back;
#override
void initState() {
super.initState();
_initializeCamera();
}
Future<CameraDescription> _getCamera(CameraLensDirection dir) async {
return await availableCameras().then(
(List<CameraDescription> cameras) => cameras.firstWhere(
(CameraDescription camera) => camera.lensDirection == dir,
),
);
}
void _initializeCamera() async {
_camera = CameraController(
await _getCamera(_direction),
defaultTargetPlatform == TargetPlatform.iOS
? ResolutionPreset.low
: ResolutionPreset.medium,
);
await _camera.initialize();
_camera.startImageStream((CameraImage image) {
if (_isDetecting) return;
_isDetecting = true;
try {
// await doSomethingWith(image)
} catch (e) {
// await handleExepction(e)
} finally {
_isDetecting = false;
}
});
}
Widget build(BuildContext context) {
return null;
}
}
This functionality was merged to https://github.com/flutter/plugins but it was not well documented.
Ref:
https://github.com/flutter/flutter/issues/26348
https://github.com/flutter/plugins/pull/965
https://github.com/bparrishMines/mlkit_demo/blob/master/lib/main.dart#L43
https://youtu.be/OAEWySye0BQ?t=1460

Answer 2

A better solution today (2022) for real-time OCR is to use the camera in a loop with a frequency of 500ms and process the image using google ML Kit's Text recognition.