Passing image to tflite model - tensorflow

I loaded tensor-flow Lite model in Flutter but i am having some problem passing the image to the model for prediction. The prediction method allows any object to pass but Flutter is reading images as Files as i am using Image Picker class and I cant find a way to convert file type into an image so that i can convert it into a (28,28,1) which is required by the model.
Thanks, any help would be appreciated.
Uint8List imageToByteListFloat32(
img.Image image, int inputSize, double mean, double std) {
var convertedBytes = Float32List(1 * inputSize * inputSize * 3);
var buffer = Float32List.view(convertedBytes.buffer);
int pixelIndex = 0;
for (var i = 0; i < inputSize; i++) {
for (var j = 0; j < inputSize; j++) {
var pixel = image.getPixel(j, i);
buffer[pixelIndex++] = (img.getRed(pixel) - mean) / std;
buffer[pixelIndex++] = (img.getGreen(pixel) - mean) / std;
buffer[pixelIndex++] = (img.getBlue(pixel) - mean) / std;
}
}
return convertedBytes.buffer.asUint8List();
}
classifyImage(File image) async {
var output = await Tflite.runModelOnBinary(
binary: imageToByteListFloat32(image, 224, 127.5, 127.5),// required
numResults: 10, // defaults to 5
threshold: 0.05, // defaults to 0.1
asynch: true // defaults to true
);
setState(() {
_loading = false;
_outputs = output;
});
}`

You can use the flutter image lib to get an image from a file.
import 'package:image/image.dart' as imageLib;
var image = imageLib.decodeImage(await imageFile.readAsBytes());
Also this github repo from shaquian might be helpful for you!
https://github.com/shaqian/flutter_tflite/blob/master/example/lib/main.dart

Related

TFlite model slow/no inference

I am trying to get an opencv pipeline working on a REV control hub for FTC with tensorflow. I have a trained TFLite model and I am processing it and retrieving the confidences. It works well on an android device as an android app, but it doesn't seem to work with the REV control hub. You may see that I have a variable named here2, this always evaluates to false unless it is before model.process, showing that the model doesn't get processed. Any idea why this is?
package org.firstinspires.ftc.teamcode.vision;
import android.app.Activity;
import android.content.Context;
import android.graphics.Bitmap;
import org.firstinspires.ftc.robotcontroller.internal.FtcRobotControllerActivity;
import org.firstinspires.ftc.robotcore.external.Telemetry;
import org.firstinspires.ftc.teamcode.ml.ModelUnquant;
import org.opencv.core.Mat;
import org.openftc.easyopencv.OpenCvPipeline;
import org.opencv.android.Utils;
import org.tensorflow.lite.DataType;
import org.tensorflow.lite.support.tensorbuffer.TensorBuffer;
import java.io.IOException;
import java.nio.ByteBuffer;
import java.nio.ByteOrder;
public class CircularTagDetectionPipeline extends OpenCvPipeline {
Telemetry telemetry;
public boolean done = false;
public int pos = -1;
public boolean here1 = false;
public boolean here2 = false;
public CircularTagDetectionPipeline(Telemetry telemetry){
this.telemetry = telemetry;
}
#Override
public Mat processFrame(Mat input) {
Bitmap bmp = Bitmap.createBitmap(input.width(), input.height(), Bitmap.Config.ARGB_8888);
Utils.matToBitmap(input, bmp);
pos = classifyImage(bmp);
done = true;
return input;
}
public int classifyImage(Bitmap image){
try {
image = Bitmap.createScaledBitmap(image, 224, 224, false);
//image.setWidth(224);
//image.setHeight(224);
ModelUnquant model = ModelUnquant.newInstance(FtcRobotControllerActivity.contextFTC);
int imageWidth = image.getWidth();
int imageHeight = image.getHeight();
TensorBuffer inputFeature0 = TensorBuffer.createFixedSize(new int[]{1, 224, 224, 3}, DataType.FLOAT32);
ByteBuffer byteBuffer = ByteBuffer.allocateDirect(4*imageWidth*imageHeight*3);
byteBuffer.order(ByteOrder.nativeOrder());
int[] intValues = new int[imageWidth*imageHeight];
image.getPixels(intValues, 0, imageWidth,0, 0, imageWidth, imageHeight);
int pixel = 0;
for(int i = 0; i < imageWidth; i++){
for(int j = 0; j < imageHeight; j++){
int val = intValues[pixel++];
byteBuffer.putFloat(((val >> 16)&0xFF)*(1.f/255.f));
byteBuffer.putFloat(((val >> 8)&0xFF)*(1.f/255.f));
byteBuffer.putFloat((val & 0xFF)*(1.f/255.f));
}
telemetry.addLine(i+"");
}
here1 = true;
inputFeature0.loadBuffer(byteBuffer);
ModelUnquant.Outputs outputs = model.process(inputFeature0);
TensorBuffer outputFeature0 = outputs.getOutputFeature0AsTensorBuffer();
here2 = true;
float[] confidences = outputFeature0.getFloatArray();
int maxPos = 0;
float maxConfidence = 0;
for(int i = 0; i < confidences.length; i++){
if(confidences[i] > maxConfidence){
maxConfidence = confidences[i];
maxPos = i;
}
}
String[] classes = {"1", "2", "3"};
int posReturn = Integer.parseInt(classes[maxPos]);
model.close();
return posReturn;
} catch (IOException e) {
e.printStackTrace();
return 0;
}
}
}
I am expecting the robot to go to execute the desired trajectory based on the confidences, but confidences never get processed. I tried on an android app and it worked, just not on the control hub.

Creating an action for moving layers on the x axis to the widest width of a selected layer?

I want to make a variable action that separates a series of layers based on the widest width of one of those selected layers. 4 sprites are 100, 100, 100, 200 pixels wide. It'd separate the layers all by 200 pixels on the x axis and make the image 800 pixels wide total.
I am able to do this easily if all layers are the same but cannot make it work with variability. Not sure if there are any photoshop wizards willing to lend me their time to make a script but it would be extremely helpful!
I'd do something like this. This presumes that there're no groups, no Background layer, layers are normal layers and all document layers are used. Before-after (largest was 150px):
If you don't need them to change Y position, change 0 - layersInfo[i].top to 0 on line 13.
function main()
{
var doc = activeDocument,
layers = doc.layers, //getting all top layers
layersInfo = getInfo().reverse(), //getting layers info and reversing the array because DOM indexes are different order than AM indexes
elWidth = getWidestElement(layersInfo); // getting widest element
doc.resizeCanvas(elWidth * layers.length, doc.height.as("px"), AnchorPosition.TOPLEFT); // resizing canvas size to new width: widest element * number of elements
for (var i = 0; i < layers.length; i++) // for every top layer...
{
doc.activeLayer = layers[i]; // selecting the layer
layers[i].translate(i * elWidth - layersInfo[i].left, 0 - layersInfo[i].top) // moving it to top left corner of each block
}
function getInfo()
{
var layers = 1,
lyrs = [];
while (true)
{
ref = new ActionReference();
ref.putIndex(charIDToTypeID('Lyr '), layers);
try
{
var desc = executeActionGet(ref);
}
catch (err)
{
break;
}
var lyr = {},
bounds = desc.getObjectValue(stringIDToTypeID("bounds"));;
lyr.top = bounds.getDouble(stringIDToTypeID("top"));
lyr.left = bounds.getDouble(stringIDToTypeID("left"));
lyr.width = bounds.getDouble(stringIDToTypeID("width"));
lyrs.push(lyr)
layers++;
}
return lyrs
}; // end of getInfo()
function getWidestElement(layers)
{
var curWidth = 0;
for (var i = 0; i < layers.length; i++)
{
if (layers[i].width > curWidth) curWidth = layers[i].width;
}
return curWidth
}; // end of getWidestElement()
}
var curUnits = app.preferences.rulerUnits;
app.preferences.rulerUnits = Units.PIXELS;
try
{
app.activeDocument.suspendHistory("temp", "main()");
}
catch (e)
{
alert(e + '\nLine: ' + e.line)
}
app.preferences.rulerUnits = curUnits;

What is the optimal render loop in Dart 2?

I am looking for ideas regarding an optimal/minimal structure for the inner render loop in Dart 2, for a 2d game (if that part matters).
Clarification / Explanation: Every framework / language has an efficient way to:
1) Deal with time.
2) Render to the screen (via memory, a canvas, an image, or whatever).
For an example, here is someone that answered this for the C# language. Being new to Flutter / Dart, my first attempt (below), is failing to work and as of right now, I can not tell where the problem is.
I have searched high and low without finding any help on this, so if you can assist, you have my eternal gratitude.
There is a post on Reddit by ‘byu/inu-no-policemen’ (a bit old). I used this to start. I suspect that it is crushing the garbage collector or leaking memory.
This is what I have so far, but it crashes pretty quickly (at least in the debugger):
import 'dart:ui';
import 'dart:typed_data';
import 'dart:math' as math;
import 'dart:async';
main() async {
var deviceTransform = new Float64List(16)
..[0] = 1.0 // window.devicePixelRatio
..[5] = 1.0 // window.devicePixelRatio
..[10] = 1.0
..[15] = 1.0;
var previous = Duration.zero;
var initialSize = await Future<Size>(() {
if (window.physicalSize.isEmpty) {
var completer = Completer<Size>();
window.onMetricsChanged = () {
if (!window.physicalSize.isEmpty) {
completer.complete(window.physicalSize);
}
};
return completer.future;
}
return window.physicalSize;
});
var world = World(initialSize.width / 2, initialSize.height / 2);
window.onBeginFrame = (now) {
// we rebuild the screenRect here since it can change
var screenRect = Rect.fromLTWH(0.0, 0.0, window.physicalSize.width, window.physicalSize.height);
var recorder = PictureRecorder();
var canvas = Canvas(recorder, screenRect);
var delta = previous == Duration.zero ? Duration.zero : now - previous;
previous = now;
var t = delta.inMicroseconds / Duration.microsecondsPerSecond;
world.update(t);
world.render(t, canvas);
var builder = new SceneBuilder()
..pushTransform(deviceTransform)
..addPicture(Offset.zero, recorder.endRecording())
..pop();
window.render(builder.build());
window.scheduleFrame();
};
window.scheduleFrame();
window.onPointerDataPacket = (packet) {
var p = packet.data.first;
world.input(p.physicalX, p.physicalY);
};
}
class World {
static var _objectColor = Paint()..color = Color(0xa0a0a0ff);
static var _s = 200.0;
static var _obejectRect = Rect.fromLTWH(-_s / 2, -_s / 2, _s, _s);
static var _rotationsPerSecond = 0.25;
var _turn = 0.0;
double _x;
double _y;
World(this._x, this._y);
void input(double x, double y) { _x = x; _y = y; }
void update(double t) { _turn += t * _rotationsPerSecond; }
void render(double t, Canvas canvas) {
var tau = math.pi * 2;
canvas.translate(_x, _y);
canvas.rotate(tau * _turn);
canvas.drawRect(_obejectRect, _objectColor);
}
}
Well, after a month of beating my face against this, I finally figured out the right question and that got me to this:
Flutter Layers / Raw
// Copyright 2015 The Chromium Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file.
// This example shows how to perform a simple animation using the raw interface
// to the engine.
import 'dart:math' as math;
import 'dart:typed_data';
import 'dart:ui' as ui;
void beginFrame(Duration timeStamp) {
// The timeStamp argument to beginFrame indicates the timing information we
// should use to clock our animations. It's important to use timeStamp rather
// than reading the system time because we want all the parts of the system to
// coordinate the timings of their animations. If each component read the
// system clock independently, the animations that we processed later would be
// slightly ahead of the animations we processed earlier.
// PAINT
final ui.Rect paintBounds = ui.Offset.zero & (ui.window.physicalSize / ui.window.devicePixelRatio);
final ui.PictureRecorder recorder = ui.PictureRecorder();
final ui.Canvas canvas = ui.Canvas(recorder, paintBounds);
canvas.translate(paintBounds.width / 2.0, paintBounds.height / 2.0);
// Here we determine the rotation according to the timeStamp given to us by
// the engine.
final double t = timeStamp.inMicroseconds / Duration.microsecondsPerMillisecond / 1800.0;
canvas.rotate(math.pi * (t % 1.0));
canvas.drawRect(ui.Rect.fromLTRB(-100.0, -100.0, 100.0, 100.0),
ui.Paint()..color = const ui.Color.fromARGB(255, 0, 255, 0));
final ui.Picture picture = recorder.endRecording();
// COMPOSITE
final double devicePixelRatio = ui.window.devicePixelRatio;
final Float64List deviceTransform = Float64List(16)
..[0] = devicePixelRatio
..[5] = devicePixelRatio
..[10] = 1.0
..[15] = 1.0;
final ui.SceneBuilder sceneBuilder = ui.SceneBuilder()
..pushTransform(deviceTransform)
..addPicture(ui.Offset.zero, picture)
..pop();
ui.window.render(sceneBuilder.build());
// After rendering the current frame of the animation, we ask the engine to
// schedule another frame. The engine will call beginFrame again when its time
// to produce the next frame.
ui.window.scheduleFrame();
}
void main() {
ui.window.onBeginFrame = beginFrame;
ui.window.scheduleFrame();
}

Screenshot using SharpDX and EasyHook transparent

I have been struggling in taking screenshots with DirectX, the problem is, it's working but seems to be missing some colors (black outline is missing for example), and some stuff that is also rendered by DX doesn't show.
I have uploaded the images (how the image should render and the rendered one) and also the code, what might be the issue?
Correct Image
Rendered Image
Greetings
public class Screenshot
{
public static void TakeScreenshot(string name = null)
{
var t = new Thread((ThreadStart) delegate
{
// destination folder
var destinationFolder = Environment.GetFolderPath(Environment.SpecialFolder.MyDocuments, Environment.SpecialFolderOption.Create) + "\\My Software\\Screenshots";
if (!Directory.Exists(destinationFolder)) Directory.CreateDirectory(destinationFolder);
// file name
string filename = name;
if (string.IsNullOrEmpty(name))
filename = "image-" + (Directory.GetFiles(destinationFolder, "*.png").Count() + 1).ToString("000") + ".png";
// # of graphics card adapter
const int numAdapter = 0;
// # of output device (i.e. monitor)
const int numOutput = 0;
// Create DXGI Factory1
var factory = new Factory1();
var adapter = factory.GetAdapter1(numAdapter);
// Create device from Adapter
var device = new Device(adapter);
// Get DXGI.Output
var output = adapter.GetOutput(numOutput);
var output1 = output.QueryInterface<Output1>();
// Width/Height of desktop to capture
int width = ((SharpDX.Rectangle)output.Description.DesktopBounds).Width;
int height = ((SharpDX.Rectangle)output.Description.DesktopBounds).Height;
// Create Staging texture CPU-accessible
var textureDesc = new Texture2DDescription
{
CpuAccessFlags = CpuAccessFlags.Read,
BindFlags = BindFlags.None,
Format = Format.B8G8R8A8_UNorm,
Width = width,
Height = height,
OptionFlags = ResourceOptionFlags.None,
MipLevels = 1,
ArraySize = 1,
SampleDescription = { Count = 1, Quality = 0 },
Usage = ResourceUsage.Staging
};
var screenTexture = new Texture2D(device, textureDesc);
// Duplicate the output
var duplicatedOutput = output1.DuplicateOutput(device);
bool captureDone = false;
for (int i = 0; !captureDone; i++)
{
try
{
SharpDX.DXGI.Resource screenResource;
OutputDuplicateFrameInformation duplicateFrameInformation;
// Try to get duplicated frame within given time
duplicatedOutput.AcquireNextFrame(10000, out duplicateFrameInformation, out screenResource);
if (i > 0)
{
// copy resource into memory that can be accessed by the CPU
using (var screenTexture2D = screenResource.QueryInterface<Texture2D>())
device.ImmediateContext.CopyResource(screenTexture2D, screenTexture);
// Get the desktop capture texture
var mapSource = device.ImmediateContext.MapSubresource(screenTexture, 0, MapMode.Read, MapFlags.None);
// Create Drawing.Bitmap
var bitmap = new Bitmap(width, height, PixelFormat.Format32bppArgb);
var boundsRect = new System.Drawing.Rectangle(0, 0, width, height);
// Copy pixels from screen capture Texture to GDI bitmap
var mapDest = bitmap.LockBits(boundsRect, ImageLockMode.WriteOnly, bitmap.PixelFormat);
var sourcePtr = mapSource.DataPointer;
var destPtr = mapDest.Scan0;
for (int y = 0; y < height; y++)
{
// Copy a single line
Utilities.CopyMemory(destPtr, sourcePtr, width * 4);
// Advance pointers
sourcePtr = IntPtr.Add(sourcePtr, mapSource.RowPitch);
destPtr = IntPtr.Add(destPtr, mapDest.Stride);
}
// Release source and dest locks
bitmap.UnlockBits(mapDest);
device.ImmediateContext.UnmapSubresource(screenTexture, 0);
// Save the output
bitmap.Save(destinationFolder + Path.DirectorySeparatorChar + filename);
// Send Message
Main.Chat.AddMessage(null, "~b~Screenshot saved as " + filename);
// Capture done
captureDone = true;
}
screenResource.Dispose();
duplicatedOutput.ReleaseFrame();
}
catch (SharpDXException e)
{
if (e.ResultCode.Code != SharpDX.DXGI.ResultCode.WaitTimeout.Result.Code)
{
throw e;
}
}
}
});
t.IsBackground = true;
t.Start();
}
}

How to acquire skeletal joint data to text file from two kinect cameras in SkeletalViewer? (C++)

I am currently working on a project to use multiple kinect cameras to acquire x,y,z coordinates of skeletal data in SkeletalViewer. I have an idea to use the KinectID or index of different kinect cameras to extract the sets of skeletal joints data into 2 different text files. But I am not sure if I am doing right. Please help to take a look at the modification below, or I will appreciate all your kind advice on other method to solve this problem.
In SkeletalViewer.h, I modified as following:
public:
FILE* mp3DFile0;
FILE* mp3DFile1;
char mText[1024];
INuiSensor * m_pNuiSensor;
BSTR m_instanceId;
array SensorIndex;
In NuiImpl.cpp, I modified as following:
1) define array
ref class MyClass {
public:
int m_i;
};
array<MyClass^>^ arrSensor() {
int i;
array< MyClass^ >^ local = gcnew array< MyClass^ >(2);
for (i = 0; i < 2; i++) {
local[i] = gcnew MyClass;
local[i]->m_i = i;
}
return local;
}
2) create array to store sensor index to do for loop later
HRESULT CSkeletalViewerApp::Nui_Init( )
{
HRESULT hr;
bool result;
//create an array to store two file pointers
FILE* mp3DFile[] = { mp3DFile0, mp3DFile1 };
fopen_s(mp3DFile0, "D:/Kinect/KinectCam0.txt", "w+");
fopen_s(mp3DFile1, "D:/Kinect/KinectCam1.txt", "w+");
.
.
if ( !m_pNuiSensor )
{
hr = NuiCreateSensorByIndex(0, &m_pNuiSensor);
//I am not sure about how to utilize this index in this case
.
.
}
if (NuiGetSensorCount(&m_pNuiSensor) > 1)
{
array< MyClass^ >^ SensorIndex;
SensorIndex = arrSensor();
}
}
3) use for loop to store data to different text file using index
void CSkeletalViewerApp::Nui_DrawSkeleton( const NUI_SKELETON_DATA & skel, int windowWidth, int windowHeight )
{
int i;
int h;
for (h = 0; h < 2; h++)
{
//when index point to the current kinect
if (SensorIndex[h] == &m_pNuiSensor)
{
for (i = 0; i < NUI_SKELETON_POSITION_COUNT; i++)
{
m_Points[i] = SkeletonToScreen(skel.SkeletonPositions[i], windowWidth, windowHeight);
memset(mText, 0, 1024);
sprintf(mText, "(%0.3f,%0.3f,%0.3f)", skel.SkeletonPositions[i].x, skel.SkeletonPositions[i].y, skel.SkeletonPositions[i].z);
if (mp3DFile[h]) {
fputs((const char*)mText, mp3DFile[h]);
}
}
if (mp3DFile[h]) {
fputs("\n", mp3DFile[h]);
}
}
}
.
.
}
I am a newbie in this Kinect programming. Thank you very much for your help! :)