Tensorflow Lite Model: Incompatible shapes for input and output array - tensorflow

I'm currently working on a Tensorflow Lite image classifier app that can recognice UNO cards. But when I'm running the float model in the class ImageClassifier, something is wrong.
The error is the next one:
java.lang.IllegalArgumentException: Cannot copy from a TensorFlowLite tensor (Identity) with shape [1, 10647, 4] to a Java object with shape [1, 15].
Here's the code that throw that error:
tflite.run(imgData, labelProbArray);
And this is how I have created imgData and labelProbArray:
private static final int DIM_BATCH_SIZE = 1;
private static final int DIM_PIXEL_SIZE = 3; //r+g+b = 1+1+1
static final int DIM_IMG_SIZE_X = 416;
static final int DIM_IMG_SIZE_Y = 416;
imgData = ByteBuffer.allocateDirect(DIM_BATCH_SIZE * DIM_IMG_SIZE_X * DIM_IMG_SIZE_Y * DIM_PIXEL_SIZE * 4); //The last value because size of float is 4
labelProbArray = new float[1][labelList.size()]; // {1, 15}
You can chech the inputs and ouputs of the .tflite file. Source.
I know you should create a buffer for the output values, but I tried to import this and didn't work:
import org.tensorflow.lite.support.tensorbuffer.TensorBuffer;
Any ideas? Thank you so much for read me^^
Edit v2:
Thanks to yyoon I realise that I didn't populate my model with metadata, so I run this line in my cmd:
python ./metadata_writer_for_image_classifier_uno.py \ --model_file=./model_without_metadata/custom.tflite \ --label_file=./model_without_metadata/labels.txt \ --export_directory=model_with_metadata
Before that, I modified this file with my data:
_MODEL_INFO = {
"custom.tflite":
ModelSpecificInfo(
name="UNO image classifier",
version="v1",
image_width=416,
image_height=416,
image_min=0,
image_max=255,
mean=[127.5],
std=[127.5],
num_classes=15)
}
And another error appeared:
ValueError: The number of output tensors (2) should match the number of output tensor metadata (1)
Idk why my model have 2 tensors outputs...

Related

FATAL EXCEPTION: inference. Cannot copy to a TensorFlowLite tensor from a Java Buffer

I was developing an Image Classifier app in Android Studio with MNIST-fashion database, but I have a little problem. When I try to run the app, I have this common error:
java.lang.IllegalArgumentException: Cannot copy to a TensorFlowLite tensor (serving_default_conv2d_input:0) with 3136 bytes from a Java Buffer with 9408 bytes.
I know this might be the mismatch of input tensor from the model and the buffer that I have in my code. But It's too confusing because my code automatically fits the size of the image from the model and all the info it needs. So I was wondering where is the error...
// Reads type and shape of input and output tensors, respectively.
int imageTensorIndex = 0;
int[] imageShape = tflite.getInputTensor(imageTensorIndex).shape(); // {1, height, width, 1}
imageSizeY = imageShape[1];
imageSizeX = imageShape[2];
DataType imageDataType = tflite.getInputTensor(imageTensorIndex).dataType();
int probabilityTensorIndex = 0;
int[] probabilityShape =
tflite.getOutputTensor(probabilityTensorIndex).shape(); // {1, 10}
DataType probabilityDataType = tflite.getOutputTensor(probabilityTensorIndex).dataType();
// Creates the input tensor.
inputImageBuffer = new TensorImage(imageDataType);
Maybe this is the problem... I'm creating imageShape like this {1, height, width, 1}, and the data type is FLOAT32, so it is supposed to be {1, height, width, 4}, right?
Another reason could be the metadata. But I populate the model with metadata and I have a .json and I don't know how to use it.
Note: If u want the note book to do the .tflite, there u go.
The tensor buffer size is determined by datasize (float32: 4bytes) * flat size of the tensor shape (1 * height * width * 1).
So the above code snippet needs to prepare an float input tensor data with the shape (1, height, width, 1) instead of the shape (1, height, width, 4).

How to read parameters of layers of .tflite model in python

I was trying to read tflite model and pull all the parameters of the layers out.
My steps:
I generated flatbuffers model representation by running (please build flatc before):
flatc -python tensorflow/tensorflow/lite/schema/schema.fbs
Result is tflite/ folder that contains layer description files (*.py) and some utilitarian files.
I successfully loaded model:
in case of import Error: set PYTHONPATH to point to the folder where tflite/ is
from tflite.Model import Model
def read_tflite_model(file):
buf = open(file, "rb").read()
buf = bytearray(buf)
model = Model.GetRootAsModel(buf, 0)
return model
I partly pulled model and node parameters out and stacked in iterating over nodes:
Model part:
def print_model_info(model):
version = model.Version()
print("Model version:", version)
description = model.Description().decode('utf-8')
print("Description:", description)
subgraph_len = model.SubgraphsLength()
print("Subgraph length:", subgraph_len)
Nodes part:
def print_nodes_info(model):
# what does this 0 mean? should it always be zero?
subgraph = model.Subgraphs(0)
operators_len = subgraph.OperatorsLength()
print('Operators length:', operators_len)
from collections import deque
nodes = deque(subgraph.InputsAsNumpy())
STEP_N = 0
MAX_STEPS = operators_len
print("Nodes info:")
while len(nodes) != 0 and STEP_N <= MAX_STEPS:
print("MAX_STEPS={} STEP_N={}".format(MAX_STEPS, STEP_N))
print("-" * 60)
node_id = nodes.pop()
print("Node id:", node_id)
tensor = subgraph.Tensors(node_id)
print("Node name:", tensor.Name().decode('utf-8'))
print("Node shape:", tensor.ShapeAsNumpy())
# which type is it? what does it mean?
type_of_tensor = tensor.Type()
print("Tensor type:", type_of_tensor)
quantization = tensor.Quantization()
min = quantization.MinAsNumpy()
max = quantization.MaxAsNumpy()
scale = quantization.ScaleAsNumpy()
zero_point = quantization.ZeroPointAsNumpy()
print("Quantization: ({}, {}), s={}, z={}".format(min, max, scale, zero_point))
# I do not understand it again. what is j, that I set to 0 here?
operator = subgraph.Operators(0)
for i in operator.OutputsAsNumpy():
nodes.appendleft(i)
STEP_N += 1
print("-"*60)
Please point me to documentation or some example of using this API.
My problems are:
I can not get documentation on this API
Iterating over Tensor objects seems not possible for me, as it doesn't have Inputs and Outputs methods. + subgraph.Operators(j=0) I do not understand what j means in here. Because of that my cycle goes through two nodes: input (once) and the next one over and over again.
Iterating over Operator objects is surely possible:
Here we iterate over them all but I can not get how to map Operator and Tensor.
def print_in_out_info_of_all_operators(model):
# what does this 0 mean? should it always be zero?
subgraph = model.Subgraphs(0)
for i in range(subgraph.OperatorsLength()):
operator = subgraph.Operators(i)
print('Outputs', operator.OutputsAsNumpy())
print('Inputs', operator.InputsAsNumpy())
I do not understand how to pull parameters out Operator object. BuiltinOptions method gives me Table object, that I do not know what to map at.
subgraph = model.Subgraphs(0)
What does this 0 mean? should it always be zero? obviously no, but what is it? Id of the subgraph? If so - I'm happy. If no, please try to explain it.

TensorFlow Model is still floating point after Post-training quantization

After applying post-training quantization, my custom CNN model was shrinked to 1/4 of its original size (from 56.1MB to 14MB). I put the image(100x100x3) that is to be predicted into ByteBuffer as 100x100x3=30,000 bytes. However, I got the following error during inference:
java.lang.IllegalArgumentException: Cannot convert between a TensorFlowLite buffer with 120000 bytes and a ByteBuffer with 30000 bytes.**
at org.tensorflow.lite.Tensor.throwExceptionIfTypeIsIncompatible(Tensor.java:221)
at org.tensorflow.lite.Tensor.setTo(Tensor.java:93)
at org.tensorflow.lite.NativeInterpreterWrapper.run(NativeInterpreterWrapper.java:136)
at org.tensorflow.lite.Interpreter.runForMultipleInputsOutputs(Interpreter.java:216)
at org.tensorflow.lite.Interpreter.run(Interpreter.java:195)
at gov.nih.nlm.malaria_screener.imageProcessing.TFClassifier_Lite.recongnize(TFClassifier_Lite.java:102)
at gov.nih.nlm.malaria_screener.imageProcessing.TFClassifier_Lite.process_by_batch(TFClassifier_Lite.java:145)
at gov.nih.nlm.malaria_screener.Cells.runCells(Cells.java:269)
at gov.nih.nlm.malaria_screener.CameraActivity.ProcessThinSmearImage(CameraActivity.java:1020)
at gov.nih.nlm.malaria_screener.CameraActivity.access$600(CameraActivity.java:75)
at gov.nih.nlm.malaria_screener.CameraActivity$8.run(CameraActivity.java:810)
at java.lang.Thread.run(Thread.java:762)
The imput image size to the model is: 100x100x3. I'm currently predicting one image at a time. So, if I'm making the Bytebuffer: 100x100x3 = 30,000 bytes. However, the log info above says the TensorFlowLite buffer has 120,000 bytes. This makes me suspect that the converted tflite model is still in float format. Is this expected behavior? How can I get a quantized model that take input image in 8 pit precision like it does in the example from TensorFlow official repository ?
In the example code, the ByteBuffer used as input for tflite.run() is in 8 bit precision for the quantized model.
But I also read from the google doc saying, "At inference, weights are converted from 8-bits of precision to floating-point and computed using floating point kernels." This two instances seems to contradict each other.
private static final int BATCH_SIZE = 1;
private static final int DIM_IMG_SIZE = 100;
private static final int DIM_PIXEL_SIZE = 3;
private static final int BYTE_NUM = 1;
imgData = ByteBuffer.allocateDirect(BYTE_NUM * BATCH_SIZE * DIM_IMG_SIZE * DIM_IMG_SIZE * DIM_PIXEL_SIZE);
imgData.order(ByteOrder.nativeOrder());
... ...
int pixel = 0;
for (int i = 0; i < DIM_IMG_SIZE; ++i) {
for (int j = 0; j < DIM_IMG_SIZE; ++j) {
final int val = intValues[pixel++];
imgData.put((byte)((val >> 16) & 0xFF));
imgData.put((byte)((val >> 8) & 0xFF));
imgData.put((byte)(val & 0xFF));
// imgData.putFloat(((val >> 16) & 0xFF) / 255.0f);
// imgData.putFloat(((val >> 8) & 0xFF) / 255.0f);
// imgData.putFloat((val & 0xFF) / 255.0f);
}
}
... ...
tfLite.run(imgData, labelProb);
Post-training quantization code:
import tensorflow as tf
import sys
import os
saved_model_dir = '/home/yuh5/Downloads/malaria_thinsmear.h5.pb'
input_arrays = ["input_2"]
output_arrays = ["output_node0"]
converter = tf.contrib.lite.TocoConverter.from_frozen_graph(saved_model_dir, input_arrays, output_arrays)
converter.post_training_quantize = True
tflite_model = converter.convert()
open("thinSmear_100.tflite", "wb").write(tflite_model)
Post-training quantization does not change the format of the input or output layers. You can run your model with data in the same format as used for training.
You may look into quantization-aware training to generate fully-quantized models, but I have no experience with it.
As for the sentence "At inference, weights are converted from 8-bits of precision to floating-point and computed using floating point kernels." This means that the weights are "de-quantized" to floating point values in memory, and computed with FP instructions, instead of performing integer operations.

Tensorflow C API placeholder / input variable setting

I am trying to use the TensorFlow C-API to run a implementation of LeNet that has been saved from a Keras/TF model, but I am having consistent problems with setting the input. Relevant piece of code is:
// Load the image with openCV
CvMat * img = cvLoadImageM(argv[1], CV_LOAD_IMAGE_COLOR );
// Create an Tensor from the image
int64_t dims4[]={1,1,28,28};
TF_Tensor * imgTensor = TF_NewTensor(TF_FLOAT,dims4,4,img,28*28*sizeof(float),NULL,NULL);
TF_Operation* init_op2 = TF_GraphOperationByName(graph, "conv2d_1_input");
TF_Operation* targets[] = {init_op2};
// Build up the inputs
TF_Output inp = {
init_op2,
0
};
TF_Output * inputs[] = {&inp};
TF_Tensor * input_values[] = {imgTensor};
printf("\nBefore\n");
TF_SessionRun(session, NULL,
&inp, inputvalues, 1, // inputs
NULL, NULL, 0, // outputs
&init_op2, 1, // targets
NULL,
status);
printf("After\n");
printf("Status %d %s\n", TF_GetCode(status), TF_Message(status));
However, anyway I try to build up the input tensor, I get the error status and message:
Status 3 You must feed a value for placeholder tensor 'conv2d_1_input' with dtype float and shape [?,1,28,28]
[[Node: conv2d_1_input = Placeholder[_output_shapes=[[?,1,28,28]], dtype=DT_FLOAT, shape=[?,1,28,28], _device=...]()]]
Any suggestions on what I am doing wrong?
In your call to TF_SessionRun, you're also providing the conv2d_1_input operation as a "target". The error message can be improved, but it's basically complaining that you're asking the session to execute a placeholder operation, which it can't - which isn't possible (see the note in the documentation for tf.placeholder)
Shouldn't you be asking for a different target or output tensor from the call to TF_SessionRun with something like:
TF_Output out = { TF_GraphOperationByName(graph, "<name_of_output_tensor>"), 0 };
TF_Tensor* outputvalues = NULL;
TF_SessionRun(session, NULL,
&inp, inputvalues, 1, // inputs
&out, &outputvalues, 1, // outputs
NULL, 0, // targets
NULL, status);
Hope that helps.

Tensorflow: how to add user custom op accepting two 1D vec tensor and output a scalar?

I'm trying below but not work.
#include "tensorflow/core/framework/op.h"
#include "tensorflow/core/framework/op_kernel.h"
using namespace tensorflow;
REGISTER_OP("Auc")
.Input("predicts: T1")
.Input("labels: T2")
.Output("z: double")
.Attr("T1: {float, double}")
.Attr("T2: {int32, int64}")
.SetIsCommutative()
.Doc(R"doc(
Given preidicts and labels output it's auc
)doc");
class AucOp : public OpKernel {
public:
explicit AucOp(OpKernelConstruction* context) : OpKernel(context) {}
void Compute(OpKernelContext* context) override {
// Grab the input tensor
const Tensor& predicts_tensor = context->input(0);
const Tensor& labels_tensor = context->input(1);
auto predicts = predicts_tensor.flat<double>();
auto labels = labels_tensor.flat<int32>();
// Create an output tensor
Tensor* output_tensor = NULL;
TensorShape output_shape;
OP_REQUIRES_OK(context, context->allocate_output(0, output_shape, &output_tensor));
output_tensor->flat<double>().setConstant(predicts(0) * labels(0));
}
};
REGISTER_KERNEL_BUILDER(Name("Auc").Device(DEVICE_CPU), AucOp);
test.py
predicts = tf.constant([0.8, 0.5, 0.12])
labels = tf.constant([-1, 1, 1])
output = tf.user_ops.auc(predicts, labels)
with tf.Session() as sess:
init = tf.initialize_all_variables()
sess.run(init)
print output.eval()
./test.py
I tensorflow/core/common_runtime/local_device.cc:40] Local device intra op parallelism threads: 8
I tensorflow/core/common_runtime/direct_session.cc:60] Direct session inter op parallelism threads: 8
F ./tensorflow/core/public/tensor.h:453] Check failed: dtype() == DataTypeToEnum::v() (1 vs. 2)
Aborted
The issue is that the predicts tensor in your Python program has type float, and your op registration accepts this as a valid type for the predicts input (since T1 can be float or double), but AucOp::Compute() assumes that the predicts input always has type double (in the call to predicts_tensor.flat<double>()). The tensorflow::Tensor class does not convert the type of elements in the tensor when you ask for values of a different type, and instead it raises a fatal error.
There are several possible solutions:
To get things working quickly, you could change the type of predicts in your Python program to tf.float64 (which is a synonym for double in the Python front-end):
predicts = tf.constant([0.8, 0.5, 0.12], dtype=tf.float64)
You could start by defining a simpler op that accepts inputs of a single type only:
REGISTER_OP("Auc")
.Input("predicts: double")
.Input("labels: int32")
...;
You could add code in the AucOp::Compute() method to test the input type and access the input values as appropriate. (Use this->input_type(i) to find the type of the ith input.
You could define a templated class AucOp<TPredict, TLabel>, then use TypeConstraint<> in the REGISTER_KERNEL_BUILDER call to define specializations for each of the four valid combinations of prediction and label types. This would look something like:
REGISTER_KERNEL_BUILDER(Name("Auc")
.Device(DEVICE_CPU)
.TypeConstraint<float>("T1")
.TypeConstraint<int32>("T2"),
AucOp<float, int32>);
// etc. for AucOp<double, int32>, AucOp<float, int64>, and AucOp<double, int64>.