I'm trying to run a super resolution model on Coral TPU. After integer only quantization, noticed a strong color degradation or gray overcast over SR images, even though they do look sharper than bicubic upscaled ones. Can anyone give me some tips on how to improve it? Code example pasted below. All images are from a super resolution training dataset. An example is linked here.
Top: SR image, Bot: Bicubic upscaled image
import tensorflow as tf
import tensorflow_hub as hub
import matplotlib.pyplot as plt
print(tf.__version__)
# set up representative dataset for quantization
rep_path = r'C:\Users\xxx\Downloads\BSR_bsds500\BSR\BSDS500\data\images'
rep_ds = tf.keras.utils.image_dataset_from_directory(
rep_path,
seed=123,
image_size=(256, 256),
batch_size=1)
def representative_data_gen():
for image_batch, _ in rep_ds:
yield [image_batch]
# load tf SR model and convert to tf lite with quantization.
model = hub.load("https://tfhub.dev/captain-pool/esrgan-tf2/1")
concrete_func = model.signatures[tf.saved_model.DEFAULT_SERVING_SIGNATURE_DEF_KEY]
#tf.function(input_signature=[tf.TensorSpec(shape=[1, 256, 256, 3], dtype=tf.float32)])
def f(input):
return concrete_func(input);
converter = tf.lite.TFLiteConverter.from_concrete_functions([f.get_concrete_function()], model)
converter.optimizations = [tf.lite.Optimize.DEFAULT]
converter.representative_dataset = representative_data_gen
converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8]
converter.inference_input_type = tf.uint8
converter.inference_output_type = tf.uint8
tflite_model_quant = converter.convert()
with tf.io.gfile.GFile('ESRGAN_quant.tflite', 'wb') as f:
f.write(tflite_model_quant)
# run inference for a test image
test_img_path = r'C:\Users\xxx\Downloads\24004.jpg'
lr = tf.io.read_file(test_img_path)
lr = tf.image.decode_jpeg(lr)
lr = tf.expand_dims(lr, axis=0)
lr = tf.cast(lr, tf.uint8)
# Load TFLite model and allocate tensors.
interpreter = tf.lite.Interpreter(model_path='.\ESRGAN_quant.tflite')
interpreter.allocate_tensors()
input_details = interpreter.get_input_details()
output_details = interpreter.get_output_details()
interpreter.set_tensor(input_details[0]['index'], lr)
interpreter.invoke()
output_data = interpreter.get_tensor(output_details[0]['index'])
sr = tf.squeeze(output_data, axis=0)
sr = tf.clip_by_value(sr, 0, 255)
sr = tf.round(sr)
sr = tf.cast(sr, tf.uint8)
plt.figure(figsize=(30, 30))
plt.subplot(2, 1, 1)
plt.title(f'ESRGAN (x4)')
plt.imshow(sr.numpy());
bicubic = tf.image.resize(lr, [1024, 1024], tf.image.ResizeMethod.BICUBIC)
bicubic = tf.cast(bicubic, tf.uint8)
plt.subplot(2, 1, 2)
plt.title('Bicubic')
plt.imshow(bicubic.numpy()[0]);
I have a saved tensorflow model the same as all models in the model zoo.
I want to convert it to tesorflow lite, I find the following way from tensorflow github (my tensorflw version is 2):
!wget http://download.tensorflow.org/models/object_detection/tf2/20200711/ssd_resnet50_v1_fpn_640x640_coco17_tpu-8.tar.gz
# extract the downloaded file
!tar -xzvf ssd_resnet50_v1_fpn_640x640_coco17_tpu-8.tar.gz
!pip install tf-nightly
import tensorflow as tf
converter = tf.lite.TFLiteConverter.from_saved_model('ssd_mobilenet_v2_320x320_coco17_tpu-8/saved_model')
converter.optimizations = [tf.lite.Optimize.DEFAULT]
converter.experimental_new_converter = True
converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS, tf.lite.OpsSet.SELECT_TF_OPS]
tflite_model = converter.convert()
open("m.tflite", "wb").write(tflite_model)
But the output and input shape of the converted model don't match the original model, check the following:
Original Model Input & Output shape
Converted Model Input & Output shape
So there is a problem here! the input / output shape should be matched the original model!
Any idea?
From Tensorflow github issues, I used their answer to solve my problem.
Link
Their approach:
!pip install tf-nightly
import tensorflow as tf
## TFLite Conversion
model = tf.saved_model.load("saved_model")
concrete_func = model.signatures[tf.saved_model.DEFAULT_SERVING_SIGNATURE_DEF_KEY]
concrete_func.inputs[0].set_shape([1, 300, 300, 3])
tf.saved_model.save(model, "saved_model_updated", signatures={"serving_default":concrete_func})
converter = tf.lite.TFLiteConverter.from_saved_model(saved_model_dir='saved_model_updated', signature_keys=['serving_default'])
converter.optimizations = [tf.lite.Optimize.DEFAULT]
converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS, tf.lite.OpsSet.SELECT_TF_OPS]
tflite_model = converter.convert()
## TFLite Interpreter to check input shape
interpreter = tf.lite.Interpreter(model_content=tflite_model)
interpreter.allocate_tensors()
# Get input and output tensors.
input_details = interpreter.get_input_details()
output_details = interpreter.get_output_details()
# Test the model on random input data.
input_shape = input_details[0]['shape']
print(input_shape)
[ 1 300 300 3]
Thank you MeghnaNatraj
The shape of both models input and output should be the same as shown below
If the model is already in saved_model format, you the code below
# if you are using same model
export_dir = 'ssd_mobilenet_v2_320x320_coco17_tpu-8/saved_model'
converter = tf.lite.TFLiteConverter.from_saved_model(export_dir)
if your model is in Keras format, use the format below
# if it's a keras model
model = tf.keras.applications.MobileNetV2(weights="imagenet", input_shape= (224, 224, 3))
converter = tf.lite.TFLiteConverter.from_keras_model(model)
In both cases, the intention is to get the converter.
I don't have the saved_model, so I will use keras model and convert it to saved_model format just use the Keras model format as an example
import pathlib #to use path
model = tf.keras.applications.MobileNetV2(weights="imagenet", input_shape= (224, 224, 3))
export_dir = 'imagenet/saved_model'
tf.saved_model.save(model, export_dir) #convert keras to saved model
converter = tf.lite.TFLiteConverter.from_saved_model(export_dir)
converter.optimizations = [tf.lite.Optimize.DEFAULT] #you can also optimize for size or latency OPTIMIZE_FOR_SIZE, OPTIMIZE_FOR_LATENCY
tflite_model = converter.convert()
#save the model
tflite_model_file = pathlib.Path('m.tflite')
tflite_model_file.write_bytes(tflite_model)
tflite_interpreter = tf.lite.Interpreter(model_path= 'm.tflite') #you can load the content with model_content=tflite_model
# get shape of tflite input and output
input_details = tflite_interpreter.get_input_details()
output_details = tflite_interpreter.get_output_details()
print("Input: {}".format( input_details[0]['shape']))
print("Output:{}".format(output_details[0]['shape']))
# get shape of the origin model
print("Input: {}".format( model.input.shape))
print("Output: {}".format(model.output.shape))
For the tflite: I have this
For the Original Model I have this
You will see the shape of both tflite and keras model are the same
Just reshape your input tensor.
You can use the resize_tensor_input function, like this:
interpreter.resize_tensor_input(input_index=0, tensor_size=[1, 640, 640, 3])
Now you input shape will be: [1, 640, 640, 3].
I'm trying to convert from pytorch to tflite for android inference for a app I'm working on that uses real-time camera data of basketball to create a heatmap of made and missed shots. It's already working for iOS. Here's a demo.
I've managed to convert from pytorch (.pth) to onnx and from onnx to a tensorflow frozen graph (.pb). Inference on that tf frozen graph checks out.
However, when I try to convert from the frozen graph to tflite, I get the following error:
RuntimeError: Inputs and outputs not all float|uint8|int16 types.Node number 2 (ADD) failed to invoke.
Input details from the interpreter [interpreter.get_input_details(), interpreter.get_output_details()] suggest the datatype is numpy.float32, which is where I'm confused. Shouldn't that qualify as float? Any suggestions/help will be much appreciated!
[{'name': 'image', 'index': 21904, 'shape': array([ 3, width, height], dtype=int32), 'dtype': <class 'numpy.float32'>, 'quantization': (0.0, 0)}]
[{'name': 'action', 'index': 7204, 'shape': array([], dtype=int32), 'dtype': <class 'numpy.float32'>, 'quantization': (0.0, 0)}]
From tensorflow's documentation:
import numpy as np
import tensorflow as tf
# Load TFLite model and allocate tensors.
interpreter = tf.lite.Interpreter(model_path="converted_model.tflite")
interpreter.allocate_tensors()
# Get input and output tensors.
input_details = interpreter.get_input_details()
output_details = interpreter.get_output_details()
# Test model on random input data.
input_shape = input_details[0]['shape']
input_data = np.array(np.random.random_sample(input_shape), dtype=np.float32)
interpreter.set_tensor(input_details[0]['index'], input_data)
interpreter.invoke()
# The function `get_tensor()` returns a copy of the tensor data.
# Use `tensor()` in order to get a pointer to the tensor.
output_data = interpreter.get_tensor(output_details[0]['index'])
print(output_data)
The inference file for tensorflow frozen graph (1.14.0) for anyone in need:
import numpy as np
import tensorflow as tf
from PIL import Image
w = ...
h = ...
class CNN(object):
def __init__(self, model_filepath):
self.model_filepath = model_filepath
self.load_graph(model_filepath = self.model_filepath)
def load_graph(self, model_filepath):
self.graph = tf.Graph()
with tf.gfile.GFile(model_filepath, 'rb') as f:
graph_def = tf.GraphDef()
graph_def.ParseFromString(f.read())
with self.graph.as_default():
self.input = tf.placeholder(np.float32, shape = [3, h, w], name='image')
tf.import_graph_def(graph_def, {'image': self.input})
self.graph.finalize()
self.sess = tf.Session(graph = self.graph)
def test(self, data):
output_tensor = self.graph.get_tensor_by_name('import/action:0')
output = self.sess.run(output_tensor, feed_dict = {self.input: data})
return output
def main():
nn = CNN(model_filepath='out_1.14.pb')
img = np.asarray(Image.open('example.jpg')).astype(np.float32)
img = img.transpose(-1, 0, 1)
ans = nn.test(data=img)
print(ans)
if __name__ == '__main__':
main()
I'm a newbie of Tensorflow. I have created CNNs of Tensorflow followingthis topic : A Guide to TF Layers: Building a Convolutional Neural Network
I want to create CNNs to using it for training traffic sign dataset. The dataset I use is : BelgiumTS. It includes two part, one part stores images for training, second parth stores images for testing. All of this is .ppm format.
I define a method to load the dataset :
def load_data(data_dir):
"""Load Data and return two numpy array"""
directories = [d for d in os.listdir(data_dir) if os.path.isdir(os.path.join(data_dir,d))]
list_labels = []
list_images = []
for d in directories:
label_dir = os.path.join(data_dir,d)
file_names = [os.path.join(label_dir,f) for f in os.listdir(label_dir) if f.endswith(".ppm")]
for f in file_names:
list_images.append(skimage.data.imread(f))
list_labels.append(int(d))
#resize images to 32x32 pixel
list_images32 = [skimage.transform.resize(image,(32,32)) for image in list_images]
#Got Error "Value passed to parameter 'input' has DataType float64 not in list of allowed values: float16, float32" if I don't add this line
list_images32 = tf.cast(list_images32,tf.float32)
images = np.array(list_images32)
labels = np.asarray(list_labels,dtype=int32)
return images,labels
And this is CNNs define :
def cnn_model_fn(features, labels, mode):
#Input layer
input_layer = tf.reshape(features["x"],[-1,32,32,1])
#Convolutional layer 1
conv1 = tf.layers.conv2d(
inputs=input_layer,
filters=32,
kernel_size=[5,5],
padding="same",
activation=tf.nn.relu)
#Pooling layer 1
pool1 = tf.layers.max_pooling2d(inputs=conv1,pool_size=[2,2],strides=2)
#Convolutional layer 2
conv2 = tf.layers.conv2d(
inputs=pool1,
filters=64,
kernel_size=[5,5],
padding="same",
activation=tf.nn.relu)
#Pooling layer 2
pool2 = tf.layers.max_pooling2d(inputs=conv2,pool_size=[2,2],strides=2)
#Dense layer
pool2_flat = tf.reshape(pool2,[-1,7*7*64])
dense = tf.layers.dense(inputs=pool2_flat,units=1024,activation=tf.nn.relu)
#Dropout
dropout = tf.layers.dropout(inputs=dense,rate=0.4,training=mode == tf.estimator.ModeKeys.TRAIN)
#Logits layer
logits = tf.layers.dense(inputs=dropout,units=10)
predictions = {
"classes": tf.argmax(input=logits,axis=1),
"probabilities": tf.nn.softmax(logits,name="softmax_tensor")
}
if mode == tf.estimator.ModeKeys.PREDICT:
return tf.estimator.EstimatorSpec(mode=mode,predictions=predictions)
#Calculate Loss Value
onehot_labels = tf.one_hot(indices=tf.cast(labels,tf.int32),depth=10)
loss = tf.losses.softmax_cross_entropy(onehot_labels=onehot_labels,logits=logits)
if mode == tf.estimator.ModeKeys.TRAIN:
optimizer = tf.train.GradientDescentOptimizer(learning_rate=0.001)
train_op = optimizer.minimize(
loss = loss,
global_step = tf.train.get_global_step())
return tf.estimator.EstimatorSpec(mode=mode,loss=loss,train_op=train_op)
eval_metric_ops = {
"accuracy": tf.metrics.accuracy(
labels=labels,predictions=predictions["classes"])}
return tf.estimator.EstimatorSpec(mode=mode,loss=loss,eval_metric_ops=eval_metric_ops)
I run my app in main :
def main(unused_argv):
# Load training and eval data
train_data_dir = "W:/Projects/AutoDrive/Training"
test_data_dir = "W:/Projects/AutoDrive/Testing"
images,labels = load_data(train_data_dir)
test_images,test_labels = load_data(test_data_dir)
# Create the Estimator
autoDrive_classifier = tf.estimator.Estimator(
model_fn=cnn_model_fn, model_dir="/tmp/autoDrive_convnet_model")
# Set up logging for predictions
# Log the values in the "Softmax" tensor with label "probabilities"
tensors_to_log = {"probabilities": "softmax_tensor"}
logging_hook = tf.train.LoggingTensorHook(
tensors=tensors_to_log, every_n_iter=50)
# Train the model
train_input_fn = tf.estimator.inputs.numpy_input_fn(
x={"x": images},
y=labels,
batch_size=100,
num_epochs=None,
shuffle=True)
autoDrive_classifier.train(
input_fn=train_input_fn,
steps=10000,
hooks=[logging_hook])
# Evaluate the model and print results
eval_input_fn = tf.estimator.inputs.numpy_input_fn(
x={"x": test_images},
y=test_labels,
num_epochs=1,
shuffle=False)
eval_results = autoDrive_classifier.evaluate(input_fn=eval_input_fn)
print(eval_results)
But when I run it, I got this error : ValueError: Argument must be a dense tensor ... got shape [4575, 32, 32, 3], but wanted [4575] Did I lost something ?
Finally, this is full code :
from __future__ import absolute_import
from __future__ import division
from __future__ import print_function
import numpy as np
import tensorflow as tf
import os
import skimage.data
import skimage.transform
import matplotlib
import matplotlib.pyplot as plt
tf.logging.set_verbosity(tf.logging.INFO)
def load_data(data_dir):
"""Load Data and return two lists"""
directories = [d for d in os.listdir(data_dir) if
os.path.isdir(os.path.join(data_dir,d))]
list_labels = []
list_images = []
for d in directories:
label_dir = os.path.join(data_dir,d)
file_names = [os.path.join(label_dir,f) for f in os.listdir(label_dir) if f.endswith(".ppm")]
for f in file_names:
list_images.append(skimage.data.imread(f))
list_labels.append(int(d))
list_images32 = [skimage.transform.resize(image,(32,32)) for image in list_images]
list_images32 = tf.cast(list_images32,tf.float32)
images = np.array(list_images32)
labels = np.asarray(list_labels,dtype=int32)
return images,labels
def cnn_model_fn(features, labels, mode):
#Input layer
input_layer = tf.reshape(features["x"],[-1,32,32,1])
#Convolutional layer 1
conv1 = tf.layers.conv2d(
inputs=input_layer,
filters=32,
kernel_size=[5,5],
padding="same",
activation=tf.nn.relu)
#Pooling layer 1
pool1 = tf.layers.max_pooling2d(inputs=conv1,pool_size=[2,2],strides=2)
#Convolutional layer 2
conv2 = tf.layers.conv2d(
inputs=pool1,
filters=64,
kernel_size=[5,5],
padding="same",
activation=tf.nn.relu)
#Pooling layer 2
pool2 = tf.layers.max_pooling2d(inputs=conv2,pool_size=[2,2],strides=2)
#Dense layer
pool2_flat = tf.reshape(pool2,[-1,7*7*64])
dense = tf.layers.dense(inputs=pool2_flat,units=1024,activation=tf.nn.relu)
#Dropout
dropout = tf.layers.dropout(inputs=dense,rate=0.4,training=mode == tf.estimator.ModeKeys.TRAIN)
#Logits layer
logits = tf.layers.dense(inputs=dropout,units=10)
predictions = {
"classes": tf.argmax(input=logits,axis=1),
"probabilities": tf.nn.softmax(logits,name="softmax_tensor")
}
if mode == tf.estimator.ModeKeys.PREDICT:
return tf.estimator.EstimatorSpec(mode=mode,predictions=predictions)
#Calculate Loss Value
onehot_labels = tf.one_hot(indices=tf.cast(labels,tf.int32),depth=10)
loss = tf.losses.softmax_cross_entropy(onehot_labels=onehot_labels,logits=logits)
if mode == tf.estimator.ModeKeys.TRAIN:
optimizer = tf.train.GradientDescentOptimizer(learning_rate=0.001)
train_op = optimizer.minimize(
loss = loss,
global_step = tf.train.get_global_step())
return tf.estimator.EstimatorSpec(mode=mode,loss=loss,train_op=train_op)
eval_metric_ops = {
"accuracy": tf.metrics.accuracy(
labels=labels,predictions=predictions["classes"])}
return tf.estimator.EstimatorSpec(mode=mode,loss=loss,eval_metric_ops=eval_metric_ops)
def main(unused_argv):
# Load training and eval data
train_data_dir = "W:/Projects/TSRecognition/Training"
test_data_dir = "W:/Projects/TSRecognition/Testing"
images,labels = load_data(train_data_dir)
test_images,test_labels = load_data(test_data_dir)
# Create the Estimator
TSRecognition_classifier = tf.estimator.Estimator(
model_fn=cnn_model_fn, model_dir="/tmp/TSRecognition_convnet_model")
# Set up logging for predictions
# Log the values in the "Softmax" tensor with label "probabilities"
tensors_to_log = {"probabilities": "softmax_tensor"}
logging_hook = tf.train.LoggingTensorHook(
tensors=tensors_to_log, every_n_iter=50)
# Train the model
train_input_fn = tf.estimator.inputs.numpy_input_fn(
x={"x": images},
y=labels,
batch_size=100,
num_epochs=None,
shuffle=True)
TSRecognition_classifier.train(
input_fn=train_input_fn,
steps=10000,
hooks=[logging_hook])
# Evaluate the model and print results
eval_input_fn = tf.estimator.inputs.numpy_input_fn(
x={"x": test_images},
y=test_labels,
num_epochs=1,
shuffle=False)
eval_results = TSRecognition_classifier.evaluate(input_fn=eval_input_fn)
print(eval_results)
if __name__ == "__main__":
tf.app.run()
Short answer for your code:
Get rid of the np.array and np.asarray calls in your load_data function. In particular, change:
list_images32 = [skimage.transform.resize(image,(32,32)) for image in list_images]
...to...
list_images32 = [skimage.transform.resize(image,(32,32)).astype(np.float32).tolist() for image in list_images]
...and return list_images32 AS IS from your load_data function. Don't "wrap it" with the np.asarray() call. The tolist() part of my suggestion is what is important. With the astype() call I'm just suggesting doing in numpy something you're doing in TensorFlow.
Simply getting rid of the np.asarray you have on list_labels should suffice for your labels.
The full answer for those that want to understand what's going on...
The "got shape...but wanted" exception is thrown from exactly one place in TensorFlow (tensor_util.py) and the reason is this function:
def _GetDenseDimensions(list_of_lists):
"""Returns the inferred dense dimensions of a list of lists."""
if not isinstance(list_of_lists, (list, tuple)):
return []
elif not list_of_lists:
return [0]
else:
return [len(list_of_lists)] + _GetDenseDimensions(list_of_lists[0])
It is trying to traverse what it assumes are nested plain Python lists or plain Python tuples; it doesn't know what to do with the Numpy array type it finds in your data structure because of the np.array/np.asarray calls.
I want to fine-tune the ResNet50 from Keras but first I found that given the same input, the prediction from ResNet50 is different from the output of the model. Actually, the value of the output seems to be 'random'. What am I doing wrong?
Thanks in advance!
Here it is my code:
import tensorflow as tf
from resnet50 import ResNet50
from keras.preprocessing import image
from imagenet_utils import preprocess_input
import numpy as np
from keras import backend as K
img_path = 'images/tennis_ball.jpg'
img = image.load_img(img_path, target_size=(224, 224))
x = image.img_to_array(img)
x = np.expand_dims(x, axis=0)
x_image = preprocess_input(x)
#Basic prediction
model_basic = ResNet50(weights='imagenet', include_top=False)
x_prediction = model_basic.predict(x_image)
#Using tensorflow to obtain the output
input_tensor = tf.placeholder(tf.float32, shape=[None, 224,224, 3], name='input_tensor')
model = ResNet50(weights='imagenet', include_top=False, input_tensor=input_tensor)
x = model.output
# Tensorflow session
session = tf.Session()
session.run(tf.global_variables_initializer())
K.set_session(session)
feed_dict = {input_tensor: x_image, K.learning_phase(): 0}
# Obatin the output given the same input
x_output = session.run(x, feed_dict=feed_dict)
# Different results
print('Value of the prediction: {}'.format(x_prediction))
print('Value of the output: {}'.format(x_output))
Here it is an example of the logs:
Value of the prediction: [[[[ 1.26408589e+00 3.91489342e-02 8.43058806e-03 ...,
5.63185453e+00 4.49339962e+00 5.13037841e-04]]]]
Value of the output: [[[[ 2.62883282 2.20199227 9.46755123 ..., 1.24660134 1.98682189
0.63490123]]]]
The problem was that session.run(tf.global_variables_initializer()) initializes your parameters to random values.
The problem was solve by using:
session = K.get_session()
instead of:
session = tf.Session()
session.run(tf.global_variables_initializer())