Tensorflow Lite gives me 0% accuracy for any image - tensorflow

Here is my Colab notebook which anyone should be able to run in 5 seconds. The predictions are always 0. I suspect the problem is in these two lines. Or perhaps my tflite model is corrupt or wrong?
label_id, prob = classify_image(interpreter, np.expand_dims(img_array, axis=0)/255 )
#label_id, prob = classify_image(interpreter, img )

Think I figured it out. Basically the output from the classify_image() function needs furthur processing to be useable. So I need to do this on the output:
label_index= np.argmax(probs_lite)
score = tf.nn.softmax(probs_lite)
print(
"This image most likely belongs to {} with a {:.2f} percent confidence."
.format(class_names[label_index], 100 * np.max(score))
)
And here's my full script in case anyone has similiar issue in the future:
import itertools, time, os, numpy as np,sys
from PIL import Image
import tensorflow as tf
def lite_model(images):
interpreter.allocate_tensors()
interpreter.set_tensor(interpreter.get_input_details()[0]['index'], images)
interpreter.invoke()
return interpreter.get_tensor(interpreter.get_output_details()[0]['index'])
class_names = ['empty_court', 'occupied_court' ]
path_to_tflite = tf.keras.utils.get_file(
"court.tflite",
"https://mysite.site/AI/court.tflite",
untar=False)
interpreter = tf.lite.Interpreter( path_to_tflite )
interpreter.allocate_tensors()
_, height, width, _ = interpreter.get_input_details()[0]['shape']
print("Image Shape (", width, ",", height, ")")
image_url = tf.keras.utils.get_file('Court', origin='https://mysite.site/share/court4.jpg' )
img = tf.keras.preprocessing.image.load_img(image_url, target_size=( width, height ) )
os.remove(image_url) # Remove the cached file
img_array = tf.keras.preprocessing.image.img_to_array(img)
probs_lite = lite_model( np.expand_dims(img_array, axis=0)/255 )[0]
print ( probs_lite )
label_index= np.argmax(probs_lite)
score = tf.nn.softmax(probs_lite)
print(
"This image most likely belongs to {} with a {:.2f} percent confidence."
.format(class_names[label_index], 100 * np.max(score))
)

Related

How to use TensorFlow lite on a raspberry pi 4 without keras?

Basically I want to convert this code snippet to code that opens a tflite model and does not use keras. I can not install keras on my raspberry pi 4 as it needs Tensorflow 2+.
model = keras.models.load_model( saved_model_path )
image_url = tf.keras.utils.get_file('Court', origin='https://squashvideo.site/share/court3.jpg' )
img = tf.keras.preprocessing.image.load_img(image_url, target_size=( 224, 224 ) )
os.remove(image_url) # Remove the cached file
img_array = tf.keras.preprocessing.image.img_to_array(img)
prediction_scores = model.predict(np.expand_dims(img_array, axis=0)/255)
score = tf.nn.softmax(prediction_scores[0])
print(
"This image most likely belongs to {} with a {:.2f} percent confidence."
.format(class_names[np.argmax(score)], 100 * np.max(score))
)
Here's what I have tried which gives the error below:
from PIL import Image
def classify_image(interpreter, image, top_k=1):
tensor_index = interpreter.get_input_details()[0]['index']
input_tensor = interpreter.tensor(tensor_index)()[0]
input_tensor[:, :] = image
interpreter.invoke()
output_details = interpreter.get_output_details()[0]
output = np.squeeze(interpreter.get_tensor(output_details['index']))
scale, zero_point = output_details['quantization']
output = scale * (output - zero_point)
ordered = np.argpartition(-output, top_k)
return [(i, output[i]) for i in ordered[:top_k]][0]
interpreter = Interpreter('/var/www/html/share/AI/court.tflite')
interpreter.allocate_tensors()
_, height, width, _ = interpreter.get_input_details()[0]['shape']
print("Image Shape (", width, ",", height, ")")
data_folder = "/var/www/html/share/"
image = Image.open(data_folder + "court1.jpg").convert('RGB').resize((width, height))
label_id, prob = classify_image(interpreter, image)
Running gives the error:
squash#court1:/var/www/html/share/AI $ python3 test.py
Image Shape ( 224 , 224 )
Traceback (most recent call last):
File "test.py", line 44, in <module>
label_id, prob = classify_image(interpreter, image)
File "test.py", line 22, in classify_image
interpreter.invoke()
File "/home/squash/.local/lib/python3.7/site-packages/tflite_runtime/interpreter.py", line 539, in invoke
self._ensure_safe()
File "/home/squash/.local/lib/python3.7/site-packages/tflite_runtime/interpreter.py", line 287, in _ensure_safe
data access.""")
RuntimeError: There is at least 1 reference to internal data
in the interpreter in the form of a numpy array or slice. Be sure to
only hold the function returned from tensor() if you are using raw
data access.
The error is in the way you are feeding data to the tflite Interpreter here:
input_tensor = interpreter.tensor(tensor_index)()[0]
input_tensor[:, :] = image
The Image.open function return an Image object. You need to convert it into binary data before feeding it to a tensor. An you should use:
interpreter.set_tensor(0, image_data)
to set the data instead of above assignment.
Think I fixed it by doing this:
img = Image.open( image_url ).convert('RGB').resize((224, 224))
img_array = np.array ( img, dtype=np.float32 )
probs_lite = lite_model( np.expand_dims(img_array, axis=0)/255 )[0]
print ( probs_lite )
print (np.argmax(probs_lite))
score = tf.nn.softmax(probs_lite)
print(
"This image most likely belongs to {} with a {:.2f} percent confidence."
.format(class_names[np.argmax(score)], 100 * np.max(score))
)

How to feed tensorflow an image from a url?

The five lines commented out below should work but do not . The prediction score is not anywhere close to what I would expect and when I do plt.imshow(img) it shows the wrong image. Here is the link to my notebook in Colab.
x, y = next(valid_generator)
image = x[0, :, :, :]
true_index = np.argmax(y[0])
plt.imshow(image)
image_url = 'https://mysite_example/share/court3.jpg'
image_url = tf.keras.utils.get_file('Court', origin=image_url )
#img = keras.preprocessing.image.load_img( image_url, target_size=( 224, 224 ) )
#img_array = keras.preprocessing.image.img_to_array(img)
#img_array = tf.expand_dims(img_array, 0)
#prediction_scores = model.predict(np.expand_dims(img_array, axis=0))
#plt.imshow(img)
# Expand the validation image to (1, 224, 224, 3) before predicting the label
prediction_scores = model.predict(np.expand_dims(image, axis=0))
predicted_index = np.argmax(prediction_scores)
print("True label: " + get_class_string_from_index(true_index))
print("Predicted label: " + get_class_string_from_index(predicted_index)
The method tf.keras.utils.get_file downloads the file from url to local cache only if the file is not already cached. So if you are using the same cache name for all the urls ("Court" in your code ?) the you will see only the first file.
Also while training you have a preprocess step of normalizing all the pixels by dividing them with 255. You have to apply the same preprocessing step during inference also.
Working Code:
_, axis = plt.subplots(1,3)
for i, image_url in enumerate(['https://squashvideo.site/share/court3.jpg',
'https://i.pinimg.com/originals/0f/c2/9b/0fc29b35532f8e2fb998f5605212ab27.jpg',
'https://thumbs.dreamstime.com/b/squash-court-photo-empty-30346175.jpg']):
image_url = tf.keras.utils.get_file('Court', origin=image_url )
img = tf.keras.preprocessing.image.load_img(image_url, target_size=( 224, 224 ) )
os.remove(image_url) # Remove the cached file
axis[i].imshow(img)
img_array = keras.preprocessing.image.img_to_array(img)
prediction_scores = model.predict(np.expand_dims(img_array, axis=0)/255)
axis[i].title.set_text(np.argmax(prediction_scores, axis=1))
Output:
As you can see, the predictions are perfect, the last image belong to class 0 (empty squash court) and the second image belong to class 1 (players playing in squash court)

Tensorflow 2.3 pipeline load all the data to the RAM

I created pipeline using tf.data API, for reading data set of images. I have a big dataset with high resolution. However, each time trying to reading all the dataset, the computer crash because the code using all the RAM. I tested the code with about 1280 images, it works without any error. But when I used all the datasets the model craches.
So, I am wondering if there is a way to make tf.data read a one or two batch in front not more than that.
This the code I am using to create the pipeline:
def decode_img(self, img):
img = tf.image.convert_image_dtype(img, tf.float32, saturate=False)
img = tf.image.resize(img, size=self.input_dim, antialias=False, name=None)
return img
def get_label(self, label):
y = np.zeros(self.n_class, dtype=np.float32)
y[label] = 1
return y
def process_path(self, file_path, label):
label = self.get_label(label)
img = Image.open(file_path)
width, height = img.size
# Setting the points for cropped image
new_hight = height // 2
new_width = width // 2
newsize = (new_width, new_hight)
img = img.resize(newsize)
if self.aug_img:
img = self.policy(img)
img = self.decode_img(np.array(img, dtype=np.float32))
return img, label
def create_pip_line(self):
def _fixup_shape(images, labels):
images.set_shape([None, None, 3])
labels.set_shape([7]) # I have 19 classes
return images, labels
tf_ds = tf.data.Dataset.from_tensor_slices((self.df["file_path"].values, self.df["class_num"].values))
tf_ds = tf_ds.map(lambda img, label: tf.numpy_function(self.process_path,
[img, label],
(tf.float32, tf.float32)),
num_parallel_calls=tf.data.experimental.AUTOTUNE)
tf_ds = tf_ds.map(_fixup_shape)
if not self.is_val:
tf_ds = tf_ds.shuffle(len(self.df), reshuffle_each_iteration=True)
tf_ds = tf_ds.batch(self.batch_size).repeat(self.epoch_num)
self.tf_ds = tf_ds.prefetch(tf.data.experimental.AUTOTUNE)
The main issue in my code was the Shuffle function. This function takes two parameters, the first one number of data to shuffle, the second one the repeat for each epoch.
However, I found the number of data that will be loaded to the memory depends on this function. Therefore, I reduced the number from all data to 100 and this makes the pipeline load 100 images and shuffles them then load another 100, and so on.
if not self.is_val:
tf_ds = tf_ds.shuffle(100, reshuffle_each_iteration=True)

tfp.mcmc.HamiltonianMonteCarlo Not working in Tensorflow Probability

I have the following code, which basically tries to fit a simple regression model using tensorflow probability. The code runs without error, but the MCMC sampler doesn't seem to be doing anything in that it returns a trace of the initial states.
import tensorflow.compat.v2 as tf
import tensorflow_probability as tfp
from tensorflow_probability import distributions as tfdimport warnings
tf.enable_v2_behavior()
plt.style.use("ggplot")
warnings.filterwarnings('ignore')
ru=4
N=102
N = 102 # number of data points
t = np.linspace(0, 4*np.pi, N)
data = 3+np.sin(t+0.001) + 0.5 + np.random.randn(N)
media_1 = ((data-min(data))/(max(data)-min(data)) ) #+ np.random.normal(0,.05, N)
y = np.repeat(ru, N) + np.random.normal(.3,.01,N) * media_1 + np.random.normal(0, .005, N)
# model
model = tfd.JointDistributionNamed(dict(
beta1 = tfd.Normal(0,1) ,
intercept = tfd.Normal(0,5 ) ,
var = tfd.Normal(0.05, 0.0005) ,
y = lambda intercept,beta1,var:
tfd.Independent(tfd.Normal(loc=intercept + beta1 * media_1, scale=var),
reinterpreted_batch_ndims=1
)
))
def target_log_prob_fn(intercept, beta1, var):
return model.log_prob({'intercept':intercept, 'beta1':beta1, 'var':var, 'y': y })
s = model.sample()
init_states = [ tf.fill([1], s['intercept'].numpy(), name='init_intercept'),
tf.fill([1], s['beta1'].numpy(), name='init_beta1'),
tf.fill([1], s['var'].numpy(), name='init_var'),]
num_results = 5000
num_burnin_steps = 3000
# Improve performance by tracing the sampler using `tf.function`
# and compiling it using XLA.
#tf.function(autograph=False, experimental_compile=True)
def do_sampling():
return tfp.mcmc.sample_chain(
num_results=num_results,
num_burnin_steps=num_burnin_steps,
current_state=init_states,
kernel=tfp.mcmc.HamiltonianMonteCarlo(
target_log_prob_fn=target_log_prob_fn,
step_size=0.1,
num_leapfrog_steps=3)
)
states, kernel_results = do_sampling()
The trace that is returned in states is exactly the same as the initial values in initial_states... Any ideas?
I can confirm that this MCMC sampler is not mixing by printing the acceptance rate with a snippet I found here
print("Acceptance rate:", kernel_results.is_accepted.numpy().mean())
That same page provides some hints about how to make your HMC kernel adaptive, which means it will automatically reduce the step size if too many proposed steps are rejected (and increase it if too many are accepted, too):
# Apply a simple step size adaptation during burnin
#tf.function
def do_sampling():
adaptive_kernel = tfp.mcmc.SimpleStepSizeAdaptation(
tfp.mcmc.HamiltonianMonteCarlo(
target_log_prob_fn=target_log_prob_fn,
step_size=0.1,
num_leapfrog_steps=3),
num_adaptation_steps=int(.8 * num_burnin_steps),
target_accept_prob=np.float64(.65))
return tfp.mcmc.sample_chain(
num_results=num_results,
num_burnin_steps=num_burnin_steps,
current_state=init_states,
kernel=adaptive_kernel,
trace_fn=lambda cs, kr: kr)
samples, kernel_results = do_sampling()
print("Acceptance rate:", kernel_results.inner_results.is_accepted.numpy().mean())
This produces a non-zero acceptance rate for me.

How to reverse the tf.image.per_image_standardization() function in tensorflow?

tf.image.per_image_standardization() in Tensorflow converts each image with zero mean & unit variance. So that this leads to a non-exploding gradients while training a deep learning model.But when we want to display the image array, how do we revert this z-score normalization step in Tensorflow?
By "display the image array" I assume you mean to display it in tensorboard. If this is the case then you don't need to do anything, tensorboard can handle images that have been standardized. If you want the original value for any other purpose why not just use the variable before you standardized it, such as:
img = tf.placeholder(...)
img_std = tf.image.per_image_standardization(img)
You can work with either img or img_std in any way you see fit.
If you somehow have a use case for denormalizing the standardized image that isn't covered above then you would need to compute the mean and standard deviation yourself, then multiply by the standard deviation and add the mean. Note that tf.image.per_image_standardization uses an adjusted_stddev that is defined in the documentation as:
adjusted_stddev = max(stddev, 1.0/sqrt(image.NumElements()))
The tf.image.per_image_standardization() layer will create some internal variables you can use to recover the original data. Please note that this is undocumented behavior and not guaranteed to stay the same. Still, for now, you can use the code below (tested) for reference how to get the relevant tensors and recover the original data:
import tensorflow as tf
import numpy as np
img_size = 3
a = tf.placeholder( shape = ( img_size, img_size, 1 ), dtype = tf.float32 )
b = tf.image.per_image_standardization( a )
with tf.Session() as sess:
tensors, tensor_names = [], []
for l in sess.graph.get_operations():
tensors.append( sess.graph.get_tensor_by_name( l.name + ":0" ) )
tensor_names.append( l.name )
#mean_t = sess.graph.get_tensor_by_name( "per_image_standardization/Mean:0" )
#variance_t = sess.graph.get_tensor_by_name( "per_image_standardization/Sqrt:0" )
foobar = np.reshape( np.array( range( img_size * img_size ), dtype = np.float32 ), ( img_size, img_size, 1 ) )
res = sess.run( tensors, feed_dict = { a : foobar } )
#for i in xrange( len( res ) ):
# print( i, tensor_names[ i ] + ":" )
# print( res[ i ] )
# print()
mean = res[ 6 ] # "per_image_standardization/Mean:0"
variance = res[ 13 ] # "per_image_standardization/Sqrt:0"
standardized = res[ 18 ] # "per_image_standardization:0"
original = standardized * variance + mean
print( original )
You can uncomment the mean_t and variance_t lines to get the reference to the relevant tensors by name. (Needs some rewrite of the sess.run() part.) You can uncomment the four lines starting with for i in xrange(... (no rewrite needed) to print all the available created tensors for your edification. :)
The above code, as is, outputs:
[[[0.]
[1.]
[2.]]
[[3.]
[4.]
[5.]]
[[6.]
[7.]
[8.]]]
Which is exactly the data that was fed to the network.