I am trying to convert a trained model in tensorflow to Open VINO Intermediate Representation.
I have a model of the form given below
class Conv3DModel(tf.keras.Model):
def __init__(self):
super(Conv3DModel, self).__init__()
# Convolutions
self.conv1 = tf.compat.v2.keras.layers.Conv3D(32, (3, 3, 3), activation='relu', name="conv1", data_format='channels_last')
self.pool1 = tf.keras.layers.MaxPool3D(pool_size=(2, 2, 2), data_format='channels_last')
self.conv2 = tf.compat.v2.keras.layers.Conv3D(64, (3, 3, 3), activation='relu', name="conv1", data_format='channels_last')
self.pool2 = tf.keras.layers.MaxPool3D(pool_size=(2, 2,2), data_format='channels_last')
# LSTM & Flatten
self.convLSTM =tf.keras.layers.ConvLSTM2D(40, (3, 3))
self.flatten = tf.keras.layers.Flatten(name="flatten")
# Dense layers
self.d1 = tf.keras.layers.Dense(128, activation='relu', name="d1")
self.out = tf.keras.layers.Dense(6, activation='softmax', name="output")
def call(self, x):
x = self.conv1(x)
x = self.pool1(x)
x = self.conv2(x)
x = self.pool2(x)
x = self.convLSTM(x)
x = self.flatten(x)
x = self.d1(x)
return self.out(x)
I tried to convert the model into IR. The model is here .
I have trained this model in tensorflow 1.15. Tensorflow 2.0 is currently not supported.
Now I tried to run the command
python3 /opt/intel/openvino/deployment_tools/model_optimizer/mo_tf.py --saved_model_dir jester_trained_models/3dcnn-basic/ --output_dir /home/deepanshu/open_vino/udacity_project_custom_model/
Now i got the following error
Model Optimizer arguments:
Common parameters:
Path to the Input Model: None
Path for generated IR: /home/deepanshu/open_vino/udacity_project_custom_model/
IR output name: saved_model
Log level: ERROR
Batch: Not specified, inherited from the model
Input layers: Not specified, inherited from the model
Output layers: Not specified, inherited from the model
Input shapes: Not specified, inherited from the model
Mean values: Not specified
Scale values: Not specified
Scale factor: Not specified
Precision of IR: FP32
Enable fusing: True
Enable grouped convolutions fusing: True
Move mean values to preprocess section: False
Reverse input channels: False
TensorFlow specific parameters:
Input model in text protobuf format: False
Path to model dump for TensorBoard: None
List of shared libraries with TensorFlow custom layers implementation: None
Update the configuration file with input/output node names: None
Use configuration file used to generate the model with Object Detection API: None
Operations to offload: None
Patterns to offload: None
Use the config file: None
Model Optimizer version: 2020.1.0-61-gd349c3ba4a
[ ERROR ] Unexpected exception happened during extracting attributes for node conv3d_model/conv_lst_m2d/bias/Read/ReadVariableOp. Original exception message: 'ascii' codec can't decode byte 0xc9 in position 1: ordinal not in range(128)
As far as I can see it is the tf.keras.layers.ConvLSTM2D(40, (3, 3)) causing problems . I am kind of stuck here . Can anyone tell me where can I proceed further ?
Thanks
Edit to the question
Now I rejected the above tensorflow implementation and used keras . My h5 model developed was converted into .pb format using this post.
Now I ran the model optimizer on this .pb file. Using the command
python3 /opt/intel/openvino/deployment_tools/model_optimizer/mo_tf.py --input_model /home/deepanshu/ml_playground/jester_freezed/tf_model.pb --output_dir /home/deepanshu/open_vino/udacity_project_custom_model/ --input_shape=[1,30,64,64,1] --data_type FP32
Now i am facing another issue . The issue here is point no. 97 on this post.
So my model contains a cycle and model optimizer does not know a way to convert it. Has anybody faced this issue before ?
Please help.
Here is the model .
Here is the defination of the model in keras
from keras.models import Sequential
from keras.layers import Conv3D , MaxPool3D,Flatten ,Dense
from keras.layers.convolutional_recurrent import ConvLSTM2D
import keras
model = Sequential()
model.add(Conv3D(32, (3, 3, 3),
name="conv1" , input_shape=(30, 64, 64,1) , data_format='channels_last',
activation='relu') )
model.add(MaxPool3D(pool_size=(2, 2, 2), data_format='channels_last'))
model.add(Conv3D(64, (3, 3, 3), activation='relu', name="conv2", data_format='channels_last'))
model.add(MaxPool3D(pool_size=(2, 2,2), data_format='channels_last'))
model.add(ConvLSTM2D(40, (3, 3)))
model.add(Flatten(name="flatten"))
model.add(Dense(128, activation='relu', name="d1"))
model.add(Dense(6, activation='softmax', name="output"))
Actually the script to convert from h5 to .pb suggested by intel was not good enough. Always use the code from here to convert your keras model to .pb.
Once you obtain your .pb file now convert your model to IR using
python3 /opt/intel/openvino/deployment_tools/model_optimizer/mo_tf.py --input_model ml_playground/try_directory/tf_model.pb --output_dir /home/deepanshu/open_vino/udacity_project_custom_model/ --input_shape=[1,30,64,64,1] --data_type FP32
After the execution of this script we can obtain the intermediate representation of the keras model.
Related
When extracting a model layer output as in the Tensorflow sequential model document example below, does the input x in the code go through the my_first_layer as well before going into my_intermediate_layer layer? Or does it directly go into the my_intermediate_layer layer without going through the my_first_layer layer?
If it directly goes into the my_intermediate_layer, the input to the my_intermediate_layer does not have the transformation done by my_first_layer Conv2D. However, it seems not right to me because the input should go through all the preceding layers.
Please help understand what layers does x go through?
Feature extraction with a Sequential model
initial_model = keras.Sequential(
[
keras.Input(shape=(250, 250, 3)),
layers.Conv2D(32, 5, strides=2, activation="relu", name="my_first_layer"),
layers.Conv2D(32, 3, activation="relu", name="my_intermediate_layer"),
layers.Conv2D(32, 3, activation="relu"),
]
)
# The model goes through the training.
...
# Feature extractor
feature_extractor = keras.Model(
inputs=initial_model.inputs,
outputs=initial_model.get_layer(name="my_intermediate_layer").output,
)
# Call feature extractor on test input.
x = tf.ones((1, 250, 250, 3))
features = feature_extractor(x)
Keras offers higher level of API, which runs on top of the TensorFlow machine learning platform. Keras offers two types of class to define the neural network model, namely 'Sequential Class' and 'Model Class.'
Sequential Class:
It groups a linear stack of layers to form a model, such that each layer has one input and one output tensor. One can add required layers to the defined model (schema-1) as shown below to execute sequentially as name suggests Keras Sequential Class,
model = tf.keras.Sequential()
model.add(tf.keras.layers.Dense(8, input_shape=(16,)))
model.add(tf.keras.layers.Dense(4))
The schema for defining a sequential model Keras-Sequential Class Definition has shown below (schema-2),
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers
model = keras.Sequential(
[
layers.Dense(2, activation="relu", name="layer1"),
layers.Dense(3, activation="relu", name="layer2"),
layers.Dense(4, name="layer3"),
]
)
# Call model on a test input
x = tf.ones((3, 3))
y = model(x)
Model Class
It allows the user to build a custom model along with many layers as shown below,
import tensorflow as tf
inputs = tf.keras.Input(shape=(3,))
x = tf.keras.layers.Dense(4, activation=tf.nn.relu)(inputs)
outputs = tf.keras.layers.Dense(5, activation=tf.nn.softmax)(x)
model = tf.keras.Model(inputs=inputs, outputs=outputs)
It allows one to create a new functional API model with additional layers Keras - Model Class as follows,
inputs = keras.Input(shape=(None, None, 3))
processed = keras.layers.RandomCrop(width=32, height=32)(inputs)
conv = keras.layers.Conv2D(filters=2, kernel_size=3)(processed)
pooling = keras.layers.GlobalAveragePooling2D()(conv)
feature = keras.layers.Dense(10)(pooling)
Note: The input tensors supports only dicts, lists or tuples but not lists of list, or dicts of dict.
I hope that this helps.
I am trying to build a CNN model to recognise human sketch using the TU-Berlin dataset. I downloaded the png zip file, imported the data to Google Colab and then split the data into train-test folders. Here is the model:
model = tf.keras.models.Sequential([
tf.keras.layers.Conv2D(filters = 64, kernel_size = (5,5),padding = 'Same',
activation ='relu', input_shape = target_dims),
tf.keras.layers.Conv2D(filters = 64, kernel_size = (5,5),padding = 'Same',
activation ='relu'),
tf.keras.layers.MaxPool2D(pool_size=(2,2)),
tf.keras.layers.Dropout(0.25),
tf.keras.layers.Conv2D(filters = 128, kernel_size = (3,3),padding = 'Same',
activation ='relu'),
tf.keras.layers.Conv2D(filters = 128, kernel_size = (3,3),padding = 'Same',
activation ='relu'),
tf.keras.layers.MaxPool2D(pool_size=(2,2), strides=(2,2)),
tf.keras.layers.Dropout(0.25),
tf.keras.layers.Conv2D(256, kernel_size=4, strides=1, activation='relu', padding='same'),
tf.keras.layers.Conv2D(256, kernel_size=4, strides=2, activation='relu', padding='same'),
tf.keras.layers.Dropout(0.25),
tf.keras.layers.Flatten(),
tf.keras.layers.Dense(512, activation = "relu"),
tf.keras.layers.Dropout(0.5),
tf.keras.layers.Dense(n_classes, activation= "softmax")
])
model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=["accuracy"])
model.fit_generator(train_generator, epochs=10, validation_data=val_generator)
And I am getting the following error:
UnimplementedError: Fused conv implementation does not support grouped convolutions for now.
[[node sequential/conv2d/Relu (defined at <ipython-input-9-36d4624b896d>:1) ]] [Op:__inference_train_function_1358]
Function call stack:
train_function
I would be grateful to any kind of help that will solve this issue. Thank you.
(PS - I am running Tensorflow 2.2.0 and no GPU)
I had a similar error, the problem was with the number of channels for my image and the number of channels I specified in the model. So check the number of dimension of your image and check the value specified in the input shape ensure they are the same
I had this same error using the facial expression recognition dataset, here's how i solved this same error.
From what i understand the dataset is gray color,
when you use ImageDataGenerator of tensorflow and flow_from_directory to generate the train and validation set,
you need to specify the color_mode as grayscale or rgb based on the dataset/images, here it will be 'grayscale',
in the model the first layer Conv2D the input_shape should be
input_shape = (height, width, 1), 1 because its grayscale.
Just mention the color_mode="grayscale" in flow from directory and check your model input (height,width,1).
Just as #grande_cifer said, the issue pops up from an incompatibility of number of image channel specified and correct number of channels of real images.
If you are not sure of the exact number of channel, I advice you specify 1 in your parameter target_dims, and forcefully convert all images when loading them to your net as grayscale, using the parameter color_mode = "grayscale" when loading the images to your net.
For more info, check keras online doc.
You will find this error in 2 cases:
when the number of channels for your image and the number of channels you specified in the model are not same . Here the solution is to make them equal.
When you use group param of Conv2D from tensorflow.keras . Here they have not implemented it with the use of group param , which is Depthwise Convolution in real (use tf.keras.layers.DepthwiseConv2D). For me the work around was pip install tf-nightly==2.10.0.dev20220406 as this package also have some unimplemented keras APIs...as this was not mentioned anywhere when I encountered this error
I hope this is useful
I'm training some models using tf.keras and want to save the trained model. There are two recommended ways of doing this, tf SavedModel and keras .h5 file. However things get really confusing with SavedModel.
Here are some short scripts to reproduce the issue:
import numpy as np
import tensorflow as tf
def my_network():
backbone_model = tf.keras.applications.InceptionResNetV2(
include_top=False, input_shape=(224, 224, 3), weights="imagenet", pooling="avg"
)
inputs = tf.keras.layers.Input(shape=(224, 224, 3), name="images")
backbone_features = backbone_model(inputs)
pre_embeddings = tf.keras.layers.Dense(
512,
activation=None,
name="pre_embeddings",
kernel_regularizer=tf.keras.regularizers.l2(),
)(backbone_features)
embeddings = tf.keras.layers.Lambda(
lambda x: tf.math.l2_normalize(x, 1, 1e-10), name="embeddings"
)(pre_embeddings)
probs = tf.keras.layers.Dense(
1000,
activation="softmax",
name="predictions",
kernel_regularizer=tf.keras.regularizers.l2(),
bias_regularizer=tf.keras.regularizers.l2(),
)(pre_embeddings)
return tf.keras.Model(inputs, [embeddings, probs], name="my_network")
img_arr = np.random.rand(1, 224, 224, 3)
resnet_model = my_network()
emb_1, _ = resnet_model.predict(img_arr)
resnet_model.save("./resnet_model.h5")
new_model = tf.keras.models.load_model('./resnet_model.h5')
emb_2, _ = new_model.predict(img_arr)
np.testing.assert_array_almost_equal(emb_1, emb_2)
The above script will work without errors. However it fails when I tried to save in the SavedModel format (by removing .h5 from model path). The model saved successfully but throws an error when loading and the error message is:
NotImplementedError: When subclassing the `Model` class, you should implement a `call` method.
I am confused because I did not use any subclassed model. As shown in the script, my network is built with functional API only.
I am trying to adapt Python code for a Convolutional Neural Network (in Keras) with 8 classes to work on 2 classes. My problem is that I get the following error message:
ValueError: Error when checking target: expected activation_6 to have
shape(None,2) but got array with shape (5760,1).
My Model is as follows (without the indentation issues):
class MiniVGGNet:
#staticmethod
def build(width, height, depth, classes):
# initialize the model along with the input shape to be
# "channels last" and the channels dimension itself
model = Sequential()
inputShape = (height, width, depth)
chanDim = -1
# if we are using "channels first", update the input shape
# and channels dimension
if K.image_data_format() == "channels_first":
inputShape = (depth, height, width)
chanDim = 1
# first CONV => RELU => CONV => RELU => POOL layer set
model.add(Conv2D(32, (3, 3), padding="same",
input_shape=inputShape))
model.add(Activation("relu"))
model.add(BatchNormalization(axis=chanDim))
model.add(Conv2D(32, (3, 3), padding="same"))
model.add(Activation("relu"))
model.add(BatchNormalization(axis=chanDim))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.25))
# second CONV => RELU => CONV => RELU => POOL layer set
model.add(Conv2D(64, (3, 3), padding="same"))
model.add(Activation("relu"))
model.add(BatchNormalization(axis=chanDim))
model.add(Conv2D(64, (3, 3), padding="same"))
model.add(Activation("relu"))
model.add(BatchNormalization(axis=chanDim))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.25))
# first (and only) set of FC => RELU layers
model.add(Flatten())
model.add(Dense(512))
model.add(Activation("relu"))
model.add(BatchNormalization())
model.add(Dropout(0.5))
# softmax classifier
model.add(Dense(classes))
model.add(Activation("softmax"))
# return the constructed network architecture
return model
Where classes = 2, and inputShape=(32,32,3).
I know that my error has something to do with my classes/use of binary_crossentropy and occurs in the model.fit line below, but haven't been able to figure out why it is problematic, or how to fix it.
By changing model.add(Dense(classes)) above to model.add(Dense(classes-1)) I can get the model to train, but then my labels size and target_names are mismatched, and I have only one category which everything is categorized as.
# import the necessary packages
from sklearn.preprocessing import LabelBinarizer
from sklearn.model_selection import train_test_split
from sklearn.metrics import classification_report
from pyimagesearch.nn.conv import MiniVGGNet
from pyimagesearch.preprocessing import ImageToArrayPreprocessor
from pyimagesearch.preprocessing import SimplePreprocessor
from pyimagesearch.datasets import SimpleDatasetLoader
from keras.optimizers import SGD
#from keras.datasets import cifar10
from imutils import paths
import matplotlib.pyplot as plt
import numpy as np
import argparse
# construct the argument parse and parse the arguments
ap = argparse.ArgumentParser()
ap.add_argument("-d", "--dataset", required=True,
help="path to input dataset")
ap.add_argument("-o", "--output", required=True,
help="path to the output loss/accuracy plot")
args = vars(ap.parse_args())
# grab the list of images that we'll be describing
print("[INFO] loading images...")
imagePaths = list(paths.list_images(args["dataset"]))
# initialize the image preprocessors
sp = SimplePreprocessor(32, 32)
iap = ImageToArrayPreprocessor()
# load the dataset from disk then scale the raw pixel intensities
# to the range [0, 1]
sdl = SimpleDatasetLoader(preprocessors=[sp, iap])
(data, labels) = sdl.load(imagePaths, verbose=500)
data = data.astype("float") / 255.0
# partition the data into training and testing splits using 75% of
# the data for training and the remaining 25% for testing
(trainX, testX, trainY, testY) = train_test_split(data, labels,
test_size=0.25, random_state=42)
# convert the labels from integers to vectors
trainY = LabelBinarizer().fit_transform(trainY)
testY = LabelBinarizer().fit_transform(testY)
# initialize the label names for the items dataset
labelNames = ["mint", "used"]
# initialize the optimizer and model
print("[INFO] compiling model...")
opt = SGD(lr=0.01, decay=0.01 / 10, momentum=0.9, nesterov=True)
model = MiniVGGNet.build(width=32, height=32, depth=3, classes=2)
model.compile(loss="binary_crossentropy", optimizer=opt,
metrics=["accuracy"])
# train the network
print("[INFO] training network...")
H = model.fit(trainX, trainY, validation_data=(testX, testY),
batch_size=64, epochs=10, verbose=1)
print ("Made it past training")
# evaluate the network
print("[INFO] evaluating network...")
predictions = model.predict(testX, batch_size=64)
print(classification_report(testY.argmax(axis=1),
predictions.argmax(axis=1), target_names=labelNames))
# plot the training loss and accuracy
plt.style.use("ggplot")
plt.figure()
plt.plot(np.arange(0, 10), H.history["loss"], label="train_loss")
plt.plot(np.arange(0, 10), H.history["val_loss"], label="val_loss")
plt.plot(np.arange(0, 10), H.history["acc"], label="train_acc")
plt.plot(np.arange(0, 10), H.history["val_acc"], label="val_acc")
plt.title("Training Loss and Accuracy on items dataset")
plt.xlabel("Epoch #")
plt.ylabel("Loss/Accuracy")
plt.legend()
plt.savefig(args["output"])
I have looked at these questions already, but cannot workout how to get around this problem based on the responses.
Stackoverflow Question 1
Stackoverflow Question 2
Stackoverflow Question 3
Any advice or help would be much appreciated, as I've spent the last couple of days on this.
Matt's comment was absolutely correct in that the problem lay with using LabelBinarizer and this hint led me to a solution that did not require me to give up using softmax, or change the last layer to have classes = 1. For posterity and for others, here's the section of code that I altered and how I was able to avoid LabelBinarizer:
from keras.utils import np_utils
from sklearn.preprocessing import LabelEncoder
# load the dataset from disk then scale the raw pixel intensities
# to the range [0,1]
sp = SimplePreprocessor (32, 32)
iap = ImageToArrayPreprocessor()
# encode the labels, converting them from strings to integers
le=LabelEncoder()
labels = le.fit_transform(labels)
data = data.astype("float") / 255.0
labels = np_utils.to_categorical(labels,2)
# partition the data into training and testing splits using 75% of
# the data for training and the remaining 25% for testing
....
I believe the problem lies in the use of LabelBinarizer.
From this example:
>>> lb = preprocessing.LabelBinarizer()
>>> lb.fit_transform(['yes', 'no', 'no', 'yes'])
array([[1],
[0],
[0],
[1]])
I gather that the output of your transformation has the same format, i. e. a single 1 or 0 encoding "is new" or "is used".
If your problem only calls for classification among these two classes, that format is preferable because it contains all the information and uses less space than the alternative, i. e. [1,0], [0,1], [0,1], [1,0].
Therefore, using classes = 1 would be correct, and the output should be a float indicating the network's confidence in a sample being in the first class. Since these values have to sum to one, the probability of it being in the second class could easily be inferred by subtracting from 1.
You would need to replace softmax with any other activation, because softmax on a single value always returns 1. I'm not completely sure about the behaviour of binary_crossentropy with a single-valued result, and you may want to try mean_squared_error as the loss.
If you are looking to expand your model to cover more than two classes, you would want to convert your target vector to a One-hot encoding. I believe inverse_transform from LabelBinarizer would do this, although that would seem to be quite a roundabout way to get there. I see that sklearn also has OneHotEncoder which may the more appropriate replacement.
NB: You can specify the activation function for any layer more easily with, for example:
Dense(36, activation='relu')
This may be helpful in keeping your code to a manageable size.
I have Keras' image_dim_ordering property set to 'tf', so I define my models as this:
model = Sequential()
model.add(ZeroPadding2D((1, 1), input_shape=(224, 224, 3)))
model.add(Convolution2D(64, 3, 3, activation='relu'))
But when I call load_weights method, it crashes because my model was saved using "th" format:
Exception: Layer weight shape (3, 3, 3, 64) not compatible with provided weight shape (64, 3, 3, 3)
How can I load these weights and automatically transpose them to fix Tensorflow's format?
I asked Francois Chollet about this (he doesn't have an SO account) and he kindly passed along this reply:
"th" format means that the convolutional kernels will have the shape (depth, input_depth, rows, cols)
"tf" format means that the convolutional kernels will have the shape (rows, cols, input_depth, depth)
Therefore you can convert from the former to the later via np.transpose(x, (2, 3, 1, 0)) where x is the value of the convolution kernel.
Here's some code to do the conversion:
from keras import backend as K
K.set_image_dim_ordering('th')
# build model in TH mode, as th_model
th_model = ...
# load weights that were saved in TH mode into th_model
th_model.load_weights(...)
K.set_image_dim_ordering('tf')
# build model in TF mode, as tf_model
tf_model = ...
# transfer weights from th_model to tf_model
for th_layer, tf_layer in zip(th_model.layers, tf_model.layers):
if th_layer.__class__.__name__ == 'Convolution2D':
kernel, bias = layer.get_weights()
kernel = np.transpose(kernel, (2, 3, 1, 0))
tf_layer.set_weights([kernel, bias])
else:
tf_layer.set_weights(tf_layer.get_weights())
In case the model contains Dense layers downstream of the Convolution2D layers, then the weight matrix of the first Dense layer would need to be shuffled as well.
You can Use This Script which auto translates theano/tensorflow backend trained model weights directly into the other 3 possible combinations of backend / dim ordering.