If I want to replicate the layers in a TFLite model using python tensorflow functions for performing a few experiments on the tensor data, how do I do that?
conv can be done by tf.nn.conv2d, but adding bias to it and then applying relu is not giving correct output.
Which all functions would work - Model - tf resnet50 converted to tflite using tensorflow lite converter and optimizations command
I went through this a few months ago. The problem with trying to replicate TFLite layers in regular Tensorflow is that the ordering for weights is different. Example for conv2d:
TFLite - [out_channels, filter_height, filter_width, in_channels]
Regular TF - [filter_height, filter_width, in_channels, out_channels]
Here is an example python implementation that takes in a TFLite tensor W and reorders it such that it can be used in regular TF:
def reorderWeights(W, in_channels, out_channels, kernel):
flatW = W.flatten()
newW = []
for j in range(in_channels*kernel*kernel):
for i in range(out_channels):
newW.append(flatW[i*(kernel*kernel*in_channels) + j])
newW = np.array(newW)
return newW.reshape(kernel, kernel, in_channels, out_channels)
Related
Im try convert old project writen on Keras to PyTorch.
Keras create_model() contains folowing code. This is (129,500,1) grayscale image as input and (None, 2, 14, 2038) as output. Output tensor used in another BiLSTM later.
from tensorflow.python.keras.applications.inception_v3 import InceptionV3
inception_model = InceptionV3(include_top=False, weights=None, input_tensor=input_tensor)
for layer in inception_model.layers:
layer.trainable = False
x = inception_model.output
How I am can convert this code to Pytorch? The main problem is "include_top=False" what do not exist in Pytorch torchvision.inception_v3 model. This flag allow Keras model work with non-standard 1 channel inputs and 4-dim last Conv block outputs.
Actually, InceptionV3 model available in PyTorch.
You can try the below code.
import torchvision
torchvision.models.inception_v3()
So I am working with different deep learning frameworks as part of my research and have observed something weird (at least I cannot explain the cause of it).
I trained a fairly simple MLP model (on mnist dataset) in Tensorflow, extracted trained weights, created the same model architecture in PyTorch and applied the trained weights to PyTorch model. Now my expectation is to get same test accuracy from both Tensorflow and PyTorch models but this isn't the case. I get different results.
So my question is: If a model is trained to some optimal value, shouldn't the trained weights produce same results every time testing is done on the same dataset (regardless of the framework used)?
PyTorch Model:
class Net(nn.Module):
def __init__(self) -> None:
super(Net, self).__init__()
self.fc1 = nn.Linear(784, 24)
self.fc2 = nn.Linear(24, 10)
def forward(self, x: Tensor) -> Tensor:
x = torch.flatten(x, 1)
x = F.relu(self.fc1(x))
x = self.fc2(x)
return x
Tensorflow Model:
def build_model() -> tf.keras.Model:
# Build model layers
model = models.Sequential()
# Flatten Layer
model.add(layers.Flatten(input_shape=(28,28)))
# Fully connected layer
model.add(layers.Dense(24, activation='relu'))
model.add(layers.Dense(10))
# compile the model
model.compile(
optimizer='sgd',
loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
metrics=['accuracy']
)
# return newly built model
return model
To extract weights from Tensorflow model and apply them to Pytorch model I use following functions:
Extract Weights:
def get_weights(model):
# fetch latest weights
weights = model.get_weights()
# transpose weights
t_weights = []
for w in weights:
t_weights.append(np.transpose(w))
# return
return t_weights
Apply Weights:
def set_weights(model, weights):
"""Set model weights from a list of NumPy ndarrays."""
state_dict = OrderedDict(
{k: torch.Tensor(v) for k, v in zip(model.state_dict().keys(), weights)}
)
self.load_state_dict(state_dict, strict=True)
Providing solution in answer section for the benefit of community. From comments
If you are using the same weights in the same manner then results
should be the same, though float rounding error should also be
accounted. Also it doesn't matter if model is trained at all. You can
think of your model architecture as a chain of matrix multiplications
with element-wise nonlinearities in between. How big is the
difference? Are you comparing model outputs, our metrics computed over
dataset? As a suggestion, intialize model with some random values in
Keras, do a forward pass for a single batch (paraphrased from jdehesa and Taras Sereda)
How can a Tensorflow feature_column be used in conjunction with a Keras model?
E.g. for a Tensorflow estimator, we can use an embedding column from Tensorflow Hub:
embedded_text_feature_column = hub.text_embedding_column(
key="sentence",
module_spec="https://tfhub.dev/google/nnlm-en-dim128/1")
estimator = tf.estimator.DNNClassifier(
hidden_units=[100],
feature_columns=[embedded_text_feature_column],
n_classes=2,
optimizer=tf.train.AdamOptimizer(learning_rate=0.001))
However, I would like to use the TF Hub text_embedding_column as input to a Keras model. E.g.
net = tf.keras.layers.Input(...) # use embedding column here
net = tf.keras.layers.Flatten()
net = Dense(100, activation='relu')(net)
net = Dense(2)(net)
Is this possible?
The answer seems to be that you don't use feature columns. Keras comes with its own set of preprocessing functions for images and text, so you can use those.
So basically the tf.feature_columns are reserved for the high level API. Then the tf.keras.preprocessing() functions are used with tf.keras models.
Here is a link to the section on preprocessing data in the keras documentation.
https://keras.io/preprocessing/text/
Here is another Stackoverflow post that has an example of this approach.
Add Tensorflow pre-processing to existing Keras model (for use in Tensorflow Serving)
The keras functional api is a viable way to do this, but if you want to use feature_columns this tutorial shows you how:
https://www.tensorflow.org/beta/tutorials/keras/feature_columns
Basically it's this DenseFeatures layer that does the job:
feature_layer = tf.keras.layers.DenseFeatures(feature_columns)
model = tf.keras.Sequential([
feature_layer,
layers.Dense(128, activation='relu'),
layers.Dense(128, activation='relu'),
layers.Dense(1, activation='sigmoid')
])
I'm building image processing network in tensorflow and I want to make use of texture loss. Texture loss seems simple to implement if you have pretrained model loaded.
I'm using TF to build the computational graph for my model and I want to incorporate Keras.application.VGG19 model to get output from layer 'block4_conv4'.
The problem is: I have two TF tensors target and result from my main model, how to feed them into keras VGG19 in the same session to compute their diff and use it in main loss for my model?
It seems following code does the trick
with tf.variable_scope("") as scope:
phi_func = VGG19(include_top=False, weights=None, input_shape=(128, 128, 3))
text_1 = phi_func(predicted)
scope.reuse_variables()
text_2 = phi_func(x)
text_loss = tf.reduce_mean((text_1 - text_2)**2)
right after session created I call phi_func.load_weights(path) to initiate weights
I am using keras==2.0.8 with tensorflow==1.3.0 backend.
Here is the example which I am confused with:
from keras.layers import Input, Reshape, Conv2DTranspose
x = Input((5000,))
y = Reshape((25, 25, 8))(x)
y = Conv2DTranspose(10, 5, padding='same', strides=2)(y)
print(y)
It's just part of my model and after these lines I use y in some tensorflow operations, but code above prints node of shape (?, ?, ?, 10). I have no idea why TF cannot deduce height and width of resulting tensor statically. (I know that keras can, but I want TF node with proper shape)
If you intend to use these tensorflow operations in a keras model, you have to use them inside Lambda layers.
In the function you create for the lambda layer, you can use the given tensor normally. Unless you have a very specific reason for tensorflow to have this fixed size explicit, there won't be any problem. Is there any special need that demands you to have the tensorflow tensor with explicit shape?
In Keras, you can always use K.shape() in a keras tensor to get its shape. Many keras backend functions can take this shape (mostly with tensorflow) as input. If you can use the keras backend functions instead of pure tensorflow functions, your code may be portable to other backends later.
Example of function:
def tensorflowPart(x):
#do tensorflow operations with the tensor x
shape = K.shape(x) #use the shape of the tensor, as a tensor
#more tensorflow operations
return result
Use the lambda layer in your model:
y = Lambda(tensorflowPart)(y)