Tensorflow lite micro neural network layers building - tensorflow

I try to run some ML on my ESP32, and I want to use Tensorflow lite micro. But I don't really understand, how they build up the layers. So here is the example how to train the person detection model:
Person detection model training
It is clear, but at the end they say:
MobileNet v1 is a stack of 14 of these depthwise separable convolution layers with an average pool, then a fully-connected layer followed by a softmax at the end.
If I check the sample code, where they build up the tf lite micro model, it only has 3 lines:
static tflite::MicroMutableOpResolver<3> micro_op_resolver;
micro_op_resolver.AddAveragePool2D();
micro_op_resolver.AddConv2D();
micro_op_resolver.AddDepthwiseConv2D();
There is the Average pool, and the depthwise layer, but where the Conv2D layer comes from? And only 1 depthwise layer is presented, but in the documentation, there is 14 depthwise layers in the model.
So the question is, is there any relation between training model, and the model I should build in tensoflow lite micro? If there is, how can I determine how to build up. And that is the question if there is no relation, in what way I need to build up the model.

They don't explicitly build the model, they rely on a model file that contains the architecture (source):
model = tflite::GetModel(g_person_detect_model_data);
Where g_person_detect_model_data.cc is generated from a tflite model (containing the architecture) with the following command (See Converting into a c source file in the Readme) :
# Install xxd if it is not available
!apt-get -qq install xxd
# Save the file as a C source file
!xxd -i vww_96_grayscale_quantized.tflite > person_detect_model_data.cc
So the code you shared doesn't build the model. What you see is that for performances reasons, they explicitly add the ops needed by the model instead of relying on the more complex tflite::AllOpsResolver.
It is indicated by this comment above the code you shared :
// Pull in only the operation implementations we need.
// This relies on a complete list of all the ops needed by this graph.
// An easier approach is to just use the AllOpsResolver, but this will
// incur some penalty in code space for op implementations that are not
// needed by this graph.
//
// tflite::AllOpsResolver resolver;
// NOLINTNEXTLINE(runtime-global-variables)
static tflite::MicroMutableOpResolver<5> micro_op_resolver;
micro_op_resolver.AddAveragePool2D();
micro_op_resolver.AddConv2D();
micro_op_resolver.AddDepthwiseConv2D();
micro_op_resolver.AddReshape();
micro_op_resolver.AddSoftmax();

Related

Trainable USE-lite-based classifier with SentencePiece input

I have heard that it is possible to use the pretrained Universal Sentence Encoder (USE) (neural language model) from TF-hub as part of a trainable model, e.g. a sentence classifier. Some versions of USE rely on SentencePiece sub-word tokenizer, which I also need. There are minimal instructions online for how to do this.
Here is how to use USE-lite with SentencePiece:
- https://tfhub.dev/google/universal-sentence-encoder-lite/2
Here is how to train a classifier based on a pretrained USE model:
- http://hunterheidenreich.com/blog/google-universal-sentence-encoder-in-keras/
- https://www.youtube.com/watch?v=gnz1CUzb5qo
And here is how to measure sentence similarity using both USE-lite and SentencePiece:
- https://github.com/tensorflow/hub/blob/master/examples/colab/semantic_similarity_with_tf_hub_universal_encoder_lite.ipynb
I have successfully reproduced the above pieces separately. I have then tried to combine the above ideas into a single POC that will build a classifier model based on USE-lite and SentencePiece, but I cannot see how to do it. I am currently stuck on the part where I modify the trainable classifier's first layer(s). I have tried to make it accept either (1) SentencePiece token IDs (in which I tokenize the text outide of the Tensorflow graph) or (2) raw text (using SentencePiece as an Op inside the Tensorflow graph). After that point, it should feed tokenized text forward into the USE-lite model, either in a lambda or in some other way. Finally, the output of USE-lite should be fed into a dense layer (or two?) ending in softmax for computing class probabilities.
I am relatively new to Tensorflow. I imagine that the above sources would be sufficient for a more experienced Tensorflow developer to merge and make work for my use-case. Let me know if you can provide any pointers. Thanks.

Visualizing Neural Network Layer Activation

Feature visualizing in tensor flow or keras is easy and can be found here. https://machinelearningmastery.com/how-to-visualize-filters-and-feature-maps-in-convolutional-neural-networks/ or Convolutional Neural Network visualization - weights or activations?
how to do this in pytorch?
I am using PyTorch with pretrained resnet18 model. All i need to input the image and get activation for specific layer(e.g. Layer2.0.conv2). Layer2.0.conv2 is specified in the pretrained model.
In simple words; how to convert link one code to PyTorch? how to get the specific layers in resnet18 PyTorch and how to get the activation for input image.
I tried this in tensorflow and it worked but not PyTorch.
You would have to register PyTorch's hooks on specific layer. See this tutorial for intro about hooks.
Basically, it allows to capture input/output of forward/backward going into the torch.nn.Module. Whole thing could be a bit complicated, there exists a library with similar goal to your (disclaimer I'm the author), called torchfunc. Especially torchfunc.hooks.recorder allows you to do what you want, see code snippet and comments below:
import torchvision
import torchfunc
my_network = torchvision.resnet18(pretrained=True)
# Recorder saving inputs to all submodules
recorder = torchfunc.hooks.recorders.ForwardPre()
# Will register hook for all submodules of resnet18
# You could specify some submodules by index or by layer type, see docs
recorder.modules(my_networks)
# Push example image through network
my_network(torch.randn(1, 3, 224, 224))
You could register recorder only for some layers (submodule) specified by index or layer type, to get necessary info, run:
# Zero image before going into the third submodule of this network
recorder.data[3][0]
# You can see all submodules and their positions by running this:
for i, submodule in enumerate(my_network.modules()):
print(i, submodule)
# Or you can just print the network to get this info
print(my_network)

Using model optimizer for tensorflow slim models

I am aiming to inference tensorflow slim model with Intel OpenVINO optimizer. Using open vino docs and slides for inference and tf slim docs for training model.
It's a multi-class classification problem. I have trained tf slim mobilnet_v2 model from scratch (using sript train_image_classifier.py). Evaluation of trained model on test set gives relatively good results to begin with (using script eval_image_classifier.py):
eval/Accuracy[0.8017]eval/Recall_5[0.9993]
However, single .ckpt file is not saved (even though at the end of train_image_classifier.py run there is a message like "model.ckpt is saved to checkpoint_dir"), there are 3 files (.ckpt-180000.data-00000-of-00001, .ckpt-180000.index, .ckpt-180000.meta) instead.
OpenVINO model optimizer requires a single checkpoint file.
According to docs I call mo_tf.py with following params:
python mo_tf.py --input_model D:/model/mobilenet_v2_224.pb --input_checkpoint D:/model/model.ckpt-180000 -b 1
It gives the error (same if pass --input_checkpoint D:/model/model.ckpt):
[ ERROR ] The value for command line parameter "input_checkpoint" must be existing file/directory, but "D:/model/model.ckpt-180000" does not exist.
Error message is clear, there are not such files on disk. But as I know most tf utilities convert .ckpt-????.meta to .ckpt under the hood.
Trying to call:
python mo_tf.py --input_model D:/model/mobilenet_v2_224.pb --input_meta_graph D:/model/model.ckpt-180000.meta -b 1
Causes:
[ ERROR ] Unknown configuration of input model parameters
It doesn't matter for me in which way I will transfer graph to OpenVINO intermediate representation, just need to reach that result.
Thanks a lot.
EDIT
I managed to run OpenVINO model optimizer on frozen graph of tf slim model. However I still have no idea why had my previous attempts (based on docs) failed.
you can try converting the model to frozen format (.pb) and then convert the model using OpenVINO.
.ckpt-meta has the metagraph. The computation graph structure without variable values.
the one you can observe in tensorboard.
.ckpt-data has the variable values,without the skeleton or structure. to restore a model we need both meta and data files.
.pb file saves the whole graph (meta+data)
As per the documentation of OpenVINO:
When a network is defined in Python* code, you have to create an inference graph file. Usually, graphs are built in a form that allows model training. That means that all trainable parameters are represented as variables in the graph. To use the graph with the Model Optimizer, it should be frozen.
https://software.intel.com/en-us/articles/OpenVINO-Using-TensorFlow
the OpenVINO optimizes the model by converting the weighted graph passed in frozen form.

Tensorflow remove layers from pretrained model

Is there a way to load a pretrained model in Tensorflow and remove the top layers in the network? I am looking at Tensorflow release r1.10
The only documentation I could find is with tf.keras.Sequential.pop
https://www.tensorflow.org/versions/r1.10/api_docs/python/tf/keras/Sequential#pop
I want to manually prune a pretrained network by removing bunch of top convolution layers and add a custom fully convoluted layer.
EDIT:
The model is ssd_mobilenet_v1_coco downloaded from Tensorflow Model Zoo. I have access to both the frozen_inference_graph.pb model file and checkpoint file.
I donot have access to the python code which is used to construct the model.
Thanks.
From inspecting the code, SSDMobileNetV1FeatureExtractor.extract_features redirects research.slim.nets:
from nets import mobilenet_v1 # nets will have to be on your PYTHONPATH
with tf.variable_scope('MobilenetV1',
reuse=self._reuse_weights) as scope:
with slim.arg_scope(
mobilenet_v1.mobilenet_v1_arg_scope(
is_training=None, regularize_depthwise=True)):
with (slim.arg_scope(self._conv_hyperparams_fn())
if self._override_base_feature_extractor_hyperparams
else context_manager.IdentityContextManager()):
_, image_features = mobilenet_v1.mobilenet_v1_base(
ops.pad_to_multiple(preprocessed_inputs, self._pad_to_multiple),
final_endpoint='Conv2d_13_pointwise',
min_depth=self._min_depth,
depth_multiplier=self._depth_multiplier,
use_explicit_padding=self._use_explicit_padding,
scope=scope)
The mobilenet_v1_base function takes a final_endpoint argument. Rather than prune the constructed graph, just construct the graph up until the endpoint you want.

convert resnet implementation from caffe to tensorflow

I want to implement resnet 50 from scratch
it is implemented in caffe by author of original paper,but i want tensorflow implementation
due to this repository :https://github.com/KaimingHe/deep-residual-networks
and therefor this image : http://ethereon.github.io/netscope/#/gist/db945b393d40bfa26006
I know every equivalent (in tensorflow),but i dont lknow the meaning of scale in place,after batch normalization,can you explain me the meaning and also "use globale state " parameter in batchnorm ?
An "in-place" layer in caffe simply hints caffe to save memory: instead of allocating memory for both input and output of the net, "in-place" layer overrides the input with the output of the layer.
Using global state in "BatchNorm" layer means using the mean/std computed during training and not updating these values any further. This is the "deployment" state of BN layer.