In classify_image.py, the input image is fed with a loaded image in
predictions = sess.run(softmax_tensor,{'DecodeJpeg/contents:0': image_data})
What if I want to add new layers to the inception model and train the whole model again? Are the variables loaded from classify_image_graph_def.pb trainable? I saw that freeze_graph.py used convert_variables_to_constants to produce freezed graph. So can those loaded weights be trained again, are they constants? And how can I connect the input('shuffle_batch:0') to the inception model to the output of tf.train.shuffle_batch?
The model used in classify_image.py has its variables frozen into constants, and doesn't have any gradient ops, so it's not easy to turn it back into something trainable. You can see how we remove one layer and replace it with something trainable here:
https://github.com/tensorflow/tensorflow/blob/master/tensorflow/examples/image_retraining/retrain.py
It's hard to generalize though. You'd be better off looking at some examples of fine-tuning here:
https://github.com/tensorflow/models/tree/master/inception#how-to-fine-tune-a-pre-trained-model-on-a-new-task
Related
A layer (....) which is an input to the Conv operator producing the output array model/re_lu_1/Relu, is lacking min/max data, which is necessary for quantization. If accuracy matters, either target a non-quantized output format, or run quantized training with your model from a floating point checkpoint to change the input graph to contain min/max information. If you don't care about accuracy, you can pass --default_ranges_min= and --default_ranges_max= for easy experimentation.
For tensorflow 1.x, if you want to quantize, you have to place it with fake quantization nodes to activate the quantization of the model.
There are 3 phases of quantization:
Training part: load your model to graph => create training graph by contrib => train and store weights ckpt
Eval part: load your model to graph without weights => create eval graph => restore graph => export to frozen model
Toco/tflite convert frozen model to quantized model
However, the most important factor is the configuration of batch_normalization in the model. After trying multiple configuration, the best one is using batch_normalization without fused option from tensorflow.keras.layers.
The reason is because Tensorflow want to avoid the folding result to be quantized. Therefore, activation behind batchnorm wont work. Details in [here][1]
In short, this layer should be attached only under tensorflow.keras.layers.Conv2D with parsed activation param, which is Relu/Relu6/Identity
If you conduct the above process: Conv2d=>Activation=>BatchNorm
the layer will not yield errors does not have MinMax information
I'm reading the official tutorial on save-load in Keras and it seems whether I used save or save_weights methods, then the optimizer parameters are going to be saved at any rate. How can save model's weights only?
model.save('./savedmodel.h5', save_format='h5', include_optimizer=False)
If save_format='tf', whether include_optimizer=False or True, it's useless as I tried.
In Keras, to save model weights, do:
model.save_weights('my_model_weights.h5')
To load model weights:
model.load_weights('my_model_weights.h5')
Also see additional example on saving/loading weights by layer name from here.
I will describe my intention here. I want to import BERT pretrained model via tf-hub function hub.module(bert_url, trainable = True) and utilize it for text classification task. I plan to use a large corpus to fine-tune weights of BERT as well as a few dense layers whose inputs are the BERT outputs. I would then like to freeze layers of BERT and train only the dense layers following BERT. How can I do this efficiently?
You mention Hub's TF1 API hub.Module, so I suppose you are writing TF1 code and using the TF1-compatible Hub assets google/bert/..., such as https://tfhub.dev/google/bert_cased_L-12_H-768_A-12/1
Are you going to have separate run of your program for the two phases of training? If so, maybe you can just drop trainable=True from the hub.Module call in the second run. This doesn't affect variable names, so you can restore the training result from the first run, including BERT's adjusted weights. (To be clear: the pre-trained weights shipped with the hub.Module are only used for initialization at the very start of training; restoring a checkpoint overrides them.)
Is there a way to load a pretrained model in Tensorflow and remove the top layers in the network? I am looking at Tensorflow release r1.10
The only documentation I could find is with tf.keras.Sequential.pop
https://www.tensorflow.org/versions/r1.10/api_docs/python/tf/keras/Sequential#pop
I want to manually prune a pretrained network by removing bunch of top convolution layers and add a custom fully convoluted layer.
EDIT:
The model is ssd_mobilenet_v1_coco downloaded from Tensorflow Model Zoo. I have access to both the frozen_inference_graph.pb model file and checkpoint file.
I donot have access to the python code which is used to construct the model.
Thanks.
From inspecting the code, SSDMobileNetV1FeatureExtractor.extract_features redirects research.slim.nets:
from nets import mobilenet_v1 # nets will have to be on your PYTHONPATH
with tf.variable_scope('MobilenetV1',
reuse=self._reuse_weights) as scope:
with slim.arg_scope(
mobilenet_v1.mobilenet_v1_arg_scope(
is_training=None, regularize_depthwise=True)):
with (slim.arg_scope(self._conv_hyperparams_fn())
if self._override_base_feature_extractor_hyperparams
else context_manager.IdentityContextManager()):
_, image_features = mobilenet_v1.mobilenet_v1_base(
ops.pad_to_multiple(preprocessed_inputs, self._pad_to_multiple),
final_endpoint='Conv2d_13_pointwise',
min_depth=self._min_depth,
depth_multiplier=self._depth_multiplier,
use_explicit_padding=self._use_explicit_padding,
scope=scope)
The mobilenet_v1_base function takes a final_endpoint argument. Rather than prune the constructed graph, just construct the graph up until the endpoint you want.
There are many examples about how to do fine-tuning with tensorflow. Almost all these examples are try to resize our images to the specified size that the existing model needs. Like for example, 224×224 is the input size that vgg19 needs. However, in keras, we can change the input size by setting the include_top to false:
base_model = VGG19(include_top=False, weights="imagenet", input_shape=(input_size, input_size, input_channels))
Then we do not have to fix the image size to be 224×224 anymore. Can we do such kind of fine-tuning by using official pre-trained models in tensorflow? I cannot find the solutions up till now, anyone help me?
Yes, it is possible to do this kind of fine-tuning. You would just have to ensure that you also fine-tune some of the first few layers (to account for changed input) of the original network in addition to the last few layers (to account for changed output).
I work with TensorFlow using Keras. If you are open to that, then there is a code snippet that shows the general fine-tuning flow here:
https://keras.io/applications/
Specifically, I had to write the following code to make it work for my case:
#img_width,img_height is the size of your new input, 3 is the number of channels
input_tensor = Input(shape=(img_width, img_height, 3))
base_model =
keras.applications.vgg19.VGG19(include_top=False,weights='imagenet', input_tensor=input_tensor)
#instantiate whatever other layers you need
model = Model(inputs=base_model.inputs, outputs=predictions)
#predictions is the new logistic layer added to account for new classes
Hope this helps.