I am trying to train a model from scratch ,in that case what should I put in fine_tune_checkpoint value in Protobuf file.
Nothing. It should initialize your entire model.
If for some reason that doesn't work for you, you can give it a checkpoint of something else entirely. The API builds the graph of the model, and compares between its variables to the variables in the checkpoint (both names and shapes), and every variable it doesn't find - it initializes.
Related
There's a fairly clear difference between a model and a frozen model. As described in model_files, relevant part: Freezing
...so there's the freeze_graph.py script that takes a graph definition and a set of checkpoints and freezes them together into a single file.
Is a "saved_model" most similar to a "frozen_model" (and not a saved
"model_checkpoint")?
Is this defined somewhere in docs I'm missing?
In prior versions of tensorflow we would save and restore model
weights, but this seems to be in context of a "model_checkpoint" not
a "saved_model", is that still correct?
I'm asking more for the design overview here, not implementation specifics.
Checkpoint file only contains variables for specific model and should be loaded with either exactly same, predefined graph or with specific assignment_map to load only chosen variables. See https://www.tensorflow.org/api_docs/python/tf/train/init_from_checkpoint
Saved model is more broad cause it contains graph that can be loaded within a session and training could be continued. Frozen graph, however, is serialized and could not be used to continue training.
You can find all the info here https://www.tensorflow.org/guide/saved_model
I have been trying to retrain the universal sentence encoder module (use) but I cannot seem to generate new embeddings from any updated/retrained module weights.
I have followed the classifier example for elmo and replaced that with the use module. The classifier works I cannot seem to figure out how to get access to the new module embeddings. I can restore the saved model but not sure how i actually call the embedding function?
For example, previously I set the path via
m=hub.module(path) then generate the embeddings via
m(sentences)
But how do I make this call with a restored classifier model? Note that I do set the parameter to trainable=True as in the example tutorial.
Do I simply make the same call and it automatically picks up the new weights that have been trained via the classifier retraining?
I have checked for similar questions but most of them are related to manipulating the meta graph. I just want to simply use the new weights to generate some new embeddings to see if they are different having being fine tuned on other data.
Maybe this isnt possible?
Any clarification would be appreciated.
So I actually ran into this problem when I was running distributed tensorflow and the loss was not changing.
The trick is that when you first download it you need to set trainable = True otherwise the weights will be locked.
So in your case, redownload the module with trainable = True to unlock the weights.
I borrowed everything from TF/Model_Zoo (the config files and the model files as is), and started the training by using my images and annotations without changing the config file other than the paths.
In the original config files, there is no detail the like:
}
freeze_variables: ".*FeatureExtractor."
}
I believe that means everything is getting retrained, is this true? When I look into the tensor-board, it looks like the weights are not changing but the biases are changing. How can I make sure that the weights are also trained?
Also, I would like to retrain only the last 2 (or last n layers) layer, just like regular transfer learning. Is this possible with TF Object Detection API?
I've successfully trained the inception v3 model on custom 200 classes from scratch. Now I have ckpt files in my output dir. How to use those models to run inference?
Preferably, load the model on GPU and pass images whenever I want while the model persists on GPU. Using TensorFlow serving is not an option for me.
Note: I've tried to freeze these models but failed to correctly put output_nodes while freezing. Used ImagenetV3/Predictions/Softmax but couldn't use it with feed_dict as I couldn't get required tensors from freezed model.
There is poor documentation on TF site & repo on this inference part.
It sounds like you're on the right track, you don't really do anything different at inference time as you do at training time except that you don't ask it to compute the optimizer at inference time, and by not doing so, no weights are ever updated.
The save and restore guide in tensorflow documentation explains how to restore a model from checkpoint:
https://www.tensorflow.org/programmers_guide/saved_model
You have two options when restoring a model, either you build the OPS again from code (usually a build_graph() method) then load the variables in from the checkpoint, I use this method most commonly. Or you can load the graph definition & variables in from the checkpoint if the graph definition was saved with the checkpoint.
Once you've loaded the graph you'll create a session and ask the graph to compute just the output. The tensor ImagenetV3/Predictions/Softmax looks right to me (I'm not immediately familiar with the particular model you're working with). You will need to pass in the appropriate inputs, your images, and possibly whatever parameters the graph requires, sometimes an is_train boolean is needed, and other such details.
Since you aren't asking tensorflow to compute the optimizer operation no weights will be updated. There's really no difference between training and inference other than what operations you request the graph to compute.
Tensorflow will use the GPU by default just as it did with training, so all of that is pretty much handled behind the scenes for you.
I have used the TensorFlow object detection API to train the SSD Inception model from scratch. The evaluation script shows that the model has learned something and now I want to use the model.
I have looked at the object detection ipynb that can feed single images to a trained model. However, this is for SSD with MobileNet. I have used the following line (after loading the meta graph) to print the tensor names of the TensorFlow model I trained.
print([str(op.name) for op in tf.get_default_graph().get_operations()] )
But it does not contain the same input or output tensor names as in the ipynb. I have also searched through the code, but many functions point toward each other and it is difficult to find what I am looking for.
How can I find the tensor names I need? Or is there another method I do not know about?
To use the graph, you need to freeze/export it, using this provided script. The resulting .pb file will contain the nodes you need. I don't know why it's organized like that, but it is.