If yes, then, how did they do that? I mean, say that I have a custom-built model via subclassing. My optimizer is a separated object. How is it that one command save weights of two different objects? In particular, how does it know that those two objects are related? Is it due to magic done by model.compile?
EDIT: I just realized that a model has an attribute model.optimizer, is that how Keras do it? make the optimizer an attribute of a model and will be saved with it?
No, model.save_weights() only saves the model's parameters. The state of the stuffs that the model was compiled with (optimizers, callbacks, losses, metrics) will not be saved.
You should use model.save() to save the model's optimizer and other training configurations. Please refer to this documentation (the most straigtforward way to save the optimizer together with the model).
If you for some reason want to use exactly model.save_weights(), please refer to this stackoverflow question on how to save model's optimizer (might be a bit tricky).
Related
Supposing I have a Keras model which is already trained. When using predict() method, I want to get the instance key value and corresponding prediction at the same time( I can pass key value as a feature/column in the input).
I wonder is it realistic to do that?
I struggled with this for a while. I'm using the tf.data.Dataset infrastructure so my first approach was to see if I could ensure that the order of the examples produced by the datasets was deterministic, but that wasn't optimal because it gave up a bunch of the parallel processing performance benefits and ended up not being the case in any event. I ended up processing predictions using model.predict_on_batch feeding in batches iterated out of the dataset manually instead of feeding the entire dataset into model.predict. That way I was able to grab the ids from the batch and associate them with the returned prediction.
I was surprised there wasn't a more ready made solution to a problem that must come up a lot. I haven't gotten up to speed on the Estimator interface or custom training/prediction loops yet, but hopefully this problem becomes trivial there.
There's a fairly clear difference between a model and a frozen model. As described in model_files, relevant part: Freezing
...so there's the freeze_graph.py script that takes a graph definition and a set of checkpoints and freezes them together into a single file.
Is a "saved_model" most similar to a "frozen_model" (and not a saved
"model_checkpoint")?
Is this defined somewhere in docs I'm missing?
In prior versions of tensorflow we would save and restore model
weights, but this seems to be in context of a "model_checkpoint" not
a "saved_model", is that still correct?
I'm asking more for the design overview here, not implementation specifics.
Checkpoint file only contains variables for specific model and should be loaded with either exactly same, predefined graph or with specific assignment_map to load only chosen variables. See https://www.tensorflow.org/api_docs/python/tf/train/init_from_checkpoint
Saved model is more broad cause it contains graph that can be loaded within a session and training could be continued. Frozen graph, however, is serialized and could not be used to continue training.
You can find all the info here https://www.tensorflow.org/guide/saved_model
from Tensorflow's documentation, there seems to be a large array of options for "running", serving, testing, and predicting using a Tensorflow model. I've made a model very similar to MNIST, where it outputs a distribution from an image. For a beginner, what would be the easiest way to take one or a few images, and send them through the model, getting an output prediction? It is mostly for experimentation purposes. Sorry if this is too redundant, but all my research has led me to so many different ways of doing this and the documentation doesn't really give any info on the pros and cons of the different methods. Thanks
I guess you are using placeholders for your model input and then using feed_dict to feed values into your model.
If that's the case the simplest way would be after you have a trained model you save it using tf.saver. Then you can have a test script where you restore your model and then sess.run on your output variable with a feed_dict of whatever you want your input to be.
Assume we have test/val/train splits. During training, we want to save some model checkpoints [save_1] that can be used to restart the training later.
In addition, we want to save another model during training that shows the best performance on the validation sets [save_2]. After done with training, we use save_2 to report the performance on the test data.
My question is that how we can have two different tf.savers during training in TensorFlow? whatever examples that I have seen, only save [save_1].
Pointer to any codes would be appreciated.
Thanks.
You can get quite close by using an Estimator to wrap your model. Specifically see the options surrounding saving multiple checkpoints in RunConfig (you don't have to throw any away). Could be combined with a ValidationMonitor to find the lowest validation error.
For operations in Tensorflow, we have the option to pick a name.
Example:
tf.argmin(input, dimension, name=None)
What does this do? Does it help with debugging? If so, how?
Defining a name for ops and vars helps you to build a logically correct graph.
You can visualize, then, you graph in Tensorboard and see if everything you defined is exactly as you thought.
In general, giving a name to a variable or an op is a good practice. Further, when you export a graph and you re-use it somewhere, it's unhandy to use the default generated names by tensorflow to interact with the graph. You'll surely prefer to work with name with a sense.
Think about something like BatchNorm/relu:0 vs BatchNorm/network_output:0. The latter is more clear and describes exactly what you meant when defined that operation