I'm trying to visualize the model in Tensorboard without training.
I checked this and that, but this still doesn't work even for the simplest model.
import tensorflow as tf
import tensorflow.keras as keras
# Both tf.__version__ tensorboard.__version__ are 2.5.0
s_model = keras.models.Sequential([
keras.layers.Flatten(input_shape=(28, 28)),
keras.layers.Dense(32, activation='relu'),
keras.layers.Dropout(0.2),
keras.layers.Dense(10, activation='softmax')
])
logdir = '.../logs'
_callbacks = keras.callbacks.TensorBoard(log_dir=logdir)
_callbacks.set_model(s_model) # This is exactly suggested in the link
When I did the above, I get the error message:
Graph visualization failed.
Error: Malformed GraphDef. This can sometimes be caused by a bad
network connection or difficulty reconciling mulitple GraphDefs; for
the latter case, please refer to
https://github.com/tensorflow/tensorboard/issues/1929.
I don't think this is a reconciliation problem because it is not a custom function, and if I compile the model, train, then I can get the graph visualization I wanted.
s_model.compile(
optimizer='adam',
loss='sparse_categorical_crossentropy',
metrics=['accuracy'])
(train_images, train_labels), _ = keras.datasets.fashion_mnist.load_data()
train_images = train_images / 255.0
logdir = '.../logs'
tensorboard_callback = keras.callbacks.TensorBoard(log_dir=logdir)
s_model.fit(
train_images,
train_labels,
batch_size=64,
epochs=5,
callbacks=[tensorboard_callback])
This gives the wanted graph visualization. But is there any other way to get graph visualization in Tensorboard without training?
Of course, I'm also aware that workaround, i.e. train with the tf.random.normal() for a while, would do the trick but I'm looking for the neat way like _callbacks.set_model(s_model)...
Related
Im using Kaggle to train my model. My model definition is as follows:
from tensorflow.keras.applications import EfficientNetB4
from tensorflow.keras.models import Model
base_model = EfficientNetB4(input_tensor=Input(shape=(IMG_HEIGHT, IMG_WIDTH, 3)),
weights='imagenet',
include_top=False,
pooling='avg'
)
x=base_model.output
output=Dense(1, activation='sigmoid')(x)
model=Model(inputs=base_model.input, outputs=output)
model.summary()
After fitting the model, I'm saving the model with this:
MODEL_DIR = "../working/tfx_model/"
version = "alpha"
export_path = os.path.join(MODEL_DIR, str(version))
print('export_path = {}\n'.format(export_path))
tf.keras.models.save_model(
model,
export_path,
overwrite=True,
include_optimizer=True,
save_format=None,
signatures=None,
options=None
)
print('\nSaved model:')
!ls -l {export_path}
Im simply importing the model to my local computer with the following:
model = load_model('models/tfx_model')
However the result of model.predict(input_image) returns an array of 0 or 1. However on Kaggle, I can have the confidence values.
I have already suspected a weird bug with EfficientNet implementation of Keras yet I haven't found anything on the subject. Also I tried saving .h5 and weights only and it had still the same issue.
The model is not over-fitted.
Kaggle environment versions:
'Tensorflow Version 2.9.2,Keras Version:2.9.0'
Local environment versions:
'Tensorflow Version 2.10.0,Keras Version:2.10.0'
It is probably due to different keras versions.
To get the probabilities try:
model.predict_proba(input_image)
After a certain keras version (I think 2.6), predict and predict_proba return probabilities but for previous versions predict returns 0 or 1
I am using optimizer.get_config() to get the final state of my adam optimizer (as in https://stackoverflow.com/a/60077159/607528) however .get_config() is returning the initial state. I assume this means one of the following
.get_config() is supposed to return the initial state
my optimizer is not updating because I've set something up wrong
my optimizer is not updating tf's adam is broken (highly unlikely)
my optimizer is updating but is being reset somewhere before I call .get_config()
something else?
Of course I originally noticed the issue in a proper project with training and validation sets etc, but here is a really simple snippet that seems to reproduce the issue:
import tensorflow as tf
import numpy as np
x=np.random.rand(100)
y=(x*3).round()
model = tf.keras.models.Sequential([
tf.keras.layers.Dense(128, activation='relu'),
tf.keras.layers.Dense(10, activation='softmax')
])
model.compile(optimizer='adam',
loss='sparse_categorical_crossentropy',
metrics=['accuracy'])
model.fit(x, y, epochs=500)
model.evaluate(x, y)
model.optimizer.get_config()
If you want to restore your training - you should save optimizer weights, not config:
weight_values = optimizer.get_weights()
with open(self.output_path+'optimizer.pkl', 'wb') as f:
pickle.dump(weight_values, f)
And then load them:
model.fit(dummy_x, dummy_y, epochs=500) # build optimizer by fitting model with dummy input - e.g. random tensors with simpliest shape
with open(self.path_to_saved_model+'optimizer.pkl', 'rb') as f:
weight_values = pickle.load(f)
optimizer.set_weights(weight_values)
I'm adding data augmentation to my tensorflow model like so:
data_augmentation = keras.Sequential([
layers.experimental.preprocessing.RandomRotation(factor=0.4, fill_mode="wrap"),
layers.experimental.preprocessing.RandomTranslation(height_factor=0.2, width_factor=0.2, fill_mode="wrap"),
layers.experimental.preprocessing.RandomFlip("horizontal"),
layers.experimental.preprocessing.RandomContrast(factor=0.2),
layers.experimental.preprocessing.RandomHeight(factor=0.2),
layers.experimental.preprocessing.RandomWidth(factor=0.2)
])
input_shape = (299, 299, 3)
inceptionV3_base = tf.keras.applications.InceptionV3(
input_shape=input_shape,
include_top=False,
weights='imagenet'
)
tf_model = tf.keras.Sequential([
tf.keras.layers.InputLayer(input_shape=input_shape),
data_augmentation,
inceptionV3_base,
tf.keras.layers.GlobalAveragePooling2D(),
tf.keras.layers.Dense(1024, activation='relu'),
tf.keras.layers.Dense(len(classes), activation='softmax')
])
Adding the data_augmentation layer to model slows down training by 13x. Am I using the keras preprocessing layers correctly?
I had the same issue - was missing ptxas (nvidia-cuda-toolkit package for me).
There are two ways of adding data augmentation:
1- Inside the model, just like the way you did.
2- Outside the model, and before training, using tf.data.Dataset.map()
Maybe trying option2 could make your model training faster. Try it!
More details here: https://keras.io/guides/preprocessing_layers/
There are many other ways of optimizing the training, such as the way you feed data into your model. Are you using tf.data.Dataset methods like cache() and prefetch(). More details can be found here: https://www.tensorflow.org/tutorials/load_data/images#configure_the_dataset_for_performance
I wish to apply different learning rates for each layer (as is done in Fastai) for Keras. All I have found in coming close to this is by modifying this line self.optimizer.apply_gradients(zip(gradients, trainable_vars)) in the keras code block found here by multiplying the gradients with its corresponding learning rate (and set the global learning rate to 1).
However, this method would only work with SGD and other simple optimizers as things such as momentum would distort this simple multiplication of the gradient.
Here is ideally how I would like to implement it:
# example data
(x_train, y_train), (x_test, y_test) = tf.keras.datasets.mnist.load_data()
x_train = x_train.reshape(-1, 784).astype("float32") / 255.0
x_test = x_test.reshape(-1, 784).astype("float32") / 255.0
lrs = [1e-2, 1e-1]
model = tf.keras.Sequential()
model.add(tf.keras.layers.Dense(20, input_dim=784, activation='relu'))
model.add(tf.keras.layers.Dense(1))
model.compile(
optimizer=tf.keras.optimizers.Adam(learning_rate=lrs), # DOESNT WORK
loss="mean_squared_error",
metrics=["mean_absolute_error"],
)
Any thoughts on how I could implement this. Another idea is to have the same number of optimizers as learning rates, and to update them in the train_step function inside the custom model.
This is possible using tensorflow addons now.
pip install tensorflow-addons
import tensorflow_addons as tfa
optimizers = [Adam(learning_rate=1e-2),
Adam(learning_rate=1e-1)]
#specifying the optimizers and layers in which it will operate
optimizers_and_layers = [
(optimizers[0], model.layers[0]),
(optimizers[1], model.layers[1]),
]
# Using Multi Optimizer from Tensorflow Addons
opt = tfa.optimizers.MultiOptimizer(optimizers_and_layers)
model.compile(
optimizer=opt,
loss="mean_squared_error",
metrics=["mean_absolute_error"],
)
WARNING:tensorflow:Sequential models without an input_shape passed to the first layer cannot reload their optimizer state. As a result, your model is starting with a freshly initialized optimizer.
while trying to load a saved model i encountered this warning from tensorflow
import tensorflow.keras as keras
import tensorflow as tf
mnist = tf.keras.datasets.mnist
(x_train, y_train),(x_test, y_test) = mnist.load_data()
x_train = tf.keras.utils.normalize(x_train, axis=1)
x_test = tf.keras.utils.normalize(x_test, axis=1)
model = tf.keras.models.Sequential()
model.add(tf.keras.layers.Flatten())
model.add(tf.keras.layers.Dense(128, activation=tf.nn.relu))
model.add(tf.keras.layers.Dense(128, activation=tf.nn.relu))
model.add(tf.keras.layers.Dense(10, activation=tf.nn.softmax))
model.compile(optimizer='adam',
loss='sparse_categorical_crossentropy',
metrics=['accuracy'])
model.fit(x_train, y_train, epochs=3)
model.save('epic_num_reader.model')
new_model = tf.keras.models.load_model('epic_num_reader.model')
predictions = new_model.predict(x_test)
Had the same problem after upgrading to TF 1.14, I fixed it changing the definition of the first layer from this:
model.add(tf.keras.layers.Flatten())
to this
model.add(tf.keras.layers.Flatten(input_shape=(28, 28)))
where 28 is the size of the input map to be flattened (mnist pixels in our case)
As the warning suggest, your first layer need the argument input_shape. In your case this would be the layer Flatten.
In the keras Documentation there is an extra section about the sequential API. See here for further information.
model.add(tf.keras.layers.Flatten(input_shape=(28, 28)))
for the first layer after tf 1.14 it is require to use input type which is the dimensions for the particular image.
Or you might get warning while retrieving model to not get proper working for your optimizer