I've created a model based on VGG16:
base_model = VGG16(weights='imagenet', include_top=False,
input_tensor=next_batch["image"],
input_shape=INPUT_SHAPE)
x = base_model.output
# model customization follows, not relevant
Note that in the snippet I specify input_tensor which comes from tf.data.Dataset.
I've trained the model for some epochs and then saved it using model.save("model.h5")
Now, when I try to load the model using load_model("model.h5") and continue training it, I get:
Traceback (most recent call last):
tensorflow.python.framework.errors_impl.InvalidArgumentError: You must
feed a value for placeholder tensor 'input_1' with dtype float and
shape [?,512,512,3]
[[{{node input_1}} = Placeholderdtype=DT_FLOAT, shape=[?,512,512,3],
_device="/job:localhost/replica:0/task:0/device:GPU:0"]]
So the question is, how to specify input_tensor on a loaded model?
After reading through the code responsible for model loading, I came up with the following solution:
class InputLayerFix:
#staticmethod
def from_config(config):
return InputLayer(input_tensor=next_batch["image"],
dtype=config["dtype"],
name=config["name"],
batch_input_shape=config["batch_input_shape"])
custom_objects = {"InputLayer": InputLayerFix}
model.load_model("model.h5", custom_objects=custom_objects)
This code relies on the fact that the deserialization routine first checks custom_objects for classes to instantiate (https://github.com/tensorflow/tensorflow/blob/r1.12/tensorflow/python/keras/utils/generic_utils.py#L155)
Related
Setting
As already mentioned in the title, I got a problem with my custom loss function, when trying to load the saved model. My loss looks as follows:
def weighted_cross_entropy(weights):
weights = K.variable(weights)
def loss(y_true, y_pred):
y_pred = K.clip(y_pred, K.epsilon(), 1-K.epsilon())
loss = y_true * K.log(y_pred) * weights
loss = -K.sum(loss, -1)
return loss
return loss
weighted_loss = weighted_cross_entropy([0.1,0.9])
So during training, I used the weighted_loss function as loss function and everything worked well. When training is finished I save the model as .h5file with the standard model.save function from keras API.
Problem
When I am trying to load the model via
model = load_model(path,custom_objects={"weighted_loss":weighted_loss})
I am getting a ValueError telling me that the loss is unknown.
Error
The error message looks as follows:
File "...\predict.py", line 29, in my_script
"weighted_loss": weighted_loss})
File "...\Continuum\anaconda3\envs\processing\lib\site-packages\keras\engine\saving.py", line 419, in load_model
model = _deserialize_model(f, custom_objects, compile)
File "...\Continuum\anaconda3\envs\processing\lib\site-packages\keras\engine\saving.py", line 312, in _deserialize_model
sample_weight_mode=sample_weight_mode)
File "...\Continuum\anaconda3\envs\processing\lib\site-packages\keras\engine\training.py", line 139, in compile
loss_function = losses.get(loss)
File "...\Continuum\anaconda3\envs\processing\lib\site-packages\keras\losses.py", line 133, in get
return deserialize(identifier)
File "...\Continuum\anaconda3\envs\processing\lib\site-packages\keras\losses.py", line 114, in deserialize
printable_module_name='loss function')
File "...\Continuum\anaconda3\envs\processing\lib\site-packages\keras\utils\generic_utils.py", line 165, in deserialize_keras_object
':' + function_name)
ValueError: Unknown loss function:loss
Questions
How can I fix this problem? May it be possible that the reason for that is my wrapped loss definition? So keras doesn't know, how to handle the weights variable?
Your loss function's name is loss (i.e. def loss(y_true, y_pred):). Therefore, when loading back the model you need to specify 'loss' as its name:
model = load_model(path, custom_objects={'loss': weighted_loss})
For full examples demonstrating saving and loading Keras models with custom loss functions or models, please have a look at the following GitHub gist files:
Custom loss function defined using a wrapper:
https://gist.github.com/ashkan-abbasi66/a81fe4c4d588e2c187180d5bae734fde
Custom loss function defined by subclassing:
https://gist.github.com/ashkan-abbasi66/327efe2dffcf9788847d26de934ef7bd
Custom model:
https://gist.github.com/ashkan-abbasi66/d5a525d33600b220fa7b095f7762cb5b
Note:
I tested the above examples on Python 3.8 with Tensorflow 2.5.
I am working on tensorflow federated (tff). The problem arises when I call the iterative process and pass on the instance of the model created.
I have declared a keras model which is used at both the server and at the client. I basically want to modify the client model's weights and biases in order to perform mathematical functions on them. These mathematical functions are performed on the models' weights and biases by using function named 'processed'.
The processed function can return weights and layers both in tf or numpy format (the funtion 'processed' just uses tf.convert_to_tensor command to convert the numpy returned weights/biases from the get/set_weights command used)
def create_keras_model():
return tf.keras.models.Sequential([
tf.keras.layers.Conv2D(filters=32, kernel_size=[5, 5],name='conv2d_1',activation=tf.nn.relu, use_bias=True, bias_initializer =tf.initializers.lecun_normal(seed=137), input_shape=(28 ,28 ,1 )),
tf.keras.layers.MaxPool2D(pool_size=[2, 2], strides=2),
tf.keras.layers.Conv2D(filters=32, kernel_size=[5 5],name='conv2d_2',activation=tf.nn.relu, use_bias = True, bias_initializer=tf.initializers.lecun_normal(seed=137)),
tf.keras.layers.MaxPool2D(pool_size=[2 2], strides=2),
tf.keras.layers.Reshape(target_shape=(4 * 3 * 3)),
tf.keras.layers.Dense(units= 50, activation=tf.nn.relu, use_bias=True, bias_initializer=tf.initializers.lecun_normal(seed=137), name='dense_1'),
tf.keras.layers.Dense(units=10 , use_bias=True, bias_initializer=tf.initializers.lecun_normal(seed=137), activation=tf.nn.softmax, name='dense_2' ),
])
I create an instant of the above Keras model named as
net_1 = create_keras_model()
Now I am accessing the weights and biases by using the following function (proccessed) and assigning them new weights and biases after some mathmatical operations
def processed(net_1,L_norm):
a=net_1.get_layer('conv2d_1').get_weights()[0]
b=net_1.get_layer('conv2d_1').get_weights()[1]
c= net_1.get_layer('conv2d_2').get_weights()[0]
d=net_1.get_layer('conv2d_2').get_weights()[1]
e= net_1.get_layer('dense_1').get_weights()[0]
f= net_1.get_layer('dense_1').get_weights()[1]
g=net_1.get_layer('dense_2').get_weights()[0]
h= net_1.get_layer('dense_2').get_weights()[1]
L1,B1,L2,B2,L3,B3,L4,B4 = processing_work(a,b,c,d,e,f,g,h,L_norm)
L1=tf.convert_to_tensor(L1)
B1=tf.convert_to_tensor(B1)
L2=tf.convert_to_tensor(L2)
B2=tf.convert_to_tensor(B2)
L3=tf.convert_to_tensor(L3)
B3=tf.convert_to_tensor(B3)
L4=tf.convert_to_tensor(L4)
B4=tf.convert_to_tensor(B4)
net_1.get_layer('conv2d_1').set_weights([L1,B1])
net_1.get_layer('conv2d_2').set_weights([L2,B2])
net_1.get_layer('dense_1').set_weights([L3,B3])
net_1.get_layer('dense_2').set_weights([L4,B4])
return net_1
Following that, I use the following function
def model_fn_of():
# We _must_ create a new model here, and _not_ capture it from an external
# scope. TFF will call this within different graph contexts.
local_model = processed(create_keras_model(),L_norm)
return tff.learning.from_keras_model(
local_model,
input_spec=preprocessed_example_dataset.element_spec,
loss=tf.keras.losses.SparseCategoricalCrossentropy(),
metrics=[tf.keras.metrics.SparseCategoricalAccuracy()])
Then I call the iterator of tff.
iterative_process = tff.learning.algorithms.build_weighted_fed_avg(
model_fn_of,
client_optimizer_fn=lambda: tf.keras.optimizers.SGD(learning_rate=0.02),
server_optimizer_fn=lambda: tf.keras.optimizers.SGD(learning_rate=1.00))
However I get the following error:
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
<ipython-input-32-651e903cc722> in <module>
2 model_fn_of,
3 client_optimizer_fn=lambda: tf.keras.optimizers.SGD(learning_rate=0.02),
----> 4 server_optimizer_fn=lambda: tf.keras.optimizers.SGD(learning_rate=1.00))
8 frames
/usr/local/lib/python3.7/dist-packages/tensorflow/python/framework/ops.py in __getattr__(self, name)
511 from tensorflow.python.ops.numpy_ops import np_config
512 np_config.enable_numpy_behavior()""".format(type(self).__name__, name))
--> 513 self.__getattribute__(name)
514
515 #staticmethod
AttributeError: 'Tensor' object has no attribute 'numpy'
Can someone please help me with this. I dont know what is wrong and from which part of the code the error is even popping out. Please note that the dtype of weights/biases of model layers before and after using the 'processed' function are same.
Thanks in Advance
Setting
As already mentioned in the title, I got a problem with my custom loss function, when trying to load the saved model. My loss looks as follows:
def weighted_cross_entropy(weights):
weights = K.variable(weights)
def loss(y_true, y_pred):
y_pred = K.clip(y_pred, K.epsilon(), 1-K.epsilon())
loss = y_true * K.log(y_pred) * weights
loss = -K.sum(loss, -1)
return loss
return loss
weighted_loss = weighted_cross_entropy([0.1,0.9])
So during training, I used the weighted_loss function as loss function and everything worked well. When training is finished I save the model as .h5file with the standard model.save function from keras API.
Problem
When I am trying to load the model via
model = load_model(path,custom_objects={"weighted_loss":weighted_loss})
I am getting a ValueError telling me that the loss is unknown.
Error
The error message looks as follows:
File "...\predict.py", line 29, in my_script
"weighted_loss": weighted_loss})
File "...\Continuum\anaconda3\envs\processing\lib\site-packages\keras\engine\saving.py", line 419, in load_model
model = _deserialize_model(f, custom_objects, compile)
File "...\Continuum\anaconda3\envs\processing\lib\site-packages\keras\engine\saving.py", line 312, in _deserialize_model
sample_weight_mode=sample_weight_mode)
File "...\Continuum\anaconda3\envs\processing\lib\site-packages\keras\engine\training.py", line 139, in compile
loss_function = losses.get(loss)
File "...\Continuum\anaconda3\envs\processing\lib\site-packages\keras\losses.py", line 133, in get
return deserialize(identifier)
File "...\Continuum\anaconda3\envs\processing\lib\site-packages\keras\losses.py", line 114, in deserialize
printable_module_name='loss function')
File "...\Continuum\anaconda3\envs\processing\lib\site-packages\keras\utils\generic_utils.py", line 165, in deserialize_keras_object
':' + function_name)
ValueError: Unknown loss function:loss
Questions
How can I fix this problem? May it be possible that the reason for that is my wrapped loss definition? So keras doesn't know, how to handle the weights variable?
Your loss function's name is loss (i.e. def loss(y_true, y_pred):). Therefore, when loading back the model you need to specify 'loss' as its name:
model = load_model(path, custom_objects={'loss': weighted_loss})
For full examples demonstrating saving and loading Keras models with custom loss functions or models, please have a look at the following GitHub gist files:
Custom loss function defined using a wrapper:
https://gist.github.com/ashkan-abbasi66/a81fe4c4d588e2c187180d5bae734fde
Custom loss function defined by subclassing:
https://gist.github.com/ashkan-abbasi66/327efe2dffcf9788847d26de934ef7bd
Custom model:
https://gist.github.com/ashkan-abbasi66/d5a525d33600b220fa7b095f7762cb5b
Note:
I tested the above examples on Python 3.8 with Tensorflow 2.5.
I have tried all the options described in the documentation but none of them allowed me to save my model in tensorflow 2.0.0 beta1. I've also tried to upgrade to the (also unstable) TF2-RC but that ruined even the code I had working in beta so I quickly rolled back for now to beta.
See a minimal reproduction code below.
What I have tried:
model.save("mymodel.h5")
NotImplementedError: Saving the model to HDF5 format requires the
model to be a Functional model or a Sequential model. It does not work
for subclassed models, because such models are defined via the body of
a Python method, which isn't safely serializable. Consider saving to
the Tensorflow SavedModel format (by setting save_format="tf") or
using save_weights.
model.save("mymodel", format='tf')
ValueError: Model <main.CVAE object at 0x7f1cac2e7c50> cannot be
saved because the input shapes have not been set. Usually, input
shapes are automatically determined from calling .fit() or .predict().
To manually set the shapes, call model._set_inputs(inputs).
3.
model._set_input(input_sample)
model.save("mymodel", format='tf')
AssertionError: tf.saved_model.save is not supported inside a traced
#tf.function. Move the call to the outer eagerly-executed context.
And this is where I am stuck now because it gives me no reasonable hint whatsoever. That's because I am NOT calling the save() function from a #tf.function, I'm already calling it from the outermost scope possible. In fact, I have no #tf.function at all in this minimal reproduction script below and still getting the same error.
So I really have no idea how to save my model, I've tried every options and they all throw errors and provide no hints.
The minimal reproduction example below works fine if you set save_model=False and it reproduces the error when save_model=True.
It may seem unnecessary in this simplified auto-encoder code example to use a subclassed model but I have lots of custom functions added to it in my original VAE code that I need it for.
Code:
import tensorflow as tf
save_model = True
learning_rate = 1e-4
BATCH_SIZE = 100
TEST_BATCH_SIZE = 10
color_channels = 1
imsize = 28
(train_images, _), (test_images, _) = tf.keras.datasets.mnist.load_data()
train_images = train_images[:5000, ::]
test_images = train_images[:1000, ::]
train_images = train_images.reshape(-1, imsize, imsize, 1).astype('float32')
test_images = test_images.reshape(-1, imsize, imsize, 1).astype('float32')
train_images /= 255.
test_images /= 255.
train_dataset = tf.data.Dataset.from_tensor_slices(train_images).batch(BATCH_SIZE)
test_dataset = tf.data.Dataset.from_tensor_slices(test_images).batch(TEST_BATCH_SIZE)
class AE(tf.keras.Model):
def __init__(self):
super(AE, self).__init__()
self.network = tf.keras.Sequential([
tf.keras.layers.InputLayer(input_shape=(imsize, imsize, color_channels)),
tf.keras.layers.Flatten(),
tf.keras.layers.Dense(50),
tf.keras.layers.Dense(imsize**2 * color_channels),
tf.keras.layers.Reshape(target_shape=(imsize, imsize, color_channels)),
])
def decode(self, input):
logits = self.network(input)
return logits
optimizer = tf.keras.optimizers.Adam(learning_rate)
model = AE()
def compute_loss(data):
logits = model.decode(data)
loss = tf.reduce_mean(tf.losses.mean_squared_error(logits, data))
return loss
def train_step(data):
with tf.GradientTape() as tape:
loss = compute_loss(data)
gradients = tape.gradient(loss, model.trainable_variables)
optimizer.apply_gradients(zip(gradients, model.trainable_variables))
return loss, 0
def test_step(data):
loss = compute_loss(data)
return loss
input_shape_set = False
epoch = 0
epochs = 20
for epoch in range(epochs):
for train_x in train_dataset:
train_step(train_x)
if epoch % 1 == 0:
loss = 0.0
num_batches = 0
for test_x in test_dataset:
loss += test_step(test_x)
num_batches += 1
loss /= num_batches
print("Epoch: {}, Loss: {}".format(epoch, loss))
if save_model:
print("Saving model...")
if not input_shape_set:
# Note: Why set input shape manually and why here:
# 1. If I do not set input shape manually: ValueError: Model <main.CVAE object at 0x7f1cac2e7c50> cannot be saved because the input shapes have not been set. Usually, input shapes are automatically determined from calling .fit() or .predict(). To manually set the shapes, call model._set_inputs(inputs).
# 2. If I set input shape manually BEFORE the first actual train step, I get: RuntimeError: Attempting to capture an EagerTensor without building a function.
model._set_inputs(train_dataset.__iter__().next())
input_shape_set = True
# Note: Why choose tf format: model.save('MNIST/Models/model.h5') will return NotImplementedError: Saving the model to HDF5 format requires the model to be a Functional model or a Sequential model. It does not work for subclassed models, because such models are defined via the body of a Python method, which isn't safely serializable. Consider saving to the Tensorflow SavedModel format (by setting save_format="tf") or using save_weights.
model.save('MNIST/Models/model', save_format='tf')
I have tried the same minimal reproduction example in tensorflow-gpu 2.0.0-rc0 and the error was more revealing than what the beta version gave me. The error in RC says:
NotImplementedError: When subclassing the Model class, you should
implement a call method.
This got me read through https://www.tensorflow.org/beta/guide/keras/custom_layers_and_models where I found examples of how to do subclassing in TF2 in a way that allows saving. I was able to resolve the error and have the model saved by replacing my 'decode' method by 'call' in the above example (although this will be more complicated with my actual code where I had various methods defined for the class). This solved the error both in beta and in rc. Strangely, the training (or the saving) got also much faster in rc.
You should change two things:
Change the decode method to call, as you pointed out
As your model is of type Sequential, and not built inside the class, you want to call the save method on the self.network attribute of the model, i.e.,
model.network.save('mymodel.h5')
alternatively, to keep things more standard, you can implement this method inside the AE class, as follows:
def save(self, save_dir):
self.network.save(save_dir)
Cheers mate
I am implementing a custom layer in Keras. If I print the shape of the input passed to the call method, I get None as the first element. Why is that? Shouldn't the first element be the batch size?
def call(self, x):
print(x.shape) # (None, ...)
When I call model.fit, I am passing the batch size
batch_size = 50
model.fit(x_train, y_train, ..., batch_size=batch_size)
So, when is the method call actually called? And what is the recommended way of getting the batch size in the method call?
None means it is a dynamic shape. It can take any value depending on the batch size you choose.
When you define a model by default it is defined to support any batch size you can choose. This is what the None means. In TensorFlow 1.* the input to your model is an instance of tf.placeholder().
If you don't use the keras.InputLayer() with specified batch size you get the first dimension None by default:
import tensorflow as tf
model = tf.keras.models.Sequential()
model.add(tf.keras.layers.Dense(units=2, input_shape=(2, )))
print(model.inputs[0].get_shape().as_list()) # [None, 2]
print(model.inputs[0].op.type == 'Placeholder') # True
When you do use keras.InputLayer() with specified batch size you can define the input placeholder with fixed batch size:
import tensorflow as tf
model = tf.keras.models.Sequential()
model.add(tf.keras.layers.InputLayer((2,), batch_size=50))
model.add(tf.keras.layers.Dense(units=2, input_shape=(2, )))
print(model.inputs[0].get_shape().as_list()) # [50, 2]
print(model.inputs[0].op.type == 'Placeholder') # True
When you specify the batch size to the model.fit() method these input placeholders have already been defined and you cannot modify their shape. The batch size for model.fit() is used only to split the data you provided to batches.
If you define your input layer with batch size 2 and then you pass different value of a batch size to the model.fit() method you will get ValueError:
import tensorflow as tf
import numpy as np
model = tf.keras.models.Sequential()
model.add(tf.keras.layers.InputLayer((2,), batch_size=2)) # <--batch_size==2
model.add(tf.keras.layers.Dense(units=2, input_shape=(2, )))
model.compile(optimizer=tf.keras.optimizers.Adam(),
loss='categorical_crossentropy')
x_train = np.random.normal(size=(10, 2))
y_train = np.array([[0, 1] for _ in range(10)])
model.fit(x_train, y_train, batch_size=3) # <--batch_size==3
This will raise:
ValueError: Thebatch_sizeargument value 3 is incompatible with the specified batch size of your Input Layer: 2
I faced the same issue and found that using tf.shape(your_variable) instead of your_variable.shape solved the problem. As tf.shape(your_variable) is dynamically evaluated later when the fit function is called.
reference
https://github.com/tensorflow/tensorflow/issues/36991#issuecomment-590448880
To get the value in integers consider output_shape variable.
Minimum working example
import tensorflow as tf
model = tf.keras.models.Sequential()
model.add(tf.keras.layers.InputLayer((2,), batch_size=50))
model.add(tf.keras.layers.Dense(units=2, input_shape=(2, )))
print(model.output_shape)
print(type(model.output_shape[0]), type(model.output_shape[1]))
Output:
(50, 2)
<class 'int'> <class 'int'>