I am trying to have a TensorFlow model that joins the representation and the classification parts separately. However, my classifier uses two inputs that come from the same representation network and the time I coded this architecture I received the error: "The list of inputs passed to the model is redundant. All inputs should only appear once.". This is the code I have.
import tensorflow as tf
def representation():
inp = tf.keras.Input(shape=[100, 300])
x = tf.keras.layers.Conv1D(300, 10, 1, 'same')(inp)
output = tf.keras.layers.Conv1D(300, 10, 1, 'same')(x)
return tf.keras.Model(inp, output)
def classifier():
inp1 = tf.keras.Input(shape=[100, 300])
inp2 = tf.keras.Input(shape=[100, 300])
output = tf.keras.layers.Activation('sigmoid')(inp1+inp2)
return tf.keras.Model([inp1,inp2], output)
repre = representation()
cla = classifier()
model = tf.keras.Model([repre.input, repre.input], cla([repre.output, repre.output]))
I used two representations like tf.keras.Model([repre1.input, repre2.input], cla([repre1.output, repre2.output])) and it works but, the representations come from different networks.
The call to tf.keras.Input in the representation() function defines a node on the neural network's computational graph. This node represents a single input that's tied to a couple of 1D convolutional layers. Using it for two inputs at the same time doesn't make sense as there is only one path from input to output in your model definition above.
Keras uses different syntax for defining models that used shared weights and multiple inputs. This page has good examples, and the most relevant detail in your case is "to share a layer in the functional API, call the same layer instance multiple times". In your case, you want to amend your code to actually define two input nodes and pretend like your representation model is a layer.
class RepresentationLayer(tf.keras.layers.Layer):
def __init__(self):
super(RepresentationLayer, self).__init__()
self.conv1 = tf.keras.layers.Conv1D(300, 10, 1, 'same')
self.conv2 = tf.keras.layers.Conv1D(300, 10, 1, 'same')
def __call__(self, inputs):
x = self.conv1(inputs)
x = self.conv2(x)
return x
def classifier_shared_representation():
inp1 = tf.keras.Input(shape=[100, 300])
inp2 = tf.keras.Input(shape=[100, 300])
rep_layer = RepresentationLayer()
rep1 = rep_layer(inp1)
rep2 = rep_layer(inp2)
output = tf.keras.layers.Activation('sigmoid')(rep1+rep2)
return tf.keras.Model(inputs=[inp1, inp2], outputs=[output])
model = classifier_shared_representation()
In the code you posted, you specify two outputs from your classification layer:
model = tf.keras.Model([repre.input, repre.input], cla([repre.output, repre.output]))
To do this in the code above, just change the tf.keras.Model call to
tf.keras.Model(inputs=[inp1, inp2], outputs=[output, output])
I'm still not sure if you want multiple outputs. It seems like your classifier() definition is just summing the representation outputs and passing them through an activation, resulting in a single output. If you do want a shared classification layer with multiple outputs, the process is the same as above.
Related
Problem
As the title suggests I have been trying to create a pipeline for training an Autoencoder model using TFX. The problem I'm having is fitting the tf.Dataset returned by the DataAccessor.tf_dataset_factory object to the Autoencoder.
Below I summarise the steps I've taken through this project, and have some Questions at the bottom if you wish to skip the background information.
Intro
TFX Pipeline
The TFX components I have used so far have been:
CsvExampleGenerator (the dataset has 82 columns, all numeric, and the sample csv has 739 rows)
StatisticsGenerator / SchemaGenerator, the schema has been edited as is now loaded in using an Importer
Transform
Trainer (this is the component I am currently having problems with)
Model
The model that I am attempting to train is based off of the example laid out here https://www.tensorflow.org/tutorials/generative/autoencoder. However, my model is being trained on tabular data, searching for anomalous results, as opposed to image data.
As I have tried a couple of solutions I have tried using both the Keras.layers and Keras.model format for defining the model and I outline both below:
Subclassing Keras.Model
class Autoencoder(keras.models.Model):
def __init__(self, features):
super(Autoencoder, self).__init__()
self.encoder = tf.keras.Sequential([
keras.layers.Dense(82, activation = 'relu'),
keras.layers.Dense(32, activation = 'relu'),
keras.layers.Dense(16, activation = 'relu'),
keras.layers.Dense(8, activation = 'relu')
])
self.decoder = tf.keras.Sequential([
keras.layers.Dense(16, activation = 'relu'),
keras.layers.Dense(32, activation = 'relu'),
keras.layers.Dense(len(features), activation = 'sigmoid')
])
def call(self, x):
inputs = [keras.layers.Input(shape = (1,), name = f) for f in features]
dense = keras.layers.concatenate(inputs)
encoded = self.encoder(dense)
decoded = self.decoder(encoded)
return decoded
Subclassing Keras.Layers
def _build_keras_model(features: List[str]) -> tf.keras.Model:
inputs = [keras.layers.Input(shape = (1,), name = f) for f in features]
dense = keras.layers.concatenate(inputs)
dense = keras.layers.Dense(32, activation = 'relu')(dense)
dense = keras.layers.Dense(16, activation = 'relu')(dense)
dense = keras.layers.Dense(8, activation = 'relu')(dense)
dense = keras.layers.Dense(16, activation = 'relu')(dense)
dense = keras.layers.Dense(32, activation = 'relu')(dense)
outputs = keras.layers.Dense(len(features), activation = 'sigmoid')(dense)
model = keras.Model(inputs = inputs, outputs = outputs)
model.compile(
optimizer = 'adam',
loss = 'mae'
)
return model
TFX Trainer Component
For creating the Trainer Component I have been mainly following the implementation details laid out here: https://www.tensorflow.org/tfx/guide/trainer
As well as following the default penguins example: https://www.tensorflow.org/tfx/tutorials/tfx/penguin_simple#write_model_training_code
run_fn defintion
def run_fn(fn_args: tfx.components.FnArgs) -> None:
tft_output = tft.TFTransformOutput(fn_args.transform_output)
train_dataset = _input_fn(
file_pattern = fn_args.train_files,
data_accessor = fn_args.data_accessor,
tf_transform_output = tft_output,
batch_size = fn_args.train_steps
)
eval_dataset = _input_fn(
file_pattern = fn_args.eval_files,
data_accessor = fn_args.data_accessor,
tf_transform_output = tft_output,
batch_size = fn_args.custom_config['eval_batch_size']
)
# model = Autoencoder(
# features = fn_args.custom_config['features']
# )
model = _build_keras_model(features = fn_args.custom_config['features'])
model.compile(optimizer = 'adam', loss = 'mse')
model.fit(
train_dataset,
steps_per_epoch = fn_args.train_steps,
validation_data = eval_dataset,
validation_steps = fn_args.eval_steps
)
...
_input_fn definition
def _apply_preprocessing(raw_features, tft_layer):
transformed_features = tft_layer(raw_features)
return transformed_features
def _input_fn(
file_pattern,
data_accessor: tfx.components.DataAccessor,
tf_transform_output: tft.TFTransformOutput,
batch_size: int) -> tf.data.Dataset:
"""
Generates features and label for tuning/training.
Args:
file_pattern: List of paths or patterns of input tfrecord files.
data_accessor: DataAccessor for converting input to RecordBatch.
tf_transform_output: A TFTransformOutput.
batch_size: representing the number of consecutive elements of returned
dataset to combine in a single batch
Returns:
A dataset that contains features where features is a
dictionary of Tensors.
"""
dataset = data_accessor.tf_dataset_factory(
file_pattern,
tfxio.TensorFlowDatasetOptions(batch_size = batch_size),
tf_transform_output.transformed_metadata.schema
)
transform_layer = tf_transform_output.transform_features_layer()
def apply_transform(raw_features):
return _apply_preprocessing(raw_features, transform_layer)
return dataset.map(apply_transform).repeat()
This differs from the _input_fn example given above as I was following the example in the next tfx tutorial found here: https://www.tensorflow.org/tfx/tutorials/tfx/penguin_tft#run_fn
Also for reference, there is no Target within the example data so there is no label_key to be passed to the tfxio.TensorFlowDatasetOptions object.
Error
When trying to run the Trainer component using a TFX InteractiveContext object I receive the following error.
ValueError: No gradients provided for any variable: ['dense_460/kernel:0', 'dense_460/bias:0', 'dense_461/kernel:0', 'dense_461/bias:0', 'dense_462/kernel:0', 'dense_462/bias:0', 'dense_463/kernel:0', 'dense_463/bias:0', 'dense_464/kernel:0', 'dense_464/bias:0', 'dense_465/kernel:0', 'dense_465/bias:0'].
From my own attempts to solve this I believe the problem lies in the way that an Autoencoder is trained. From the Autoencoder example linked here https://www.tensorflow.org/tutorials/generative/autoencoder the data is fitted like so:
autoencoder.fit(x_train, x_train,
epochs=10,
shuffle=True,
validation_data=(x_test, x_test))
therefore it stands to reason that the tf.Dataset should also mimic this behaviour and when testing with plain Tensor objects I have been able to recreate the error above and then solve it when adding the target to be the same as the training data in the .fit() function.
Things I've Tried So Far
Duplicating Train Dataset
model.fit(
train_dataset,
train_dataset,
steps_per_epoch = fn_args.train_steps,
validation_data = eval_dataset,
validation_steps = fn_args.eval_steps
)
Raises error due to Keras not accepting a 'y' value when a dataset is passed.
ValueError: `y` argument is not supported when using dataset as input.
Returning a dataset that is a tuple with itself
def _input_fn(...
dataset = data_accessor.tf_dataset_factory(
file_pattern,
tfxio.TensorFlowDatasetOptions(batch_size = batch_size),
tf_transform_output.transformed_metadata.schema
)
transform_layer = tf_transform_output.transform_features_layer()
def apply_transform(raw_features):
return _apply_preprocessing(raw_features, transform_layer)
dataset = dataset.map(apply_transform)
return dataset.map(lambda x: (x, x))
This raises an error where the keys from the features dictionary don't match the output of the model.
ValueError: Found unexpected keys that do not correspond to any Model output: dict_keys(['feature_string', ...]). Expected: ['dense_477']
At this point I switched to using the keras.model Autoencoder subclass and tried to add output keys to the Model using an output which I tried to create dynamically in the same way as the inputs.
def call(self, x):
inputs = [keras.layers.Input(shape = (1,), name = f) for f in x]
dense = keras.layers.concatenate(inputs)
encoded = self.encoder(dense)
decoded = self.decoder(encoded)
outputs = {}
for feature_name in x:
outputs[feature_name] = keras.layers.Dense(1, activation = 'sigmoid')(decoded)
return outputs
This raises the following error:
TypeError: Cannot convert a symbolic Keras input/output to a numpy array. This error may indicate that you're trying to pass a symbolic value to a NumPy call, which is not supported. Or, you may be trying to pass Keras symbolic inputs/outputs to a TF API that does not register dispatching, preventing Keras from automatically converting the API call to a lambda layer in the Functional Model.
I've been looking into solving this issue but am no longer sure if the data is being passed correctly and am beginning to think I'm getting side-tracked from the actual problem.
Questions
Has anyone managed to get an Autoencoder working when connected via TFX examples?
Did you alter the tf.Dataset or handled the examples in a different way to the _input_fn demonstrated?
So I managed to find an answer to this and wanted to leave what I found here in case anyone else stumbles onto a similar problem.
It turns out my feelings around the error were correct and the solution did indeed lie in how the tf.Dataset object was presented.
This can be demonstrated when I ran some code which simulated the incoming data using randomly generated tensors.
tensors = [tf.random.uniform(shape = (1, 82)) for i in range(739)]
# This gives us a list of 739 tensors which hold 1 value for 82 'features' simulating the dataset I had
dataset = tf.data.Dataset.from_tensor_slices(tensors)
dataset = dataset.map(lambda x : (x, x))
# This returns a dataset which marks the training set and target as the same
# which is what the Autoecnoder model is looking for
model.fit(dataset ...)
Following this I proceeded to do the same thing with the dataset returned by the _input_fn. Given that the tfx DataAccessor object returns a features_dict however I needed to combine the tensors in that dict together to create a single tensor.
This is how my _input_fn looks now:
def create_target_values(features_dict: Dict[str, tf.Tensor]) -> tuple:
value_tensor = tf.concat(list(features_dict.values()), axis = 1)
return (features_dict, value_tensor)
def _input_fn(
file_pattern,
data_accessor: tfx.components.DataAccessor,
tf_transform_output: tft.TFTransformOutput,
batch_size: int) -> tf.data.Dataset:
"""
Generates features and label for tuning/training.
Args:
file_pattern: List of paths or patterns of input tfrecord files.
data_accessor: DataAccessor for converting input to RecordBatch.
tf_transform_output: A TFTransformOutput.
batch_size: representing the number of consecutive elements of returned
dataset to combine in a single batch
Returns:
A dataset that contains (features, target_tensor) tuple where features is a
dictionary of Tensors, and target_tensor is a single Tensor that is a concatenated tensor of all the
feature values.
"""
dataset = data_accessor.tf_dataset_factory(
file_pattern,
tfxio.TensorFlowDatasetOptions(batch_size = batch_size),
tf_transform_output.transformed_metadata.schema
)
dataset = dataset.map(lambda x: create_target_values(features_dict = x))
return dataset.repeat()
All the documentation for Keras pre-processing seems to assume a single Input. If you have a model with multiple Inputs:
x_norm = preprocessing.Normalization()
y_norm = preprocessing.Normalization()
x = layers.Input(shape=(1,))
x = x_norm(x)
y = layers.Input(shape=(1,))
y = y_norm(y)
concated = layers.Concatenate()([x, y])
output = layers.Dense(1)(concated)
model = keras.Model(inputs=[x, y], outputs=output)
It's unclear how to use adapt() on a Dataset to "train" each preprocessing layer (i.e. x_norm and y_norm). With a single Input and preprocessing layer (e.g. preprocessing_layer) you simply do:
preprocessing_layer.adapt(dataset)
But in the case of multiple inputs how do I select the right input feature to use in adapt()?
The best I've come up with so far is:
normalization_layers = {
'x': preprocessing.Normalization(),
'y': preprocessing.Normalization(),
}
for batch in dataset:
for name, layer in normalization_layers.items():
layer.adapt(batch[0][name])
I don't know if this is efficient and TensorFlow gives a warning about a tf.function (inside adapt()) being called in a loop.
i am new to deep learning, and tensorflow. and it seems that all example online are very simple sequential model. but i have a lit bit complex model that i am trying to implement using tensorflow 2.1. so in nutshell, i have two models A, B that i am trying to combined its output and used as input for model C. please refer to diagram i attached to have more clear understanding of model architecture that i am trying to achieve
here is un-completed code so far i am still getting a lot of errors. any suggestions of how i can implement below network. thanks in advance.
def model_a():
model_small = Sequential()
model_small.add(Conv1D(filters=64, kernel_size=50, activation=None, input_shape=(3000, 1)))
model_small.add(MaxPool1D(pool_size=8, strides=8))
model_small.add(Dropout(0.5))
model_small.add(Conv1D(filters=128, kernel_size=8, strides=1, activation=None))
model_small.add(Conv1D(filters=128, kernel_size=8, strides=1, activation=None))
model_small.add(Conv1D(filters=128, kernel_size=8, strides=1, activation=None))
model_small.add(MaxPool1D(pool_size=4, strides=4))
model_small.add(Flatten())
return model_small
#return model_small.add(Flatten())
def model_c():
model = Sequential()
model.add(Bidirectional(LSTM(512)))
model.add(Dropout(0.5))
model.add(Dense(4, activation='sigmoid'))
def model_b():
model_large = Sequential()
# the number of strides in this layer is too large at 50?
model_large.add(Conv1D(filters=64, kernel_size=400, activation=None, input_shape=(3000, 1)))
model_large.add(MaxPool1D(pool_size=4))
model_large.add(Dropout(0.5))
model_large.add(Conv1D(filters=128, kernel_size=6, activation=None))
model_large.add(Conv1D(filters=128, kernel_size=6, activation=None))
model_large.add(Conv1D(filters=128, kernel_size=6, strides=1, activation=None))
model_large.add(MaxPool1D(pool_size=2))
model_large.add(Flatten())
return model_large
#return model_large.add(Flatten())
def final_model():
input1 = model_a()
input2 = model_b()
model_concat = concatenate([input1.output, input2.output], axis=1)
model_concat = Dropout(0.5)(model_concat)
# try to fix model_c here but i don't how
model= Model(inputs=[input1.input, input2.input], outputs=model_concat)
return model
model_2 = final_model()
model_2.compile(
loss=tf.keras.losses.sparse_categorical_crossentropy,
optimizer=tf.keras.optimizers.SGD(learning_rate=0.01),
metrics=['accuracy'] # can add more metrics
)
model_2.fit(x=INPUT, epochs=10)
You're on the right track. You can do it two ways: build one model split into functional parts, or build multiple models and link them together. Functionally they're the same thing, since a layer is a model; a model can be a layer.
I'm going to use some simple Dense layers to represent model_a through model_c. As a caveat, there's a bug that you might run into doing this my way if you use the sequential api, so I'll demonstrate with a functional api, but I assure you it's just as easy to use (the model is just defined after the layers, rather than before).
As one model split into functions:
import tensorflow as tf
def model_a(x):
return tf.keras.layers.Dense(56,name='model_a_whatever')(x) # returning output of layer
def model_b(x):
x = tf.keras.layers.Dense(56,name='model_b_whatever')(x)
return tf.keras.layers.Dense(56,name='model_b_more_whatever')(x)
def model_c(x):
x = tf.keras.layers.Dense(56,name='model_c_whatever')(x)
x = tf.keras.layers.Dense(56,name='model_c_more_whatever')(x)
return tf.keras.layers.Dense(56,name='model_c_even_more_whatever')(x)
# in through the input layer
main_input = tf.keras.layers.Input(shape=(12,34,56),name='INPUT')
# now through functions containing different models' layers
left = model_a(main_input)
right = model_b(main_input)
# concatenate their outputs
concatenated = tf.keras.layers.Concatenate(axis=-1)([left,right])
# now through function containing layers of model c
left = model_c(concatenated)
# and the juke right to a fully connected layer
right = tf.keras.layers.Dense(56,name='FC')(concatenated)
# then add the outputs and apply softmax activation
added = tf.keras.layers.Add(name='add')([left,right])
outputs = tf.keras.layers.Activation('softmax',name='Softmax')(added)
# now define the model
model = tf.keras.models.Model(main_input,outputs) # Model(input layer, final output))
print(model.summary())
tf.keras.utils.plot_model(model, to_file='just_a_model.png')
The diagram will look more cluttered than yours, since all layers will be visible:
As many models joined together:
# as separate models linked together
def build_model_a():
a = tf.keras.layers.Input(shape=(12,34,56),name='model_a_input')
b = tf.keras.layers.Dense(56,name='model_a_whatever')(a) # whatever layers
return tf.keras.models.Model(a,b,name='MODEL_A') # returning model, not just layer output
def build_model_b():
a = tf.keras.layers.Input(shape=(12,34,56),name='model_b_input')
b = tf.keras.layers.Dense(56,name='model_b_whatever')(a)
b = tf.keras.layers.Dense(56,name='model_b_more_whatever')(b)
return tf.keras.models.Model(a,b,name='MODEL_B')
def build_model_c():
a = tf.keras.layers.Input(shape=(12,34,112),name='model_c_input') # axis 2 is doubled because concatenation.
b = tf.keras.layers.Dense(56,name='model_c_whatever')(a)
b = tf.keras.layers.Dense(56,name='model_c_more_whatever')(b)
b = tf.keras.layers.Dense(56,name='model_c_even_more_whatever')(b)
return tf.keras.models.Model(a,b,name='MODEL_C')
# define the main input
main_input = tf.keras.layers.Input(shape=(12,34,56),name='INPUT')
# build the models
model_a = build_model_a()
model_b = build_model_b()
model_c = build_model_c()
# pass input through models a and b
a = model_a(main_input)
b = model_b(main_input)
# concatenate their outputs
ab = tf.keras.layers.Concatenate(axis=-1,name='Concatenate')([a,b])
# pass through model c and fully-connected layer
c = model_c(ab)
d = tf.keras.layers.Dense(56,name='FC')(ab)
# add their outputs and apply softmax activation
add = tf.keras.layers.Add(name="add")([c,d])
outputs = tf.keras.layers.Activation('softmax',name='Softmax')(add)
model = tf.keras.models.Model(main_input,outputs)
print(model.summary())
tf.keras.utils.plot_model(model, to_file='multi_model.png')
Although this is functionally the same network as in the first case, the diagram now matches yours:
Either method will work. As you can see, the first method is just a code-cleanup, really; putting separate data-pipelines into functions for clarity. If you want to get more complicated, like having different loss functions for sub-models and such, then the second method might simplify the process. The bug I mentioned only happens if you use the sequential api with the second method.
I'm following the section "Losses and Metrics Based on Model Internals" on chapter 12 of "Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow, 2nd Edition - Aurélien Geron", in which he shows how to add custom losses and metrics that do not depend on labels and predictions.
To illustrate this, we add a custom "reconstruction loss" by adding a layer on top of the upper hidden layer which should reproduce the input. The loss is the mean squared difference betweeen the reconstruction loss and the inputs.
He shows the code for adding the custom loss, which works nicely, but even following his description I cannot make add the metric, since it raises `ValueError". He says:
Similarly, you can add a custom metric based on model internals by
computing it in any way you want, as long as the result is the output of a
metric object. For example, you can create a keras.metrics.Mean object
in the constructor, then call it in the call() method, passing it the
recon_loss, and finally add it to the model by calling the model’s
add_metric() method.
This is the code(I have added #MINE for the lines I have added myself)
import tensorflow as tf
from tensorflow import keras
class ReconstructingRegressor(keras.models.Model):
def __init__(self, output_dim, **kwargs):
super().__init__(**kwargs)
self.hidden = [keras.layers.Dense(30, activation="selu",
kernel_initializer="lecun_normal")
for _ in range(5)]
self.out = keras.layers.Dense(output_dim)
self.reconstruction_mean = keras.metrics.Mean(name="reconstruction_error") #MINE
def build(self, batch_input_shape):
n_inputs = batch_input_shape[-1]
self.reconstruct = keras.layers.Dense(n_inputs)
super().build(batch_input_shape)
def call(self, inputs, training=None):
Z = inputs
for layer in self.hidden:
Z = layer(Z)
reconstruction = self.reconstruct(Z)
recon_loss = tf.reduce_mean(tf.square(reconstruction - inputs))
self.add_loss(0.05 * recon_loss)
if training: #MINE
result = self.reconstruction_mean(recon_loss) #MINE
else: #MINE
result = 0. #MINE, I have also tried different things here,
#but the help showed a similar sample to this.
self.add_metric(result, name="foo") #MINE
return self.out(Z)
Then compiling and fitting the model:
training_set_size=10
X_dummy = np.random.randn(training_set_size, 8)
y_dummy = np.random.randn(training_set_size, 1)
model = ReconstructingRegressor(1)
model.compile(loss="mse", optimizer="nadam")
history = model.fit(X_dummy, y_dummy, epochs=2)
Which throws:
ValueError: in converted code:
<ipython-input-296-878bdeb30546>:26 call *
self.add_metric(result, name="foo") #MINE
C:\Users\Kique\Anaconda3\envs\piz3\lib\site-packages\tensorflow_core\python\keras\engine\base_layer.py:1147 add_metric
self._symbolic_add_metric(value, aggregation, name)
C:\Users\Kique\Anaconda3\envs\piz3\lib\site-packages\tensorflow_core\python\keras\engine\base_layer.py:1867 _symbolic_add_metric
'We do not support adding an aggregated metric result tensor that '
ValueError: We do not support adding an aggregated metric result tensor that is not the output of a `tf.keras.metrics.Metric` metric instance. Without having access to the metric instance we cannot reset the state of a metric after every epoch during training. You can create a `tf.keras.metrics.Metric` instance and pass the result here or pass an un-aggregated result with `aggregation` parameter set as `mean`. For example: `self.add_metric(tf.reduce_sum(inputs), name='mean_activation', aggregation='mean')`
Having read that, I tried similar things to solve that issue but it just led to different errors. How can I solve this? What is the "correct" way to do this?
I'm using conda on Windows, with tensorflow-gpu 2.1.0 installed.
The problem is just right here:
def call(self, inputs, training=None):
Z = inputs
for layer in self.hidden:
Z = layer(Z)
reconstruction = self.reconstruct(Z)
recon_loss = tf.reduce_mean(tf.square(reconstruction - inputs))
self.add_loss(0.05 * recon_loss)
if training:
result = self.reconstruction_mean(recon_loss)
else:
result = 0.#<---Here!
self.add_metric(result, name="foo")
return self.out(Z)
The error says that add_metric only gets a metric derived from tf.keras.metrics.Metric but 0 is a scalar, not a metric type.
My proposed solution is to simply do that:
def call(self, inputs, training=None):
Z = inputs
for layer in self.hidden:
Z = layer(Z)
reconstruction = self.reconstruct(Z)
recon_loss = tf.reduce_mean(tf.square(reconstruction - inputs))
self.add_loss(0.05 * recon_loss)
if training:
result = self.reconstruction_mean(recon_loss)
self.add_metric(result, name="foo")
return self.out(Z)
This way, your mean reconstruction_error will be shown only in training time.
Since you work with eager mode, you should create your layer with dynamic=True as below:
model = ReconstructingRegressor(1,dynamic=True)
model.compile(loss="mse", optimizer="nadam")
history = model.fit(X_dummy, y_dummy, epochs=2, batch_size=10)
P.S - pay attention, that when calling model.fit or model.evaluate you should also make sure that the batch size divides your train set (since this is a stateful network). So, call those function like this: model.fit(X_dummy, y_dummy, epochs=2, batch_size=10) or model.evaluate(X_dummy,y_dummy, batch_size=10).
Good Luck!
from keras.models import load_model
import h5py
# sq_model.save_weights('sq_model_weights.h5')
# res_model.save_weights('res_model_weights.h5')
# model.save('my_model.h5')
# dense_model.save_weights('dense_model_v3_weights.h5')
sq_model.load_weights('sq_model_weights.h5')
res_model.load_weights('res_model_weights.h5')
dense_model.load_weights('dense_model_v2_weights.h5')
models = [sq_model, res_model, dense_model]
model_input = Input((3,32,32))
def ensemble(models, model_input):
outputs = [model.outputs[0] for model in models]
y = Average()(outputs)
model = Model(inputs = model_input, outputs = y, name='ensemble')
return model
ensemble_model = ensemble(models,model_input)
I am getting the following error when I run the above code:
RuntimeError: Graph disconnected: cannot obtain value for tensor Tensor("input_2:0", shape=(?, 3, 32, 32), dtype=float32) at layer "input_2". The following previous layers were accessed without issue: []
You have three models, each of them with a separate input. In your call to
model = Model(inputs = model_input, outputs = y, name='ensemble')
you specify a new Model. Its input should be your model_input, and the outputs should be your averaged outputs.
But you forgot to actually connect your three models to your input. So you have a disconnected model containing the loose input layer model_input and the ensemble, with each of the three models contained in the ensemble waiting for an input on its own input layer (so 4 input layers in total).
Changing
outputs = [model.outputs[0] for model in models]
to
outputs = [model(model_inputs) for model in models]
should do the trick. It calls each of the models on model_input and gives the corresponding outputs.
Changing
outputs = [model.outputs[0] for model in models]
to
outputs = [model(model_input) for model in models]
worked for me