Related
Recently, i was in an interview and i was asked to build a neural netowrk using tensorflow which meets the following requirements:
The input layer of the model must have an input shape of (32, 10, 1)
The model must have an output shape of (32, 10, 1)
and in response, i provided the following solution:
import tensorflow as tf
from tensorflow.keras.layers import Dense, Conv1D
model = tf.keras.models.Sequential([
Conv1D(filters=32, kernel_size=1, activation='relu', input_shape=(32, 10, 1)),
Dense(30, activation='relu'),
Dense(10, activation='relu'),
tf.keras.layers.Dense(1)
])
and in order to prove that my model can pass the requeirment, i printed the input-shape and out-put shape of each model using the below code:
for layer in model.layers:
print('input shape: ',layer.input_shape, 'output shape: ',layer.output_shape)
and here is what i got in the output:
input shape: (None, 32, 10, 1) output shape: (None, 32, 10, 32)
input shape: (None, 32, 10, 32) output shape: (None, 32, 10, 30)
input shape: (None, 32, 10, 30) output shape: (None, 32, 10, 10)
input shape: (None, 32, 10, 10) output shape: (None, 32, 10, 1)
Sadly and apparently my answer to this question was not correct and i don't know how to build such model ?
As you can see, my model has 4 dimentions and the input and output layer start by None.
Is it the problem ?
I am not 100% sure but for me really seems like you did not explicitly declared the input layer, I really think at the shape's command response we should not see a 'None' on it.
Two possible solutions I found at this source, which the best one seems to be the following (not tested):
inputs = Input(shape=(32, 10, 1))
x = Conv1D(filters=32, kernel_size=1)(inputs)
x = Dense(30, "relu")(x)
outputs = Dense(10, "relu")(x)
model = Model(inputs=inputs, outputs=outputs, name="my_model_name")
Let's see if that makes any sense.
Thanks to #Pedro Silva and #AloneTogether i came out with a possible solution as below. So, in the Input or Conv1D layer the input_shape does not include the Batch_size of the input data. The input_shape only specifies the shape of each Data point or (entry of data) and if we need to specify the Batch_size then we cn use the batch_size parameter in the layer. So, if we develop the mode as :
import tensorflow as tf
from tensorflow.keras.layers import Dense, Conv1D,Input
from tensorflow.keras.models import Model
model = tf.keras.models.Sequential([
Conv1D(filters=32, kernel_size=1, activation='relu', input_shape=(10, 1),batch_size=32),
Dense(30, activation='relu'),
Dense(10, activation='relu'),
tf.keras.layers.Dense(1)
])
for layer in model.layers:
print('input shape: ',layer.input_shape, 'output shape: ',layer.output_shape)
or this:
inputs = Input(shape=(10, 1),batch_size=32)
x = Conv1D(filters=32, kernel_size=1)(inputs)
x = Dense(30, "relu")(x)
outputs = Dense(10, "relu")(x)
model = Model(inputs=inputs, outputs=outputs, name="my_model_name")
for layer in model.layers:
print('input shape: ',layer.input_shape, 'output shape: ',layer.output_shape)
Then in both cases, the model has the following shape of input and output:
input shape: (32, 10, 1) output shape: (32, 10, 1)
input shape: (32, 10, 1) output shape: (32, 10, 32)
input shape: (32, 10, 32) output shape: (32, 10, 30)
input shape: (32, 10, 30) output shape: (32, 10, 10)
I am trying to define a model happyModel()
# GRADED FUNCTION: happyModel
def happyModel():
"""
Implements the forward propagation for the binary classification model:
ZEROPAD2D -> CONV2D -> BATCHNORM -> RELU -> MAXPOOL -> FLATTEN -> DENSE
Note that for simplicity and grading purposes, you'll hard-code all the values
such as the stride and kernel (filter) sizes.
Normally, functions should take these values as function parameters.
Arguments:
None
Returns:
model -- TF Keras model (object containing the information for the entire training process)
"""
model = tf.keras.Sequential(
[
## ZeroPadding2D with padding 3, input shape of 64 x 64 x 3
tf.keras.layers.ZeroPadding2D(padding=(3,3), data_format=(64,64,3)),
## Conv2D with 32 7x7 filters and stride of 1
tf.keras.layers.Conv2D(32, (7, 7), strides = (1, 1), name = 'conv0'),
## BatchNormalization for axis 3
tf.keras.layers.BatchNormalization(axis = 3, name = 'bn0'),
## ReLU
tf.keras.layers.Activation('relu'),
## Max Pooling 2D with default parameters
tf.keras.layers.MaxPooling2D((2, 2), name='max_pool0'),
## Flatten layer
tf.keras.layers.Flatten(),
## Dense layer with 1 unit for output & 'sigmoid' activation
tf.keras.layers.Dense(1, activation='sigmoid', name='fc'),
# YOUR CODE STARTS HERE
# YOUR CODE ENDS HERE
]
)
return model
and following code is for creating the object of this model defined above:
happy_model = happyModel()
# Print a summary for each layer
for layer in summary(happy_model):
print(layer)
output = [['ZeroPadding2D', (None, 70, 70, 3), 0, ((3, 3), (3, 3))],
['Conv2D', (None, 64, 64, 32), 4736, 'valid', 'linear', 'GlorotUniform'],
['BatchNormalization', (None, 64, 64, 32), 128],
['ReLU', (None, 64, 64, 32), 0],
['MaxPooling2D', (None, 32, 32, 32), 0, (2, 2), (2, 2), 'valid'],
['Flatten', (None, 32768), 0],
['Dense', (None, 1), 32769, 'sigmoid']]
comparator(summary(happy_model), output)
I got following error:
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
<ipython-input-67-f33284fd82fe> in <module>
1 happy_model = happyModel()
2 # Print a summary for each layer
----> 3 for layer in summary(happy_model):
4 print(layer)
5
~/work/release/W1A2/test_utils.py in summary(model)
30 result = []
31 for layer in model.layers:
---> 32 descriptors = [layer.__class__.__name__, layer.output_shape, layer.count_params()]
33 if (type(layer) == Conv2D):
34 descriptors.append(layer.padding)
/opt/conda/lib/python3.7/site-packages/tensorflow/python/keras/engine/base_layer.py in output_shape(self)
2177 """
2178 if not self._inbound_nodes:
-> 2179 raise AttributeError('The layer has never been called '
2180 'and thus has no defined output shape.')
2181 all_output_shapes = set(
AttributeError: The layer has never been called and thus has no defined output shape.
I suspect my calling of ZeroPadding2D() is not right. The project seems to require the input shape of ZeroPadding2D() to be 64X64X3. I tried many formats but could not fix the problem. Anyone can give a pointer? Thanks a lot.
In your model definition, there's an issue with the following layer:
tf.keras.layers.ZeroPadding2D(padding=(3,3), data_format=(64,64,3)),
First, you didn't define any input layer also, the data_format is a string, one of channels_last (default) or channels_first, source. The correct way to define the above model as follows:
def happyModel():
model = tf.keras.Sequential(
[
## ZeroPadding2D with padding 3, input shape of 64 x 64 x 3
tf.keras.layers.ZeroPadding2D(padding=(3,3),
input_shape=(64, 64, 3), data_format="channels_last"),
....
....
happy_model = happyModel()
happy_model.summary()
Model: "sequential_2"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
zero_padding2d_4 (ZeroPaddin (None, 70, 70, 3) 0
_________________________________________________________________
conv0 (Conv2D) (None, 64, 64, 32) 4736
_________________________________________________________________
bn0 (BatchNormalization) (None, 64, 64, 32) 128
_________________________________________________________________
activation_2 (Activation) (None, 64, 64, 32) 0
_________________________________________________________________
max_pool0 (MaxPooling2D) (None, 32, 32, 32) 0
_________________________________________________________________
flatten_16 (Flatten) (None, 32768) 0
_________________________________________________________________
fc (Dense) (None, 1) 32769
=================================================================
Total params: 37,633
Trainable params: 37,569
Non-trainable params: 64
Per the documentation for tf.keras.Sequential() (https://www.tensorflow.org/api_docs/python/tf/keras/Sequential):
"Optionally, the first layer can receive an input_shape argument"
So instead of
tf.keras.layers.ZeroPadding2D(padding=(3,3), data_format=(64,64,3))
if you want to specify input shape it should be
tf.keras.layers.ZeroPadding2D(padding=(3,3), input_shape=(64,64,3))
model = tf.keras.Sequential([
# YOUR CODE STARTS HERE
tf.keras.layers.ZeroPadding2D(padding=(3, 3), input_shape=(64,64,3), data_format="channels_last"),
tf.keras.layers.Conv2D(32, (7, 7), strides = (1, 1)),
tf.keras.layers.BatchNormalization(axis=3),
tf.keras.layers.ReLU(),
tf.keras.layers.MaxPooling2D(),
tf.keras.layers.Flatten(),
tf.keras.layers.Dense(1, activation='sigmoid'),
# YOUR CODE ENDS HERE
])
return model
Try it working perfectly......
model = tf.keras.Sequential(
[
## ZeroPadding2D with padding 3, input shape of 64 x 64 x 3
## Conv2D with 32 7x7 filters and stride of 1
## BatchNormalization for axis 3
## ReLU
## Max Pooling 2D with default parameters
## Flatten layer
## Dense layer with 1 unit for output & 'sigmoid' activation
# YOUR CODE STARTS HERE
tfl.ZeroPadding2D(padding=(3,3), input_shape=(64,64,3),data_format="channels_last"),
tfl.Conv2D(32, (7, 7), strides = (1, 1), name = 'conv0'),
tfl.BatchNormalization(axis = 3, name = 'bn0'),
tfl.ReLU(),
tfl.MaxPooling2D((2, 2), name='max_pool0'),
tfl.Flatten(),
tfl.Dense(1, activation='sigmoid', name='fc'),
# YOUR CODE ENDS HERE
])
It is working you can try it.
I have the sample following model
from tensorflow.keras import models
from tensorflow.keras import layers
sample_model = models.Sequential()
sample_model.add(layers.Dense(32, input_shape=(4,)))
sample_model.add(layers.Dense(16, input_shape = (44,)))
sample_model.compile(loss="binary_crossentropy",
optimizer="adam", metrics = ["accuracy"])
IP for the model:
sam_x = np.random.rand(10,4)
sam_y = np.array([0,1,1,0,1,0,0,1,0,1,])
sample_model.fit(sam_x,sam_y)
The confusion is the fit should have thrown an error of shape mismatch as the expected_input_shape for the 2nd Dense Layer is given as (None,44) but the output for the 1st Dense Layer (which is the input of the 2nd Dense Layer) will be of shape (None,32). But it ran successfully.
I don't understand why there was no error. Any clarifications will be helpful
The input_shape keyword argument has an effect only on the first layer of a Sequential. The shape of the input of the other layers will be derived from their previous layer.
That behaviour is hinted in the doc of tf.keras.layers.InputShape:
When using InputLayer with Keras Sequential model, it can be skipped by moving the input_shape parameter to the first layer after the InputLayer.
And in the Sequential Model guide.
The behaviour can be confirmed by looking at the source of the Sequential.add method:
if not self._layers:
if isinstance(layer, input_layer.InputLayer):
# Case where the user passes an Input or InputLayer layer via `add`.
set_inputs = True
else:
batch_shape, dtype = training_utils.get_input_shape_and_dtype(layer)
if batch_shape:
# Instantiate an input layer.
x = input_layer.Input(
batch_shape=batch_shape, dtype=dtype, name=layer.name + '_input')
# This will build the current layer
# and create the node connecting the current layer
# to the input layer we just created.
layer(x)
set_inputs = True
If there is no layers yet in the model, then an Input will be added to the model with the shape derived from the first layer of the model. This is done only if no layer is present yet in the model.
That shape is either fully known (if input_shape has been passed to the first layer of the model) or will be fully known once the model is built (for example, with a call to model.build(input_shape)).
The thing is after checking the input shape of the model from the first layer, it won't check or deal with other declared input shape inside that same model. For example, if you write your model the following way
sample_model.add(layers.Dense(32, input_shape=(4,)))
sample_model.add(layers.Dense(16, input_shape = (44,)))
sample_model.add(layers.Dense(8, input_shape = (32,)))
The program will always check the first declared input shape layer and discard the rest. So, if you start your first layer with input_shape = (44,), you need to pass exact feature numbers to your model as input such as:
sam_x = np.random.rand(10,44)
sam_y = np.array([0,1,1,0,1,0,0,1,0,1,])
sample_model.fit(sam_x,sam_y)
Additionally, if you look at the Functional API, unlike the Sequential model, you must create and define a standalone Input layer that specifies the shape of input data. It's not learnable but simply a spec layer. It's a kind of gateway of the input data for the model. That means even if we define input_shape inside the other layers, they all will be discarded. For example:
nputs = keras.Input(shape=(4,))
dense = layers.Dense(64, input_shape=(8,)) # dicard input_shape
x = dense(inputs)
x = layers.Dense(64, input_shape=(16,))(x) # dicard input_shape
outputs = layers.Dense(10)(x)
model = keras.Model(inputs=inputs, outputs=outputs, name="mnist_model")
Here is a more complex example with Conv2D and MNIST.
encoder_input = keras.Input(shape=(28, 28, 1),)
x = layers.Conv2D(16, 3, activation="relu", input_shape=[32,32,3])(encoder_input)
x = layers.Conv2D(32, 3, activation="relu", input_shape=[64,64,3])(x)
x = layers.MaxPooling2D(3)(x)
x = layers.Conv2D(32, 3, activation="relu", input_shape=[224,321,3])(x)
x = layers.Conv2D(16, 3, activation="relu", input_shape=[420,32,3])(x)
x = layers.GlobalMaxPooling2D()(x)
out = layers.Dense(10, activation='softmax')(x)
encoder = keras.Model(encoder_input, out, name="encoder")
encoder.summary()
Model: "encoder"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
input_15 (InputLayer) [(None, 28, 28, 1)] 0
_________________________________________________________________
conv2d_8 (Conv2D) (None, 26, 26, 16) 160
_________________________________________________________________
conv2d_9 (Conv2D) (None, 24, 24, 32) 4640
_________________________________________________________________
max_pooling2d_2 (MaxPooling2 (None, 8, 8, 32) 0
_________________________________________________________________
conv2d_10 (Conv2D) (None, 6, 6, 32) 9248
_________________________________________________________________
conv2d_11 (Conv2D) (None, 4, 4, 16) 4624
_________________________________________________________________
global_max_pooling2d_2 (Glob (None, 16) 0
_________________________________________________________________
dense_56 (Dense) (None, 10) 170
=================================================================
Total params: 18,842
Trainable params: 18,842
Non-trainable params: 0
def pre_process(image, label):
return (image / 256)[...,None].astype('float32'),
tf.keras.utils.to_categorical(label, num_classes=10)
(x, y), (_, _) = tf.keras.datasets.mnist.load_data('mnist')
encoder.compile(
loss = tf.keras.losses.CategoricalCrossentropy(),
metrics = tf.keras.metrics.CategoricalAccuracy(),
optimizer = tf.keras.optimizers.Adam())
encoder.fit(x, y, batch_size=256)
4s 14ms/step - loss: 1.4303 - categorical_accuracy: 0.5279
I think Keras will create (or preserves to create) an additional Input Layer - but as the second dense layer is added using model.add() it will automatically be connected to the layer before, and thus the extra input layer stays unconnected and is not part of the model.
(I agree that it would be nice of Keras to hint at unconnected layers, I sometimes created unconnected layers when using the functional API and changed the inputs. Keras doesn't remind me that I had jumped several layers, I just wondered why the summary() was so short...)
For each convolution activation map, I want to concatenate a layer of constants -- more particularly, I want to concatenate a meshgrid. (This is to reproduce a paper from Uber.)
For example, say I have an activation map of (?, 256, 256, 32); then I want to concatenate a constants layer of shape (?, 256, 256, 1).
This is how I am doing this:
from keras import layers
import tensorflow as tf
import numpy as np
input_layer = layers.Input((256, 256, 3))
conv = layers.Conv2D(32, 3, padding='same')(input_layer)
print('conv:', conv.shape)
xx, yy = np.mgrid[:256, :256] # [(256, 256), (256, 256)]
xx = tf.constant(xx, np.float32)
yy = tf.constant(yy, np.float32)
xx = tf.reshape(xx, (-1, 256, 256, -1))
yy = tf.reshape(yy, (-1, 256, 256, -1))
print('xx:', xx.shape, 'yy:', yy.shape)
concat = layers.Concatenate()([conv, xx, yy])
print('concat:', concat.shape)
conv2 = layers.Conv2D(32, 3, padding='same')(concat)
print('conv2:', conv2.shape)
But I get the error:
conv: (?, 256, 256, 32)
xx: (?, 256, 256, ?) yy: (?, 256, 256, ?)
concat: (?, 256, 256, ?)
Traceback (most recent call last):
File "temp.py", line 21, in <module>
conv2 = layers.Conv2D(32, 3, padding='same')(concat)
[...]
raise ValueError('The channel dimension of the inputs '
ValueError: The channel dimension of the inputs should be defined. Found `None`.
The problem is that my constants layer is (?, 256, 256, ?), instead of (?, 256, 256, 1), and then the next convolution layers errors-out.
I have tried other things without success.
PS: The paper I was trying to implement is already implemented here.
The problem is that tf.reshape can't infer the shape of more than one dimension (i.e. using -1 for more than one dimension results in undefined dimensions ?). Since you want the shapes of xx and yy to be (?, 256, 256, 1), you could reshape these tensors as follows:
xx = tf.reshape(xx, (-1, 256, 256, 1))
yy = tf.reshape(yy, (-1, 256, 256, 1))
The resulting shapes will be (1, 256, 256, 1). Now, conv is (?, 256, 256, 32), and keras.layers.Concatenate requires the shapes of all the inputs to match except for the concat axis. You could then use tf.tile to repeat tensors xx and yy along the first dimension in order to match the batch size:
xx = tf.tile(xx, [tf.shape(conv)[0], 1, 1, 1])
yy = tf.tile(yy, [tf.shape(conv)[0], 1, 1, 1])
The shapes of xx and yy are now (?, 256, 256, 1), and the tensors can be concatenated because their first dimension matches the batch size.
ValueError: Error when checking input: expected input_1 to have shape (None, 299, 299, 3) but got array with shape (32, 229, 229, 3)
Here is the core part of the code:
train_datagen = ImageDataGenerator()
train_batches = train_datagen.flow_from_directory('../train/',
target_size=(229, 229))
test_datagen = ImageDataGenerator()
test_batches = test_datagen.flow_from_directory('../test/',
target_size=(229, 229))
#
model = keras.applications.InceptionV3(weights="imagenet")
#
clf_model.compile(optimizer='rmsprop',
loss='categorical_crossentropy',
metrics=['accuracy'])
train_history = clf_model.fit_generator(train_batches,
steps_per_epoch=train_batches.samples/train_batches.batch_size,
epochs=1,
validation_data = test_batches,
validation_steps=test_batches.samples/test_batches.batch_size )
Looks a weird Error. I saw other peoples error were like
Error when checking input: expected input_1 to have shape (None, 10, 3) but got array with shape (1, 3, 10)
or
Error when checking input: expected input_1 to have shape (None, 10, 3) but got array with shape (1, 4, 16)
However, in my error, the shape looks fine. Keras complains about the shape is not None... Pls help. thanks
Your input shape is not math:
train_datagen = ImageDataGenerator()
train_batches = train_datagen.flow_from_directory('../train/',
target_size=(229, 229))
test_datagen = ImageDataGenerator()
test_batches = test_datagen.flow_from_directory('../test/',
target_size=(229, 229))
change target_size=(229,229) to target_size=(299,299) for train_batches and test_batches.