I get a dense layer shape error with Keras Version 2.1.0. This problem only happens with this version of Keras (2.1.0). I am in no position to upgrade the version since it's on a cluster so I am trying to find a fix for the time being. My model is defined as below.
model = Sequential()
model.add(Conv2D(32, kernel_size=(3, 3),
activation='relu',
input_shape=(32, 32, 3)))
model.add(Conv2D(64, (3, 3), activation='relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.25))
model.add(Flatten())
model.add(Dense(128, activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(10, activation='softmax'))
model.compile(loss='categorical_crossentropy',
optimizer=config["optimizer"],
metrics=['accuracy'])
I have done one hot encoding as shown below.
y_train = keras.utils.to_categorical(y_train, 10)
y_test = keras.utils.to_categorical(y_test, 10)
The model summary is
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
conv2d_1 (Conv2D) (None, 30, 30, 32) 896
_________________________________________________________________
conv2d_2 (Conv2D) (None, 28, 28, 64) 18496
_________________________________________________________________
max_pooling2d_1 (MaxPooling2 (None, 14, 14, 64) 0
_________________________________________________________________
dropout_1 (Dropout) (None, 14, 14, 64) 0
_________________________________________________________________
flatten_1 (Flatten) (None, 12544) 0
_________________________________________________________________
dense_1 (Dense) (None, 128) 1605760
_________________________________________________________________
dropout_2 (Dropout) (None, 128) 0
_________________________________________________________________
dense_2 (Dense) (None, 10) 1290
=================================================================
Total params: 1,626,442
Trainable params: 1,626,442
Non-trainable params: 0
_____________________________________
The error I get is :
ValueError: Error when checking target: expected dense_2 to have 2
dimensions, but got array with shape (50000, 1, 10)
The exact same code works perfectly in Keras 2.2.4
Related
I'm trying to train my keras model but shapes are incompatible.
The error says
ValueError: Shapes (None, 3) and (None, 3, 3) are incompatible
My train set's shape is (2000, 3, 768) and lable's shape is (2000, 3).
What is the wrong the point?
Model define & fit code
input_shape = x_train.shape[1:]
model = my_dnn(input_shape, 3)
model.fit(x_train, y_train, epochs=25, verbose=1)
Model code
def my_dnn(input, num_classes):
model = Sequential()
model.add(tf.keras.Input(input))
model.add(Dense(1024))
model.add(Activation('relu'))
model.add(Dropout(0.5))
model.add(Dense(512))
model.add(Activation('relu'))
model.add(Dense(225))
model.add(Activation('relu'))
model.add(Dense(100))
model.add(Activation('relu'))
model.add(Dense(num_classes))
model.add(Activation('sigmoid'))
model.compile( loss='categorical_crossentropy',
optimizer='adam',
metrics=[tf.keras.metrics.SparseCategoricalAccuracy()])
return model
In addition to what's said, it seems you are carrying the second dimension of the input data until the end of the model. So your model summary is something like this:
Layer (type) Output Shape Param #
=================================================================
dense_1 (Dense) (None, 3, 1024) 787456
_________________________________________________________________
activation_1 (Activation) (None, 3, 1024) 0
_________________________________________________________________
dropout_1 (Dropout) (None, 3, 1024) 0
_________________________________________________________________
dense_2 (Dense) (None, 3, 512) 524800
_________________________________________________________________
activation_2 (Activation) (None, 3, 512) 0
_________________________________________________________________
dense_3 (Dense) (None, 3, 225) 115425
_________________________________________________________________
activation_3 (Activation) (None, 3, 225) 0
_________________________________________________________________
dense_4 (Dense) (None, 3, 100) 22600
_________________________________________________________________
activation_4 (Activation) (None, 3, 100) 0
_________________________________________________________________
dense_5 (Dense) (None, 3, 3) 303
_________________________________________________________________
activation_5 (Activation) (None, 3, 3) 0
=================================================================
Total params: 1,450,584
Trainable params: 1,450,584
Non-trainable params: 0
As you can see, the output shape of the model (None, 3, 3) is not compatible with the label's shape (None, 3), and at some point, you need to use a Flatten layer.
There are two possible reasons:
Your problem is multi-class classification, hence you need softmax instead of sigmoid + accuracy or CategoricalAccuracy() as a metric.
Your problem is multi-label classification, hence you need binary_crossentropy and tf.keras.metrics.BinaryAccuracy()
Depending on how your dataset is built/the task you are trying to solve, you need to opt for one of those.
For case 1, ensure your data is OHE(one-hot encoded).
Also, Marco Cerliani and Amir (in the comment below) point out that the data output needs to be in a 2D format rather than 3D : you should either preprocess the data accordingly before feeding it to the network or use, as suggested in the comment below, a Flatten() at a point (probably before the final Dense())
Hi wish to enquire some help regarding neural networks, i am doing a school project whereby i am required to build deep fake detection neural network. I am unsure on why by adding more layers into the neural. My Accuracy during training goes from 0.7 in the first epoch and jumps to 1.0 in the second to fifth epoch which is overfittin and the loss value goes to a weird number, Wish to seek advice on how i could adjust the neural network to suit deepfake detections.
Thank you all for the time in reading
import tensorflow as tf
from keras.preprocessing.image import ImageDataGenerator
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, Dropout, Activation, Flatten, Conv2D, MaxPooling2D, Dropout
model = Sequential()
model.add(Conv2D(32, (3,3), input_shape = (256,256,3)))
model.add(Activation("relu"))
model.add(MaxPooling2D(pool_size=(2,2)))
model.add(Conv2D(64, (3,3)))
model.add(Activation("relu"))
model.add(MaxPooling2D(pool_size=(2,2)))
model.add(Conv2D(64, (3,3)))
model.add(Activation("relu"))
model.add(Dropout(0.20))
model.add(Conv2D(64, (3,3)))
model.add(Activation("relu"))
model.add(Dropout(0.20))
model.add(Conv2D(64, (3,3)))
model.add(Activation("relu"))
model.add(Dropout(0.20))
model.add(Conv2D(64, (3,3)))
model.add(Activation("relu"))
#flatten the layer conv 2d dense is 1d data set
model.add(Flatten()) #convets 3d feature maps to 1D feature Vectors
model.add(Dense(64))
model.add(Activation('relu'))
model.add(Dense(1))
model.add(Activation('sigmoid'))
model.compile(loss="binary_crossentropy", optimizer="adam", metrics=['accuracy'])
model.fit(X, y, batch_size=32, epochs=5)
Model Summary
Model: "sequential"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
conv2d (Conv2D) (None, 254, 254, 32) 896
_________________________________________________________________
activation (Activation) (None, 254, 254, 32) 0
_________________________________________________________________
max_pooling2d (MaxPooling2D) (None, 127, 127, 32) 0
_________________________________________________________________
conv2d_1 (Conv2D) (None, 125, 125, 64) 18496
_________________________________________________________________
activation_1 (Activation) (None, 125, 125, 64) 0
_________________________________________________________________
max_pooling2d_1 (MaxPooling2 (None, 62, 62, 64) 0
_________________________________________________________________
conv2d_2 (Conv2D) (None, 60, 60, 64) 36928
_________________________________________________________________
activation_2 (Activation) (None, 60, 60, 64) 0
_________________________________________________________________
dropout (Dropout) (None, 60, 60, 64) 0
_________________________________________________________________
conv2d_3 (Conv2D) (None, 58, 58, 64) 36928
_________________________________________________________________
activation_3 (Activation) (None, 58, 58, 64) 0
_________________________________________________________________
dropout_1 (Dropout) (None, 58, 58, 64) 0
_________________________________________________________________
conv2d_4 (Conv2D) (None, 56, 56, 64) 36928
_________________________________________________________________
activation_4 (Activation) (None, 56, 56, 64) 0
_________________________________________________________________
dropout_2 (Dropout) (None, 56, 56, 64) 0
_________________________________________________________________
conv2d_5 (Conv2D) (None, 54, 54, 64) 36928
_________________________________________________________________
activation_5 (Activation) (None, 54, 54, 64) 0
_________________________________________________________________
flatten (Flatten) (None, 186624) 0
_________________________________________________________________
dense (Dense) (None, 64) 11944000
_________________________________________________________________
activation_6 (Activation) (None, 64) 0
_________________________________________________________________
dense_1 (Dense) (None, 1) 65
_________________________________________________________________
activation_7 (Activation) (None, 1) 0
=================================================================
Total params: 12,111,169
Trainable params: 12,111,169
Non-trainable params: 0
_________________________________________________________________
You have to specify more stuff inside each layer, not only the size and number of filters. This will help you to increase the model performance.
For example, you could use adam from keras_optimizers, which will help to increase the accuracy during training the model. Also, l2 from keras.regularizers will help you to reduce overfitting. Which means you can't increase the accuracy just by increasing the epochs, you must first build a good model before starting the training
I am using this reference project/paper https://feedforward.github.io/blog/depthmap-prediction-from-a-single/ for predicting the depth estimation from 2D images. I cannot understand how the coarse depth image (Coarse7) is formed after the fully connected layer.
I am using an color input image of size (576,172)
import pandas as pd
import matplotlib.pyplot as plt
import tensorflow as tf
import keras
from keras.models import Sequential, Model
from keras.applications import vgg16
from keras.layers import Dense, Dropout, Activation, Flatten, Conv2D, MaxPooling2D, InputLayer,UpSampling2D
from keras.layers.normalization import BatchNormalization
import cv2
model=Sequential()
model.add(Conv2D(96,(11,11),strides=(4,4),input_shape=new_arr.shape,padding='same'))
# model.add(BatchNormalization())
model.add(Activation("relu"))
model.add(MaxPooling2D(pool_size=(2,2)))
model.add(Conv2D(256,(5,5),padding='same'))
# model.add(BatchNormalization())
model.add(Activation("relu"))
model.add(MaxPooling2D(pool_size=(2,2)))
model.add(Conv2D(384,(3,3),padding='same'))
# model.add(BatchNormalization())
model.add(Activation("relu"))
# model.add(MaxPooling2D(pool_size=(2,2)))
model.add(Conv2D(384,(3,3),padding='same'))
# model.add(BatchNormalization())
model.add(Activation("relu"))
# model.add(MaxPooling2D(pool_size=(2,2)))
model.add(Dense(256))
# model.add(BatchNormalization())
model.add(Activation("relu"))
model.add(MaxPooling2D(pool_size=(2,2)))
model.add(Dropout(0.4))
model.add(Flatten())
model.add(Dense(4096))
# model.add(BatchNormalization())
model.add(Activation("relu"))
model.add(Dropout(0.4))
# model.add(UpSampling2D(size=(2, 2)))
# model.add(Conv2D(128, 5, padding='same',activation='relu', kernel_initializer='glorot_normal'))
model.summary()
This the model summary
```Layer (type) Output Shape Param #
=================================================================
conv2d_94 (Conv2D) (None, 43, 144, 96) 34944
_________________________________________________________________
activation_137 (Activation) (None, 43, 144, 96) 0
_________________________________________________________________
max_pooling2d_79 (MaxPooling (None, 21, 72, 96) 0
_________________________________________________________________
conv2d_95 (Conv2D) (None, 21, 72, 256) 614656
_________________________________________________________________
activation_138 (Activation) (None, 21, 72, 256) 0
_________________________________________________________________
max_pooling2d_80 (MaxPooling (None, 10, 36, 256) 0
_________________________________________________________________
conv2d_96 (Conv2D) (None, 10, 36, 384) 885120
_________________________________________________________________
activation_139 (Activation) (None, 10, 36, 384) 0
_________________________________________________________________
conv2d_97 (Conv2D) (None, 10, 36, 384) 1327488
_________________________________________________________________
activation_140 (Activation) (None, 10, 36, 384) 0
_________________________________________________________________
dense_44 (Dense) (None, 10, 36, 256) 98560
_________________________________________________________________
activation_141 (Activation) (None, 10, 36, 256) 0
_________________________________________________________________
max_pooling2d_81 (MaxPooling (None, 5, 18, 256) 0
_________________________________________________________________
dropout_44 (Dropout) (None, 5, 18, 256) 0
_________________________________________________________________
flatten_14 (Flatten) (None, 23040) 0
_________________________________________________________________
dense_45 (Dense) (None, 4096) 94375936
_________________________________________________________________
activation_142 (Activation) (None, 4096) 0
_________________________________________________________________
dropout_45 (Dropout) (None, 4096) 0
=================================================================
Total params: 97,336,704
Trainable params: 97,336,704
Non-trainable params: 0
After you have an flattend image or a dense layer you can reshape it to an image size with keras.layers.Reshape.
Than just upsample it to your output size.
https://www.tensorflow.org/api_docs/python/tf/keras/layers/Reshape
I hope that helped
I am beginner in Keras. I am tring to build a model for which i am using Sequential model. When i am tring to reduce the input size from 28 to 14 or lesser by using maxpooling function then the maxpooling function results does't display on call to the model.summary() function. I am tring to achive an accuracy of 0.99 or above after traing i.e, on call to model.score() the accuracy result should be 0.99 or above. Model build my me so far can be seen here
from keras.layers import Activation, MaxPooling2D
model = Sequential()
model.add(Convolution2D(32, 3, 3, activation='relu', input_shape=(28,28,1)))
model.add(Convolution2D(32, 1, activation='relu'))
MaxPooling2D(pool_size=(2, 2))
model.add(Convolution2D(32, 26))
model.add(Convolution2D(10, 1))
model.add(Flatten())
model.add(Activation('softmax'))
model.summary()
Output -
Layer (type) Output Shape Param #
=================================================================
conv2d_29 (Conv2D) (None, 26, 26, 32) 320
_________________________________________________________________
conv2d_30 (Conv2D) (None, 26, 26, 32) 1056
_________________________________________________________________
conv2d_31 (Conv2D) (None, 1, 1, 32) 692256
_________________________________________________________________
conv2d_32 (Conv2D) (None, 1, 1, 10) 330
_________________________________________________________________
flatten_7 (Flatten) (None, 10) 0
_________________________________________________________________
activation_7 (Activation) (None, 10) 0
=================================================================
Total params: 693,962
Trainable params: 693,962
Non-trainable params: 0
____________________________
Batch size i am using is 32 and number of epoch is 10.
model.compile(loss='categorical_crossentropy',
optimizer='adam',
metrics=['accuracy'])
model.fit(X_train, Y_train, batch_size=32, nb_epoch=10, verbose=1)
score = model.evaluate(X_test, Y_test, verbose=0)
print(score)
Output after training -
[0.09016687796734459, 0.9814]
You are not adding the Maxpooling2D layer to your model...
model.add(MaxPooling2D(pool_size=(2, 2)))
Also, the output of your maxpooling will have shape (None, 13, 13, 32), the convolutional kernel in the next layer (in your case 26) can't be larger than the dimensions your current (13). Your code should be something like this:
from keras.layers import Activation, MaxPooling2D, Dense
model = Sequential()
model.add(Convolution2D(32, 3, 3, activation='relu', input_shape=(28,28,1)))
model.add(Convolution2D(32, 1, activation='relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Convolution2D(32, 8))
model.add(Convolution2D(10, 6))
model.add(Flatten())
model.add(Activation('softmax'))
print(model.summary())
Output
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
conv2d_1 (Conv2D) (None, 26, 26, 32) 320
_________________________________________________________________
conv2d_2 (Conv2D) (None, 26, 26, 32) 1056
_________________________________________________________________
max_pooling2d_1 (MaxPooling2 (None, 13, 13, 32) 0
_________________________________________________________________
conv2d_3 (Conv2D) (None, 6, 6, 32) 65568
_________________________________________________________________
conv2d_4 (Conv2D) (None, 1, 1, 10) 11530
_________________________________________________________________
flatten_1 (Flatten) (None, 10) 0
_________________________________________________________________
activation_1 (Activation) (None, 10) 0
=================================================================
Total params: 78,474
Trainable params: 78,474
Non-trainable params: 0
___________________________________
P.S.: I would consider using smaller kernel sizes and a FC layer at the output, as it is a more practical solution in most cases than trying to match convolution output shapes
I have a keras (with tensorflow backend) model which is defined like so:
INPUT_SHAPE = [4740, 3540, 1]
model = Sequential()
model.add(Conv2D(32, kernel_size=(3, 3),
activation='relu',
input_shape=INPUT_SHAPE))
model.add(Conv2D(2, (4, 4), strides=(1, 1), activation='relu'))
model.add(MaxPooling2D(pool_size=(4, 4)))
model.add(Conv2D(4, (4, 4), strides=(1, 1), activation='relu'))
model.add(MaxPooling2D(pool_size=(4, 4)))
model.add(Conv2D(8, (4, 4), strides=(1, 1), activation='relu'))
model.add(MaxPooling2D(pool_size=(4, 4)))
model.add(Conv2D(16, (4, 4), strides=(1, 1), activation='relu'))
model.add(MaxPooling2D(pool_size=(4, 4)))
model.add(Conv2D(32, (4, 4), strides=(1, 1), activation='relu'))
model.add(MaxPooling2D(pool_size=(4, 4)))
model.add(Dropout(0.25))
model.add(Flatten())
model.add(Dense(128, activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(num_classes, activation='softmax'))
model.compile(loss=keras.losses.categorical_crossentropy,
optimizer=keras.optimizers.Adadelta(),
metrics=['accuracy'])
This model has only 37,506 trainable params. Yet somehow it is able to deplete K80's 12GB vram resource on model.fit() if a batch size is more then 1.
Why does this model need so much memory?
And how do I calculate memory requirements properly?
The function from How to determine needed memory of Keras model? gives me 2.15 GB per 1 element in a batch. So at least I should be able to make a batch of 5.
EDIT: model.summary()
Layer (type) Output Shape Param #
=================================================================
conv2d_1 (Conv2D) (None, 4738, 3538, 32) 320
_________________________________________________________________
conv2d_2 (Conv2D) (None, 4735, 3535, 2) 1026
_________________________________________________________________
max_pooling2d_1 (MaxPooling2 (None, 1183, 883, 2) 0
_________________________________________________________________
conv2d_3 (Conv2D) (None, 1180, 880, 4) 132
_________________________________________________________________
max_pooling2d_2 (MaxPooling2 (None, 295, 220, 4) 0
_________________________________________________________________
conv2d_4 (Conv2D) (None, 292, 217, 8) 520
_________________________________________________________________
max_pooling2d_3 (MaxPooling2 (None, 73, 54, 8) 0
_________________________________________________________________
conv2d_5 (Conv2D) (None, 70, 51, 16) 2064
_________________________________________________________________
max_pooling2d_4 (MaxPooling2 (None, 17, 12, 16) 0
_________________________________________________________________
conv2d_6 (Conv2D) (None, 14, 9, 32) 8224
_________________________________________________________________
max_pooling2d_5 (MaxPooling2 (None, 3, 2, 32) 0
_________________________________________________________________
dropout_1 (Dropout) (None, 3, 2, 32) 0
_________________________________________________________________
flatten_1 (Flatten) (None, 192) 0
_________________________________________________________________
dense_1 (Dense) (None, 128) 24704
_________________________________________________________________
dropout_2 (Dropout) (None, 128) 0
_________________________________________________________________
dense_2 (Dense) (None, 4) 516
=================================================================
Total params: 37,506
Trainable params: 37,506
Non-trainable params: 0
_________________________________________________________________
The output shape of the first layer is B*4738*3538*32 (B is the batch size), which will take around 1GB * B memory. The gradients and other activations will probably take some memory too. Maybe increasing the stride for the first layer will help.