How to customize AlexNet for difference uses - tensorflow

Question

So I've been working on learning some machine learning this past week and I have been messing around with my own regression CNN with inputs of color images 128x128 and an output of a rating. Although my dataset is small, 400ish total, I got alright results with a little overfitting (mean_absolute_error of 0.5 for training and 0.9 for testing with scale 1-10) with the model showed below:
model = keras.Sequential([
keras.layers.Conv2D(32, kernel_size=(5, 5), strides=(1, 1), activation='relu', input_shape=(128, 128, 3)),
keras.layers.Dropout(0.15),
keras.layers.MaxPooling2D(pool_size=(2, 2), strides=(2, 2)),
keras.layers.Conv2D(64, kernel_size=(5, 5), activation='relu'),
keras.layers.Dropout(0.15),
keras.layers.MaxPooling2D(pool_size=(2, 2)),
keras.layers.Flatten(),
keras.layers.Dense(1000, activation='relu'),
keras.layers.Dropout(0.4),
keras.layers.Dense(1)
])
However, not being satisfied with the results, I wanted to try out tried and true models. So I used AlexNet:
model = keras.Sequential([
keras.layers.Conv2D(filters=96, kernel_size=(11, 11), strides=(4, 4), activation='relu', input_shape=(128, 128, 3), padding='same'),
keras.layers.MaxPooling2D(pool_size=(2, 2), strides=(2, 2), padding='same'),
keras.layers.Conv2D(filters=256, kernel_size=(11, 11), strides=(1, 1), activation='relu', padding='same'),
keras.layers.MaxPooling2D(pool_size=(2, 2), strides=(2, 2), padding='same'),
keras.layers.Conv2D(filters=384, kernel_size=(3, 3), strides=(1, 1), activation='relu', padding='same'),
keras.layers.Conv2D(filters=384, kernel_size=(3, 3), strides=(1, 1), activation='relu', padding='same'),
keras.layers.Conv2D(filters=256, kernel_size=(3, 3), strides=(1, 1), activation='relu', padding='same'),
keras.layers.MaxPooling2D(pool_size=(2, 2), strides=(2, 2), padding='same'),
keras.layers.Flatten(),
keras.layers.Dense(4096, activation='relu'),
keras.layers.Dropout(0.4),
keras.layers.Dense(4096, activation='relu'),
keras.layers.Dropout(0.4),
keras.layers.Dense(1000, activation='relu'),
keras.layers.Dropout(0.4),
keras.layers.Dense(1)
])
However, it converged much slower and pretty much plateaued at MAE of 1.2 for train and 0.9 for test. Although this does show less overfitting, I thought it was strange that I still got the same test results. Is my implemention of AlexNet flawed or is this just not the right application for AlexNet. I understand it is usually used for classification, but I figured it might be worth trying with regression. Any info/suggestions/criticisms help, thanks!

Answer 1

I don't see anything clearly wrong with your AlexNet implementation. But I'd like to point few things.
The way dropout used in the first model
It is not a standard things to apply Dropout like that after a convolution output. When you apply dropout this way the outputs in the Convolution output gets randomly switched off. But unlike a fully connected layer, convolution outputs have a "spatial" structure. What I'm trying to say is it make me a more sense to switch of full channels than switching off random neurons. I think an illustration would help. Think of one channel output corresponding to a single neuron of a fully connected layer (not the best analogy but it helps to understand my proposal).
Or the other option is to be rid of Dropout after convolution outputs and only have Dropout after fully connected layers.
Time taken to converge for AlexNet
AlexNet is significantly large than the model 1, meaning way more parameters than your first model. So it makes sense of it to take longer to converge.
Why the accuracy low?
One thing I can think of is the size of the output just before the Flatten() layer. In model 1, it is 32x32, where with Alexnet it is, 4x4 which is very small. So your fully connected layers have very little information coming from the Convolution layers. This might be causing AlexNet to underperform (Just a speculation).