Loss and accuracy don't change during the training phase - tensorflow

Question

I built a model to colorize a grayscale image, during the training phase i feed the network 100 RGB images of a forest, and then i convert the images to the LAB color space to split the training set to L and AB,
Based on the trained AB data, the model will predict these two channels for the grayscale input image during the testing phase.
Now i have a problem, i trained the model with a different architecture than this one with 10 images, the loss decreased to 0.0035 and it worked good, for that, i wanted to increase the size of the dataset to acquire a better result, but in exchange, the loss and the accuracy kept being constant and the model output is a mess,
My code is the following, i wish anyone can direct me of what i am doing wrong, is it because of the optimizer? the loss function? the batch size? or anything else i'm not aware of,
Thank you in advance.
# Import images
MODEL_NAME = 'forest'
X = []
Y = []
for filename in os.listdir('forest/'):
if (filename != '.DS_Store'):
image = img_to_array(load_img("/Users/moos/Desktop/Project-Master/forest/" + filename))
image = np.array(image, dtype=float)
imL = rgb2lab(1.0 / 255 * image)[:, :,0]
X.append(imL)
imAB = rgb2lab(1.0 / 255 * image)[:, :,1:]
imAB = imAB/128
Y.append(imAB)
X = np.array(X)
Y = np.array(Y)
X = X.reshape(1, 256 , np.size(X)/256, 1)
Y = Y.reshape(1, 256, np.size(Y)/256/2, 2)
# Building the neural network
model = Sequential()
model.add(InputLayer(input_shape=(256, np.size(X)/256, 1)))
model.add(Conv2D(8, (3, 3), activation='relu', padding='same', strides=2))
model.add(Conv2D(32, (3, 3), activation='relu', padding='same'))
model.add(Conv2D(32, (3, 3), activation='relu', padding='same', strides=2))
model.add(Conv2D(64, (3, 3), activation='relu', padding='same', strides=1))
model.add(Conv2D(64, (3, 3), activation='relu', padding='same', strides=2))
model.add(Conv2D(128, (3, 3), activation='relu', padding='same', strides=1))
model.add(Conv2D(128, (3, 3), activation='relu', padding='same', strides=1))
model.add(Conv2D(64, (3, 3), activation='relu', padding='same', strides=1))
model.add(Conv2D(32, (3, 3), activation='relu', padding='same', strides=1))
model.add(UpSampling2D((2, 2)))
model.add(Conv2D(8, (3, 3), activation='relu', padding='same'))
model.add(Conv2D(8, (3, 3), activation='relu', padding='same'))
model.add(UpSampling2D((2, 2)))
model.add(Conv2D(2, (3, 3), activation='relu', padding='same'))
model.add(Conv2D(2, (3, 3), activation='tanh', padding='same'))
model.add(UpSampling2D((2, 2)))
# Finish model
model.compile(optimizer='rmsprop',loss='mse', metrics=['acc'])
#Train the neural network
model.fit(x=X, y=Y, batch_size=100, epochs=1000)
print(model.evaluate(X, Y, batch_size=100))
output
Epoch 1/1000 1/1 [==============================] - 7s 7s/step - loss: 0.0214 - acc: 0.4987
Epoch 2/1000 1/1 [==============================] - 7s 7s/step - loss: 0.0214 - acc:
0.4987
Epoch 3/1000 1/1 [==============================] - 9s 9s/step - loss: 0.0214 - acc: 0.4987
Epoch 4/1000 1/1 [==============================] - 8s 8s/step - loss: 0.0214 - acc:
0.4987 . . . .

Answer 1

First of all, I have simplified the image loading code, and also normalized (subtract mean, divide by standard deviation) all channels (L, A, B) separately, also renamed the variables, that usually helps a lot. (5 minute free Coursera video about normalizing inputs (will bug you to subscribe but just click that away.).) So the loading part now looks like this:
# Import images
MODEL_NAME = 'forest'
imgLABs = []
for filename in os.listdir('./forest/'):
if (filename != '.DS_Store'):
image = img_to_array( load_img("./forest/" + filename) )
imgLABs.append( rgb2lab( image / 255.0 ) )
imgLABs_arr = np.array( imgLABs )
L, A, B = imgLABs_arr[ :, :, :, 0 : 1 ], imgLABs_arr[ :, :, :, 1 : 2 ], imgLABs_arr[ :, :, :, 2 : 3 ]
L_mean, L_std = np.mean( L ), np.std( L )
A_mean, A_std = np.mean( A ), np.std( A )
B_mean, B_std = np.mean( B ), np.std( B )
L, A, B = ( L - L_mean ) / L_std, ( A - A_mean ) / A_std, ( B - B_mean ) / B_std
AB = np.concatenate( ( A, B ), axis = 3)
Also changed the model around, added more feature depth, and a few max pool layers (don't forget to include them in the imports, not shown). Note that the activation function at the final few layers are set to None to allow for negative values, since we're expecting normalized results:
# Building the neural network
model = Sequential()
model.add(InputLayer( input_shape = L.shape[ 1: ] ) )
model.add(Conv2D(64, (3, 3), activation='relu', padding='same', strides=2,
kernel_initializer='truncated_normal'))
model.add(MaxPooling2D( (3, 3), strides = 1, padding='same' ) )
model.add(Conv2D(64, (3, 3), activation='relu', padding='same',
kernel_initializer='truncated_normal'))
model.add(Conv2D(64, (3, 3), activation='relu', padding='same', strides=2,
kernel_initializer='truncated_normal'))
model.add(Conv2D(64, (3, 3), activation='relu', padding='same', strides=1,
kernel_initializer='truncated_normal'))
model.add(Conv2D(64, (3, 3), activation='relu', padding='same', strides=2,
kernel_initializer='truncated_normal'))
model.add(MaxPooling2D( (3, 3), strides = 1, padding='same' ) )
model.add(Conv2D(64, (3, 3), activation='relu', padding='same', strides=1,
kernel_initializer='truncated_normal'))
model.add(Conv2D(64, (3, 3), activation='relu', padding='same', strides=1,
kernel_initializer='truncated_normal'))
model.add(Conv2D(64, (3, 3), activation='relu', padding='same', strides=1,
kernel_initializer='truncated_normal'))
model.add(Conv2D(64, (3, 3), activation='relu', padding='same', strides=1,
kernel_initializer='truncated_normal'))
model.add(UpSampling2D((2, 2)))
model.add(Conv2D(32, (3, 3), activation='relu', padding='same',
kernel_initializer='truncated_normal'))
model.add(Conv2D(32, (3, 3), activation='relu', padding='same',
kernel_initializer='truncated_normal'))
model.add(UpSampling2D((2, 2)))
model.add(Conv2D(32, (3, 3), activation=None, padding='same',
kernel_initializer='truncated_normal'))
model.add(Conv2D(2, (3, 3), activation=None, padding='same',
kernel_initializer='truncated_normal'))
model.add(UpSampling2D((2, 2)))
# Finish model
optimizer = optimizers.RMSprop( lr = 0.0005, decay = 1e-5 )
model.compile( optimizer=optimizer, loss='mse', metrics=['acc'] )
#Train the neural network
model.fit( x=L, y=AB, batch_size=1, epochs=1800 )
model.save("forest-model-v2.h5")
Note the learning rate of 0.0005, I've experimented with some values, this looked best. Then the learning rate decay can help later in the training, reducing the learning rate as we go along. Also, I've changed the batch_size to 1 - this is very specific to this network and is not generally recommended. But here you mostly have straight convolutions, so it makes sense to update the kernels after each exemplar, as every exemplar itself is affecting the weights from each pixel. But if you change the architecture, then this might not make sense any more, and you should change the batch size back. I've also increased the epochs to 1,800, because it runs fairly quickly on my machine and I had the time to run it. It reaches its maximum around 1,000 though.
With all that, here's the output from the training (first and last few lines only):
Epoch 1/1800
100/100 [==============================] - 6s 63ms/step - loss: 1.0554 - acc: 0.5217
Epoch 2/1800
100/100 [==============================] - 1s 13ms/step - loss: 1.1097 - acc: 0.5703
...
Epoch 1000/1800
100/100 [==============================] - 1s 13ms/step - loss: 0.0533 - acc: 0.9338
...
Epoch 1800/1800
100/100 [==============================] - 1s 13ms/step - loss: 0.0404 - acc: 0.9422
To print the re-colored image I used the following code, please note 5 is just an arbitrary index of the image I picked from the 100; also we need to add back the means and standard deviations for L, A and B (you have to treat these six numbers as part of your network, when you want to use it for actual recoloring - you need to preprocess the input with L_std, L_mean, and then postprocess the output with A, B means and std-s):
predicted = model.predict( x = L[ 5 : 6 ], batch_size = 1, verbose = 1 )
plt.imshow( lab2rgb( np.concatenate(
( ( L[ 5 ] * L_std ) + L_mean,
( predicted[ 0, :, :, 0 : 1 ] * A_std ) + A_mean,
( predicted[ 0, :, :, 1 : 2 ] * B_std ) + B_mean),
axis = 2 ) ) )
img_pred = lab2rgb( np.concatenate(
( ( L[ 5 ] * L_std ) + L_mean,
( predicted[ 0, :, :, 0 : 1 ] * A_std ) + A_mean,
( predicted[ 0, :, :, 1 : 2 ] * B_std ) + B_mean),
axis = 2 ) )
img_orig = lab2rgb( np.concatenate(
( ( L[ 5 ] * L_std ) + L_mean,
( A[ 5 ] * A_std ) + A_mean,
( B[ 5 ] * B_std ) + B_mean ),
axis = 2 ) )
diff = img_orig - img_pred
plt.imshow( diff * 10 )
And with all that the images are (original; greyscale network input; network output (colors restored); difference between original and restored):
Pretty neat! :) Mainly some of the detail on the mountains what's lost only. Since it's only 100 training images, it might be seriously overfitted, though. Still, I hope this gives you a good start!