Training a Keras model for object location - tensorflow

Question

I am trying to use Keras to train a model for location a certain mountain peak in the images of a moving wide-angle camera. This is my training data, I have 329 images labeled:
The position does not change, so except for lens distortion and rotation, is always looks the same. Peace of cake for a human.
These plots are generated from my training data generator function to rule out any issues in that regard:
train_generator = traingui_generator([], frames_labeled, labels, batch_size)
im,la = next(train_generator)
fig = plt.figure(figsize=(16,10))
for n, (i,l) in enumerate(zip(im,la)):
a = plt.subplot(2,2,n+1)
plt.imshow(i)
plt.plot(l[0],l[1],'r+')
plt.xlim((1440,2880))
plt.ylim((0,1440))
a.invert_yaxis()
plt.show()
With the help of the Keras documentation examples, I came up with this model:
from keras import Input
from keras.models import Model
from keras.optimizers import Adam
from keras.layers import Cropping2D, Rescaling, Conv2D, MaxPooling2D, Flatten, Dense, AveragePooling2D,Resizing
inputs = Input(shape=input_shape)
x = Cropping2D(((0,0),(1440,0)))(inputs)
x = Rescaling(scale=1.0 / 255)(x)
x = Resizing(720,720)(x)
x = Conv2D(32, (3, 3), activation='relu')(x)
x = MaxPooling2D((2, 2))(x)
x = Conv2D(64, (3, 3), activation='relu')(x)
x = MaxPooling2D((2, 2))(x)
x = Flatten()(x)
x = Dense(64, activation='relu')(x)
location = Dense(2)(x)
# Define the model with the input and output layers
model = Model(inputs=inputs, outputs=location)
model.compile(optimizer=Adam(learning_rate=0.001),
loss='mse', metrics=['accuracy'])
batch_size = 4
num_train_samples = 329
num_epochs = 10
train_generator = traingui_generator([], frames_labeled, labels, batch_size)
model.fit(train_generator, steps_per_epoch=num_train_samples // batch_size, epochs=num_epochs)
This is my learning process:
27/27 [==============================] - 87s 3s/step - loss: 1126492.5000 - accuracy: 0.9877
Epoch 2/20
27/27 [==============================] - 87s 3s/step - loss: 750418.5000 - accuracy: 1.0000
Epoch 3/20
27/27 [==============================] - 87s 3s/step - loss: 702527.0625 - accuracy: 1.0000
Epoch 4/20
27/27 [==============================] - 87s 3s/step - loss: 651334.3125 - accuracy: 1.0000
Epoch 5/20
27/27 [==============================] - 87s 3s/step - loss: 591387.7500 - accuracy: 1.0000
Epoch 6/20
27/27 [==============================] - 88s 3s/step - loss: 495730.0625 - accuracy: 0.9722
Epoch 7/20
27/27 [==============================] - 87s 3s/step - loss: 322107.7500 - accuracy: 0.8981
Epoch 8/20
27/27 [==============================] - 87s 3s/step - loss: 213287.5312 - accuracy: 0.8981
Epoch 9/20
27/27 [==============================] - 87s 3s/step - loss: 151553.3281 - accuracy: 0.9475
Epoch 10/20
27/27 [==============================] - 87s 3s/step - loss: 114601.3828 - accuracy: 0.9506
Epoch 11/20
27/27 [==============================] - 87s 3s/step - loss: 96194.7031 - accuracy: 0.9475
Epoch 12/20
27/27 [==============================] - 88s 3s/step - loss: 69348.9922 - accuracy: 0.9321
Epoch 13/20
27/27 [==============================] - 87s 3s/step - loss: 65372.2852 - accuracy: 0.9475
Epoch 14/20
27/27 [==============================] - 88s 3s/step - loss: 58215.0547 - accuracy: 0.9043
Epoch 15/20
27/27 [==============================] - 87s 3s/step - loss: 57038.0078 - accuracy: 0.9475
Epoch 16/20
27/27 [==============================] - 87s 3s/step - loss: 47969.5234 - accuracy: 0.9660
Epoch 17/20
27/27 [==============================] - 87s 3s/step - loss: 45780.5820 - accuracy: 0.9383
Epoch 18/20
27/27 [==============================] - 88s 3s/step - loss: 39562.1836 - accuracy: 0.9660
Epoch 19/20
27/27 [==============================] - 87s 3s/step - loss: 51684.8164 - accuracy: 0.9537
Epoch 20/20
27/27 [==============================] - 88s 3s/step - loss: 45646.8398 - accuracy: 0.9815
I notice that the loss function is quite often increasing during the epochs and the accuracy isn't stable, so I already went down with the learning_rate by one order of magnitude.
Unfortunately, the result is not particularly good:
These plots are generated with the same piece of code as above, just replacing la with the output of model.predict(im).
Note that this on the training data, so I would rather rule out overfitting, if I understand correctly.
I have also tried just two convolutional layers, without success.
Now I read that I should modify the model, but I am lacking guidance in what direction.
Should I just play around randomly, each time waiting for it to finish? Where would I start to adjust? Filter number? Kernel size of convolution or pooling? Number of layers? Number pf epochs? Is the training data even enough?
Or is there something fundamentally wrong in what I am doing?
The size of the image is unfortunately not really open for discussion, I would even like to remove the resize because there will be other, smaller feature that I would like to detect, and add the other halve sphere, such that my final data would have 1440x2880 pixel..
If necessary, I would have access to quite powerful hardware, though, if RAM is an issue, and I would not have too much of a problem if the training took hours or more, if the problem REALLY requires it.
I would really appreciate if someone experienced could give me a push into the right direction for this concrete problem. I have no idea what to expect.
Edit: Given that my images are much larger than most examples and I read that the first layers are meant capture low frequency features, I increased the kernel size of the first layers
x = Conv2D(32, (15, 15), activation='relu')(x)
x = MaxPooling2D((2, 2))(x)
x = Conv2D(64, (5, 5), activation='relu')(x)
x = MaxPooling2D((2, 2))(x)
x = Conv2D(128, (3, 3), activation='relu')(x)
x = MaxPooling2D((2, 2))(x)
If anything, this made things worse.