How can I use Convolutional LSTM in Keras to realize position estimation? - tensorflow

Question

I used LSTM in Keras with Tensorflow.
I would like to realize position estimation.
I want to input the movie (1 scene is 15 frame) and estimate the position of moving the square in the movie.
Input is 15 frame. Output is 2 variable (x, y).
In the following code the estimation accuracy is too bad. What should I do?
And, I don't understand AveragePooling3D/Reshape (Without this it will not perform.).
# We create a layer which take as input movies of shape
# (n_frames, width, height, channels) and returns a movie
# of identical shape.
seq = Sequential()
seq.add(ConvLSTM2D(filters=40, kernel_size=(3, 3),
input_shape=(None, 80, 80, 1),
padding='same', return_sequences=True))
seq.add(BatchNormalization())
seq.add(ConvLSTM2D(filters=40, kernel_size=(3, 3),
padding='same', return_sequences=True))
seq.add(BatchNormalization())
seq.add(ConvLSTM2D(filters=40, kernel_size=(3, 3),
padding='same', return_sequences=True))
seq.add(BatchNormalization())
seq.add(ConvLSTM2D(filters=40, kernel_size=(3, 3),
padding='same', return_sequences=True))
seq.add(BatchNormalization())
#seq.add(Flatten())
seq.add(AveragePooling3D((1, 80, 80)))
seq.add(Reshape((-1, 40)))
seq.add(Dense(2))
#seq.add(Conv3D(filters=1, kernel_size=(3, 3, 3),
# activation='sigmoid',
# padding='same', data_format='channels_last'))
seq.compile(loss='mean_squared_error', optimizer='adam')
def generate_movies(n_samples=1200, n_frames=15):
row = 80
col = 80
noisy_movies = np.zeros((n_samples, n_frames, row, col, 1), dtype=np.float)
shifted_movies = np.zeros((n_samples, n_frames, row, col, 1),
dtype=np.float)
square_x_y = np.zeros((n_samples, n_frames, 2), dtype=np.float)
for i in range(n_samples):
for j in range(1):
# Initial position
xstart = np.random.randint(20, 60)
ystart = np.random.randint(20, 60)
# Direction of motion
directionx = np.random.randint(0, 3) - 1
directiony = np.random.randint(0, 3) - 1
# Size of the square
w = np.random.randint(2, 4)
for t in range(n_frames):
x_shift = xstart + directionx * t
y_shift = ystart + directiony * t
noisy_movies[i, t, x_shift - w: x_shift + w,
y_shift - w: y_shift + w, 0] += 1
# Make it more robust by adding noise.
# The idea is that if during inference,
# the value of the pixel is not exactly one,
# we need to train the network to be robust and still
# consider it as a pixel belonging to a square.
if np.random.randint(0, 2):
noise_f = (-1)**np.random.randint(0, 2)
noisy_movies[i, t,
x_shift - w - 1: x_shift + w + 1,
y_shift - w - 1: y_shift + w + 1,
0] += noise_f * 0.1
# Shift the ground truth by 1
x_shift = xstart + directionx * (t + 1)
y_shift = ystart + directiony * (t + 1)
shifted_movies[i, t, x_shift - w: x_shift + w,
y_shift - w: y_shift + w, 0] += 1
square_x_y[i, t, 0] = x_shift/row
square_x_y[i, t, 1] = y_shift/col
# Cut to a 40x40 window
#noisy_movies = noisy_movies[::, ::, 20:60, 20:60, ::]
#shifted_movies = shifted_movies[::, ::, 20:60, 20:60, ::]
#noisy_movies[noisy_movies >= 1] = 1
#shifted_movies[shifted_movies >= 1] = 1
return noisy_movies, shifted_movies, square_x_y
# Train the network
noisy_movies, shifted_movies, sq_x_y = generate_movies(n_samples = 1200)
seq.fit(noisy_movies[:1000], sq_x_y[:1000], batch_size=10,
epochs=1, validation_split=0.05)

Answer 1

See your seq.summary() to understand the shapes.
Your last ConvLSTM2D layer is outputting something with shape
(movies, length, side1, side2, channels)
(None, None, 80, 80, 40)
For the Dense layer to work, you will need to keep the "movies" and the "length" dimension, and collapse the others into one.
Notice that the dimension 40 (the channels) is important for it represents different "concepts", while the 80s (the sides) are purely 2D positional.
Since you don't want to touch the "length" dimension, you need an AveragePooling2D (not 3D). But since you're going to detect a very distinct positional feature and its location, I suggest you don't collapse the spatial dimensions at all. It should be better to just reshape and add a Dense that will take into account these positions.
So instead of the AveragePooling I suggest you use:
seq.add(Reshape((-1, 80*80*40)) #new shape is = (movies, length, 80*80*40)
Then you add a Dense layer that still has a notion of the positions:
seq.add(Dense(2)) #output shape is (movies, length, 2)
This is not guaranteed to make your model perform well, but I believe "better".
Other things you can do are adding more dense layers and smoothly go from the 80*80*40 features to 2.
This is more complex, but you might study the functional API model and learn about "U-nets" and try to make something like a u-net there.