CNN for automated car in GTA poor performance - tensorflow

Question

Recently I have been trying to build an automated car in gta v using a CNN model. I started out by collecting about 30k images from the game by driving around while capturing the scene and the key that was pressed at the current time. I also made sure to keep the dataset balanced by limiting the amount of data for each label to be equal.
An example of a random image in the dataset: IMAGE.
The labels are the basic driving inputs -LABELS.
Using this dataset on various models the accuracy never went above 50-60% on the validation test (test accuracy is even lower). Trying to fix this issue I tried cropping the images from the dataset to only include the center of the image which contains the road and drop the outlying data (scenery, buildings etc..). Also tried using RGB pictures as data instead of greyscale, also tested out collecting data from a specific location and testing it in the same place, different model architectures, different parameters and still no luck.
All the models were tested out in-game by constantly capturing the image of the road in-game and using it as an input to the model, then the output of the model would be the input for the game. All models seem to behave in the same general way which is basically outputting the same label – mostly ‘WA’, until it crashes into a wall.
I would love to get some tips on what I may be doing wrong or on what I can do to improve performance and let me know if you need any more information regarding this project to help out.
Thanks in advance.
TWO OF THE MODELS I TRIED:
model = Sequential()
model.add(Conv2D(filters=96, kernel_size=11, strides=4, activation='relu', input_shape=(IMAGE_HEIGHT, IMAGE_WIDTH, IMAGE_CHANNELS), padding='same'))
model.add(MaxPooling2D(pool_size=(3, 3), strides=2))
model.add(Conv2D(filters=256, kernel_size=5, activation='relu', padding='same'))
model.add(MaxPooling2D(pool_size=(3, 3), strides=2))
model.add(Conv2D(filters=384, kernel_size=3, activation='relu', padding='same'))
model.add(Conv2D(filters=384, kernel_size=3, activation='relu', padding='same'))
model.add(Conv2D(filters=256, kernel_size=3, activation='relu', padding='same'))
model.add(MaxPooling2D(pool_size=(3, 3), strides=2))
model.add(Dense(4096, activation='tanh'))
model.add(Dropout(0.5))
model.add(Dense(4096, activation='tanh'))
model.add(Dropout(0.5))
model.add(Flatten())
model.add(Dense(7, activation='sigmoid'))
model = Sequential()
model.add(Conv2D(filters=12, kernel_size=11, activation='relu', input_shape=(IMAGE_HEIGHT, IMAGE_WIDTH, IMAGE_CHANNELS), padding='same'))
model.add(MaxPooling2D(pool_size=(3,3)))
model.add(Conv2D(filters=256, kernel_size=5, activation='relu', padding='same'))
model.add(MaxPooling2D(pool_size=(3,3)))
model.add(Conv2D(filters=384, kernel_size=3, activation='relu', padding='same'))
model.add(Conv2D(filters=256, kernel_size=5, activation='relu', padding='same'))
model.add(MaxPooling2D(pool_size=(3,3)))
model.add(Flatten())
model.add(Dense(7, activation='sigmoid'))
The code:
filenames = os.listdir("dataset")
labels = []
for filename in filenames:
label = filename.split('.')[1]
labels.append(label)
df = pd.DataFrame({
'filename': filenames,
'category': labels
})
model = model1()
print(model.summary())
model.compile(loss='categorical_crossentropy', optimizer=Adam(learning_rate=0.001), metrics=['accuracy'])
train_df, validate_df = train_test_split(df, test_size=0.40, random_state=42)
train_df = train_df.reset_index(drop=True)
validate_df = validate_df.reset_index(drop=True)
total_train = train_df.shape[0]
total_validate = validate_df.shape[0]
batch_size=32
train_datagen = ImageDataGenerator(rescale=1./255)
train_generator = train_datagen.flow_from_dataframe(
train_df,
"dataset",
x_col='filename',
y_col='category',
target_size=IMAGE_SIZE,
class_mode='categorical',
batch_size=batch_size
)
validation_datagen = ImageDataGenerator( rescale=1./255,)
validation_generator = validation_datagen.flow_from_dataframe(
validate_df,
"dataset",
x_col='filename',
y_col='category',
target_size=IMAGE_SIZE,
class_mode='categorical',
batch_size=batch_size
)
epochs = 25
log_dir = "logs/fit/" + datetime.datetime.now().strftime("%Y%m%d-%H%M%S")
tensorboard_callback = tf.keras.callbacks.TensorBoard(log_dir=log_dir, histogram_freq=1)
history = model.fit(
train_generator,
epochs=epochs,
validation_data=validation_generator,
validation_steps=total_validate // batch_size,
steps_per_epoch=total_train // batch_size, shuffle=True, callbacks=[tensorboard_callback])
model.save("model.h5")