Keras model classifies images as the same class - tensorflow

Question

I am using Keras' pre-trained VGGNet16 as my base model, but I need to add layers on the end to make it work for my data. I've got the data pre-processed and formatted, so I'll jump to the part of the code actually involving the CNN.
import tensorflow as tf
from tensorflow.keras.layers import Dense, Dropout, Activation, Flatten, Conv2D, MaxPooling2D, AveragePooling2D
from tensorflow.keras.utils import to_categorical
from tensorflow.keras.models import Sequential
from tensorflow.keras.optimizers import Adam
import tensorflow.keras as K
.
.
.
input_t = K.Input(shape=(224,224,3))
base_model = K.applications.VGG16(include_top=False,weights="imagenet",input_tensor=input_t)
for layer in base_model.layers:
layer.trainable = False
model = K.models.Sequential()
model.add(base_model)
model.add(Flatten())
model.add(Dense(num_classes,activation='softmax'))
#Training
model.compile(loss = 'binary_crossentropy',
optimizer = Adam(learning_rate=1e-5),
metrics = ['accuracy'])
#Fit
model.fit(x_train,
y_train,
epochs = 3,
validation_split = 0.1,
validation_data =(x_test,y_test))
model.summary()
This code yields the following output.
Epoch 1/3
14/14 [==============================] - 48s 3s/step - loss: 0.5887 - accuracy: 0.2287 - val_loss: 0.4951 - val_accuracy: 0.3000
Epoch 2/3
14/14 [==============================] - 48s 3s/step - loss: 0.5170 - accuracy: 0.2220 - val_loss: 0.4972 - val_accuracy: 0.2800
Epoch 3/3
14/14 [==============================] - 48s 3s/step - loss: 0.4982 - accuracy: 0.2265 - val_loss: 0.4975 - val_accuracy: 0.2200
Model: "sequential"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
vgg16 (Functional) (None, 7, 7, 512) 14714688
_________________________________________________________________
flatten (Flatten) (None, 25088) 0
_________________________________________________________________
dense (Dense) (None, 5) 125445
=================================================================
Total params: 14,840,133
Trainable params: 125,445
Non-trainable params: 14,714,688
_________________________________________________________________
Test loss: 0.4997282326221466
Test accuracy: 0.2720000147819519
All of my test images are being classified into the same class. Is there any reason for this?
Edit: Modified question upon realizing the model is running properly

Answer 1

Perhaps this isn't a great solution, but I fixed this by making the last six layers trainable with the others trainable
for layer in base_model.layers[:-6]:
layer.trainable = False