Keras / Tensorflow not utilising all CPU cores - tensorflow

Question

I have a rather straightforward script for training and validating. I'm using tensorflow-gpu and I can see GPU:0 being used. However, the python process itself appears to be using just a single core with just around 90% utilisation. My GPU isn't getting maxed out during training either. It gets fully utilised during validation, however.
I wonder whether the use of a single core is preventing the GPU from being utilised more. Is there a way to use more CPU cores? I've tried setting config.intra_op_parallelism_threads = 4, but still only a single core is used.
Here's my script:
import model
from keras.optimizers import SGD
from keras.callbacks import EarlyStopping, ModelCheckpoint
from keras.preprocessing.image import ImageDataGenerator
from visual_callbacks import AccLossPlotter
import numpy as np
def main():
np.random.seed(45)
nb_class = 2
width, height = 224, 224
sn = model.SqueezeNet(nb_classes=nb_class, inputs=(3, height, width))
print('Build model')
sgd = SGD(lr=0.001, decay=0.0002, momentum=0.9, nesterov=True)
sn.compile(
optimizer=sgd, loss='categorical_crossentropy', metrics=['accuracy'])
print(sn.summary())
# Training
train_data_dir = 'data/train'
validation_data_dir = 'data/validation'
nb_train_samples = 2000
nb_validation_samples = 800
nb_epoch = 500
# Generator
train_datagen = ImageDataGenerator(
rescale=1./255,
shear_range=0.2,
zoom_range=0.2,
horizontal_flip=True)
#train_datagen = ImageDataGenerator(rescale=1./255)
test_datagen = ImageDataGenerator(rescale=1./255)
train_generator = train_datagen.flow_from_directory(
train_data_dir,
target_size=(width, height),
batch_size=32,
class_mode='categorical')
validation_generator = test_datagen.flow_from_directory(
validation_data_dir,
target_size=(width, height),
batch_size=32,
class_mode='categorical')
# Instantiate AccLossPlotter to visualise training
plotter = AccLossPlotter(graphs=['acc', 'loss'], save_graph=True)
early_stopping = EarlyStopping(monitor='val_loss', patience=3, verbose=0)
checkpoint = ModelCheckpoint(
'weights.{epoch:02d}-{val_loss:.2f}.h5',
monitor='val_loss',
verbose=0,
save_best_only=True,
save_weights_only=True,
mode='min',
period=1)
sn.fit_generator(
train_generator,
samples_per_epoch=nb_train_samples,
nb_epoch=nb_epoch,
validation_data=validation_generator,
nb_val_samples=nb_validation_samples,
callbacks=[plotter, checkpoint])
sn.save_weights('weights.h5')
if __name__ == '__main__':
main()
input('Press ENTER to exit...')

Answer 1

You can not utilize both CPU as well as GPU simultaneously. When you are using GPU for computation, your CPU not doing the actual computation, it is only doing the book-keeping job for GPU kernels. And for doing book-keeping, CPU does not have to utilize all the cores (single core is enough).
My GPU isn't getting maxed out during training either. It gets fully utilised during validation, however.
That is because during training you are calculating the gradients and doing back-prop which are not massively parallel processes compare to simple forward pass(you have to update weights after every batch forward pass). And so those can not fully utilize the GPU. But during the validation you only calculating the forward pass and that is why during validation GPU's are fully utilized.
Although you may get more GPU utilization if you increase the batch_size.