Can KerasClassifier wtih TF model works with sklearn.cross_val_score when setting n_job=-1 and TF runs on a single GPU? - tensorflow

Question

I have this sample code and it can only runs with n_jobs=1.
Tensorflow backend is running on a GPU.
When I run with n_jobs=-1 on method cross_val_score, the program jams/stops working or give any output, after output 4 lines Epoch 1/100 (as I have a 4 core CPU I assume it will use all 4 cores to do CV and each trys to start a tf session on GPU)
from keras.wrappers.scikit_learn import KerasClassifier
from sklearn.model_selection import cross_val_score
from keras.models import Sequential
from keras.layers import Dense
from keras.layers import Dropout
def build_classifier():
classifier = Sequential()
classifier.add(Dense(units = 6, kernel_initializer = 'uniform', activation = 'relu', input_dim = 11))
classifier.add(Dropout(0.3))
classifier.add(Dense(units = 6, kernel_initializer = 'uniform', activation = 'relu'))
# classifier.add(Dropout(0.3))
classifier.add(Dense(units = 1, kernel_initializer = 'uniform', activation = 'sigmoid'))
classifier.compile(optimizer = 'adam', loss = 'binary_crossentropy', metrics = ['accuracy'])
return classifier
classifier = KerasClassifier(build_fn = build_classifier, batch_size = 100, epochs = 100, verbose=0)
accuracies = cross_val_score(estimator = classifier, X = X_train, y = y_train, cv = 10, n_jobs = 1)
I have also tried to limit the TF GPU usage in this way but n_job=-1 still won't work.
np.random.seed(123)
tf.set_random_seed(123)
config = tf.ConfigProto(inter_op_parallelism_threads=1)
config.gpu_options.per_process_gpu_memory_fraction = 0.1 # in my case this setting will use around 1G memory on GPU
set_session(tf.Session(config=config))

Answer 1

I have the same issue, I use the below lines of code
Configure GPU to use all the memory
config = tf.ConfigProto(allow_soft_placement=True)
config.gpu_options.per_process_gpu_memory_fraction = 1.0
set_session(tf.Session(config=config))
def build_classifier():
classifier = Sequential()
classifier.add(Dense(units = 6, kernel_initializer = 'uniform', activation = 'relu', input_dim = 11))
classifier.add(Dense(units = 6, kernel_initializer = 'uniform', activation = 'relu'))
classifier.add(Dense(units = 1, kernel_initializer = 'uniform', activation = 'sigmoid'))
classifier.compile(optimizer = 'adam', loss = 'binary_crossentropy', metrics = ['accuracy'])
return classifier
classifier = KerasClassifier(build_fn = build_classifier, batch_size = 10, epochs = 100)
accuracies = cross_val_score(estimator = classifier, X = X_train, y = y_train, cv = 10)
mean = accuracies.mean()
variance = accuracies.std()
then I removed the n_jobs = -1 then I tried to run it again and check the GPU utilization using GPU-Z below is a photo from the run.
Maybe your question is you don't feel the performance enhancement using GPU. To answer this question I run the same code with CPU and GPU.
GPU at least in my average experiment 3:1 CPU. I believe it should take less than the time but this is the max performance achieved.
You can also found some good discussions Run Keras with GPU