Tensorflow 2.0.0 MirroredStrategy NCCL problem - tensorflow

Question

I'm trying to train with multi-gpu using tf.distribute.MirroredStrategy().
After several attempt to apply to my custom code, it has some error about NcclAllReduce.
So I copied mnist tutorial using tf.distribute from tensorflow page, running it has same error. logs and my environments are below
My env
sys.platform----------Window 10
Python----------3.7.6
Numpy----------1.18.1
TensorFlow----------2.0.0
TF CUDA support-----------True
GPU----------2 GPU, both are Quadro GV100
INFO:tensorflow:batch_all_reduce: 8 all-reduces with algorithm = nccl, num_packs = 1, agg_small_grads_max_bytes = 0 and agg_small_grads_max_group = 10
INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
INFO:tensorflow:batch_all_reduce: 8 all-reduces with algorithm = nccl, num_packs = 1, agg_small_grads_max_bytes = 0 and agg_small_grads_max_group = 10
INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
---------------------------------------------------------------------------
InvalidArgumentError Traceback (most recent call last)
<ipython-input-12-a7ead7e91ea5> in <module>
19 num_batches = 0
20 for x in train_dist_dataset:
---> 21 total_loss += distributed_train_step(x)
22 num_batches += 1
23 train_loss = total_loss / num_batches
~\Anaconda3\envs\tf-MSTO-DL\lib\site-packages\tensorflow_core\python\eager\def_function.py in __call__(self, *args, **kwds)
455
456 tracing_count = self._get_tracing_count()
--> 457 result = self._call(*args, **kwds)
458 if tracing_count == self._get_tracing_count():
459 self._call_counter.called_without_tracing()
~\Anaconda3\envs\tf-MSTO-DL\lib\site-packages\tensorflow_core\python\eager\def_function.py in _call(self, *args, **kwds)
518 # Lifting succeeded, so variables are initialized and we can run the
519 # stateless function.
--> 520 return self._stateless_fn(*args, **kwds)
521 else:
522 canon_args, canon_kwds = \
~\Anaconda3\envs\tf-MSTO-DL\lib\site-packages\tensorflow_core\python\eager\function.py in __call__(self, *args, **kwargs)
1821 """Calls a graph function specialized to the inputs."""
1822 graph_function, args, kwargs = self._maybe_define_function(args, kwargs)
-> 1823 return graph_function._filtered_call(args, kwargs) # pylint: disable=protected-access
1824
1825 #property
~\Anaconda3\envs\tf-MSTO-DL\lib\site-packages\tensorflow_core\python\eager\function.py in _filtered_call(self, args, kwargs)
1139 if isinstance(t, (ops.Tensor,
1140 resource_variable_ops.BaseResourceVariable))),
-> 1141 self.captured_inputs)
1142
1143 def _call_flat(self, args, captured_inputs, cancellation_manager=None):
~\Anaconda3\envs\tf-MSTO-DL\lib\site-packages\tensorflow_core\python\eager\function.py in _call_flat(self, args, captured_inputs, cancellation_manager)
1222 if executing_eagerly:
1223 flat_outputs = forward_function.call(
-> 1224 ctx, args, cancellation_manager=cancellation_manager)
1225 else:
1226 gradient_name = self._delayed_rewrite_functions.register()
~\Anaconda3\envs\tf-MSTO-DL\lib\site-packages\tensorflow_core\python\eager\function.py in call(self, ctx, args, cancellation_manager)
509 inputs=args,
510 attrs=("executor_type", executor_type, "config_proto", config),
--> 511 ctx=ctx)
512 else:
513 outputs = execute.execute_with_cancellation(
~\Anaconda3\envs\tf-MSTO-DL\lib\site-packages\tensorflow_core\python\eager\execute.py in quick_execute(op_name, num_outputs, inputs, attrs, ctx, name)
65 else:
66 message = e.message
---> 67 six.raise_from(core._status_to_exception(e.code, message), None)
68 except TypeError as e:
69 keras_symbolic_tensors = [
~\Anaconda3\envs\tf-MSTO-DL\lib\site-packages\six.py in raise_from(value, from_value)
InvalidArgumentError: No OpKernel was registered to support Op 'NcclAllReduce' used by {{node Adam/NcclAllReduce}}with these attrs: [reduction="sum", shared_name="c1", T=DT_FLOAT, num_devices=2]
Registered devices: [CPU, GPU]
Registered kernels:
<no registered kernels>
[[Adam/NcclAllReduce]] [Op:__inference_distributed_train_step_1755]

Answer 1

There are several options for cross_device_ops, it seems
strategy = tf.distribute.MirroredStrategy(
cross_device_ops=tf.distribute.NcclAllReduce())
would produce a NCCL error depending on your architecture and configuration.
This option was meant for NVIDIA DGX-1 architecture and might underperform on other architectures :
strategy = tf.distribute.MirroredStrategy(
cross_device_ops=tf.distribute.HierarchicalCopyAllReduce())
Should work :
strategy = tf.distribute.MirroredStrategy(
cross_device_ops=tf.distribute.ReductionToOneDevice())
So that it can be advised to try the different options.
See Tensorflow Multi-GPU - NCCL

Answer 2

NCCL drivers do not work with Windows. To my knowledge they only work with Linux. I have read that there might be a NCCL driver equivalent for Windows but have not been able to find them myself. If you want to stay with Windows you can try HierarchicalCopyAllReduce.
mirrored_strategy = tf.distribute.MirroredStrategy(devices=["/job:localhost/replica:0/task:0/device:GPU:0", "/job:localhost/replica:0/task:0/device:GPU:1"],
cross_device_ops = tf.distribute.HierarchicalCopyAllReduce())
with mirrored_strategy.scope():

Answer 3

You can try:
strategy = tf.distribute.MultiWorkerMirroredStrategy()
For me (1 CPU - 3 GPU setup on Windows WSL2) it's faster than any of the other methods.

Answer 4

I had this exact problem with tensorflow 2.2.0 when using tf.distribbute.MirrorStrategy and found that I was running the wrong version of Cuda - I had been runing 10.0 and it should have been cuda 10.1.
I think worth checking which version you are running and if it's right.
For tensorflow 2.0.0 you should be using cuda 10.0. See here for combinations between the two libraries