Converting tensorflow session-based code to distribute.Mirroredstrategy - tensorflow

Question

I'm pretty new to Tensorflow and I'll be the first to admit I'm a bit confused and turned around and might very well be barking up the wrong tree.
First: This is NOT a question about getting my GPUs working and seen by tensorflow(TF); I have verified from inside the container the GPU's are detected by TF. (using tensorflow/tensorflow:1.13.1-gpu-py3)
2020-02-20 22:24:25.233916: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-02-20 22:24:25.233933: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 0 1 2 3 4 5
2020-02-20 22:24:25.233939: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0: N Y Y Y Y Y
2020-02-20 22:24:25.233943: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 1: Y N Y Y Y Y
2020-02-20 22:24:25.233947: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 2: Y Y N Y Y Y
2020-02-20 22:24:25.233950: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 3: Y Y Y N Y Y
2020-02-20 22:24:25.233954: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 4: Y Y Y Y N Y
2020-02-20 22:24:25.233958: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 5: Y Y Y Y Y N
2020-02-20 22:24:25.234135: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 7623 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1070, pci bus id: 0000:01:00.0, compute capability: 6.1)
2020-02-20 22:24:25.234370: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 7624 MB memory) -> physical GPU (d
evice: 1, name: GeForce GTX 1070, pci bus id: 0000:02:00.0, compute capability: 6.1)
2020-02-20 22:24:25.234516: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:2 with 7624 MB memory) -> physical GPU (d
evice: 2, name: GeForce GTX 1070, pci bus id: 0000:04:00.0, compute capability: 6.1)
2020-02-20 22:24:25.234623: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:3 with 7624 MB memory) -> physical GPU (d
evice: 3, name: GeForce GTX 1070, pci bus id: 0000:05:00.0, compute capability: 6.1)
2020-02-20 22:24:25.234832: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:4 with 7624 MB memory) -> physical GPU (d
evice: 4, name: GeForce GTX 1070, pci bus id: 0000:07:00.0, compute capability: 6.1)
2020-02-20 22:24:25.234949: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:5 with 7624 MB memory) -> physical GPU (d
evice: 5, name: GeForce GTX 1070, pci bus id: 0000:08:00.0, compute capability: 6.1)
Second: The code as-is does work; I have successfully run the training model, but of course it only used the first GPU.
The code I'm using is a GAN project and uses a 'with' block for training:
with tf.Session() as session:
# Time stamp
localtime = time.asctime( time.localtime(time.time()) )
print("Starting TensorFlow session...")
print("Local current time :", localtime)
# Start TensorFlow session...
session.run(tf.global_variables_initializer())
.
.
.
I've been going in circles (and crazy) trying to figure out how to use the recommended tf.distribute.MirroredStrategy() to do parallel training across my GPUs. Everything I've come across so far leads in circles or stops short of applicable examples.
Is there a straightforward way to modify the session code to use the mirrored strategy? Is there just a more basic way to get the session calls to train across multiple GPUs?

Answer 1

It's doable, but the short answer to this is no. There's no straightforward way.
Everything I've been able to find requires a good understanding of tensorflow and a fair amount of work to port a 'stock' tensorflow session over to multi GPU. It requires running multiple sessions with assigned GPUs and figuring out how to coordinate the training data between everything.
Switching to the newer strategy paradigm (or keras multigpu) requires figuring out how to express the learning model in layers vs sessions. Again, something that requires a pretty solid handle on TF.
If you're starting from scratch, I'd say look into keras or the mirrored strategy from the beginning.
Managing sessions and coordinating data is a bit of a headache (thats why there's nice wrappers now) but it's been done a lot.
If you're a beginner like me let me save you some frustration; it's a big task either go a different way or buckle up no easy answer.
https://jhui.github.io/2017/03/07/TensorFlow-GPU/