Related
How can I write code DSSIM+MAE loss function from formula:
Loss = αMAE + (1-α)DSSIM
with
Mean Absolute Error(MAE) = (1/M) * ∑|yi – xi|
DSSIM = 1-SSIM
SSIM = (numerator1 * numerator2) / (denominator1 * denominator2)
numerator1 = 2 * μ12 + C1 #μ12 = μ1 * μ2
numerator2 = 2 * σ12 + C2
denominator1 = μ1_sq + μ2_sq + C1
denominator2 = σ1_sq + σ2_sq + C2
where α is a trade-off parameter between MAE and DSSIM,
M is the total number of pixels in the image,
μ is the mean value of the image,
σ is the standard variation of the image, and σx,y is the covariance of x and y two images. c1 and c2 are two variables that stabilize
the division with a weak denominator. In our implementation,
I set α = 0.75, c1 = (0.01L)^2 and c2 = (0.03L)^2
where L is the dynamic range of the pixel values in the image.
So, this is my code
def custom_loss (y_true,y_pred):
M = 512 #M = total number of pixels in the sCT image
sum = 0
y_pred = tf.cast(y_pred, tf.int32)
y_true = tf.cast(y_true, tf.int32)
print(y_pred.shape)
print(y_true.shape)
y_pred = y_pred[0]
y_true = y_true[0]
for i in range(n):
sum = sum+abs(y_true[i] - y_pred[i])
my_mae = sum / n
dssim = tf.reduce_mean((1 - tf.image.ssim(y_true,y_pred, max_val=512,
filter_size=11,filter_sigma=1.5, k1=0.01, k2=0.03)) / 2)
my_mae = tf.cast(my_mae, tf.float32)
return (0.75*my_mae) + (1 - 0.75*dssim)
it have error when I run
model.compile(optimizer='adam',loss= custom_loss,metrics=['accuracy'])
error is
Traceback (most recent call last):
File "C:/Users/CRA01/Desktop/Unet/custom loss.py", line 85, in
history = model.fit(Training_CBCT_dataset,Training_pCT_dataset,validation_split=0.2,batch_size=1, epochs=5, callbacks=[model_save_callback])
File "C:\Users\CRA01\miniconda3\envs\tf_2.9\lib\site-packages\keras\utils\traceback_utils.py", line 67, in error_handler
raise e.with_traceback(filtered_tb) from None
File "C:\Users\CRA01\AppData\Local\Temp_autograph_generated_fileqb1dimxg.py", line 15, in tf__train_function
retval = ag__.converted_call(ag__.ld(step_function), (ag__.ld(self), ag__.ld(iterator)), None, fscope)
ValueError: in user code:
File "C:\Users\CRA01\miniconda3\envs\tf_2.9\lib\site-packages\keras\engine\training.py", line 1051, in train_function *
return step_function(self, iterator)
File "C:\Users\CRA01\miniconda3\envs\tf_2.9\lib\site-packages\keras\engine\training.py", line 1040, in step_function **
outputs = model.distribute_strategy.run(run_step, args=(data,))
File "C:\Users\CRA01\miniconda3\envs\tf_2.9\lib\site-packages\keras\engine\training.py", line 1030, in run_step **
outputs = model.train_step(data)
File "C:\Users\CRA01\miniconda3\envs\tf_2.9\lib\site-packages\keras\engine\training.py", line 893, in train_step
self.optimizer.minimize(loss, self.trainable_variables, tape=tape)
File "C:\Users\CRA01\miniconda3\envs\tf_2.9\lib\site-packages\keras\optimizers\optimizer_v2\optimizer_v2.py", line 539, in minimize
return self.apply_gradients(grads_and_vars, name=name)
File "C:\Users\CRA01\miniconda3\envs\tf_2.9\lib\site-packages\keras\optimizers\optimizer_v2\optimizer_v2.py", line 640, in apply_gradients
grads_and_vars = optimizer_utils.filter_empty_gradients(grads_and_vars)
File "C:\Users\CRA01\miniconda3\envs\tf_2.9\lib\site-packages\keras\optimizers\optimizer_v2\utils.py", line 73, in filter_empty_gradients
raise ValueError(f"No gradients provided for any variable: {variable}. "
ValueError: No gradients provided for any variable: (['conv2d/kernel:0', 'conv2d/bias:0', 'conv2d_1/kernel:0', 'conv2d_1/bias:0', 'conv2d_2/kernel:0', 'conv2d_2/bias:0', 'conv2d_3/kernel:0', 'conv2d_3/bias:0', 'conv2d_4/kernel:0', 'conv2d_4/bias:0', 'conv2d_5/kernel:0', 'conv2d_5/bias:0', 'conv2d_6/kernel:0', 'conv2d_6/bias:0', 'conv2d_7/kernel:0', 'conv2d_7/bias:0', 'conv2d_8/kernel:0', 'conv2d_8/bias:0', 'conv2d_9/kernel:0', 'conv2d_9/bias:0', 'conv2d_10/kernel:0', 'conv2d_10/bias:0', 'conv2d_11/kernel:0', 'conv2d_11/bias:0', 'conv2d_12/kernel:0', 'conv2d_12/bias:0', 'conv2d_13/kernel:0', 'conv2d_13/bias:0', 'conv2d_14/kernel:0', 'conv2d_14/bias:0', 'conv2d_15/kernel:0', 'conv2d_15/bias:0', 'conv2d_16/kernel:0', 'conv2d_16/bias:0', 'conv2d_17/kernel:0', 'conv2d_17/bias:0', 'conv2d_18/kernel:0', 'conv2d_18/bias:0', 'conv2d_19/kernel:0', 'conv2d_19/bias:0', 'conv2d_20/kernel:0', 'conv2d_20/bias:0', 'conv2d_21/kernel:0', 'conv2d_21/bias:0', 'conv2d_22/kernel:0', 'conv2d_22/bias:0', 'conv2d_23/kernel:0', 'conv2d_23/bias:0', 'conv2d_24/kernel:0', 'conv2d_24/bias:0', 'conv2d_25/kernel:0', 'conv2d_25/bias:0', 'conv2d_26/kernel:0', 'conv2d_26/bias:0'],). Provided `grads_and_vars` is ((None, <tf.Variable 'conv2d/kernel:0' shape=(3, 3, 1, 32) dtype=float32>), (None, <tf.Variable 'conv2d/bias:0' shape=(32,) dtype=float32>), (None, <tf.Variable 'conv2d_1/kernel:0' shape=(3, 3, 32, 32) dtype=float32>), (None, <tf.Variable 'conv2d_1/bias:0' shape=(32,) dtype=float32>), (None, <tf.Variable 'conv2d_2/kernel:0' shape=(3, 3, 32, 64) dtype=float32>), (None, <tf.Variable 'conv2d_2/bias:0' shape=(64,) dtype=float32>), (None, <tf.Variable 'conv2d_3/kernel:0' shape=(3, 3, 64, 64) dtype=float32>), (None, <tf.Variable 'conv2d_3/bias:0' shape=(64,) dtype=float32>), (None, <tf.Variable 'conv2d_4/kernel:0' shape=(3, 3, 64, 128) dtype=float32>), (None, <tf.Variable 'conv2d_4/bias:0' shape=(128,) dtype=float32>), (None, <tf.Variable 'conv2d_5/kernel:0' shape=(3, 3, 128, 128) dtype=float32>), (None, <tf.Variable 'conv2d_5/bias:0' shape=(128,) dtype=float32>), (None, <tf.Variable 'conv2d_6/kernel:0' shape=(3, 3, 128, 256) dtype=float32>), (None, <tf.Variable 'conv2d_6/bias:0' shape=(256,) dtype=float32>), (None, <tf.Variable 'conv2d_7/kernel:0' shape=(3, 3, 256, 256) dtype=float32>), (None, <tf.Variable 'conv2d_7/bias:0' shape=(256,) dtype=float32>), (None, <tf.Variable 'conv2d_8/kernel:0' shape=(3, 3, 256, 512) dtype=float32>), (None, <tf.Variable 'conv2d_8/bias:0' shape=(512,) dtype=float32>), (None, <tf.Variable 'conv2d_9/kernel:0' shape=(3, 3, 512, 512) dtype=float32>), (None, <tf.Variable 'conv2d_9/bias:0' shape=(512,) dtype=float32>), (None, <tf.Variable 'conv2d_10/kernel:0' shape=(3, 3, 512, 1024) dtype=float32>), (None, <tf.Variable 'conv2d_10/bias:0' shape=(1024,) dtype=float32>), (None, <tf.Variable 'conv2d_11/kernel:0' shape=(3, 3, 1024, 1024) dtype=float32>), (None, <tf.Variable 'conv2d_11/bias:0' shape=(1024,) dtype=float32>), (None, <tf.Variable 'conv2d_12/kernel:0' shape=(3, 3, 1024, 2048) dtype=float32>), (None, <tf.Variable 'conv2d_12/bias:0' shape=(2048,) dtype=float32>), (None, <tf.Variable 'conv2d_13/kernel:0' shape=(3, 3, 2048, 2048) dtype=float32>), (None, <tf.Variable 'conv2d_13/bias:0' shape=(2048,) dtype=float32>), (None, <tf.Variable 'conv2d_14/kernel:0' shape=(3, 3, 3072, 1024) dtype=float32>), (None, <tf.Variable 'conv2d_14/bias:0' shape=(1024,) dtype=float32>), (None, <tf.Variable 'conv2d_15/kernel:0' shape=(3, 3, 1024, 1024) dtype=float32>), (None, <tf.Variable 'conv2d_15/bias:0' shape=(1024,) dtype=float32>), (None, <tf.Variable 'conv2d_16/kernel:0' shape=(3, 3, 1536, 512) dtype=float32>), (None, <tf.Variable 'conv2d_16/bias:0' shape=(512,) dtype=float32>), (None, <tf.Variable 'conv2d_17/kernel:0' shape=(3, 3, 512, 512) dtype=float32>), (None, <tf.Variable 'conv2d_17/bias:0' shape=(512,) dtype=float32>), (None, <tf.Variable 'conv2d_18/kernel:0' shape=(3, 3, 768, 256) dtype=float32>), (None, <tf.Variable 'conv2d_18/bias:0' shape=(256,) dtype=float32>), (None, <tf.Variable 'conv2d_19/kernel:0' shape=(3, 3, 256, 256) dtype=float32>), (None, <tf.Variable 'conv2d_19/bias:0' shape=(256,) dtype=float32>), (None, <tf.Variable 'conv2d_20/kernel:0' shape=(3, 3, 384, 128) dtype=float32>), (None, <tf.Variable 'conv2d_20/bias:0' shape=(128,) dtype=float32>), (None, <tf.Variable 'conv2d_21/kernel:0' shape=(3, 3, 128, 128) dtype=float32>), (None, <tf.Variable 'conv2d_21/bias:0' shape=(128,) dtype=float32>), (None, <tf.Variable 'conv2d_22/kernel:0' shape=(3, 3, 192, 64) dtype=float32>), (None, <tf.Variable 'conv2d_22/bias:0' shape=(64,) dtype=float32>), (None, <tf.Variable 'conv2d_23/kernel:0' shape=(3, 3, 64, 64) dtype=float32>), (None, <tf.Variable 'conv2d_23/bias:0' shape=(64,) dtype=float32>), (None, <tf.Variable 'conv2d_24/kernel:0' shape=(3, 3, 96, 32) dtype=float32>), (None, <tf.Variable 'conv2d_24/bias:0' shape=(32,) dtype=float32>), (None, <tf.Variable 'conv2d_25/kernel:0' shape=(3, 3, 32, 32) dtype=float32>), (None, <tf.Variable 'conv2d_25/bias:0' shape=(32,) dtype=float32>), (None, <tf.Variable 'conv2d_26/kernel:0' shape=(1, 1, 32, 1) dtype=float32>), (None, <tf.Variable 'conv2d_26/bias:0' shape=(1,) dtype=float32>)).
Using tensorflow, I'm trying to reimplement the following architecture (for now I'm focusing on the Generator part):
What I've done for now has been defining the generator in the following way:
N_Z = 128
generator = [
tf.keras.layers.Dense(units=6144, activation="relu"),
tf.keras.layers.Reshape(target_shape=(6, 4, 256)),
tf.keras.layers.Conv2DTranspose(
filters=128, kernel_size=(5,5), strides=(2, 2), padding="SAME", activation="relu"
),
tf.keras.layers.Conv2DTranspose(
filters=128, kernel_size=(3,3), strides=(2, 1), padding="SAME", activation="relu"
),
tf.keras.layers.Conv2DTranspose(
filters=64, kernel_size=(3,3), strides=(1, 1), padding="SAME", activation="relu"
),
tf.keras.layers.Conv2DTranspose(
filters=64, kernel_size=(3,3), strides=(2, 1), padding="SAME", activation="relu"
),
tf.keras.layers.Conv2DTranspose(
filters=32, kernel_size=(3,3), strides=(1, 1), padding="SAME", activation="relu"
)
tf.keras.layers.Conv2DTranspose(
filters=32, kernel_size=(3,3), strides=(2, 1), padding="SAME", activation="relu"
)
tf.keras.layers.Conv2DTranspose(
filters=1, kernel_size=(3,3), strides=(1, 1), padding="SAME", activation="relu"
)
]
Generator = tf.keras.models.Sequential(generator)
But if I take some random noise and let the model process it, this is the final shape I get back:
noise = tf.random.normal((64,128))
result = Generator(noise)
result.shape
TensorShape([64, 28, 28, 1])
What am I doing wrong here? I was also checking the original implementation to see additional details but I can't find anything that makes me understand.
It is easy you need to see input-output, it required some help at the top levels.
[ Sample ]:
"""""""""""""""""""""""""""""""""""""""""""""""""""""""""
: Model Initialize
"""""""""""""""""""""""""""""""""""""""""""""""""""""""""
model = tf.keras.models.Sequential([
tf.keras.layers.InputLayer(input_shape=( 6144 )),
tf.keras.layers.Dense( 48 * 128, activation="linear" ),
tf.keras.layers.BatchNormalization( momentum=0.99, epsilon=0.00001 ),
tf.keras.layers.Reshape(target_shape=( 6, 4, 256 )),
tf.keras.layers.Conv2DTranspose(
filters=128, kernel_size=(5,5), strides=(2, 2), padding="same", activation="relu"
),
tf.keras.layers.Resizing( 11, 8, interpolation='bilinear', crop_to_aspect_ratio=False ),
tf.keras.layers.Reshape(target_shape=(11, 8, 128)),
tf.keras.layers.Conv2DTranspose(
filters=128, kernel_size=(3,3), strides=(2, 1), padding="SAME", activation="relu"
),
tf.keras.layers.Resizing( 22, 8, interpolation='bilinear', crop_to_aspect_ratio=False ),
tf.keras.layers.Reshape(target_shape=(22, 8, 128)),
tf.keras.layers.Conv2DTranspose(
filters=64, kernel_size=(3,3), strides=(1, 1), padding="SAME", activation="relu"
),
tf.keras.layers.Resizing( 22, 8, interpolation='bilinear', crop_to_aspect_ratio=False ),
tf.keras.layers.Reshape(target_shape=(22, 8, 64)),
tf.keras.layers.Conv2DTranspose(
filters=64, kernel_size=(3,3), strides=(2, 1), padding="SAME", activation="relu"
),
tf.keras.layers.Resizing( 43, 8, interpolation='bilinear', crop_to_aspect_ratio=False ),
tf.keras.layers.Reshape(target_shape=(43, 8, 64)),
tf.keras.layers.Conv2DTranspose(
filters=32, kernel_size=(3,3), strides=(1, 1), padding="SAME", activation="relu"
),
tf.keras.layers.Resizing( 43, 8, interpolation='bilinear', crop_to_aspect_ratio=False ),
tf.keras.layers.Reshape(target_shape=(43, 8, 32)),
tf.keras.layers.Conv2DTranspose(
filters=32, kernel_size=(3,3), strides=(2, 1), padding="SAME", activation="relu"
),
tf.keras.layers.Resizing( 85, 8, interpolation='bilinear', crop_to_aspect_ratio=False ),
tf.keras.layers.Reshape(target_shape=(85, 8, 32)),
])
model.summary()
[ Output ]:
Model: "sequential"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
dense (Dense) (None, 6144) 37754880
batch_normalization (BatchN (None, 6144) 24576
ormalization)
reshape (Reshape) (None, 6, 4, 256) 0
conv2d_transpose (Conv2DTra (None, 12, 8, 128) 819328
nspose)
resizing (Resizing) (None, 11, 8, 128) 0
reshape_1 (Reshape) (None, 11, 8, 128) 0
conv2d_transpose_1 (Conv2DT (None, 22, 8, 128) 147584
ranspose)
resizing_1 (Resizing) (None, 22, 8, 128) 0
reshape_2 (Reshape) (None, 22, 8, 128) 0
conv2d_transpose_2 (Conv2DT (None, 22, 8, 64) 73792
ranspose)
resizing_2 (Resizing) (None, 22, 8, 64) 0
reshape_3 (Reshape) (None, 22, 8, 64) 0
conv2d_transpose_3 (Conv2DT (None, 44, 8, 64) 36928
ranspose)
resizing_3 (Resizing) (None, 43, 8, 64) 0
reshape_4 (Reshape) (None, 43, 8, 64) 0
conv2d_transpose_4 (Conv2DT (None, 43, 8, 32) 18464
ranspose)
resizing_4 (Resizing) (None, 43, 8, 32) 0
reshape_5 (Reshape) (None, 43, 8, 32) 0
conv2d_transpose_5 (Conv2DT (None, 86, 8, 32) 9248
ranspose)
resizing_5 (Resizing) (None, 85, 8, 32) 0
reshape_6 (Reshape) (None, 85, 8, 32) 0
=================================================================
Total params: 38,884,800
Trainable params: 38,872,512
Non-trainable params: 12,288
_________________________________________________________________
2022-04-03 03:37:10.354570: I tensorflow/stream_executor/cuda/cuda_dnn.cc:368] Loaded cuDNN version 8100
(1, 85, 8, 32)
1/1 [==============================] - 2s 2s/step - loss: 0.0000e+00 - accuracy: 1.0000 - val_loss: 0.0000e+00 - val_accuracy: 1.0000
I'm new in TensorFlow and I'm trying to create a classifier using Keras. My training data is spitted into two files:
- one with training examples, each example is a vector of 64 floats
- second with labels, each label is an int within range (0,..,SIZE) (SIZE is 100) and it describes a class.
Both files are quire large and I can't fit them into memory so I've used tf.Dataset. I create two Datasets (one for features and one for labels) and them merge them using tf.data.Dataset.zip().
However during training I have "IndexError: list index out of range" error.
But when I print input data it looks fine.
This is the code:
model = tf.keras.Sequential()
model.add(layers.Dense(256, activation='relu'))
model.add(layers.Dense(64, activation='relu'))
model.add(layers.Dense(SIZE, activation='softmax'))
opt = tf.keras.optimizers.Adagrad()
model.compile(optimizer=opt,
loss='categorical_crossentropy',
metrics=['accuracy'])
#read data to: data and label
dataset = tf.data.Dataset.zip((data, label))
#iterator = dataset.make_one_shot_iterator()
#next_element = iterator.get_next()
#print(next_element[0])
#print("\n\n")
#print(next_element[1])
model.fit(dataset, epochs=50)
The error message is:
Epoch 1/50
Traceback (most recent call last):
File "<ipython-input-129-f200a4503ff9>", line 1, in <module>
runfile('D:/ai/collab-filter/dan.py', wdir='D:/ai/collab-filter')
File "C:\Users\DELL\Anaconda3\lib\site-packages\spyder_kernels\customize\spydercustomize.py", line 786, in runfile
execfile(filename, namespace)
File "C:\Users\DELL\Anaconda3\lib\site-packages\spyder_kernels\customize\spydercustomize.py", line 110, in execfile
exec(compile(f.read(), filename, 'exec'), namespace)
File "D:/ai/myModel.py", line 256, in <module>
model.fit(dataset, epochs=50)
File "C:\Users\DELL\Anaconda3\lib\site-packages\tensorflow\python\keras\engine\training.py", line 694, in fit
initial_epoch=initial_epoch)
File "C:\Users\DELL\Anaconda3\lib\site-packages\tensorflow\python\keras\engine\training.py", line 1433, in fit_generator
steps_name='steps_per_epoch')
File "C:\Users\DELL\Anaconda3\lib\site-packages\tensorflow\python\keras\engine\training_generator.py", line 256, in model_iteration
batch_size = int(nest.flatten(batch_data)[0].shape[0])
File "C:\Users\DELL\Anaconda3\lib\site-packages\tensorflow\python\framework\tensor_shape.py", line 868, in __getitem__
return self._dims[key]
IndexError: list index out of range
When I uncomment printing of the example I get (for me it looks good):
(<tf.Tensor: id=281899, shape=(), dtype=float32, numpy=0.0011093982>, <tf.Tensor: id=281900, shape=(), dtype=float32, numpy=0.008290171>, <tf.Tensor: id=281901, shape=(), dtype=float32, numpy=0.010696268>, <tf.Tensor: id=281902, shape=(), dtype=float32, numpy=-0.0068962937>, <tf.Tensor: id=281903, shape=(), dtype=float32, numpy=0.0020472356>, <tf.Tensor: id=281904, shape=(), dtype=float32, numpy=0.0041239075>, <tf.Tensor: id=281905, shape=(), dtype=float32, numpy=-0.0018036675>, <tf.Tensor: id=281906, shape=(), dtype=float32, numpy=-0.007521228>, <tf.Tensor: id=281907, shape=(), dtype=float32, numpy=0.012179799>, <tf.Tensor: id=281908, shape=(), dtype=float32, numpy=-0.008569455>, <tf.Tensor: id=281909, shape=(), dtype=float32, numpy=-0.005547243>, <tf.Tensor: id=281910, shape=(), dtype=float32, numpy=-0.024963537>, <tf.Tensor: id=281911, shape=(), dtype=float32, numpy=-0.0047834134>, <tf.Tensor: id=281912, shape=(), dtype=float32, numpy=-0.0073425>, <tf.Tensor: id=281913, shape=(), dtype=float32, numpy=-0.0049664816>, <tf.Tensor: id=281914, shape=(), dtype=float32, numpy=0.0012769673>, <tf.Tensor: id=281915, shape=(), dtype=float32, numpy=-0.008846987>, <tf.Tensor: id=281916, shape=(), dtype=float32, numpy=0.002845391>, <tf.Tensor: id=281917, shape=(), dtype=float32, numpy=-0.0012304187>, <tf.Tensor: id=281918, shape=(), dtype=float32, numpy=-0.0073605254>, <tf.Tensor: id=281919, shape=(), dtype=float32, numpy=-0.019149099>, <tf.Tensor: id=281920, shape=(), dtype=float32, numpy=0.0053162603>, <tf.Tensor: id=281921, shape=(), dtype=float32, numpy=0.00018294304>, <tf.Tensor: id=281922, shape=(), dtype=float32, numpy=-0.007135446>, <tf.Tensor: id=281923, shape=(), dtype=float32, numpy=0.019139009>, <tf.Tensor: id=281924, shape=(), dtype=float32, numpy=0.0031176396>, <tf.Tensor: id=281925, shape=(), dtype=float32, numpy=0.016997647>, <tf.Tensor: id=281926, shape=(), dtype=float32, numpy=-0.017783713>, <tf.Tensor: id=281927, shape=(), dtype=float32, numpy=-0.0033694915>, <tf.Tensor: id=281928, shape=(), dtype=float32, numpy=0.02030162>, <tf.Tensor: id=281929, shape=(), dtype=float32, numpy=-0.01870913>, <tf.Tensor: id=281930, shape=(), dtype=float32, numpy=-0.0057595233>, <tf.Tensor: id=281931, shape=(), dtype=float32, numpy=0.013816875>, <tf.Tensor: id=281932, shape=(), dtype=float32, numpy=-0.00463876>, <tf.Tensor: id=281933, shape=(), dtype=float32, numpy=-0.023181098>, <tf.Tensor: id=281934, shape=(), dtype=float32, numpy=0.0064159813>, <tf.Tensor: id=281935, shape=(), dtype=float32, numpy=-0.0018356718>, <tf.Tensor: id=281936, shape=(), dtype=float32, numpy=0.014198529>, <tf.Tensor: id=281937, shape=(), dtype=float32, numpy=-0.019970264>, <tf.Tensor: id=281938, shape=(), dtype=float32, numpy=-0.013106668>, <tf.Tensor: id=281939, shape=(), dtype=float32, numpy=0.01739781>, <tf.Tensor: id=281940, shape=(), dtype=float32, numpy=-0.0075084846>, <tf.Tensor: id=281941, shape=(), dtype=float32, numpy=-0.007515852>, <tf.Tensor: id=281942, shape=(), dtype=float32, numpy=0.008860749>, <tf.Tensor: id=281943, shape=(), dtype=float32, numpy=0.011078904>, <tf.Tensor: id=281944, shape=(), dtype=float32, numpy=0.0031385398>, <tf.Tensor: id=281945, shape=(), dtype=float32, numpy=0.00069636817>, <tf.Tensor: id=281946, shape=(), dtype=float32, numpy=0.016473386>, <tf.Tensor: id=281947, shape=(), dtype=float32, numpy=0.010464343>, <tf.Tensor: id=281948, shape=(), dtype=float32, numpy=0.009564337>, <tf.Tensor: id=281949, shape=(), dtype=float32, numpy=-0.00023193806>, <tf.Tensor: id=281950, shape=(), dtype=float32, numpy=-0.0043777116>, <tf.Tensor: id=281951, shape=(), dtype=float32, numpy=0.0033248402>, <tf.Tensor: id=281952, shape=(), dtype=float32, numpy=0.0020942744>, <tf.Tensor: id=281953, shape=(), dtype=float32, numpy=0.00989055>, <tf.Tensor: id=281954, shape=(), dtype=float32, numpy=0.000547247>, <tf.Tensor: id=281955, shape=(), dtype=float32, numpy=-0.0011691392>, <tf.Tensor: id=281956, shape=(), dtype=float32, numpy=-0.033643395>, <tf.Tensor: id=281957, shape=(), dtype=float32, numpy=-0.0014932752>, <tf.Tensor: id=281958, shape=(), dtype=float32, numpy=0.012660088>, <tf.Tensor: id=281959, shape=(), dtype=float32, numpy=0.0124913>, <tf.Tensor: id=281960, shape=(), dtype=float32, numpy=-0.010591994>, <tf.Tensor: id=281961, shape=(), dtype=float32, numpy=-0.030872872>, <tf.Tensor: id=281962, shape=(), dtype=float32, numpy=-0.0014752604>)
tf.Tensor(
[0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0], shape=(100,), dtype=int32)
I was able to solve this issue. Like I said before, I'm new in TensorFlow and I wasn't aware that the result of tf.data.experimental.CsvDataset() (I use it to read my data) can't be used directly to train the model. The result is in the form of a tuple of 64 Tensors, but now I'm aware that the training example has to have at least 2 dimensions (the first one can be 1), when the first dimension is the batch_size and the second is the size of the example itself. In my code the first Tensor (for example <tf.Tensor: id=281899, shape=(), dtype=float32, numpy=0.0011093982>) was treated as the first training example, so its shape was () which caused the error.
When I was trying to fix the problem I changed some things, that caused other error, which led me to the solution. Each example has to be resized.
This is my current version of function to read training examples (DATA_SIZE is 64):
def expMap(*item) :
item2 = tf.reshape(item, [1, DATA_SIZE])
return item2
def generate_trainExp(nameD):
filename = os.path.join(DATA, "exps{}.txt".format(nameD))
record_defaults = [tf.float32] * DATA_SIZE
dataset = tf.data.experimental.CsvDataset(filename, record_defaults, field_delim=' ')
dataset = dataset.map(expMap)
return dataset
.....
data = generate_trainExp(filename)
In expMap() the input item can be also transformed to vector of 64 floats before the reshape instruction (item = tf.convert_to_tensor(item, dtype=tf.float32)).
I want to make this repo https://github.com/ildoonet/tf-pose-estimation run with Intel Movidius, so I tried convert the pb model using mvNCCompile.
The problem is mvNCCompile require a fixed input shape but the model I have is a dynamic one.
I tried this
graph_path = 'models/graph/mobilenet_thin/graph_opt.pb'
with tf.gfile.GFile(graph_path, 'rb') as f:
graph_def = tf.GraphDef()
graph_def.ParseFromString(f.read())
graph = tf.get_default_graph()
tf.import_graph_def(graph_def, name='TfPoseEstimator')
x = graph.get_tensor_by_name('TfPoseEstimator/image:0')
x.set_shape([1, 368, 368, 3])
x = graph.get_tensor_by_name('TfPoseEstimator/MobilenetV1/Conv2d_0/Conv2D:0')
x.set_shape([1, 368, 368, 24])
and got this
(<tf.Tensor 'TfPoseEstimator/MobilenetV1/Conv2d_0/weights:0' shape=(3, 3, 3, 24) dtype=float32>,)
(<tf.Tensor 'TfPoseEstimator/image:0' shape=(1, 368, 368, 3) dtype=float32>,)
(<tf.Tensor 'TfPoseEstimator/MobilenetV1/Conv2d_0/Conv2D:0' shape=(1, 368, 368, 24) dtype=float32>,)
(<tf.Tensor 'TfPoseEstimator/MobilenetV1/Conv2d_0/Conv2D_bn_offset:0' shape=(24,) dtype=float32>,)
(<tf.Tensor 'TfPoseEstimator/MobilenetV1/Conv2d_0/BatchNorm/FusedBatchNorm:0' shape=(?, ?, ?, 24) dtype=float32>,)
(<tf.Tensor 'TfPoseEstimator/MobilenetV1/Conv2d_0/Relu:0' shape=(?, ?, ?, 24) dtype=float32>,)
(<tf.Tensor 'TfPoseEstimator/MobilenetV1/Conv2d_1_depthwise/depthwise_weights:0' shape=(3, 3, 24, 1) dtype=float32>,)
(<tf.Tensor 'TfPoseEstimator/MobilenetV1/Conv2d_1_pointwise/weights:0' shape=(1, 1, 24, 48) dtype=float32>,)
(<tf.Tensor 'TfPoseEstimator/MobilenetV1/Conv2d_1_depthwise/depthwise:0' shape=(?, ?, ?, 24) dtype=float32>,)
(<tf.Tensor 'TfPoseEstimator/MobilenetV1/Conv2d_1_pointwise/Conv2D:0' shape=(?, ?, ?, 48) dtype=float32>,)
(<tf.Tensor 'TfPoseEstimator/MobilenetV1/Conv2d_1_pointwise/Conv2D_bn_offset:0' shape=(48,) dtype=float32>,)
(<tf.Tensor 'TfPoseEstimator/MobilenetV1/Conv2d_1_pointwise/BatchNorm/FusedBatchNorm:0' shape=(?, ?, ?, 48) dtype=float32>,)
(<tf.Tensor 'TfPoseEstimator/MobilenetV1/Conv2d_1_pointwise/Relu:0' shape=(?, ?, ?, 48) dtype=float32>,)
(<tf.Tensor 'TfPoseEstimator/MobilenetV1/Conv2d_2_depthwise/depthwise_weights:0' shape=(3, 3, 48, 1) dtype=float32>,)
(<tf.Tensor 'TfPoseEstimator/MobilenetV1/Conv2d_2_pointwise/weights:0' shape=(1, 1, 48, 96) dtype=float32>,)
(<tf.Tensor 'TfPoseEstimator/MobilenetV1/Conv2d_2_depthwise/depthwise:0' shape=(?, ?, ?, 48) dtype=float32>,)
(<tf.Tensor 'TfPoseEstimator/MobilenetV1/Conv2d_2_pointwise/Conv2D:0' shape=(?, ?, ?, 96) dtype=float32>,)
(<tf.Tensor 'TfPoseEstimator/MobilenetV1/Conv2d_2_pointwise/Conv2D_bn_offset:0' shape=(96,) dtype=float32>,)
(<tf.Tensor 'TfPoseEstimator/MobilenetV1/Conv2d_2_pointwise/BatchNorm/FusedBatchNorm:0' shape=(?, ?, ?, 96) dtype=float32>,)
(<tf.Tensor 'TfPoseEstimator/MobilenetV1/Conv2d_2_pointwise/Relu:0' shape=(?, ?, ?, 96) dtype=float32>,)
(<tf.Tensor 'TfPoseEstimator/MobilenetV1/Conv2d_3_depthwise/depthwise_weights:0' shape=(3, 3, 96, 1) dtype=float32>,)
(<tf.Tensor 'TfPoseEstimator/MobilenetV1/Conv2d_3_pointwise/weights:0' shape=(1, 1, 96, 96) dtype=float32>,)
(<tf.Tensor 'TfPoseEstimator/MobilenetV1/Conv2d_3_depthwise/depthwise:0' shape=(?, ?, ?, 96) dtype=float32>,)
(<tf.Tensor 'TfPoseEstimator/MobilenetV1/Conv2d_3_pointwise/Conv2D:0' shape=(?, ?, ?, 96) dtype=float32>,)
(<tf.Tensor 'TfPoseEstimator/MobilenetV1/Conv2d_3_pointwise/Conv2D_bn_offset:0' shape=(96,) dtype=float32>,)
(<tf.Tensor 'TfPoseEstimator/MobilenetV1/Conv2d_3_pointwise/BatchNorm/FusedBatchNorm:0' shape=(?, ?, ?, 96) dtype=float32>,)
(<tf.Tensor 'TfPoseEstimator/MobilenetV1/Conv2d_3_pointwise/Relu:0' shape=(?, ?, ?, 96) dtype=float32>,)
(<tf.Tensor 'TfPoseEstimator/MobilenetV1/Conv2d_4_depthwise/depthwise_weights:0' shape=(3, 3, 96, 1) dtype=float32>,)
(<tf.Tensor 'TfPoseEstimator/MobilenetV1/Conv2d_4_pointwise/weights:0' shape=(1, 1, 96, 192) dtype=float32>,)
(<tf.Tensor 'TfPoseEstimator/MobilenetV1/Conv2d_4_depthwise/depthwise:0' shape=(?, ?, ?, 96) dtype=float32>,)
(<tf.Tensor 'TfPoseEstimator/MobilenetV1/Conv2d_4_pointwise/Conv2D:0' shape=(?, ?, ?, 192) dtype=float32>,)
(<tf.Tensor 'TfPoseEstimator/MobilenetV1/Conv2d_4_pointwise/Conv2D_bn_offset:0' shape=(192,) dtype=float32>,)
(<tf.Tensor 'TfPoseEstimator/MobilenetV1/Conv2d_4_pointwise/BatchNorm/FusedBatchNorm:0' shape=(?, ?, ?, 192) dtype=float32>,)
(<tf.Tensor 'TfPoseEstimator/MobilenetV1/Conv2d_4_pointwise/Relu:0' shape=(?, ?, ?, 192) dtype=float32>,)
(<tf.Tensor 'TfPoseEstimator/MobilenetV1/Conv2d_5_depthwise/depthwise_weights:0' shape=(3, 3, 192, 1) dtype=float32>,)
(<tf.Tensor 'TfPoseEstimator/MobilenetV1/Conv2d_5_pointwise/weights:0' shape=(1, 1, 192, 192) dtype=float32>,)
(<tf.Tensor 'TfPoseEstimator/MobilenetV1/Conv2d_5_depthwise/depthwise:0' shape=(?, ?, ?, 192) dtype=float32>,)
(<tf.Tensor 'TfPoseEstimator/MobilenetV1/Conv2d_5_pointwise/Conv2D:0' shape=(?, ?, ?, 192) dtype=float32>,)
(<tf.Tensor 'TfPoseEstimator/MobilenetV1/Conv2d_5_pointwise/Conv2D_bn_offset:0' shape=(192,) dtype=float32>,)
(<tf.Tensor 'TfPoseEstimator/MobilenetV1/Conv2d_5_pointwise/BatchNorm/FusedBatchNorm:0' shape=(?, ?, ?, 192) dtype=float32>,)
(<tf.Tensor 'TfPoseEstimator/MobilenetV1/Conv2d_5_pointwise/Relu:0' shape=(?, ?, ?, 192) dtype=float32>,)
(<tf.Tensor 'TfPoseEstimator/MobilenetV1/Conv2d_6_depthwise/depthwise_weights:0' shape=(3, 3, 192, 1) dtype=float32>,)
(<tf.Tensor 'TfPoseEstimator/MobilenetV1/Conv2d_6_pointwise/weights:0' shape=(1, 1, 192, 384) dtype=float32>,)
(<tf.Tensor 'TfPoseEstimator/MobilenetV1/Conv2d_6_depthwise/depthwise:0' shape=(?, ?, ?, 192) dtype=float32>,)
(<tf.Tensor 'TfPoseEstimator/MobilenetV1/Conv2d_6_pointwise/Conv2D:0' shape=(?, ?, ?, 384) dtype=float32>,)
(<tf.Tensor 'TfPoseEstimator/MobilenetV1/Conv2d_6_pointwise/Conv2D_bn_offset:0' shape=(384,) dtype=float32>,)
(<tf.Tensor 'TfPoseEstimator/MobilenetV1/Conv2d_6_pointwise/BatchNorm/FusedBatchNorm:0' shape=(?, ?, ?, 384) dtype=float32>,)
(<tf.Tensor 'TfPoseEstimator/MobilenetV1/Conv2d_6_pointwise/Relu:0' shape=(?, ?, ?, 384) dtype=float32>,)
(<tf.Tensor 'TfPoseEstimator/MobilenetV1/Conv2d_7_depthwise/depthwise_weights:0' shape=(3, 3, 384, 1) dtype=float32>,)
(<tf.Tensor 'TfPoseEstimator/MobilenetV1/Conv2d_7_pointwise/weights:0' shape=(1, 1, 384, 384) dtype=float32>,)
(<tf.Tensor 'TfPoseEstimator/MobilenetV1/Conv2d_7_depthwise/depthwise:0' shape=(?, ?, ?, 384) dtype=float32>,)
(<tf.Tensor 'TfPoseEstimator/MobilenetV1/Conv2d_7_pointwise/Conv2D:0' shape=(?, ?, ?, 384) dtype=float32>,)
(<tf.Tensor 'TfPoseEstimator/MobilenetV1/Conv2d_7_pointwise/Conv2D_bn_offset:0' shape=(384,) dtype=float32>,)
(<tf.Tensor 'TfPoseEstimator/MobilenetV1/Conv2d_7_pointwise/BatchNorm/FusedBatchNorm:0' shape=(?, ?, ?, 384) dtype=float32>,)
(<tf.Tensor 'TfPoseEstimator/MobilenetV1/Conv2d_7_pointwise/Relu:0' shape=(?, ?, ?, 384) dtype=float32>,)
(<tf.Tensor 'TfPoseEstimator/MobilenetV1/Conv2d_8_depthwise/depthwise_weights:0' shape=(3, 3, 384, 1) dtype=float32>,)
(<tf.Tensor 'TfPoseEstimator/MobilenetV1/Conv2d_8_pointwise/weights:0' shape=(1, 1, 384, 384) dtype=float32>,)
(<tf.Tensor 'TfPoseEstimator/MobilenetV1/Conv2d_8_depthwise/depthwise:0' shape=(?, ?, ?, 384) dtype=float32>,)
(<tf.Tensor 'TfPoseEstimator/MobilenetV1/Conv2d_8_pointwise/Conv2D:0' shape=(?, ?, ?, 384) dtype=float32>,)
(<tf.Tensor 'TfPoseEstimator/MobilenetV1/Conv2d_8_pointwise/Conv2D_bn_offset:0' shape=(384,) dtype=float32>,)
(<tf.Tensor 'TfPoseEstimator/MobilenetV1/Conv2d_8_pointwise/BatchNorm/FusedBatchNorm:0' shape=(?, ?, ?, 384) dtype=float32>,)
(<tf.Tensor 'TfPoseEstimator/MobilenetV1/Conv2d_8_pointwise/Relu:0' shape=(?, ?, ?, 384) dtype=float32>,)
(<tf.Tensor 'TfPoseEstimator/MobilenetV1/Conv2d_9_depthwise/depthwise_weights:0' shape=(3, 3, 384, 1) dtype=float32>,)
(<tf.Tensor 'TfPoseEstimator/MobilenetV1/Conv2d_9_pointwise/weights:0' shape=(1, 1, 384, 384) dtype=float32>,)
(<tf.Tensor 'TfPoseEstimator/MobilenetV1/Conv2d_9_depthwise/depthwise:0' shape=(?, ?, ?, 384) dtype=float32>,)
(<tf.Tensor 'TfPoseEstimator/MobilenetV1/Conv2d_9_pointwise/Conv2D:0' shape=(?, ?, ?, 384) dtype=float32>,)
(<tf.Tensor 'TfPoseEstimator/MobilenetV1/Conv2d_9_pointwise/Conv2D_bn_offset:0' shape=(384,) dtype=float32>,)
(<tf.Tensor 'TfPoseEstimator/MobilenetV1/Conv2d_9_pointwise/BatchNorm/FusedBatchNorm:0' shape=(?, ?, ?, 384) dtype=float32>,)
(<tf.Tensor 'TfPoseEstimator/MobilenetV1/Conv2d_9_pointwise/Relu:0' shape=(?, ?, ?, 384) dtype=float32>,)
(<tf.Tensor 'TfPoseEstimator/MobilenetV1/Conv2d_10_depthwise/depthwise_weights:0' shape=(3, 3, 384, 1) dtype=float32>,)
(<tf.Tensor 'TfPoseEstimator/MobilenetV1/Conv2d_10_pointwise/weights:0' shape=(1, 1, 384, 384) dtype=float32>,)
(<tf.Tensor 'TfPoseEstimator/MobilenetV1/Conv2d_10_depthwise/depthwise:0' shape=(?, ?, ?, 384) dtype=float32>,)
(<tf.Tensor 'TfPoseEstimator/MobilenetV1/Conv2d_10_pointwise/Conv2D:0' shape=(?, ?, ?, 384) dtype=float32>,)
(<tf.Tensor 'TfPoseEstimator/MobilenetV1/Conv2d_10_pointwise/Conv2D_bn_offset:0' shape=(384,) dtype=float32>,)
(<tf.Tensor 'TfPoseEstimator/MobilenetV1/Conv2d_10_pointwise/BatchNorm/FusedBatchNorm:0' shape=(?, ?, ?, 384) dtype=float32>,)
(<tf.Tensor 'TfPoseEstimator/MobilenetV1/Conv2d_10_pointwise/Relu:0' shape=(?, ?, ?, 384) dtype=float32>,)
(<tf.Tensor 'TfPoseEstimator/MobilenetV1/Conv2d_11_depthwise/depthwise_weights:0' shape=(3, 3, 384, 1) dtype=float32>,)
(<tf.Tensor 'TfPoseEstimator/MobilenetV1/Conv2d_11_pointwise/weights:0' shape=(1, 1, 384, 384) dtype=float32>,)
(<tf.Tensor 'TfPoseEstimator/MobilenetV1/Conv2d_11_depthwise/depthwise:0' shape=(?, ?, ?, 384) dtype=float32>,)
(<tf.Tensor 'TfPoseEstimator/MobilenetV1/Conv2d_11_pointwise/Conv2D:0' shape=(?, ?, ?, 384) dtype=float32>,)
(<tf.Tensor 'TfPoseEstimator/MobilenetV1/Conv2d_11_pointwise/Conv2D_bn_offset:0' shape=(384,) dtype=float32>,)
(<tf.Tensor 'TfPoseEstimator/MobilenetV1/Conv2d_11_pointwise/BatchNorm/FusedBatchNorm:0' shape=(?, ?, ?, 384) dtype=float32>,)
(<tf.Tensor 'TfPoseEstimator/MobilenetV1/Conv2d_11_pointwise/Relu:0' shape=(?, ?, ?, 384) dtype=float32>,)
Another layers beside TfPoseEstimator/image:0 and TfPoseEstimator/MobilenetV1/Conv2d_0/Conv2D:0 still have ? shape.
I'm very new in Tensorflow so this might be a stupid question, but how to change the input shape of a saved model?
I manage to solve this problem using this.
import tensorflow as tf
if __name__ == '__main__':
graph_path = 't/tf_model.pb'
with tf.gfile.GFile(graph_path, 'rb') as f:
graph_def = tf.GraphDef()
graph_def.ParseFromString(f.read())
graph = tf.get_default_graph()
tf_new_image = tf.placeholder(shape=(1, 368, 368, 3), dtype='float32', name='new_image')
tf.import_graph_def(graph_def, name='TfPoseEstimator', input_map={"image:0": tf_new_image})
tf.train.write_graph(graph, "t", "mobilenet_thin_model.pb", as_text=False)
With tf2.x I belive you can change it to concrete func:
imported = tf.saved_model.load('/path/to/saved_model')
concrete_func = imported.signatures["serving_default"]
concrete_func.inputs[0].set_shape([1, 368, 368, 3])
I'm using TF 1.10.0 on Ubuntu 16.04.
I'm using the Estimator API to build a language model and want to preserve the last hidden states to initialize the states for next batch. It looks something like this(Ref: https://stackoverflow.com/a/41240243/6948766):
init_state = get_state_variables(params.batch_size, lstm_cell)
# list of [batch_size, max_steps, lstm_dim]
with tf.variable_scope(direction):
_lstm_output_unpacked, final_state = tf.nn.static_rnn(
lstm_cell,
tf.unstack(lstm_input, axis=1),
initial_state=init_state,
dtype=DTYPE)
self.state_update_op.append(
get_state_update_op(init_state, final_state))
And training code looks like this:
dist_trategy = tf.contrib.distribute.MirroredStrategy()
run_config = tf.estimator.RunConfig(
train_distribute=dist_trategy)
estimator = tf.estimator.Estimator(model_fn=model.model_fn,
model_dir='model/',
params=params,
config=run_config)
But I get the following errors:
Traceback (most recent call last):
File "src/train_eval.py", line 64, in <module>
main()
File "src/train_eval.py", line 60, in main
run(p, arg.mode)
File "src/train_eval.py", line 47, in run
params, 'train'), max_steps=1000000)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/estimator/estimator.py", line 356, in train
loss = self._train_model(input_fn, hooks, saving_listeners)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/estimator/estimator.py", line 1179, in _train_model
return self._train_model_distributed(input_fn, hooks, saving_listeners)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/estimator/estimator.py", line 1290, in _train_model_distributed
self.config)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/distribute.py", line 718, in call_for_each_tower
return self._call_for_each_tower(fn, *args, **kwargs)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/contrib/distribute/python/mirrored_strategy.py", line 552, in _call_for_each_tower
return _call_for_each_tower(self, fn, *args, **kwargs)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/contrib/distribute/python/mirrored_strategy.py", line 183, in _call_for_each_tower
coord.join(threads)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/coordinator.py", line 389, in join
six.reraise(*self._exc_info_to_raise)
File "/usr/local/lib/python3.5/dist-packages/six.py", line 693, in reraise
raise value
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/coordinator.py", line 297, in stop_on_exception
yield
File "/usr/local/lib/python3.5/dist-packages/tensorflow/contrib/distribute/python/mirrored_strategy.py", line 166, in _call_for_each_tower
merge_args = values.regroup({t.device: t.merge_args for t in threads})
File "/usr/local/lib/python3.5/dist-packages/tensorflow/contrib/distribute/python/values.py", line 585, in regroup
for i in range(len(v0)))
File "/usr/local/lib/python3.5/dist-packages/tensorflow/contrib/distribute/python/values.py", line 585, in <genexpr>
for i in range(len(v0)))
File "/usr/local/lib/python3.5/dist-packages/tensorflow/contrib/distribute/python/values.py", line 576, in regroup
(len(v), len(v0), v, v0))
AssertionError: len(v) == 33, len(v0) == 2, v: [(<tensorflow.python.framework.ops.IndexedSlices object at 0x7fe818e9c518>, MirroredVariable({'/replica:0/task:0/device:GPU:3': <tf.Variable 'char_embed/replica_3:0' shape=(11204, 64) dtype=float32>, '/replica:0/task:0/device:GPU:2': <tf.Variable 'char_embed/replica_2:0' shape=(11204, 64) dtype=float32>, '/replica:0/task:0/device:GPU:0': <tf.Variable 'char_embed:0' shape=(11204, 64) dtype=float32>, '/replica:0/task:0/device:GPU:1': <tf.Variable 'char_embed/replica_1:0' shape=(11204, 64) dtype=float32>})), (<tf.Tensor 'clip_by_global_norm/clip_by_global_norm/_1:0' shape=(1, 1, 64, 32) dtype=float32>, MirroredVariable({'/replica:0/task:0/device:GPU:3': <tf.Variable 'CNN/W_cnn_0/replica_3:0' shape=(1, 1, 64, 32) dtype=float32>, '/replica:0/task:0/device:GPU:2': <tf.Variable 'CNN/W_cnn_0/replica_2:0' shape=(1, 1, 64, 32) dtype=float32>, '/replica:0/task:0/device:GPU:0': <tf.Variable 'CNN/W_cnn_0:0' shape=(1, 1, 64, 32) dtype=float32>, '/replica:0/task:0/device:GPU:1': <tf.Variable 'CNN/W_cnn_0/replica_1:0' shape=(1, 1, 64, 32) dtype=float32>})), (<tf.Tensor 'clip_by_global_norm/clip_by_global_norm/_2:0' shape=(32,) dtype=float32>, MirroredVariable({'/replica:0/task:0/device:GPU:3': <tf.Variable 'CNN/b_cnn_0/replica_3:0' shape=(32,) dtype=float32>, '/replica:0/task:0/device:GPU:2': <tf.Variable 'CNN/b_cnn_0/replica_2:0' shape=(32,) dtype=float32>, '/replica:0/task:0/device:GPU:0': <tf.Variable 'CNN/b_cnn_0:0' shape=(32,) dtype=float32>, '/replica:0/task:0/device:GPU:1': <tf.Variable 'CNN/b_cnn_0/replica_1:0' shape=(32,) dtype=float32>})), (<tf.Tensor 'clip_by_global_norm/clip_by_global_norm/_3:0' shape=(1, 2, 64, 32) dtype=float32>, MirroredVariable({'/replica:0/task:0/device:GPU:3': <tf.Variable 'CNN/W_cnn_1/replica_3:0' shape=(1, 2, 64, 32) dtype=float32>, '/replica:0/task:0/device:GPU:2': <tf.Variable 'CNN/W_cnn_1/replica_2:0' shape=(1, 2, 64, 32) dtype=float32>, '/replica:0/task:0/device:GPU:0': <tf.Variable 'CNN/W_cnn_1:0' shape=(1, 2, 64, 32) dtype=float32>, '/replica:0/task:0/device:GPU:1': <tf.Variable 'CNN/W_cnn_1/replica_1:0' shape=(1, 2, 64, 32) dtype=float32>})), (<tf.Tensor 'clip_by_global_norm/clip_by_global_norm/_4:0' shape=(32,) dtype=float32>, MirroredVariable({'/replica:0/task:0/device:GPU:3': <tf.Variable 'CNN/b_cnn_1/replica_3:0' shape=(32,) dtype=float32>, '/replica:0/task:0/device:GPU:2': <tf.Variable 'CNN/b_cnn_1/replica_2:0' shape=(32,) dtype=float32>, '/replica:0/task:0/device:GPU:0': <tf.Variable 'CNN/b_cnn_1:0' shape=(32,) dtype=float32>, '/replica:0/task:0/device:GPU:1': <tf.Variable 'CNN/b_cnn_1/replica_1:0' shape=(32,) dtype=float32>})), (<tf.Tensor 'clip_by_global_norm/clip_by_global_norm/_5:0' shape=(1, 3, 64, 64) dtype=float32>, MirroredVariable({'/replica:0/task:0/device:GPU:3': <tf.Variable 'CNN/W_cnn_2/replica_3:0' shape=(1, 3, 64, 64) dtype=float32>, '/replica:0/task:0/device:GPU:2': <tf.Variable 'CNN/W_cnn_2/replica_2:0' shape=(1, 3, 64, 64) dtype=float32>, '/replica:0/task:0/device:GPU:0': <tf.Variable 'CNN/W_cnn_2:0' shape=(1, 3, 64, 64) dtype=float32>, '/replica:0/task:0/device:GPU:1': <tf.Variable 'CNN/W_cnn_2/replica_1:0' shape=(1, 3, 64, 64) dtype=float32>})), (<tf.Tensor 'clip_by_global_norm/clip_by_global_norm/_6:0' shape=(64,) dtype=float32>, MirroredVariable({'/replica:0/task:0/device:GPU:3': <tf.Variable 'CNN/b_cnn_2/replica_3:0' shape=(64,) dtype=float32>, '/replica:0/task:0/device:GPU:2': <tf.Variable 'CNN/b_cnn_2/replica_2:0' shape=(64,) dtype=float32>, '/replica:0/task:0/device:GPU:0': <tf.Variable 'CNN/b_cnn_2:0' shape=(64,) dtype=float32>, '/replica:0/task:0/device:GPU:1': <tf.Variable 'CNN/b_cnn_2/replica_1:0' shape=(64,) dtype=float32>})), (<tf.Tensor 'clip_by_global_norm/clip_by_global_norm/_7:0' shape=(1, 4, 64, 128) dtype=float32>, MirroredVariable({'/replica:0/task:0/device:GPU:3': <tf.Variable 'CNN/W_cnn_3/replica_3:0' shape=(1, 4, 64, 128) dtype=float32>, '/replica:0/task:0/device:GPU:2': <tf.Variable 'CNN/W_cnn_3/replica_2:0' shape=(1, 4, 64, 128) dtype=float32>, '/replica:0/task:0/device:GPU:0': <tf.Variable 'CNN/W_cnn_3:0' shape=(1, 4, 64, 128) dtype=float32>, '/replica:0/task:0/device:GPU:1': <tf.Variable 'CNN/W_cnn_3/replica_1:0' shape=(1, 4, 64, 128) dtype=float32>})), (<tf.Tensor 'clip_by_global_norm/clip_by_global_norm/_8:0' shape=(128,) dtype=float32>, MirroredVariable({'/replica:0/task:0/device:GPU:3': <tf.Variable 'CNN/b_cnn_3/replica_3:0' shape=(128,) dtype=float32>, '/replica:0/task:0/device:GPU:2': <tf.Variable 'CNN/b_cnn_3/replica_2:0' shape=(128,) dtype=float32>, '/replica:0/task:0/device:GPU:0': <tf.Variable 'CNN/b_cnn_3:0' shape=(128,) dtype=float32>, '/replica:0/task:0/device:GPU:1': <tf.Variable 'CNN/b_cnn_3/replica_1:0' shape=(128,) dtype=float32>})), (<tf.Tensor 'clip_by_global_norm/clip_by_global_norm/_9:0' shape=(1, 5, 64, 256) dtype=float32>, MirroredVariable({'/replica:0/task:0/device:GPU:3': <tf.Variable 'CNN/W_cnn_4/replica_3:0' shape=(1, 5, 64, 256) dtype=float32>, '/replica:0/task:0/device:GPU:2': <tf.Variable 'CNN/W_cnn_4/replica_2:0' shape=(1, 5, 64, 256) dtype=float32>, '/replica:0/task:0/device:GPU:0': <tf.Variable 'CNN/W_cnn_4:0' shape=(1, 5, 64, 256) dtype=float32>, '/replica:0/task:0/device:GPU:1': <tf.Variable 'CNN/W_cnn_4/replica_1:0' shape=(1, 5, 64, 256) dtype=float32>})), (<tf.Tensor 'clip_by_global_norm/clip_by_global_norm/_10:0' shape=(256,) dtype=float32>, MirroredVariable({'/replica:0/task:0/device:GPU:3': <tf.Variable 'CNN/b_cnn_4/replica_3:0' shape=(256,) dtype=float32>, '/replica:0/task:0/device:GPU:2': <tf.Variable 'CNN/b_cnn_4/replica_2:0' shape=(256,) dtype=float32>, '/replica:0/task:0/device:GPU:0': <tf.Variable 'CNN/b_cnn_4:0' shape=(256,) dtype=float32>, '/replica:0/task:0/device:GPU:1': <tf.Variable 'CNN/b_cnn_4/replica_1:0' shape=(256,) dtype=float32>})), (<tf.Tensor 'clip_by_global_norm/clip_by_global_norm/_11:0' shape=(1, 6, 64, 512) dtype=float32>, MirroredVariable({'/replica:0/task:0/device:GPU:3': <tf.Variable 'CNN/W_cnn_5/replica_3:0' shape=(1, 6, 64, 512) dtype=float32>, '/replica:0/task:0/device:GPU:2': <tf.Variable 'CNN/W_cnn_5/replica_2:0' shape=(1, 6, 64, 512) dtype=float32>, '/replica:0/task:0/device:GPU:0': <tf.Variable 'CNN/W_cnn_5:0' shape=(1, 6, 64, 512) dtype=float32>, '/replica:0/task:0/device:GPU:1': <tf.Variable 'CNN/W_cnn_5/replica_1:0' shape=(1, 6, 64, 512) dtype=float32>})), (<tf.Tensor 'clip_by_global_norm/clip_by_global_norm/_12:0' shape=(512,) dtype=float32>, MirroredVariable({'/replica:0/task:0/device:GPU:3': <tf.Variable 'CNN/b_cnn_5/replica_3:0' shape=(512,) dtype=float32>, '/replica:0/task:0/device:GPU:2': <tf.Variable 'CNN/b_cnn_5/replica_2:0' shape=(512,) dtype=float32>, '/replica:0/task:0/device:GPU:0': <tf.Variable 'CNN/b_cnn_5:0' shape=(512,) dtype=float32>, '/replica:0/task:0/device:GPU:1': <tf.Variable 'CNN/b_cnn_5/replica_1:0' shape=(512,) dtype=float32>})), (<tf.Tensor 'clip_by_global_norm/clip_by_global_norm/_13:0' shape=(1, 7, 64, 1024) dtype=float32>, MirroredVariable({'/replica:0/task:0/device:GPU:3': <tf.Variable 'CNN/W_cnn_6/replica_3:0' shape=(1, 7, 64, 1024) dtype=float32>, '/replica:0/task:0/device:GPU:2': <tf.Variable 'CNN/W_cnn_6/replica_2:0' shape=(1, 7, 64, 1024) dtype=float32>, '/replica:0/task:0/device:GPU:0': <tf.Variable 'CNN/W_cnn_6:0' shape=(1, 7, 64, 1024) dtype=float32>, '/replica:0/task:0/device:GPU:1': <tf.Variable 'CNN/W_cnn_6/replica_1:0' shape=(1, 7, 64, 1024) dtype=float32>})), (<tf.Tensor 'clip_by_global_norm/clip_by_global_norm/_14:0' shape=(1024,) dtype=float32>, MirroredVariable({'/replica:0/task:0/device:GPU:3': <tf.Variable 'CNN/b_cnn_6/replica_3:0' shape=(1024,) dtype=float32>, '/replica:0/task:0/device:GPU:2': <tf.Variable 'CNN/b_cnn_6/replica_2:0' shape=(1024,) dtype=float32>, '/replica:0/task:0/device:GPU:0': <tf.Variable 'CNN/b_cnn_6:0' shape=(1024,) dtype=float32>, '/replica:0/task:0/device:GPU:1': <tf.Variable 'CNN/b_cnn_6/replica_1:0' shape=(1024,) dtype=float32>})), (<tf.Tensor 'clip_by_global_norm/clip_by_global_norm/_15:0' shape=(2048, 512) dtype=float32>, MirroredVariable({'/replica:0/task:0/device:GPU:3': <tf.Variable 'CNN_proj/W_proj/replica_3:0' shape=(2048, 512) dtype=float32>, '/replica:0/task:0/device:GPU:2': <tf.Variable 'CNN_proj/W_proj/replica_2:0' shape=(2048, 512) dtype=float32>, '/replica:0/task:0/device:GPU:0': <tf.Variable 'CNN_proj/W_proj:0' shape=(2048, 512) dtype=float32>, '/replica:0/task:0/device:GPU:1': <tf.Variable 'CNN_proj/W_proj/replica_1:0' shape=(2048, 512) dtype=float32>})), (<tf.Tensor 'clip_by_global_norm/clip_by_global_norm/_16:0' shape=(512,) dtype=float32>, MirroredVariable({'/replica:0/task:0/device:GPU:3': <tf.Variable 'CNN_proj/b_proj/replica_3:0' shape=(512,) dtype=float32>, '/replica:0/task:0/device:GPU:2': <tf.Variable 'CNN_proj/b_proj/replica_2:0' shape=(512,) dtype=float32>, '/replica:0/task:0/device:GPU:0': <tf.Variable 'CNN_proj/b_proj:0' shape=(512,) dtype=float32>, '/replica:0/task:0/device:GPU:1': <tf.Variable 'CNN_proj/b_proj/replica_1:0' shape=(512,) dtype=float32>})), (<tf.Tensor 'clip_by_global_norm/clip_by_global_norm/_17:0' shape=(2048, 2048) dtype=float32>, MirroredVariable({'/replica:0/task:0/device:GPU:3': <tf.Variable 'CNN_high_0/W_carry/replica_3:0' shape=(2048, 2048) dtype=float32>, '/replica:0/task:0/device:GPU:2': <tf.Variable 'CNN_high_0/W_carry/replica_2:0' shape=(2048, 2048) dtype=float32>, '/replica:0/task:0/device:GPU:0': <tf.Variable 'CNN_high_0/W_carry:0' shape=(2048, 2048) dtype=float32>, '/replica:0/task:0/device:GPU:1': <tf.Variable 'CNN_high_0/W_carry/replica_1:0' shape=(2048, 2048) dtype=float32>})), (<tf.Tensor 'clip_by_global_norm/clip_by_global_norm/_18:0' shape=(2048,) dtype=float32>, MirroredVariable({'/replica:0/task:0/device:GPU:3': <tf.Variable 'CNN_high_0/b_carry/replica_3:0' shape=(2048,) dtype=float32>, '/replica:0/task:0/device:GPU:2': <tf.Variable 'CNN_high_0/b_carry/replica_2:0' shape=(2048,) dtype=float32>, '/replica:0/task:0/device:GPU:0': <tf.Variable 'CNN_high_0/b_carry:0' shape=(2048,) dtype=float32>, '/replica:0/task:0/device:GPU:1': <tf.Variable 'CNN_high_0/b_carry/replica_1:0' shape=(2048,) dtype=float32>})), (<tf.Tensor 'clip_by_global_norm/clip_by_global_norm/_19:0' shape=(2048, 2048) dtype=float32>, MirroredVariable({'/replica:0/task:0/device:GPU:3': <tf.Variable 'CNN_high_0/W_transform/replica_3:0' shape=(2048, 2048) dtype=float32>, '/replica:0/task:0/device:GPU:2': <tf.Variable 'CNN_high_0/W_transform/replica_2:0' shape=(2048, 2048) dtype=float32>, '/replica:0/task:0/device:GPU:0': <tf.Variable 'CNN_high_0/W_transform:0' shape=(2048, 2048) dtype=float32>, '/replica:0/task:0/device:GPU:1': <tf.Variable 'CNN_high_0/W_transform/replica_1:0' shape=(2048, 2048) dtype=float32>})), (<tf.Tensor 'clip_by_global_norm/clip_by_global_norm/_20:0' shape=(2048,) dtype=float32>, MirroredVariable({'/replica:0/task:0/device:GPU:3': <tf.Variable 'CNN_high_0/b_transform/replica_3:0' shape=(2048,) dtype=float32>, '/replica:0/task:0/device:GPU:2': <tf.Variable 'CNN_high_0/b_transform/replica_2:0' shape=(2048,) dtype=float32>, '/replica:0/task:0/device:GPU:0': <tf.Variable 'CNN_high_0/b_transform:0' shape=(2048,) dtype=float32>, '/replica:0/task:0/device:GPU:1': <tf.Variable 'CNN_high_0/b_transform/replica_1:0' shape=(2048,) dtype=float32>})), (<tf.Tensor 'clip_by_global_norm/clip_by_global_norm/_21:0' shape=(2048, 2048) dtype=float32>, MirroredVariable({'/replica:0/task:0/device:GPU:3': <tf.Variable 'CNN_high_1/W_carry/replica_3:0' shape=(2048, 2048) dtype=float32>, '/replica:0/task:0/device:GPU:2': <tf.Variable 'CNN_high_1/W_carry/replica_2:0' shape=(2048, 2048) dtype=float32>, '/replica:0/task:0/device:GPU:0': <tf.Variable 'CNN_high_1/W_carry:0' shape=(2048, 2048) dtype=float32>, '/replica:0/task:0/device:GPU:1': <tf.Variable 'CNN_high_1/W_carry/replica_1:0' shape=(2048, 2048) dtype=float32>})), (<tf.Tensor 'clip_by_global_norm/clip_by_global_norm/_22:0' shape=(2048,) dtype=float32>, MirroredVariable({'/replica:0/task:0/device:GPU:3': <tf.Variable 'CNN_high_1/b_carry/replica_3:0' shape=(2048,) dtype=float32>, '/replica:0/task:0/device:GPU:2': <tf.Variable 'CNN_high_1/b_carry/replica_2:0' shape=(2048,) dtype=float32>, '/replica:0/task:0/device:GPU:0': <tf.Variable 'CNN_high_1/b_carry:0' shape=(2048,) dtype=float32>, '/replica:0/task:0/device:GPU:1': <tf.Variable 'CNN_high_1/b_carry/replica_1:0' shape=(2048,) dtype=float32>})), (<tf.Tensor 'clip_by_global_norm/clip_by_global_norm/_23:0' shape=(2048, 2048) dtype=float32>, MirroredVariable({'/replica:0/task:0/device:GPU:3': <tf.Variable 'CNN_high_1/W_transform/replica_3:0' shape=(2048, 2048) dtype=float32>, '/replica:0/task:0/device:GPU:2': <tf.Variable 'CNN_high_1/W_transform/replica_2:0' shape=(2048, 2048) dtype=float32>, '/replica:0/task:0/device:GPU:0': <tf.Variable 'CNN_high_1/W_transform:0' shape=(2048, 2048) dtype=float32>, '/replica:0/task:0/device:GPU:1': <tf.Variable 'CNN_high_1/W_transform/replica_1:0' shape=(2048, 2048) dtype=float32>})), (<tf.Tensor 'clip_by_global_norm/clip_by_global_norm/_24:0' shape=(2048,) dtype=float32>, MirroredVariable({'/replica:0/task:0/device:GPU:3': <tf.Variable 'CNN_high_1/b_transform/replica_3:0' shape=(2048,) dtype=float32>, '/replica:0/task:0/device:GPU:2': <tf.Variable 'CNN_high_1/b_transform/replica_2:0' shape=(2048,) dtype=float32>, '/replica:0/task:0/device:GPU:0': <tf.Variable 'CNN_high_1/b_transform:0' shape=(2048,) dtype=float32>, '/replica:0/task:0/device:GPU:1': <tf.Variable 'CNN_high_1/b_transform/replica_1:0' shape=(2048,) dtype=float32>})), (<tf.Tensor 'clip_by_global_norm/clip_by_global_norm/_25:0' shape=(1024, 16384) dtype=float32>, MirroredVariable({'/replica:0/task:0/device:GPU:3': <tf.Variable 'fw/rnn/multi_rnn_cell/cell_0/lstm_cell/kernel/replica_3:0' shape=(1024, 16384) dtype=float32>, '/replica:0/task:0/device:GPU:2': <tf.Variable 'fw/rnn/multi_rnn_cell/cell_0/lstm_cell/kernel/replica_2:0' shape=(1024, 16384) dtype=float32>, '/replica:0/task:0/device:GPU:0': <tf.Variable 'fw/rnn/multi_rnn_cell/cell_0/lstm_cell/kernel:0' shape=(1024, 16384) dtype=float32>, '/replica:0/task:0/device:GPU:1': <tf.Variable 'fw/rnn/multi_rnn_cell/cell_0/lstm_cell/kernel/replica_1:0' shape=(1024, 16384) dtype=float32>})), (<tf.Tensor 'clip_by_global_norm/clip_by_global_norm/_26:0' shape=(16384,) dtype=float32>, MirroredVariable({'/replica:0/task:0/device:GPU:3': <tf.Variable 'fw/rnn/multi_rnn_cell/cell_0/lstm_cell/bias/replica_3:0' shape=(16384,) dtype=float32>, '/replica:0/task:0/device:GPU:2': <tf.Variable 'fw/rnn/multi_rnn_cell/cell_0/lstm_cell/bias/replica_2:0' shape=(16384,) dtype=float32>, '/replica:0/task:0/device:GPU:0': <tf.Variable 'fw/rnn/multi_rnn_cell/cell_0/lstm_cell/bias:0' shape=(16384,) dtype=float32>, '/replica:0/task:0/device:GPU:1': <tf.Variable 'fw/rnn/multi_rnn_cell/cell_0/lstm_cell/bias/replica_1:0' shape=(16384,) dtype=float32>})), (<tf.Tensor 'clip_by_global_norm/clip_by_global_norm/_27:0' shape=(4096, 512) dtype=float32>, MirroredVariable({'/replica:0/task:0/device:GPU:3': <tf.Variable 'fw/rnn/multi_rnn_cell/cell_0/lstm_cell/projection/kernel/replica_3:0' shape=(4096, 512) dtype=float32>, '/replica:0/task:0/device:GPU:2': <tf.Variable 'fw/rnn/multi_rnn_cell/cell_0/lstm_cell/projection/kernel/replica_2:0' shape=(4096, 512) dtype=float32>, '/replica:0/task:0/device:GPU:0': <tf.Variable 'fw/rnn/multi_rnn_cell/cell_0/lstm_cell/projection/kernel:0' shape=(4096, 512) dtype=float32>, '/replica:0/task:0/device:GPU:1': <tf.Variable 'fw/rnn/multi_rnn_cell/cell_0/lstm_cell/projection/kernel/replica_1:0' shape=(4096, 512) dtype=float32>})), (<tf.Tensor 'clip_by_global_norm/clip_by_global_norm/_28:0' shape=(1024, 16384) dtype=float32>, MirroredVariable({'/replica:0/task:0/device:GPU:3': <tf.Variable 'fw/rnn/multi_rnn_cell/cell_1/lstm_cell/kernel/replica_3:0' shape=(1024, 16384) dtype=float32>, '/replica:0/task:0/device:GPU:2': <tf.Variable 'fw/rnn/multi_rnn_cell/cell_1/lstm_cell/kernel/replica_2:0' shape=(1024, 16384) dtype=float32>, '/replica:0/task:0/device:GPU:0': <tf.Variable 'fw/rnn/multi_rnn_cell/cell_1/lstm_cell/kernel:0' shape=(1024, 16384) dtype=float32>, '/replica:0/task:0/device:GPU:1': <tf.Variable 'fw/rnn/multi_rnn_cell/cell_1/lstm_cell/kernel/replica_1:0' shape=(1024, 16384) dtype=float32>})), (<tf.Tensor 'clip_by_global_norm/clip_by_global_norm/_29:0' shape=(16384,) dtype=float32>, MirroredVariable({'/replica:0/task:0/device:GPU:3': <tf.Variable 'fw/rnn/multi_rnn_cell/cell_1/lstm_cell/bias/replica_3:0' shape=(16384,) dtype=float32>, '/replica:0/task:0/device:GPU:2': <tf.Variable 'fw/rnn/multi_rnn_cell/cell_1/lstm_cell/bias/replica_2:0' shape=(16384,) dtype=float32>, '/replica:0/task:0/device:GPU:0': <tf.Variable 'fw/rnn/multi_rnn_cell/cell_1/lstm_cell/bias:0' shape=(16384,) dtype=float32>, '/replica:0/task:0/device:GPU:1': <tf.Variable 'fw/rnn/multi_rnn_cell/cell_1/lstm_cell/bias/replica_1:0' shape=(16384,) dtype=float32>})), (<tf.Tensor 'clip_by_global_norm/clip_by_global_norm/_30:0' shape=(4096, 512) dtype=float32>, MirroredVariable({'/replica:0/task:0/device:GPU:3': <tf.Variable 'fw/rnn/multi_rnn_cell/cell_1/lstm_cell/projection/kernel/replica_3:0' shape=(4096, 512) dtype=float32>, '/replica:0/task:0/device:GPU:2': <tf.Variable 'fw/rnn/multi_rnn_cell/cell_1/lstm_cell/projection/kernel/replica_2:0' shape=(4096, 512) dtype=float32>, '/replica:0/task:0/device:GPU:0': <tf.Variable 'fw/rnn/multi_rnn_cell/cell_1/lstm_cell/projection/kernel:0' shape=(4096, 512) dtype=float32>, '/replica:0/task:0/device:GPU:1': <tf.Variable 'fw/rnn/multi_rnn_cell/cell_1/lstm_cell/projection/kernel/replica_1:0' shape=(4096, 512) dtype=float32>})), (<tensorflow.python.framework.ops.IndexedSlices object at 0x7fe818e9c390>, MirroredVariable({'/replica:0/task:0/device:GPU:3': <tf.Variable 'softmax/W/replica_3:0' shape=(556527, 512) dtype=float32>, '/replica:0/task:0/device:GPU:2': <tf.Variable 'softmax/W/replica_2:0' shape=(556527, 512) dtype=float32>, '/replica:0/task:0/device:GPU:0': <tf.Variable 'softmax/W:0' shape=(556527, 512) dtype=float32>, '/replica:0/task:0/device:GPU:1': <tf.Variable 'softmax/W/replica_1:0' shape=(556527, 512) dtype=float32>})), (<tensorflow.python.framework.ops.IndexedSlices object at 0x7fe818e9c0b8>, MirroredVariable({'/replica:0/task:0/device:GPU:3': <tf.Variable 'softmax/b/replica_3:0' shape=(556527,) dtype=float32>, '/replica:0/task:0/device:GPU:2': <tf.Variable 'softmax/b/replica_2:0' shape=(556527,) dtype=float32>, '/replica:0/task:0/device:GPU:0': <tf.Variable 'softmax/b:0' shape=(556527,) dtype=float32>, '/replica:0/task:0/device:GPU:1': <tf.Variable 'softmax/b/replica_1:0' shape=(556527,) dtype=float32>}))], v0: [(<tensorflow.python.framework.ops.IndexedSlices object at 0x7fe8127cfe10>, MirroredVariable({'/replica:0/task:0/device:GPU:3': <tf.Variable 'softmax/W/replica_3:0' shape=(556527, 512) dtype=float32>, '/replica:0/task:0/device:GPU:2': <tf.Variable 'softmax/W/replica_2:0' shape=(556527, 512) dtype=float32>, '/replica:0/task:0/device:GPU:0': <tf.Variable 'softmax/W:0' shape=(556527, 512) dtype=float32>, '/replica:0/task:0/device:GPU:1': <tf.Variable 'softmax/W/replica_1:0' shape=(556527, 512) dtype=float32>})), (<tensorflow.python.framework.ops.IndexedSlices object at 0x7fe8127cfe80>, MirroredVariable({'/replica:0/task:0/device:GPU:3': <tf.Variable 'softmax/b/replica_3:0' shape=(556527,) dtype=float32>, '/replica:0/task:0/device:GPU:2': <tf.Variable 'softmax/b/replica_2:0' shape=(556527,) dtype=float32>, '/replica:0/task:0/device:GPU:0': <tf.Variable 'softmax/b:0' shape=(556527,) dtype=float32>, '/replica:0/task:0/device:GPU:1': <tf.Variable 'softmax/b/replica_1:0' shape=(556527,) dtype=float32>}))]
Thanks in advance.