Tensorflow vgg16 prediction dramatically slow - tensorflow

Question

I use vgg.h5 model + Keras (Tensorflow backend on GPU) for real-time objects classification. It works good.
Then I try to use pure tensorflow graphs with weights from vgg.h5:
I've parsed vgg.h5 with h5py and have received weights for all layers in numpay.array format
I've built a graph (I store kernels and biases in tf.Variable)
But I can't receive output prediction vector. After investigation I've found out that all convolutional layers work, but the first full connection layer output (fc1 with 25088 x 4096 weights matrix) in vgg16 calculated for about 5 minutes. It's not appropriate for real time classification.
So, maybe anyone has experience of building vgg16 from scratch in tensorflow and can help? Why tensorflow as Keras backend works good, but pure tensorflow (with the same weights) can't calculate full connection output? Is there any additional optimisation in Keras for realising full connection (Dense) layers?

Answer 1

Here is a test variant of your code, instrumented with printing shapes of tensors in several places:
import tensorflow as tf
import numpy as np
with tf.Session() as sess:
# mock the previous layer's output with a placeholder
pool5_input = tf.placeholder(dtype = tf.float32, shape = (None,7,7,512))
# insert a print operation to print the shape
pool5 = tf.Print(pool5_input, [ tf.shape(pool5_input) ], "pool5 shape is ", summarize = 4)
layer_name = 'fc1'
wd = tf.Variable(np.ones((25088, 4096), dtype='float32'), trainable=False, name=layer_name+'_wd')
bd = tf.Variable(np.ones((4096,), dtype='float32'), trainable=False, name=layer_name+'_bd')
layer_shape = [-1, wd.get_shape().as_list()[0]]
print('layer_shape:', layer_shape)
fc1_flat = tf.reshape(pool5, shape=layer_shape)
fc1_flat = tf.Print(fc1_flat, [ tf.shape(fc1_flat) ], "fc1_flat shape is ")
fc1 = tf.nn.relu( tf.nn.bias_add( tf.matmul(fc1_flat, wd, name=layer_name), bd ) )
fc1 = tf.Print(fc1, [ tf.shape(fc1) ], "fc1 shape is ")
import time
sess.run(tf.global_variables_initializer())
# evaluate network for in input of (minibatch_size, 7, 7, 512)
minibatch_size = 32
start = time.time()
output = sess.run(fc1, feed_dict = { pool5_input: np.ones((minibatch_size, 7, 7, 512), dtype = 'float32')})
elapsed = time.time() - start
print("time to evaluate fully connected layer for minibatch size %d: %.3f seconds" % (minibatch_size, elapsed))
print("output shape is",output.shape)
I get the following output:
layer_shape: [-1, 25088]
...: I tensorflow/core/kernels/logging_ops.cc:79] pool5 shape is [32 7 7 512]
...: I tensorflow/core/kernels/logging_ops.cc:79] fc1_flat shape is [32 25088]
...: I tensorflow/core/kernels/logging_ops.cc:79] fc1 shape is [32 4096]
time to evaluate fully connected layer for minibatch size 32: 0.329 seconds
output shape is (32, 4096)
so for me it takes less than a second (on a GPU) for a minibatch size of 32.
You could insert similar tf.Print() statements into your code and verify that you have the same (or similar) dimensions. By multiplying the sizes of the dimensions you can see how much memory is used at each stage.