tensorflow 2.5x slower than pytorch on vgg16 architecture - tensorflow

Question

So I'm trying to get into tensorflow and liking it so far.
Today I upgraded to cuda 8, cudnn 5.1 and tensorflow 0.12.1. Using a Maxwell Titan X GPU.
Using the following short code of loading the pretrained vgg16:
import tensorflow as tf
from tensorflow.contrib import slim
from tensorflow.contrib.slim import nets
tf.reset_default_graph()
input_images = tf.placeholder(tf.float32, [None, 224, 224, 3], 'image')
preds = nets.vgg.vgg_16(input_images, is_training=False)[0]
saver = tf.train.Saver()
config = tf.ConfigProto(log_device_placement=True,
gpu_options=tf.GPUOptions(per_process_gpu_memory_fraction = 0.5))
sess = tf.InteractiveSession(config=config)
saver.restore(sess, './vgg_16.ckpt')
_in = np.random.randn(16, 224, 224, 3).astype(np.float32)
I then time the forward pass :
%timeit sess.run(preds, feed_dict={input_images: _in})
I get 160ms per batch (forward pass only), which seems 2.5x slower than the respective configuration in torch according to this benchmark (and also slower than MatconvNet).
The operations seem correctly assigned to the gpu, and the cuda libraries properly found, what else am I missing?
Edit : cudnn and cuda properly found
I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcublas.so locally
I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcudnn.so locally
I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcufft.so locally
I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcuda.so.1 locally
I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcurand.so locally
I tensorflow/core/common_runtime/gpu/gpu_device.cc:885] Found device 0 with properties:
name: GeForce GTX TITAN X
major: 5 minor: 2 memoryClockRate (GHz) 1.076
pciBusID 0000:04:00.0
Total memory: 11.92GiB
Free memory: 11.81GiB
Also the feeding does not seem to be the problem since replacing input_images by tf.random_uniform((16, 224, 224, 3), maxval=255)does not change the timing.
Edit 2: So I compared to the pytorch version running on the same machine and I get (batches of 16x224x224x3) :
Resnet-50 : pytorch 48ms vs tf 58 ms (OK)
VGG16 : pytorch 65ms vs tf 160ms (not OK)

Answer 1

Tested recently on cuda 9.0, tensorflow 1.9 and pytorch 0.4.1, the differences are now negligible for the same operations.
See the proper timing here.