Trouble restoring checkpointed TensorFlow net - tensorflow

Question

I have built an auto encoder to "convert" the activations from VGG19.relu4_1 into pixels. I use the new convenience functions in tensorflow.contrib.layers (as in TF 0.10rc0). The code is have similar layout as TensorFlow's CIFAR10 tutorial with a train.py that does the training and checkpoints the model to disk and one eval.py that polls for new checkpoints files and run inference on them.
My problem is that the evaluation is never as good as the training, neither in terms of the value of the loss function nor when I look at the output images (even when running on the same images as the training does). This makes me think it has something to do with the restore process.
When I look at the output from the training in TensorBoard it looks good (eventually) so I don't think there is anything wrong with my net per se.
My net looks like this:
import tensorflow.contrib.layers as contrib
bn_params = {
"is_training": is_training,
"center": True,
"scale": True
}
tensor = contrib.convolution2d_transpose(vgg_output, 64*4, 4,
stride=2,
normalizer_fn=contrib.batch_norm,
normalizer_params=bn_params,
scope="deconv1")
tensor = contrib.convolution2d_transpose(tensor, 64*2, 4,
stride=2,
normalizer_fn=contrib.batch_norm,
normalizer_params=bn_params,
scope="deconv2")
.
.
.
And in train.py I do this to save the checkpoint:
variable_averages = tf.train.ExponentialMovingAverage(mynet.MOVING_AVERAGE_DECAY)
variables_averages_op = variable_averages.apply(tf.trainable_variables())
with tf.control_dependencies([apply_gradient_op, variables_averages_op]):
train_op = tf.no_op(name='train')
while training:
# train (with batch normalization's is_training = True)
if time_to_checkpoint:
saver.save(sess, checkpoint_path, global_step=step)
In eval.py I do this:
# run code that creates the net
variable_averages = tf.train.ExponentialMovingAverage(
mynet.MOVING_AVERAGE_DECAY)
saver = tf.train.Saver(variable_averages.variables_to_restore())
while polling:
# sleep and check for new checkpoint files
with tf.Session() as sess:
init = tf.initialize_all_variables()
init_local = tf.initialize_local_variables()
sess.run([init, init_local])
saver.restore(sess, checkpoint_path)
# run inference (with batch normalization's is_training = False)
The blue is the training loss, and the orange is the eval loss.

Answer 1

The problem was that I used the tf.train.AdamOptimizer() directly. During the optimization it didn't call the operations defined in contrib.batch_norm to calculate the running mean/variance of the input so the mean/variance was always 0.0/1.0.
The solution is to add a dependency to the GraphKeys.UPDATE_OPS collection. There already is a function defined in the contrib module that does this (optimize_loss())