Linear Regression with Neural Networks in Tensorflow and normalization - tensorflow

Question

I've been following this tutorial:
https://blog.altoros.com/using-linear-regression-in-tensorflow.html
I'm aware there's better ways to do linear regression, but I'm using this as a base to do multi-variate regression and multi-variate non-linear regression to try to understand TensorFlow.
Without normalizing my data at all, I get 'nan' with GradientDescentOptimizer. I'm curious about why this is. Why is normalization so important that the model won't run at all? And what about subtracting mean and dividing by standard deviation suddenly makes it work so well?
After normalizing data, I'd like to recover the original value.
Each set of data seems to be normalized separately with its own stddev and mean parameters: the training data X, training data Y, test data X, and test data Y.
However, when I run the network on new data, I'm assuming when I predict new values, I have to normalize the input again. In that case, how do I make sense of the predicted Y? Am I supposed to use the training data's standard deviation and mean to unnormalize, or the new data's standard deviation and mean? I am confused what the model is actually fitting to when I give it normalized training data, and how to interpret W and b. I originally wanted to fit to Y = mx + b, and want to know what m and b really are.
Because I trained on training data, I assumed that I would need to store the training_data's pre-normalization standard deviation and mean and unnormalize any results from the network using this value. But in fact, when I use the new data's standard deviation and mean to unnormalize I get more reasonable values. I don't think it's worth posting that code because I just have a fundamental misunderstanding of what I need to do, but this is the basic code I'm using anyway.
import tensorflow as tf
import numpy
import matplotlib.pyplot as plt
# Train a data set
# X: size data
size_data = [ 2104, 1600, 2400, 1416, 3000, 1985, 1534, 1427,
1380, 1494, 1940, 2000, 1890, 4478, 1268, 2300,
1320, 1236, 2609, 3031, 1767, 1888, 1604, 1962,
3890, 1100, 1458, 2526, 2200, 2637, 1839, 1000,
2040, 3137, 1811, 1437, 1239, 2132, 4215, 2162,
1664, 2238, 2567, 1200, 852, 1852, 1203 ]
# Y: price data (set to 5x + 30)
price_data = [5*c + 30 for c in size_data]
size_data = numpy.asarray(size_data)
price_data = numpy.asarray(price_data)
# Test a data set
size_data_test = [ 1600, 1494, 1236, 1100, 3137, 2238 ]
price_data_test = [5*c + 30 for c in size_data_test]
size_data_test = numpy.asarray(size_data_test)
price_data_test = numpy.asarray(price_data_test)
def normalize(array):
std = array.std()
mean = array.mean()
return (array - mean) / std, std, mean
# Normalize a data set
size_data_n, size_data_n_std, size_data_n_mean = normalize(size_data)
price_data_n, price_data_n_std, price_data_n_mean = normalize(price_data)
size_data_test_n, size_data_test_n_std, size_data_test_n_mean = normalize(size_data_test)
price_data_test_n, price_data_test_n_std, price_data_test_n_mean = normalize(price_data_test)
# Display a plot
#plt.plot(size_data, price_data, 'ro', label='Samples data')
#plt.legend()
#plt.draw()
samples_number = price_data_n.size
# TF graph input
X = tf.placeholder("float")
Y = tf.placeholder("float")
# Create a model
# Set model weights
W = tf.Variable(numpy.random.randn(), name="weight")
b = tf.Variable(numpy.random.randn(), name="bias")
# Set parameters
learning_rate = 0.05
training_iteration = 200
# Construct a linear model
model = tf.add(tf.mul(X, W), b)
# Minimize squared errors
cost_function = tf.reduce_sum(tf.pow(model - Y, 2))/(2 * samples_number) #L2 loss
optimizer = tf.train.GradientDescentOptimizer(learning_rate).minimize(cost_function) #Gradient descent
#optimizer = tf.train.AdagradOptimizer(learning_rate).minimize(cost_function)
# Initialize variables
init = tf.initialize_all_variables()
# Launch a graph
with tf.Session() as sess:
sess.run(init)
display_step = 20
# Fit all training data
for iteration in range(training_iteration):
for (x, y) in zip(size_data_n, price_data_n):
sess.run(optimizer, feed_dict={X: x, Y: y})
# Display logs per iteration step
if iteration % display_step == 0:
print("Iteration:", '%04d' % (iteration + 1), "cost=", "{:.9f}".format(sess.run(cost_function, feed_dict={X:size_data_n, Y:price_data_n})),\
"W=", sess.run(W), "b=", sess.run(b))
tuning_cost = sess.run(cost_function, feed_dict={X: size_data_n, Y: price_data_n})
print("Tuning completed:", "cost=", "{:.9f}".format(tuning_cost), "W=", sess.run(W), "b=", sess.run(b))
# Validate a tuning model
testing_cost = sess.run(cost_function, feed_dict={X: size_data_test_n, Y: price_data_test_n})
print("Testing data cost:" , testing_cost)
Y_predicted = sess.run(model, feed_dict={X: size_data_test_n, Y: price_data_test_n})
print("%-20s%-20s%-20s%-20s" % ("Test X", "Actual", "Target", "Error(%)"))
print('Normalized')
for i in range(len(size_data_test_n)):
err = 100.0 * abs(Y_predicted[i] - price_data_test_n[i]) / abs(price_data_test_n[i])
print("%-20f%-20f%-20f%-20f" % (size_data_test_n[i], Y_predicted[i], price_data_test_n[i], err))
print('Unnormalized')
for i in range(len(size_data_test_n)):
orig_size_data_test_i = size_data_test_n[i] * size_data_test_n_std + size_data_test_n_mean
orig_price_data_test_i = price_data_test_n[i] * price_data_test_n_std + price_data_test_n_mean
# ??? which one is correct for getting unnormalized predicted Y?
#orig_Y_predicted_i = Y_predicted[i] * price_data_n_std + price_data_n_mean
orig_Y_predicted_i = Y_predicted[i] * price_data_test_n_std + price_data_test_n_mean
orig_err = 100.0 * abs(orig_Y_predicted_i - orig_price_data_test_i) / abs(orig_price_data_test_i)
print("%-20f%-20f%-20f%-20f" % (orig_size_data_test_i, orig_Y_predicted_i, orig_price_data_test_i, orig_err))
# Display a plot
plt.figure()
plt.plot(size_data, price_data, 'ro', label='Samples')
plt.plot(size_data_test, price_data_test, 'go', label='Testing samples')
plt.plot(size_data_test, (sess.run(W) * size_data_test_n + sess.run(b))*price_data_n_std + price_data_n_mean , label='Fitted test line')
plt.legend()
plt.show()