Network diverges with NaN in simple TensorFlow example - tensorflow

Question

I am trying to follow the example from Stanford series on TF by implementing a quadratic linear regression.
Y = W*X*X + u*X + b
The dataset can be found in Cengage dataset; and the code is the following:
import numpy as np
import matplotlib.pyplot as plt
import tensorflow as tf
import xlrd
DATA = 'data\\slr05.xls'
# Read data
data = xlrd.open_workbook(DATA, encoding_override='utf-8')
sheet = data.sheet_by_index(0)
dataset = np.asarray([sheet.row_values(i) for i in range(1, sheet.nrows)])
n_samples = sheet.nrows - 1
X = tf.placeholder('float', name = 'X')
Y = tf.placeholder('float', name = 'Y')
W = tf.Variable(0.0, name = 'weights')
b = tf.Variable(0.0, name = 'bias')
u = tf.Variable(0.0, name = 'u_weight')
Y_ = X*X*W + X*u + b
loss = tf.square(Y - Y_, name = 'loss')
optimizer = tf.train.GradientDescentOptimizer(0.0001).minimize(loss)
init = tf.global_variables_initializer()
loss_average = []
# Start the Session
with tf.Session() as sess:
sess.run(init)
for i in range(10):
for x, y in dataset:
print(sess.run([optimizer, Y_, W, b, u, X, Y], feed_dict = {X:x, Y:y}))
loss_average.append(sess.run(loss, feed_dict = {X:x, Y:y}))
The final W, b, and u values that I get are nan. I tried to check step-by-step why this is happening. So, in the output below I have included the [optimizer, Y_, W, b, u, X, Y]
and after a few row iterations I get:
[None, 3.9304674e+33, -1.0271335e+33, -7.7725354e+29, -2.8294217e+31, 36.2, 41.]
[None, -1.619979e+36, inf, 3.2321854e+32, 1.2834338e+34, 39.7, 147]
Apparently, during optimization the W ends up to 'inf', which breaks down the regression output.
Any, idea what have I done wrong?

Answer 1

You have an exploding gradient problem here. That's because your X and Y, and consequently difference values are in the magnitude of 101, so the square differences (you loss) are of magnitude 102. When you introduce the X2 into the regression, your difference values will be in the magnitude of 102, their squares of magnitude 104. Therefore the gradients will be much larger and the network diverges violently.
To correct for this, you can reduce the learning rate by a factor of 10-3, to put the gradients roughly back where they were, and lo and behold, this code (tested):
import numpy as np
import matplotlib.pyplot as plt
import tensorflow as tf
import xlrd
DATA = 'slr05.xls'
# Read data
data = xlrd.open_workbook(DATA, encoding_override='utf-8')
sheet = data.sheet_by_index(0)
dataset = np.asarray([sheet.row_values(i) for i in range(1, sheet.nrows)])
n_samples = sheet.nrows - 1
X = tf.placeholder('float', name = 'X')
Y = tf.placeholder('float', name = 'Y')
W = tf.Variable(0.0, name = 'weights')
b = tf.Variable(0.0, name = 'bias')
u = tf.Variable(0.0, name = 'u_weight')
Y_ = X*X*W + X*u + b
#Y_ = X * u + b
loss = tf.square(Y - Y_, name = 'loss')
optimizer = tf.train.GradientDescentOptimizer(0.0000001).minimize(loss)
init = tf.global_variables_initializer()
loss_average = []
# Start the Session
with tf.Session() as sess:
sess.run(init)
for i in range(10):
for x, y in dataset:
print(sess.run([optimizer, loss, Y_, W, b, u, X, Y], feed_dict = {X:x, Y:y}))
loss_average.append(sess.run(loss, feed_dict = {X:x, Y:y}))
will obediently and orderly converge, as nice networks do, outputting (last 5 lines only):
[None, 1313.2705, 9.760924, 0.06911032, 0.0014081484, 0.010015297, array(11.9, dtype=float32), array(46., dtype=float32)]
[None, 1174.7083, 7.7259817, 0.06986606, 0.0014150032, 0.010087272, array(10.5, dtype=float32), array(42., dtype=float32)]
[None, 1217.4297, 8.1083145, 0.07066501, 0.0014219815, 0.01016194, array(10.7, dtype=float32), array(43., dtype=float32)]
[None, 657.74097, 8.353538, 0.07126329, 0.0014271108, 0.010217336, array(10.8, dtype=float32), array(34., dtype=float32)]
[None, 299.5538, 1.6923765, 0.07134304, 0.0014305722, 0.010233952, array(4.8, dtype=float32), array(19., dtype=float32)]