Multivariate linear regression using Tensorflow, Keras, Numpy - numpy

Question

I've been studying machine learning and I've become stuck on creating a code for multivariate linear regression.
Here's my training set:
And here is the current code I have at the moment
from keras.models import Sequential
from keras.layers import Dense
from keras.optimizers import SGD
import matplotlib.pyplot as plt
import numpy as np
# Training set
train_x = np.numpy([[400, 180, 200], [430, 140, 305], [405, 255, 300],
[180, 180, 180], [220, 100, 160], [405, 255, 300],
[500, 350, 440], [1500, 900, 200], [1500, 900, 900],
[1000, 1000, 1000]], dtype=float)
train_y = np.array([4.20, 4.85, 6, 3.50, 2.70, 6.50, 11, 20.5, 39.8, 35.3], dtype=float)
# Create Keras model
model = Sequential()
model.add(Dense(1, input_dim=3))
# Gradient descent algorithm
sgd = SGD(0.00000005)
model.compile(loss='mse', optimizer=sgd)
history = model.fit(train_x, train_y, epochs=20000)
plt.plot(history.history['loss'])
plt.xlabel("No. of Iterations")
plt.ylabel("J(Theta1 Theta0)/Cost")
plt.show()
predict = np.array([[100, 100, 100]])
print(model.predict(predict))
When running this the cost function does decrease but doesn't seem to diverge. Also the prediction seems to be quite off as well. (The predict array has lower numbers compared to all data within the training set but it seem to get a price that's higher than some within the training set) Also for some reason I've had to lower my learning rate to a ridiculously low number.
I have a feeling that maybe I'm creating my train_x array wrongly?...

Answer 1

Looking at this code, I can see two problems that might result with bad predictions and the lack of divergence:
Lack of Layers:
A neural network works by optimising weights that are applied on inputs. With the lack of possible inputs to be updated, it has low flexibility and is unable to learn. In this case, there is only one neuron in the layer. I suggest adding more layers such as the one below:
model.add(Dense(25)
Low learning rate
In your example, you used stochastic gradient descent with a learning rate of 0.00000005. I believe that this value is too small for divergence, especially for an algorithm such as SGD. I suggest Adam with a learning rate of 0.1.
Putting all of this together I have a program with:
from keras.models import Sequential
from keras.layers import Dense
from keras.optimizers import adam
import matplotlib.pyplot as plt
import numpy as np
# Training set
train_x = np.array([[400, 180, 200], [430, 140, 305], [405, 255, 300],
[180, 180, 180], [220, 100, 160], [405, 255, 300],
[500, 350, 440], [1500, 900, 200], [1500, 900, 900],
[1000, 1000, 1000]], dtype=float)
train_y = np.array([4.20, 4.85, 6, 3.50, 2.70, 6.50, 11, 20.5, 39.8, 35.3], dtype=float)
# Create Keras model
model = Sequential()
model.add(Dense(1, input_dim=3))
model.add(Dense(25))
model.add(Dense(25))
model.add(Dense(25))
model.add(Dense(1))
# Gradient descent algorithm
adam = adam(0.1)
model.compile(loss='mse', optimizer=adam)
history = model.fit(train_x, train_y, epochs=1000)
plt.plot(history.history['loss'])
plt.xlabel("No. of Iterations")
plt.ylabel("J(Theta1 Theta0)/Cost")
plt.show()
predict = np.array([[100, 100, 100]])
print(model.predict(predict))
This program allowed for faster divergence (only 1000 epochs) and a lower final loss value as compared to the original post.