I am learning ML through TensorFlow (tfjs).
My first test was to train my model to predict cos(x) as a function of x (from 0 to 2*Math.PI*4 aka 4 periods)
feature: 2000 values of x (random)
label: 2000 values of cos(x)
model:
const model = tf.sequential({
layers: [
tf.layers.dense({ inputShape: [1], units: 22, activation: 'tanh' }),
tf.layers.dense({ units: 1 }),
]
});
model.compile({
optimizer: tf.train.adam(0.01),
loss: 'meanSquaredError',
metrics: ['mae']
});
...
await model.fit(feature, label, {
epochs: 500,
validationSplit: 0.2,
})
The result is quite "fun":
Now I would like to know how to enhance my model to fit with the periodicity nature of cos(x) (without using the mathematical periodicity of cos(x) like y = cos(x modulo 2PI) ).
Is it possible for my model to "understand" that there is a periodicity ?
I think that the network you built is too small to learn the periodic behaviour of the cosine function (try increasing the number of hidden units and/or adding hidden layers), also I don't think a regular (fully connected neural network) is the right choice if you want to learn a function that has a periodic sequential nature, try using an RNN or LSTM for this.
Related
I would like to use online training with a deep LSTM model.
When using online training, my batch size is 1.
I would also like the model to be stateful.
As such, I don't want my input to be a sequence, but rather a single state. Thus the number of steps in my input is also 1. I would like the model to learn the sequence based on the individual input states
Thus, I've constructed the following model (Tensorflow JS):
this.model = tf.sequential();
this.model.add(tf.layers.lstm({ units: 256, returnSequences: false, batchInputShape: [1, 1, input_dim], stateful: true }));
this.model.add(tf.layers.dense({ units: output_dim}));
this.model.compile({ optimizer: 'adam', loss: 'meanSquaredError' });
this.model.summary();
This model works perfectly fine, however when I want to stack more LSTM layers, for example trying the following model:
this.model = tf.sequential();
this.model.add(tf.layers.lstm({ units: 256, returnSequences: true, batchInputShape: [1, 1, input_dim], stateful: true }));
this.model.add(tf.layers.lstm({ units: 128, returnSequences: false}));
this.model.add(tf.layers.dense({ units: output_dim}));
this.model.compile({ optimizer: 'adam', loss: 'meanSquaredError' });
Above I've changed returnSequences to true in the first LSTM layer so that its output shape can be accepted by the second LSTM layer.
When attempting to fit on the same input and output that works with the first model on the second model, I receive the following error:
Error: Argument tensors passed to stack must be a `Tensor[]` or `TensorLike[]`
What does this error mean?
As a beginner, I have tried build a really simple multi-class classifier in tensorflowJS which is suppose to predict the direction of my eye sight.
Step 1: I created data set in the browser to train my model where I am storing images of my eyes rendered by webcam on a HTML5 canvas. I use arrow keys to label my images as 0=left,1=normal and 2=right. To train the model, I convert these lables using tf.onHot() before passing to the method.
// data collection
let imageArray = [];
let labelArray = [];
let collectData = (label) => {
const img = tf.tidy(() => {
const captureImg = getImage();
//console.log(captureImg.shape)
return captureImg;
})
imageArray.push(img)
labelArray.push(label) //--- labels are 0,1,2
}
// label conversion
let labelSet = tf.oneHot(tf.tensor1d(labelArray, 'int32'), 3);
Step 2: Instead of loading any per-trained model, I used my custom model that I built using tensorflowJS.
let createModel = () => {
const model = tf.sequential();
let config_one = {
kernelSize: 3,
filters: 40,
strides: 1,
activation: 'relu',
inputShape: [imageHeight, imageWidth, imageChannels]
}
model.add(tf.layers.conv2d(config_one));
let config_two = {
poolSize: [2, 2],
strides: [2, 2],
}
model.add(tf.layers.maxPooling2d(config_two));
model.add(tf.layers.flatten());
model.add(tf.layers.dropout(0.2));
// Two output values x and y
let congfig_output = {
units: 3,
activation: 'tanh',
}
model.add(tf.layers.dense(congfig_output));
// Use ADAM optimizer with learning rate of 0.0005 and MSE loss
let config_compile = {
optimizer: tf.train.adam(0.00005),
loss: 'categoricalCrossentropy',
}
model.compile(config_compile);
tf.memory()
return model;
}
Problems : There are several problems I am facing right now.
When I use meanSquared as loss function and adam learning rate 0.000005, my model starts predicting but it only predicts two of the eye's state normal and left/right so to do multi-class classification, I changed loss function to categoricalCrossentropy but the result is still same or sometime worst.
I tried other combination of hyper parameters but no luck. The worst situation I got into was my loss function was showing only three constant values repeatedly.
My browser would crashed in some case where - if - I pass too much data or use other type of optimizer in compile config such as sgd or anything else. When I did a quick search on google, I found I can use tf.memory() to check any memory leak which could be causing browser crash but that line didn't log anything in the console.
I was adjusting various values and parameters in the code and training the model which made it work sometimes, partially, and most of the time didn't even work. It was all hit and trial. Eventually I learned about parameters to use for loss function in the compile method and activation function in con2d input layer but other stuff is still confusing such as - number of epochs, batch size, learning rate in adam etc.
I understood or I think I understood these - kernalsize, filters, strides, inputshape but still have no idea how to decide number of layers various hyper parameters etc.
Edit - this is what I get after updating the code as per the suggestion. I still don't proper classification. I am training with minimum of 1000+ images.
A. I still get the loss recurring with fixed valeus
B. Accuracy is also repeating itself with 1, 0.5 and 0
function getImage() {
return tf.tidy(function () {
const image = tf.browser.fromPixels($('#eyes')[0]);
const batchedImage = image.expandDims(0);
const norm = batchedImage.toFloat().div(tf.scalar(255)).sub(tf.scalar(1));
return norm;
});
}
Here are the console output
Sample images -
Most obvious thing to me that is wrong with this is your output layer's activation function, where you use tanh you should be using softmax instead. Next, your learning rate is way to low try setting it to 0.001 which is a good default to use.
You also probably don't need dropout as you have not gotten any results to justify that the model is overfitting. You could also add in more convolutional layers to this, try the example below.
model.add(tf.layers.conv2d({
inputShape: [28, 28, 1],
kernelSize: 5,
filters: 8,
strides: 1,
activation: 'relu',
}));
model.add(tf.layers.maxPooling2d({
poolSize: [2, 2],
strides: [2, 2],
}));
model.add(tf.layers.conv2d({
kernelSize: 5,
filters: 16,
strides: 1,
activation: 'relu',
}));
model.add(tf.layers.maxPooling2d({
poolSize: [2, 2],
strides: [2, 2],
}));
model.add(tf.layers.flatten());
model.add(tf.layers.dense({
units: 3,
activation: 'softmax',
}));
const LEARNING_RATE = 0.001;
const optimizer = tf.train.adam(LEARNING_RATE);
model.compile({
optimizer: optimizer,
loss: 'categoricalCrossentropy',
metrics: ['accuracy'],
});
I've been trying to set up a simple reinforcement learning example using tfjs. However, when trying to train the model I am running into the following error:
Uncaught (in promise) Error: Error when checking target: expected dense_Dense5 to have shape [,1], but got array with shape [3,4]
I built the model up as following:
const NUM_OUTPUTS = 4;
const model = tf.sequential();
//First hidden Layer, which also defines the input shape of the model
model.add(
tf.layers.dense({
units: LAYER_1_UNITS,
batchInputShape: [null, NUM_INPUTS],
activation: "relu",
})
);
// Second hidden Layer
model.add(tf.layers.dense({ units: LAYER_2_UNITS, activation: "relu" }));
// Third hidden Layer
model.add(tf.layers.dense({ units: LAYER_3_UNITS, activation: "relu" }));
// Fourth hidden Layer
model.add(tf.layers.dense({ units: LAYER_4_UNITS, activation: "relu" }));
// Defining the output Layer of the model
model.add(tf.layers.dense({ units: NUM_OUTPUTS, activation: "relu" }));
model.compile({
optimizer: tf.train.adam(),
loss: "sparseCategoricalCrossentropy",
metrics: "accuracy",
});
The training is done by a function that calculates the Q-values for some examples:
batch.forEach((sample) => {
const { state, nextState, action, reward } = sample;
// We let the model predict the rewards of the current state.
const current_Q: tf.Tensor = <tf.Tensor>model.predict(state);
// We also let the model predict the rewards for the next state, if there was a next state in the
//game.
let future_reward = tf.zeros([NUM_ACTIONS]);
if (nextState) {
future_reward = <Tensor>model.predict(nextState);
}
let totalValue =
reward + discountFactor * future_reward.max().dataSync()[0];
current_Q.bufferSync().set(totalValue, 0, action);
// We can now push the state to the input collector
x = x.concat(Array.from(state.dataSync()));
// For the labels/outputs, we push the updated Q values
y = y.concat(Array.from(current_Q.dataSync()));
});
await model.fit(
tf.tensor2d(x, [batch.length, NUM_INPUTS]),
tf.tensor2d(y, [batch.length, NUM_OUTPUTS]),
{
batchSize: batch.length,
epochs: 3,
}
);
This appeared to be the right way to provide the examples to the fit function, seeing as when logging the model, the shape of the last dense layer is correct:
Log of the shape of dense_Dense5
However it results in the error shown above, where instead of the expected shape [3,4] it checks for the shape [,1]. I really dont understand where this shape is suddenly coming from and would much appreciate some help with this!
For a better overview, you can simply view/check out the whole project from its Github repo:
Github Repo
The tensorflow code in question is in the AI folder.
EDIT:
Providing a summary of the model plus some info of the shape of the tensor im providing for y in model.fit(x,y) :
Solved: Issue occured due to using the wrong loss function. Moving from categoricalCrossEntropy to meanSquaredError fixed the issue with the shape of the output layer mismatching the batch shape.
I am attempting to load a frozen graph created through TensorFlow Keras (.pb) onto a memory limited microcontroller. Via Hyperopt(...) I am optimizing my training hyperparameters, however I would like to include the size of the resulting model as part of the search space. At the moment, I'm including using the weighted loss as an input to hyperopt, something like the following:
def optimizer(args):
...
training_history = nn.train_v00(....)
file_bytes = int(os.path.getsize(frozen_graph_filepath)) # returns size in bytes
final_loss = training_history.history['loss'][-1]
weighted_loss = final_loss*(file_bytes/(100*1024)) # Want model smaller than 100KB
if OPTIMIZATION_TARGET == 'loss':
return_struct = {
'status': STATUS_OK,
'loss': weighted_loss,
'epochs': epochs,
'metrics': {
'accuracy': final_acc
}
}
return return_struct
space = {
'learning_rate': hp.loguniform('learning_rate', np.log(0.00001), np.log(0.05)),
'dropout': hp.uniform('dropout', 0, 0.5),
'batch_size': hp.quniform('batch_size', 32, 128, 2),
'input_seq_len': hp.quniform('seq_len', 32, N_ADC_SAMPLES, 2),
'cnn_n_filters': hp.quniform('cnn_n_filters', 1, 10, 1),
'cnn_kernel_size': hp.quniform('cnn_kernel_size', 1, 10, 1),
'cnn_pool_size': hp.quniform('cnn_pool_size', 1, 10, 1)
}
t = Trials()
best = fmin(optimizer, space, algo=tpe.suggest, max_evals=MAX_EVALS, trials=t)
From what I have found thus far, there isn't a way to directly backpropagate the size of the model back through the training, but is there a better way of doing this?
Thanks for the consideration!
If I understand you correctly this is a question about how to craft your loss function, not about hyperopt directly. Correct? If so, if you indeed have a hard cut-off at 100kb, I wouldn't scale your training loss (as it might lead to strange artifacts like a poor performing, but very tiny model taking the first spot).
How about you report the regular training loss if the model size is smaller than 100kb and otherwise return an astronomically large number? Just an idea.
I am trying to learn and practice on Tensorflow.js.
So, I tried to train a neural network on a [,2] shaped array as x (as I understood, this would simulate a problem where I have x samples that each one has 2 variables) and a [,1] array as y (what would mean if I'm correct, that the combination of my 2 variables generate 1 output).
And I tried to code it:
const model = tf.sequential();
model.add(tf.layers.dense({ units: 2, inputShape: [2] }));
model.add(tf.layers.dense({ units: 64, inputShape: [2] }));
model.add(tf.layers.dense({ units: 1, inputShape: [64] }));
// Prepare the model for training: Specify the loss and the optimizer.
model.compile({ loss: 'meanSquaredError', optimizer: 'sgd' });
// Generate some synthetic data for training.
const xs = tf.tensor([[1,5], [2,10], [3,15], [4,20], [5,25], [6,30], [7,35], [8,40]], [8, 2]);
const ys = tf.tensor([1, 2, 3, 4, 5, 6, 7, 8], [8, 1]);
// Train the model using the data.
model.fit(xs, ys, { epochs: 100 }).then(() => {
// Use the model to do inference on a data point the model hasn't seen before:
// Open the browser devtools to see the output
model.predict(tf.tensor([10, 50], [1, 2])).print();
});
But, what I am facing is that when I try to predict the [10,50] input, I have the following console output:
Tensor
[[NaN],]
So, I think my problem might be very simple, but I am really stuck with this and probably it is a matter of some background knowledge I'm missing.
Thank you!
The first layer takes the shape of the input data
model.add(tf.layers.dense({ units: 2, inputShape: [2] }))
The inputShape is [2], which means that your input x is of shape [2].
The last layer unit value gives the dimension of the output y.
model.add(tf.layers.dense({ units: 1, inputShape: [64] }));
So the shape of y should be [1]
In this case, the NaN prediction is related to the number of epochs for your training. If you decrease it to 2 or 3, it will return a numerical value. Actually, the error is related to how your optimizer is updating the weights. Alternatively, you can change the optimizer to adam and it will be fine.
I think I am late but I hope this helps someone.
I got same problem once and it was because I am getting training and testing data from file using "fs" dependency and I solved the problem by just doing this to the returned variable before returning it to the main function to start training:
JSON.parse(JSON.stringify(data))
I don't know the reason but for some reason the tensorflow model only accepts JSON Array and not any JavaScript array so just by doing this you are converting your array to json array instead of leaving it as JavaScript Array.
Hope this saves someone's time.
I dealt with this same issue for the past 2 days, and the problem was that I trained my model with a GPU (using Google Colab) and performed inference on the CPU. After changing the settings on Google Colab to use no hardware acceleration, my problem was fixed!
Hope this helps someone in the future.