Output probability of prediction in tensorflow.js - tensorflow

I have a model.json generated from tensorflow via tensorflow.js coverter
In the original implementation of model in tensorflow in python, it is built like this:
model = models.Sequential([
base_model,
layers.Dropout(0.2),
layers.Flatten(),
layers.Dense(128, activation='relu'),
layers.Dense(num_classes)
])
In tensorflow, the probability can be generated by score = tf.nn.softmax(predictions[0]), according to the tutorial on official website.
How do I get this probability in tensorflow.js?
I have copied the codes template as below:
$("#predict-button").click(async function () {
if (!modelLoaded) { alert("The model must be loaded first"); return; }
if (!imageLoaded) { alert("Please select an image first"); return; }
let image = $('#selected-image').get(0);
// Pre-process the image
console.log( "Loading image..." );
let tensor = tf.browser.fromPixels(image, 3)
.resizeNearestNeighbor([224, 224]) // change the image size
.expandDims()
.toFloat()
// RGB -> BGR
let predictions = await model.predict(tensor).data();
console.log(predictions);
let top5 = Array.from(predictions)
.map(function (p, i) { // this is Array.map
return {
probability: p,
className: TARGET_CLASSES[i] // we are selecting the value from the obj
};
}).sort(function (a, b) {
return b.probability - a.probability;
}).slice(0, 2);
console.log(top5);
$("#prediction-list").empty();
top5.forEach(function (p) {
$("#prediction-list").append(`<li>${p.className}: ${p.probability.toFixed(6)}</li>`);
});
How should I modify the above code?
The output is just the same as the value of variable 'predictions':
Float32Array(5)
0: -2.5525975227355957
1: 7.398464679718018
2: -3.252196788787842
3: 4.710395812988281
4: -4.636396408081055
buffer: (...)
byteLength: (...)
byteOffset: (...)
length: (...)
Symbol(Symbol.toStringTag): (...)
__proto__: TypedArray
0: {probability: 7.398464679718018, className: "Sunflower"}
1: {probability: 4.710395812988281, className: "Rose"}
length: 2
__proto__: Array(0)
Please help!!!
Thanks!

In order to extract the probabilities from the logits of the model using a softmax function you can do the following:
This is the array of logits that are also the predictions you get from the model
const logits = [-2.5525975227355957, 7.398464679718018, -3.252196788787842, 4.710395812988281, -4.636396408081055]
You can call tf.softmax() on the array of values
const probabilities = tf.softmax(logits)
Result:
[0.0000446, 0.9362511, 0.0000222, 0.0636765, 0.0000056]
Then if you wanted to get the index with the highest probability you can make use of tf.argMax():
const results = tf.argMax(probabilities).dataSync()[0]
Result:
1
Edit
I am not too familiar with jQuery so this might not be correct. But here is how I would get the probabilities of the outputs in descending order:
let probabilities = tf.softmax(predictions).dataSync();
$("#prediction-list").empty();
probabilities.forEach(function(p, i) {
$("#prediction-list").append(
`<li>${TARGET_CLASSES[i]}: ${p.toFixed(6)}</li>`
);
});

Related

Tensorflow: converting H5 layer model to TFJS version leads to Unknown layer: TensorFlowOpLayer error when it works in TS

I'm trying to run the converted model from the repository: https://github.com/HasnainRaz/Fast-SRGAN. Well, the conversion was successful. But when I tried to initialize the model, I saw the error: "Unknown layer: TensorFlowOpLayer.". If we will investigate the saved model, we can see TensorFlowOpLayer:
The model structure
As I understood it is this peace of code:
keras.layers.UpSampling2D(size=2, interpolation='bilinear')(layer_input).
I decided to write my own class "TensorFlowOpLayer".
import * as tf from '#tensorflow/tfjs';
export class TensorFlowOpLayer extends tf.layers.Layer {
constructor() {
super({});
}
computeOutputShape(shape: Array<number>) {
return [1, null, null, 32];
}
call(input_3): tf.Tensor {
const result = tf.layers.upSampling2d({ size: [2, 2], dataFormat: 'channelsLast', interpolation: 'bilinear' }).apply(input_3) as tf.Tensor;
return result;
}
static get className() {
return 'TensorFlowOpLayer';
}
}
But it doesn't work. Can someone help me to understand how to write to the method "computeOutputShape"?
And second misunderstanding, why on the picture above we see the next order of layers:
Conv2D -> TensorFlowOpLayer -> PReLU
As I understood the TensorFlowOpLayer layer is "UpSampling2D" in the python code. The H5 model was investigated through the site: https://netron.app
u = keras.layers.UpSampling2D(size=2, interpolation='bilinear')(layer_input)
u = keras.layers.Conv2D(self.gf, kernel_size=3, strides=1, padding='same')(u)
u = keras.layers.PReLU(shared_axes=[1, 2])(u)
The initializing of the model in TS:
async loadModel() {
this.model = await tf.loadLayersModel('/assets/fast_srgan/model.json');
const inputs = tf.layers.input({shape: [null, null, 32]});
const outputs = this.model.apply(inputs) as tf.SymbolicTensor;
this.model = tf.model({inputs: inputs, outputs: outputs});
console.log("Model has been loaded");
}
like in python code:
from tensorflow import keras
# Load the model
model = keras.models.load_model('models/generator.h5')
# Define arbitrary spatial dims, and 3 channels.
inputs = keras.Input((None, None, 3))
# Trace out the graph using the input:
outputs = model(inputs)
# Override the model:
model = keras.models.Model(inputs, outputs)
Then, how is it used:
tf.tidy(() => {
let img = tf.browser.fromPixels(this.imgLr.nativeElement, 3);
img = tf.div(img, 255.0);
img = tf.image.resizeNearestNeighbor(img, [96, 96]);
img = tf.expandDims(img, 0);
let sr = this.model.predict(img) as tf.Tensor;
});
like in python code:
def predict(img):
# Rescale to 0-1.
lr = tf.math.divide(img, 255)
# Get super resolution image
sr = model.predict(tf.expand_dims(lr, axis=0))
return sr[0]
When I added my own class "TensorFlowOpLayer" I see the next error:
"expected input1 to have shape [null,null,null,32] but got array with shape [1,96,96,3]."
Solved the issue. The issue related to the version of the code and the saved model. The author of the code refactored the code and didn't change the saved model. I rewrote the needed class:
import * as tf from '#tensorflow/tfjs';
export class DepthToSpace extends tf.layers.Layer {
constructor() {
super({});
}
computeOutputShape(shape: Array<number>) {
return [null, ...shape.slice(1, 3).map(x => x * 2), 32];
}
call(input): tf.Tensor {
input = input[0];
const result = tf.depthToSpace(input, 2);
return result;
}
static get className() {
return 'TensorFlowOpLayer';
}
}
and it works.
The author's original code is:
u = keras.layers.Conv2D(filters, kernel_size=3, strides=1, padding='same')(layer_input)
u = tf.nn.depth_to_space(u, 2)
u = keras.layers.PReLU(shared_axes=[1, 2])(u)

TensorflowJS: how to reset input/output shapes for pretrained model in TFJS

For the pre-trained model in python we can reset input/output shapes:
from tensorflow import keras
# Load the model
model = keras.models.load_model('models/generator.h5')
# Define arbitrary spatial dims, and 3 channels.
inputs = keras.Input((None, None, 3))
# Trace out the graph using the input:
outputs = model(inputs)
# Override the model:
model = keras.models.Model(inputs, outputs)
The source code
I'm trying to do the same in TFJS:
// Load the model
this.model = await tf.loadLayersModel('/assets/fast_srgan/model.json');
// Define arbitrary spatial dims, and 3 channels.
const inputs = tf.layers.input({shape: [null, null, 3]});
// Trace out the graph using the input.
const outputs = this.model.apply(inputs) as tf.SymbolicTensor;
// Override the model.
this.model = tf.model({inputs: inputs, outputs: outputs});
TFJS does not support one of the layers in the model:
...
u = keras.layers.Conv2D(filters, kernel_size=3, strides=1, padding='same')(layer_input)
u = tf.nn.depth_to_space(u, 2) # <- TFJS does not support this layer
u = keras.layers.PReLU(shared_axes=[1, 2])(u)
...
I wrote my own:
import * as tf from '#tensorflow/tfjs';
export class DepthToSpace extends tf.layers.Layer {
constructor() {
super({});
}
computeOutputShape(shape: Array<number>) {
// I think the issue is here
// because the error occurs during initialization of the model
return [null, ...shape.slice(1, 3).map(x => x * 2), 32];
}
call(input): tf.Tensor {
const result = tf.depthToSpace(input[0], 2);
return result;
}
static get className() {
return 'TensorFlowOpLayer';
}
}
Using the model:
tf.tidy(() => {
let img = tf.browser.fromPixels(this.imgLr.nativeElement, 3);
img = tf.div(img, 255);
img = tf.expandDims(img, 0);
let sr = this.model.predict(img) as tf.Tensor;
sr = tf.mul(tf.div(tf.add(sr, 1), 2), 255).arraySync()[0];
tf.browser.toPixels(sr as tf.Tensor3D, this.imgSrCanvas.nativeElement);
});
but I get the error:
Error: Input 0 is incompatible with layer p_re_lu: expected axis 1 of input shape to have value 96 but got shape 1,128,128,32.
The pre-trained model was trained with 96x96 pixels images. If I use the 96x96 image, it works. But if I try to use other sizes (for example 128x128), It doesn't work. In python, we can easily reset input/output shapes. Why it doesn't work in JS?
To define a new model from the layers of the previous model, you need to use tf.model
this.model = tf.model({inputs: inputs, outputs: outputs});
I tried to debug this class:
import * as tf from '#tensorflow/tfjs';
export class DepthToSpace extends tf.layers.Layer {
constructor() {
super({});
}
computeOutputShape(shape: Array<number>) {
return [null, ...shape.slice(1, 3).map(x => x * 2), 32];
}
call(input): tf.Tensor {
const result = tf.depthToSpace(input[0], 2);
return result;
}
static get className() {
return 'TensorFlowOpLayer';
}
}
and saw: when I do not try to rewrite the size, the computeOutputShape, method works only twice, and it works 4 times when I try to reset inputs/outputs. Well, then I opened the model's JSON file and changed inputs from [null, 96, 96, 32] to [null, 128, 128, 32] and removed these lines:
// Define arbitrary spatial dims, and 3 channels.
const inputs = tf.layers.input({shape: [null, null, 3]});
// Trace out the graph using the input.
const outputs = this.model.apply(inputs) as tf.SymbolicTensor;
// Override the model.
this.model = tf.model({inputs: inputs, outputs: outputs});
And now it works with 128x128 images. It looks like the piece of code above, adds the layers instead of rewriting them.

How to use vectors created by P5 createVector as a tensor in tensorflow.js

I am using p5 to return the vector path of a drawn line. All the vectors in the line are pushed into an array that holds all the vectors. I'm trying to use this as a tensor but I keep getting an error saying
Error when checking model input: the Array of Tensors that you are passing to your model is not the size the model expected. Expected to see 1 Tensor(s), but instead got the following list of Tensor(s):
When I opened the array on the dev tool, each vector was printed like this:
0: Vector {p5: p5, x: 0.5150300601202404, y: -0.25450901803607207, z: 0}
could it be the p5 text in the vector array that's giving me the error? Here's my model and fit code:
let vectorpath = []; //vector path array
// model, setting layers till next '-----'
const model = tf.sequential();
model.add(tf.layers.dense({units: 4, inputShape: [2, 2], activation: 'sigmoid'}));
model.add(tf.layers.dense({units: 2, activation: 'sigmoid'}));
console.log(JSON.stringify(model.outputs[0].shape));
model.weights.forEach(w => {
console.log(w.name, w.shape);
});
// -----
//this is under the draw function so it is continually updated
const labels = tf.randomUniform([0, 1]);
function onBatchEnd(batch, logs) {
console.log('Accuracy', logs.acc);
}
model.fit(vectorpath, labels, {
epochs: 5,
batchSize: 32,
callbacks: {onBatchEnd}
}).then(info => {
console.log('Final accuracy', info.history.acc);
});
What could be causing the error? and how can I fix it?
The question's pretty vague but I'm really just not sure.

TFJS predict vs Python predict

I trained my model using Keras in Python and I converted my model to a tfjs model to use it in my webapp. I also wrote a small prediction script in python to validate my model on unseen data. In python it works perfectly, but when I'm trying to predict in my webapp it goes wrong.
This is the code I use in Python to create tensors and predict based on these created tensors:
input_dict = {name: tf.convert_to_tensor([value]) for name, value in sample_v.items()}
predictions = model.predict(input_dict)
classes = predictions.argmax(axis=-1)
In TFJS however it seems I can't pass a dict (or object) to the predict function, but if I write code to convert it to a tensor array (like I found on some places online), it still doesn't seem to work.
Object.keys(input).forEach((k) => {
input[k] = tensor1d([input[k]]);
});
console.log(Object.values(input));
const prediction = await model.executeAsync(Object.values(input));
console.log(prediction);
If I do the above, I get the following error: The shape of dict['key_1'] provided in model.execute(dict) must be [-1,1], but was [1]
If I then convert it to this code:
const input = { ...track.audioFeatures };
Object.keys(input).forEach((k) => {
input[k] = tensor2d([input[k]], [1, 1]);
});
console.log(Object.values(input));
I get the error that some dtypes have to be int32 but are float32. No problem, I can set the dtype manually:
const input = { ...track.audioFeatures };
Object.keys(input).forEach((k) => {
if (k === 'int_key') {
input[k] = tensor2d([input[k]], [1, 1], 'int32');
} else {
input[k] = tensor2d([input[k]], [1, 1]);
}
});
console.log(Object.values(input));
I still get the same error, but if I print it, I can see the datatype is set to int32.
I'm really confused as to why this is and why I can't just do like python and just put a dict (or object) in TFJS, and how to fix the issues I'm having.
Edit 1: Complete Prediction Snippet
const model = await loadModel();
const input = { ...track.audioFeatures };
Object.keys(input).forEach((k) => {
if (k === 'time_signature') {
input[k] = tensor2d([parseInt(input[k], 10)], [1, 1], 'int32');
} else {
input[k] = tensor2d([input[k]], [1, 1]);
}
});
console.log(Object.values(input));
const prediction = model.predict(Object.values(input));
console.log(prediction);
Edit 2: added full errormessage

Tensorflow map_fn gives error: ValueError: No attr named '_XlaCompile'

I try to implement the 'batch hard' batches as described in https://arxiv.org/pdf/1703.07737.pdf to use with a triplet loss. So input is of shape [batch_size, 32] and output should be a list representing triplets, so [[batch_size, 32], [batch_size, 32], [batch_size, 32]] when each individual example is of size (32,).
I implemented this with the following function, so basically using tf.map_fn:
def batch_hard(inputs):
"""
Batch Hard triplets as described in https://arxiv.org/pdf/1703.07737.pdf.
For each sample in input the hardest positive and hardest negative
in the given batch will be selected. A triplet is returned.
"""
class_ids, f_anchor = inputs[0], inputs[1]
def body(x):
class_id, f = x[0], x[1]
same_class = tf.equal(class_ids, class_id)
positive = same_class
negative = tf.logical_not(same_class)
positive = tf.squeeze(positive)
negative = tf.squeeze(negative)
positive.set_shape([None])
negative.set_shape([None])
samples_pos = tf.boolean_mask(f_anchor, positive)
samples_neg = tf.boolean_mask(f_anchor, negative)
# Select hardest positive example
distances = euclidean_distance(samples_pos, f)
hardest_pos = samples_pos[tf.argmax(distances)]
# Select hardest negative example
distances = euclidean_distance(samples_neg, f)
hardest_neg = samples_neg[tf.argmin(distances)]
return [hardest_pos, hardest_neg]
[f_pos, f_neg] = tf.map_fn(body, inputs, dtype=[tf.float32, tf.float32])
return [f_anchor, f_pos, f_neg]
This works perfectly when I only perform a forward pass, with no train_op specified . However when I add this line train_op = optimizer.minimize(loss, global_step=global_step) the following error occurs:
Traceback (most recent call last):
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/gradients_impl.py", line 348, in _MaybeCompile
xla_compile = op.get_attr("_XlaCompile")
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py", line 2003, in get_attr
raise ValueError("No attr named '" + name + "' in " + str(self._node_def))
ValueError: No attr named '_XlaCompile' in name: "map/while/strided_slice"
op: "StridedSlice"
input: "map/while/boolean_mask/Gather"
input: "map/while/strided_slice/stack"
input: "map/while/strided_slice/stack_1"
input: "map/while/strided_slice/Cast"
attr {
key: "Index"
value {
type: DT_INT64
}
}
attr {
key: "T"
value {
type: DT_FLOAT
}
}
attr {
key: "begin_mask"
value {
i: 0
}
}
attr {
key: "ellipsis_mask"
value {
i: 0
}
}
attr {
key: "end_mask"
value {
i: 0
}
}
attr {
key: "new_axis_mask"
value {
i: 0
}
}
attr {
key: "shrink_axis_mask"
value {
i: 1
}
}
Does anyone has an idea what goes wrong?
A full example of this issue is here https://gist.github.com/anonymous/0b5e9194ebf09be7ad2f0a740bf369b8
Edit: It seems the problems is in these lines
hardest_pos = samples_pos[tf.argmax(distances)]
replacing it with something like
hardest_pos = tf.zeros(32)
gives no errors, however how to solve this?