I'm trying to run the converted model from the repository: https://github.com/HasnainRaz/Fast-SRGAN. Well, the conversion was successful. But when I tried to initialize the model, I saw the error: "Unknown layer: TensorFlowOpLayer.". If we will investigate the saved model, we can see TensorFlowOpLayer:
The model structure
As I understood it is this peace of code:
keras.layers.UpSampling2D(size=2, interpolation='bilinear')(layer_input).
I decided to write my own class "TensorFlowOpLayer".
import * as tf from '#tensorflow/tfjs';
export class TensorFlowOpLayer extends tf.layers.Layer {
constructor() {
super({});
}
computeOutputShape(shape: Array<number>) {
return [1, null, null, 32];
}
call(input_3): tf.Tensor {
const result = tf.layers.upSampling2d({ size: [2, 2], dataFormat: 'channelsLast', interpolation: 'bilinear' }).apply(input_3) as tf.Tensor;
return result;
}
static get className() {
return 'TensorFlowOpLayer';
}
}
But it doesn't work. Can someone help me to understand how to write to the method "computeOutputShape"?
And second misunderstanding, why on the picture above we see the next order of layers:
Conv2D -> TensorFlowOpLayer -> PReLU
As I understood the TensorFlowOpLayer layer is "UpSampling2D" in the python code. The H5 model was investigated through the site: https://netron.app
u = keras.layers.UpSampling2D(size=2, interpolation='bilinear')(layer_input)
u = keras.layers.Conv2D(self.gf, kernel_size=3, strides=1, padding='same')(u)
u = keras.layers.PReLU(shared_axes=[1, 2])(u)
The initializing of the model in TS:
async loadModel() {
this.model = await tf.loadLayersModel('/assets/fast_srgan/model.json');
const inputs = tf.layers.input({shape: [null, null, 32]});
const outputs = this.model.apply(inputs) as tf.SymbolicTensor;
this.model = tf.model({inputs: inputs, outputs: outputs});
console.log("Model has been loaded");
}
like in python code:
from tensorflow import keras
# Load the model
model = keras.models.load_model('models/generator.h5')
# Define arbitrary spatial dims, and 3 channels.
inputs = keras.Input((None, None, 3))
# Trace out the graph using the input:
outputs = model(inputs)
# Override the model:
model = keras.models.Model(inputs, outputs)
Then, how is it used:
tf.tidy(() => {
let img = tf.browser.fromPixels(this.imgLr.nativeElement, 3);
img = tf.div(img, 255.0);
img = tf.image.resizeNearestNeighbor(img, [96, 96]);
img = tf.expandDims(img, 0);
let sr = this.model.predict(img) as tf.Tensor;
});
like in python code:
def predict(img):
# Rescale to 0-1.
lr = tf.math.divide(img, 255)
# Get super resolution image
sr = model.predict(tf.expand_dims(lr, axis=0))
return sr[0]
When I added my own class "TensorFlowOpLayer" I see the next error:
"expected input1 to have shape [null,null,null,32] but got array with shape [1,96,96,3]."
Solved the issue. The issue related to the version of the code and the saved model. The author of the code refactored the code and didn't change the saved model. I rewrote the needed class:
import * as tf from '#tensorflow/tfjs';
export class DepthToSpace extends tf.layers.Layer {
constructor() {
super({});
}
computeOutputShape(shape: Array<number>) {
return [null, ...shape.slice(1, 3).map(x => x * 2), 32];
}
call(input): tf.Tensor {
input = input[0];
const result = tf.depthToSpace(input, 2);
return result;
}
static get className() {
return 'TensorFlowOpLayer';
}
}
and it works.
The author's original code is:
u = keras.layers.Conv2D(filters, kernel_size=3, strides=1, padding='same')(layer_input)
u = tf.nn.depth_to_space(u, 2)
u = keras.layers.PReLU(shared_axes=[1, 2])(u)
I use the following function now, but I don't think it works, but I can't understand the description of the cupy kernel definition. This function is very memory intensive and time-consuming when it comes to huge data.
def cupy_sum(self, bins):
bidx = cupy.cumsum(bins) -1,
return cupy.diff(cupy.r_[0, cupy.cumsum(self)[bidx]])
Refer to other examples and write the following code, do not know if there is a problem.
sum_section_kernel = cp.ElementwiseKernel(
'raw T bins, raw T dats',
'float32 out',
'''
T bin_f = bins[i ];
T bin_l = bins[i+1];
T biv = 0;
for(size_t j=bin_f; j<bin_l; j++){
biv += dats[j];
}
out = biv;
''',
'summe')
a = cp.array([4, 3, 5], dtype=cp.float32)
b = cp.array([1, 1, 1.1, 1, 2, 2, 2, 3, 3, 3, 3, 3], dtype=cp.float32)
y = cp.empty(3, dtype=cp.float32)
a = cp.r_[0,a.cumsum()]
out = sum_section_kernel(a, b, y)
print(out)
> [ 4.100 6.000 15.000]
The example has been put in the above, and the speed has not been improved, but I think there is still the advantage of saving memory.
I trained my model using Keras in Python and I converted my model to a tfjs model to use it in my webapp. I also wrote a small prediction script in python to validate my model on unseen data. In python it works perfectly, but when I'm trying to predict in my webapp it goes wrong.
This is the code I use in Python to create tensors and predict based on these created tensors:
input_dict = {name: tf.convert_to_tensor([value]) for name, value in sample_v.items()}
predictions = model.predict(input_dict)
classes = predictions.argmax(axis=-1)
In TFJS however it seems I can't pass a dict (or object) to the predict function, but if I write code to convert it to a tensor array (like I found on some places online), it still doesn't seem to work.
Object.keys(input).forEach((k) => {
input[k] = tensor1d([input[k]]);
});
console.log(Object.values(input));
const prediction = await model.executeAsync(Object.values(input));
console.log(prediction);
If I do the above, I get the following error: The shape of dict['key_1'] provided in model.execute(dict) must be [-1,1], but was [1]
If I then convert it to this code:
const input = { ...track.audioFeatures };
Object.keys(input).forEach((k) => {
input[k] = tensor2d([input[k]], [1, 1]);
});
console.log(Object.values(input));
I get the error that some dtypes have to be int32 but are float32. No problem, I can set the dtype manually:
const input = { ...track.audioFeatures };
Object.keys(input).forEach((k) => {
if (k === 'int_key') {
input[k] = tensor2d([input[k]], [1, 1], 'int32');
} else {
input[k] = tensor2d([input[k]], [1, 1]);
}
});
console.log(Object.values(input));
I still get the same error, but if I print it, I can see the datatype is set to int32.
I'm really confused as to why this is and why I can't just do like python and just put a dict (or object) in TFJS, and how to fix the issues I'm having.
Edit 1: Complete Prediction Snippet
const model = await loadModel();
const input = { ...track.audioFeatures };
Object.keys(input).forEach((k) => {
if (k === 'time_signature') {
input[k] = tensor2d([parseInt(input[k], 10)], [1, 1], 'int32');
} else {
input[k] = tensor2d([input[k]], [1, 1]);
}
});
console.log(Object.values(input));
const prediction = model.predict(Object.values(input));
console.log(prediction);
Edit 2: added full errormessage
I am trying to implement a structure search mechanism, find blocks and wrap them in a block.
I am new to machine learning, at first I started with the brain.js This library is quite simple and clear, I realized what was happening from the first time, the library is suitable for simple tasks.
But unfortunately, this library is not functional, earlier I asked how to find blocks: How to take the data?
I decided to try tensorflow, but for understanding this library is difficult, I still do not understand how it learns, because there is input and what the result should be.
Here is an example of how I tried to do a search for a brain.js
https://jsfiddle.net/eoy7krzj/
<html>
<head>
<script src="https://cdn.rawgit.com/BrainJS/brain.js/5797b875/browser.js"></script>
</head>
<body>
<div>
<button onclick="train()">train</button><button onclick="Generate.next(); Generate.draw();">generate</button><button onclick="calculate()">calculate</button>
</div>
<canvas id="generate" style="border: 1px solid #000"></canvas>
</body>
<script type="text/javascript">
var trainData = [];
function randomInteger(min, max) {
var rand = min - 0.5 + Math.random() * (max - min + 1)
//rand = Math.round(rand);
return rand;
}
function getRandomColor() {
var letters = '0123456789ABCDEF';
var color = '#';
for (var i = 0; i < 6; i++) {
color += letters[Math.floor(Math.random() * 16)];
}
return color;
}
var Generate = new function(){
var canvas = document.getElementById('generate');
var ctx = canvas.getContext('2d');
var elem = {
input: [],
output: []
}
var size = {
width: 240,
height: 140
}
canvas.width = 500;
canvas.height = 250;
this.next = function(){
this.build();
trainData.push({
input: elem.input,
output: elem.output
});
}
this.clear = function(){
ctx.clearRect(0, 0, canvas.width, canvas.height);
}
this.draw = function(){
this.clear();
this.item(elem.input, function(item){
ctx.strokeStyle = "green";
ctx.strokeRect(item[0], item[1], item[2], item[3]);
})
this.item(elem.output, function(item){
ctx.strokeStyle = "blue";
ctx.strokeRect(item[0], item[1], item[2], item[3]);
})
}
this.item = function(where, call){
for (var i = 0; i < where.length; i+=4) {
var input = [
where[i],
where[i+1],
where[i+2],
where[i+3],
];
this.denormalize(input);
call(input)
}
}
this.normalize = function(input){
input[0] = input[0] / 500;
input[1] = input[1] / 250;
input[2] = input[2] / 500;
input[3] = input[3] / 250;
}
this.denormalize = function(input){
input[0] = input[0] * 500;
input[1] = input[1] * 250;
input[2] = input[2] * 500;
input[3] = input[3] * 250;
}
this.empty = function(add){
var data = [];
for (var i = 0; i < add; i++) {
data = data.concat([0,0,0,0]);
}
return data;
}
this.build = function(){
var output = [];
var input = [];
size.width = randomInteger(100,500);
size.height = randomInteger(50,250);
var lines = 1;//Math.round(size.height / 100);
var line_size = 0;
var line_offset = 0;
for(var i = 0; i < lines; i++){
line_size = randomInteger(30,Math.round(size.height / lines));
var columns = Math.round(randomInteger(1,3));
var columns_width = 0;
var columns_offset = 0;
for(var c = 0; c < columns; c++){
columns_width = randomInteger(30,Math.round(size.width / columns));
var item = [
columns_offset + 10,
line_offset + 10,
columns_width - 20,
line_size - 20
];
this.normalize(item);
input = input.concat(item);
columns_offset += columns_width;
}
var box = [
0,
line_offset,
columns_offset,
line_size
]
this.normalize(box);
output = output.concat(box);
line_offset += line_size + 10;
}
elem.input = input.concat(this.empty(5 - Math.round(input.length / 4)));
elem.output = output.concat(this.empty(2 - Math.round(output.length / 4)));
}
this.get = function(){
return elem.input;
}
this.calculate = function(result, stat){
console.log('brain:',result);
this.item(result, function(item){
ctx.strokeStyle = "red";
ctx.strokeRect(item[0], item[1], item[2], item[3]);
})
}
this.train = function(){
for(var i = 0; i < 1400; i++){
this.next();
}
}
}
Generate.train();
Generate.log = true;
var net,stat;
function train(){
net = new brain.NeuralNetwork({ hiddenLayers: [4],activation: 'tanh'});
stat = net.train(trainData,{log: true, iterations: 1250,learningRate: 0.0001,errorThresh:0.0005});
console.log('stat:',stat)
}
function calculate(){
Generate.calculate(net.run(Generate.get()))
}
</script>
</html>
My goal is to train the network to find the elements and show their sizes.
Procedure: Click to train Click generate Click to calculate
The blue block wraps the green blocks, this should be the result, the red block shows that it has found a neural network.
That's what interests me:
Can tensorflow find blocks?
The data should be in the form of pictures, or numerical data?
How do you advise to start?
I would be very grateful if someone would put a small example on how to receive data, in what format and how to train)
Edit
I give the size and position of the green blocks, the goal is to find where the green blocks are and their total size, as an example this is shown by the blue block.
Neural Network
The neural network has a fix input that are the number of green blocks. Lets suppose we are going to find 3 blocks in a picture. The model will have an InputShape of [3, 4] for each block has 4 coordinates (x, y, w, h). The predicted box can be the min(x), min(y), max(x+w), max(y+h). This bounding box will wrap the boxes.
A sample data can be
features = [[[1, 2, 3, 4], [2, 4, 5, 6], [3, 4, 2, 2]]]
labels = [[1, 2, 7, 10]]
const random = _ => Math.floor(Math.random()*100)
const generate = _ => {
xarr = Array.from({length: 3}, _ => random())
yarr = Array.from({length: 3}, _ => random())
features = xarr.map((x, i) => ([x, yarr[i], x + random(), yarr[i] + random()]))
labels = features.reduce((acc, f) => ([Math.min(acc[0], f[0]), Math.min(acc[1], f[1]), Math.max(acc[0] + acc[2], f[0] + f[2]), Math.max(acc[0] + acc[3], f[1] + f[3])]) )
return {features, labels}
}
(async () => {
const model = tf.sequential();
model.add(tf.layers.dense({units: 20, inputShape: [3, 4], activation: 'relu'}));
model.add(tf.layers.reshape({targetShape: [60]}));
model.add(tf.layers.dense({units: 4, activation: 'relu'}));
model.summary();
// Prepare the model for training: Specify the loss and the optimizer.
model.compile({loss: 'meanSquaredError', optimizer: 'adam'});
// Generate some synthetic data for training.
let x = [];
let y = [];
for (let i = 0; i < 5; i++) {
const data = generate();
x.push(data.features);
y.push(data.labels);
}
const xs = tf.tensor3d(x);
const ys = tf.tensor2d(y);
console.log(xs.shape);
console.log(ys.shape);
// Train the model using the data then do inference on a data point the
// model hasn't seen:
xs.print()
ys.print()
await model.fit(xs, ys, {epochs: 100});
model.predict(tf.tensor([[[1, 2, 3, 4], [2, 4, 5, 6], [3, 4, 2, 2]]])).print();
})();
<html>
<head>
<!-- Load TensorFlow.js -->
<script src="https://cdn.jsdelivr.net/npm/#tensorflow/tfjs#latest"> </script>
</head>
<body>
</body>
</html>
Convolutionnal filters
The previous model will generate boxes that wraps up boxes whose coordinates are given to the model. But if we are to find out which position are the matching boxes, one can use a convolution filter.
Let's suppose we want to match the following data [[1, 2], [5, 6]] in a tensor.
This data can be a cropped picture that we want to see if it exists or not in a big picture and if yes, how many times it appears. Using a convolution filter of [[1, 1], [1, 1]], we will have a result of 14 at the top left coordinates (x, y) where there is a match. Filtering over this value (14) will return the index of the coordinates of interest.
(async() => {
// tf.setBackend('cpu')
const arr = Array.from({length: 16}, (_, k) => k+1)
const x = tf.tensor([...arr, ...arr.reverse()], [8, 4]); // input image 2d
x.print()
const filter = tf.ones([2, 2]) // input filter 2d
const conv = x.reshape([1, ...x.shape, 1]).conv2d(filter.reshape([...filter.shape, 1, 1]), 1, 'same').squeeze()
conv.print() // conv
const part = tf.tensor([[1, 2], [5, 6]]) // searched tensor
const mask = conv.equal(part.sum()).asType('bool');
const coords = await tf.whereAsync(mask);
coords.print(); // (0, 0) and (4, 0) are the top left coordinates of part of x that matches the part tensor
// how many elements matches
console.log(coords.shape[0])
// filter coords
const [a, b] = coords.lessEqual(x.shape.map((a, i) => a - part.shape[i] )).split(2, 1); // because of padding 'same'
const filterMask = a.mul(b)
const filterCoords = await tf.whereAsync(filterMask);
filterCoords.print()
const newCoords = coords.gather(filterCoords.split(2, 1)[0].reshape([2]))
newCoords.print()
const matchIndexes = await newCoords.unstack().reduce(async (a, c) => {
const cropped = x.slice(await c.data(), part.shape)
const sameElements = (await tf.whereAsync(cropped.equal(part).asType('bool')))
if(tf.util.sizeFromShape(part.shape) * 2 === (await sameElements.data()).length) {
a.push(await c.data())
}
return a
}, [])
console.log('matching index', matchIndexes) // only [0, 0]
})()
<html>
<head>
<!-- Load TensorFlow.js -->
<script src="https://cdn.jsdelivr.net/npm/#tensorflow/tfjs#latest"> </script>
</head>
<body>
</body>
</html>
To be more thorough, the convolutional filters is not enough to tell if there is a match. Actually a part of the tensor with the following values [[5, 6], [2, 1]] will also output 14. To make sure of outputting only the correct index, one can slice the input tensor at the given coordinates and check values bitwise if possible when the tensor processed are not big or just randomly some few elements.
How do I mutate value of a tensor in Tensorflow.js? For example if I have a tensor created like this:
const a = tf.tensor1d([1,2,3,4])
How do I change the value of the third element of the tensor? I know that tensors are immutable and variables are mutable.
Doing this: const a = tf.variable(tf.tensor1d([1,2,3,4])) doesn't seem to solve the problem. I cannot do:
const a = a[0].assign(5)
I am able to do this in python tensorflow like this:
a = tf.Variable([1,2,3,4])
a = a[0].assign(100)
with tf.Session() as sess:
sess.run(tf.global_variables_iniliazer())
print sess.run(a)
This outputs [100, 2,3,4]
Does tf.buffer work for you?
// Create a buffer and set values at particular indices.
const a = tf.tensor1d([1, 2, 3, 4]);
const buffer = tf.buffer(a.shape, a.dtype, a.dataSync());
buffer.set(5, 0);
const b = buffer.toTensor();
// Convert the buffer back to a tensor.
b.print();
Tensor
[5, 2, 3, 4]
I had to do this using mulStrict and addStrict which do element-wise multiplication and addition.
const a = tf.tensor1d([1,2,3,4]);
tf.mulStrict(a, tf.tensor1d([0,1,1,1]))
.addStrict(tf.tensor1d([100, 0, 0, 0]);
This was based on asnwer
here