Is it possible to have different loss-functions defined on different "segments" of a deep network (in the following sense):
Suppose that I have some input-output pairs (x_n,y_n) and (x_n,z_n) and I want to train a deep network of the from f_k∘...∘f_1 (each f_i is a feed-forward layer) such that
MSE(f_k∘...∘f_1(x_n) - y_n) is minimized
MSE(f_k∘f_k-1(x_n) - z_n) is also minimized.
Analogy/ Example/ Intuition/ Motivation:
Suppose I want the output of my network to approximately be like x^2 then f_k∘f_k-1(x_n)~~ x_n^2 and z_n:= x_n^2. Then f_k∘...∘f_1(x_n) a feed-forward network whose output layer is approximately the function x^2.
How can you do this in TensorFlow/Keras?
You can achieve that by defining a two output model.
It essentially minimizes the weighted average of the two losses.
A restriction is that the input shape of the first and (n-1)-th layers must be same, since both receives the same input x.
from tensorflow.keras.models import Sequential, Model
from tensorflow.keras.layers import Dense, Input
input_shape = (1,)
model1 = Sequential([
Dense(5, activation="relu", input_shape=input_shape),
Dense(8, activation="relu"),
Dense(input_shape[0])
])
model2 = Sequential([
Dense(15, activation="relu", input_shape=input_shape),
Dense(1)
])
x = Input(input_shape)
y = model2(model1(x))
z = model2(x)
model = Model(inputs=x, outputs=[y, z])
model.compile("adam", "mean_squared_error", loss_weight=[0.5, 0.5])
import numpy as np
n = 1000
x = np.random.normal(size=n * input_shape[0]).reshape((n, input_shape[0]))
y = x**2
z = x**2
history = model.fit(x, [y, z], epochs=100)
Visualization:
import matplotlib.pyplot as plt
plt.plot(history.history["loss"])
yhat, zhat = model.predict(x)
plt.scatter(x, yhat)
plt.scatter(x, zhat)
Related
This is a regression problem, where I want to generate 5 float values from each image of size 224 x 224. So I use fully connected networks with 5 nodes in the last layer. But doing so in keras gives me the following error:
import keras, os
import numpy as np
from tensorflow.keras.models import Model
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.layers import Dense, GlobalAveragePooling2D
from tensorflow.keras.applications.inception_v3 import InceptionV3
## data_list = list of four 224x224 numpy arrays
inception = InceptionV3(weights='imagenet', include_top=False)
x = inception.output
x = GlobalAveragePooling2D()(x)
x = Dense(1024, activation='relu')(x)
predictions = Dense(5, activation='relu')(x)
y = [np.random.random(5),np.random.random(5),np.random.random(5),np.random.random(5)]
model = Model(inputs=inception.input, outputs=predictions)
opt = Adam(lr=0.001)
model.compile(optimizer=opt, loss="mae")
model.fit(data_list, y, verbose=0, epochs=100)
Error:
ValueError: Data cardinality is ambiguous:
x sizes: 224, 224, 224, 224
y sizes: 5, 5, 5, 5
Make sure all arrays contain the same number of samples.
What could be going wrong?
Convert data_list and y to numpy arrays or tensors.
In your code the list is treated as four inputs while your model has one input - https://keras.io/api/models/model_training_apis/
Add these lines:
import tensorflow as tf
data_list = tf.stack(data_list)
y = tf.stack(y)
Try this
model.fit(np.array(data_list), np.array(y), verbose=0, epochs=100)
I am using TF2 (2.3.0) NN to approximate the function y which solves the ODE: y'+3y=0
I have defined cutsom loss class and function in which I am trying to differentiate the single output with respect to the single input so the equation holds, provided that y_true is zero:
from tensorflow.keras.losses import Loss
import tensorflow as tf
class CustomLossOde(Loss):
def __init__(self, x, model, name='ode_loss'):
super().__init__(name=name)
self.x = x
self.model = model
def call(self, y_true, y_pred):
with tf.GradientTape() as tape:
tape.watch(self.x)
y_p = self.model(self.x)
dy_dx = tape.gradient(y_p, self.x)
loss = tf.math.reduce_mean(tf.square(dy_dx + 3 * y_pred - y_true))
return loss
but running the following NN:
import tensorflow as tf
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.models import Model
from tensorflow.keras.layers import Dense
from tensorflow.keras import Input
from custom_loss_ode import CustomLossOde
num_samples = 1024
x_train = 4 * (tf.random.uniform((num_samples, )) - 0.5)
y_train = tf.zeros((num_samples, ))
inputs = Input(shape=(1,))
x = Dense(16, 'tanh')(inputs)
x = Dense(8, 'tanh')(x)
x = Dense(4)(x)
y = Dense(1)(x)
model = Model(inputs=inputs, outputs=y)
loss = CustomLossOde(model.input, model)
model.compile(optimizer=Adam(learning_rate=0.01, beta_1=0.9, beta_2=0.99),loss=loss)
model.run_eagerly = True
model.fit(x_train, y_train, batch_size=16, epochs=30)
for now I am getting 0 loss from the fisrt epoch, which doesn't make any sense.
I have printed both y_true and y_test from within the function and they seem OK so I suspect that the problem is in the gradien which I didn't succeed to print.
Apprecitate any help
Defining a custom loss with the high level Keras API is a bit difficult in that case. I would instead write the training loop from scracth, as it allows a finer grained control over what you can do.
I took inspiration from those two guides :
Advanced Automatic Differentiation
Writing a training loop from scratch
Basically, I used the fact that multiple tape can interact seamlessly. I use one to compute the loss function, the other to calculate the gradients to be propagated by the optimizer.
import tensorflow as tf
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.models import Model
from tensorflow.keras.layers import Dense
from tensorflow.keras import Input
num_samples = 1024
x_train = 4 * (tf.random.uniform((num_samples, )) - 0.5)
y_train = tf.zeros((num_samples, ))
inputs = Input(shape=(1,))
x = Dense(16, 'tanh')(inputs)
x = Dense(8, 'tanh')(x)
x = Dense(4)(x)
y = Dense(1)(x)
model = Model(inputs=inputs, outputs=y)
# using the high level tf.data API for data handling
x_train = tf.reshape(x_train,(-1,1))
dataset = tf.data.Dataset.from_tensor_slices((x_train,y_train)).batch(1)
opt = Adam(learning_rate=0.01, beta_1=0.9, beta_2=0.99)
for step, (x,y_true) in enumerate(dataset):
# we need to convert x to a variable if we want the tape to be
# able to compute the gradient according to x
x_variable = tf.Variable(x)
with tf.GradientTape() as model_tape:
with tf.GradientTape() as loss_tape:
loss_tape.watch(x_variable)
y_pred = model(x_variable)
dy_dx = loss_tape.gradient(y_pred, x_variable)
loss = tf.math.reduce_mean(tf.square(dy_dx + 3 * y_pred - y_true))
grad = model_tape.gradient(loss, model.trainable_variables)
opt.apply_gradients(zip(grad, model.trainable_variables))
if step%20==0:
print(f"Step {step}: loss={loss.numpy()}")
I'm a newbie at ML and I'm struggling with a model.
In order to try to understand what was wrong with a bigger model, I wanted to create a simple one which goal is just to provide the double value of the input, but I couldn't succeed even in this simples problem, since the model compile but doesn't learn.
Can someone please help me? I'm just frustrated since I have no clue why this simples model cannot learn.
import numpy as np
from tensorflow.keras import Sequential, Model
from tensorflow.keras.layers import Input, Dense
x = [i for i in range(1, 21)]
y = [2 * i for i in range(1, 21)]
x = np.array(x)
y = np.array(y)
model = Sequential()
model.add(Input(shape=1))
model.add(Dense(units=1, activation='relu'))
model.compile(optimizer='adam', loss='mse', metrics=['accuracy'])
model.fit(x, y, shuffle=True, epochs=10, validation_data=(x, y))```
This is because you used a ReLU activation function, which results a derivative of 0 for the parameter. A linear activation function will solve the problem. it fits well:)
You could also try altering the initializer for the parameter somehow.
import numpy as np
from tensorflow.keras import Sequential, Model
from tensorflow.keras.layers import Input, Dense
x = [i for i in range(1, 21)]
y = [2 * i for i in range(1, 21)]
x = np.array(x)
y = np.array(y)
model = Sequential()
model.add(Input(shape=1))
model.add(Dense(units=1, activation=None))
model.compile(optimizer='adam', loss='mse', metrics=['accuracy'])
model.fit(x, y, shuffle=True, epochs=100, validation_data=(x, y))
bce = tf.keras.losses.BinaryCrossentropy()
ll=bce(y_test[0], model.predict(X_test[0].reshape(1,-1)))
print(ll)
<tf.Tensor: shape=(), dtype=float32, numpy=0.04165391>
print(model.input)
<tf.Tensor 'dense_1_input:0' shape=(None, 195) dtype=float32>
model.output
<tf.Tensor 'dense_3/Sigmoid:0' shape=(None, 1) dtype=float32>
grads=K.gradients(ll, model.input)[0]
print(grads)
None
So here i have Trained a 2 hidden layer neural network, input has 195 features and output is 1 size. I wanted to feed the neural network with validation instances named as X_test one by one with their correct labels in y_test and for each instance calculate the gradients of the output with respect to input, the grads upon printing gives me a None. Your help is appreciated.
One can do this using tf.GradientTape. I wrote the following code to learn a sin wave, and get its derivative in the spirit of this question. I think, it should be possible to extend the following codes in order to compute partial derivatives.
Importing the needed libraries:
import numpy as np
from tensorflow.keras.layers import Dense, Input
from tensorflow.keras.models import Model, Sequential
from tensorflow.keras.optimizers import Adam
from tensorflow.keras import losses
import tensorflow as tf
Create the data:
x = np.linspace(0, 6*np.pi, 2000)
y = np.sin(x)
Defining a Keras NN:
def model_gen(Input_shape):
X_input = Input(shape=Input_shape)
X = Dense(units=64, activation='sigmoid')(X_input)
X = Dense(units=64, activation='sigmoid')(X)
X = Dense(units=1)(X)
model = Model(inputs=X_input, outputs=X)
return model
Training the model:
model = model_gen(Input_shape=(1,))
opt = Adam(lr=0.01, beta_1=0.9, beta_2=0.999, decay=0.001)
model.compile(loss=losses.mean_squared_error, optimizer=opt)
model.fit(x,y, epochs=200)
To obtain the gradient of the network w.r.t. the input:
x = list(x)
x = tf.constant(x)
with tf.GradientTape() as t:
t.watch(x)
y = model(x)
dy_dx = t.gradient(y, x)
dy_dx.numpy()
One can further visualise dy_dx to make sure of how smooth the derivative is. Finally, note that one get a smoother derivative when one uses a smooth activation (e.g. sigmoid) instead of Relu as noted here.
I have built and trained a Sequential model.
Now before each model.predict call I want to upload the data into GPU, do some operations and then call model.predict using the output stored in GPU without downloading to memory and handover to keras model for it to upload to gpu again.
Edit:
I would like to use opencv operations on the input image in gpu and use the output directly to call model.predict if possible.
You can easily achieve this by adding the operations as Lambda layers on the top of the model.
Here is a very simple example.. you can extend from here:
import numpy as np
from keras import backend as K
from keras.models import Sequential, Model
from keras.layers import Dense, Lambda, Input, merge
X = np.random.random((1000,5))
Y = np.random.random((1000,1))
inp = Input(shape = (5,))
d1 = Dense(60, input_dim=5, init='normal', activation='relu')
d2 = Dense(1, init='normal', activation='sigmoid')
out = d2(d1(inp))
model = Model(input=[inp], output=[out])
model.compile(loss='binary_crossentropy', optimizer='adam')
model.summary()
model.fit(X, Y, nb_epoch=1)
X1 = np.random.random((10,3))
X2 = np.random.random((10,2))
inp1 = Input(shape = (3,))
inp2 = Input(shape = (2,))
p1 = Lambda(lambda x: K.sqrt(x))(inp1)
p2 = Lambda(lambda x: K.tf.exp(x))(inp2)
mer = merge([p1, p2], mode='concat')
out2 = d2(d1(mer))
model2 = Model(input=[inp1, inp2], output=[out2])
model2.summary()
ypred = model2.predict([X1, X2])
print ypred.shape
Here from model.summary() you can see both the models are sharing the upper layers so in essence is using the weights already learnt during the training of the first model