This is related to an implementation issue in TensorFlow. I have a tensor H of size (batch_size, time_steps, 256) (but batch_size and time_steps are None during build time).
I want to compute a tensor A of the shape (batch_size, time_steps, time_steps, n_dim) which is a Cartesian product across the time dimension. Thus, in simple terms:
A[:,i,j,:] = some_func(H[:,i,:], H[:,j,:])
Since the number of time_steps is None at time of building graph, what would be an elegant way to compute A?
You can use TensorFlow's broadcasting properties to construct cartesian products between tensors by expanding one tensor with an additional dimension and multiplying it with the other tensor.
import tensorflow as tf
A = tf.placeholder(shape=(None, None, 256), dtype=tf.float32)
B = tf.placeholder(shape=(None, None, 256), dtype=tf.float32)
A_ = tf.expand_dims(A, axis=1)
A_*B
Gives
<tf.Tensor 'mul_1:0' shape=(?, ?, ?, 256) dtype=float32>
I have a solution for the case when the size of time_steps axis is specified (i.e. is not None). We can easily use K.repeat_elements and K.tile to form the tensors for cartesian product:
from keras import layers, models
from keras import backend as K
def some_func(a, b):
# define the some_func here
return a + b
def cart_prod(x):
shp = K.int_shape(x)[1]
x_rep = K.repeat_elements(x, shp, axis=1)
x_tile = K.tile(x, [1, shp, 1])
res = some_func(x_rep, x_tile)
return K.reshape(res, [-1, shp, shp, K.shape(res)[-1]])
inp = layers.Input((3, 2))
out = layers.Lambda(cart_prod)(inp)
model = models.Model(inp, out)
model.predict(np.arange(6).reshape(1, 3, 2))
Output:
array([[[[ 0., 2.],
[ 2., 4.],
[ 4., 6.]],
[[ 2., 4.],
[ 4., 6.],
[ 6., 8.]],
[[ 4., 6.],
[ 6., 8.],
[ 8., 10.]]]], dtype=float32)
Related
I'm using sigmoid and binary_crossentropy for multi-label classification. A very similar question asked here. And the following custom metric was suggested:
from keras import backend as K
def full_multi_label_metric(y_true, y_pred):
comp = K.equal(y_true, K.round(y_pred))
return K.cast(K.all(comp, axis=-1), K.floatx())
But I do not want to use all() because for one single sample with a true label of [1, 0, 0, 1, 1] and a predicted label of [0, 0, 0, 1, 1] I do not consider the prediction accuracy as zero (due to the the fact that the labels for the last four classes have been predicted correctly).
Here is my model:
# expected input data shape: (batch_size, timesteps, data_dim)
model = Sequential()
model.add(Masking(mask_value=-9999, input_shape=(197, 203)))
model.add(LSTM(512, return_sequences=True))
model.add(Dense(20, activation='sigmoid'))
model.compile(loss='binary_crossentropy',
optimizer=SGD(lr=1e-3, decay=1e-4, momentum=0.9, nesterov=True),
metrics = ['accuracy'])
print(model.summary())
Here is my y_pred for one example:
pred = model.predict(X_test)
y_pred = pred[0,196,:]
y_pred
array([2.6081860e-01, 9.9079555e-01, 1.4816311e-01, 8.6009043e-01,
2.6759505e-04, 3.0792636e-01, 2.6738405e-02, 8.5339689e-01,
5.1105350e-02, 1.5427300e-01, 6.7039116e-05, 1.7909735e-02,
6.4140558e-04, 3.5133284e-01, 5.3054303e-02, 1.2765944e-01,
2.9298663e-04, 6.3041472e-01, 5.8620870e-03, 5.9656668e-01],
dtype=float32)
Here is my y_true for one example:
y_true = Y_test[0,0,:]
y_true
array([1., 0., 0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 0., 1., 1., 0., 0.,
0., 0., 1.])
My question is: How can I set a Keras custom metric function so that each element in y_pred should be compared to the each element in y_true, then an accuracy measure will be given during training? I want to use this metric in metrics = [X])?
Unless I'm mistaken the default binary_crossentropy metric/loss already does what you need. Taking your example
import tensorflow as tf
from tensorflow import keras
y_true = tf.constant([[1, 0, 0, 1, 1]], dtype=tf.int32)
y_pred = tf.constant([[0.6, 0, 0, 1, 1]], dtype=tf.float32)
m = keras.metrics.binary_crossentropy(y_true, y_pred)
m.numpy()
The output is [-log(0.6) / 5].
i.e. The metric/loss for the batch takes into account the losses for each of the 20 outputs of the model. Which I'm assuming represent time steps.
As a metric, it is much more common to use binary_accuracy.
Example:
y_true = tf.constant([[1, 0, 0, 1, 1]], dtype=tf.int32)
y_pred = tf.constant([[0.1, 0, 0, 1, 1]], dtype=tf.float32)
keras.metrics.binary_accuracy(tf.cast(y_true, tf.float32), y_pred)
One can get a better idea of the performance of the model via an ROC metric (https://www.tensorflow.org/api_docs/python/tf/keras/metrics/AUC) which measures the curve at various thresholds see an explanation at https://towardsdatascience.com/understanding-auc-roc-curve-68b2303cc9c5. Personally, I tend to use an accuracy metric while training and look at the precision/recall curve after the model is trained in order to check that it behaves as expected and select the prediction threshold.
I want to shift a tensor in a given axis. It's easy to do this in pandas or numpy. Like this:
import numpy as np
import pandas as pd
data = np.arange(0, 6).reshape(-1, 2)
pd.DataFrame(data).shift(1).fillna(0).values
Output is:
array([[0., 0.],
[0., 1.],
[2., 3.]])
But in tensorflow, the closest solution I found is tf.roll. But it shift the last row to the first row. (I don't want that). So I have to use something like
tf.roll + tf.slice(remove the last row) + tf.concat(add tf.zeros to the first row).
It's really ugly.
Is there a better way to handle shift in tensorflow or keras?
Thanks.
I think I find a better way for this problem.
We could use tf.roll, then apply tf.math.multiply to set the first row to zeros.
Sample code is as follows:
Original tensor:
A = tf.cast(tf.reshape(tf.range(27), (-1, 3, 3)), dtype=tf.float32)
A
Output:
<tf.Tensor: id=117, shape=(3, 3, 3), dtype=float32, numpy=
array([[[ 0., 1., 2.],
[ 3., 4., 5.],
[ 6., 7., 8.]],
[[ 9., 10., 11.],
[12., 13., 14.],
[15., 16., 17.]],
[[18., 19., 20.],
[21., 22., 23.],
[24., 25., 26.]]], dtype=float32)>
Shift (like pd.shift):
B = tf.concat((tf.zeros((1, 3)), tf.ones((2, 3))), axis=0)
C = tf.expand_dims(B, axis=0)
tf.math.multiply(tf.roll(A, 1, axis=1), C)
Output:
<tf.Tensor: id=128, shape=(3, 3, 3), dtype=float32, numpy=
array([[[ 0., 0., 0.],
[ 0., 1., 2.],
[ 3., 4., 5.]],
[[ 0., 0., 0.],
[ 9., 10., 11.],
[12., 13., 14.]],
[[ 0., 0., 0.],
[18., 19., 20.],
[21., 22., 23.]]], dtype=float32)>
Try this:
import tensorflow as tf
input = tf.constant([[0, 1, 3], [4, 5, 6], [7, 8, 9]])
shifted_0dim = input[1:]
shifted_1dim = input[:, 1:]
shifted2 = input[2:]
Generalizing the accepted answer to arbitrary tensor shapes, desired shift, and axis to shift:
import tensorflow as tf
def tf_shift(tensor, shift=1, axis=0):
dim = len(tensor.shape)
if axis > dim:
raise ValueError(
f'Value of axis ({axis}) must be <= number of tensor axes ({dim})'
)
mask_dim = dim - axis
mask_shape = tensor.shape[-mask_dim:]
zero_dim = min(shift, mask_shape[0])
mask = tf.concat(
[tf.zeros(tf.TensorShape(zero_dim) + mask_shape[1:]),
tf.ones(tf.TensorShape(mask_shape[0] - zero_dim) + mask_shape[1:])],
axis=0
)
for i in range(dim - mask_dim):
mask = tf.expand_dims(mask, axis=0)
return tf.multiply(
tf.roll(tensor, shift, axis),
mask
)
EDIT:
This code above doesn't allow for negative shift values, and is pretty slow. Here is a more efficient version utilizing tf.roll and tf.concat without creating a mask and multiplying the tensor of interest by it.
import tensorflow as tf
def tf_shift(values: tf.Tensor, shift: int = 1, axis: int = 0):
pad = tf.zeros([val if i != axis else abs(shift) for i, val in enumerate(values.shape)],
dtype=values.dtype)
size = [-1 if i != axis else val - abs(shift) for i, val in enumerate(values.shape)]
if shift > 0:
shifted = tf.concat(
[pad, tf.slice(values, [0] * len(values.shape), size)],
axis=axis
)
elif shift < 0:
shifted = tf.concat(
[tf.slice(values, [0 if i != axis else abs(shift) for i, _ in enumerate(values.shape)], size), pad],
axis=axis
)
else:
shifted = values
return shifted
Assuming a 2d tensor, this function should mimic a Dataframe shift:
def shift_tensor(tensor, periods, fill_value):
num_row = len(tensor)
num_col = len(tensor[0])
pad = tf.fill([periods, num_col], fill_value)
if periods > 0:
shifted_tensor = tf.concat((pad, tensor[:(num_row - periods), :]), axis=0)
else:
shifted_tensor = tf.concat((tensor[:(num_row - periods), :], pad), axis=0)
return shifted_tensor
I'm trying to implement the model described by Professor Andrew Ng for object detection (explanation starts at 10:00).
He describes the first element of the output vector as the probability that an object was detected, followed by the coordinates of the bounding box of the object matched (when one is matched). The last part of the output vector is a softmax of all the classes your model knows.
As he explains it, using a simple squared error for the case when there is a detection is fine, and just the squares difference of y^[0] - y[0]. I get that this is a naive approach. I'm just wanting to implement this for the learning experience.
My questions
How do I implement this conditional loss in tensorflow?
How do I handle this conditional about y^[0] when dealing with a batch.
How do I implement this conditional loss in tensorflow?
You can convert the loss function to:
Error = mask[0]*(y^[0]-y[0])**2 + mask[1]*(y^[1]-y[1])**2 ... mask[n]*(y^[n]-y[n])**2),
where mask = [1, 1,...1] for y[0] = 1 and [1, 0, ...0] for y[0] = 0
How do I handle this conditional about y^[0] when dealing with a
batch.
For a batch, you can construct the mask on the fly like:
mask = tf.concat([tf.ones((tf.shape(y)[0],1)),y[:,0][...,None]*y[:,1:]], axis=1)
Code:
y_hat_n = np.array([[3, 3, 3, 3], [3,3,3,3]])
y_1 = np.array([[1, 1, 1, 1], [1,1,1,1]])
y_0 = np.array([[0, 1, 1, 1], [0,1,1,1]])
y = tf.placeholder(tf.float32,[None, 4])
y_hat = tf.placeholder(tf.float32,[None, 4])
mask = tf.concat([tf.ones((tf.shape(y)[0],1)),y[:,0][...,None]*y[:,1:]], axis=1)
error = tf.losses.mean_squared_error(mask*y, mask*y_hat)
with tf.Session() as sess:
print(sess.run([mask,error], {y:y_0, y_hat:y_hat_n}))
print(sess.run([mask,error], {y:y_1, y_hat:y_hat_n}))
# Mask and error
#[array([[1., 0., 0., 0.],
# [1., 0., 0., 0.]], dtype=float32), 2.25]
#[array([[1., 1., 1., 1.],
# [1., 1., 1., 1.]], dtype=float32), 4.0]
input: length(placeholder), index(1D tensor)
output: 0-1 1D tensor
example: length 5, index [0,1,3], output tensor should be [1,1,0,1,0]
I have tried scatter_add, which requires Variable which requires known shape, and embedding_lookup from matrix with [length, length], which is not effective when length is large.
Any ideas?
Try tf.sparse_to_dense:
output_size = tf.placeholder(tf.int32, [1])
index = tf.constant([0, 1, 3])
ones = tf.ones([tf.size(index)])
result = tf.sparse_to_dense(index, output_size, ones)
with tf.Session() as sess:
sess.run(result, feed_dict={output_size: [5]})
Outputs: array([ 1., 1., 0., 1., 0.], dtype=float32)
I have two vectors A & B having dimensions (1, 100) & (784, 100) respectively. I thought A would be broadcast along the raw to the same dimension as B, but got error that "Dimensions must be equal". Can you please explain why?
Broadcasting of matrices with the same rank (i.e. 2) seems to work as it says on the tin:
import tensorflow as tf
tf.__version__
# 1.3.0
A = tf.constant([[1, 2], [3, 4], [5, 6]], dtype=tf.float32)
B = tf.constant([[1, -1]], dtype=tf.float32)
sess = tf.Session()
sess.run(A * B)
# array([[ 1., -2.],
# [ 3., -4.],
# [ 5., -6.]], dtype=float32)
sess.run(tf.multiply(A, B))
# array([[ 1., -2.],
# [ 3., -4.],
# [ 5., -6.]], dtype=float32)