Clipping(Filtering) tf.placeholder values - tensorflow

I want to change my tf.placeholder values such that:
values < SmallConstant is set to 0.
It's not exactly clipping, so I can't use: tf.clip_by_value()
I tried the suggestion in Conditional assignment of tensor values in TensorFlow, and this is what I have so far:
x = tf.placeholder(tf.float32, None)
condition = tf.less(x, tf.constant(SmallConst))
tf.assign(x, tf.where(condition, tf.zeros_like(x), x))
However, on running this, I get an error saying
AttributeError: 'Tensor' object has no attribute 'assign'
It seems tf.assign() can be done on tf.Variable but not on tf.placeholder.
Is there any other way I can do this?
Thank you!

Yes, it's even easier than you think:
x = tf.placeholder(tf.float32, None)
# create a bool tensor the same shape as x
condition = x < SmallConst
# create tensor same shape as x, with values greater than SmallConst set to 0
to_remove = x*tf.to_float(condition)
# set all values of x less than SmallConst to 0
x_clipped = x - to_remove
I'd normally just put that into one line like:
x_clipped = x - x*tf.to_float(x < small_const)
note: using tf.to_float on a tensor of type bool will give you 0.0s in place of Falses and 1.0s in place of Trues
Additional information for cleaner code:
The numerical operators (e.g. <, >=, +, - etc, but not ==) are overloaded for tensorflow tensors such that you can use native python variables with tensors to get a new tensor that is the result of that operation. So tf.constant() is actually fairly rarely actually needed. Example of this in action:
a = tf.placeholder(tf.int32)
b = a + 1
c = a > 0
print(b) # gives "<tf.Tensor 'add:0' shape=<unknown> dtype=int32>"
print(c) # gives "<tf.Tensor 'Greater:0' shape=<unknown> dtype=bool>"
sess.run(b, {a: 1}) # gives scalar int32 numpy array with value 2
sess.run(c, {a: 1}) # gives scalar bool numpy array with value True
This is also true of numpy.
tf.assign() only works on Variables because it will
Update 'ref' by assigning 'value' to it.
Tensors in tensorflow are immutable. The result of any operation on a tensor is another tensor, but the original tensor will never change. Variables, however are mutable, and you change their value with tf.assign()

Related

How to create Keras ZeroTensor of specific shape

I am a total beginner with tensorflow.keras and I am wondering how I could create a constant zero tensor of a specific shape.
For example with this:
zeros = tf.keras.backend.zeros((someTensor.shape[0], someTensor.shape[1], someTensor.shape[2], channels))
concat = tf.kerasbackend.concatenate([someTensor, zeros], axis=3)
The operation tf.keras.backend.zeros fails with:
ValueError: Cannot convert a partially known TensorShape to a Tensor
I guess thats because the batch size is unknown during graph building. How can I create a ZeroTensor or any other constant tensor when I don't know the batchsize at that moment? Or is there some kind of unknown(?) value that I can specify?
It's strange because you are using a tuple of tensors and integers. Sort of weird.
You should:
shape = K.shape(someTensor)
ch = K.variable([channels]) #I think K.constant also works.
newShape = K.concatenate([shape[:3], ch])
zeros = K.zeros(newShape)
Now, if this doesn't work because of unknown shapes, a dirty workaround would be:
#if someTensor is 3D
zeros = K.zeros_like(someTensor)
zeros = K.stack([zeros] * channels, axis=-1)
#if someTensor is 4D
zeros = K.zeros_like(someTensor[:,:,:,0])
zeros = K.stack([zeros]*channels, axis=-1)

gather values from 2dim tensor in tensorflow

Hi tensorflow beginner here... I'm trying to get the value of a certain elements in an 2 dim tensor, in my case class scores from a probability matrix.
The probability matrix is (1000,81) with batchsize 1000 and number of classes 81. ClassIDs is (1000,) and contains the index for the highest class score for each sample. How do I get the corresponding class score from the probability matrix using tf.gather?
class_ids = tf.cast(tf.argmax(probs, axis=1), tf.int32)
class_scores = tf.gather_nd(probs,class_ids)
class_scores should be a tensor of shape (1000,) containing the highest class_score for each sample.
Right now I'm using a workaround that looks like this:
class_score_count = []
for i in range(probs.shape[0]):
prob = probs[i,:]
class_score = prob[class_ids[i]]
class_score_count.append(class_score)
class_scores = tf.stack(class_score_count, axis=0)
Thanks for the help!
You can do it with tf.gather_nd like this:
class_ids = tf.cast(tf.argmax(probs, axis=1), tf.int32)
# If shape is not dynamic you can use probs.shape[0].value instead of tf.shape(probs)[0]
row_ids = tf.range(tf.shape(probs)[0], dtype=tf.int32)
idx = tf.stack([row_ids, class_ids], axis=1)
class_scores = tf.gather_nd(probs, idx)
You could also just use tf.reduce_max, even though it would actually compute the maximum again it may not be much slower if your data is not too big:
class_scores = tf.reduce_max(probs, axis=1)
you need to run the tensor class_ids to get the values
the values will be a bumpy array
you can access numpy array normally by a loop
you have to do something like this :
predictions = sess.run(tf.argmax(probs, 1), feed_dict={x: X_data})
predictions variable has all the information you need
tensorflow only returns those tensor values which you run explicitly
I think this is what the batch_dims argument for tf.gather is for.

How can I compare if column equals in a matrix multiplication mannar?

I am using Keras (tensorflow as backend). What I want to do is to write a lambda layer that gets 2 tensor input and compare every combination of 2 column of them using Indicator function and produce a new tensor with 0-1 value. Here is an example.
Input: x = K.variable(np.array([[1,2,3],[2,3,4]])),
y = K.variable(np.array([[1,2,3],[2,3,4]]))
Output
z=K.variable(np.array[[1,0],[0,1]])
As far as I know, tensorflow provides tf.equal() to compare tensor in a elementwise way. But if I apply it here, I get
>>> z=tf.equal(x,y)
>>> K.eval(z)
array([[True, True, True],
[True, True, True]], dtype=bool)
It only compares tensor in same position.
So my questions are:
1. Is there a tensorflow API to get my desired output or if I need to write my own function to complete it?
2. If it is the latter one, then there is another problem. I noticed that in keras the input is mini-batch, so the input format looks like: (None, m, n). When writing my own method, how can I tackle with the first dimension, which is None?
Any reply would be appreciated!
You could use broadcasting.
import numpy as np
import tensorflow as tf
x = tf.constant(np.array([[1,2,3],[2,3,4]]))
y = tf.constant(np.array([[1,2,3],[2,3,4]]))
x_ = tf.expand_dims(x, 0)
y_ = tf.expand_dims(y, 1)
res = tf.reduce_all(tf.equal(x_, y_), axis=-1)
sess = tf.Session()
sess.run(res)

(Tensorflow)Does the op assign change the gradient computation?

I use the op "assign" to change the value of variables instead of "=", but I found the gradient I got is quite different. Could anyone tell me the difference and why? thanks!
Like change w = w1 to op1 = tf.assign(w, w1) sess.run(op1)
= and tf.assign are different operations.
= is a python operation, in which you assign a python value to a python variable
tf.assign is a Tensorflow operation that assigns the value to the variable ref and returns the assign operation.
= is executed in python and doesn't affect the computation graph.
tf.assign is a node in the computational graph.
To understand, let's run this simple script
import tensorflow as tf
x = tf.Variable(1)
y = tf.Variable(2)
x = y
print(x.name, y.name)
a = tf.Variable(1)
b = tf.Variable(2)
# override a, otherwise a content is 1
a = a.assign(b)
print(a.name, b.name)
init = tf.global_variables_initializer()
with tf.Session() as sess:
sess.run(init)
print(sess.run([x, y, a, b]))
print(x.name, y.name) outputs Variable_1:0 Variable_1:0
because = is executed in python and you've overwritten the variable x.
print(a.name, b.name) outputs Assign:0 Variable_3:0 because you defined an assign op in the computational graph, now a is an assign op.
When you run the defined graph, you get:
[2, 2, 2, 2]
But these values are computed differently: one is a computation in the graph, the others no.
If you forgot to assign a to the assign op created with tf.assign (thus you change the line a = a.assign(b) to a.assign(b)), then when you evaluate the graph, you'll get:
[2, 2, 1, 2]

Setting all negative values of a tensor to zero (in tensorflow)

Here's my problem. I have a tensor X and I want to set all negative values to zero. In numpy, I would do the following np.maximum(0, X). Is there any way to achieve the same effect in tensorflow? I tried tf.maximum(tf.fill(X.get_shape(), 0.0), X), but this throws ValueError: Cannot convert a partially known TensorShape to a Tensor: (?,).
PS. X is a 1-D tensor of shape (?,).
As it happens, your problem is exactly the same as computing the rectifier activation function, and TensorFlow has a built-in operator, tf.nn.relu(), that does exactly what you need:
X_with_negatives_set_to_zero = tf.nn.relu(X)
You can use tf.clip_by_value function as follows:
t = tf.clip_by_value(t, min_val, max_val)
It will clip tensor t in the range [min_val, max_val]. Here you can set min_val to 0 to clip all negative values and set those to 0. More documentation about clip_by_value.
A simple solution is to use the cast function keras documentation (as suggested by #ldavid)
X = tf.cast(X > 0, X.dtype) * X
Moreover this can be adapted to any threshold level with :
X = tf.cast(X > threshold, X.dtype) * X
One possible solution could be this (although it's not the best):
class TensorClass(object):
def __init__(tensor_values):
self.test_tensor = tf.Variable(tensor_values, name="test_tensor")
test_session = tf.Session()
with test_session.as_default():
tc = TensorClass([1, -1, 2, -2, 3])
test_session.run(tf.initialize_all_variables())
test_tensor_value = test_session.run(tc.test_tensor)
print(test_tensor_value) # Will print [1, -1, 2, -2, 3]
new_test_tensor_value = [element * int(element > 0) for element in test_tensor_value]
test_tensor_value_assign_op = tf.assign(tc.test_tensor, new_test_tensor_value)
test_session.run(test_tensor_value_assign_op)
test_tensor_value = test_session.run(tc.test_tensor)
print(test_tensor_value) # Will print [1 0 2 0 3]
While this does what you need, it's not done in tensorflow. We are pulling out a tensorflow variable, changing it, and putting it back again.
For performance critical things, don't use this because it's not very efficient.