What does it mean if this is the return value for tf.losses.softmax_cross_entropy_loss?
<tf.Tensor 'softmax_cross_entropy_loss/value:0' shape=() dtype=float32>
Does the fact that is states value:0 mean and shape=() mean that nothing was computed?
Nothing has been computed because you are displaying the tensors in the graph before any data has been passed through them. Let's say
sce = tf.losses.softmax_cross_entropy_loss(input)
Then to actually get the loss value you have to feed data into it using
sess = tf.Session()
...
loss = sess.run(sce, feed_dict)
where feed_dict is the dictionary for your data. Loss will now return the actual numerical loss value.
value is just an indicator for the group of computations that the value belongs to. For example: tf.reduce_mean returns tf.Tensor 'Mean_1:0' shape=() dtype=float32 because it is a mean calculation. The 0 does not mean its current value is 0, it is just used for indexing.
Additionally, your tensor shape is () because the single loss value doesn't have a batch size, x or y directions, or channels (assuming you are working with 4D tensors) so that is also ok.
Related
The API of sampled_softmax_loss goes like:
tf.nn.sampled_softmax_loss(
weights,
biases,
labels,
inputs,
num_sampled,
num_classes,
num_true=1,
sampled_values=None,
...
)
I've noticed that arg sampled_values is the one which determines what negatives samples we take and it's returned by a _candidate_sampler function like tf.random.fixed_unigram_candidate_sampler.
And in tf.random.fixed_unigram_candidate_sampler we can decide the probability of each sample chosen as negative sample.
How can I assign certain sample as negative sample on purpose?
For instance, in the case of recommender system, I'd like to add some hard negative sample to the model. So I want the hard negative samples been chosen for sure, not by probability like in _candidate_sampler function
How can I assign certain samples as negative samples when using sampled_softmax_loss in TensorFlow?
You need to understand that the sampler candidates function is only a remarks function and your question is right about how to create a negative sampler.
You don't need to create a negative sampler when you assigned a unique. The sampler is (sampled_candidates, true_expected_count, sampled_expected_count). Hard negative is when you add contrast values to significant the candidates. In this way, you can have it with distributions.
Random Uniform Candidates Sampler
Candidate Sampling
Sampled SoftMax
Simple: It is weight and bias are varies, and functions are the same.
import tensorflow as tf
weights = tf.zeros([4, 1])
biases = tf.zeros([4])
labels = tf.ones([4, 1])
inputs = tf.zeros([4, 1])
num_sampled = 1
num_classes = 1
true_classes = tf.ones([4, 4], dtype=tf.int64)
num_true = 4
num_sampled = 1
unique =True
range_max = 1
sampler = tf.random.uniform_candidate_sampler(
true_classes,
num_true,
num_sampled,
unique,
range_max,
seed=None,
name=None
)
loss_fn = tf.nn.sampled_softmax_loss(
weights,
biases,
labels,
inputs,
num_sampled,
num_classes,
num_true=1,
sampled_values=sampler,
remove_accidental_hits=True,
seed=None,
name='sampled_softmax_loss'
)
print( loss_fn )
Output: Value output as examples, and ran three times.
tf.Tensor([6.437752 6.437752 6.437752 6.437752], shape=(4,), dtype=float32)
tf.Tensor([6.437752 6.437752 6.437752 6.437752], shape=(4,), dtype=float32)
tf.Tensor([6.437752 6.437752 6.437752 6.437752], shape=(4,), dtype=float32)
My codes are as follow:
v = tf.Variable(initial_value=v, trainable=True)
v.shape is (1, 768)
In the model:
inputs_sents = keras.Input(shape=(50,3))
inputs_events = keras.Input(shape=(50,768))
x_1 = tf.matmul(v,tf.transpose(inputs_events))
x_2 = tf.matmul(x_1,inputs_sents)
But I got an error,
ValueError: Dimensions must be equal, but are 768 and 50 for
'{{node BatchMatMulV2_3}} =
BatchMatMulV2[T=DT_FLOAT,
adj_x=false,
adj_y=false](BatchMatMulV2_3/ReadVariableOp,
Transpose_3)' with input shapes: [1,768], [768,50,?]
I think it takes consideration of the batch? But how shall I deal with this?
v is a trainable vector (or 2d array with first dimension being 1), I want it to be trained in the training process.
PS: This is the result I got using the codes provided by the first answer, I think it is incorrect cause keras already takes consideration of the first batch dimension.
Plus, from the keras documentation,
shape: A shape tuple (integers), not including the batch size. For instance, shape=(32,) indicates that the expected input will be batches of 32-dimensional vectors. Elements of this tuple can be None; 'None' elements represent dimensions where the shape is not known.
https://keras.io/api/layers/core_layers/input/
Should I rewrite my codes without keras?
The shape of a batch is denoted by None:
import numpy as np
inputs_sents = keras.Input(shape=(None,1,3))
inputs_events = keras.Input(shape=(None,1,768))
v = np.ones(shape=(1,768), dtype=np.float32)
v = tf.Variable(initial_value=v, trainable=True)
x_1 = tf.matmul(v,tf.transpose(inputs_events))
x_2 = tf.matmul(x_1,inputs_sents)
I have a rank 3 tensor of shape (100, 257, 121) that we'll call y_pred.
I have extracted one rank 2 tensor from this of shape (257, 121) that we'll call y_element.
Is there a method similar to tensorflow.not_equal() that will compare y_element to every other rank 2 tensor element along axis 0 of y_pred, and return a tensor of bools of shape (100)?
Calling tensorflow.not_equal(y_pred, y_element) does return a tensor of bools, but of the same shape as y_pred, suggesting it's doing something like comparing the y_element tensor to all 3109700 values in y_pred instead.
y_pred = tf.Variable(tf.ones((100, 257, 121)))
y_element = tf.Variable(tf.ones((257, 121)))
y_element[-1,:].assign(tf.zeros(121))
tf.reduce_all(tf.equal(y_pred, tf.expand_dims(y_element,0)), axis=[1,2])
this piece of code is based on the method you introduced and it compares element-wise the 2 tensors on the axis zeros. it returns an array of shape equal to the first axis (100 in our case). every element of the tensor of rank 3 is compared with the tensor of rank 2. True is return if ALL the tensor are equal otherwise false
I would like to define a Tensorflow operation that allows me to, given a tensor, return a boolean tensor of the same size where all values in the tensor greater than 0 are set to 1, and all other values are set to 0.
I have tried using tf.cond, tf.where, x>0, but I'm getting the following error:
ValueError: No gradients provided for any variable, check your graph
for ops that do not support gradients, between variables
Are there Tensorflow operation(s) that will allow me to perform this binary thresholding that are also differentiable/have defined gradients?
Here is the code that is causing the error:
x1 and x2 are tensors of shape (32, 128, 128, 1):
diff = tf.abs(x1-x2)
diff = tf.to_float(diff > 0.0)
y = tf.reduce_mean(tf.reduce_sum(diff, axis=[1, 2, 3]))
Thanks
When we do tf.embedding_lookup, it returns a vector (not matrix).
In [244]: one_hot_label = tf.nn.embedding_lookup(np.eye(vocab_size), Y[labels_i])
In [245]: one_hot_label
Out[245]: <tf.Tensor 'embedding_lookup_43975:0' shape=(20, 8000) dtype=float64>
I need to reshape this (20,8000) tensor into (20,8000,1). How should I do it?
I'm not asking for hard-cord (20,8000,1) using tf.reshape. I'm asking in general how to convert 2d -> 3d or higher.
You can use tf.expand_dims: this operation inserts a dimension of 1 into the tensor's shape.
one_hot_label = tf.expand_dims(one_hot_label, axis=2)