I have a vector x = <C elements>. I want to multiple it with a matrix Z = <C x C elements>. This will give an output of <1xC> matrix ( <C> vector).
I can just multiply them if I have one samples only for each.
But during training, I have tensors x = <NxC> and Z = <NxCxC>, where 'N' is my mini-batch size. How can I achieve the above computation in this case. Plain tf.matmul() return error complaining about dimensions.
ValueError: Shape must be rank 2 but is rank 3 for 'Channel/MatMul' (op: 'MatMul') with input shapes: [?,2], [?,2,2].
Thanks,
Vishnu Raj
Here is one way to do it:
Expand the last dimension of tensor x. Its shape is now (N, C).
Multiply tensors Z and x with broadcasting. The shape of the resulting tensor is (N, C, C).
Sum over the last dimension of that tensor.
For example:
import tensorflow as tf
N = 2
C = 3
Z = tf.ones((N, C, C)) # Shape=(N, C, C)
x = tf.reshape(tf.range(0, N*C, dtype=tf.float32), shape=(N, C)) # Shape=(N, C)
mul = tf.reduce_sum(Z * x[:, :, None], axis=-1) # Shape=(N, C)
with tf.Session() as sess:
print(sess.run(mul))
Related
I have an idea for a tensor operation that would not be difficult to implement via iteration, with batch size one. However I would like to parallelize it as much as possible.
I have two tensors with shape (n, 5) called X and Y. X is actually supposed to represent 5 one-dimensional tensors with shape (n, 1): (x_1, ..., x_n). Ditto for Y.
I would like to compute a tensor with shape (n, 25) where each column represents the output of the tensor operation f(x_i, y_j), where f is fixed for all 1 <= i, j <= 5. The operation f has output shape (n, 1), just like x_i and y_i.
I feel it is important to clarify that f is essentially a fully-connected layer from the concatenated [...x_i, ...y_i] tensor with shape (1, 10), to an output layer with shape (1,5).
Again, it is easy to see how to do this manually with iteration and slicing. However this is probably very slow. Performing this operation in batches, where the tensors X, Y now have shape (n, 5, batch_size) is also desirable, particularly for mini-batch gradient descent.
It is difficult to really articulate here why I desire to create this network; I feel it is suited for my domain of 'itemized tabular data' and cuts down significantly on the number of weights per operation, compared to a fully connected network.
Is this possible using tensorflow? Certainly not using just keras.
Below is an example in numpy per AloneTogether's request
import numpy as np
features = 16
batch_size = 256
X_batch = np.random.random((features, 5, batch_size))
Y_batch = np.random.random((features, 5, batch_size))
# one tensor operation to reduce weights in this custom 'layer'
f = np.random.random((features, 2 * features))
for b in range(batch_size):
X = X_batch[:, :, b]
Y = Y_batch[:, :, b]
for i in range(5):
x_i = X[:, i:i+1]
for j in range(5):
y_j = Y[:, j:j+1]
x_i_y_j = np.concatenate([x_i, y_j], axis=0)
# f(x_i, y_j)
# implemented by a fully-connected layer
f_i_j = np.matmul(f, x_i_y_j)
All operations you need (concatenation and matrix multiplication) can be batched.
Difficult part here is, that you want to concatenate features of all items in X with features of all items in Y (all combinations).
My recommended solution is to expand the dimensions of X to [batch, features, 5, 1], expand dimensions of Y to [batch, features, 1, 5]
Than tf.repeat() both tensors so their shapes become [batch, features, 5, 5].
Now you can concatenate X and Y. You will have a tensor of shape [batch, 2*features, 5, 5]. Observe that this way all combinations are built.
Next step is matrix multiplication. tf.matmul() can also do batch matrix multiplication, but I use here tf.einsum() because I want more control over which dimensions are considered as batch.
Full code:
import tensorflow as tf
import numpy as np
batch_size=3
features=6
items=5
x = np.random.uniform(size=[batch_size,features,items])
y = np.random.uniform(size=[batch_size,features,items])
f = np.random.uniform(size=[2*features,features])
x_reps= tf.repeat(x[:,:,:,tf.newaxis], items, axis=3)
y_reps= tf.repeat(y[:,:,tf.newaxis,:], items, axis=2)
xy_conc = tf.concat([x_reps,y_reps], axis=1)
f_i_j = tf.einsum("bfij, fg->bgij", xy_conc,f)
f_i_j = tf.reshape(f_i_j , [batch_size,features,items*items])
I have a vector as input for a layer.
For this vector I would like to calculate the cosine similariy to several other vectors (that can be arranged in a matrix)
Example (other vectors: c1,c2,c3 ...):
Input:
v
(len(v) = len(c1) = len(c2) ...)
Output:
[cosinsSimilarity(v,c1),cosineSimilarity(v,c2),cosineSimilarity(v,c3),consinSimilarity(v,...)]
I think the problem could be solved by an approach like the following:
cosineSimilarity (v, matrix (c1, c2, c3, ...))
but unfortunately I have no idea how I can implement that in a keras layer with input_shape(1,len(v)) and output_shape(1,columns(matrix))
okay it was so easy now. I simply inserted this lambda layer
because the mean function also works for vector - matrix multiplication.
def cosine_similarity(x):
#shape x: (10,)
y = tf.constant([c1,c2])
#shape c1,c2: (10,)
#shape y: (2,10)
x = K.l2_normalize(x, -1)
y = K.l2_normalize(y, -1)
s = K.mean(x * y, axis=-1, keepdims=False) * 10
return s
input is in my case a vector with shape (10,). Output is a vector with the cosine-similarity-values of the input vector to c1 and c2 with shape (2,)
I have two 2-D tensors of shape say m X d and n X d. What is the optimized(i.e. without for loops) or the tensorflow way of evaluating the pairwise euclidean distance between these two tensors so that I get an output tensor of shape m X n. I need it for creating the squared term of a Gaussian kernel for ultimately having a covariance matrix of size m x n.
The equivalent unoptimized numpy code would look like this
difference_squared = np.zeros((x.shape[0], x_.shape[0]))
for row_iterator in range(difference_squared.shape[0]):
for column_iterator in range(difference_squared.shape[1]):
difference_squared[row_iterator, column_iterator] = np.sum(np.power(x[row_iterator]-x_[column_iterator], 2))
I found the answer by taking help from here. Assuming the two tensors are x1 and x2, and their dimensions are m X d and n X d, their pair-wise Euclidean distance is given by
tile_1 = tf.tile(tf.expand_dims(x1, 0), [n, 1, 1])
tile_2 = tf.tile(tf.expand_dims(x2, 1), [1, m, 1])
pairwise_euclidean_distance = tf.reduce_sum(tf.square(tf.subtract(tile_1, tile_2)), 2))
I ma trying to understand tf.rank function in tensorflow. From the documentation here, I understood that rank should return the number of distinct elements in the tensor.
Here x and weights are 2 distinct 2*2 tensors with 4 distinct elemnts in each of them. However, rank() function outputs are:
Tensor("Rank:0", shape=(), dtype=int32) Tensor("Rank_1:0", shape=(),
dtype=int32)
Also, for the tensor x, I used tf.constant() with dtype = float to convert ndarray into float32 tensor but the rank() still outputs as int32.
g = tf.Graph()
with g.as_default():
weights = tf.Variable(tf.truncated_normal([2,2]))
x = np.asarray([[1 , 2], [3 , 4]])
x = tf.constant(x, dtype = tf.float32)
y = tf.matmul(weights, x)
print (tf.rank(x), tf.rank(weights))
with tf.Session(graph = g) as s:
tf.initialize_all_variables().run()
print (s.run(weights), s.run(x))
print (s.run(y))
How should I interpret the output.
Firstly, tf.rank returns the dimension of a tensor, not the number of elements. For instance, the output from tf.rank called for the 2x2 matrix would be 2.
To print the rank of a tensor, create an appropriate node, e.g. rank = tf.rank(x) and then evaluate this node using a Session.run(), as you've done for weights and x. Execution of print (tf.rank(x), tf.rank(weights)) expectedly prints out description of tensors, as tf.rank(x), tf.rank(weights) are nodes of the graph, not the variables with defined values.
I know tensorflow can calculate expressions like [ [a,b,c] ] x [ [x],[y],[z] ] when the elements are primitive data type (integer or float).
Is it possible to perform a similar computation when each of a, b and c is a 1x3 matrix and x, y and z are 3x1 matrices?
Can TensorFlow calculate and optimize this formula?
The tf.batch_matmul() operator can perform matrix multiplications on batches of matrices. In this case, you would have a tensor abc of shape (3, 1, 3) (where abc[0, :, :] = a, abc[1, :, :] = b, etc.) and a tensor xyz of shape (3, 3, 1) (where xyz[0, :, :] = x, etc.).
abc = ...
xyz = ...
result = tf.batch_matmul(abc, xyz)
print result.get_shape() # ==> "(3, 1, 1)"
result is a 3-D tensor with contents equivalent to tf.pack([tf.matmul(a, x), tf.matmul(b, y), tf.matmul(c, z)]).