Given a Tensor whose shape is known only at the compute time, how to convert a Tensor of shape, for e.g., (n1, n2, m*n) to (n1, n2, m, n) assuming there is a prior condition that, the last dimension is strictly a multiple of n.
Since, the only information we have is that the last dimension to be a multiple of n, the following line of code simply does the trick. Here, x is a tensor variable whose shape is computed at the compilation of network.
x = C.reshape(x, x.shape[:-1] + (-1, n))
Related
I have a tensor phi = np.random.rand(n, n, 3) and a matrix D = np.random.rand(3, 3). I want to multiply the matrix D along the last axis of phi so that the output has shape (n, n, 3). I have tried this
np.einsum("klj,ij->kli", phi, D)
But I am not confident in this notation at all. Basically I want to do
res = np.zeros_like(phi)
for i in range(n):
for j in range(n):
res[i, j, :] = D.dot(phi[i, j, :])
You are treating phi as an n, n array of vectors, each of which is to be left-multiplied by D. So you want to keep the n, n portion of the shape exactly as-is. The last (only) dimension of the vectors should be multiplied and summed with the last dimension of the matrix (the vectors are implicitly 3x1):
np.einsum('ijk,lk->ijl', phi, D)
OR
np.einsum('ij,klj->kli', D, phi)
It's likely much simpler to use broadcasting with np.matmul (the # operator):
np.squeeze(D # phi[..., None])
You can omit the squeeze if you don't mind the extra unit dimension at the end.
I have N Gaussian distributions (multivariate) with N different means (covariance is the same for all of them) in D dimensions.
I also have N evaluation points, where I want to evaluate each of these (log) PDFs.
This means I need to get a NxN matrix, call it "kernels". That is, the (i,j)-th entry is the j-th Gaussian evaluated at the i-th point. A naive approach is:
from torch.distributions.multivariate_normal import MultivariateNormal
import numpy as np
# means contains all N means as rows and is thus N x D
# same for eval_points
# cov is not a problem , just a DxD matrix that is equal for all N Gaussians
kernels = np.empty((N,N))
for i in range(N):
for j in range(N):
kernels[i][j] = MultivariateNormal(means[j], cov).log_prob(eval_points[i])
Now one for loop we can get rid of easily, since for example if we wanted all the evaluations of the first Gaussian , we simply do:
MultivariateNormal(means[0], cov).log_prob(eval_points).squeeze()
and this gives us a N x 1 list of values, that is the first Gaussian evaluated at all N points.
My problem is that , in order to get the full N x N matrix , this doesn't work:
kernels = MultivariateNormal(means, cov).log_prob(eval_points).squeeze()
It doesn't figure out that it should evaluate each mean with all evaluation points in eval_points, and it doesn't return a NxN matrix with these which would be what I want. Therefore, I am not able to get rid of the second for loop, over all N Gaussians.
You are passing wrong shaped tensors to MultivariateNormal's constructor. You should pass a collection of mean vectors of shape (N, D) and a collection of precision matrix cov of shape (N, D, D) for N D-dimensional gaussian.
You are passing mu of shape (N, D) but your precision matrix is not well-shaped. You will need to repeat the precision matrix N number of times before passing it to the MultivariateNormal constructor. Here's one way to do it.
N = 10
D = 3
# means contains all N means as rows and is thus N x D
# same for eval_points
# cov is not a problem , just a DxD matrix that is equal for all N Gaussians
mu = torch.from_numpy(np.random.randn(N, D))
cov = torch.from_numpy(make_spd_matrix(D, D))
cov_n = cov[None, ...].repeat_interleave(N, 0)
assert cov_n.shape == (N, D, D)
kernels = MultivariateNormal(mu, cov_n)
I have a tensor T of shape Batch_Size x Num_Items x Item_Dimension and another tensor P of shape Batch_Size x Num_Items, where the Num_Items values in each batch of P sum to 1 (a probability distribution of items for each batch). I want to sample without replacement N items from T according to probability distribution P. The resulting tensor should be of shape Batch_Size x N x Item_Dimension. How would I do this?
Take a look at
https://github.com/tensorflow/tensorflow/issues/9260
Though note I believe you need logits instead of probs for Gumbel max sampling.
Given a tensor A of shape [?, n, l] and a tensor D of shape [?, n], I want to divide each row of tensor a of shape [n,l] of A by the corresponding scalar of D.
I thought I could somehow do this by using the broadcasting behavior of tf.div. Unfortunatley I am stuck.
Here you need to extend D to match the dimension of A:
res = A / D[...,tf.newaxis])
In Tensorflow, what is the simplest way to recursively apply a set of ops where each recursive step takes as input the output of the last step? The critical feature here is that the number of recursion steps is specified at evaluation, and can vary between examples.
For example, for input tensors X, 5, and some function F, I'd like to compute
F( F( F( F( F(X) ) ) ) )
But the same model should also be able to compute X, 3 as:
F( F( F( X ) ) )
Unfortunately, while loops will not work, because they enforce strict shape invariants, meaning that all dimensions of tensor X have to be known at compile time, but I won't know the example size (first dimension of X) until runtime.
Thanks for your help!
What you want is easily achieved in TensorFlow using while loops, namely the tf.while_loop construct. tf.while_loop has a shape_invariants parameter, which lets you specify a (possibly partial) shape for every loop variable. That shape only has to be less specific than the shape that loop variable can take during the loop. Here is a complete example where F is appending a value to a tensor.
import tensorflow as tf
sess = tf.InteractiveSession()
i = tf.constant(0, dtype=tf.int32)
x = tf.constant(0, shape=[0], dtype=tf.int32)
i, x = tf.while_loop(lambda i, _: i < 10,
lambda i, x: (i + 1, tf.concat_v2([x, tf.expand_dims(i, 0)], 0)),
[i, x],
shape_variants=[i.get_shape(), tf.TensorShape([None])])
print(x.eval())
[0 1 2 3 4 5 6 7 8 9]