I've been trying to figure out how to implement Show for my Tensor type for ages. Tensor is a thin wrapper round either a single value, or arbitrarily-nested Vects of values
import Data.Vect
Shape : Nat -> Type
Shape rank = Vect rank Nat
array_type: (shape: Shape rank) -> (dtype: Type) -> Type
array_type [] dtype = dtype
array_type (d :: ds) dtype = Vect d (array_type ds dtype)
data Tensor : (shape: Shape rank) -> (dtype: Type) -> Type where
MkTensor : array_type shape dtype -> Tensor shape dtype
Show dtype => Show (Tensor shape dtype) where
show (MkTensor x) = show x
I get
When checking right hand side of Prelude.Show.Main.Tensor shape dtype implementation of Prelude.Show.Show, method show with expected type
String
Can't find implementation for Show (array_type shape dtype)
which is understandable given array_type's not trivial. I believe that it should be showable, as I can show highly-nested Vects in the REPL as long their elements are Show. I guess Idris just doesn't know it's an arbitrarily nested Vect.
If I pull in some implicit parameters and case split on rank/shape, I get somewhere
Show dtype => Show (Tensor {rank} shape dtype) where
show {rank = Z} {shape = []} (MkTensor x) = show x -- works
show {rank = (S Z)} {shape = (d :: [])} (MkTensor x) = show x -- works
show {rank = (S k)} {shape = (d :: ds)} (MkTensor x) = show x -- doesn't work
and I can indefinitely expand this to higher and higher rank explicitly, where the RHS is always just show x, but I can't figure out how to get this to type check for all ranks. I'd guess some recursive thing is required.
EDIT to be clear, I want to know how to do this by using Idris' implementation of Show for Vects. I want to avoid having to construct an implementation manually myself.
If you want to go via the Show (Vect n a) implementations, you can do that as well, by defining a Show implementation that requires that there is a Show for the vector:
import Data.Vect
import Data.List
Shape : Nat -> Type
Shape rank = Vect rank Nat
array_type: (shape: Shape rank) -> (dtype: Type) -> Type
array_type [] dtype = dtype
array_type (d :: ds) dtype = Vect d (array_type ds dtype)
data Tensor : (shape: Shape rank) -> (dtype: Type) -> Type where
MkTensor : array_type shape dtype -> Tensor shape dtype
Show (array_type shape dtype) => Show (Tensor {rank} shape dtype) where
show (MkTensor x) = show x
For all choices of shape, the Show (array_type shape dtype) constraint will reduce to Show dtype, so e.g. this works:
myTensor : Tensor [1, 2, 3] Int
myTensor = MkTensor [[[1, 2, 3], [4, 5, 6]]]
*ShowVect> show myTensor
"[[[1, 2, 3], [4, 5, 6]]]" : String
You were on the right track: you can do it by writing a recursive function over the nested vectors, and then lifting it to your Tensor type in the Show implementation:
showV : Show dtype => array_type shape dtype -> String
showV {rank = 0} {shape = []} x = show x
showV {rank = (S k)} {shape = d :: ds} xs = combine $ map showV xs
where
combine : Vect n String -> String
combine ss = "[" ++ concat (intersperse ", " . toList $ ss) ++ "]"
Show dtype => Show (Tensor {rank} shape dtype) where
show (MkTensor x) = showV x
Example:
λΠ> show $ the (Tensor [1, 2, 3] Int) $ MkTensor [[[1, 2, 3], [4, 5, 6]]]
"[[[1, 2, 3], [4, 5, 6]]]"
Related
I want to write a function that takes in a collection (I'm not too fussed what kind) of dependent types, and returns another collection of the same types but perhaps with different values. The elements are of the form
data Tensor : (Vect r Nat) -> Type where
For example, a function that accepts a (Tensor [2, 3, 4], Tensor [2], Tensor []) or (Tensor [3],) and returns values of the same type.
What I've tried
Using dependent pairs: accept a List (s ** Tensor s). I don't then know how to constrain the output to have the same types.
Using tuples, but I'm not sure how to fix the element type to be Tensor
You can write a function that is indexed by the complete shape of all the Tensors, not just any one of them. The shape of each Tensor is a List Nat, so the shape of a list of them is a List (List Nat):
import Data.Vect
data Tensor : (Vect r Nat) -> Type where
data Tensors : List (List Nat) -> Type where
Nil : Tensors []
Cons : Tensor (fromList ns) -> Tensors nss -> Tensors (ns :: nss)
Here's an example of a shape-preserving map function:
mapTensors
: ({0 r : Nat} -> {0 ns : Vect r Nat} -> Tensor ns -> Tensor ns) ->
Tensors nss -> Tensors nss
mapTensors f Nil = Nil
mapTensors f (Cons t ts) = Cons (f t) (mapTensors f ts)
I have an B x M x N tensor, X, and I have and B x 1 tensor, Y, which corresponds to the index of tensor X at dimension=1 that I want to keep. What is the shorthand for this slice so that I can avoid a loop?
Essentially I want to do this:
Z = torch.zeros(B,N)
for i in range(B):
Z[i] = X[i][Y[i]]
the following code is similar to the code in the loop. the difference is that instead of sequentially indexing the array Z,X and Y we are indexing them in parallel using the array i
B, M, N = 13, 7, 19
X = np.random.randint(100, size= [B,M,N])
Y = np.random.randint(M , size= [B,1])
Z = np.random.randint(100, size= [B,N])
i = np.arange(B)
Y = Y.ravel() # reducing array to rank-1, for easy indexing
Z[i] = X[i,Y[i],:]
this code can be further simplified as
-> Z[i] = X[i,Y[i],:]
-> Z[i] = X[i,Y[i]]
-> Z[i] = X[i,Y]
-> Z = X[i,Y]
pytorch equivalent code
B, M, N = 5, 7, 3
X = torch.randint(100, size= [B,M,N])
Y = torch.randint(M , size= [B,1])
Z = torch.randint(100, size= [B,N])
i = torch.arange(B)
Y = Y.ravel()
Z = X[i,Y]
The answer provided by #Hammad is short and perfect for the job. Here's an alternative solution if you're interested in using some less known Pytorch built-ins. We will use torch.gather (similarly you can achieve this with numpy.take).
The idea behind torch.gather is to construct a new tensor-based on two identically shaped tensors containing the indices (here ~ Y) and the values (here ~ X).
The operation performed is Z[i][j][k] = X[i][Y[i][j][k]][k].
Since X's shape is (B, M, N) and Y shape is (B, 1) we are looking to fill in the blanks inside Y such that Y's shape becomes (B, 1, N).
This can be achieved with some axis manipulation:
>>> Y.expand(-1, N)[:, None] # expand to dim=1 to N and unsqueeze dim=1
The actual call to torch.gather will be:
>>> X.gather(dim=1, index=Y.expand(-1, N)[:, None])
Which you can reshape to (B, N) by adding in [:, 0].
This function can be very effective in tricky scenarios...
I'm trying to have a layer in keras that takes a flat tensor x (doesn't have zero value in it and shape = (batch_size, units)) multiplied by a mask (of the same shape), and it will sort it in the way that masked values will be placed first in the output (the order of the elements value doesn't matter). For clarity here is an example (batch_size = 1, units = 8):
It seems simple but the problem is that I can't find a good solution. Any code or idea is appreciated.
My current code is as below, If you know a more efficient way please let me know.
class Sort(keras.layers.Layer):
def call(self, inputs):
x = inputs.numpy()
nonx, nony = x.nonzero() # idxs of nonzero elements
zero = [np.where(x == 0)[0][0], np.where(x == 0)[1][0]] # idx of first zero
x_shape = tf.shape(inputs)
result = np.zeros((x_shape[0], x_shape[1], 2), dtype = 'int') # mapping matrix
result[:, :, 0] += zero[0]
result[:, :, 1] += zero[1]
p = np.zeros((x_shape[0]), dtype = 'int')
for i, j in zip(nonx, nony):
result[i, p[i]] = [i, j]
p[i] += 1
y = tf.gather_nd(inputs, result)
return y
Fairly new to numpy/python here, trying to figure out some less c-like, more numpy-like coding styles.
Background
I've got some code done that takes a fixed set of x values and multiple sets of corresponding y value sets and tries to find which set of the y values are the "most linear".
It does this by going through each set of y values in a loop, calculating and storing the residual from a straight line fit of those y's against the x's, then once the loop has finished finding the index of the minimum residual value.
...sorry this might make a bit more sense with the code below.
import numpy as np
import numpy.polynomial.polynomial as poly
# set of x values
xs = [1,22,33,54]
# multiple sets of y values for each of the x values in 'xs'
ys = np.array([[1, 22, 3, 4],
[2, 3, 1, 5],
[3, 2, 1, 1],
[34,23, 5, 4],
[23,24,29,33],
[5,19, 12, 3]])
# array to store the residual from a linear fit of each of the y's against x
residuals = np.empty(ys.shape[0])
# loop through the xs's and calculate the residual of a linear fit for each
for i in range(ys.shape[0]):
_, stats = poly.polyfit(xs, ys[i], 1, full=True)
residuals[i] = stats[0][0]
# the 'most linear' of the ys's is at np.argmin:
print('most linear at', np.argmin(residuals))
Question
I'd like to know if it's possible to "numpy'ize" that into a single expression, something like
residuals = get_residuals(xs, ys)
...I've tried:
I've tried the following, but no luck (it always passes the full arrays in, not row by row):
# ------ ok try to do it without a loop --------
def wrap(x, y):
_, stats = poly.polyfit(x, y, 1, full=True)
return stats[0][0]
res = wrap(xs, ys) # <- fails as passes ys as full 2D array
res = wrap(np.broadcast_to(xs, ys.shape), ys) # <- fails as passes both as 2D arrays
Could anyone give any tips on how to numpy'ize that?
From the numpy.polynomial.polynomial.polyfit docs (not to be confused with numpy.polyfit which is not interchangable)
:
x : array_like, shape (M,)
y : array_like, shape (M,) or (M, K)
Your ys needs to be transposed to have ys.shape[0] equal to xs.shape
def wrap(x, y):
_, stats = poly.polyfit(x, y.T, 1, full=True)
return stats[0]
res = wrap(xs, ys)
res
Out[]: array([284.57337884, 5.54709898, 0.41399317, 91.44641638,
6.34982935, 153.03515358])
I have a 2D tensor A with shape [batch_size, D] , and a 1D tensor B with shape [batch_size]. Each element of B is a column index of A, for each row of A, eg. B[i] in [0,D).
What is the best way in tensorflow to get the values A[B]
For example:
A = tf.constant([[0,1,2],
[3,4,5]])
B = tf.constant([2,1])
with desired output:
some_slice_func(A, B) -> [2,4]
There is another constraint. In practice, batch_size is actually None.
Thanks in advance!
I was able to get it working using a linear index:
def vector_slice(A, B):
""" Returns values of rows i of A at column B[i]
where A is a 2D Tensor with shape [None, D]
and B is a 1D Tensor with shape [None]
with type int32 elements in [0,D)
Example:
A =[[1,2], B = [0,1], vector_slice(A,B) -> [1,4]
[3,4]]
"""
linear_index = (tf.shape(A)[1]
* tf.range(0,tf.shape(A)[0]))
linear_A = tf.reshape(A, [-1])
return tf.gather(linear_A, B + linear_index)
This feels slightly hacky though.
If anyone knows a better (as in clearer or faster) please also leave an answer! (I won't accept my own for a while)
Code for what #Eugene Brevdo said:
def vector_slice(A, B):
""" Returns values of rows i of A at column B[i]
where A is a 2D Tensor with shape [None, D]
and B is a 1D Tensor with shape [None]
with type int32 elements in [0,D)
Example:
A =[[1,2], B = [0,1], vector_slice(A,B) -> [1,4]
[3,4]]
"""
B = tf.expand_dims(B, 1)
range = tf.expand_dims(tf.range(tf.shape(B)[0]), 1)
ind = tf.concat([range, B], 1)
return tf.gather_nd(A, ind)
the least hacky way is probably to build a proper 2d index by concatenating range(batch_size) and B, to get a batch_size x 2 matrix. then pass this to tf.gather_nd.
The simplest approach is to do:
def tensor_slice(target_tensor, index_tensor):
indices = tf.stack([tf.range(tf.shape(index_tensor)[0]), index_tensor], 1)
return tf.gather_nd(target_tensor, indices)
Consider to use tf.one_hot, tf.math.multiply and tf.reduce_sum to solve it.
e.g.
def vector_slice (inputs, inds, axis = None):
axis = axis if axis is not None else tf.rank(inds) - 1
inds = tf.one_hot(inds, inputs.shape[axis])
for i in tf.range(tf.rank(inputs) - tf.rank(inds)):
inds = tf.expand_dims(inds, axis = -1)
inds = tf.cast(inds, dtype = inputs.dtype)
x = tf.multiply(inputs, inds)
return tf.reduce_sum(x, axis = axis)