Mask and reshape a matrix using Tensorflow - tensorflow

I would like to use a mask to select vectors in mat variable. Below is the resultant matrix that I expcxt to get:
[[0, 0, 2, 2],
[4, 4, 5, 5],
[8, 8, 9, 9]]
Below is the code that I use:
mat = tf.constant([[[0, 0], [1, 1], [2, 2]],
[[4, 4], [5, 5], [6, 6]],
[[7, 7], [8, 8], [9, 9]]])
mask = tf.constant([[True, False, True],
[True, True, False],
[False, True, True]])
masked = tf.boolean_mask(mat, mask)
res = tf.keras.layers.Reshape([4])(masked)
with tf.Session() as sess:
print(res.eval())
However, I get an error:
tensorflow.python.framework.errors_impl.InvalidArgumentError: Input to reshape is a tensor with 12 values, but the requested shape has 24
[[{{node reshape/Reshape}}]]
How should I handle this issue?

If you just want reshape it then using tf.reshape instead, because tf.keras.layers.Reshape will take first dimension as batch size:
#TF2
import tensorflow as tf
mat = tf.constant([[[0, 0], [1, 1], [2, 2]],
[[4, 4], [5, 5], [6, 6]],
[[7, 7], [8, 8], [9, 9]]])
mask = tf.constant([[True, False, True],
[True, True, False],
[False, True, True]])
masked = tf.boolean_mask(mat, mask)
res = tf.reshape(masked, (-1, 4))
print(res)
If you want using tf.keras.layers.Reshape then you can do following:
import tensorflow as tf
mat = tf.constant([[[0, 0], [1, 1], [2, 2]],
[[4, 4], [5, 5], [6, 6]],
[[7, 7], [8, 8], [9, 9]]])
mask = tf.constant([[True, False, True],
[True, True, False],
[False, True, True]])
masked = tf.boolean_mask(mat, mask)
masked = tf.expand_dims(masked, axis=0)
res = tf.keras.layers.Reshape((-1, 4))(masked)
res = tf.squeeze(res, axis=0)
print(res)
Outputs of both:
tf.Tensor(
[[0 0 2 2]
[4 4 5 5]
[8 8 9 9]], shape=(3, 4), dtype=int32)

Related

Create 2d tensor of points using tensorflow

I'm trying to make a tensor with all the points between a certain range.
For example
min_x = 5
max_x = 7
min_y = 3
max_y = 5
points = get_points(min_x, max_x, min_y, max_y)
print(point) # [[5, 3], [5, 4], [5, 5], [6, 3], [6, 4], [6, 5], [7, 3], [7, 4], [7, 5]]
I'm trying to do this inside a tensorflow function. AKA #tf.function
Also all the inputs to get_points need to be tensors.
Thanks, I'm new to tensorflow as you can tell.
You can use tf.meshgrid, then stack x and y along the last dim after reshaping these two tensors.
min_x = 5
max_x = 7
min_y = 3
max_y = 5
def get_points(min_x, max_x, min_y, max_y):
x, y = tf.meshgrid(tf.range(min_x, max_x+1),tf.range(min_y, max_y+1))
_x = tf.reshape(x, (-1,1))
_y = tf.reshape(y, (-1,1))
return tf.squeeze(tf.stack([_x, _y], axis=-1))
res = get_points(min_x, max_x, min_y, max_y)
K.eval(res)
# array([[5, 3],
# [6, 3],
# [7, 3],
# [5, 4],
# [6, 4],
# [7, 4],
# [5, 5],
# [6, 5],
# [7, 5]], dtype=int32)

Delete specified column index from every row of 2d numpy array

I have a numpy array A as follows:
array([[1, 2, 3],
[4, 5, 6],
[7, 8, 9]])
and another numpy array column_indices_to_be_deleted as follows:
array([1, 0, 2])
I want to delete the element from every row of A specified by the column indices in column_indices_to_be_deleted. So, column index 1 from row 0, column index 0 from row 1 and column index 2 from row 2 in this case, to get a new array that looks like this:
array([[1, 3],
[5, 6],
[7, 8]])
What would be the simplest way of doing that?
One way with masking created with broadcatsed-comparison -
In [43]: a # input array
Out[43]:
array([[1, 2, 3],
[4, 5, 6],
[7, 8, 9]])
In [44]: remove_idx # indices to be removed from each row
Out[44]: array([1, 0, 2])
In [45]: n = a.shape[1]
In [46]: a[remove_idx[:,None]!=np.arange(n)].reshape(-1,n-1)
Out[46]:
array([[1, 3],
[5, 6],
[7, 8]])
Another mask based approach with the mask created with array-assignment -
In [47]: mask = np.ones(a.shape,dtype=bool)
In [48]: mask[np.arange(len(remove_idx)), remove_idx] = 0
In [49]: a[mask].reshape(-1,a.shape[1]-1)
Out[49]:
array([[1, 3],
[5, 6],
[7, 8]])
Another with np.delete -
In [64]: m,n = a.shape
In [66]: np.delete(a.flat,remove_idx+n*np.arange(m)).reshape(m,-1)
Out[66]:
array([[1, 3],
[5, 6],
[7, 8]])

Random valid data items in numpy array

Suppose I have a numpy array as follows:
data = np.array([[1, 3, 8, np.nan], [np.nan, 6, 7, 9], [np.nan, 0, 1, 2], [5, np.nan, np.nan, 2]])
I would like to randomly select n-valid items from the array, including their indices.
Does numpy provide an efficient way of doing this?
Example
data = np.array([[1, 3, 8, np.nan], [np.nan, 6, 7, 9], [np.nan, 0, 1, 2], [5, np.nan, np.nan, 2]])
n = 5
Get valid indices
y_val, x_val = np.where(~np.isnan(data))
n_val = y_val.size
Pick random subset of size n by index
pick = np.random.choice(n_val, n)
Apply index to valid coordinates
y_pick, x_pick = y_val[pick], x_val[pick]
Get corresponding data
data_pick = data[y_pick, x_pick]
Admire
data_pick
# array([2., 8., 1., 1., 2.])
y_pick
# array([3, 0, 0, 2, 3])
x_pick
# array([3, 2, 0, 2, 3])
Find nonzeros by :
In [37]: a = np.array(np.nonzero(data)).reshape(-1,2)
In [38]: a
Out[38]:
array([[0, 0],
[0, 0],
[1, 1],
[1, 1],
[2, 2],
[2, 3],
[3, 3],
[3, 0],
[1, 2],
[3, 0],
[1, 2],
[3, 0],
[2, 3],
[0, 1],
[2, 3]])
Now pick a random choice :
In [44]: idx = np.random.choice(np.arange(len(a)))
In [45]: data[a[idx][0],a[idx][1]]
Out[45]: 2.0

How to combined two arrays by interating with numpy? [duplicate]

I'd like to turn an open mesh returned by the numpy ix_ routine to a list of coordinates
eg, for:
In[1]: m = np.ix_([0, 2, 4], [1, 3])
In[2]: m
Out[2]:
(array([[0],
[2],
[4]]), array([[1, 3]]))
What I would like is:
([0, 1], [0, 3], [2, 1], [2, 3], [4, 1], [4, 3])
I'm pretty sure I could hack it together with some iterating, unpacking and zipping, but I'm sure there must be a smart numpy way of achieving this...
Approach #1 Use np.meshgrid and then stack -
r,c = np.meshgrid(*m)
out = np.column_stack((r.ravel('F'), c.ravel('F') ))
Approach #2 Alternatively, with np.array() and then transposing, reshaping -
np.array(np.meshgrid(*m)).T.reshape(-1,len(m))
For a generic case with for generic number of arrays used within np.ix_, here are the modifications needed -
p = np.r_[2:0:-1,3:len(m)+1,0]
out = np.array(np.meshgrid(*m)).transpose(p).reshape(-1,len(m))
Sample runs -
Two arrays case :
In [376]: m = np.ix_([0, 2, 4], [1, 3])
In [377]: p = np.r_[2:0:-1,3:len(m)+1,0]
In [378]: np.array(np.meshgrid(*m)).transpose(p).reshape(-1,len(m))
Out[378]:
array([[0, 1],
[0, 3],
[2, 1],
[2, 3],
[4, 1],
[4, 3]])
Three arrays case :
In [379]: m = np.ix_([0, 2, 4], [1, 3],[6,5,9])
In [380]: p = np.r_[2:0:-1,3:len(m)+1,0]
In [381]: np.array(np.meshgrid(*m)).transpose(p).reshape(-1,len(m))
Out[381]:
array([[0, 1, 6],
[0, 1, 5],
[0, 1, 9],
[0, 3, 6],
[0, 3, 5],
[0, 3, 9],
[2, 1, 6],
[2, 1, 5],
[2, 1, 9],
[2, 3, 6],
[2, 3, 5],
[2, 3, 9],
[4, 1, 6],
[4, 1, 5],
[4, 1, 9],
[4, 3, 6],
[4, 3, 5],
[4, 3, 9]])

Calling reshape on an LSTMStateTuple turns it into a tensor

I was using dynamic_rnn with an LSTMCell, which put out an LSTMStateTuple containing the inner state. Calling reshape on this object (by my mistake) results in a tensor without causing any error at graph creation. I didn't get any error at runtime when feeding input through the graph, either.
Code:
cell = tf.contrib.rnn.LSTMCell(size, state_is_tuple=True, ...)
outputs, states = tf.nn.dynamic_rnn(cell, inputs, ...)
print(states) # state is an LSTMStateTuple
states = tf.reshape(states, [-1, size])
print(states) # state is a tensor of shape [?, size]
Is this a bug (I ask because it's not documented anywhere)? What is the reshaped tensor holding?
I have conducted a similar experiment which may gives you some hints:
>>> s = tf.constant([[0, 0, 0, 1, 1, 1],
[2, 2, 2, 3, 3, 3]])
>>> t = tf.constant([[4, 4, 4, 5, 5, 5],
[6, 6, 6, 7, 7, 7]])
>>> g = tf.reshape((s, t), [-1, 3]) # <tf.Tensor 'Reshape_1:0' shape=(8, 3) dtype=int32>
>>> sess.run(g)
array([[0, 0, 0],
[1, 1, 1],
[2, 2, 2],
[3, 3, 3],
[4, 4, 4],
[5, 5, 5],
[6, 6, 6],
[7, 7, 7]], dtype=int32)
We can see that it just concatenates the two tensors in the first dimension and performs the reshaping. Since the LSTMStateTuple is like a namedtuple then it has the same effect as tuple and I think this is also what happens in your case.
Let's go further,
>>> st = tf.contrib.rnn.LSTMStateTuple(s, t)
>>> gg = tf.reshape(st, [-1, 3])
>>> sess.run(gg)
array([[0, 0, 0],
[1, 1, 1],
[2, 2, 2],
[3, 3, 3],
[4, 4, 4],
[5, 5, 5],
[6, 6, 6],
[7, 7, 7]], dtype=int32)
We can see that if we create a LSTMStateTuple, the result verifies our assumption.