Related
I'm trying to update rank3 tensor using tensor_scatter_nd_update fn.
For rank2 tensor I used the following snippet:
tensor = tf.zeros((3, 2))
indices = [[0], [2]]
updates = [[5, 5], [10, 10]]
output = tf.tensor_scatter_nd_update(tensor, indices, updates)
print(output)
tf.Tensor(
[[ 5. 5.]
[ 0. 0.]
[10. 10.]], shape=(3, 2), dtype=float32)
Now I want to perform the same op for batches -> basically on rank3 tensor:
tensor = tf.zeros((4, 3, 2))
indices = [[[0], [2]],
[[1], [0]],
[[0], [1]],
[[0], [2]]]
updates = [[[5, 5], [10, 10]],
[[1, 1], [7, 7]],
[[3, 3], [2, 2]],
[[5, 5], [1, 1]]]
output = tf.tensor_scatter_nd_update(tensor, indices, updates)
However, I'm getting a shape mismatch error:
Inner dimensions of output shape must match inner dimensions of updates shape Output: [4,3,2] updates: [4,2,2]
If you want to update by the first dimension, for example you want to change [0], [2] and [3], then each of the updates has shape (1, 3, 2) but you have three updates so (3, 3, 2):
tensor = tf.zeros((4, 3, 2))
indices = [[0], [2], [3]] # shape (3, 1)
updates = [[[1, 1], [2, 2], [3, 3]], # <- this is [0] and will go to output[0,:,:]
[[10, 10], [10, 10], [10, 10]], # <- this is [2]
[[101,102], [103, 104], [105, 106]]] # shape (3, 3, 2)
output = tf.tensor_scatter_nd_update(tensor, indices, updates)
output:
tf.Tensor(
[[[ 1. 1.]
[ 2. 2.]
[ 3. 3.]]
[[ 0. 0.]
[ 0. 0.]
[ 0. 0.]]
[[ 10. 10.]
[ 10. 10.]
[ 10. 10.]]
[[101. 102.]
[103. 104.]
[105. 106.]]], shape=(4, 3, 2), dtype=float32)
If you want to update by the first two dimensions, for example you want to update the positions [0, 1] and [2, 2], then each of updates has shape (1, 1, 2), but you have two updates, so updates will have shape (2, 1, 2):
tensor = tf.zeros((4, 3, 2))
indices = [[[0,1]], [[2,2]]] # shape (2, 1, 2)
updates = [[[1, 1]], # <- this will go to output[0,1,:]
[[10, 10]]] # shape (2, 1, 2)
output = tf.tensor_scatter_nd_update(tensor, indices, updates)
output:
tf.Tensor(
[[[ 0. 0.]
[ 1. 1.]
[ 0. 0.]]
[[ 0. 0.]
[ 0. 0.]
[ 0. 0.]]
[[ 0. 0.]
[ 0. 0.]
[10. 10.]]
[[ 0. 0.]
[ 0. 0.]
[ 0. 0.]]], shape=(4, 3, 2), dtype=float32)
It is simply as Matrix multiplication that follows the sample it reveals your works, and the index indicates the target.
As in programming tools built carefully it will consider the orders and the number of data contain as a basic program.
The outer and inner is indicates the multiplying matrix dimensions as we learn in mathematics and science programs. Sample: ( 8, 2 ) * ( 2 * 8 )
Specify types of inputs, index can be int, scatters expecting float32 since the calculation of Eigaint and their relationship.
For more than 2 dimensions you had options to convert it back to 2 dimensions or vary each index input, see some example create function and loop though the indexes.
Sample: Simple solution
tensor = tf.zeros((8, 2))
tensor = tf.cast( tensor, dtype=tf.int32 )
indices = [[[0], [2]],
[[1], [0]],
[[0], [1]],
[[0], [2]]]
indices = tf.constant(indices, shape=(8, 1), dtype=tf.int32)
updates = [[[5, 5], [10, 10]],
[[1, 1], [7, 7]],
[[3, 3], [2, 2]],
[[5, 5], [1, 1]]]
updates = tf.constant(updates, shape=(8, 2), dtype=tf.int32)
output = tf.tensor_scatter_nd_update(tensor, indices, updates)
print(output)
Output
tf.Tensor(
[[5 5]
[2 2]
[1 1]
[0 0]
[0 0]
[0 0]
[0 0]
[0 0]], shape=(8, 2), dtype=int32)
Sample 2: Transform input is easiest way
tensor = tf.zeros((4, 3, 2))
tensor = tf.cast( tensor, dtype=tf.int32 )
tensor = tf.constant(tensor, shape=(12, 2), dtype=tf.int32)
indices = [[[0], [2], [1]],
[[1], [0], [1]],
[[0], [1], [1]],
[[0], [2], [1]]]
indices = tf.constant(indices, shape=(12, 1), dtype=tf.int32)
updates = [[[5, 5, 1], [10, 10, 1]],
[[1, 1, 1], [7, 7, 1]],
[[3, 3, 1], [2, 2, 1]],
[[5, 5, 1], [1, 1, 1]]]
updates = tf.constant(updates, shape=(12, 2), dtype=tf.int32)
output = tf.tensor_scatter_nd_update(tensor, indices, updates)
print(output)
Output
tf.Tensor(
[[5 5]
[1 1]
[1 1]
[0 0]
[0 0]
[0 0]
[0 0]
[0 0]
[0 0]
[0 0]
[0 0]
[0 0]], shape=(12, 2), dtype=int32)
Sample 3: Repeating where you can create recursive
global index
index = 5
#tf.function
def reverse_count( index_input ):
global index
index = index - 1
if index > 0 :
return True
else :
return False
tensor = tf.zeros((3, 2))
tensor = tf.cast( tensor, dtype=tf.int32 )
index = 5
indices = [[[0], [2], [1]],
[[1], [0], [1]],
[[0], [1], [1]],
[[0], [2], [1]]]
indices = tf.constant(indices, shape=(4, 3, 1), dtype=tf.int32)
updates = [[5, 5], [10, 10], [1, 1]]
updates = tf.constant(updates, shape=(3, 2), dtype=tf.int32)
print( index )
indice = tf.where([reverse_count( index )], indices[index - 1,:,:1], tf.constant([[-1, -1, -1]], shape=(3, 1)).numpy() )
print( 'tensor' )
print( tensor )
print( 'indice' )
print( indice )
print( 'updates' )
print( updates )
output = tf.tensor_scatter_nd_update(tensor, indice, updates)
print( 'output' )
print( output )
Output
5
tensor
tf.Tensor(
[[0 0]
[0 0]
[0 0]], shape=(3, 2), dtype=int32)
indice
tf.Tensor(
[[0]
[2]
[1]], shape=(3, 1), dtype=int32)
updates
tf.Tensor(
[[ 5 5]
[10 10]
[ 1 1]], shape=(3, 2), dtype=int32)
output
tf.Tensor(
[[ 5 5]
[ 1 1]
[10 10]], shape=(3, 2), dtype=int32)
I am confused about the difference between x[:] and x[...] in Numpy.
For example, I have this 2-d array
[[4, 1, 9],
[5, 2, 0]]
When I tried to print out x[:] and x[...], they both gave me the same output:
[[4, 1, 9],
[5, 2, 0]]
However, when I tried to broadcast it by adding one dimension
print(np.broadcast_to(x[:,None],(2,3,3)))
print(np.broadcast_to(x[...,None],(2,3,3)))
They gave me different results.
[[[4 1 9]
[4 1 9]
[4 1 9]]
[[5 2 0]
[5 2 0]
[5 2 0]]]
[[[4 4 4]
[1 1 1]
[9 9 9]]
[[5 5 5]
[2 2 2]
[0 0 0]]]
I am trying to figure out the difference but cannot.
In [91]: x = np.array([[4, 1, 9],
...: [5, 2, 0]])
In [92]: x
Out[92]:
array([[4, 1, 9],
[5, 2, 0]])
These just make a whole slice, a view of the original:
In [93]: x[:]
Out[93]:
array([[4, 1, 9],
[5, 2, 0]])
In [94]: x[...]
Out[94]:
array([[4, 1, 9],
[5, 2, 0]])
In [95]: x[:,:]
Out[95]:
array([[4, 1, 9],
[5, 2, 0]])
Trailing : are added as needed, but you can't supply more than the number of dimensions:
In [96]: x[:,:,:]
Traceback (most recent call last):
File "<ipython-input-96-9d8949edcb06>", line 1, in <module>
x[:,:,:]
IndexError: too many indices for array: array is 2-dimensional, but 3 were indexed
None adds a dimension:
In [97]: x[:,None].shape # after the first
Out[97]: (2, 1, 3)
In [98]: x[...,None].shape # at the end
Out[98]: (2, 3, 1)
In [99]: x[:,:,None].shape # after the 2nd
Out[99]: (2, 3, 1)
In [100]: x[:,None,:].shape # same as 97
Out[100]: (2, 1, 3)
In [101]: x[None].shape # same as [None,:,:] [None,...]
Out[101]: (1, 2, 3)
with a scalar index
In [102]: x[1,:] # same as x[1], x[1,...]
Out[102]: array([5, 2, 0])
In [103]: x[...,1] # same as x[:,1]
Out[103]: array([1, 2])
I'm trying to concatenate 2 arrays element wise. I have the concatenation working to produce the correct shape but it has not been applied element wise.
So i have this array
[0, 1]
[2, 3]
[4, 5]
I want to append each element in the array with each element. the target result would be
[0, 1, 0, 1]
[0, 1, 2, 3]
[0, 1, 4, 5]
[2, 3, 0, 1]
[2, 3, 2, 3]
[2, 3, 4, 5]
[4, 5, 0, 1]
[4, 5, 2, 3]
[4, 5, 4, 5]
i think i may need to change an axis but then i can't get the broadcasting to work.
any help would be greatly appreciated. lots to learn in numpy !
a = np.arange(6).reshape(3, 2))
b = np.concatenate((a, a), axis=1)
One way would be stacking replicated versions created with np.repeat and np.tile -
In [52]: n = len(a)
In [53]: np.hstack((np.repeat(a,n,axis=0),np.tile(a,(n,1))))
Out[53]:
array([[0, 1, 0, 1],
[0, 1, 2, 3],
[0, 1, 4, 5],
[2, 3, 0, 1],
[2, 3, 2, 3],
[2, 3, 4, 5],
[4, 5, 0, 1],
[4, 5, 2, 3],
[4, 5, 4, 5]])
Another would be with broadcasted-assignment, since you mentioned broadcasting -
def create_mesh(a):
m,n = a.shape
out = np.empty((m,m,2*n),dtype=a.dtype)
out[...,:n] = a[:,None]
out[...,n:] = a
return out.reshape(-1,2*n)
One solution is to build on senderle's cartesian_product to extend this to 2D arrays. Here's how I usually do this:
# Your input array.
arr
# array([[0, 1],
# [2, 3],
# [4, 5]])
idxs = cartesian_product(*[np.arange(len(arr))] * 2)
arr[idxs].reshape(idxs.shape[0], -1)
# array([[0, 1, 0, 1],
# [0, 1, 2, 3],
# [0, 1, 4, 5],
# [2, 3, 0, 1],
# [2, 3, 2, 3],
# [2, 3, 4, 5],
# [4, 5, 0, 1],
# [4, 5, 2, 3],
# [4, 5, 4, 5]])
Suppose I have a numpy array as follows:
data = np.array([[1, 3, 8, np.nan], [np.nan, 6, 7, 9], [np.nan, 0, 1, 2], [5, np.nan, np.nan, 2]])
I would like to randomly select n-valid items from the array, including their indices.
Does numpy provide an efficient way of doing this?
Example
data = np.array([[1, 3, 8, np.nan], [np.nan, 6, 7, 9], [np.nan, 0, 1, 2], [5, np.nan, np.nan, 2]])
n = 5
Get valid indices
y_val, x_val = np.where(~np.isnan(data))
n_val = y_val.size
Pick random subset of size n by index
pick = np.random.choice(n_val, n)
Apply index to valid coordinates
y_pick, x_pick = y_val[pick], x_val[pick]
Get corresponding data
data_pick = data[y_pick, x_pick]
Admire
data_pick
# array([2., 8., 1., 1., 2.])
y_pick
# array([3, 0, 0, 2, 3])
x_pick
# array([3, 2, 0, 2, 3])
Find nonzeros by :
In [37]: a = np.array(np.nonzero(data)).reshape(-1,2)
In [38]: a
Out[38]:
array([[0, 0],
[0, 0],
[1, 1],
[1, 1],
[2, 2],
[2, 3],
[3, 3],
[3, 0],
[1, 2],
[3, 0],
[1, 2],
[3, 0],
[2, 3],
[0, 1],
[2, 3]])
Now pick a random choice :
In [44]: idx = np.random.choice(np.arange(len(a)))
In [45]: data[a[idx][0],a[idx][1]]
Out[45]: 2.0
For example:
array = [[1, 2, 3], [4, 5, 6]]
slice = [[0, 0, 1], [0, 1, 2]]
output = [[1, 1, 2], [4, 5,6]]
I've tried array[slice], but that didn't work. I also couldn't get tf.gather or tf.gather_nd to work, although these initially seemed like the correct functions to use. Note that these are all tensors in-graph.
How can I select these values in my array according to slice?
You need to add a dimension to your slice tensor which you can do with tf.pack and then we can use tf.gather_nd no problem.
import tensorflow as tf
tensor = tf.constant([[1, 2, 3], [4, 5, 6]])
old_slice = tf.constant([[0, 0, 1], [0, 1, 2]])
# We need to add a dimension - we need a tensor of rank 2, 3, 2 instead of 2, 3
dims = tf.constant([[0, 0, 0], [1, 1, 1]])
new_slice = tf.pack([dims, old_slice], 2)
out = tf.gather_nd(tensor, new_slice)
If we run the follow code:
with tf.Session() as sess:
sess.run(tf.initialize_all_variables())
run_tensor, run_slice, run_out = sess.run([tensor, new_slice, out])
print 'Input tensor:'
print run_tensor
print 'Correct param for gather_nd:'
print run_slice
print 'Output:'
print run_out
This should give the correct output:
Input tensor:
[[1 2 3]
[4 5 6]]
Correct param for gather_nd:
[[[0 0]
[0 0]
[0 1]]
[[1 0]
[1 1]
[1 2]]]
Output:
[[1 1 2]
[4 5 6]]
An even easier way to calculate the results, which is also of more general nature, is to directly leverage the batch_dims argument of tf.gather:
>>> array = tf.constant([[1,2,3], [4,5,6]])
>>> slice = tf.constant([[0,0,1], [0,1,2]])
>>> output = tf.constant([[1,1,2], [4,5,6]])
>>> tf.gather(array, slice, batch_dims=1, axis=1)
<tf.Tensor: shape=(2, 3), dtype=int32, numpy=
array([[1, 1, 2],
[4, 5, 6]], dtype=int32)>