Related
I'm following this example on doc
In [42]: x = torch.tensor([1,2,3])
In [45]: x.repeat(4,2)
Out[45]: tensor([[1, 2, 3, 1, 2, 3],
[1, 2, 3, 1, 2, 3],
[1, 2, 3, 1, 2, 3],
[1, 2, 3, 1, 2, 3]])
In [46]: x.repeat(4,2).shape
Out[46]: torch.Size([4, 6])
So far, so good.
But why does repeating just 1 time on 3rd dimension expand 3rd dim to 3 (not 1)?
[On the doc]
>>> x.repeat(4, 2, 1).size()
torch.Size([4, 2, 3])
Double checking.
In [43]: x.repeat(4,2,1)
Out[43]:
tensor([[[1, 2, 3],
[1, 2, 3]],
[[1, 2, 3],
[1, 2, 3]],
[[1, 2, 3],
[1, 2, 3]],
[[1, 2, 3],
[1, 2, 3]]])
Why does it behave this way?
It expands the size([3]) tensor it only once along first dim. The (4,2,1) is the number of times you want to repeat a (3,) tensor. So, the final tensor is (4,2,3), because you repeat the (3,) once over last axis, twice over second last and 4 times over the first axis.
x = torch.tensor([1, 2, 3])
x.shape
torch.Size([3])
Then,
xx = x.repeat(4,2,1)
xx.shape
torch.Size([4, 2, 3])
I'm trying to concatenate 2 arrays element wise. I have the concatenation working to produce the correct shape but it has not been applied element wise.
So i have this array
[0, 1]
[2, 3]
[4, 5]
I want to append each element in the array with each element. the target result would be
[0, 1, 0, 1]
[0, 1, 2, 3]
[0, 1, 4, 5]
[2, 3, 0, 1]
[2, 3, 2, 3]
[2, 3, 4, 5]
[4, 5, 0, 1]
[4, 5, 2, 3]
[4, 5, 4, 5]
i think i may need to change an axis but then i can't get the broadcasting to work.
any help would be greatly appreciated. lots to learn in numpy !
a = np.arange(6).reshape(3, 2))
b = np.concatenate((a, a), axis=1)
One way would be stacking replicated versions created with np.repeat and np.tile -
In [52]: n = len(a)
In [53]: np.hstack((np.repeat(a,n,axis=0),np.tile(a,(n,1))))
Out[53]:
array([[0, 1, 0, 1],
[0, 1, 2, 3],
[0, 1, 4, 5],
[2, 3, 0, 1],
[2, 3, 2, 3],
[2, 3, 4, 5],
[4, 5, 0, 1],
[4, 5, 2, 3],
[4, 5, 4, 5]])
Another would be with broadcasted-assignment, since you mentioned broadcasting -
def create_mesh(a):
m,n = a.shape
out = np.empty((m,m,2*n),dtype=a.dtype)
out[...,:n] = a[:,None]
out[...,n:] = a
return out.reshape(-1,2*n)
One solution is to build on senderle's cartesian_product to extend this to 2D arrays. Here's how I usually do this:
# Your input array.
arr
# array([[0, 1],
# [2, 3],
# [4, 5]])
idxs = cartesian_product(*[np.arange(len(arr))] * 2)
arr[idxs].reshape(idxs.shape[0], -1)
# array([[0, 1, 0, 1],
# [0, 1, 2, 3],
# [0, 1, 4, 5],
# [2, 3, 0, 1],
# [2, 3, 2, 3],
# [2, 3, 4, 5],
# [4, 5, 0, 1],
# [4, 5, 2, 3],
# [4, 5, 4, 5]])
A= np.random.randint(5, size=(25, 4, 4))
U= np.unique(A, axis =0 )
results = np.where((A==U[0]).all(axis=-1))
Using this Where function matches individual rows, I would like to match the entire 4x4 array not just individual rows.
here are example results:
(array([ 1, 97, 97, 97, 97], dtype=int64), array([0, 0, 1, 2, 3], dtype=int64))
if all four rows were matched the results would contain the same index 4 times as its for the index 97 above, a single row was matched with the index "1".
I assume if the entire array was matched then just one index would have been returned.
An example of desired output if multiple indexes are supplied for one array:
(array([97, 97, 97, 97], dtype=int64), array([0, 1, 2, 3], dtype=int64)
np.where((A.reshape(A.shape[0],-1) == U[0].reshape(-1)).all(axis=1))
Let's see an example
>>> A = np.random.randint(5, size=(25, 4, 4))
>>> A[:3,...]
array([[[0, 2, 0, 1],
[1, 0, 3, 0],
[4, 1, 1, 2],
[0, 1, 0, 0]],
[[1, 3, 2, 3],
[2, 4, 2, 1],
[3, 3, 2, 3],
[4, 2, 1, 1]],
[[4, 0, 3, 3],
[1, 0, 4, 4],
[0, 0, 2, 3],
[4, 1, 2, 2]]])
>>> U = np.unique(A, axis=0)
>>> U[0]
array([[0, 2, 0, 1],
[1, 0, 3, 0],
[4, 1, 1, 2],
[0, 1, 0, 0]])
Now you to want find U[0] in A if I understood correctly. It's easier to match row by row, so let's reshape the 4x4 arrays into rows
>>> A.reshape(A.shape[0], -1)[:3,...]
array([[0, 2, 0, 1, 1, 0, 3, 0, 4, 1, 1, 2, 0, 1, 0, 0],
[1, 3, 2, 3, 2, 4, 2, 1, 3, 3, 2, 3, 4, 2, 1, 1],
[4, 0, 3, 3, 1, 0, 4, 4, 0, 0, 2, 3, 4, 1, 2, 2]])
>>> U[0].reshape(-1)
array([0, 2, 0, 1, 1, 0, 3, 0, 4, 1, 1, 2, 0, 1, 0, 0])
Now we can compare them with np.where but if we're not careful we'll get an elementwise comparison, so we need to use np.all(axis=1) to be sure to compare them row by row:
>>> np.where(np.all(A.reshape(25, -1) == U[0].reshape(-1), axis=1))
(array([0]),)
EDIT it just occurred to me you can use multiple axes with np.all and avoid reshaping altogether:
np.where((A == U[0]).all(axis=(1,2)))
Can someone explain to me what the second line of this code does?
objp = np.zeros((48,3), np.float32)
objp[:,:2] = np.mgrid[0:8,0:6].T.reshape(-1,2)
Can someone explain to me what exactly the np.mgrid[0:8,0:6] part of the code is doing and what exactly the T.reshape(-1,2) part of the code is doing?
Thanks and good job!
The easiest way to see these is to use smaller values for mgrid:
In [11]: np.mgrid[0:2,0:3]
Out[11]:
array([[[0, 0, 0],
[1, 1, 1]],
[[0, 1, 2],
[0, 1, 2]]])
In [12]: np.mgrid[0:2,0:3].T # (matrix) transpose
Out[12]:
array([[[0, 0],
[1, 0]],
[[0, 1],
[1, 1]],
[[0, 2],
[1, 2]]])
In [13]: np.mgrid[0:2,0:3].T.reshape(-1, 2) # reshape to an Nx2 matrix
Out[13]:
array([[0, 0],
[1, 0],
[0, 1],
[1, 1],
[0, 2],
[1, 2]])
Then objp[:,:2] = sets the 0th and 1th columns of objp to this result.
The second line creates a multi-dimensional mesh grid, transposes it, reshapes it so that it represents two columns and inserts it into the first two columns of the objp array.
Breakdown:
np.mgrid[0:8,0:6] creates the following mgrid:
>> np.mgrid[0:8,0:6]
array([[[0, 0, 0, 0, 0, 0],
[1, 1, 1, 1, 1, 1],
[2, 2, 2, 2, 2, 2],
[3, 3, 3, 3, 3, 3],
[4, 4, 4, 4, 4, 4],
[5, 5, 5, 5, 5, 5],
[6, 6, 6, 6, 6, 6],
[7, 7, 7, 7, 7, 7]],
[[0, 1, 2, 3, 4, 5],
[0, 1, 2, 3, 4, 5],
[0, 1, 2, 3, 4, 5],
[0, 1, 2, 3, 4, 5],
[0, 1, 2, 3, 4, 5],
[0, 1, 2, 3, 4, 5],
[0, 1, 2, 3, 4, 5],
[0, 1, 2, 3, 4, 5]]])
The .T transposes the matrix, and the .reshape(-1,2) then reshapes it into two a two-column array shape. These two columns are then the correct shape to replace two columns in the original array.
I have a 2-d numpy array as follows:
a = np.array([[1,5,9,13],
[2,6,10,14],
[3,7,11,15],
[4,8,12,16]]
I want to extract it into patches of 2 by 2 sizes with out repeating the elements.
The answer should exactly be the same. This can be 3-d array or list with the same order of elements as below:
[[[1,5],
[2,6]],
[[3,7],
[4,8]],
[[9,13],
[10,14]],
[[11,15],
[12,16]]]
How can do it easily?
In my real problem the size of a is (36, 72). I can not do it one by one. I want programmatic way of doing it.
Using scikit-image:
import numpy as np
from skimage.util import view_as_blocks
a = np.array([[1,5,9,13],
[2,6,10,14],
[3,7,11,15],
[4,8,12,16]])
print(view_as_blocks(a, (2, 2)))
You can achieve it with a combination of np.reshape and np.swapaxes like so -
def extract_blocks(a, blocksize, keep_as_view=False):
M,N = a.shape
b0, b1 = blocksize
if keep_as_view==0:
return a.reshape(M//b0,b0,N//b1,b1).swapaxes(1,2).reshape(-1,b0,b1)
else:
return a.reshape(M//b0,b0,N//b1,b1).swapaxes(1,2)
As can be seen there are two ways to use it - With keep_as_view flag turned off (default one) or on. With keep_as_view = False, we are reshaping the swapped-axes to a final output of 3D, while with keep_as_view = True, we will keep it 4D and that will be a view into the input array and hence, virtually free on runtime. We will verify it with a sample case run later on.
Sample cases
Let's use a sample input array, like so -
In [94]: a
Out[94]:
array([[2, 2, 6, 1, 3, 6],
[1, 0, 1, 0, 0, 3],
[4, 0, 0, 4, 1, 7],
[3, 2, 4, 7, 2, 4],
[8, 0, 7, 3, 4, 6],
[1, 5, 6, 2, 1, 8]])
Now, let's use some block-sizes for testing. Let's use a blocksize of (2,3) with the view-flag turned off and on -
In [95]: extract_blocks(a, (2,3)) # Blocksize : (2,3)
Out[95]:
array([[[2, 2, 6],
[1, 0, 1]],
[[1, 3, 6],
[0, 0, 3]],
[[4, 0, 0],
[3, 2, 4]],
[[4, 1, 7],
[7, 2, 4]],
[[8, 0, 7],
[1, 5, 6]],
[[3, 4, 6],
[2, 1, 8]]])
In [48]: extract_blocks(a, (2,3), keep_as_view=True)
Out[48]:
array([[[[2, 2, 6],
[1, 0, 1]],
[[1, 3, 6],
[0, 0, 3]]],
[[[4, 0, 0],
[3, 2, 4]],
[[4, 1, 7],
[7, 2, 4]]],
[[[8, 0, 7],
[1, 5, 6]],
[[3, 4, 6],
[2, 1, 8]]]])
Verify view with keep_as_view=True
In [20]: np.shares_memory(a, extract_blocks(a, (2,3), keep_as_view=True))
Out[20]: True
Let's check out performance on a large array and verify the virtually free runtime claim as discussed earlier -
In [42]: a = np.random.rand(2000,3000)
In [43]: %timeit extract_blocks(a, (2,3), keep_as_view=True)
1000000 loops, best of 3: 801 ns per loop
In [44]: %timeit extract_blocks(a, (2,3), keep_as_view=False)
10 loops, best of 3: 29.1 ms per loop
Here's a rather cryptic numpy one-liner to generate your 3-d array, called result1 here:
In [60]: x
Out[60]:
array([[2, 1, 2, 2, 0, 2, 2, 1, 3, 2],
[3, 1, 2, 1, 0, 1, 2, 3, 1, 0],
[2, 0, 3, 1, 3, 2, 1, 0, 0, 0],
[0, 1, 3, 3, 2, 0, 3, 2, 0, 3],
[0, 1, 0, 3, 1, 3, 0, 0, 0, 2],
[1, 1, 2, 2, 3, 2, 1, 0, 0, 3],
[2, 1, 0, 3, 2, 2, 2, 2, 1, 2],
[0, 3, 3, 3, 1, 0, 2, 0, 2, 1]])
In [61]: result1 = x.reshape(x.shape[0]//2, 2, x.shape[1]//2, 2).swapaxes(1, 2).reshape(-1, 2, 2)
result1 is like a 1-d array of 2-d arrays:
In [68]: result1.shape
Out[68]: (20, 2, 2)
In [69]: result1[0]
Out[69]:
array([[2, 1],
[3, 1]])
In [70]: result1[1]
Out[70]:
array([[2, 2],
[2, 1]])
In [71]: result1[5]
Out[71]:
array([[2, 0],
[0, 1]])
In [72]: result1[-1]
Out[72]:
array([[1, 2],
[2, 1]])
(Sorry, I don't have time at the moment to give a detailed breakdown of how it works. Maybe later...)
Here's a less cryptic version that uses a nested list comprehension. In this case, result2 is a python list of 2-d numpy arrays:
In [73]: result2 = [x[2*j:2*j+2, 2*k:2*k+2] for j in range(x.shape[0]//2) for k in range(x.shape[1]//2)]
In [74]: result2[5]
Out[74]:
array([[2, 0],
[0, 1]])
In [75]: result2[-1]
Out[75]:
array([[1, 2],
[2, 1]])