I have a numpy array A as follows:
array([[1, 2, 3],
[4, 5, 6],
[7, 8, 9]])
and another numpy array column_indices_to_be_deleted as follows:
array([1, 0, 2])
I want to delete the element from every row of A specified by the column indices in column_indices_to_be_deleted. So, column index 1 from row 0, column index 0 from row 1 and column index 2 from row 2 in this case, to get a new array that looks like this:
array([[1, 3],
[5, 6],
[7, 8]])
What would be the simplest way of doing that?
One way with masking created with broadcatsed-comparison -
In [43]: a # input array
Out[43]:
array([[1, 2, 3],
[4, 5, 6],
[7, 8, 9]])
In [44]: remove_idx # indices to be removed from each row
Out[44]: array([1, 0, 2])
In [45]: n = a.shape[1]
In [46]: a[remove_idx[:,None]!=np.arange(n)].reshape(-1,n-1)
Out[46]:
array([[1, 3],
[5, 6],
[7, 8]])
Another mask based approach with the mask created with array-assignment -
In [47]: mask = np.ones(a.shape,dtype=bool)
In [48]: mask[np.arange(len(remove_idx)), remove_idx] = 0
In [49]: a[mask].reshape(-1,a.shape[1]-1)
Out[49]:
array([[1, 3],
[5, 6],
[7, 8]])
Another with np.delete -
In [64]: m,n = a.shape
In [66]: np.delete(a.flat,remove_idx+n*np.arange(m)).reshape(m,-1)
Out[66]:
array([[1, 3],
[5, 6],
[7, 8]])
Related
I have two numpy matrices (6 rows and 3 columns) :
a = np.array([[1,2,4],[3,6,2],[3,4,7],[9,7,7],[6,3,1],[3,5,9]])
b = np.array([[4,5,2],[9,2,5],[1,5,6],[4,5,6],[1,2,6],[6,4,3]])
a = array([[1, 2, 4],
[3, 6, 2],
[3, 4, 7],
[9, 7, 7],
[6, 3, 1],
[3, 5, 9]])
b = array([[4, 5, 2],
[9, 2, 5],
[1, 5, 6],
[4, 5, 6],
[1, 2, 6],
[6, 4, 3]])
I would like to calculate the pearson correlation coefficient between the first column of a and b, the second column of a and b and the third column of a and b.
The result would be a vector of 3 (3 correlation coeff).
One way using numpy.corrcoef and diagonal:
corr = np.corrcoef(a.T, b.T).diagonal(a.shape[1])
corr
Output:
array([-0.2324843 , -0.03631365, -0.18057878])
I'm new in numpy, I understand the methods of "Joining arrays" in lower shape such as (n1, n2) beacause we can visualize, like a matrix.
But I don't undestand the logic in higher dimensions (n0, ...., n_{d-1}) of course I can't visualize that. To visualize I usually imagine a multidimensional array like a tree, so (n0, ...., n_{d-1}) means that at level (axis) i of tree every node has n_{i} children. So at level 0 (the root) we have n0 children and so on.
In substance what is the formal exact definiton of "Joining arrays" algorithms?
https://numpy.org/doc/stable/reference/routines.array-manipulation.html
Let's see I can illustrate some basic array operations.
First make a 2d array. Start with a 1d, [0,1,...5], and reshape it to (2,3):
In [1]: x = np.arange(6).reshape(2,3)
In [2]: x
Out[2]:
array([[0, 1, 2],
[3, 4, 5]])
I can join 2 copies of x along the 1st dimension (vstack, v for vertical also does this):
In [3]: np.concatenate([x,x], axis=0)
Out[3]:
array([[0, 1, 2],
[3, 4, 5],
[0, 1, 2],
[3, 4, 5]])
Note that the result is (4,3); no new dimension.
Or join them 'horizontally':
In [4]: np.concatenate([x,x], axis=1)
Out[4]:
array([[0, 1, 2, 0, 1, 2], # (2,6) shape
[3, 4, 5, 3, 4, 5]])
But if I supply them to np.array I make a 3d array (2,2,3) shape:
In [5]: np.array([x,x])
Out[5]:
array([[[0, 1, 2],
[3, 4, 5]],
[[0, 1, 2],
[3, 4, 5]]])
This action of np.array is really no different from making a 2d array from nested lists, np.array([[1,2],[3,4]]). We could just add a layer of nesting, just like Out[5} without the line breaks. I tend to think of this 3d array as having 2 blocks, each with 2 rows and 3 columns. But the names are just a convenience.
stack acts like np.array, making a 3d array. It actually changes the input arrays to (1,2,3) shape, and concatenates on the first axis.
In [6]: np.stack([x,x])
Out[6]:
array([[[0, 1, 2],
[3, 4, 5]],
[[0, 1, 2],
[3, 4, 5]]])
stack lets us join the array in other ways
In [7]: np.stack([x,x], axis=1) # expand to (2,1,3) and concatante
Out[7]:
array([[[0, 1, 2],
[0, 1, 2]],
[[3, 4, 5],
[3, 4, 5]]])
In [8]: np.stack([x,x], axis=2) # expand to (2,3,1) and concatenate
Out[8]:
array([[[0, 0],
[1, 1],
[2, 2]],
[[3, 3],
[4, 4],
[5, 5]]])
concatenate and the other stack functions don't add anything new to basic numpy arrays. They just provide a way(s) of making a new array from existing ones. There aren't any special algorithms.
If it helps you could think of these join functions as creating a new "blank" array, and filling it with copies of the source arrays. For example that last stack can be done with:
In [9]: res = np.zeros((2,3,2), int)
In [10]: res
Out[10]:
array([[[0, 0],
[0, 0],
[0, 0]],
[[0, 0],
[0, 0],
[0, 0]]])
In [11]: res[:,:,0] = x
In [12]: res[:,:,1] = x
In [13]: res
Out[13]:
array([[[0, 0],
[1, 1],
[2, 2]],
[[3, 3],
[4, 4],
[5, 5]]])
I want to delete all row after the second row, however, when I try to apply the following code, the function delete only the third and the 5th rows and keep the forth any idea on how to improve this without doing a loop
arr1 = array([[1, 2, 3], [4, 5, 6], [7, 8, 9]])
arr2 = array([[10, 11, 12], [13, 14, 15]])
arr1 = concatenate((arr1, arr2), axis=0)
print(arr1)
print(delete(arr1, (2, 4), axis=0))
If the data you want to delete is contiguous (like in your example), using numpy's array indexing is arguably the easiest way to achieve what you want.
import numpy as np
arr1 = np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9]])
arr2 = np.array([[10, 11, 12], [13, 14, 15]])
arr3 = np.r_[arr1, arr2]
# First dimension corresponds to rows, second dimension corresponds to columns.
print(arr3[:2, :])
You can just pass a tuple as the second argument.
>>> Arr = array([[1, 2, 3], [4, 5, 6], [7, 8, 9]])
>>> delete(Arr, (0, 2), axis=0)
array([[4, 5, 6]])
Btw, delete(Arr, (3), axis=0) does not work in your example, since the maximum index for this array is 2.
Concerning your edit, the error is the same as above: you are using an index (3 or 5) which does not correspond to an actual index of the array. Arr[3] or Arr[5] does not make any sense for an array of shape (3,3).
I used the following code and it works well
arr1 = array([[1, 2, 3], [4, 5, 6], [7, 8, 9]])
arr2 = array([[10, 11, 12], [13, 14, 15]])
arr1 = concatenate((arr1, arr2), axis=0)
print(arr1)
print(delete(arr1, (range(2, 5)), axis=0))
Suppose I have a numpy array as follows:
data = np.array([[1, 3, 8, np.nan], [np.nan, 6, 7, 9], [np.nan, 0, 1, 2], [5, np.nan, np.nan, 2]])
I would like to randomly select n-valid items from the array, including their indices.
Does numpy provide an efficient way of doing this?
Example
data = np.array([[1, 3, 8, np.nan], [np.nan, 6, 7, 9], [np.nan, 0, 1, 2], [5, np.nan, np.nan, 2]])
n = 5
Get valid indices
y_val, x_val = np.where(~np.isnan(data))
n_val = y_val.size
Pick random subset of size n by index
pick = np.random.choice(n_val, n)
Apply index to valid coordinates
y_pick, x_pick = y_val[pick], x_val[pick]
Get corresponding data
data_pick = data[y_pick, x_pick]
Admire
data_pick
# array([2., 8., 1., 1., 2.])
y_pick
# array([3, 0, 0, 2, 3])
x_pick
# array([3, 2, 0, 2, 3])
Find nonzeros by :
In [37]: a = np.array(np.nonzero(data)).reshape(-1,2)
In [38]: a
Out[38]:
array([[0, 0],
[0, 0],
[1, 1],
[1, 1],
[2, 2],
[2, 3],
[3, 3],
[3, 0],
[1, 2],
[3, 0],
[1, 2],
[3, 0],
[2, 3],
[0, 1],
[2, 3]])
Now pick a random choice :
In [44]: idx = np.random.choice(np.arange(len(a)))
In [45]: data[a[idx][0],a[idx][1]]
Out[45]: 2.0
Is is possible to filter the data after groupby aggregation ?
I have aggregated the sum after applying groupby function, and want to see the rows where the sum is between some values.
Here is a basic code
A = pd.DataFrame([
[1, 2],
[2, 3],
[1, 6],
[2, 7],
[3, 5],
[2, 9],
[4, 7],
[3, 5],
[3, 9],
[3, 4]
], columns=['id', 'val'])
display(A)
display(A.groupby(['id']).agg({'val': ['sum', 'count']}))
I want count of val between 1 and 4 after aggregation
I dint understand if you wanted the sum between 1 and 4 or the count. So here is how i made it for the two options:
import pandas as pd
A = pd.DataFrame([
[1, 2],
[2, 3],
[1, 6],
[2, 7],
[3, 5],
[2, 9],
[4, 7],
[3, 5],
[3, 9],
[3, 4],
[1,2],
[1,2],
[1,2],
[1,2],
[1,2],
], columns=['id', 'val'])
s = A.groupby(['id']).agg({'val': ['sum', 'count']})
# If you want the count
s[(s['val']['count']<=4) & (s['val']['count']>=1)]
# If you want the sum
s[(s['val']['sum']<=4) & (s['sum']['count']>=1)]