Related
How can I retrieve a column vector from a 2d array given an indicator column vector?
Suppose I have
X = np.array([[1, 4, 6],
[8, 2, 9],
[0, 3, 7],
[6, 5, 1]])
and
S = np.array([0, 2, 1, 2])
Is there an elegant way to get from X and S the result array([1, 9, 3, 1]), which is equivalent to
np.array([x[s] for x, s in zip(X, S)])
You can achieve this using np.take_along_axis:
>>> np.take_along_axis(X, S[..., None], axis=1)
array([[1],
[9],
[3],
[1]])
You need to make sure both array arguments are of the same shape (or broadcasting can be applied), hence the S[..., None] broadcasting.
Of course your can reshape the returned value with a [:, 0] slice.
>>> np.take_along_axis(X, S[..., None], axis=1)[:, 0]
array([1, 9, 3, 1])
Alternatively you can just use indexing with an arangement:
>>> X[np.arange(len(S)), S[np.arange(len(S))]]
array([1, 9, 3, 1])
I believe this is also equivalent to np.diag(X[:, S]) but with unnecessary copying...
For 2d arrays
# Mention row numbers as one list and S which is column number as other
X[[0, 1, 2, 3], S]
# more general
X[np.indices(S.shape), S]
indexing_basics
I want to delete all row after the second row, however, when I try to apply the following code, the function delete only the third and the 5th rows and keep the forth any idea on how to improve this without doing a loop
arr1 = array([[1, 2, 3], [4, 5, 6], [7, 8, 9]])
arr2 = array([[10, 11, 12], [13, 14, 15]])
arr1 = concatenate((arr1, arr2), axis=0)
print(arr1)
print(delete(arr1, (2, 4), axis=0))
If the data you want to delete is contiguous (like in your example), using numpy's array indexing is arguably the easiest way to achieve what you want.
import numpy as np
arr1 = np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9]])
arr2 = np.array([[10, 11, 12], [13, 14, 15]])
arr3 = np.r_[arr1, arr2]
# First dimension corresponds to rows, second dimension corresponds to columns.
print(arr3[:2, :])
You can just pass a tuple as the second argument.
>>> Arr = array([[1, 2, 3], [4, 5, 6], [7, 8, 9]])
>>> delete(Arr, (0, 2), axis=0)
array([[4, 5, 6]])
Btw, delete(Arr, (3), axis=0) does not work in your example, since the maximum index for this array is 2.
Concerning your edit, the error is the same as above: you are using an index (3 or 5) which does not correspond to an actual index of the array. Arr[3] or Arr[5] does not make any sense for an array of shape (3,3).
I used the following code and it works well
arr1 = array([[1, 2, 3], [4, 5, 6], [7, 8, 9]])
arr2 = array([[10, 11, 12], [13, 14, 15]])
arr1 = concatenate((arr1, arr2), axis=0)
print(arr1)
print(delete(arr1, (range(2, 5)), axis=0))
trainX.size == 43120000
trainX = trainX.reshape([-1, 28, 28, 1])
(1)Does reshape accept a list as an argment instead of a tuple?
(2)Are the following two statements equivalent?
trainX = trainX.reshape([-1, 28, 28, 1])
trainX = trainX.reshape((55000, 28, 28, 1))
Try the variations:
In [1]: np.arange(12).reshape(3,4)
Out[1]:
array([[ 0, 1, 2, 3],
[ 4, 5, 6, 7],
[ 8, 9, 10, 11]])
In [2]: np.arange(12).reshape([3,4])
Out[2]:
array([[ 0, 1, 2, 3],
[ 4, 5, 6, 7],
[ 8, 9, 10, 11]])
In [3]: np.arange(12).reshape((3,4))
Out[3]:
array([[ 0, 1, 2, 3],
[ 4, 5, 6, 7],
[ 8, 9, 10, 11]])
With the reshape method, the shape can be arguments, a tuple or a list. In the reshape function is has to be in a list or tuple, to separate them from the first array argument
In [4]: np.reshape(np.arange(12), (3,4))
Out[4]:
array([[ 0, 1, 2, 3],
[ 4, 5, 6, 7],
[ 8, 9, 10, 11]])
and yes, one -1 can be used. The total size of the reshape is fixed, so one value can be deduced from the others.
In [5]: np.arange(12).reshape(-1,4)
Out[5]:
array([[ 0, 1, 2, 3],
[ 4, 5, 6, 7],
[ 8, 9, 10, 11]])
The method documentation has this note:
Unlike the free function numpy.reshape, this method on ndarray allows
the elements of the shape parameter to be passed in as separate arguments.
For example, a.reshape(10, 11) is equivalent to
a.reshape((10, 11)).
It's a builtin function, but the signature looks like x.reshape(*shape), and it tries to be flexible as long as the values make sense.
From the numpy documentation:
newshape : int or tuple of ints
The new shape should be compatible with the original shape. If an
integer, then the result will be a 1-D array of that length. One shape
dimension can be -1. In this case, the value is inferred from the
length of the array and remaining dimensions.
So yes, -1 for one dimension is fine and your two statements are equivalent. About the tuple requirement,
>>> import numpy as np
>>> a = np.arange(9)
>>> a
array([0, 1, 2, 3, 4, 5, 6, 7, 8])
>>> a.reshape([3,3])
array([[0, 1, 2],
[3, 4, 5],
[6, 7, 8]])
>>>
So apparently a list is good as well.
I have tensor T of shape [batch_size, A] with values and tensor S of shape [batch_size] with shift parameters.
I would like to shift values in T[b] by S[b] positions to the right, the last S[b] elements of T[b] should be dropped and new elements should be set to 0.
So basically want to do something like:
for i in range(batch_size):
T[i] = zeros[:S[i]] + T[i, :A-S[i]]
Example:
For:
T = [[1, 2, 3], [4, 5, 6]]
S = [1, 2]
Return:
T' = [[0, 1, 2], [0, 0, 4]]
Is there some easy way to do it?
You can use tf.concat and tf.stack for that purpose:
T_shift = tf.zeros((batch_size, A), tf.float32)
tmp = []
for i in xrange(batch_size):
tmp.append(tf.concat([T_shift[i, :S[i, 0]],T[i, :17 - S[i,0]]], axis = 0))
T_shift = tf.stack(tmp)
If you are working in Tensorflow 2, you can use the tf.roll for that purpose:
"The elements are shifted positively (towards larger indices) by the
offset of shift along the dimension of axis. Negative shift values
will shift elements in the opposite direction. Elements that roll
passed the last position will wrap around to the first and vice versa.
Multiple shifts along multiple axes may be specified."
tf.roll(
input, shift, axis, name=None
)
# 't' is [0, 1, 2, 3, 4]
roll(t, shift=2, axis=0) ==> [3, 4, 0, 1, 2]
# shifting along multiple dimensions
# 't' is [[0, 1, 2, 3, 4], [5, 6, 7, 8, 9]]
roll(t, shift=[1, -2], axis=[0, 1]) ==> [[7, 8, 9, 5, 6], [2, 3, 4, 0, 1]]
# shifting along the same axis multiple times
# 't' is [[0, 1, 2, 3, 4], [5, 6, 7, 8, 9]]
roll(t, shift=[2, -3], axis=[1, 1]) ==> [[1, 2, 3, 4, 0], [6, 7, 8, 9, 5]]
If I have a matrix Xy that I want to split into a matrix X and an array y, I usually do this
X, y = Xy[:, :-1], Xy[:, -1]
Is there a better way to do this using scikit-learn or numpy? I feel like it's a very common operation.
You can use NumPy built-in np.split -
X, y = np.split(Xy,[-1],axis=1) # Or simply : np.split(Xy,[-1],1)
Sample run -
In [93]: Xy
Out[93]:
array([[6, 2, 0, 5, 2],
[6, 3, 7, 0, 0],
[3, 2, 3, 1, 3],
[1, 3, 7, 1, 7]])
In [94]: X, y = np.split(Xy,[-1],axis=1)
In [95]: X
Out[95]:
array([[6, 2, 0, 5],
[6, 3, 7, 0],
[3, 2, 3, 1],
[1, 3, 7, 1]])
In [96]: y
Out[96]:
array([[2],
[0],
[3],
[7]])
Note that np.split would produce y as 2D. To have a 1D slice, we need to use np.squeeze(y) there.
Also, these slices would be views into original array, so no additional memory required there -
In [104]: np.may_share_memory(Xy, X)
Out[104]: True
In [105]: np.may_share_memory(Xy, y)
Out[105]: True
np.split uses np.array_split. That in turn does:
sub_arys = []
sary = _nx.swapaxes(ary, axis, 0)
for i in range(Nsections):
st = div_points[i]
end = div_points[i + 1]
sub_arys.append(_nx.swapaxes(sary[st:end], axis, 0))
swapaxes is needed with axis=1; or without the swapping:
sub_arys = []
for ...:
sub_arys.append(ary[:, st:end])
return sub_arys
i.e. the same as:
In [388]: ary=np.arange(12).reshape(3,4)
In [389]: [ary[:,0:3], ary[:,3:4]]
Out[389]:
[array([[ 0, 1, 2],
[ 4, 5, 6],
[ 8, 9, 10]]),
array([[ 3],
[ 7],
[11]])]
split like this keeps the original number of dimensions.
Wrapping your code in a function gives something that will be as fast, if not faster:
def xysplit(ary):
return ary[:,:-1], ary[:,-1]
X, y = xysplit(ary)
produces:
array([[ 0, 1, 2],
[ 4, 5, 6],
[ 8, 9, 10]]),
array([ 3, 7, 11])
When I commented that this seems to be more common in sklearn contexts I had in mind questions like
Python ValueError: non-broadcastable output operand with shape (124,1) doesn't match the broadcast shape (124,13)
X = df_wine.iloc[:, 1:].values
y = df_wine.iloc[:, 0].values
....
X_train, X_test, y_train, y_test = train_test_split(X, y, ...
X and y are 2d and 1d arrays, pulled in this case from a columns of a pandas dataframe. train_test_split is used to split X and y into training and testing groups. If there is a special X,y splitter, it would be in the sklearn package, not numpy.
Python - NumPy array_split adds a dminesion
train_inputs = train[:,: -1]
train_outputs = train[:, -1]