Im trying to solve for x1 x2 x3 and x4 for this matrix but I keep getting errors.
Matrix A contains all the coefficients for x1 x2 x3 x4 respectively and Matrix B contains what it is equal to.
I wrote the following code which in theory should work but it keeps saying I provided 5 arguments or something like that
import numpy as np
a = np.matrix([2, 5, 6, 4], [5, 10, 9, 5], [7, 17.5, 21, 14], [0, 0, 2, 5])
b = np.matrix([23.5, 34, 82.25, -13])
x = np.linalg.solve(a,b)
print(x)
I shouldn't have to do this, since you should show the full traceback with the error:
In [396]: a = np.matrix([2, 5, 6, 4], [5, 10, 9, 5], [7, 17.5, 21, 14], [0, 0,
...: 2, 5])
...: b = np.matrix([23.5, 34, 82.25, -13])
...:
...: x = np.linalg.solve(a,b)
Traceback (most recent call last):
File "<ipython-input-396-710e1fc00100>", line 1, in <module>
a = np.matrix([2, 5, 6, 4], [5, 10, 9, 5], [7, 17.5, 21, 14], [0, 0, 2, 5])
TypeError: __new__() takes from 2 to 4 positional arguments but 5 were given
Look at that error message! See the np.matrix? Now go to np.matrix docs, and you'll see that the you need to provide ONE list of lists. And extra lists are interpreted as added arguments.
Thus you should use: (note the added [] - they are important.
In [397]: a = np.matrix([[2, 5, 6, 4], [5, 10, 9, 5], [7, 17.5, 21, 14], [0, 0,
...: 2, 5]])
...: b = np.matrix([23.5, 34, 82.25, -13])
...:
...: x = np.linalg.solve(a,b)
Traceback (most recent call last):
File "<ipython-input-397-b90e1785a311>", line 4, in <module>
x = np.linalg.solve(a,b)
File "<__array_function__ internals>", line 180, in solve
File "/usr/local/lib/python3.8/dist-packages/numpy/linalg/linalg.py", line 393, in solve
r = gufunc(a, b, signature=signature, extobj=extobj)
ValueError: solve: Input operand 1 has a mismatch in its core dimension 0, with gufunc signature (m,m),(m,n)->(m,n) (size 1 is different from 4)
In [398]: a.shape
Out[398]: (4, 4)
In [399]: b.shape
Out[399]: (1, 4)
Note the shape of b. solve doesn't like that mix of shapes. A (4,1) would probably work. But since we looked at np.matrix docs, lets follow its recommendations, and switch to np.array:
In [400]: a = np.array([[2, 5, 6, 4], [5, 10, 9, 5], [7, 17.5, 21, 14], [0, 0,
...: 2, 5]])
...: b = np.array([23.5, 34, 82.25, -13])
...:
...: x = np.linalg.solve(a,b)
Traceback (most recent call last):
File "<ipython-input-400-b1b6c06db25c>", line 4, in <module>
x = np.linalg.solve(a,b)
File "<__array_function__ internals>", line 180, in solve
File "/usr/local/lib/python3.8/dist-packages/numpy/linalg/linalg.py", line 393, in solve
r = gufunc(a, b, signature=signature, extobj=extobj)
File "/usr/local/lib/python3.8/dist-packages/numpy/linalg/linalg.py", line 88, in _raise_linalgerror_singular
raise LinAlgError("Singular matrix")
LinAlgError: Singular matrix
In [401]: a
Out[401]:
array([[ 2. , 5. , 6. , 4. ],
[ 5. , 10. , 9. , 5. ],
[ 7. , 17.5, 21. , 14. ],
[ 0. , 0. , 2. , 5. ]])
In [402]: np.linalg.det(a)
Out[402]: 0.0
I assume you know enough linear algebra to understand that problem, and undertake your own fix.
Related
I am trying to access a pytorch tensor by a matrix of indices and I recently found this bit of code that I cannot find the reason why it is not working.
The code below is split into two parts. The first half proves to work, whilst the second trips an error. I fail to see the reason why. Could someone shed some light on this?
import torch
import numpy as np
a = torch.rand(32, 16)
m, n = a.shape
xx, yy = np.meshgrid(np.arange(m), np.arange(m))
result = a[xx] # WORKS for a torch.tensor of size M >= 32. It doesn't work otherwise.
a = torch.rand(16, 16)
m, n = a.shape
xx, yy = np.meshgrid(np.arange(m), np.arange(m))
result = a[xx] # IndexError: too many indices for tensor of dimension 2
and if I change a = np.random.rand(16, 16) it does work as well.
To whoever comes looking for an answer: it looks like its a bug in pyTorch.
Indexing using numpy arrays is not well defined, and it works only if tensors are indexed using tensors. So, in my example code, this works flawlessly:
a = torch.rand(M, N)
m, n = a.shape
xx, yy = torch.meshgrid(torch.arange(m), torch.arange(m), indexing='xy')
result = a[xx] # WORKS
I made a gist to check it, and it's available here
First, let me give you a quick insight into the idea of indexing a tensor with a numpy array and another tensor.
Example: this is our target tensor to be indexed
numpy_indices = torch.tensor([[0, 1, 2, 7],
[0, 1, 2, 3]]) # numpy array
tensor_indices = torch.tensor([[0, 1, 2, 7],
[0, 1, 2, 3]]) # 2D tensor
t = torch.tensor([[1, 2, 3, 4], # targeted tensor
[5, 6, 7, 8],
[9, 10, 11, 12],
[13, 14, 15, 16],
[17, 18, 19, 20],
[21, 22, 23, 24],
[25, 26, 27, 28],
[29, 30, 31, 32]])
numpy_result = t[numpy_indices]
tensor_result = t[tensor_indices]
Indexing using a 2D numpy array: the index is read like pairs (x,y) tensor[row,column] e.g. t[0,0], t[1,1], t[2,2], and t[7,3].
print(numpy_result) # tensor([ 1, 6, 11, 32])
Indexing using a 2D tensor: walks through the index tensor in a row-wise manner and each value is an index of a row in the targeted tensor.
e.g. [ [t[0],t[1],t[2],[7]] , [[0],[1],[2],[3]] ] see the example below, the new shape of tensor_result after indexing is (tensor_indices.shape[0],tensor_indices.shape[1],t.shape[1])=(2,4,4).
print(tensor_result) # tensor([[[ 1, 2, 3, 4],
# [ 5, 6, 7, 8],
# [ 9, 10, 11, 12],
# [29, 30, 31, 32]],
# [[ 1, 2, 3, 4],
# [ 5, 6, 7, 8],
# [ 9, 10, 11, 12],
# [ 13, 14, 15, 16]]])
If you try to add a third row in numpy_indices, you will get the same error you have because the index will be represented by 3D e.g., (0,0,0)...(7,3,3).
indices = np.array([[0, 1, 2, 7],
[0, 1, 2, 3],
[0, 1, 2, 3]])
print(numpy_result) # IndexError: too many indices for tensor of dimension 2
However, this is not the case with indexing by tensor and the shape will be bigger (3,4,4).
Finally, as you see the outputs of the two types of indexing are completely different. To solve your problem, you can use
xx = torch.tensor(xx).long() # convert a numpy array to a tensor
What happens in the case of advanced indexing (rows of numpy_indices > 3 ) as your situation is still ambiguous and unsolved and you can check 1 , 2, 3.
for example I got many sub-arrays by splitting one array A based on list B:
A = np.array([[1,1,1],
[2,2,2],
[2,3,4],
[5,8,10],
[5,9,9],
[7,9,6],
[1,1,1],
[2,2,2],
[9,2,4],
[9,3,6],
[10,3,3],
[11,2,2]])
B = np.array([5,7])
C = np.split(A,B.cumsum()[:-1])
>>>print(C)
>>>array([[1,1,1],
[1,2,2],
[2,3,4],
[5,8,10],
[5,9,9]]),
array([[7,9,6],
[1,1,1],
[2,2,2],
[9,2,4],
[9,3,6],
[10,3,3],
[11,2,2]])
How can I find get the rows only appeared once in all the sub-arrays (delete those who appeared twice)? so that I can get the result like: (because [1,1,1] and [2,2,2] appeared twice in C )
>>>array([[2,3,4],
[5,8,10],
[5,9,9]]),
array([[7,9,6],
[9,2,4],
[9,3,6],
[10,3,3],
[11,2,2]])
You can use np.unique to identify the duplicates:
_, i, c = np.unique(A, axis=0, return_index=True, return_counts=True)
idx = np.isin(np.arange(len(A)), i[c==1])
out = [a[i] for a,i in zip(np.split(A, B.cumsum()[:-1]),
np.split(idx, B.cumsum()[:-1]))]
output:
[array([[ 2, 3, 4],
[ 5, 8, 10],
[ 5, 9, 9]]),
array([[ 7, 9, 6],
[ 9, 2, 4],
[ 9, 3, 6],
[10, 3, 3],
[11, 2, 2]])]
For Python iterables, sum() is applicable to append multiple slices from left to right.
import numpy as np
_list = list(range(15))
print("iterables is {}".format(_list))
print(sum(
[ _list[_slice] for _slice in np.s_[1:3, 5:7, 9:11] ],
start=[]
))
---
List is [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14]
[1, 2, 5, 6, 9, 10]
It cannot be simply apply to numpy array.
import numpy as np
_list = np.arange(15)
print("List is {}\n".format(_list))
print(sum(
[ _list[_slice] for _slice in np.s_[1:3, 5:7, 9:11] ],
start=[]
))
---
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
<ipython-input-66-a9d278e659c8> in <module>
3 print("List is {}\n".format(_list))
4
----> 5 print(sum(
6 [ _list[_slice] for _slice in np.s_[1:3, 5:7, 9:11] ],
7 start=[]
ValueError: operands could not be broadcast together with shapes (0,) (2,)
I suppose numpy way is something like below.
import numpy as np
a = np.arange(15).astype(np.int32)
print("array is {}\n".format(a))
print([a[_slice] for _slice in slices])
np.concatenate([a[_slice] for _slice in slices])
---
array is [ 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14]
[array([1, 2], dtype=int32), array([5, 6], dtype=int32), array([ 9, 10], dtype=int32)]
array([ 1, 2, 5, 6, 9, 10], dtype=int32)
Question
Is there a way to be able to apply sum(). Is there better way other than np.concatenate?
In [38]: np.s_[1:3, 5:7, 9:11]
Out[38]: (slice(1, 3, None), slice(5, 7, None), slice(9, 11, None))
np.r_ can make a composite index - basically a concatenate of aranges:
In [39]: np.r_[1:3, 5:7, 9:11]
Out[39]: array([ 1, 2, 5, 6, 9, 10])
Alternatively, create the slice objects, index and concatenate:
In [40]: x = np.s_[1:3, 5:7, 9:11]
In [41]: y = np.arange(20)
In [42]: np.concatenate([y[s] for s in x])
Out[42]: array([ 1, 2, 5, 6, 9, 10])
When I looked at this in the past, performance is similar.
Ways of creating the indices with list join:
In [46]: list(range(1,3))+list(range(5,7))+list(range(9,11))
Out[46]: [1, 2, 5, 6, 9, 10]
In [50]: sum([list(range(i,j)) for i,j in [(1,3),(5,7),(9,11)]],start=[])
Out[50]: [1, 2, 5, 6, 9, 10]
sum(..., start=[]) is just a list way of concatenating, using the + definition for lists.
In [55]: alist = []
In [56]: for i,j in [(1,3),(5,7),(9,11)]: alist.extend(range(i,j))
In [57]: alist
Out[57]: [1, 2, 5, 6, 9, 10]
Lets say I have a Python Numpy array a.
a = numpy.array([1,2,3,4,5,6,7,8,9,10,11])
I want to create a matrix of sub sequences from this array of length 5 with stride 3. The results matrix hence will look as follows:
numpy.array([[1,2,3,4,5],[4,5,6,7,8],[7,8,9,10,11]])
One possible way of implementing this would be using a for-loop.
result_matrix = np.zeros((3, 5))
for i in range(0, len(a), 3):
result_matrix[i] = a[i:i+5]
Is there a cleaner way to implement this in Numpy?
Approach #1 : Using broadcasting -
def broadcasting_app(a, L, S ): # Window len = L, Stride len/stepsize = S
nrows = ((a.size-L)//S)+1
return a[S*np.arange(nrows)[:,None] + np.arange(L)]
Approach #2 : Using more efficient NumPy strides -
def strided_app(a, L, S ): # Window len = L, Stride len/stepsize = S
nrows = ((a.size-L)//S)+1
n = a.strides[0]
return np.lib.stride_tricks.as_strided(a, shape=(nrows,L), strides=(S*n,n))
Sample run -
In [143]: a
Out[143]: array([ 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11])
In [144]: broadcasting_app(a, L = 5, S = 3)
Out[144]:
array([[ 1, 2, 3, 4, 5],
[ 4, 5, 6, 7, 8],
[ 7, 8, 9, 10, 11]])
In [145]: strided_app(a, L = 5, S = 3)
Out[145]:
array([[ 1, 2, 3, 4, 5],
[ 4, 5, 6, 7, 8],
[ 7, 8, 9, 10, 11]])
Starting in Numpy 1.20, we can make use of the new sliding_window_view to slide/roll over windows of elements.
And coupled with a stepping [::3], it simply becomes:
from numpy.lib.stride_tricks import sliding_window_view
# values = np.array([1,2,3,4,5,6,7,8,9,10,11])
sliding_window_view(values, window_shape = 5)[::3]
# array([[ 1, 2, 3, 4, 5],
# [ 4, 5, 6, 7, 8],
# [ 7, 8, 9, 10, 11]])
where the intermediate result of the sliding is:
sliding_window_view(values, window_shape = 5)
# array([[ 1, 2, 3, 4, 5],
# [ 2, 3, 4, 5, 6],
# [ 3, 4, 5, 6, 7],
# [ 4, 5, 6, 7, 8],
# [ 5, 6, 7, 8, 9],
# [ 6, 7, 8, 9, 10],
# [ 7, 8, 9, 10, 11]])
Modified version of #Divakar's code with checking to ensure that memory is contiguous and that the returned array cannot be modified. (Variable names changed for my DSP application).
def frame(a, framelen, frameadv):
"""frame - Frame a 1D array
a - 1D array
framelen - Samples per frame
frameadv - Samples between starts of consecutive frames
Set to framelen for non-overlaping consecutive frames
Modified from Divakar's 10/17/16 11:20 solution:
https://stackoverflow.com/questions/40084931/taking-subarrays-from-numpy-array-with-given-stride-stepsize
CAVEATS:
Assumes array is contiguous
Output is not writable as there are multiple views on the same memory
"""
if not isinstance(a, np.ndarray) or \
not (a.flags['C_CONTIGUOUS'] or a.flags['F_CONTIGUOUS']):
raise ValueError("Input array a must be a contiguous numpy array")
# Output
nrows = ((a.size-framelen)//frameadv)+1
oshape = (nrows, framelen)
# Size of each element in a
n = a.strides[0]
# Indexing in the new object will advance by frameadv * element size
ostrides = (frameadv*n, n)
return np.lib.stride_tricks.as_strided(a, shape=oshape,
strides=ostrides, writeable=False)
Is it possible to look up entries from an nd array without throwing an IndexError?
I'm hoping for something like:
>>> a = np.arange(10) * 2
>>> a[[-4, 2, 8, 12]]
IndexError
>>> wrap(a, default=-1)[[-4, 2, 8, 12]]
[-1, 4, 16, -1]
>>> wrap(a, default=-1)[200]
-1
Or possibly more like get_with_default(a, [-4, 2, 8, 12], default=-1)
Is there some builtin way to do this? Can I ask numpy not to throw the exception and return garbage, which I can then replace with my default value?
np.take with clip mode, sort of does this
In [155]: a
Out[155]: array([ 0, 2, 4, 6, 8, 10, 12, 14, 16, 18])
In [156]: a.take([-4,2,8,12],mode='raise')
...
IndexError: index 12 is out of bounds for size 10
In [157]: a.take([-4,2,8,12],mode='wrap')
Out[157]: array([12, 4, 16, 4])
In [158]: a.take([-4,2,8,12],mode='clip')
Out[158]: array([ 0, 4, 16, 18])
Except you don't have much control over the return value - here indexing on 12 return 18, the last value. And treated the -4 as out of bounds in the other direction, returning 0.
One way of adding the defaults is to pad a first
In [174]: a = np.arange(10) * 2
In [175]: ind=np.array([-4,2,8,12])
In [176]: np.pad(a, [1,1], 'constant', constant_values=-1).take(ind+1, mode='clip')
Out[176]: array([-1, 4, 16, -1])
Not exactly pretty, but a start.
This is my first post on any stack exchange site so forgive me for any stylistic errors (hopefully there are only stylistic errors). I am interested in the same feature but could not find anything from numpy better than np.take mentioned by hpaulj. Still np.take doesn't do exactly what's needed. Alfe's answer works but would need some elaboration in order to handle n-dimensional inputs. The following is another workaround that generalizes to the n-dimensional case. The basic idea is similar the one used by Alfe: create a new index with the out of bounds indices masked out (in my case) or disguised (in Alfe's case) and use it to index the input array without raising an error.
def take(a,indices,default=0):
#initialize mask; will broadcast to length of indices[0] in first iteration
mask = True
for i,ind in enumerate(indices):
#each element of the mask is only True if all indices at that position are in bounds
mask = mask & (0 <= ind) & (ind < a.shape[i])
#create in_bound indices
in_bound = [ind[mask] for ind in indices]
#initialize result with default value
result = default * np.ones(len(mask),dtype=a.dtype)
#set elements indexed by in_bound to their appropriate values in a
result[mask] = a[tuple(in_bound)]
return result
And here is the output from Eric's sample problem:
>>> a = np.arange(10)*2
>>> indices = (np.array([-4,2,8,12]),)
>>> take(a,indices,default=-1)
array([-1, 4, 16, -1])
You can restrict the range of the indexes to the size of your value array you want to index in using np.maximum() and np.minimum().
Example:
I have a heatmap like
h = np.array([[ 2, 3, 1],
[ 3, -1, 5]])
and I have a palette of RGB values I want to use to color the heatmap. The palette only names colors for the values 0..4:
p = np.array([[0, 0, 0], # black
[0, 0, 1], # blue
[1, 0, 1], # purple
[1, 1, 0], # yellow
[1, 1, 1]]) # white
Now I want to color my heatmap using the palette:
p[h]
Currently this leads to an error because of the values -1 and 5 in the heatmap:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
IndexError: index 5 is out of bounds for axis 0 with size 5
But I can limit the range of the heatmap:
p[np.maximum(np.minimum(h, 4), 0)]
This works and gives me the result:
array([[[1, 0, 1],
[1, 1, 0],
[0, 0, 1]],
[[1, 1, 0],
[0, 0, 0],
[1, 1, 1]]])
If you really need to have a special value for the indexes which are out of bound, you could implement your proposed get_with_default() like this:
def get_with_default(values, indexes, default=-1):
return np.concatenate([[default], values, [default]])[
np.maximum(np.minimum(indexes, len(values)), -1) + 1]
a = np.arange(10) * 2
get_with_default(a, [-4, 2, 8, 12], default=-1)
Will return:
array([-1, 4, 16, -1])
as wanted.