I am implementing a co-occurrence matrix for an image to be able to detect the edges of an image through the change in brightness. So I made a 256x256 numpy matrix to store the co-occurrences, and then I wrote a function that turns all of the values of occurrences in the matrix to 0 if the change between them is less than a certain value like 30, ie the difference between the i and j of the matrix is less than 30 then the value inside that cell is turned into 0.
Here is the function, it takes the co-occurrence matrix and turn the values into 0.
def nullify(matrix):
for i in range (0,matrix.shape[0]):
for j in range(0,matrix.shape[1]):
if(abs(i-j)<30):
matrix[i,j]=0
return matrix
But for some reason it turn the entire matrix into 0's, the function work perfectly when I'm using a smaller matrix like a 3x3.
This is the function that I use to calculate the Cooccurrence
def calculateCooccurrence(im):
Horizontal = np.zeros((256, 256))
for i in range (0,im.size[0]):
for j in range (0,im.size[1]-1):
pixelRGB = im.getpixel((i,j))
R,G,B = pixelRGB
brightness = int(sum([R,G,B])/3)
pixelRGB1 = im.getpixel((i,j+1))
R1,G1,B1 = pixelRGB
brightness1 = int(sum([R1,G1,B1])/3)
Horizontal[brightness,brightness1]+=1
Vertical = np.zeros((256, 256))
for i in range (0,im.size[0]-1):
for j in range (0,im.size[1]):
pixelRGB = im.getpixel((i,j))
R,G,B = pixelRGB
brightness = int(sum([R,G,B])/3)
pixelRGB1 = im.getpixel((i+1,j))
R1,G1,B1 = pixelRGB
brightness1 = int(sum([R1,G1,B1])/3)
Vertical[brightness,brightness1]+=1
return Horizontal,Vertical
And this is what I do exactly
horiz,vertic=calculateCooccurrence(im2)
horizon=nullify(horiz)
there are some things to point about your code. I can't tell why your entire matrix turns into zeros, but these are things that should help you:
This might be due to formatting in stackoverflow, but your matrix is returned after the first iteration of the i-loop.
You actually are not working on the values of the matrix. You iterate over the range 0...256. This will set all values close to the diagonal to 0. Based on your text where you say that you want to detect the edges, I am not sure if this is what you actually want to do.
instead of creating a variable named difference you can also put it simply in the if-statement if(abs(i-j)<30:
Edit: Found the problem: Your code works intended, which is the problem. All the elements are on the diagonal. I just used a test image myself and found that np.sum(matrix) and np.trace(matrix) returned the same result. So when your code eliminates all elements along the diagonal, it turns all elements to 0.
Your code seems to work properly for Python 3.7.7 and NumPy 1.18.1:
>>> import numpy as np
>>> matrix = np.ones((256, 256))
>>> def nullify(matrix):
... for i in range (0,matrix.shape[0]):
... for j in range(0,matrix.shape[1]):
... if(abs(i-j)<30):
... matrix[i,j]=0
... return matrix
...
>>> nullify(matrix)
array([[0., 0., 0., ..., 1., 1., 1.],
[0., 0., 0., ..., 1., 1., 1.],
[0., 0., 0., ..., 1., 1., 1.],
...,
[1., 1., 1., ..., 0., 0., 0.],
[1., 1., 1., ..., 0., 0., 0.],
[1., 1., 1., ..., 0., 0., 0.]])
You can see that the elements far away from the diagonal remain unchanged. You should provide a minimal working example of what goes wrong in your code.
Related
How do I convert a numpy matrix of values to numpy matrix of row indexes where a certain condition is met?
Let's say
A = array([[ 0., 5., 0.],[ 0., 0., 3.],[ 0., 0., 0.]])
If there is a condition that I want to use here -- if an element is greater than 0 then replace it by row index+1, how would I do it?
So output should be,
B = array([[0., 1., 0.],[0., 0., 2.],[0., 0., 0.]])
Not sure if I am using np.where correctly. Thanks.
Using numpy.where
np.where(A>0, np.arange(1, A.shape[0]+1)[:, None], A)
array([[0., 1., 0.],
[0., 0., 2.],
[0., 0., 0.]])
Or you can use arithmetic (won't work if you have values less than 0):
(A > 0) * np.arange(1, A.shape[0]+1)[:, None]
I want to completely get rid of for loops in my code.
I have a complex numpy array stack1 of dimension OxMxN This is a stack of MxN arrays stacked in the 1st dimension. For each MxN array that we call A I want to compute the matrix multiplication:
for k in range(stack1.shape[0]):
A=stack1[k,:,:]
newstack[k,:,:]=A.dot( numpy.conj(numpy.transpose(A)) )
I tried
newstack = stack1 # np.conj(stack1.T)
but I run in an issue because the dimensions won't match
We can use einsum -
np.einsum('ijk,ilk->ijl',stack1,np.conj(stack1))
We can also use np.matmul -
np.matmul(stack1,np.conj(stack1).swapaxes(1,2))
On Python 3.x, simplifies with # operator -
stack1 # np.conj(stack1).swapaxes(1,2)
Just try to correct your for loop
a=[]
for k in range(stack1.shape[0]):
A=stack1[k,:,:]
a.append(A.dot( numpy.conj(numpy.transpose(A)) ))
np.array(a)
Out[399]:
array([[[0., 0., 0.],
[0., 0., 0.],
[0., 0., 0.]],
[[0., 0., 0.],
[0., 0., 0.],
[0., 0., 0.]]])
I am trying to understand the correct usage of cntk.metrics.classification_error() and use it to verify a batch of predictions against their ground truths.
The below toy example (based on the Python API docs):
import numpy as np
from cntk.metrics import classification_error
predictions = np.asarray([[1., 2., 3., 4.],[1., 2., 3., 4.],[1., 2., 3., 4.]], dtype=np.float32)
labels = np.asarray([[0., 0., 0., 1.],[0., 0., 0., 1.],[0., 0., 1., 0.]], dtype=np.float32)
classification_error(predictions, labels).eval()
yields the following result:
array([[ 0., 0., 1.],
[ 0., 0., 1.],
[ 0., 0., 1.]], dtype=float32)
Is there a way I can obtain a vector rather than a square matrix which appears inefficient given I would like to process a large batch?
I've tried using the axis keyword when calling classification_error(), but whether I set axis=0 or axis=1 I get an empty result.
This happens because CNTK is trying to be user-friendly and ends up being confused about the types :-) You can tell because the classification error is not even correct.
If you add a little bit of typing information it gets the semantics right.
p = C.input(4)
y = C.input(4)
classification_error(p, y).eval({p:predictions, y:labels})
array([[ 0.],
[ 0.],
[ 1.]], dtype=float32)
We will work on a fix that will prevent the confusion.
First off, I'm no mathmatician. I admit that. Yet I still need to understand how ScyPy's sparse matrices work arithmetically in order to switch from a dense NumPy matrix to a SciPy sparse matrix in an application I have to work on. The issue is memory usage. A large dense matrix will consume tons of memory.
The formula portion at issue is where a matrix is added to a scalar.
A = V + x
Where V is a square matrix (its large, say 60,000 x 60,000) and sparsely populated. x is a float.
The operation with NumPy will (if I'm not mistaken) add x to each field in V. Please let me know if I'm completely off base, and x will only be added to non-zero values in V.
With a SciPy, not all sparse matrices support the same features, like scalar addition. dok_matrix (Dictionary of Keys) supports scalar addition, but it looks like (in practice) that it's allocating each matrix entry, effectively rendering my sparse dok_matrix as a dense matrix with more overhead. (not good)
The other matrix types (CSR, CSC, LIL) don't support scalar addition.
I could try constructing a full matrix with the scalar value x, then adding that to V. I would have no problems with matrix types as they all seem to support matrix addition. However I would have to eat up a lot of memory to construct x as a matrix, and the result of the addition could end up being fully populated matrix as well.
There must be an alternative way to do this that doesn't require allocating 100% of a sparse matrix.
I'm will to accept that large amounts of memory are needed, but I thought I would seek some advice first. Thanks.
Admittedly sparse matrices aren't really in my wheelhouse, but ISTM the best way forward depends on the matrix type. If you're DOK:
>>> S = dok_matrix((5,5))
>>> S[2,3] = 10; S[4,1] = 20
>>> S.todense()
matrix([[ 0., 0., 0., 0., 0.],
[ 0., 0., 0., 0., 0.],
[ 0., 0., 0., 10., 0.],
[ 0., 0., 0., 0., 0.],
[ 0., 20., 0., 0., 0.]])
Then you could update:
>>> S.update(zip(S.keys(), np.array(S.values()) + 99))
>>> S
<5x5 sparse matrix of type '<type 'numpy.float64'>'
with 2 stored elements in Dictionary Of Keys format>
>>> S.todense()
matrix([[ 0., 0., 0., 0., 0.],
[ 0., 0., 0., 0., 0.],
[ 0., 0., 0., 109., 0.],
[ 0., 0., 0., 0., 0.],
[ 0., 119., 0., 0., 0.]])
Not particularly performant, but is O(nonzero).
OTOH, if you have something like COO, CSC, or CSR, you can modify the data attribute directly:
>>> C = S.tocoo()
>>> C
<5x5 sparse matrix of type '<type 'numpy.float64'>'
with 2 stored elements in COOrdinate format>
>>> C.data
array([ 119., 109.])
>>> C.data += 1000
>>> C
<5x5 sparse matrix of type '<type 'numpy.float64'>'
with 2 stored elements in COOrdinate format>
>>> C.todense()
matrix([[ 0., 0., 0., 0., 0.],
[ 0., 0., 0., 0., 0.],
[ 0., 0., 0., 1109., 0.],
[ 0., 0., 0., 0., 0.],
[ 0., 1119., 0., 0., 0.]])
Note that you're probably going to want to add an additional
>>> C.eliminate_zeros()
to handle the possibility that you've added a negative number and so there's now a 0 which is actually being recorded. By itself, that should work fine, but the next time you did the C.data += some_number trick, it would add somenumber to that zero you introduced.
Assume you have the following 1D-Array:
array([1,2,3,4,5])
I want to perform different (simple) calculations between each combination of numbers (such as addition, subtraction, etc.) resulting in a Matrix-type output (without duplication), i.e. for the above array, the output should be as below if we wanted to calculated the pairwise difference:
array([0,-,-,-,-],
[1,0,-,-,-],
[2,1,0,-,-],
[3,2,1,0,-],
[4,3,2,1,0])
Of course one could use brute force with two for loops but I feel like there is a better way, I just can't seem to find the right method.
For anyone interested, I managed to find a solution using pairwise_distances from scikit-learn. This will by default just calculate the absolute distance between any pair, but it is possible to supply a custom function that takes two arguments, i.e. two numbers of a pair, for more elaborate calculations. It will require a slight reshape for 1D arrays.
from sklearn.metrics import pairwise_distances
def custom_calc(x,y):
return (y-x)
a = np.array([1,2,3,4,5])
matrix = pairwise_distances(a.reshape(-1,1), metric=custom_calc)
matrix will look as follows:
array([[0., 1., 2., 3., 4.],
[1., 0., 1., 2., 3.],
[2., 1., 0., 1., 2.],
[3., 2., 1., 0., 1.],
[4., 3., 2., 1., 0.]])
Make use of numpy broadcasting to calculate the pairwise difference. Like this no loops are needed. For that to happen the operation has to be done between a row- and a column-vector of the same array.
import numpy as np
x = np.arange(1,6, dtype=np.float)
# x[:,None] adds a second axis to the array
mat = x[:,None]-x
this yields:
array([[ 0., -1., -2., -3., -4.],
[ 1., 0., -1., -2., -3.],
[ 2., 1., 0., -1., -2.],
[ 3., 2., 1., 0., -1.],
[ 4., 3., 2., 1., 0.]])