Changing numpy array with array of indices - numpy

I have an array in numpy:
A=np.zeros((5,10))
and I want to change one value in each row with ones according to another array N=np.array([7, 2, 9, 4, 5])
like: A[:,N]=1;
0 0 0 0 0 1 0 0 0 0
0 0 1 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 1
0 0 0 0 1 0 0 0 0 0
0 0 0 0 0 1 0 0 0 0
How can I do that?


Since you want to set a single element per row, you need to fancy-index the first axis using arange(5). this can be thought of as setting indices (I0[0], N[0])=(0,7), (I0[1],N[1])=(1,2), ...
I0 = np.arange(A.shape[0])
A[I0, N] = 1
A
=>
array([[ 0., 0., 0., 0., 0., 0., 0., 1., 0., 0.],
[ 0., 0., 1., 0., 0., 0., 0., 0., 0., 0.],
[ 0., 0., 0., 0., 0., 0., 0., 0., 0., 1.],
[ 0., 0., 0., 0., 1., 0., 0., 0., 0., 0.],
[ 0., 0., 0., 0., 0., 1., 0., 0., 0., 0.]])
A.nonzero()
=> (array([0, 1, 2, 3, 4]), array([7, 2, 9, 4, 5]))

Related

merge two thresholds for two 3D arrays into a list

I have the first 3D array of size (50,250,250) that includes data points (1,2,3,4,5). I set up a threshold that is 3, where the data points above should equal to 1 and below it equal to 0. the only exception is when the data points are equal to 3, it has to test the second threshold (threshold1=50) that is based on the second 3D array of size (50,250,250). my equation is how to include the two thresholds in my code! In other words, the for loop will check every datapoint in array 1 and perform the first threshold testing, if the datapoint is equal to 3, the for loop should check the counterpart of that datapoint in the second array for the second threshold testing! I have tried the below code, but the results did not make sense
res1=[]
f1=numpy.ones((250, 250))
threshold=3
threshold1=30
for i in array1:
i = i.data
ii= f1*i
ii[ii < threshold] = 0
ii[ii > threshold] = 1
res1.append(ii)
if ii[ii == threshold]:
for j in array2:
j = j.data
jj[jj < threshold1] = 0
jj[jj > threshold1] = 1
res1.append(jj)
Array1:
array([[[0., 0., 0., ..., 0., 0., 0.],
[0., 0., 0., ..., 0., 0., 0.],
[0., 0., 0., ..., 0., 0., 0.],
...,
[3., 3., 3., ..., 0., 0., 0.],
[3., 3., 3., ..., 0., 0., 0.],
[3., 3., 3., ..., 0., 0., 0.]],
[[0., 0., 0., ..., 0., 0., 1.],
[0., 0., 0., ..., 0., 0., 0.],
[0., 0., 0., ..., 0., 0., 0.],
...,
[3., 3., 3., ..., 0., 0., 0.],
[3., 3., 3., ..., 0., 0., 0.],
[3., 3., 3., ..., 0., 0., 0.]],
Array2:[[ nan, nan, nan, ..., nan,
0.9839769, 1.7042577],
[ nan, nan, nan, ..., nan,
nan, nan],
[ nan, nan, nan, ..., 3.2351596,
2.0924768, 1.7604152],
...,
[ nan, nan, nan, ..., 158.48865 ,
158.48865 , 125.888 ],
[ nan, nan, nan, ..., 158.48865 ,
158.48865 , 158.48865 ],
[ nan, nan, nan, ..., 125.88556 ,
158.48865 , 158.48865 ]],
the produced list (rest1)
`[array([[0., 0., 0., ..., 0., 0., 0.],
[0., 0., 0., ..., 0., 0., 0.],
[0., 0., 0., ..., 0., 0., 0.],
...,
[1., 1., 1., ..., 0., 0., 0.],
[1., 1., 1., ..., 0., 0., 0.],
[1., 1., 1., ..., 0., 0., 0.]]),
array([[0., 0., 0., ..., 0., 0., 0.],
[0., 0., 0., ..., 0., 0., 0.],
[0., 0., 0., ..., 0., 0., 0.],
...,
[1., 1., 1., ..., 0., 0., 0.],
[1., 1., 1., ..., 0., 0., 0.],
[1., 1., 1., ..., 0., 0., 0.]]),
array([[0., 0., 0., ..., 0., 0., 0.],
[0., 0., 0., ..., 0., 0., 0.],
[0., 0., 0., ..., 0., 0., 0.],`

IIUC, for your second if condition, you are trying to see whether there is at least a 3 value in that array1, and then you will choose that 2D array of the same position. In that case, you should use in operator.
for i in range(len(array1)):
if threshold in array1[i]:
array2[i][array2[i] < threshold1] = 0
array2[i][array2[i] > threshold1] = 1
res1.append(array2[i])
else:
array1[i][array1[i] < threshold] = 0
array1[i][array1[i] > threshold] = 1
res1.append(array1[i])
The above method is a bit lengthy for numpy. There's a numpy way to do this, too.
array1[array1 < threshold] = 0
array1[array1 > threshold] = 1
array2_condition = np.unique(np.argwhere(array1 == 3)[:,0]) # return the index of array1 if 3 in array1
chosen_array2 = array2[array2_condition]
chosen_array2[chosen_array2 < threshold1] = 0
chosen_array2[chosen_array2 > threshold1] = 1
array2[array2_condition] = chosen_array2 # if you still want array2 values to be changed
res1 = array1
res1[array2_condition] = chosen_array2 # Final result
Update
As was mentioned by the OP, every 2D array has at least a 3 in it. So, the array2_condition is not applicable. Instead, we will modify the array2_condition and use a for loop to change the elements.
res1 = array1
res1[res1 < threshold] = 0
res1[res1 > threshold] = 1
array2_condition = np.argwhere(array1 == 3)
for data in array2_condition:
if array2[tuple(data)] > threshold1:
res1[tuple(data)] = 1
elif array2[tuple(data)] < threshold1:
res12[tuple(data)] = 0

Adding two matrices with different shapes using numpy

I have a matrix of zeros with shape (8,30)
matrix
Out[312]:
array([[0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
[0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
[0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
[0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
[0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
[0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
[0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
[0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]])
Then i have another matrix with shape (4,30) like this:
matrix2
Out[313]:
array([[0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0,
0, 1, 0, 0, 0, 0, 0, 0],
[0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
1, 0, 0, 0, 0, 0, 0, 0],
[0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0,
0, 1, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 1]])
What is the quickest way, using numpy, to add to matrix the part corresponding to matrix2 ?
My desired output would be
array([[0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0,
0, 1, 0, 0, 0, 0, 0, 0],
[0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
1, 0, 0, 0, 0, 0, 0, 0],
[0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0,
0, 1, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 1],
[0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
[0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
[0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
[0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]])
still with shape (8,30).

You can use:
matrix[:matrix2.shape[0]] += matrix2
Output:
[[0 0 0 1 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0]
[0 1 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0]
[0 1 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0]
[0 0 0 1 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1]
[0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]
[0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]
[0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]
[0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]]

Error computing KL divergence in Scipy

I am trying to calculate KL divergence using the entropy function of scipy.
My p is:
array([[ 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1.],
[ 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
[ 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
[ 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
[ 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]])
and q is:
array([[ 0.05242718, 0.04436347, 0.04130855, 0.04878344, 0.04310538,
0.02856853, 0.03303122, 0.02517992, 0.08525434, 0.03450324,
0.14580068, 0.1286993 , 0.28897473],
[ 0.65421444, 0.11592199, 0.0642645 , 0.02989768, 0.01385762,
0.01756484, 0.01024294, 0.00891479, 0.01140301, 0.00718939,
0.00938009, 0.01070139, 0.04644726],
[ 0.65984136, 0.13251236, 0.06345234, 0.02891162, 0.02429709,
0.02025307, 0.01073064, 0.01170066, 0.00678652, 0.00703361,
0.00560414, 0.00651137, 0.02236522],
[ 0.32315928, 0.23900077, 0.05460232, 0.03953635, 0.02901102,
0.01294443, 0.02372061, 0.02092882, 0.01188251, 0.01377188,
0.02976672, 0.05854314, 0.14313218],
[ 0.7717858 , 0.09692616, 0.03415596, 0.01713088, 0.01108141,
0.0128005 , 0.00847301, 0.01049734, 0.0052889 , 0.00514799,
0.00442508, 0.00485477, 0.01743218]], dtype=float32)
When I do:
entropy(p[0],q[0])
I am getting the following error:
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
<ipython-input-201-563ea7d4decf> in <module>()
4 print('p0:',p[0])
5 print('q0:',q[0])
----> 6 entropy(p[0],q[0])
/Users/freelancer/anaconda/envs/py35/lib/python3.5/site-packages/matplotlib/mlab.py in entropy(y, bins)
1570 y = np.zeros((len(x)+2,), x.dtype)
1571 y[1:-1] = x
-> 1572 dif = np.diff(y)
1573 up = (dif == 1).nonzero()[0]
1574 dn = (dif == -1).nonzero()[0]
/Users/freelancer/anaconda/envs/py35/lib/python3.5/site-packages/numpy/lib/function_base.py in histogram(a, bins, range, normed, weights, density)
781 if (np.diff(bins) < 0).any():
782 raise ValueError(
--> 783 'bins must increase monotonically.')
784
785 # Initialize empty histogram
ValueError: bins must increase monotonically.
Why is it?

This works with the example arrays:
import scipy as sp
sp.stats.entropy(p[0], q[0])
Looking at the stack trace in the error massage, it becomes apparent that you did not call scipy's entropy function but matplotlib's entropy, which works differently.
Here is the relevant part:
/Users/freelancer/anaconda/envs/py35/lib/python3.5/site-packages/matplotlib/mlab.pyin entropy(y, bins)

Index variable range in numpy

I have a numpy zero matrix A of the shape (2, 5).
A = [[ 0., 0., 0., 0., 0.],
[ 0., 0., 0., 0., 0.]]
I have another array seq of size 2. This is same as the first axis of A.
seq = [2, 3]
I want to create another matrix B which looks like this:
B = [[ 1., 1., 0., 0., 0.],
[ 1., 1., 1., 0., 0.]]
B is constructed by changing the first seq[i] elements in the ith row of A with 1.
This is a toy example. A and seq can be large so efficiency is required. I would be extra thankful if someone knows how to do this in tensorflow.

You can do this in TensorFlow (and with some analogous code in NumPy) as follows:
seq = [2, 3]
b = tf.expand_dims(tf.range(5), 0) # A 1 x 5 matrix.
seq_matrix = tf.expand_dims(seq, 1) # A 2 x 1 matrix.
b_bool = tf.greater(seq_matrix, b) # A 2 x 5 bool matrix.
B = tf.to_int32(b_bool) # A 2 x 5 int matrix.
Example output:
In [7]: b = tf.expand_dims(tf.range(5), 0)
[[0 1 2 3 4]]
In [21]: b_bool = tf.greater(seq_matrix, b)
In [22]: op = sess.run(b_bool)
In [23]: print(op)
[[ True True False False False]
[ True True True False False]]
In [24]: bint = tf.to_int32(b_bool)
In [25]: op = sess.run(bint)
In [26]: print(op)
[[1 1 0 0 0]
[1 1 1 0 0]]

This #mrry's solution, expressed a little differently
In [667]: [[2],[3]]>np.arange(5)
Out[667]:
array([[ True, True, False, False, False],
[ True, True, True, False, False]], dtype=bool)
In [668]: ([[2],[3]]>np.arange(5)).astype(int)
Out[668]:
array([[1, 1, 0, 0, 0],
[1, 1, 1, 0, 0]])
The idea is to compare [2,3] with [0,1,2,3,4] in an 'outer' broadcasting sense. The result is boolean which can be easily changed to 0/1 integers.
Another approach would be to use cumsum (or another ufunc.accumulate function):
In [669]: A=np.zeros((2,5))
In [670]: A[range(2),[2,3]]=1
In [671]: A
Out[671]:
array([[ 0., 0., 1., 0., 0.],
[ 0., 0., 0., 1., 0.]])
In [672]: A.cumsum(axis=1)
Out[672]:
array([[ 0., 0., 1., 1., 1.],
[ 0., 0., 0., 1., 1.]])
In [673]: 1-A.cumsum(axis=1)
Out[673]:
array([[ 1., 1., 0., 0., 0.],
[ 1., 1., 1., 0., 0.]])
Or a variation starting with 1's:
In [681]: A=np.ones((2,5))
In [682]: A[range(2),[2,3]]=0
In [683]: A
Out[683]:
array([[ 1., 1., 0., 1., 1.],
[ 1., 1., 1., 0., 1.]])
In [684]: np.minimum.accumulate(A,axis=1)
Out[684]:
array([[ 1., 1., 0., 0., 0.],
[ 1., 1., 1., 0., 0.]])

How to create a diagonal multi-dimensional (ie greater than 2) in numpy

Is there a higher (than two) dimensional equivalent of diag?
L = [...] # some arbitrary list.
A = ndarray.diag(L)
will create a diagonal 2-d matrix shape=(len(L), len(L)) with elements of L on the diagonal.
I'd like to do the equivalent of:
length = len(L)
A = np.zeros((length, length, length))
for i in range(length):
A[i][i][i] = L[i]
Is there a slick way to do this?
Thanks!

You can use diag_indices to get the indices to be set. For example,
x = np.zeros((3,3,3))
L = np.arange(6,9)
x[np.diag_indices(3,ndim=3)] = L
gives
array([[[ 6., 0., 0.],
[ 0., 0., 0.],
[ 0., 0., 0.]],
[[ 0., 0., 0.],
[ 0., 7., 0.],
[ 0., 0., 0.]],
[[ 0., 0., 0.],
[ 0., 0., 0.],
[ 0., 0., 8.]]])
Under the hood diag_indices is just the code Jaime posted, so which to use depends on whether you want it spelled out in a numpy function, or DIY.

You can use fancy indexing:
In [2]: a = np.zeros((3,3,3))
In [3]: idx = np.arange(3)
In [4]: a[[idx]*3] = 1
In [5]: a
Out[5]:
array([[[ 1., 0., 0.],
[ 0., 0., 0.],
[ 0., 0., 0.]],
[[ 0., 0., 0.],
[ 0., 1., 0.],
[ 0., 0., 0.]],
[[ 0., 0., 0.],
[ 0., 0., 0.],
[ 0., 0., 1.]]])
For a more general approach, you could set the diagonal of an arbitrarily sized array doing something like:
def set_diag(arr, values):
idx = np.arange(np.min(arr.shape))
arr[[idx]*arr.ndim] = values