one-hot encoding and existing data - numpy

I have a numpy array (N,M) where some of the columns should be one-hot encoded. Please help to make a one-hot encoding using numpy and/or tensorflow.
Example:
[
[ 0.993, 0, 0.88 ]
[ 0.234, 1, 1.00 ]
[ 0.235, 2, 1.01 ]
.....
]
The 2nd column here ( with values 3 and 2 ) should be one hot-encoded, I know that there are only 3 distinct values ( 0, 1, 2 ).
The resulting array should look like:
[
[ 0.993, 0.88, 0, 0, 0 ]
[ 0.234, 1.00, 0, 1, 0 ]
[ 0.235, 1.01, 1, 0, 0 ]
.....
]
Like that I would be able to feed this array into the tensorflow.
Please notice that 2nd column was removed and it's one-hot version was appended in the end of each sub-array.
Any help would be highly appreciated.
Thanks in advance.
Update:
Here is what I have right now:
Well, not exactly...
1. I have more than 3 columns in the array...but I still want to do it only with 2nd..
2. First array is structured, ie it's shape is (N,)
Here is what I have:
def one_hot(value, max_value):
value = int(value)
a = np.zeros(max_value, 'uint8')
if value != 0:
a[value] = 1
return a
# data is structured array with the shape of (N,)
# it has strings, ints, floats inside..
# was get by np.genfromtxt(dtype=None)
unique_values = dict()
unique_values['categorical1'] = 1
unique_values['categorical2'] = 2
for row in data:
row[col] = unique_values[row[col]]
codes = np.zeros((data.shape[0], len(unique_values)))
idx = 0
for row in data:
codes[idx] = one_hot(row[col], len(unique_values)) # could be optimised by not creating new array every time
idx += 1
data = np.c_[data[:, [range(0, col), range(col + 1, 32)]], codes[data[:, col].astype(int)]]
Also trying to concatenate via:
print data.shape # shape (5000,)
print codes.shape # shape (5000,3)
data = np.concatenate((data, codes), axis=1)

Here's one approach -
In [384]: a # input array
Out[384]:
array([[ 0.993, 0. , 0.88 ],
[ 0.234, 1. , 1. ],
[ 0.235, 2. , 1.01 ]])
In [385]: codes = np.array([[0,0,0],[0,1,0],[1,0,0]]) # define codes here
In [387]: codes
Out[387]:
array([[0, 0, 0], # encoding for 0
[0, 1, 0], # encoding for 1
[1, 0, 0]]) # encoding for 2
# Slice out the second column and append one-hot encoded array
In [386]: np.c_[a[:,[0,2]], codes[a[:,1].astype(int)]]
Out[386]:
array([[ 0.993, 0.88 , 0. , 0. , 0. ],
[ 0.234, 1. , 0. , 1. , 0. ],
[ 0.235, 1.01 , 1. , 0. , 0. ]])

Related

Addressing polynomial multiplication and division "overflow" issue

I have a list of the coefficient to degree 1 polynomials, with a[i][0]*x^1 + a[i][1]
a = np.array([[ 1. , 77.48514702],
[ 1. , 0. ],
[ 1. , 2.4239275 ],
[ 1. , 1.21848739],
[ 1. , 0. ],
[ 1. , 1.18181818],
[ 1. , 1.375 ],
[ 1. , 2. ],
[ 1. , 2. ],
[ 1. , 2. ]])
And running into issues with the following operation,
np.polydiv(reduce(np.polymul, a), a[0])[0] != reduce(np.polymul, a[1:])
where
In [185]: reduce(np.polymul, a[1:])
Out[185]:
array([ 1. , 12.19923307, 63.08691612, 179.21045388,
301.91486027, 301.5756213 , 165.35814595, 38.39582615,
0. , 0. ])
and
In [186]: np.polydiv(reduce(np.polymul, a), a[0])[0]
Out[186]:
array([ 1.00000000e+00, 1.21992331e+01, 6.30869161e+01, 1.79210454e+02,
3.01914860e+02, 3.01575621e+02, 1.65358169e+02, 3.83940472e+01,
1.37845155e-01, -1.06809521e+01])
First of all the remainder of np.polydiv(reduce(np.polymul, a), a[0]) is way bigger than 0, 827.61514239 to be exact, and secondly, the last two terms to quotient should be 0, but way larger from 0. 1.37845155e-01, -1.06809521e+01.
I'm wondering what are my options to improve the accuracy?
There is a slightly complicated way to keep the product first and then divide structure.
By first employ n points and evaluate on a.
xs = np.linspace(0, 1., 10)
ys = np.array([np.prod(list(map(lambda r: np.polyval(r, x), a))) for x in xs])
then do the division on ys instead of coefficients.
ys = ys/np.array([np.polyval(a[0], x) for x in xs])
finally recover the coefficient using polynomial interpolation with xs and ys
from scipy.interpolate import lagrange
lagrange(xs, ys)

Compare numpy arrays of different shapes

I have two numpy arrays of shapes (4,4) and (9,4)
matrix1 = array([[ 72. , 72. , 72. , 72. ],
[ 72.00396729, 72.00396729, 72.00396729, 72.00396729],
[596.29998779, 596.29998779, 596.29998779, 596.29998779],
[708.83398438, 708.83398438, 708.83398438, 708.83398438]])
matrix2 = array([[ 72.02400208, 77.68997192, 115.6057663 , 105.64997101],
[120.98195648, 77.68997192, 247.19802856, 105.64997101],
[252.6330719 , 77.68997192, 337.25634766, 105.64997101],
[342.63256836, 77.68997192, 365.60125732, 105.64997101],
[ 72.02400208, 113.53997803, 189.65515137, 149.53997803],
[196.87202454, 113.53997803, 308.13119507, 149.53997803],
[315.3480835 , 113.53997803, 405.77023315, 149.53997803],
[412.86999512, 113.53997803, 482.0453186 , 149.53997803],
[ 72.02400208, 155.81002808, 108.98254395, 183.77003479]])
I need to compare all the rows of matrix2 with every row of matrix1. How can this be done without looping in the elements of matrix1?
If it is about element-wise comparison of the rows, then check this example:
# Generate sample arrays
a = np.random.randint(0, 5, size = (4, 3))
b = np.random.randint(-1, 6, size = (5, 3))
# Compare
a == b[:, None]
The last line does the comparison for you. The output array will have shape (num_of_b_rows, num_of_a_rows, common_num_of_cols): in this case, (5, 4, 3).

tensorflow get boxes pair with maximum IOU but discard boxes with all zeros

This question is built from this question: (tensorflow remember the index after calculating getting the maximum box). I find discarding boxes with all zeros particularly hard, so I am posting a new one.
Complete description:
Assume that I have two arrays of boxes, each of which has the shape (?, b1, 4) and (?, b2, 4) respectively (treat ? as a unknown batch size):
box1: [[[1,2,3,4], [2,3,4,5], [3,4,5,6], [0,0,0,0], [0,0,0,0]...]...]
box2: [[[4,3,2,1], [3,2,5,4], [4,3,5,6]...]...]
(the number above are set arbitarily)
Note that box1 may or may not have fake box([0,0,0,0]) at the end.
I want to:
in each batch, for each non-fake box A in box1 (that is, boxes that does not contain all zeros), find in box2 the box B which has the maximum IOU (intersection over union) with A (in the same batch, of course), and then append the tuple (A, B) to a list list_max.
append to list_nonmax all the boxes in box2 that does not have maximum IOU with any box in box1 (separated by batch, of course)
You can assume that:
b1 and b2 are both python variables, not tensorflow tensor.
methods for calculating IOU between single box or between batch of boxes already exists and can be used literally:
iou_single_box(box1, box2) : both box1 and box2 are of shape (4,).
iou_multiple_boxes(bbox1, bbox2) : both bbox1 and bbox2 are of shape (b1, 4) and (b2, 4) respectively.
iou_batch_boxes(bbbox1, bbbox2) : both bbbox1 and bbbox2 are of shape (?, b1, 4) and (?, b2, 4) respectively (treat ? as a unknown batch size).
You can take a look at the question (tensorflow remember the index after calculating getting the maximum box) I post previously. I only add one constraint:
I don't want any fake box in box1 to match against any box in box2. when getting list_max and list_nonmax
Note that the number of fake box is not set.
****: I know this question is quite complicated. I do all these because Tensorflow cannot handle dynamic-length array (you have to have a deterministic b1 for box1 at runtime). So I pad [0, 0, 0, 0] at the end of box1 to make the length fixed.
I believe this is easily doable with tf.boolean_mask() like this code (tested):
from __future__ import print_function
import tensorflow as tf
box1 = tf.reshape( tf.constant( range( 16 ), dtype = tf.float32 ), ( 2, 2, 4 ) )
box1 = tf.concat( [ box1, tf.zeros( ( 2, 2, 4 ) ) ], axis = 1 )
box2 = tf.reshape( tf.constant( range( 2, 26 ), dtype = tf.float32 ), ( 2, 3, 4 ) )
batch_size = box1.get_shape().as_list()[ 0 ]
def dummy_iou_batch_boxes( box1, box2 ):
b1s, b2s = box1.get_shape().as_list(), box2.get_shape().as_list()
return tf.constant( [ [ [9.0,8,7], [1,2,3], [ 0, 10, 0 ], [ 0, 0, 0 ],
[0 ,1,2], [0,5,0], [ 0, 0, 0 ], [ 0, 0, 0 ] ] ] )
iou = dummy_iou_batch_boxes( box1, box2 )
val, idx = tf.nn.top_k( iou, k = 1 )
idx = tf.reshape( idx, ( batch_size, box1.get_shape().as_list()[ 1 ] ) )
one_hot_idx = tf.one_hot( idx, depth = box2.get_shape().as_list()[ 1 ] )
# for listmax
full_idx = tf.where( tf.equal( 1.0, one_hot_idx ) )
box1_idx = full_idx[ :, 0 : 2 ]
box2_idx = full_idx[ :, 0 : 3 : 2 ]
box12 = tf.gather_nd( box1, box1_idx )
box22 = tf.gather_nd( box2, box2_idx )
list_max_raw = tf.stack( [ box12, box22 ], axis = 1 )
# filter out for a = [ 0, 0, 0, 0 ]
nonzero_mask = tf.reduce_any( tf.not_equal( 0.0, list_max_raw ), axis = 2 )[ :, 0 ]
list_max = tf.boolean_mask( list_max_raw, nonzero_mask )
# for list nonmax
nonzero_mask = tf.cast( tf.reduce_any( tf.not_equal( 0.0, box1 ), axis = 2 ), tf.float32 )[ ..., None ]
filtered_one_hot = one_hot_idx * nonzero_mask
active_box2 = tf.sign( tf.reduce_sum( filtered_one_hot, axis = 1 ) )
nonactive_box2 = 1.0 - active_box2
nonactive_box2_idx = tf.where( tf.equal( 1.0, nonactive_box2 ) )
list_nonmax = tf.gather_nd( box2, nonactive_box2_idx )
with tf.Session() as sess:
res = sess.run( [ box1, box2, list_max ] )
print( "Input boxes: " )
for v in res[ : 2 ]:
print( v )
print( " ", "=" * 40 )
print( "List max: " )
for v in res[ 2 : ]:
print( v )
print( " ", "=" * 40 )
res = sess.run( [ list_nonmax ] )
print( "List nonmax: " )
for v in res:
print( v )
print( " ", "=" * 40 )
will output
Input boxes:
[[[ 0. 1. 2. 3.]
[ 4. 5. 6. 7.]
[ 0. 0. 0. 0.]
[ 0. 0. 0. 0.]]
[[ 8. 9. 10. 11.]
[12. 13. 14. 15.]
[ 0. 0. 0. 0.]
[ 0. 0. 0. 0.]]]
========================================
[[[ 2. 3. 4. 5.]
[ 6. 7. 8. 9.]
[10. 11. 12. 13.]]
[[14. 15. 16. 17.]
[18. 19. 20. 21.]
[22. 23. 24. 25.]]]
========================================
List max:
[[[ 0. 1. 2. 3.]
[ 2. 3. 4. 5.]]
[[ 4. 5. 6. 7.]
[10. 11. 12. 13.]]
[[ 8. 9. 10. 11.]
[22. 23. 24. 25.]]
[[12. 13. 14. 15.]
[18. 19. 20. 21.]]]
========================================
List nonmax:
[[ 6. 7. 8. 9.]
[14. 15. 16. 17.]]
========================================

bad result from numpy corrcoef and minimum spanning tree

I have this code:
mm = np.array([[1, 4, 7, 8], [2, 2, 8, 4], [1, 13, 1, 5]])
mm = np.column_stack(mm)
mmCov = np.cov(mm, rowvar=0)
print("covariance\n", mmCov)
# my code to get correlations
mmResCor = np.zeros(shape=(3, 3))
for i in range(len(mmCov)):
for j in range(len(mmCov[i])):
mmResCor[i][j] = mmCov[i][j] / (math.sqrt(mmCov[i][i] * mmCov[j] [j]))
print("correlaciones a mano\n", mmResCor)
mmCor = np.corrcoef(mmCov, rowvar=0)
print("correlations\n", mmCor)
X = csr_matrix(mmCor)
XX = minimum_spanning_tree(X)
print("minimun spanning tree\n", XX)
first: each column represents a variable, with observations in the rows
numpy corrcoef use this relation with covariance matrix:
R_{ij} = \frac{ C_{ij} } { \sqrt{ C_{ii} * C_{jj} } }
when I use numpy corrcoef I get this matrix
correlations
[[ 1. 0.8660254 -0.82603319]
[ 0.8660254 1. -0.99717646]
[-0.82603319 -0.99717646 1. ]]
but when I apply "my code" to get the same result...
mmResCor = np.zeros(shape=(3, 3))
for i in range(len(mmCov)):
for j in range(len(mmCov[i])):
mmResCor[i][j] = mmCov[i][j] / (math.sqrt(mmCov[i][i] * mmCov[j][j]))
I get this matrix
correlaciones a mano
[[ 1. 0.67082039 0. ]
[ 0.67082039 1. -0.5 ]
[ 0. -0.5 1. ]]
why do I get differents results if its suppose I am doing the same?
One more question:
When I apply minimun_spanning_tree I get this:
minimun spanning tree
(0, 2) -0.826033187631
(1, 2) -0.997176464953
Is there any way to represent these or can I save this result in some variables?
The np.corrcoef should take the data as the input. You're passing the covariance matrix as input. If you pass the data, you get the same result as your manual computation:
>>> np.corrcoef(mm, rowvar=0)
array([[ 1. , 0.67082039, 0. ],
[ 0.67082039, 1. , -0.5 ],
[ 0. , -0.5 , 1. ]])
Regarding the minimum spanning tree, I'm not sure what your question is, but the output XX is a sparse matrix which stores a matrix representation of the tree.

How to efficiently prepare matrices (2-d array) for multiple arguments?

If you want to evaluate a 1-d array for multiple arguments efficiently i.e. without for-loop, you can do this:
x = array([1, 2, 3])
def gen_1d_arr(x):
arr = array([2 + x, 2 - x,])
return arr
gen_1d_arr(x).T
and you get:
array([[ 3, 1],
[ 4, 0],
[ 5, -1]])
Okay, but how do you do this for 2-d array like below:
def gen_2d_arr(x):
arr = array([[2 + x, 2 - x,],
[2 * x, 2 / x]])
return arr
and obtain this?:
array([[[ 3. , 1. ],
[ 2. , 2. ]],
[[ 4. , 0. ],
[ 4. , 1. ]],
[[ 5. , -1. ],
[ 6. , 0.66666667]]])
Also, is this generally possible for n-d arrays?
Look at what you get with your function
In [274]: arr = np.array([[2 + x, 2 - x,],
[2 * x, 2 / x]])
In [275]: arr
Out[275]:
array([[[ 3. , 4. , 5. ],
[ 1. , 0. , -1. ]],
[[ 2. , 4. , 6. ],
[ 2. , 1. , 0.66666667]]])
In [276]: arr.shape
Out[276]: (2, 2, 3)
The 3 comes from x. The middle 2 comes from [2+x, 2-x] pairs, and the 1st 2 from the outer list.
Looks like what you want is a (3,2,2) array. One option is to apply a transpose or axis swap to arr.
arr.transpose([2,0,1])
The basic operation of np.array([arr1,arr2]) is to construct a new array with a new dimension in front, i.e. with shape (2, *arr1(shape)).
There are other operations that combine arrays. np.concatenate and its variants hstack, vstack, dstack, column_stack, join arrays. .reshape() and [None,...], atleast_nd etc add dimensions. Look at the code of the stack functions to get some ideas on how to combine arrays using these tools.
On the question of efficiency, my time tests show that concatenate operations are generally faster than np.array. Often np.array converts its inputs to lists, and reparses the values. This gives it more power in cooercing arrays to specific dtypes, but at the expense of time. But I'd only worry about this with large arrays where construction time is significant.