how to resize and subtract numpy arrays in c++ - numpy

I have two numpy 3D-array in python with different height and width. I want to pass them to my C-Extension. How can I resize and subtract them in c++? Please see the comments in the code.
static PyObject *my_func(PyObject *self, PyObject *args)
{
Py_Initialize();
import_array();
PyObject *arr1;
PyObject *arr2;
if(!PyArg_ParseTuple(args, "OO", &arr1, &arr2))
{
return NULL;
}
//How can I do this?
//resize arr1 to [100, 100, 3]
//resize arr2 to [100, 100, 3]
//res = arr1 - arr2
//return res
}

Start by making the desired shape. It's easier to do this as a tuple than a list:
PyObject* shape = Py_BuildValue("iii",100,100,3);
Check this against NULL to ensure do error has occurred and handle if it has.
You can call the numpy resize function on both arrays to resize them. Unless you are certain that the data isn't shared then you need to call numpy.resize rather than the .resize method of the arrays. This involves importing the module and getting the resize attribute:
PyObject* np = PyImport_ImportModule("numpy");
PyObject* resize = PyObject_GetAttrString(np,"resize");
PyObject* resize_result = PyObject_CallFunctionObjArgs(resize,arr1, shape,NULL);
I've omitted all the error checking, which you should do after each stage.
Make sure you decrease the reference counts on the various PyObjects once you don't need them any more.
Use PyNumber_Subtract to do the subtraction (do it on the result from resize).
Addition: A shortcut for calling resize that should avoid most of the intermediates:
PyObject* np = PyImport_ImportModule("numpy");
// error check against null
PyObject* resize_result = PyObject_CallMethod(np,"resize","O(iii)",arr1,100,100,3);
(The "(iii)" creates the shape tuple rather than needing to do it separately.)
If you are certain that arr1 and arr2 are the only owners of the data then you can call the numpy .resize method either by the normal C API function calls or the specific numpy function PyArray_Resize.

Related

Indexing in Rust ndarray crate based on a boolean mask

I would like to efficiently index into an ndarray using a boolean mask. To better convey what I mean I have some working numpy code and then my attempt in rust ndarray which works but is extremely inefficient.
Numpy:
import numpy as np
shape = (100, 100, 100)
grouping_array = np.random.randint(0, 100, size=shape)
data_array = np.random.rand(*shape)
for i in range(1, 100):
ith_mean = data_array[grouping_array == i].mean()
print(ith_mean)
Rust ndarray:
fn group_means(
data: &Array<f32, IxDyn>,
grouping_var: &Array<f32, IxDyn>,
n_groups: i32,
) {
for group in 1..n_groups {
let index_array = grouping_var.mapv(|x| x == roi as f32);
let roi_data = Array::from_iter(
image_data
.iter()
.zip(index_array.iter())
.map(|(x, y)| if *y { *x } else { 0. })
);
let mean_roi = roi_data.mean().unwrap();
println!("group {}; mean {}", group, mean_roi);
}
}
Here each iteration in the n_groups loop takes about as long as the whole numpy script which is done in less than a second. Is there a better way to do this in the rust-ndarray version?
This is likely not a surprise to others, but since my grouping_var array should (in my use case) always be 3D array, I changed its type (and therefore also index_array) from &Array<f32, IxDyn> to &Array<f32, Ix3> which dramatically improved performance.

Series.agg() works differently when passing function

Called a Function inside agg() of Series, from below snippet of code, in first call it's printing int number for variable "a", and in second call it's coming as Series. I am not able to figure it out the reason for this behaviour.
import pandas as pd
ser = pd.Series([1,2,3,4,5])
def find_second_last(a):
print(a)
return a.iloc[-2]
ser.agg(find_second_last)
.iloc with single position without [] will return the int by default
a.iloc[[-2]]# return pd.Series
a.iloc[-2] # return int
a.iloc[1:] # return pd.Series

Why does cythons in-place division of numpy arrays use conversion to python floats?

I tried to normalize a vector stored as numpy array, but cython -a shows unexpected conversions to Python values in this code.
Minimal example:
import numpy as np
cimport cython
cimport numpy as np
#cython.wraparound(False)
#cython.boundscheck(False)
cdef vec_diff(np.ndarray[double, ndim=1] vec1, double m):
vec1/=m
return vec1
Cython 0.29.6 run with the -a option generates the following code for the line vec1/=m:
__pyx_t_1 = PyFloat_FromDouble(__pyx_v_m); if (unlikely(!__pyx_t_1)) __PYX_ERR(0, 8, __pyx_L1_error)
__Pyx_GOTREF(__pyx_t_1);
__pyx_t_2 = __Pyx_PyNumber_InPlaceDivide(((PyObject *)__pyx_v_vec1), __pyx_t_1); if (unlikely(!__pyx_t_2)) __PYX_ERR(0, 8, __pyx_L1_error)
__Pyx_GOTREF(__pyx_t_2);
__Pyx_DECREF(__pyx_t_1); __pyx_t_1 = 0;
if (!(likely(((__pyx_t_2) == Py_None) || likely(__Pyx_TypeTest(__pyx_t_2, __pyx_ptype_5numpy_ndarray))))) __PYX_ERR(0, 8, __pyx_L1_error)
__pyx_t_3 = ((PyArrayObject *)__pyx_t_2);
{
__Pyx_BufFmt_StackElem __pyx_stack[1];
__Pyx_SafeReleaseBuffer(&__pyx_pybuffernd_vec1.rcbuffer->pybuffer);
__pyx_t_4 = __Pyx_GetBufferAndValidate(&__pyx_pybuffernd_vec1.rcbuffer->pybuffer, (PyObject*)__pyx_t_3, &__Pyx_TypeInfo_double, PyBUF_FORMAT| PyBUF_STRIDES, 1, 0, __pyx_stack);
if (unlikely(__pyx_t_4 < 0)) {
PyErr_Fetch(&__pyx_t_5, &__pyx_t_6, &__pyx_t_7);
if (unlikely(__Pyx_GetBufferAndValidate(&__pyx_pybuffernd_vec1.rcbuffer->pybuffer, (PyObject*)__pyx_v_vec1, &__Pyx_TypeInfo_double, PyBUF_FORMAT| PyBUF_STRIDES, 1, 0, __pyx_stack) == -1)) {
Py_XDECREF(__pyx_t_5); Py_XDECREF(__pyx_t_6); Py_XDECREF(__pyx_t_7);
__Pyx_RaiseBufferFallbackError();
} else {
PyErr_Restore(__pyx_t_5, __pyx_t_6, __pyx_t_7);
}
__pyx_t_5 = __pyx_t_6 = __pyx_t_7 = 0;
}
__pyx_pybuffernd_vec1.diminfo[0].strides = __pyx_pybuffernd_vec1.rcbuffer->pybuffer.strides[0]; __pyx_pybuffernd_vec1.diminfo[0].shape = __pyx_pybuffernd_vec1.rcbuffer->pybuffer.shape[0];
if (unlikely(__pyx_t_4 < 0)) __PYX_ERR(0, 8, __pyx_L1_error)
}
__pyx_t_3 = 0;
__Pyx_DECREF_SET(__pyx_v_vec1, ((PyArrayObject *)__pyx_t_2));
__pyx_t_2 = 0;
where the first line __pyx_t_1 = PyFloat_FromDouble(__pyx_v_m); has PyFloat_FromDouble highlighted in dark red.
Given that I have told cython that the array contains double values, why does it have to convert to a python float?
Note: Memoryviews do not support the /= operation (would require a loop)
Because this isn't something that Cython does anything special for or optimises at all. All it's doing is calling __Pyx_PyNumber_InPlaceDivide on the Numpy array, which calls the Numpy array's __idiv__ operator.
Since it's calling a Python operator it needs to pass a Python object as the second argument, and hence it needs to convert your double to a Python float.
The Numpy __idiv__ operator is almost certainly written in C so likely to be pretty fast (although there is a little overhead calling it) so there's not a lot of value in Cython doing anything except delegating to Numpy's code.
Memoryviews don't define the whole-array operators (they're just ways to access memory so don't make any claims about meaningful mathematical operations) and hence the fact that it doesn't work is consistent with how Cython deals with these operators.

TypeError: unhashable type: 'numpy.ndarray' - How to get data from data frame by querying radius from ball tree?

How to get data by querying radius from ball tree? For example
from sklearn.neighbors import BallTree
import pandas as pd
bt = BallTree(df[['lat','lng']], metric="haversine")
for idx, row in df.iterrow():
res = df[bt.query_radius(row[['lat','lng']],r=1)]
I want to get those rows in df that are in radius r=1. But it throws type error
TypeError: unhashable type: 'numpy.ndarray'
Following the first answer I got index out of range when iterating over the rows
5183
(5219, 25)
5205
(5219, 25)
5205
(5219, 25)
5221
(5219, 25)
Traceback (most recent call last):
File "/Users/Chu/Documents/dssg2018/sa4.py", line 45, in <module>
df.loc[idx,word]=len(df.iloc[indices[idx]][df[word]==1])/\
IndexError: index 5221 is out of bounds for axis 0 with size 5219
And the code is
bag_of_words = ['beautiful','love','fun','sunrise','sunset','waterfall','relax']
for idx,row in df.iterrows():
for word in bag_of_words:
if word in row['caption']:
df.loc[idx, word] = 1
else:
df.loc[idx, word] = 0
bt = BallTree(df[['lat','lng']], metric="haversine")
indices = bt.query_radius(df[['lat','lng']],r=(float(10)/40000)*360)
for idx,row in df.iterrows():
for word in bag_of_words:
if word in row['caption']:
print(idx)
print(df.shape)
df.loc[idx,word]=len(df.iloc[indices[idx]][df[word]==1])/\
np.max([1,len(df.iloc[indices[idx]][df[word]!=1])])
The error is not in the BallTree, but the indices returned by it are not used properly for putting it into index.
Do it this way:
for idx, row in df.iterrows():
indices = bt.query_radius(row[['lat','lng']].values.reshape(1,-1), r=1)
res = df.iloc[[x for b in indices for x in b]]
# Do what you want to do with res
This will also do (since we are sending only a single point each time):
res = df.iloc[indices[0]]
Explanation:
I'm using scikit 0.20. So the code you wrote above:
df[bt.query_radius(row[['lat','lng']],r=1)]
did not work for me. I needed to make it a 2-d array by using reshape().
Now bt.query_radius() returns array of array of indices within the radius r specified as mentioned in the documentation:
ind : array of objects, shape = X.shape[:-1]
each element is a numpy integer array listing the indices of neighbors of the corresponding point. Note that unlike the results of
a k-neighbors query, the returned neighbors are not sorted by distance
by default.
So we needed to iterate two arrays to reach the actual indices of the data.
Now once we got the indices, in a pandas Dataframe, iloc is the way to access data with indices.
Update:
You dont need to query the bt each time for individual points. You can send all the df at once to return a 2-d array containing the indices of points within the radius to the point specified that index.
indices = bt.query_radius(df, r=1)
for idx, row in df.iterrows():
nearest_points_index = indices[idx]
res = df.iloc[nearest_points_index]
# Do what you want to do with res

how band read array just return me 0, how can i do?

I am new in GDAL. I use landsat image in classification but I apply these code line band data just give me 0, what I don't understand:
raster_dataset = gdal.Open(‘LC81850552017065LGN00_B3.tif’, gdal.GA_ReadOnly)
geo_transform = raster_dataset.GetGeoTransform()
proj = raster_dataset.GetProjectionRef()
bands_data = []
for b in range(1, raster_dataset.RasterCount+1):
band = raster_dataset.GetRasterBand(b)
bands_data.append(band.ReadAsArray())
bands_data = np.dstack(bands_data)
You are reassigning bands_data from a list to a numpy array, which is causing your trouble. Perform the np.dstack operation after all bands have been read in (after the loop has been completed).
See the numpy docs to better understand how dstack works.