Making my cython code more efficient - numpy
I've written a python program which I try to cythonize.
Is there any suggestion how to make the for-loop more efficient, as this is taking 99% of the time?
This is the for-loop:
for i in range(l):
b1[i] = np.nanargmin(locator[i,:]) # Closer point
locator[i, b1[i]] = NAN # Do not consider Closer point
b2[i] = np.nanargmin(locator[i,:]) # 2nd Closer point
Adjacents[i,0] = np.array((Existed_Pips[b1[i]]), dtype=np.double)
Adjacents[i,1] = np.array((Existed_Pips[b2[i]]), dtype=np.double)
This is the rest of the code:
import numpy as np
cimport numpy as np
from libc.math cimport NAN #, isnan
def PIPs(np.ndarray[np.double_t, ndim=1, mode='c'] ys, unsigned int nofPIPs, unsigned int typeofdist):
cdef:
unsigned int currentstate, j, i
np.ndarray[np.double_t, ndim=1, mode="c"] D
np.ndarray[np.int64_t, ndim=1, mode="c"] Existed_Pips
np.ndarray[np.int_t, ndim=1, mode="c"] xs
np.ndarray[np.double_t, ndim=2] Adjacents, locator, Adjy, Adjx, Raw_Fire_PIPs, Raw_Fem_PIPs
np.ndarray[np.int_t, ndim=2, mode="c"] PIP_points, b1, b2
cdef unsigned int l = len(ys)
xs = np.arange(0,l, dtype=np.int) # Column vector with xs
PIP_points = np.zeros((l,1), dtype=np.int) # Binary indexation
PIP_points[0] = 1 # One indicate the PIP points.The first two PIPs are the first and the last observation.
PIP_points[-1] = 1
Adjacents = np.zeros((l,2), dtype=np.double)
currentstate = 2 # Initial PIPs
while currentstate <= nofPIPs: # for eachPIPs in range(nofPIPs)
Existed_Pips = np.flatnonzero(PIP_points)
currentstate = len(Existed_Pips)
locator = np.full((l,currentstate), NAN, dtype=np.double) #np.int*
for j in range(currentstate):
locator[:,j] = np.absolute(xs-Existed_Pips[j])
b1 = np.zeros((l,1), dtype=np.int)
b2 = np.zeros((l,1), dtype=np.int)
for i in range(l):
b1[i] = np.nanargmin(locator[i,:]) # Closer point
locator[i, b1[i]] = NAN # Do not consider Closer point
b2[i] = np.nanargmin(locator[i,:]) # 2nd Closer point
Adjacents[i,0] = np.array((Existed_Pips[b1[i]]), dtype=np.double)
Adjacents[i,1] = np.array((Existed_Pips[b2[i]]), dtype=np.double)
##Calculate Distance
Adjx = Adjacents
Adjy = np.array([ys[np.array(Adjacents[:,0], dtype=np.int)], ys[np.array(Adjacents[:,1], dtype=np.int)]]).transpose()
Adjx[Existed_Pips,:] = NAN # Existed PIPs are not candidates for new PIP.
Adjy[Existed_Pips,:] = NAN
if typeofdist == 1: #Euclidean Distance
##[D] = EDist(ys,xs,Adjx,Adjy)
ED = np.power(np.power((Adjx[:,1]-xs),2) + np.power((Adjy[:,1]-ys),2),(0.5)) + np.power(np.power((Adjx[:,0]-xs),2) + np.power((Adjy[:,0]-ys),2),(0.5))
EDmax = np.nanargmax(ED)
PIP_points[EDmax]=1
currentstate=currentstate+1
return np.array([Existed_Pips, ys[Existed_Pips]]).transpose()
A couple of suggestions:
Take the calls to np.nanargmin out of the loop (use the axis parameter to let you operate on the whole array at once. This reduces the number of Python function calls you have to make:
b1 = np.nanargmin(locator,axis=1)
locator[np.arange(locator.shape[0]),b1] = np.nan
b2 = np.nanargmin(locator,axis=1)
Your assignment to Adjacents is odd - you seem to be creating a length-1 array for the right-hand side first. Instead just do
Adjacents[i,0] = Existed_Pips[b1[i]]
# ...
However, in this case, you can also take both lines outside the loop, eliminating the entire loop:
Adjacents = np.vstack((Existing_Pips[b1], Existings_Pips[b2])).T
All of this is relying on numpy, rather than Cython, for the speed-up, but it probably beats your version.
Related
What is wrong with my cython implementation of erosion operation of mathematical morphology
I have produced a naive implementation of "erosion". The performance is not relevant since I just trying to understand the algorithm. However, the output of my implementation does not match the one I get from scipy.ndimage. What is wrong with my implementation ? Here is my implementation with a small test case: import numpy as np from PIL import Image # a small image to play with a cross structuring element imgmat = np.array([ [0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0], [0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0], [0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,1,1,1,1,1,1,0,0,0,0,0,0,0,0,0], [0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,1,0,0,0,1,0,0,0,0,0,0,0,0,0], [0,0,0,0,0,1,1,0,0,1,1,1,1,0,0,0,1,1,0,0,1,0,0,0,0,0,0,0,0,0], [0,0,0,0,0,1,1,0,0,1,1,1,1,0,0,0,1,1,1,0,1,0,0,0,0,0,0,0,0,0], [0,0,0,0,0,1,1,0,0,1,1,1,1,0,0,0,1,1,1,1,1,0,0,0,0,0,0,0,0,0], [0,0,0,0,0,1,1,0,0,1,1,1,1,0,0,0,1,1,1,1,1,0,0,0,0,0,0,0,0,0], [0,0,0,0,0,1,1,0,0,1,1,1,1,0,0,0,0,1,1,1,1,0,0,0,0,0,0,0,0,0], [0,0,0,0,0,1,1,0,0,1,1,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0], [0,0,0,0,0,1,1,0,0,1,1,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0], [0,0,0,0,0,1,1,0,0,0,0,0,0,0,0,1,1,1,1,1,0,0,1,0,1,1,0,0,0,0], [0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,1,0,0,1,0,1,1,0,0,0,0], [0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,1,0,1,1,0,0,0,0], [0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,1,0,1,1,0,0,0,0], [0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,1,1,1,1,0,0,0,0], ]) imgmat2 = np.where(imgmat == 0, 0, 255).astype(np.uint8) imarr = Image.fromarray(imgmat2).resize((100, 200)) imarr = np.array(imgrrr) imarr = np.where(imarr == 0, 0, 1) se_mat3 = np.array([ [0,1,0], [1,1,1], [0,1,0] ]) se_mat31 = np.where(se_mat3 == 1, 0, 1) The imarr is . My implementation of erosion: %%cython -a import numpy as np cimport numpy as cnp cdef erosionC(cnp.ndarray[cnp.int_t, ndim=2] img, cnp.ndarray[cnp.int_t, ndim=2] B, cnp.ndarray[cnp.int_t, ndim=2] X): """ X: image coordinates struct_element_mat: black and white image, black region is considered as the shape of structuring element This operation checks whether (B *includes* X) = $B \subset X$ as per defined in Serra (Jean), « Introduction to mathematical morphology », Computer Vision, Graphics, and Image Processing, vol. 35, nᵒ 3 (septembre 1986). URL : https://linkinghub.elsevier.com/retrieve/pii/0734189X86900022.. doi: 10.1016/0734-189X(86)90002-2 Consulted le 6 août 2020, p. 283‑305. """ cdef cnp.ndarray[cnp.int_t, ndim=1] a, x, bx cdef cnp.ndarray[cnp.int_t, ndim=2] Bx, B_frame, Xcp, b cdef bint check a = B[0] # get an anchor point from the structuring element coordinates B_frame = B - a # express the se element coordinates in with respect to anchor point Xcp = X.copy() b = img.copy() for x in X: # X contains the foreground coordinates in the image Bx = B_frame + x # translate relative coordinates with respect to foreground coordinates considering it as the anchor point check = True # this is erosion so if any of the se coordinates is not in foreground coordinates we consider it a miss for bx in Bx: # Bx contains all the translated coordinates of se if bx not in Xcp: check = False if check: b[x[0], x[1]] = 1 # if there is a hit else: b[x[0], x[1]] = 0 # if there is no hit return b def erosion(img: np.ndarray, struct_el_mat: np.ndarray, foregroundValue = 0): B = np.argwhere(struct_el_mat == 0) X = np.argwhere(img == foregroundValue) nimg = erosionC(img, B, X) return np.where(nimg == 1, 255, 0) The calling code for both is: from scipy import ndimage as nd err = nd.binary_erosion(imarr, se_mat3) imerrCustom = erosion(imarr, se_mat31, foregroundValue=1) err produces imerrCustom produces
In the end, I am still not sure about it, but after having read several papers more, I assume that my interpretation of X as foreground coordinates was an error. It should have probably been the entire image that is being iterated. As I have stated I am not sure if this interpretation is correct as well. But I made a new implementation which iterates over the image, and it gives a more plausible result. I am sharing it in here, hoping that it might help someone: %%cython -a import numpy as np cimport numpy as cnp cdef dilation_c(cnp.ndarray[cnp.uint8_t, ndim=2] X, cnp.ndarray[cnp.uint8_t, ndim=2] SE): """ X: boolean image SE: structuring element matrix origin: coordinate of the origin of the structuring element This operation checks whether (B *hits* X) = $B \cap X \not = \emptyset$ as per defined in Serra (Jean), « Introduction to mathematical morphology », Computer Vision, Graphics, and Image Processing, vol. 35, nᵒ 3 (septembre 1986). URL : https://linkinghub.elsevier.com/retrieve/pii/0734189X86900022.. doi: 10.1016/0734-189X(86)90002-2 Consulted le 6 août 2020, p. 283‑305. The algorithm adapts DILDIRECT of Najman (Laurent) et Talbot (Hugues), Mathematical morphology: from theory to applications, 2013. ISBN : 9781118600788, p. 329 to the formula given in Jähne (Bernd), Digital image processing, 6th rev. and ext. ed, Berlin ; New York, 2005. TA1637 .J34 2005. ISBN : 978-3-540-24035-8. """ cdef cnp.ndarray[cnp.uint8_t, ndim=2] O cdef list elst cdef int r, c, X_rows, X_cols, SE_rows, SE_cols, se_r, se_c cdef cnp.ndarray[cnp.int_t, ndim=1] bp cdef list conds cdef bint check, b, p, cond O = np.zeros_like(X) X_rows, X_cols = X.shape[:2] SE_rows, SE_cols = SE.shape[:2] # a boolean convolution for r in range(0, X_rows-SE_rows): for c in range(0, X_cols - SE_cols): conds = [] for se_r in range(SE_rows): for se_c in range(SE_cols): b = <bint>SE[se_r, se_c] p = <bint>X[se_r+r, se_c+c] conds.append(b and p) O[r,c] = <cnp.uint8_t>any(conds) return O def dilation_erosion( img: np.ndarray, struct_el_mat: np.ndarray, foregroundValue: int = 1, isErosion: bool = False): """ img: image matrix struct_el: NxN mesh grid of the structuring element whose center is SE's origin structuring element is encoded as 1 foregroundValue: value to be considered as foreground in the image """ B = struct_el_mat.astype(np.uint8) if isErosion: X = np.where(img == foregroundValue, 0, 1).astype(np.uint8) else: X = np.where(img == foregroundValue, 1, 0).astype(np.uint8) nimg = dilation_c(X, B) foreground, background = (255, 0) if foregroundValue == 1 else (0, 1) if isErosion: return np.where(nimg == 1, background, foreground).astype(np.uint8) else: return np.where(nimg == 1, foreground, background).astype(np.uint8) # return nimg
SOM kmean optimization ValueError: all the input arrays must have same number of dimensions
I am trying to merge kmeans into SOM finding the best match unit. During clustering points to return the numbers of clusters for each point I encounter this error "ValueError: all the input arrays must have same number of dimensions" in line 159 distances_from_center = np.concatenate((distances_from_center, [dist(teacher,nodes)])) I am trying to optimize the SOM using the fast kmeans approach. N = 8 # linear size of 2D map M = 8 n_teacher = 10000 # # of teacher signal np.random.seed(100)# test seed for random number def main(): # initialize node vectors nodes = np.random.rand(N,M,3)# node array. each node has 3-dim weight vector #nodes = centers_initiation(n_teacher, 4) #initial out put #TODO; make out put function to simplify here plt.imshow(nodes, interpolation='none') plt.savefig("init.png") """""" """ Learning """ """""" # teacher signal teachers = np.random.rand(n_teacher,3) for i in range(n_teacher): train(nodes, teachers, i) # intermediate out put if i%200 ==0 or i< 100: #out put for i<100 or each 1000 iteration plt.imshow(nodes, interpolation='none') plt.savefig(str(i)+".png") #output plt.imshow(nodes, interpolation='none') plt.savefig("final.png") def train(nodes, teachers, i): bmu = best_matching_unit(nodes, teachers[i]) #print bmu for x in range(N): for y in range(M): c = np.array([x,y])# coordinate of unit d = np.linalg.norm(c-bmu) L = learning_ratio(i) S = learning_radius(i,d) for z in range(3): #TODO clear up using numpy function nodes[x,y,z] += L*S*(teachers[i,z] - nodes[x,y,z]) def dist(x, y): # euclidean distance if len(x.shape) == 1: d = np.sqrt(np.sum((x - y) ** 2)) else: d = np.sqrt(np.sum((x - y) ** 2, axis=1)) return d def centers_initiation(teacher, number_of_centers): # initialization of clusters centers as most distant points. return cluster centers (point) dist_per_point = np.empty((0, 0), int) dist_for_point = 0 index_of_deleted_point = 0 for point in teacher: for other_point in np.delete(teacher, index_of_deleted_point, axis=0): dist_for_point += dist(point, other_point) dist_per_point = np.append(dist_per_point, dist_for_point) dist_for_point = 0 index_of_deleted_point += 1 ordered_points_by_min = np.array( [key for key, value in sorted(enumerate(dist_per_point), key=lambda p: p[1], reverse=True)]) return teacher[ordered_points_by_min[0:number_of_centers]] def get_cluster_number(teacher, nodes): # clustering points. return numbers of clusters for each point distances_from_centers = np.zeros((0, nodes.shape[0]), int) for point in teacher: distances_from_center = np.array([]) for center in nodes: distances_from_center = np.concatenate((distances_from_center, [dist(teacher,nodes)])) distances_from_centers = np.concatenate((distances_from_centers, [distances_from_center]), axis=0) nearest_center_number = np.argmin(distances_from_centers, axis=1) return nearest_center_number def best_matching_unit(teacher, nodes): clusters = get_cluster_number(teacher, nodes) clusters_centers_shift = 1 new_centers = np.zeros(nodes.shape) counter = 0 while np.sum(clusters_centers_shift) != 0: counter += 1 for i in xrange(nodes.shape[0]): new_centers[i] = np.mean(teacher[:][clusters == i], axis=0) clusters_centers_shift = dist(new_centers, nodes) clusters = get_cluster_number(teacher, new_centers) nodes = np.copy(new_centers) return clusters def neighbourhood(t):#neighbourhood radious halflife = float(n_teacher/4) #for testing initial = float(N/2) return initial*np.exp(-t/halflife) def learning_ratio(t): halflife = float(n_teacher/4) #for testing initial = 0.1 return initial*np.exp(-t/halflife) def learning_radius(t, d): # d is distance from BMU s = neighbourhood(t) return np.exp(-d**2/(2*s**2)) main()
Evaluating the squared term of a gaussian kernel for having a covariance matrix for multi-dimensional inputs [duplicate]
I have the following code. It is taking forever in Python. There must be a way to translate this calculation into a broadcast... def euclidean_square(a,b): squares = np.zeros((a.shape[0],b.shape[0])) for i in range(squares.shape[0]): for j in range(squares.shape[1]): diff = a[i,:] - b[j,:] sqr = diff**2.0 squares[i,j] = np.sum(sqr) return squares
You can use np.einsum after calculating the differences in a broadcasted way, like so - ab = a[:,None,:] - b out = np.einsum('ijk,ijk->ij',ab,ab) Or use scipy's cdist with its optional metric argument set as 'sqeuclidean' to give us the squared euclidean distances as needed for our problem, like so - from scipy.spatial.distance import cdist out = cdist(a,b,'sqeuclidean')
I collected the different methods proposed here, and in two other questions, and measured the speed of the different methods: import numpy as np import scipy.spatial import sklearn.metrics def dist_direct(x, y): d = np.expand_dims(x, -2) - y return np.sum(np.square(d), axis=-1) def dist_einsum(x, y): d = np.expand_dims(x, -2) - y return np.einsum('ijk,ijk->ij', d, d) def dist_scipy(x, y): return scipy.spatial.distance.cdist(x, y, "sqeuclidean") def dist_sklearn(x, y): return sklearn.metrics.pairwise.pairwise_distances(x, y, "sqeuclidean") def dist_layers(x, y): res = np.zeros((x.shape[0], y.shape[0])) for i in range(x.shape[1]): res += np.subtract.outer(x[:, i], y[:, i])**2 return res # inspired by the excellent https://github.com/droyed/eucl_dist def dist_ext1(x, y): nx, p = x.shape x_ext = np.empty((nx, 3*p)) x_ext[:, :p] = 1 x_ext[:, p:2*p] = x x_ext[:, 2*p:] = np.square(x) ny = y.shape[0] y_ext = np.empty((3*p, ny)) y_ext[:p] = np.square(y).T y_ext[p:2*p] = -2*y.T y_ext[2*p:] = 1 return x_ext.dot(y_ext) # https://stackoverflow.com/a/47877630/648741 def dist_ext2(x, y): return np.einsum('ij,ij->i', x, x)[:,None] + np.einsum('ij,ij->i', y, y) - 2 * x.dot(y.T) I use timeit to compare the speed of the different methods. For the comparison, I use vectors of length 10, with 100 vectors in the first group, and 1000 vectors in the second group. import timeit p = 10 x = np.random.standard_normal((100, p)) y = np.random.standard_normal((1000, p)) for method in dir(): if not method.startswith("dist_"): continue t = timeit.timeit(f"{method}(x, y)", number=1000, globals=globals()) print(f"{method:12} {t:5.2f}ms") On my laptop, the results are as follows: dist_direct 5.07ms dist_einsum 3.43ms dist_ext1 0.20ms <-- fastest dist_ext2 0.35ms dist_layers 2.82ms dist_scipy 0.60ms dist_sklearn 0.67ms While the two methods dist_ext1 and dist_ext2, both based on the idea of writing (x-y)**2 as x**2 - 2*x*y + y**2, are very fast, there is a downside: When the distance between x and y is very small, due to cancellation error the numerical result can sometimes be (very slightly) negative.
Another solution besides using cdist is the following difference_squared = np.zeros((a.shape[0], b.shape[0])) for dimension_iterator in range(a.shape[1]): difference_squared = difference_squared + np.subtract.outer(a[:, dimension_iterator], b[:, dimension_iterator])**2.
how to use Apache Commons Math Optimization in Jython?
I want to transfer Matlab code to Jython version, and find that the fminsearch in Matlab might be replaced by Apache-Common-Math-Optimization. I'm coding on the Mango Medical Image script manager, which uses Jython 2.5.3 as coding language. And the Math version is 3.6.1. Here is my code: def f(x,y): return x^2+y^2 sys.path.append('/home/shujian/APPs/Mango/lib/commons-math3-3.6.1.jar') sys.add_package('org.apache.commons.math3.analysis') from org.apache.commons.math3.analysis import MultivariateFunction sys.add_package('org.apache.commons.math3.optim.nonlinear.scalar.noderiv') from org.apache.commons.math3.optim.nonlinear.scalar.noderiv import NelderMeadSimplex,SimplexOptimizer sys.add_package('org.apache.commons.math3.optim.nonlinear.scalar') from org.apache.commons.math3.optim.nonlinear.scalar import ObjectiveFunction sys.add_package('org.apache.commons.math3.optim') from org.apache.commons.math3.optim import MaxEval,InitialGuess sys.add_package('org.apache.commons.math3.optimization') from org.apache.commons.math3.optimization import GoalType initialSolution=[2.0,2.0] simplex=NelderMeadSimplex([2.0,2.0]) opt=SimplexOptimizer(2**(-6), 2**(-10)) solution=opt.optimize(MaxEval(300),ObjectiveFunction(f),simplex,GoalType.MINIMIZE,InitialGuess([2.0,2.0])) skewParameters2 = solution.getPointRef() print skewParameters2; And I got the error below: TypeError: optimize(): 1st arg can't be coerced to I'm quite confused about how to use the optimization in Jython and the examples are all Java version.
I've given up this plan and find another method to perform the fminsearch in Jython. Below is the Jython version code: import sys sys.path.append('.../jnumeric-2.5.1_ra0.1.jar') #add the jnumeric path import Numeric as np def nelder_mead(f, x_start, step=0.1, no_improve_thr=10e-6, no_improv_break=10, max_iter=0, alpha=1., gamma=2., rho=-0.5, sigma=0.5): ''' #param f (function): function to optimize, must return a scalar score and operate over a numpy array of the same dimensions as x_start #param x_start (float list): initial position #param step (float): look-around radius in initial step #no_improv_thr, no_improv_break (float, int): break after no_improv_break iterations with an improvement lower than no_improv_thr #max_iter (int): always break after this number of iterations. Set it to 0 to loop indefinitely. #alpha, gamma, rho, sigma (floats): parameters of the algorithm (see Wikipedia page for reference) return: tuple (best parameter array, best score) ''' # init dim = len(x_start) prev_best = f(x_start) no_improv = 0 res = [[np.array(x_start), prev_best]] for i in range(dim): x=np.array(x_start) x[i]=x[i]+step score = f(x) res.append([x, score]) # simplex iter iters = 0 while 1: # order res.sort(key=lambda x: x[1]) best = res[0][1] # break after max_iter if max_iter and iters >= max_iter: return res[0] iters += 1 # break after no_improv_break iterations with no improvement print '...best so far:', best if best < prev_best - no_improve_thr: no_improv = 0 prev_best = best else: no_improv += 1 if no_improv >= no_improv_break: return res[0] # centroid x0 = [0.] * dim for tup in res[:-1]: for i, c in enumerate(tup[0]): x0[i] += c / (len(res)-1) # reflection xr = x0 + alpha*(x0 - res[-1][0]) rscore = f(xr) if res[0][1] <= rscore < res[-2][1]: del res[-1] res.append([xr, rscore]) continue # expansion if rscore < res[0][1]: xe = x0 + gamma*(x0 - res[-1][0]) escore = f(xe) if escore < rscore: del res[-1] res.append([xe, escore]) continue else: del res[-1] res.append([xr, rscore]) continue # contraction xc = x0 + rho*(x0 - res[-1][0]) cscore = f(xc) if cscore < res[-1][1]: del res[-1] res.append([xc, cscore]) continue # reduction x1 = res[0][0] nres = [] for tup in res: redx = x1 + sigma*(tup[0] - x1) score = f(redx) nres.append([redx, score]) res = nres And the test example is as below: def f(x): return x[0]**2+x[1]**2+x[2]**2 print nelder_mead(f,[3.4,2.3,2.2]) Actually, the original version is for python, and the link below is the source: https://github.com/fchollet/nelder-mead
Passing numpy integer array to c code
I'm trying to write Cython code to dump a dense feature matrix, target vector pair to libsvm format faster than sklearn's built in code. I get a compilation error complaining about a type issue with passing the target vector (a numpy array of ints) to the relevant c function. Here's the code: import numpy as np cimport numpy as np cimport cython cdef extern from "cdump.h": int filedump( double features[], int numexemplars, int numfeats, int target[], char* outfname) #cython.boundscheck(False) #cython.wraparound(False) def fastdumpdense_libsvmformat(np.ndarray[np.double_t,ndim=2] X, y, outfname): if X.shape[0] != len(y): raise ValueError("X and y need to have the same number of points") cdef int numexemplars = X.shape[0] cdef int numfeats = X.shape[1] cdef bytes py_bytes = outfname.encode() cdef char* outfnamestr = py_bytes cdef np.ndarray[np.double_t, ndim=2, mode="c"] X_c cdef np.ndarray[np.int_t, ndim=1, mode="c"] y_c X_c = np.ascontiguousarray(X, dtype=np.double) y_c = np.ascontiguousarray(y, dtype=np.int) retval = filedump( &X_c[0,0], numexemplars, numfeats, &y_c[0], outfnamestr) return retval When I attempt to compile this code using distutils, I get the error cythoning fastdump_svm.pyx to fastdump_svm.cpp Error compiling Cython file: ------------------------------------------------------------ ... cdef np.ndarray[np.double_t, ndim=2, mode="c"] X_c cdef np.ndarray[np.int_t, ndim=1, mode="c"] y_c X_c = np.ascontiguousarray(X, dtype=np.double) y_c = np.ascontiguousarray(y, dtype=np.int) retval = filedump( &X_c[0,0], numexemplars, numfeats, &y_c[0], outfnamestr) ^ ------------------------------------------------------------ fastdump_svm.pyx:24:58: Cannot assign type 'int_t *' to 'int *' Any idea how to fix this error? I originally was following the paradigm of passing y_c.data, which works, but this is apparently not the recommended way.
You can also use dtype=np.dtype("i") when initiating a numpy array to match the C int on your machine. cdef int [:] y_c c_array = np.ascontiguousarray(y, dtype=np.dtype("i"))
The problem is that numpy.int_t is not the same as int, you can easily check this by having your program print sizeof(numpy.int_t) and sizeof(int). int is a c int, defined by the c standard as being at least 16 bits, but it's 32 bits on my machine. numpy.int_t is usually 32 bits or 64 bits depending on whether you're using a 32 or 64 bit version of numpy, but of course there is some exception (probably for windows users). If you want to know which numpy dtype matches your c_int you can do np.dtype(cytpes.c_int). So to pass your numpy array to c code you can do: import ctypes cdef np.ndarray[int, ndim=1, mode="c"] y_c y_c = np.ascontiguousarray(y, dtype=ctypes.c_int) retval = filedump( &X_c[0,0], numexemplars, numfeats, &y_c[0], outfnamestr)