I have a list of varying size that contains numpy arrays with the same data type and shape. I would like to process this data using a function written in Cython without copying the data. Both memoryviews and the Python buffer protocol seem to support this kind of data using indirect for the first dimension. So I was hoping that something like this could work:
%%cython
from cython.view cimport indirect
def test(list a):
cdef double[::indirect, :] x
x = a
x[0, 0] = 42
Unfortunately it doesn't.
Is there a way to convert this list of numpy arrays into such a memoryview?
Related
Import numpy as np
A=np.array([1,2,3])
Is there any way to achive A[1,:], in MATLAB it is fine
If you want to treat your numpy array as 2 dimensional array like in MatLab, you have to tell it explicitly, by creating a new array and using np.newaxis .
import numpy as np
A=np.array([1,2,3])
print(A);
B = A[np.newaxis,:]
print(B)
# Here you go
print(B[0,:])
Test it on Online Python
Side note:
I wrote B[0,:], not B[1,:], because Python array indices are 0-based, not 1-based like MatLab.
I created a typed memoryview in cython and would like to multiply it by a scalar:
import numpy as np
import math
cimport numpy as np
def foo():
N = 10
cdef np.double_t [:, :] A = np.ones(shape=(N,N),dtype=np.double_)
cdef int i,j
cdef double pi = math.pi
for i in range(N):
for j in range(N):
A[i,j] *= pi
return A
def bar():
N = 10
cdef np.double_t [:, :] A = np.ones(shape=(N,N),dtype=np.double_)
cdef double pi = math.pi
A *= pi
return A
Function foo() does this task but is not very convenient/readable.
The line A *= pi in function bar() does however not compile: Invalid operand types for '*' (double_t[:, :]; double).
Is there a way to perform such a broadcasting operation on a cython memoryview?
No, memoryviews don't do this. A memoryview is literally just a way to access individual elements of an array quickly. It has no really concept of the mathematical operations that can be performed on the array.
In the case of your bar function, any attempt to type it is probably actually going to make it worse (i.e. it'll spend extra time checking the type, but ultimately the work is done in ordinary calls to Numpy function).
There's a number of (not 100% satisfactory) ways of getting a Numpy array from a memoryview:
np.asarray(memview) - this should be done without copying (provided you aren't using the esoteric indirect memory layout). It might be worth adding an assertion to check that no copy was made though.
memview.base - be slightly careful with this. If the memoryview is a result of slicing then .base will be the original unsliced object.
Keep a parallel numpy array and memoryview variable:
Anp = np.array(...)
cdef double[:] Amview = Anp
because the memoryview is a view of some memory, modifications to the array will be reflected in the memoryview and vice-versa. (Reassigning the array variable, e.g. Anp = something_else, won't be reflected though).
In summary, memoryviews are designed for one main job: being able to access individual elements quickly. If that's not what you're doing then you probably don't want to use a memoryview.
I want to implement an operation on two matrices that is similar to matrix multiplication in that it each element of the resulting matrix is a function of the ith row of the first matrix and the jth column of the second matrix.
I would like to be able to do this using numpy and/or pandas using vectorized computations.
In other words:
How do I implemenent $A \bigotimes B = C$
where $C_{ij} = sum_k f(a_{ik}, b_{kj})$ in numpy and/or pandas?
I created a 3 dimensional object using numpy.random module such as
import numpy as np
b = np.random.randn(4,4,3)
Why can't we cast type float to b?
TypeError
actual code
You can't float(b) because b isn't a number, it's a multidimensional array/matrix. If you're trying to convert every element to a Python float, that's a bad idea because numpy numbers are more precise, but if you really want to do that for whatever reason, you can do b.tolist(), which returns a Python list of floats. However, I don't believe you can have a numpy matrix of native Python types because that doesn't make any sense.
I am very new to Cython, yet am already experiencing extraordinary speedups just copying my .py to .pyx (and cimport cython, numpy etc) and importing into ipython3 with pyximport.
Many tutorials start in this approach with the next step being to add cdef declarations for every data type, which I can do for the iterators in my for loops etc.
But unlike most Pandas Cython tutorials or examples I am not apply functions so to speak, more manipulating data using slices, sums and division (etc).
So the question is: Can I increase the speed at which my code runs by stating that my DataFrame only contains floats (double), with columns that are int and rows that are int?
How to define the type of an embedded list? i.e [[int,int],[int]]
Here is an example that generates the AIC score for a partitioning of a DF, sorry it is so verbose:
cimport cython
import numpy as np
cimport numpy as np
import pandas as pd
offcat = [
"breakingPeace",
"damage",
"deception",
"kill",
"miscellaneous",
"royalOffences",
"sexual",
"theft",
"violentTheft"
]
def partitionAIC(EmpFrame, part, OffenceEstimateFrame, ReturnDeathEstimate=False):
"""EmpFrame is DataFrame of ints, part is nested list of ints, OffenceEstimate frame is DF of float"""
"""partOf/block is a list of ints"""
"""ll, AIC, is series/frame of floats"""
##Cython cdefs
cdef int DFlen
cdef int puns
cdef int DeathPun
cdef int k
cdef int pId
cdef int punish
DFlen = EmpFrame.shape[1]
puns = 2
DeathPun = 0
PartitionModel = pd.DataFrame(index = EmpFrame.index, columns = EmpFrame.columns)
for partOf in part:
Grouping = [puns*x + y for x in partOf for y in list(range(0,puns))]
PartGroupSum = EmpFrame.iloc[:,Grouping].sum(axis=1)
for punish in range(0,puns):
PunishGroup = [x*puns+punish for x in partOf]
punishPunishment = ((EmpFrame.iloc[:,PunishGroup].sum(axis = 1) + 1/puns).div(PartGroupSum+1)).values[np.newaxis].T
PartitionModel.iloc[:,PunishGroup] = punishPunishment
PartitionModel = PartitionModel*OffenceEstimateFrame
if ReturnDeathEstimate:
DeathProbFrame = pd.DataFrame([[part]], index=EmpFrame.index, columns=['Partition'])
for pId,block in enumerate(part):
DeathProbFrame[pId] = PartitionModel.iloc[:,block[::puns]].sum(axis=1)
DeathProbFrame = DeathProbFrame.apply(lambda row: sorted( [ [format("%6.5f"%row[idx])]+[offcat[X] for X in x ]
for idx,x in enumerate(row['Partition'])],
key=lambda x: x[0], reverse=True),axis=1)
ll = (EmpFrame*np.log(PartitionModel.convert_objects(convert_numeric=True))).sum(axis=1)
k = (len(part))*(puns-1)
AIC = 2*k-2*ll
if ReturnDeathEstimate:
return AIC, DeathProbFrame
else:
return AIC
My advice is to do as much as possible in pandas. This is kinda standard advice "get it working first, then care about performance if it really matters". So let's suppose you've done that (hopefully you've written some tests too), and it's too slow:
Profile your code. (See this SO answer, or use %prun in ipython).
The output of prun should drive what bit to improve next.
pandas (make your code more pandorable, this can help a lot).
numpy (not creating intermediary Series/DataFrames, being careful about dtypes)
cython (the last resort).
Now, if it is a line to do with slicing (it probably isn't) put that tiny part in cython, I like to remove single python function calls to cython function. On that point stuff with cython should use numpy not pandas, I don't think pandas is not going to lower to C (cython can't infer types).
Putting your entire code into cython won't actually help that much, you want to only put the specific lines, or function calls, which are performance sensitive. Keeping cython focussed is the only way to have a good time.
Read the enhancing performance section of the pandas docs*! Here this process (prun -> cythonize -> type) is gone over step-by-step with a real-life example.
*Full-disclose I wrote it that section of the docs! :)