numpy.corrcoeff() MemoryError - numpy
Can't understand MemoryError I get using numpy.corrcoeff() to find correlation coefficient between 2 vectors smin & smax as following:
import numpy as np
from numpy import random as rn
r=0.01
sigma=0.2
T=1
K=1
N=252
h=T/N
M = 50000
Z = rn.randn(M,N)
S=np.ones((M,N+1))
smax=np.ones((M,1))
smin=np.ones((M,1))
for i in range(0,N):
S[:,i+1]=S[:,i]*(np.exp((r-(sigma**2)/2)*h+sigma*Z[:,i]*np.sqrt(h)))
for j in range(0,M):
smax[j,:]=np.exp(-r*T)*(np.max(S[j,:])>K)*(np.max(S[j,:])-K)
smin[j,:]=np.exp(-r*T)*(np.min(S[j,:])<K)*(K-np.min(S[j,:]))
c=np.corrcoef(smax,smin)
print(c)
if there is another way to find correlation coeff.,like using pandas it's also good.
The shape of your arrays here is what is the problem. The function documentation states that x is a "1-D or 2-D array containing multiple variables and observations. Each row of x represents a variable, and each column a single observation of all those variables." and that y is an additional set of variables and observations. So this is trying to allocate an array of size (10000, 10000), which is huge.
If you just want to calculate the pearson correlation coefficient between two one dimensional vectors, you can use a much simpler formula than what is implemented here. This documentation has the formula I am referring to.
https://hydroerr.readthedocs.io/en/stable/api/HydroErr.HydroErr.pearson_r.html#HydroErr.HydroErr.pearson_r
But to be able to still use the numpy version you need to pass in the observations and predictions in the same parameter x, and x and y need to be 1D arrays.
import numpy as np
simulated_array = np.random.rand(50000)
observed_array = np.random.rand(50000)
c = np.corrcoef([simulated_array, observed_array])[1, 0]
More explanation about this here.
Related
How to detect multivariate outliers within large dataset?
How do I detect multivariate outliers within large data with more than 50 variables. Do i need to plot all of the variables or do i have to group them based independent and dependent variables or do i need an algorithm for this?
We do have a special type of distance formula that we use to find multivariate outliers. It is called Mahalanobis Distance. The MD is a metric that establishes the separation between a distribution D and a data point x by generalizing the z-score, the MD indicates how far x is from the D mean in terms of standard deviations. You can use the below function to find out outliers. It returns the index of outliers. from scipy.stats import chi2 import scipy as sp import numpy as np def mahalanobis_method(df): #M-Distance x_minus_mean = df - np.mean(df) cov = np.cov(df.values.T) #Covariance inv_covmat = sp.linalg.inv(cov) #Inverse covariance left_term = np.dot(x_minus_mean, inv_covmat) mahal = np.dot(left_term, x_minus_mean.T) md = np.sqrt(mahal.diagonal()) #Flag as outliers outliers = [] #Cut-off point C = np.sqrt(chi2.ppf((1-0.001), df=df.shape[1])) #degrees of freedom = number of variables for i, v in enumerate(md): if v > C: outliers.append(i) else: continue return outliers, md If you want to study more about Mahalanobis Distance and its formula you can read this blog. So, how to understand the above formula? Let’s take the (x – m)^T . C^(-1) term. (x – m) is essentially the distance of the vector from the mean. We then divide this by the covariance matrix (or multiply by the inverse of the covariance matrix). If you think about it, this is essentially a multivariate equivalent of the regular standardization (z = (x – mu)/sigma).
To find an inverse matrix of A with LU decomposition
The task asks me to generate A matrix with 50 columns and 50 rows with a random library of seed 1007092020 in the range [0,1]. import numpy as np np.random.seed(1007092020) A = np.random.randint(2, size=(3,3)) Then I have to find an inverse matrix of A with LU decomposition. No idea how to do that.
If you need matrix A to be a 50 x 50 matrix with random floating numbers, then you can make that with the following code : import numpy as np np.random.seed(1007092020) A = np.random.random((50,50)) Instead, if you want integers in the range 0,1 (1 included), you can do this A = np.random.randint(0,2,(50,50)) If you want to compute the inverse using LU decomposition, you can use SciPy. It should be noted that since you are generating random matrices, it is possible that your matrix does not have an inverse. In that case, you can not find the inverse. Here's some code that will work in case A does have an inverse. from scipy.linalg import lu p,l,u = lu(A, permute_l = False) Now that we have the lower (l) and upper (u) triangular matrices, we can find the inverse of A by the following equation : A^-1 = U^-1 L^-1 l = np.dot(p,l) l_inv = np.linalg.inv(l) u_inv = np.linalg.inv(u) A_inv = np.dot(u_inv,l_inv)
Efficient way to calculate the pairwise matrix product between one tensor and all the rolling of another tensor
Suppose we have two tensors:
tensor A whose shape is (d,m,n)
tensor B whose shape is (d,n,l).
If we want to get the pairwise matrix product of the right-most matrix of A and B, I think we can use np.einsum('dmn,...nl->d...ml',A,B) whose size is (d,d,m,l). However, I would like to get the pairwise product of not all the pairs.
Import a parameter k, 1<=k<=d, I want to get the following pairwise matrix product:
from
A(0,...)#B(0,...)
to
A(0,...)#B(k-1,...)
;
from
A(1,...)#B(1,...)
to
A(1,...)#B(k,...)
;
....
;
from
A(d-2,...)#B(d-2,...),
A(d-2,...)#B(d-1,...)
to
A(d-2,...)#B(k-3,...)
;
from
A(d-1,...)#B(d-1,...)
to
A(d-1,...)#B(k-2,...)
.
Note here we we use a rolling way to deal with tensor B. (like numpy.roll).
Finally, we actually get a tensor whose shape is (d,k,m,l).
What's the most efficient way to do this.
I know several ways like:
First get np.einsum('dmn,...nl->d...ml',A,B), then use a mask to extract the (d,k) pairs.
tile B first, then use einsum in some way.
But I think there exists a better way.
I doubt you can do much better than a for loop. Here is, for example, a vectorized version using einsum and stride_tricks compared to a double for loop:
Code:
from simple_benchmark import BenchmarkBuilder, MultiArgument
import numpy as np
from numpy.lib.stride_tricks import as_strided
B = BenchmarkBuilder()
#B.add_function()
def loopy(A,B,k):
d,m,n = A.shape
l = B.shape[-1]
out = np.empty((d,k,m,l),int)
for i in range(d):
for j in range(k):
out[i,j] = A[i]#B[(i+j)%d]
return out
#B.add_function()
def vectory(A,B,k):
d,m,n = A.shape
l = B.shape[-1]
BB = np.concatenate([B,B[:k-1]],0)
BB = as_strided(BB,(d,k,n,l),np.repeat(BB.strides,(2,1,1)))
return np.einsum("ikl,ijln->ijkn",A,BB)
#B.add_arguments('d x k x m x n x l')
def argument_provider():
for exp in range(10):
d,k,m,n,l = (np.r_[1.6,1.5,1.5,1.5,1.5]**exp*(4,2,2,2,2)).astype(int)
print(d,k,m,n,l)
A = np.random.randint(0,10,(d,m,n))
B = np.random.randint(0,10,(d,n,l))
yield k*d*m*n*l,MultiArgument([A,B,k])
r = B.run()
r.plot()
import pylab
pylab.savefig('diagwa.png')
Python: AttributeError: "'numpy.float64' object has no attribute 'tanh'"
I have seen couple of questions with similar title, however I am afraid, none of them could satisfactorily answer my question and that is, how do I take tan inverse or lets say exp of a numpy ndarray? For instance, piece of my code looks similar to this- import numpy as np from numpy import ndarray,zeros,array,dot,exp import itertools def zetta_G(x,spr_g,theta_g,c_g): #this function computes estimated g: #c_g is basically a matrix of dim equal to g and whose elements contains list of centers that describe the fuzzy system for each element of g: m,n=c_g.shape[0],c_g.shape[1] #creating an empty matrix of dim mxn to hold regressors: zetta_g=zeros((m,n),dtype=ndarray) #creating an empty matrix of dim mxn to hold estimated g: z_g=np.zeros((m,n),dtype=ndarray) #for filling rows for k in range(m): #for filling columns for p in range(n): #container to hold-length being equal to number of inputs(e1,e2,e3 etc) Mu=[[] for i in range(len(x))] for i in range(len(x)): #filling that with number of zeros equal to len of center Mu[i]=np.zeros(len(c_g[k][p])) #creating an empty list for holding rules M=[] #piece of code for creating rules-all possible combinations for i in range(len(x)): for j in range(len(c_g[k][p])): Mu[i][j]=exp(-.5*((x[i]-c_g[k][p][j])/spr_g[k][p])**2) b=list(itertools.product(*Mu)) for i in range(len(b)): M.append(reduce(lambda x,y:x*y,b[i])) M=np.array(M) S=np.sum(M) #import pdb;pdb.set_trace() zetta_g[k][p]=M/S z_g[k][p]=dot(M/S,theta_g[k][p]) return zetta_g,z_g if __name__=='__main__': x=[1.2,.2,.4] cg11,cg12,cg13,cg21,cg22,cg23,cg31,cg32,cg33=[-10,-8,-6,-4,-2,0,2,4,6,8,10],[-10,-8,-6,-4,-2,0,2,4,6,8,10],[-10,-8,-6,-4,-2,0,2,4,6,8,10],[-10,-8,-6,-4,-2,0,2,4,6,8,10],[-10,-8,-6,-4,-2,0,2,4,6,8,10],[-12,-9,-6,-3,0,3,6,9,12],[-6.5,-4.5,-2.5,0,2.5,4.5,6.5],[-5,-4,-3,-2,-1,0,1,2,3,4,5],[-3.5,-2.5,-1.5,0,1.5,2.5,3.5] C,spr_f=array([[-10,-8,-6,-4,-2,0,2,4,6,8,10],[-10,-8,-6,-4,-2,0,2,4,6,8,10],[-10,-8,-6,-4,-2,0,2,4,6,8,10]]),[2.2,2,2.1] c_g=array([[cg11,cg12,cg13],[cg21,cg22,cg23],[cg31,cg32,cg33]]) spr_g=array([[2,2.1,2],[2.1,2.2,3],[2.5,1,1.5]]) theta_g=np.zeros((c_g.shape[0],c_g.shape[1]),dtype=ndarray) #import pdb;pdb.set_trace() N=0 for i in range(c_g.shape[0]): for j in range(c_g.shape[1]): length=len(c_g[i][j])**len(x) theta_g[i][j]=np.random.sample(length) N=N+(len(c_g[i][j]))**len(x) zetta_g,z_g=zetta_G(x,spr_g,theta_g,c_g) #zetta is a function that accepts following args-- x: which is a list of certain dim, spr_g: is a matrix of dimension similar to theta_g and c_g. theta_g and c_g are numpy matrices with lists as individual elements print(zetta_g) print(z_g) inv=np.tanh(z_g) print(inv)
In [89]: a=np.array([[1],[3],[2]],dtype=np.ndarray) In [90]: a Out[90]: array([[1], [3], [2]], dtype=object) Note that the dtype is object, not ndarray. If the dtype isn't one of the recognized numeric or string types, it is object, a generic pointer, just like the elements of a list. In [91]: np.tanh(a) AttributeError: 'int' object has no attribute 'tanh' np.tanh is trying to delegate the task to the elements of array. Commonly math on object dtype arrays is performed by list like iteration on the elements. It does not do the fast compiled numeric numpy math. If a is ordinary number array: In [95]: np.tanh(np.array([[1],[3],[2]])) Out[95]: array([[0.76159416], [0.99505475], [0.96402758]]) With object dtype arrays, your ability to do numeric calculations is limited. Some things work, others don't. It's hit-or-miss. Here's a first stab at cleaning up your code; it's not tested. def zetta_G(x,spr_g,theta_g,c_g): m,n=c_g.shape[0],c_g.shape[1] #creating an empty matrix of dim mxn to hold regressors: zetta_g=zeros((m,n),dtype=object) #creating an empty matrix of dim mxn to hold estimated g: z_g=np.zeros((m,n),dtype=object) #for filling rows for k in range(m): #for filling columns for p in range(n): #container to hold-length being equal to number of inputs(e1,e2,e3 etc) Mu = np.zeros((len(x), len(c_g[k,p]))) #creating an empty list for holding rules for i in range(len(x)): Mu[i,:]=exp(-.5*((x[i]-c_g[k,p,:])/spr_g[k,p])**2) # probably can calc Mu without any loop M = [] b=list(itertools.product(*Mu)) for i in range(len(b)): M.append(reduce(lambda x,y:x*y,b[i])) M=np.array(M) S=np.sum(M) zetta_g[k,p]=M/S z_g[k,p]=dot(M/S,theta_g[k,p]) return zetta_g,z_g Running your code, and adding some .shape displays I see that z_g is (3,3) and contains just single numbers. So it can be initialed as a plain 2d float array: z_g=np.zeros((m,n)) theta_g is (3,3), but with variable length array elements print([i.shape for i in theta_g.flat]) [(1331,), (1331,), (1331,), (1331,), (1331,), (729,), (343,), (1331,), (343,)] zetta_g matches in shapes If I change: x=np.array([1.2,.2,.4]) I can calculate Mu without a loop with: Mu = exp(-.5*((x[:,None]-np.array(c_g[k,p])[None,:])/spr_g[k,p])**2) c_g is a (3,3) array with variable length lists; I can vectorize the ((x[i]-c_g[k,p][j]) expression with: x[:,None]-np.array(c_g[k,p])[None,:] Not a big time saver here since x has 4 elements and c_g elements are only 7-11 long. But cleaner. In this running code I don't see a tanh, so I don't know what kinds of arrays are using that.
You set type of array's elements to dtype=np.ndarray. Replace type to, let say, dtype=np.float64 or any numeric type.
Numpy / Scipy - Sparse matrix to vector
I have sparse CSR matrices (from a product of two sparse vector) and I want to convert each matrix to a flat vector. Indeed, I want to avoid using any dense representation or iterating over indexes. So far, the only solution that came up was to iterate over non null elements by using coo representation: import numpy from scipy import sparse as sp matrices = [sp.csr_matrix([[1,2],[3,4]])]*3 vectorSize = matrices[0].shape[0]*matrices[0].shape[1] flatMatrixData = [] flatMatrixRows = [] flatMatrixCols = [] for i in range(len(matrices)): matrix = matrices[i].tocoo() flatMatrixData += matrix.data.tolist() flatMatrixRows += [i]*matrix.nnz flatMatrixCols += [r+c*2 for r,c in zip(matrix.row, matrix.col)] flatMatrix = sp.coo_matrix((flatMatrixData,(flatMatrixRows, flatMatrixCols)), shape=(len(matrices), vectorSize), dtype=numpy.float64).tocsr() It is indeed unsatisfying and inelegant. Does any one know how to achieve this in an efficient way?
Your flatMatrix is (3,4); each row is [1 3 2 4]. If a submatrix is x, then the row is x.A.T.flatten(). F = sp.vstack([x.T.tolil().reshape((1,vectorSize)) for x in matrices]) F is the same (dtype is int). I had to convert each submatrix to lil since csr has not implemented reshape (in my version of sparse). I don't know if other formats work. Ideally sparse would let you do the whole range of numpy array (or matrix) manipulations, but it isn't there yet. Given the small dimensions in this example, I won't speculate on the speed of the alternatives.