Matrix associativity in tensorflow tf.matmul issues - numpy
We have a matrix M with shape (n, n) and a matrix C with shape (n, n). In this example, both matrices are generate using numpy's np.random.normal(). We are encountering some unexpected behaviour in the associativity property of matrix multiplication in both numpy and tensorflow. This is causing my model (written in tensorflow) to give some unexpected errors when computing the Cholesky decomposition of covariance matrices.
Below is a simple working example of this (the model actually uses 3D and 4D tensors):
import numpy as np
import tensorflow as tf
# Generate random matrices sampled from normal distribution
M = np.random.normal(0, 0.1, size=(5, 5))
C = np.random.normal(0, 0.1, size=(5, 5))
First we check numpy output:
r1 = np.matmul(np.matmul(M, C),np.transpose(M))
r2 = np.matmul(M, np.matmul(C, np.transpose(M)))
r3 = (M # C) # np.transpose(M)
r4 = M # (C # np.transpose(M))
print("r1 == r2: ", np.all(r1 == r2))
print("r1 == r3: ", np.all(r1 == r3))
print("r1 == r4: ", np.all(r1 == r4))
print("r2 == r3: ", np.all(r2 == r3))
print("r2 == r4: ", np.all(r2 == r4))
print("r3 == r4: ", np.all(r3 == r4))
We get the following result:
Numpy's result.
The we check tensorflow output:
# Check tensorflow
t1 = tf.linalg.matmul(tf.linalg.matmul(M, C), M, transpose_b = True)
t2 = tf.linalg.matmul(M, tf.linalg.matmul(C, M, transpose_b = True))
print("t1 == t2: ", np.all(t1.numpy() == t2.numpy()))
print(t1.numpy() - t2.numpy())
We get the following result:
Tensorflow's results.
I found a similar thread describing a similar issue here, but with no answer.
Do you think that differences in the order of approximately e-19 can have such an impact that causes these weight matrices to not have a Cholesky decomposition?
Thanks in advance.
Related
Align the Truncated SVD from sklearn.decomposition and np.linalg.svd
=========update========== I read an infomation in this book: The matrix that is actually returned by TruncatedSVD is the dot product of the U andS matrices. Then i try to just multiply U and Sigma: US = U.dot(Sigma) print("==>> US: ", US) this time it produce the same result, just with sign flipping. So why Truncated SVD doesn't need multiplying VT ? ==========previous question=========== I am learning SVD, i found numpy and sklearn both provide some related APIs, then i try to use them to do dimensional reduction, below are the code: import numpy as np np.set_printoptions(precision=2, suppress=True) A = np.array([ [1,1,1,0,0], [3,3,3,0,0], [4,4,4,0,0], [5,5,5,0,0], [0,2,0,4,4], [0,0,0,5,5], [0,1,0,2,2]]) U, s, VT = np.linalg.svd(A) print("==>> U: ", U) print("==>> VT: ", VT) # create m x n Sigma matrix Sigma = np.zeros((A.shape[0], A.shape[1])) # populate Sigma with n x n diagonal matrix square_len = min((A.shape[0], A.shape[1])) Sigma[:square_len, :square_len] = np.diag(s) print("==>> Sigma: ", Sigma) n_elements = 2 U = U[:, :n_elements] Sigma = Sigma[:n_elements, :n_elements] VT = VT[:n_elements, :n_elements] # reconstruct B = U.dot(Sigma.dot(VT)) print("==>> B: ", B) The output B is : ==>> B: [[ 0.99 1.01] [ 2.98 3.04] [ 3.98 4.05] [ 4.97 5.06] [ 0.36 1.29] [-0.37 0.73] [ 0.18 0.65]] then this is sklearn code: import numpy as np from sklearn.decomposition import TruncatedSVD A = np.array([ [1,1,1,0,0], [3,3,3,0,0], [4,4,4,0,0], [5,5,5,0,0], [0,2,0,4,4], [0,0,0,5,5], [0,1,0,2,2]]).astype(float) svd = TruncatedSVD(n_components=2) svd.fit(A) # Fit model on training data A print("==>> right singular vectors: ", svd.components_) print("==>> svd.singular_values_: ", svd.singular_values_) B = svd.transform(A) # Perform dimensionality reduction on A. print("==>> B: ", B) its last output result is: ==>> B: [[ 1.72 -0.22] [ 5.15 -0.67] [ 6.87 -0.9 ] [ 8.59 -1.12] [ 1.91 5.62] [ 0.9 6.95] [ 0.95 2.81]] As we can see, they produce different result (but i notice their singular values are the same, both are 12.48 9.51), how to make them same, does i misunderstand something ?
I think the correct way to perform a dimensionality reduction of the array A with np.linalg.svd is: U, s, V = np.linalg.svd(A) VT = V.T B = A#VT[:,:n_elements] Now B is: array([[-1.72, 0.22], [-5.15, 0.67], [-6.87, 0.9 ], [-8.59, 1.12], [-1.91, -5.62], [-0.9 , -6.95], [-0.95, -2.81]]) That is exactly what you get from the TruncatedSVD, but with negative sign.
What is wrong with my cython implementation of erosion operation of mathematical morphology
I have produced a naive implementation of "erosion". The performance is not relevant since I just trying to understand the algorithm. However, the output of my implementation does not match the one I get from scipy.ndimage. What is wrong with my implementation ? Here is my implementation with a small test case: import numpy as np from PIL import Image # a small image to play with a cross structuring element imgmat = np.array([ [0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0], [0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0], [0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,1,1,1,1,1,1,0,0,0,0,0,0,0,0,0], [0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,1,0,0,0,1,0,0,0,0,0,0,0,0,0], [0,0,0,0,0,1,1,0,0,1,1,1,1,0,0,0,1,1,0,0,1,0,0,0,0,0,0,0,0,0], [0,0,0,0,0,1,1,0,0,1,1,1,1,0,0,0,1,1,1,0,1,0,0,0,0,0,0,0,0,0], [0,0,0,0,0,1,1,0,0,1,1,1,1,0,0,0,1,1,1,1,1,0,0,0,0,0,0,0,0,0], [0,0,0,0,0,1,1,0,0,1,1,1,1,0,0,0,1,1,1,1,1,0,0,0,0,0,0,0,0,0], [0,0,0,0,0,1,1,0,0,1,1,1,1,0,0,0,0,1,1,1,1,0,0,0,0,0,0,0,0,0], [0,0,0,0,0,1,1,0,0,1,1,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0], [0,0,0,0,0,1,1,0,0,1,1,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0], [0,0,0,0,0,1,1,0,0,0,0,0,0,0,0,1,1,1,1,1,0,0,1,0,1,1,0,0,0,0], [0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,1,0,0,1,0,1,1,0,0,0,0], [0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,1,0,1,1,0,0,0,0], [0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,1,0,1,1,0,0,0,0], [0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,1,1,1,1,0,0,0,0], ]) imgmat2 = np.where(imgmat == 0, 0, 255).astype(np.uint8) imarr = Image.fromarray(imgmat2).resize((100, 200)) imarr = np.array(imgrrr) imarr = np.where(imarr == 0, 0, 1) se_mat3 = np.array([ [0,1,0], [1,1,1], [0,1,0] ]) se_mat31 = np.where(se_mat3 == 1, 0, 1) The imarr is . My implementation of erosion: %%cython -a import numpy as np cimport numpy as cnp cdef erosionC(cnp.ndarray[cnp.int_t, ndim=2] img, cnp.ndarray[cnp.int_t, ndim=2] B, cnp.ndarray[cnp.int_t, ndim=2] X): """ X: image coordinates struct_element_mat: black and white image, black region is considered as the shape of structuring element This operation checks whether (B *includes* X) = $B \subset X$ as per defined in Serra (Jean), « Introduction to mathematical morphology », Computer Vision, Graphics, and Image Processing, vol. 35, nᵒ 3 (septembre 1986). URL : https://linkinghub.elsevier.com/retrieve/pii/0734189X86900022.. doi: 10.1016/0734-189X(86)90002-2 Consulted le 6 août 2020, p. 283‑305. """ cdef cnp.ndarray[cnp.int_t, ndim=1] a, x, bx cdef cnp.ndarray[cnp.int_t, ndim=2] Bx, B_frame, Xcp, b cdef bint check a = B[0] # get an anchor point from the structuring element coordinates B_frame = B - a # express the se element coordinates in with respect to anchor point Xcp = X.copy() b = img.copy() for x in X: # X contains the foreground coordinates in the image Bx = B_frame + x # translate relative coordinates with respect to foreground coordinates considering it as the anchor point check = True # this is erosion so if any of the se coordinates is not in foreground coordinates we consider it a miss for bx in Bx: # Bx contains all the translated coordinates of se if bx not in Xcp: check = False if check: b[x[0], x[1]] = 1 # if there is a hit else: b[x[0], x[1]] = 0 # if there is no hit return b def erosion(img: np.ndarray, struct_el_mat: np.ndarray, foregroundValue = 0): B = np.argwhere(struct_el_mat == 0) X = np.argwhere(img == foregroundValue) nimg = erosionC(img, B, X) return np.where(nimg == 1, 255, 0) The calling code for both is: from scipy import ndimage as nd err = nd.binary_erosion(imarr, se_mat3) imerrCustom = erosion(imarr, se_mat31, foregroundValue=1) err produces imerrCustom produces
In the end, I am still not sure about it, but after having read several papers more, I assume that my interpretation of X as foreground coordinates was an error. It should have probably been the entire image that is being iterated. As I have stated I am not sure if this interpretation is correct as well. But I made a new implementation which iterates over the image, and it gives a more plausible result. I am sharing it in here, hoping that it might help someone: %%cython -a import numpy as np cimport numpy as cnp cdef dilation_c(cnp.ndarray[cnp.uint8_t, ndim=2] X, cnp.ndarray[cnp.uint8_t, ndim=2] SE): """ X: boolean image SE: structuring element matrix origin: coordinate of the origin of the structuring element This operation checks whether (B *hits* X) = $B \cap X \not = \emptyset$ as per defined in Serra (Jean), « Introduction to mathematical morphology », Computer Vision, Graphics, and Image Processing, vol. 35, nᵒ 3 (septembre 1986). URL : https://linkinghub.elsevier.com/retrieve/pii/0734189X86900022.. doi: 10.1016/0734-189X(86)90002-2 Consulted le 6 août 2020, p. 283‑305. The algorithm adapts DILDIRECT of Najman (Laurent) et Talbot (Hugues), Mathematical morphology: from theory to applications, 2013. ISBN : 9781118600788, p. 329 to the formula given in Jähne (Bernd), Digital image processing, 6th rev. and ext. ed, Berlin ; New York, 2005. TA1637 .J34 2005. ISBN : 978-3-540-24035-8. """ cdef cnp.ndarray[cnp.uint8_t, ndim=2] O cdef list elst cdef int r, c, X_rows, X_cols, SE_rows, SE_cols, se_r, se_c cdef cnp.ndarray[cnp.int_t, ndim=1] bp cdef list conds cdef bint check, b, p, cond O = np.zeros_like(X) X_rows, X_cols = X.shape[:2] SE_rows, SE_cols = SE.shape[:2] # a boolean convolution for r in range(0, X_rows-SE_rows): for c in range(0, X_cols - SE_cols): conds = [] for se_r in range(SE_rows): for se_c in range(SE_cols): b = <bint>SE[se_r, se_c] p = <bint>X[se_r+r, se_c+c] conds.append(b and p) O[r,c] = <cnp.uint8_t>any(conds) return O def dilation_erosion( img: np.ndarray, struct_el_mat: np.ndarray, foregroundValue: int = 1, isErosion: bool = False): """ img: image matrix struct_el: NxN mesh grid of the structuring element whose center is SE's origin structuring element is encoded as 1 foregroundValue: value to be considered as foreground in the image """ B = struct_el_mat.astype(np.uint8) if isErosion: X = np.where(img == foregroundValue, 0, 1).astype(np.uint8) else: X = np.where(img == foregroundValue, 1, 0).astype(np.uint8) nimg = dilation_c(X, B) foreground, background = (255, 0) if foregroundValue == 1 else (0, 1) if isErosion: return np.where(nimg == 1, background, foreground).astype(np.uint8) else: return np.where(nimg == 1, foreground, background).astype(np.uint8) # return nimg
Slow computation on google colab while solving partial differential equation
I 'm using google colab to solve the homogeneous heat equation. I had made a program earlier with scipy using sparse matrices which worked upto N = 10(hyperparameter) but I need to run it for like N = 4... 1000 and thus it won't work on my pc. I therefore converted the code to tensorflow and here I 'm unable to use sparse matrices like I could in sympy but even the GPU/TPU computation is also slow and slower than my pc. Problems that I'm facing in the code and require solution for 1) tf.contrib is removed and thus I 've to use an older version of tensorflow for odeint function. Where is it in 2.0? 2)If the computation can be computed with sparse matrices it could be good since matrices are tridiagonal.I know about sparse_dense_mul() function but that returns dense tensor and it wouldn't do the job. The "func" function applies time independent boundary conditions and then requires matrix multiplication of (nxn) with (nX1) which gives (nX1) with multiple matrices. Also the program was running faster without I created the class. Also it's giving this WARNING: Logging before flag parsing goes to stderr. W0829 09:12:24.415445 139855355791232 lazy_loader.py:50] The TensorFlow contrib module will not be included in TensorFlow 2.0. For more information, please see: * https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md * https://github.com/tensorflow/addons * https://github.com/tensorflow/io (for I/O related ops) If you depend on functionality not listed there, please file an issue. W0829 09:12:24.645356 139855355791232 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensorflow/contrib/integrate/python/ops/odes.py:233: div (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Deprecated in favor of operator or tf.math.divide. when I run code for loop in range(2, 10) and tqdm does not display and cell keeps running forever but it works fine for in (2, 5) and tqdm bar does appears. #find a way to use sparse matrices class Heat: def __init__(self, N): self.N = N self.H = 1/N self.A = ts.to_dense(ts.SparseTensor(indices=[[0, 0], [0, 1]] + \ [[i, i+j] for i in range(1, N) for j in [-1, 0, 1]] +[[N, N-1], [N, N]], values=self.H*np.array([1/3, 1/6] + [1/6, 2/3, 1/6]*(N-1) + [1/6, 1/3], dtype=np.float32), dense_shape=(N+1, N+1 ))) self.D = ts.to_dense(ts.SparseTensor(indices=[[0, 0], [0, 1]] + [[i, i+j] \ for i in range(1, N) for j in [-1, 0, 1]] +[[N, N-1], [N, N]], values=N*np.array([1-(1), -1 -(-1)] + [-1, 2, -1]*(N-1) + [-1-(-1), 1-(1)], dtype=np.float32), dense_shape=(N+1, N+1))) self.domain = tf.linspace(0.0, 1.0, N+1) def f(k): if k == 0: return (1 + math.pi**2)*(math.pi*self.H - math.sin(math.pi*self.H))/(math.pi**2*self.H) elif k == N: return -(1 + math.pi**2)*(-math.pi*self.H + math.sin(math.pi*self.H))/(math.pi**2*self.H) else: return -2*(1 + math.pi**2)*(math.cos(math.pi*self.H) - 1)*math.sin(math.pi*self.H*k)/(math.pi**2*self.H) self.F = tf.constant([f(k) for k in range(N+1)], shape=(N+1,), dtype=tf.float32) #caution! shape changed caution caution 1, N+1(problem) is different from N+1, self.exact = tm.scalar_mul(scalar=np.exp(1), x=tf.sin(math.pi*self.domain)) def error(self): return np.linalg.norm(self.exact.numpy() - self.approx, 2) def func (self, y, t): y = tf.Variable(y) y = y[0].assign(0.0) y = y[self.N].assign(0.0) if self.N**2> 100: y_dash = tl.matvec(tf.linalg.inv(self.A), tl.matvec(a=tm.negative(self.D), b=y, a_is_sparse=True) + tm.scalar_mul(scalar=math.exp(t), x=self.F)) #caution! shape changed F is (1, N+1) others too else: y_dash = tl.matvec(tf.linalg.inv(self.A), tl.matvec(a=tm.negative(self.D), b=y) + tm.scalar_mul(scalar=math.exp(t), x=self.F)) #caution! shape changed F is (1, N+1) others too y_dash = tf.Variable(y_dash) #!!y_dash performs Hadamard product like multiplication not matrix-like multiplication;returns 2-D y_dash = y_dash[0].assign(0.0) y_dash = y_dash[self.N].assign(0.0) return y_dash def algo_1(self): self.approx = tf.contrib.integrate.odeint( func=self.func, y0=tf.sin(tm.scalar_mul(scalar=math.pi, x=self.domain)), t=tf.constant([0.0, 1.0]), rtol=1e-06, atol=1e-12, method='dopri5', options={"max_num_steps":10**10}, full_output=False, name=None ).numpy()[1] def algo_2(self): self.approx = tf.contrib.integrate.odeint_fixed( func=self.func, y0=tf.sin(tm.scalar_mul(scalar=math.pi, x=self.domain)), t=tf.constant([0.0, 1.0]), dt=tf.constant([self.H**2], dtype=tf.float32), method='rk4', name=None ).numpy()[1] df = pd.DataFrame(columns=["NumBasis", "Errors"]) Ns = [2**r for r in range(2, 10)] l =[] for i in tqdm_notebook(Ns): heateqn = Heat(i) heateqn.algo_1() l.append([i, heateqn.error()]) df.append({"NumBasis":i, "Errors":heateqn.error()}, ignore_index=True) tf.keras.backend.clear_session()
Evaluating the squared term of a gaussian kernel for having a covariance matrix for multi-dimensional inputs [duplicate]
I have the following code. It is taking forever in Python. There must be a way to translate this calculation into a broadcast... def euclidean_square(a,b): squares = np.zeros((a.shape[0],b.shape[0])) for i in range(squares.shape[0]): for j in range(squares.shape[1]): diff = a[i,:] - b[j,:] sqr = diff**2.0 squares[i,j] = np.sum(sqr) return squares
You can use np.einsum after calculating the differences in a broadcasted way, like so - ab = a[:,None,:] - b out = np.einsum('ijk,ijk->ij',ab,ab) Or use scipy's cdist with its optional metric argument set as 'sqeuclidean' to give us the squared euclidean distances as needed for our problem, like so - from scipy.spatial.distance import cdist out = cdist(a,b,'sqeuclidean')
I collected the different methods proposed here, and in two other questions, and measured the speed of the different methods: import numpy as np import scipy.spatial import sklearn.metrics def dist_direct(x, y): d = np.expand_dims(x, -2) - y return np.sum(np.square(d), axis=-1) def dist_einsum(x, y): d = np.expand_dims(x, -2) - y return np.einsum('ijk,ijk->ij', d, d) def dist_scipy(x, y): return scipy.spatial.distance.cdist(x, y, "sqeuclidean") def dist_sklearn(x, y): return sklearn.metrics.pairwise.pairwise_distances(x, y, "sqeuclidean") def dist_layers(x, y): res = np.zeros((x.shape[0], y.shape[0])) for i in range(x.shape[1]): res += np.subtract.outer(x[:, i], y[:, i])**2 return res # inspired by the excellent https://github.com/droyed/eucl_dist def dist_ext1(x, y): nx, p = x.shape x_ext = np.empty((nx, 3*p)) x_ext[:, :p] = 1 x_ext[:, p:2*p] = x x_ext[:, 2*p:] = np.square(x) ny = y.shape[0] y_ext = np.empty((3*p, ny)) y_ext[:p] = np.square(y).T y_ext[p:2*p] = -2*y.T y_ext[2*p:] = 1 return x_ext.dot(y_ext) # https://stackoverflow.com/a/47877630/648741 def dist_ext2(x, y): return np.einsum('ij,ij->i', x, x)[:,None] + np.einsum('ij,ij->i', y, y) - 2 * x.dot(y.T) I use timeit to compare the speed of the different methods. For the comparison, I use vectors of length 10, with 100 vectors in the first group, and 1000 vectors in the second group. import timeit p = 10 x = np.random.standard_normal((100, p)) y = np.random.standard_normal((1000, p)) for method in dir(): if not method.startswith("dist_"): continue t = timeit.timeit(f"{method}(x, y)", number=1000, globals=globals()) print(f"{method:12} {t:5.2f}ms") On my laptop, the results are as follows: dist_direct 5.07ms dist_einsum 3.43ms dist_ext1 0.20ms <-- fastest dist_ext2 0.35ms dist_layers 2.82ms dist_scipy 0.60ms dist_sklearn 0.67ms While the two methods dist_ext1 and dist_ext2, both based on the idea of writing (x-y)**2 as x**2 - 2*x*y + y**2, are very fast, there is a downside: When the distance between x and y is very small, due to cancellation error the numerical result can sometimes be (very slightly) negative.
Another solution besides using cdist is the following difference_squared = np.zeros((a.shape[0], b.shape[0])) for dimension_iterator in range(a.shape[1]): difference_squared = difference_squared + np.subtract.outer(a[:, dimension_iterator], b[:, dimension_iterator])**2.
hessian of a variable returned by tf.concat() is None
Let x and y be vectors of length N, and z is a function z = f(x,y). In Tensorflow v1.0.0, tf.hessians(z,x) and tf.hessians(z,y) both returns an N by N matrix, which is what I expected. However, when I concatenate the x and y into a vector p of size 2*N using tf.concat, and run tf.hessian(z, p), it returns error "ValueError: None values not supported." I understand this is because in the computation graph x,y ->z and x,y -> p, so there is no gradient between p and z. To circumvent the problem, I can create p first, slice it into x and y, but I will have to change a ton of my code. Is there a more elegant way? related question: Slice of a variable returns gradient None import tensorflow as tf import numpy as np N = 2 A = tf.Variable(np.random.rand(N,N).astype(np.float32)) B = tf.Variable(np.random.rand(N,N).astype(np.float32)) x = tf.Variable(tf.random_normal([N]) ) y = tf.Variable(tf.random_normal([N]) ) #reshape to N by 1 x_1 = tf.reshape(x,[N,1]) y_1 = tf.reshape(y,[N,1]) #concat x and y to form a vector with length of 2*N p = tf.concat([x,y],axis = 0) #define the function z = 0.5*tf.matmul(tf.matmul(tf.transpose(x_1), A), x_1) + 0.5*tf.matmul(tf.matmul(tf.transpose(y_1), B), y_1) + 100 #works , hx and hy are both N by N matrix hx = tf.hessians(z,x) hy = tf.hessians(z,y) #this gives error "ValueError: None values not supported." #expecting a matrix of size 2*N by 2*N hp = tf.hessians(z,p)
Compute the hessian by its definition. gxy = tf.gradients(z, [x, y]) gp = tf.concat([gxy[0], gxy[1]], axis=0) hp = [] for i in range(2*N): hp.append(tf.gradients(gp[i], [x, y])) Because tf.gradients computes the sum of (dy/dx), so when computing the second partial derivative, one should slice the vector into scalars and then compute the gradient. Tested on tf1.0 and python2.