I am having difficulty to understand why neither B or C equals A. How can I extract a specific row from f and calculate the same result as in A?
import numpy as np
L = np.array([
[2.66667,1.33333],
[0.8,1.6]
])
f = np.array([[0.5,0.333333]])
A = L*f.T
B = L*f[0,:]
C = L*f[0,:].T
print(20*'*')
print(A)
print(20*'*')
print(B)
print(20*'*')
print(C)
print(20*'*')
Output:
********************
[[1.333335 0.666665 ]
[0.2666664 0.5333328]]
********************
[[1.333335 0.44444289]
[0.4 0.5333328 ]]
********************
[[1.333335 0.44444289]
[0.4 0.5333328 ]]
********************
L = np.array([
[2.66667,1.33333],
[0.8,1.6]
])
f = np.array([[0.5,0.333333]])
L is (2,2), f is (1,2). f.T is (2,1)
A = L*f.T
This broadcasts the (2,2) with (2,1), replicating the the f.T columns to (2,2).
B = L*f[0,:]
C = L*f[0,:].T
f[0,:] is (2,) shape, 1d. f[0,:].T is the same because there's only one 1 axis to switch.
Here (2,2) times (2,) => (2,2) times (1,2), That's the same as L*f.
In A you are multiplying the columns of L by f:
In [235]: L[0,:]*f[0,:]
Out[235]: array([1.333335 , 0.44444289])
In B it's the rows of L:
In [236]: L[:,0]*f[0,:]
Out[236]: array([1.333335 , 0.2666664])
The mix and match of rows and columns could be clearer with L was (2,3) shaped.
edit
In [34]: np.array([L[:,i]*f[0] for i in range(2)]).T
Out[34]:
array([[1.333335 , 0.666665 ],
[0.2666664, 0.5333328]])
In [35]: np.array([L[i,:]*f[0] for i in range(2)])
Out[35]:
array([[1.333335 , 0.44444289],
[0.4 , 0.5333328 ]])
I got [235] and [236] switched.
Related
I want to solve an optimization problem in Julia.
I am trying to define a binary variable x_{r,i}
Thereby, the length of the sets of both indices is not the same.
Let's say there is r_a and r_b, but for r_a there are i_1 and i_2 whereas for r_b there are i_1, i_2 and i_3 so in the end I want to get X_a_1, X_a_2 and X_b_1, X_b_2, X_b_3
The set of indices i varies for different indices r.
Is there any way to define variable x with these indices in Julia?
This is what I tried:
R=["a","b"]
I=Dict("a" => [1,2],"b"=>[1,2,3])
m = Model(CPLEX.Optimizer)
#variables m begin
X[R,[I]], Bin
end
You were on the right track. Create a SparseAxisArray:
julia> R = ["a", "b"]
2-element Vector{String}:
"a"
"b"
julia> I = Dict("a" => [1, 2], "b" => [1, 2, 3])
Dict{String, Vector{Int64}} with 2 entries:
"b" => [1, 2, 3]
"a" => [1, 2]
julia> model = Model();
julia> #variable(model, x[r in R, i in I[r]])
JuMP.Containers.SparseAxisArray{VariableRef, 2, Tuple{String, Int64}} with 5 entries:
[a, 1] = x[a,1]
[a, 2] = x[a,2]
[b, 1] = x[b,1]
[b, 2] = x[b,2]
[b, 3] = x[b,3]
JuMP supports dense arrays with custom indexing.
julia> indices = [Symbol.(:a, 1:2);Symbol.(:b, 1:3)];
julia> #variable(m, x[indices], Bin)
1-dimensional DenseAxisArray{VariableRef,1,...} with index sets:
Dimension 1, [:a1, :a2, :b1, :b2, :b3]
And data, a 5-element Vector{VariableRef}:
x[a1]
x[a2]
x[b1]
x[b2]
x[b3]
=========update==========
I read an infomation in this book:
The matrix that is actually returned by TruncatedSVD is the dot product of the U andS matrices.
Then i try to just multiply U and Sigma:
US = U.dot(Sigma)
print("==>> US: ", US)
this time it produce the same result, just with sign flipping. So why Truncated SVD doesn't need multiplying VT ?
==========previous question===========
I am learning SVD, i found numpy and sklearn both provide some related APIs, then i try to use them to do dimensional reduction, below are the code:
import numpy as np
np.set_printoptions(precision=2, suppress=True)
A = np.array([
[1,1,1,0,0],
[3,3,3,0,0],
[4,4,4,0,0],
[5,5,5,0,0],
[0,2,0,4,4],
[0,0,0,5,5],
[0,1,0,2,2]])
U, s, VT = np.linalg.svd(A)
print("==>> U: ", U)
print("==>> VT: ", VT)
# create m x n Sigma matrix
Sigma = np.zeros((A.shape[0], A.shape[1]))
# populate Sigma with n x n diagonal matrix
square_len = min((A.shape[0], A.shape[1]))
Sigma[:square_len, :square_len] = np.diag(s)
print("==>> Sigma: ", Sigma)
n_elements = 2
U = U[:, :n_elements]
Sigma = Sigma[:n_elements, :n_elements]
VT = VT[:n_elements, :n_elements]
# reconstruct
B = U.dot(Sigma.dot(VT))
print("==>> B: ", B)
The output B is :
==>> B: [[ 0.99 1.01]
[ 2.98 3.04]
[ 3.98 4.05]
[ 4.97 5.06]
[ 0.36 1.29]
[-0.37 0.73]
[ 0.18 0.65]]
then this is sklearn code:
import numpy as np
from sklearn.decomposition import TruncatedSVD
A = np.array([
[1,1,1,0,0],
[3,3,3,0,0],
[4,4,4,0,0],
[5,5,5,0,0],
[0,2,0,4,4],
[0,0,0,5,5],
[0,1,0,2,2]]).astype(float)
svd = TruncatedSVD(n_components=2)
svd.fit(A) # Fit model on training data A
print("==>> right singular vectors: ", svd.components_)
print("==>> svd.singular_values_: ", svd.singular_values_)
B = svd.transform(A) # Perform dimensionality reduction on A.
print("==>> B: ", B)
its last output result is:
==>> B: [[ 1.72 -0.22]
[ 5.15 -0.67]
[ 6.87 -0.9 ]
[ 8.59 -1.12]
[ 1.91 5.62]
[ 0.9 6.95]
[ 0.95 2.81]]
As we can see, they produce different result (but i notice their singular values are the same, both are 12.48 9.51), how to make them same, does i misunderstand something ?
I think the correct way to perform a dimensionality reduction of the array A with np.linalg.svd is:
U, s, V = np.linalg.svd(A)
VT = V.T
B = A#VT[:,:n_elements]
Now B is:
array([[-1.72, 0.22],
[-5.15, 0.67],
[-6.87, 0.9 ],
[-8.59, 1.12],
[-1.91, -5.62],
[-0.9 , -6.95],
[-0.95, -2.81]])
That is exactly what you get from the TruncatedSVD, but with negative sign.
I found this fast script here in Stack Overflow for perform PCA with a given numpy array.
I don't know how to plot this in 3D, and also plot in 3D the Cumulative Explained Variances and the Number of Components. This fast script was perform with covariance method, and not with singular value decomposition, maybe that's the reason why I can't get my Cumulative Variances?
I tried to plotting with this, but it doesn't work.
This is the code and my output:
from numpy import array, dot, mean, std, empty, argsort
from numpy.linalg import eigh, solve
from numpy.random import randn
from matplotlib.pyplot import subplots, show
def cov(X):
"""
Covariance matrix
note: specifically for mean-centered data
note: numpy's `cov` uses N-1 as normalization
"""
return dot(X.T, X) / X.shape[0]
# N = data.shape[1]
# C = empty((N, N))
# for j in range(N):
# C[j, j] = mean(data[:, j] * data[:, j])
# for k in range(j + 1, N):
# C[j, k] = C[k, j] = mean(data[:, j] * data[:, k])
# return C
def pca(data, pc_count = None):
"""
Principal component analysis using eigenvalues
note: this mean-centers and auto-scales the data (in-place)
"""
data -= mean(data, 0)
data /= std(data, 0)
C = cov(data)
E, V = eigh(C)
key = argsort(E)[::-1][:pc_count]
E, V = E[key], V[:, key]
U = dot(data, V)
print(f'Eigen Values: {E}')
print(f'Eigen Vectors: {V}')
print(f'Key: {key}')
print(f'U: {U}')
print(f'shape: {U.shape}')
return U, E, V
data = dftransformed.transpose() # df tranpose and convert to numpy
trans = pca(data, 3)[0]
fig, (ax1, ax2) = subplots(1, 2)
ax1.scatter(data[:50, 0], data[:50, 1], c = 'r')
ax1.scatter(data[50:, 0], data[50:, 1], c = 'b')
ax2.scatter(trans[:50, 0], trans[:50, 1], c = 'r')
ax2.scatter(trans[50:, 0], trans[50:, 1], c = 'b')
show()
I understand the eigen values & eigen vectors, but I can't understand this key value, the user didn't comment this section of code in the answer, anyone knows what means each variable printed?
output:
Eigen Values: [126.30390621 68.48966957 26.03124927]
Eigen Vectors: [[-0.05998409 0.05852607 -0.03437937]
[ 0.00807487 0.00157143 -0.12352761]
[-0.00341751 0.03819162 0.08697668]
...
[-0.0210582 0.06601974 -0.04013712]
[-0.03558994 0.02953385 0.01885872]
[-0.06728424 -0.04162485 -0.01508154]]
Key: [439 438 437]
U: [[-12.70954048 8.97405411 -2.79812235]
[ -4.90853527 4.36517107 0.54129243]
[ -2.49370123 0.48341147 7.26682759]
[-16.07860635 6.16100749 5.81777637]
[ -1.81893291 6.48443689 -5.8655646 ]
[ 9.03939039 2.64196391 4.22056618]
[-14.71731064 9.19532016 -2.79275543]
[ 1.60998654 8.37866823 0.86207034]
[ -4.4503797 10.12688097 -5.12453656]
[ 12.16293556 2.2594413 -2.11730311]
[-15.76505125 9.48537581 -2.73906772]
[ -2.54289959 9.86768111 -4.84802992]
[ -5.78214902 9.21901651 -8.13594627]
[ -1.35428398 5.85550586 6.30553987]
[ 12.87261987 0.96283606 -3.26982121]
[ 24.57767477 -4.28214631 6.29510659]
[ 4.13941679 3.3688288 3.01194055]
[ -2.98318764 1.32775227 7.62610929]
[ -4.44461549 -1.49258339 1.39080386]
[ -0.10590795 -0.3313904 8.46363066]
[ 6.05960739 1.03091753 5.10875657]
[-21.27737352 -3.44453629 3.25115921]
[ -1.1183025 0.55238687 10.75611405]
[-10.6359291 7.58630341 -0.55088259]
[ 4.52557492 -8.05670864 2.23113833]
[-11.07822559 1.50970501 4.66555889]
[ -6.89542628 -19.24672805 -3.71322812]
[ -0.57831362 -17.84956249 -5.52002876]
[-12.70262277 -14.05542691 -2.72417438]
[ -7.50263129 -15.83723295 -3.2635125 ]
[ -7.52780216 -17.60790567 -2.00134852]
[ -5.34422731 -17.29394266 -2.69261597]
[ 9.40597893 0.21140292 2.05522806]
[ 12.12423431 -2.80281266 7.81182024]
[ 19.51224195 4.7624575 -11.20523383]
[ 22.38102384 0.82486072 -1.64716468]
[ -8.60947699 4.12597477 -6.01885407]
[ 9.56268414 1.18190655 -5.44074124]
[ 14.97675455 3.31666971 -3.30012109]
[ 20.47530869 -1.95896058 -1.91238615]]
shape: (40, 3)
trans = pca(data, 3)[0] is the U data, since [0] selects the first index of the returned data, and pca returns U, E, V
ax2.scatter(trans[:50, 0], trans[:50, 1], c = 'r') plots the first 50 rows of column 0 against the first 50 rows of column 1, and ax2.scatter(trans[50:, 0], trans[50:, 1], c = 'b') does the same for rows from 50 to the end. This from the sample data given in this fast script, but your data only has shape: (40, 3) (e.g. only 40 rows of data).
In order to plot trans as a 3d scatter plot, extract each of the 3 columns into a separate variable and plot as a scatter plot.
# imports as shown in the linked answer
from numpy import array, dot, mean, std, empty, argsort
from numpy.linalg import eigh, solve
from numpy.random import randn
from matplotlib.pyplot import subplots, show
# other imports
import numpy as np
# test data from linked answer (e.g. this fast script)
np.random.seed(365) # makes data repeatable
data = array([randn(8) for k in range(150)]) # creates array with shape (150, 8)
data[:50, 2:4] += 5 # adds 5 to first 50 rows of columns 2:4
data[50:, 2:5] += 5 # adds 5 to to rows from 50 of columns 2:5
# function call
trans = pca(data, 3)[0] # [0] gets U returned by pca(...)
# extract each column to a separate variable
x = trans[:, 0] # all rows of column 0
y = trans[:, 1] # all rows of column 1
z = trans[:, 2] # all rows of column 2
# plot 3d scatter plot
fig = plt.figure()
ax = fig.add_subplot(111, projection='3d')
ax.scatter(x, y, z)
Given a numpy ndarray with dimensions m by n (where n>m), how can I find the linearly independent columns?
One way is to use the LU decomposition. The factor U will be of the same size as your matrix, but will be upper-triangular. In each row of U, pick the first nonzero element: these are pivot elements, which belong to linearly independent columns. A self-contained example:
import numpy as np
from scipy.linalg import lu
A = np.array([[1, 2, 3], [2, 4, 2]]) # example for testing
U = lu(A)[2]
lin_indep_columns = [np.flatnonzero(U[i, :])[0] for i in range(U.shape[0])]
Output: [0, 2], which means the 0th and 2nd columns of A form a basis for its column space.
#user6655984's answer inspired this code, where I developed a function instead of the author's last line of code (finding pivot columns of U) so that it can handle more diverse A's.
Here it is:
import numpy as np
from scipy import linalg as LA
np.set_printoptions(precision=1, suppress=True)
A = np.array([[1, 4, 1, -1],
[2, 5, 1, -2],
[3, 6, 1, -3]])
P, L, U = LA.lu(A)
print('P', P, '', 'L', L, '', 'U', U, sep='\n')
Output:
P
[[0. 1. 0.]
[0. 0. 1.]
[1. 0. 0.]]
L
[[1. 0. 0. ]
[0.3 1. 0. ]
[0.7 0.5 1. ]]
U
[[ 3. 6. 1. -3. ]
[ 0. 2. 0.7 -0. ]
[ 0. 0. -0. -0. ]]
I came up with this function:
def get_indices_for_linearly_independent_columns_of_A(U: np.ndarray) -> list:
# I should first convert all "-0."s to "0." so that nonzero() can find them.
U_copy = U.copy()
U_copy[abs(U_copy) < 1.e-7] = 0
# Because some rows in U may not have even one nonzero element,
# I have to find the index for the first one in two steps.
index_of_all_nonzero_cols_in_each_row = (
[U_copy[i, :].nonzero()[0] for i in range(U_copy.shape[0])]
)
index_of_first_nonzero_col_in_each_row = (
[indices[0] for indices in index_of_all_nonzero_cols_in_each_row
if len(indices) > 0]
)
# Because two rows or more may have the same indices
# for their first nonzero element, I should remove duplicates.
unique_indices = sorted(list(set(index_of_first_nonzero_col_in_each_row)))
return unique_indices
Finally:
col_sp_A = A[:, get_indices_for_linearly_independent_columns_of_A(U)]
print(col_sp_A)
Output:
[[1 4]
[2 5]
[3 6]]
Try this one
def LU_decomposition(A):
"""
Perform LU decompostion of a given matrix
Args:
A: the given matrix
Returns: P, L and U, s.t. PA = LU
"""
assert A.shape[0] == A.shape[1]
N = A.shape[0]
P_idx = np.arange(0, N, dtype=np.int16).reshape(-1, 1)
for i in range(N - 1):
pivot_loc = np.argmax(np.abs(A[i:, [i]])) + i
if pivot_loc != i:
A[[i, pivot_loc], :] = A[[pivot_loc, i], :]
P_idx[[i, pivot_loc], :] = P_idx[[pivot_loc, i], :]
A[i + 1:, i] /= A[i, i]
A[i + 1:, i + 1:] -= A[i + 1:, [i]] * A[[i], i + 1:]
U, L, P = np.zeros_like(A), np.identity(N), np.zeros((N, N), dtype=np.int16)
for i in range(N):
L[i, :i] = A[i, :i]
U[i, i:] = A[i, i:]
P[i, P_idx[i][0]] = 1
return P.astype(np.float64), L, U
def get_bases(A):
assert A.ndim == 2
Q = gaussian_elimination(A)
M, N = Q.shape
pivot_idxs = []
for i in range(M):
j = i
while j < N and abs(Q[i, j]) < 1e-5:
j += 1
if j < N:
pivot_idxs.append(j)
return A[:, list(set(pivot_idxs))]
X is an n by d matrix, W is an m by d matrix, for every row in X I want to compute the squared Euclidean distance with every row in W, so the results will be an n by m matrix.
If there's only one row in W, this is easy
x = tensor.TensorType("float64", [False, False])()
w = tensor.TensorType("float64", [False])()
z = tensor.sum((x-w)**2, axis=1)
fn = theano.function([x, w], z)
print fn([[1,2,3], [2,2,2]], [2,2,2])
# [ 2. 0.]
What do I do when W is a matrix (in Theano)?
Short answer, use scipy.spatial.distance.cdist
Long answer, if you don't have scipy, is to broadcast subtract and then norm by axis 0.
np.linalg.norm(X[:,:,None]-W[:,None,:], axis=0)
Really long answer, of you have an ancient version of numpy without a vecorizable linalg.norm (i.e. you're using Abaqus) is
np.sum((X[:,:,None]-W[:,None,:])**2, axis=0).__pow__(0.5)
Edit by OP
In Theano we can make X and W both 3d matrices and make the corresponding axes broadcastable like
x = tensor.TensorType("float64", [False, True, False])()
w = tensor.TensorType("float64", [True, False, False])()
z = tensor.sum((x-w)**2, axis=2)
fn = theano.function([x, w], z)
print fn([[[0,1,2]], [[1,2,3]]], [[[1,1,1], [2,2,2]]])
# [[ 2. 5.]
# [ 5. 2.]]
Luckily the the number of rows in W can be known in advance, so I'm temporally doing
x = tensor.TensorType("float64", [False, False])()
m = 2
w = tensor.as_tensor([[2,2,2],[1,2,3]])
res_list = []
for i in range(m):
res_list.append(ten.sum((x-w[i,:])**2, axis=1))
z = tensor.stack(res_list)
fn = theano.function([x], z)
print fn([[1,2,3], [2,2,2], [2,3,4]])
# [[ 2. 0. 5.]
# [ 0. 2. 3.]]
Other answers are welcome!