I have two 2-D matrices which have a shared axis.
I want to get a 3-D array that holds the results of every pairwise multiplication made between all the combinations of vectors from each matrix along that shared axis.
What is the best way to achieve this? (assuming that the matrices are big)
As an illustration, let's say I have 100 technicians and 1000 customers.
For each of these individuals I have a 1-D array with ones and zeros representing their availability on a each day of the week.
That's a 7x100 matrix for the technicians, a 7x1000 matrix for the customers.
import numpy as np
technicians = np.random.randint(low=0,high=2,size=(7,100))
customers = np.random.randint(low=0,high=2,size=(7,1000))
result = solution(technicians, customers)
result.shape # (7,100,1000)
I want to find for each technician-customer couple the days they are both available.
If I perform a pairwise multiplication between each combination of technician availability and customer availability I get a 1-D arrays that shows for each couple whether they are both available on these days. Together they create the 3-D array I'm aiming for, shaped something like 7x100x1000.
Thanks!
Try
ans = technicians.reshape((7, 1, 100)) * customers.reshape((7, 1000, 1))
We make use of numpy.broadcasting.
General Broadcasting Rules: When operating on two arrays, NumPy
compares their shapes element-wise. It starts with the trailing
dimensions, and works its way forward. Two dimensions are compatible
when
(1) they are equal, or (2) one of them is 1
Now, we are matching the shape of technicians and customers as
technician : 7 x 1 x 100
customers : 7 x 1000 x 1
Result (3d array): 7 x 1000 x 100
using reshape. Then, we can apply elementwise multiplication with *.
Related
I am attempting to create many matrices in Python, each of which is a 4 x 4 matrix. Each matrix is a matrix filled with 0s, but there is an input that randomly enters the number of 8's that will be put into that matrix. For example, if a 3 is input, then 3 of the 16 entries will be 8s, and the rest will be 0's.
It would also be helpful to show how this could be looped over a sequence of values, to generate many different matrices. Thank you.
Here's a code snippet to do it for one matrix:
# Create m x n matrix with 0s
a = np.zeros((m,n))
# Get n random indices to replace with 8
x_indices = np.random.choice(a.shape[0], n, replace=False)
y_indices = np.random.choice(a.shape[1], n, replace=False)
# Replace those positions with 8
a[x_index, y_index] = 8
You can loop it over to get many different matrices
I would like to set a custom weight constrain in Keras / TensorFlow.
I have a weight matrix W of shape 2dxd for data dimension d (this is the input layer, which takes a datapoint of dimension d and uses its squared values additionally to the normal ones) and I want the diagonal entries of the dxd matrix, which consists of the first d rows of W and the diagonal entries of the dxd matrix, which consists of the last d rows of W to be the maximum weights in the corresponding row.
Is there any easy / cheap way to implement this?
Edit: I splitted w in two diagonal matrices and used
w-tf.linalg.diag(tf.linalg.diag_part(w))+tf.linalg.diag(tf.keras.backend.max(w,axis=0))
as constraint but I don't think this is the best (cheapest) implementation
I know for a 4D vector, shape should be (4, 1) which is actually represented in 4D space but ndim is 2, and for some ndarray to be in 4 dimension, its shape should be something like (2, 3, 4, 5).
So, Is it like dimensional concept differs between vector and matrices (or arrays)? I'm trying to understand from mathematical perspective and how it's derived to pandas programming.
The dimensionality of a mathematical object is usually determined by the number of independent parameters in that particular object. For example, a 4-D vector is mathematically 4 dimensional because it contains 4 independent elements (unless some relation between them has been specified). Such a vector, if represented as a column vector in numpy, would have a shape (4, 1) because it has 4 rows and 1 column. The transpose of this vector, a row vector, has shape (4, ) because it has 4 columns and only 1 row, and the row-style view is default, so if there is 1 row, it's not explicitly mentioned.
Note however, that the column vector and row vector are dimensionally equivalent mathematically. Both have 4 dimensions.
For a 3 x 3 matrix, the most general mathematical dimension is 9, because it has 9 independent elements in general. The shape of a corresponding numpy array would be (3, 3). If you're looking for the maximum number of elements in any numpy array, ndarray.size is the way to go.
ndarray.ndim, however, yields the number of axes in a numpy array. That is, the number of directions along which values are placed (sloppy terminology!). So for the 3 x 3 matrix, ndim yields 2. For an array of shape (3, 7, 2, 1), ndim would yield 4. But, as we already discussed, the mathematical dimension would generally be 3 x 7 x 2 x 1 = 42 (So this is a matrix in 42-dimensional space! But the numpy array has just 4 dimensions). Thereby, as you might've already noticed, ndarray.size is just the product of the numbers in ndarray.shape.
Note that these are not just concepts of programming. We are used to saying "2-D matrices" in mathematics, but that is not to be confused with the space in which the matrices reside.
I am writing a simple linear regression cost function (Python) for a simple neural network. I have come across the following two alternate ways of summing the error (cost) over m examples using numpy (np) matrices.
The cost function is:
def compute_cost(X, Y, W):
m = Y.size;
H = h(X,W)
error = H-Y
J = (1/(2*m)) * np.sum(error **2, axis=0) #1 (sum squared error over m examples)
return J
X is the input matrix.
Y is the output matrix (labels).
W is the weights matrix.
It seems that the statement:
J = (1/(2*m)) * np.sum(error **2, axis=0) #1 (sum squared error over m examples)
can be replaced by:
J = (1/(2*m)) * np.dot(error.T, error) #2
with the same result.
I do not understand why np.dot is equivalent to summing over m examples or just why the two statements give the same result. Could you please provide some leads and also point me to some link(s) where I can read more and understand this relationship between np.sum and np.dot.
There's nothing special, just simple linear algebra.
According to numpy documentation, np.dot(a,b) performs different operation on different types of inputs.
If both a and b are 1-D arrays, it is inner product of vectors
(without complex conjugation).
If both a and b are 2-D arrays, it is
matrix multiplication, but using matmul or a # b is preferred.
If your error is 1-D array, then the transpose error.T is equal to error, then the operation np.dot is the inner product of them, which equals to the sum of each element to the power of 2.
If your error is 2-D array, then you should follow matrix multiplication principle, so each row of error.T will multiply by each column of error. When your error is a column vector, then the result will be a 1*1 matrix, which is similar to a scalar. when your error is a 1-by-N row vector, then it returns an N-by-N matrix.
For each location in the result matrix, instead of storing the dot product of the corresponding row and column in the argument matrices, I would like like to store the element wise product, which will be a vector extending into a third dimension.
One idea would be to convert the argument matrices to vectors with vector entries, and then take their outer product, but I'm not sure how to do this either.
EDIT:
I figured it out before I saw there was a reply. Here is my solution:
def newdot(A, B):
A = A.reshape((1,) + A.shape)
B = B.reshape((1,) + B.shape)
A = A.transpose(2, 1, 0)
B = B.transpose(1, 0, 2)
return A * B
What I am doing is taking apart each row and column pair that will have their outer product taken, and forming two lists of them, which then get their contents matrix multiplied together in parallel.
It's a little convoluted (and difficult to explain) but this function should get you what you're looking for:
def f(m1, m2):
return (m2.A.T * m1.A.reshape(m1.shape[0],1,m1.shape[1]))
m3 = m1 * m2
m3_el = f(m1, m2)
m3[i,j] == sum(m3_el[i,j,:])
m3 == m3_el.sum(2)
The basic idea is to turn the matrices into arrays and do element-by-element multiplication. One of the arrays gets reshaped to have a size of one in its middle dimension, and array broadcasting rules expand this dimension out to match the height of the other array.