I want to compute the dot product between two numpy arrays.
For example, my arrays have shape of (3,) and (1,), so from basic math understanding I should an vector of shape (3,1). However using numpy dot would not get the result like that. In general, my input would have the size of (x,n) and (n,x) and I would like to get the shape (x,x) or scalar if x=1.
The only real issue here is that you're using arrays of size (3,) and (1,) but you should be using (3,1) and (1,1). With that it behaves exactly as you want/expect:
>>> np.dot([3, 2, 1], [1])
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ValueError: shapes (3,) and (1,) not aligned: 3 (dim 0) != 1 (dim 0)
>>> np.dot([[3], [2], [1]], [[1]])
array([[3],
[2],
[1]])
For (x, n) and (n, x) shapes:
>>> x = 5
>>> n = 4
>>> A = np.ones((x, n))
>>> B = np.ones((n, x))
>>> A.dot(B)
array([[ 4., 4., 4., 4., 4.],
[ 4., 4., 4., 4., 4.],
[ 4., 4., 4., 4., 4.],
[ 4., 4., 4., 4., 4.],
[ 4., 4., 4., 4., 4.]])
>>> A.dot(B).shape
(5, 5)
Again, exactly as you want/expect. Note that in numpy, an array with shape (n,) is a zero dimensional array, while an array with shape (n,1) is a one dimensional array. Single dimensional arrays are necessary for operations on them to behave like you would expect.
Related
Starting with a 3d array (like a 2d image with RGB). I'd like to change the color based on the value of another 2d matrix.
import numpy as np
img=np.zeros((2,2,3)) # a black image
print('\nimg=',list(img))
b=np.array([[1,2],[3,4]]) # some 2d array of values
#img=np.where(b==1,[9,9,9],img) # ValueError: operands could not be broadcast together with shapes (2,2) (3,) (2,2,3)
#print(img)
# Trying to color the coordinate where b==1 with the RGB color 9,9,9
whatIwant=np.array([[9,9,9],[0,0,0],[0,0,0],[0,0,0]])
print('\nwhatIwant=\n',list(whatIwant))
expected output:
img=[array([[0., 0., 0.],
[0., 0., 0.]]), array([[0., 0., 0.],
[0., 0., 0.]])]
whatIwant=
[array([9, 9, 9]), array([0, 0, 0]), array([0, 0, 0]), array([0, 0, 0])]
In [13]: img=np.zeros((2,2,3)) # a black image
In [14]: b=np.array([[1,2],[3,4]]) # some 2d array of values
boolean test array:
In [15]: b==1
Out[15]:
array([[ True, False],
[False, False]])
A boolean mask has to match all dimensions, or just one:
In [16]: img[b]
Traceback (most recent call last):
File "<ipython-input-16-228af24ace6b>", line 1, in <module>
img[b]
IndexError: index 2 is out of bounds for axis 0 with size 2
But if we get the indices of the True value(s):
In [17]: idx = np.nonzero(b==1)
In [18]: idx
Out[18]: (array([0]), array([0]))
we can use that to index the 3d array, for get or for set:
In [19]: img[idx]
Out[19]: array([[0., 0., 0.]])
In [20]: img[idx]=[9,8,7]
In [21]: img
Out[21]:
array([[[9., 8., 7.],
[0., 0., 0.]],
[[0., 0., 0.],
[0., 0., 0.]]])
Sometimes it's easier to unpack the nonzero tuple:
In [22]: I,J = np.nonzero(b==1)
In [23]: I,J
Out[23]: (array([0]), array([0]))
In [24]: img[I,J,:]
Out[24]: array([[9., 8., 7.]])
I want to shift a tensor in a given axis. It's easy to do this in pandas or numpy. Like this:
import numpy as np
import pandas as pd
data = np.arange(0, 6).reshape(-1, 2)
pd.DataFrame(data).shift(1).fillna(0).values
Output is:
array([[0., 0.],
[0., 1.],
[2., 3.]])
But in tensorflow, the closest solution I found is tf.roll. But it shift the last row to the first row. (I don't want that). So I have to use something like
tf.roll + tf.slice(remove the last row) + tf.concat(add tf.zeros to the first row).
It's really ugly.
Is there a better way to handle shift in tensorflow or keras?
Thanks.
I think I find a better way for this problem.
We could use tf.roll, then apply tf.math.multiply to set the first row to zeros.
Sample code is as follows:
Original tensor:
A = tf.cast(tf.reshape(tf.range(27), (-1, 3, 3)), dtype=tf.float32)
A
Output:
<tf.Tensor: id=117, shape=(3, 3, 3), dtype=float32, numpy=
array([[[ 0., 1., 2.],
[ 3., 4., 5.],
[ 6., 7., 8.]],
[[ 9., 10., 11.],
[12., 13., 14.],
[15., 16., 17.]],
[[18., 19., 20.],
[21., 22., 23.],
[24., 25., 26.]]], dtype=float32)>
Shift (like pd.shift):
B = tf.concat((tf.zeros((1, 3)), tf.ones((2, 3))), axis=0)
C = tf.expand_dims(B, axis=0)
tf.math.multiply(tf.roll(A, 1, axis=1), C)
Output:
<tf.Tensor: id=128, shape=(3, 3, 3), dtype=float32, numpy=
array([[[ 0., 0., 0.],
[ 0., 1., 2.],
[ 3., 4., 5.]],
[[ 0., 0., 0.],
[ 9., 10., 11.],
[12., 13., 14.]],
[[ 0., 0., 0.],
[18., 19., 20.],
[21., 22., 23.]]], dtype=float32)>
Try this:
import tensorflow as tf
input = tf.constant([[0, 1, 3], [4, 5, 6], [7, 8, 9]])
shifted_0dim = input[1:]
shifted_1dim = input[:, 1:]
shifted2 = input[2:]
Generalizing the accepted answer to arbitrary tensor shapes, desired shift, and axis to shift:
import tensorflow as tf
def tf_shift(tensor, shift=1, axis=0):
dim = len(tensor.shape)
if axis > dim:
raise ValueError(
f'Value of axis ({axis}) must be <= number of tensor axes ({dim})'
)
mask_dim = dim - axis
mask_shape = tensor.shape[-mask_dim:]
zero_dim = min(shift, mask_shape[0])
mask = tf.concat(
[tf.zeros(tf.TensorShape(zero_dim) + mask_shape[1:]),
tf.ones(tf.TensorShape(mask_shape[0] - zero_dim) + mask_shape[1:])],
axis=0
)
for i in range(dim - mask_dim):
mask = tf.expand_dims(mask, axis=0)
return tf.multiply(
tf.roll(tensor, shift, axis),
mask
)
EDIT:
This code above doesn't allow for negative shift values, and is pretty slow. Here is a more efficient version utilizing tf.roll and tf.concat without creating a mask and multiplying the tensor of interest by it.
import tensorflow as tf
def tf_shift(values: tf.Tensor, shift: int = 1, axis: int = 0):
pad = tf.zeros([val if i != axis else abs(shift) for i, val in enumerate(values.shape)],
dtype=values.dtype)
size = [-1 if i != axis else val - abs(shift) for i, val in enumerate(values.shape)]
if shift > 0:
shifted = tf.concat(
[pad, tf.slice(values, [0] * len(values.shape), size)],
axis=axis
)
elif shift < 0:
shifted = tf.concat(
[tf.slice(values, [0 if i != axis else abs(shift) for i, _ in enumerate(values.shape)], size), pad],
axis=axis
)
else:
shifted = values
return shifted
Assuming a 2d tensor, this function should mimic a Dataframe shift:
def shift_tensor(tensor, periods, fill_value):
num_row = len(tensor)
num_col = len(tensor[0])
pad = tf.fill([periods, num_col], fill_value)
if periods > 0:
shifted_tensor = tf.concat((pad, tensor[:(num_row - periods), :]), axis=0)
else:
shifted_tensor = tf.concat((tensor[:(num_row - periods), :], pad), axis=0)
return shifted_tensor
Im trying to understand numpy where condition.
>>> import numpy as np
>>> x = np.arange(9.).reshape(3, 3)
>>> x
array([[ 0., 1., 2.],
[ 3., 4., 5.],
[ 6., 7., 8.]])
>>> np.where( x > 5 )
(array([2, 2, 2]), array([0, 1, 2]))
IN the above case, what does the output actually mean, array([0,1,2]) I actually see in the input what is array([2,2,2])
Th first array indicates the row number and the second array indicates the corresponding column number.
If the array is following:
array([[ 0., 1., 2.],
[ 3., 4., 5.],
[ 6., 7., 8.]])
Then the following
(array([2, 2, 2]), array([0, 1, 2]))
Can be interpreted as
array(2,0) => 6
array(2,1) => 7
array (2,2) => 8
You might also want to know where those values appear visually in your array. In such cases, you can return the array's value where the condition is True and a null value where they are false. In the example below, the value of x is returned at the position where x>5, otherwise assign -1.
x = np.arange(9.).reshape(3, 3)
np.where(x>5, x, -1)
array([[-1., -1., -1.],
[-1., -1., -1.],
[ 6., 7., 8.]])
Three elements found, located at (2,0),(2,1),(2,2)..
By the way, tryhelp(np.where()) will help you a lot.
I'm trying to feed 1D numpy arrays (flattend images) via a generator into a H5py data file in order to create training and validation matrices.
The following code was adapted from a solution (can't find it now) in which the data attribute of H5py's File objects's create_dataset function is provided data in the form of a call to np.fromiter which has a generator function as one of its arguments.
from scipy.misc import imread
import h5py
import numpy as np
import os
# Creating h5 data file
f = h5py.File('../data.h5', 'w')
# Source directory for image data
src = '/datasets/aic540/train/images/'
# Showing quantity and dimensionality of data
images = os.listdir(src)
ex_img = imread(src + images[0])
flat_img = ex_img.flatten()
print "# of images is {}".format(len(images))
print "image shape is {}".format(ex_img.shape)
print "flattened image shape is {}".format(flat_img.shape)
# Creating generator to feed in data to h5py's `create_dataset` function
gen = (imread(src + i).flatten().astype(np.int8) for i in os.listdir(src))
# Creating h5 dataset
f.create_dataset(name='training',
#shape=(59482, 1555200),
data=np.fromiter(gen, dtype=np.int8))
Output:
# of images is 59482
image shape is (540, 960, 3)
flattened image shape is (1555200,)
Traceback (most recent call last):
File "process_images.py", line 30, in <module>
data=np.fromiter(gen, dtype=np.int8))
ValueError: setting an array element with a sequence.
I've read when searching for this error in this context that the problem is that np.fromiter() needs a list and not a generator function (which seems opposed to the function that the name "fromiter" implies) -- wrapping the generator in a list call list(gen) allows the code to run but it, of course, uses up all the memory in the expansion of this list before the call to create_dataset is made.
How do I use a generator to feed data into an H5py data file?
If my approach is entirely wrong, what is the correct way to build a very large numpy matrix that doesn't fit in memory -- using H5py or otherwise?
The with a sequence error comes from what you are trying to feed fromiter, not the generator part.
In py3, range is generator like:
In [15]: np.fromiter(range(3),dtype=int)
Out[15]: array([0, 1, 2])
In [16]: np.fromiter((2*x for x in range(3)),dtype=int)
Out[16]: array([0, 2, 4])
But if I start with a 2d array (which imread produces, right?), and create a generator expression as you do:
In [17]: gen = (np.ones((2,3)).flatten().astype(np.int8) for i in range(3))
In [18]: list(gen)
Out[18]:
[array([1, 1, 1, 1, 1, 1], dtype=int8),
array([1, 1, 1, 1, 1, 1], dtype=int8),
array([1, 1, 1, 1, 1, 1], dtype=int8)]
I generate a list of arrays.
In [19]: gen = (np.ones((2,3)).flatten().astype(np.int8) for i in range(3))
In [21]: np.fromiter(gen, np.int8)
...
ValueError: setting an array element with a sequence.
np.fromiter creates a 1d array from an iterator that provides 'numbers' one at a time, not something that dishes out lists or arrays.
In any case, npfromiter creates a full array; not some sort of generator. There's nothing like an array 'generator'.
Even without chunking you can write data to the file by 'row' or other slice.
In [28]: f = h5py.File('test.h5', 'w')
In [29]: data = f.create_dataset(name='test',shape=(100,10))
In [30]: for i in range(100):
...: data[i,:] = np.arange(i,i+10)
...:
In [31]: data
Out[31]: <HDF5 dataset "test": shape (100, 10), type "<f4">
The equivalent in your case is to load an image, reshape it, and write it immediately to the h5py dataset. No need to collect all the images in an array or list.
read 10 rows:
In [33]: data[:10,:]
Out[33]:
array([[ 0., 1., 2., 3., 4., 5., 6., 7., 8., 9.],
[ 1., 2., 3., 4., 5., 6., 7., 8., 9., 10.],
[ 2., 3., 4., 5., 6., 7., 8., 9., 10., 11.],
[ 3., 4., 5., 6., 7., 8., 9., 10., 11., 12.],
[ 4., 5., 6., 7., 8., 9., 10., 11., 12., 13.],
[ 5., 6., 7., 8., 9., 10., 11., 12., 13., 14.],
[ 6., 7., 8., 9., 10., 11., 12., 13., 14., 15.],
[ 7., 8., 9., 10., 11., 12., 13., 14., 15., 16.],
[ 8., 9., 10., 11., 12., 13., 14., 15., 16., 17.],
[ 9., 10., 11., 12., 13., 14., 15., 16., 17., 18.]], dtype=float32)
Enabling chunking might help with really large datasets, but I don't experience in that area.
>>> import numpy as np
>>> from scipy.sparse import *
>>> x1 = np.eye(3, dtype=float)
>>> x2 = csr_matrix(x1, dtype=float, shape =x1.shape)
>>> assert x2.todense().any()==x1.any() ## holds true
>>> w = np.ones((3,1))
>>> dw1 = w - x1[:,0]
>>> dw2 = w - x2[:,0]
which gives me
>>> print dw1
[[ 0. 1. 1.]
[ 0. 1. 1.]
[ 0. 1. 1.]]
while
>>> print dw2
[[ 0.]
[ 1.]
[ 1.]]
My question is why dw1 and dw2 differ? Should they defer, Is it a bug? Many Thanks!
It's because those slices are 1D and 2D respectively -
In [23]: x1[:,0]
Out[23]: array([ 1., 0., 0.])
In [24]: x2[:,0].toarray()
Out[24]:
array([[ 1.],
[ 0.],
[ 0.]])
In [29]: x1[:,0].ndim
Out[29]: 1
In [30]: x2[:,0].toarray().ndim
Out[30]: 2
Also, w is a 2D array -
In [33]: w
Out[33]:
array([[ 1.],
[ 1.],
[ 1.]])
In [34]: w.ndim
Out[34]: 2
So, with broadcasting the subtractions from w are performed along different axes of w namely second and first axis respectively.
This is a slicing/indexing issue. The questionable line here is
w - x1[:, 0]
which has nothing to do with sparseness. You have sliced x1, obtaining a 1D array. When this gets subtracted from w, numpy broadcasts this array back into a 3 by 3 matrix (because it equates the number of columns of both terms), which I guess is not what you wanted.
It looks like you just wanted the submatrix consisting of the first column of x1. This would be
w - x1[:, [0]]
returning
array([[ 0.],
[ 1.],
[ 1.]])
consistent with the other result.
In case of a sparse matrix, you automatically get a submatrix (not a 1D array) because indexing works differently for those.