Tensorflow: batching labels with tf.train.batch - tensorflow

i have a piece of code
la=[0,0,0,0,0,0,1,1,1,1]
onehot = tf.one_hot(la, depth=2) #[[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[0,1],[0,1],[0,1],[0,1]]
image_batch,labels_batch=tf.train.batch([resized_image,onehot],batch_size=2,num_threads=1)
when i run
print(s.run([tf.shape(image_batch),labels_batch]))
it is batching all labes at a time,like
[array([ 2, 50, 50, 3]), array([[[ 1., 0.],
[ 1., 0.],
[ 1., 0.],
[ 1., 0.],
[ 1., 0.],
[ 1., 0.],
[ 0., 1.],
[ 0., 1.],
[ 0., 1.],
[ 0., 1.]],
[[ 1., 0.],
[ 1., 0.],
[ 1., 0.],
[ 1., 0.],
[ 1., 0.],
[ 1., 0.],
[ 0., 1.],
[ 0., 1.],
[ 0., 1.],
[ 0., 1.]]], dtype=float32)]
it should output something like
[array([ 2, 50, 50, 3]), array([[[ 1., 0.],
[[ 1., 0.]]], dtype=float32)]
doesn't it? as batch size is 2 and taking 2 images and it's corresponding labels at a time.
i'm new to CNN and machine learning.thanks beforehand.


According to the Tensorflow documentation of tf.train.batch (https://www.tensorflow.org/api_docs/python/tf/train/batch),
Since the enqueue_many=False by default and your input onehot have the shape of [10, 2], then the output (here labels_batch) shape become [batch_size, 10, 2].
if the enqueue_many=True, then only the output (here labels_batch) will become [batch_size,2].
Hope this helps.

Related

Defining a 2-d numpy array from values in 3-d numpy array

I have a 3-D numpy array representing a model domain of 39 layers, 279 rows, 153 columns. The values in the array are either 0 or 1 and signify if the cell in the domain is inactive or active, respectively. I am trying to create a 2-D array of shape 279 rows and 153 columns where the array values equal the layer number for the uppermost active layer in the grid. Essentially, at each row, col location I want to loop through the layers to find the first one that is a 1 and not a 0 and then put that layer number in the 2-D array at that row, col location. For example:
If a four layer (layers 0-3) array looks like this:
array([[[ 0., 1., 0., 0.],
[ 1., 0., 0., 0.],
[ 1., 0., 0., 0.]],
[[ 0., 1., 1., 0.],
[ 1., 1., 0., 0.],
[ 1., 1., 0., 0.]],
[[ 0., 0., 1., 1.],
[ 0., 1., 1., 0.],
[ 0., 1., 1., 0.]],
[[ 0., 0., 1., 1.],
[ 0., 1., 1., 1.],
[ 0., 1., 1., 1.]]])
The 2-D array should look like this:
array([[[ 0., 0., 1., 2.],
[ 0., 1., 2., 3.],
[ 0., 1., 2., 3.]],
If the row-col location is not active (not equal to 1) in any layer , the value in the resulting array should be 0 (like at 1,1), the same as if it were active in layer 0.
I have tried modifying a couple of solutions where the z-axis values are summed, or averaged, but can't seem to figure out how to get exactly what I am looking for.

You could try numpy.argmax:
import numpy as np
a = np.array([[[ 0., 1., 0., 0.],
[ 1., 0., 0., 0.],
[ 1., 0., 0., 0.]],
[[ 0., 1., 1., 0.],
[ 1., 1., 0., 0.],
[ 1., 1., 0., 0.]],
[[ 0., 0., 1., 1.],
[ 0., 1., 1., 0.],
[ 0., 1., 1., 0.]],
[[ 0., 0., 1., 1.],
[ 0., 1., 1., 1.],
[ 0., 1., 1., 1.]]])
print(np.argmax(a,0))
array([[0, 0, 1, 2],
[0, 1, 2, 3],
[0, 1, 2, 3]])
This works because argmax returns the first max value when searching over the defined axis (in this case the 0th axis).

Implementing BandRNN with pytorch and tensorflow

So I am trying to figure out how to train my matrix in a way that I will get a BandRNN.
BandRnn is a diagonalRNN model with a different number of connections per neuron.
For example:
C is the number of connections per neuron.
I found out that there is a way to turn off some of the gradients in a for loop, in a way that prevents them from being trained as follows:
for p in model.input.parameters():
p.requires_grad = False
But I can't find a proper way to do so, in a way that will make my matrix become a BandRNN.
Hopefully, someone will be able to help me with this issue.

As far as I know you can only activate/deactivate requires_grad on a tensor, and not on distinct components of that tensor. Instead what you could do is zero out the values outside the band.
First create a mask for the band, you could use torch.ones with torch.diagflat:
>>> torch.diagflat(torch.ones(5), offset=1)
By setting the right dimension for torch.ones as well as the right offset you can generate offset diagonal matrices with consistent shapes.
>>> N = 10; i = -1
>>> torch.diagflat(torch.ones(N-abs(i)), offset=i)
tensor([[0., 0., 0., 0., 0.],
[1., 0., 0., 0., 0.],
[0., 1., 0., 0., 0.],
[0., 0., 1., 0., 0.],
[0., 0., 0., 1., 0.]])
>>> N = 10; i = 0
>>> torch.diagflat(torch.ones(N-abs(i)), offset=i)
tensor([[1., 0., 0., 0., 0.],
[0., 1., 0., 0., 0.],
[0., 0., 1., 0., 0.],
[0., 0., 0., 1., 0.],
[0., 0., 0., 0., 1.]])
>>> N = 10; i = 1
>>> torch.diagflat(torch.ones(N-abs(i)), offset=i)
tensor([[0., 1., 0., 0., 0.],
[0., 0., 1., 0., 0.],
[0., 0., 0., 1., 0.],
[0., 0., 0., 0., 1.],
[0., 0., 0., 0., 0.]])
You get the point, summing these matrices element-wise allows use to get a mask:
>>> N = 10; b = 3
>>> mask = sum(torch.diagflat(torch.ones(N-abs(i)), i) for i in range(-b//2,b//2+1))
>>> mask
tensor([[1., 1., 0., 0., 0.],
[1., 1., 1., 0., 0.],
[1., 1., 1., 1., 0.],
[0., 1., 1., 1., 1.],
[0., 0., 1., 1., 1.]])
Then you can zero out the values outside the band on your nn.Linear:
>>> m = nn.Linear(N, N)
>>> m.weight.data = m.weight * mask
>>> m.weight
Parameter containing:
tensor([[-0.3321, -0.3377, -0.0000, -0.0000, -0.0000],
[-0.4197, 0.1729, 0.2101, 0.0000, 0.0000],
[ 0.3467, 0.2857, -0.3919, -0.0659, 0.0000],
[ 0.0000, -0.4060, 0.0908, 0.0729, -0.1318],
[ 0.0000, -0.0000, -0.4449, -0.0029, -0.1498]], requires_grad=True)
Note, you might need to perform this on each forward pass as the parameters outside the band might get updated to non-zero values during the training. Of course, you can initialize mask once and keep it in memory.
It would be more convenient to wrap everything into a custom nn.Module.

The use of meshgrid in pytorch/numpy

Below code is a snippet taken from https://blog.paperspace.com/how-to-implement-a-yolo-v3-object-detector-from-scratch-in-pytorch-part-3/, and I am confused as to what it is trying to achieve.
grid = np.arange(grid_size)
a,b = np.meshgrid(grid, grid)
x_offset = torch.FloatTensor(a).view(-1,1)
y_offset = torch.FloatTensor(b).view(-1,1)
if CUDA:
x_offset = x_offset.cuda()
y_offset = y_offset.cuda()
x_y_offset = torch.cat((x_offset, y_offset), 1).repeat(1,num_anchors).view(-1,2).unsqueeze(0)
I tried the case when grid_size = 3, and it outputed:
tensor([[0., 1.],
[2., 0.],
[1., 2.],
[0., 1.],
[2., 0.],
[0., 0.],
[1., 1.],
[1., 2.],
[2., 2.],
[0., 1.],
[2., 0.],
[1., 2.],
[0., 1.],
[2., 0.],
[0., 0.],
[1., 1.],
[1., 2.],
[2., 2.],
[0., 1.],
[2., 0.],
[1., 2.],
[0., 1.],
[2., 0.],
[0., 0.],
[1., 1.],
[1., 2.],
[2., 2.]])
I cannot quite see what is the pattern here. According to the description in the given link I think I should really expect something like:
tensor([[0,0],
[0,0],
[0,0],
[0,1],
[0,1],
[0,1],
[0,2],
[0,2],
...]])

If you are expecting the second output you show, simply change
x_y_offset = (
torch.cat((x_offset, y_offset), 1).repeat(1, num_anchors).view(-1, 2).unsqueeze(0)
)
to
x_y_offset = (
torch.cat((y_offset, x_offset), 1).repeat(1, num_anchors).view(-1, 2).unsqueeze(0)
)
It just has to do with the ordering of the output of meshgrid.

Numpy arange a diagonal array

I would like to create a square numpy array such that it starts counting from the diagonal.
Do you know a one-liner for that?
Example with 5x5:
array([[ 1., 2., 3., 4., 5.],
[ 0., 1., 2., 3., 4.],
[ 0., 0., 1., 2., 3.],
[ 0., 0., 0., 1., 2.],
[ 0., 0., 0., 0., 1.]])

In [49]: np.identity(5).cumsum(axis=1).cumsum(axis=1)
Out[49]:
array([[ 1., 2., 3., 4., 5.],
[ 0., 1., 2., 3., 4.],
[ 0., 0., 1., 2., 3.],
[ 0., 0., 0., 1., 2.],
[ 0., 0., 0., 0., 1.]]

>>> mat = np.vstack((np.concatenate((np.zeros(i),np.arange(1,5-i+1))) for i in range(0,5)))
>>> mat
array([[1., 2., 3., 4., 5.],
[0., 1., 2., 3., 4.],
[0., 0., 1., 2., 3.],
[0., 0., 0., 1., 2.],
[0., 0., 0., 0., 1.]])

Numpy broadcasting to the 4th dimension: ... vs. : vs None

In a montecarlo simulation I have the following 7 pokercards for 2 players and 3 different montecarlo runs.
self.cards:
array([[[ 6., 12.],
[ 1., 6.],
[ 3., 3.],
[ 8., 8.],
[ 1., 1.],
[ 4., 4.],
[ 2., 2.]],
[[ 6., 7.],
[ 1., 1.],
[ 3., 3.],
[ 2., 2.],
[ 12., 12.],
[ 5., 5.],
[ 10., 10.]],
[[ 6., 3.],
[ 1., 11.],
[ 2., 2.],
[ 6., 6.],
[ 12., 12.],
[ 6., 6.],
[ 7., 7.]]])
The corresponding suits are:
self.suits
array([[[ 2., 1.],
[ 1., 2.],
[ 2., 2.],
[ 2., 2.],
[ 1., 1.],
[ 2., 2.],
[ 2., 2.]],
[[ 2., 0.],
[ 1., 3.],
[ 2., 2.],
[ 0., 0.],
[ 1., 1.],
[ 1., 1.],
[ 1., 1.]],
[[ 2., 2.],
[ 1., 0.],
[ 3., 3.],
[ 2., 2.],
[ 1., 1.],
[ 1., 1.],
[ 1., 1.]]])
Now I would like to 'merge' the arrays in a way that the cards array is expanded to the 4th dimension having a size of 4: 0 containing all suits==1, 1 all suits==2, 2 all suits==3 and 3 all suits ==4
I can easily create 4 different arrays:
club_cards=(self.suits == 1) * self.cards
diamond_cards=(self.suits == 2) * self.cards
heart_cards=(self.suits == 3) * self.cards
spade_cards=(self.suits == 4) * self.cards
and then stack them together:
stacked_array=np.stack((club_cards,diamond_cards, heart_cards, spade_cards),axis=0)
The result as expected has a shape of (4, 3, 8, 2)
array([[[[ 1., 12.],
[ 1., 1.],
[ 0., 0.],
[ 0., 0.],
[ 0., 0.],
[ 0., 0.],
[ 0., 0.],
[-11., 0.]],
[[ 12., 12.],
[ 10., 10.],
[ 5., 5.],
[ 1., 0.],
[ 0., 0.],
[ 0., 0.],
[ 0., 0.],
[ 0., 0.]],
[[ 12., 12.],
[ 7., 7.],
[ 6., 6.],
[ 1., 0.],
[ 0., 0.],
[ 0., 0.],
[ 0., 0.],
[ 0., 0.]]],
[[[ 8., 8.],
[ 6., 6.],
[ 4., 4.],
[ 3., 3.],
[ 2., 2.],
[ 0., 0.],
[ 0., 0.],
[ -4., -4.]],
[[ 6., 3.],
[ 3., 0.],
[ 0., 0.],
[ 0., 0.],
[ 0., 0.],
[ 0., 0.],
[ 0., 0.],
[ -6., -9.]],
[[ 6., 6.],
[ 6., 3.],
[ 0., 0.],
[ 0., 0.],
[ 0., 0.],
[ 0., 0.],
[ 0., 0.],
[ -6., -6.]]],
[[[ 0., 0.],
[ 0., 0.],
[ 0., 0.],
[ 0., 0.],
[ 0., 0.],
[ 0., 0.],
[ 0., 0.],
[-12., -12.]],
[[ 0., 1.],
[ 0., 0.],
[ 0., 0.],
[ 0., 0.],
[ 0., 0.],
[ 0., 0.],
[ 0., 0.],
[-12., -11.]],
[[ 2., 2.],
[ 0., 0.],
[ 0., 0.],
[ 0., 0.],
[ 0., 0.],
[ 0., 0.],
[ 0., 0.],
[-10., -10.]]],
[[[ 0., 0.],
[ 0., 0.],
[ 0., 0.],
[ 0., 0.],
[ 0., 0.],
[ 0., 0.],
[ 0., 0.],
[-12., -12.]],
[[ 0., 0.],
[ 0., 0.],
[ 0., 0.],
[ 0., 0.],
[ 0., 0.],
[ 0., 0.],
[ 0., 0.],
[-12., -12.]],
[[ 0., 0.],
[ 0., 0.],
[ 0., 0.],
[ 0., 0.],
[ 0., 0.],
[ 0., 0.],
[ 0., 0.],
[-12., -12.]]]])
While this might make sense in the above case, it is not always possible, especially if there are more than 4 cases that need to be stacked together, which brings me to my question:
How can I do this with broadcasting? Below my specific questions:
I have tried a few things.
Let's focus on the first step to get the booleans in doing suits==np.arange(4) (the second step is just a multiplication with the cards which will need to be broadcast in the same way as the suits). My understanding is that we want to add a dimension for the suits array, so shouldn't we signal this with the 3 dot notation: self.suits[...,:,:,:]==np.arange(4)? Instead the following seems to almost work: self.suits[:,:,:,None]==np.arange(4) (except that it adds the dimension at the wrong place). The following doesn't work either: self.suits[None,:,:,:]==np.arange(4). How can I extend the array in the first dimension so the results are the same as in the above stack?
In what circumstances do I need the ... and when the None? I would expect to use the ... as this would signal that this dimension needs to be expanded as necessary (in this case to a size of 4)? Why does this seem to be incorrect and a None is used instead?

You are stacking the indivdual card results along axis=0. So, when porting to a broadcasting based solution, we can create a range array of those scalars 1, 2, 3, 4 in a 4D array with all axes being singleton dimensions (dims with length = 1) except the first one. There could be different ways to create such a 4D array. One way would be : np.arange(1,5)[:,None,None,None], where we create a 1D array with np.arange and simply add three singleton dims as the last three ones with np.newaxis/None.
We perform equality comparison with this 4D array against b, which would allow internally broadcasting of b elements along the last three dims. Then, we multiply it with a as also done in the original code and get the desired output.
Thus, the implementation would be -
out = a*(b == np.arange(1,5)[:,None,None,None])
When/how to use ...(ellipsis) :
We use ...(ellipsis), when trying to add new axes into a multi-dimensional array and we don't want to specify colons per dim. Thus, to make a a 4D array with the last dim being a singleton, we would do : a[:,:,:,None]. Too much of typing! So, we use ... there to help us out : a[...,None]. Please note that this ... notation is used irrespective of the number of dimensions. So, if a were a 5D array and to add a new axis into it as the last one, we would do a[:,:,:,:,:,None] or simply with ellipsis : a[...,None]. Neat huh!