I would like to dynamically broadcast multiplying of Tensors along the dynamic dimension, which is batch size. I could hardcode batch size, but then inference on batch size equal 1 is impossible.
img_placeholder = tf.placeholder(tf.float32, (None, 28, 28, 1), name='real_img')
random_part = tf.placeholder(tf.float32, (None), name='random_part')
During training time I will feed each placeholder with the same size of the first dimension. I would like to have result = random_part * img_placeholder such that result[0] = random_part[0] * img_placeholder[0].
Currently just multiplying ends with Tensor of size BATCH_SIZE, 28, 28, BATCH_SIZE.
If broadcasting won't work as I would like (because there is dynamic dimension size) is there any possibility to do it in other way?
TensorFlow uses same broadcasting rules as numpy, so in your example shape (20) first gets extended on the left with singleton dimensions to become (1, 1, 1, 20), and then singleton dimensions are matched one both sides to have shape (20, 3, 3, 20).
If you want (20) to be treated as a batch dimension, you should reshape it to have the same rank
>>> a=np.ones((5,))
>>> b=np.ones((5,28,28,1))
>>> (a*b).shape
(5, 28, 28, 5)
>>> b.shape
(5, 28, 28, 1)
>>> a.shape
(5,)
>>> a2 = np.reshape(a, [len(a), 1, 1, 1])
>>> a2.shape
(5, 1, 1, 1)
>>> (a2*b).shape
(5, 28, 28, 1)
Related
I have two tensors with different shapes, in one it is shape=(None, 20, 32) and in another, it is shape=(None, 5, 32). Let say the first is the embedding of 20 words and the second is the embeddings of 5 words. Now I would like to contact them and have a tensor with the shape of (None, 25, 32). When I try tf.concat([t1,t2], 0) I get the following exception: Dimension 0 in both shapes must be equal, but are 20 and 5
Assuming that None will represent the same number of samples, you just need to specify that concatenation axis to be 1.
import tensorflow as tf
a = tf.random.uniform((4, 5, 32))
b = tf.random.uniform((4, 20, 32))
tf.concat([a, b], axis=1).shape
TensorShape([4, 25, 32])
I have a 2d tensor of shape [32,768] also a 3d tensor of [32,512,768]. I want the stack them and get output to have shape of [32,512,1536].
If I expand dimensions at axis=1 for 2d and concat. I am getting [32,513,768]. So how to get [32,512,1536] as my output shape of tensor?
Short answer: you will have to repeat the 2D tensor along axis 1 512 times to get a 3D tensor of shape [32, 512, 768]. This 3D tensor when concatenated with the other 3D tensor along the last dimension will give a tensor of shape [32, 512, 1536]. You need to make sure this repetition in desired.
Longer extension:
Let's take a much simpler case:
Take a 1D tensor (1, 2, 3, 4, 5). Say you need to concatenate this to a 2D tensor of shape [2, 5], say ((6, 7, 8, 9, 10), (11, 12, 13, 14, 15)). Note that this is a simplified version of your problem, with smaller tensors and no batch dimension.
One way to combine these tensors is to get a tensor of shape [3, 5]. Here, you would expand the 1D tensor to a 2D tensor having shape [1, 5], and concatenate along axis 0. This will give the result ((1, 2, 3, 4, 5), (6, 7, 8, 9, 10), (11, 12, 13, 14, 15)). When applied to your problem, this gives the resulting [32, 513, 768] tensor you have.
The second way would give the tensor ((1, 2, 3, 4, 5, 6, 7, 8, 9, 10), (1, 2, 3, 4, 5, 11, 12, 13, 14, 15)), having shape [2, 10]. As you can see, this requires (1, 2, 3, 4, 5) to be repeated twice. So, you'll have to expand the 1D tensor to get the shape [1, 5], and repeat it to get a tensor of shape [2, 5]. This tensor can then be concatenated with the other 2D tensor. In your case, you will expand the 2D tensor to shape [32, 1, 768], then repeat it 512 times along axis 1 to get a tensor of shape [32, 512, 768], which will be concatenated with the other 3D tensor.
When going for the second method, ensure that you really want to repeat the smaller tensor across all entries of the second tensor.
You can try this:
import torch
a = torch.ones([32,768])
b = torch.ones([32,512,768])
result = torch.cat([a[:, None, :].repeat(1, 512,1), b], dim=2)
I'm interested in using the Networkx Python package to perform network analysis on convolutional neural networks. To achieve this I want to extract the edge and weight information from Keras model objects and put them into a Networkx Digraph object where it can be (1) written to a graphml file and (2) be subject to the graph analysis tools available in Networkx.
Before jumping in further, let me clarify and how to consider pooling. Pooling (examples: max, or average) means that the entries within a convolution window will be aggregated, creating an ambiguity on 'which' entry would be used in the graph I want to create. To resolve this, I would like every possible choice included in the graph as I can account for this later as needed.
For the sake of example, let's consider doing this with VGG16. Keras makes it pretty easy to access the weights while looping over the layers.
from keras.applications.vgg16 import VGG16
model = VGG16()
for layer_index, layer in enumerate(model.layers):
GW = layer.get_weights()
if layer_index == 0:
print(layer_index, layer.get_config()['name'], layer.get_config()['batch_input_shape'])
elif GW:
W, B = GW
print(layer_index, layer.get_config()['name'], W.shape, B.shape)
else:
print(layer_index, layer.get_config()['name'])
Which will print the following:
0 input_1 (None, 224, 224, 3)
1 block1_conv1 (3, 3, 3, 64) (64,)
2 block1_conv2 (3, 3, 64, 64) (64,)
3 block1_pool
4 block2_conv1 (3, 3, 64, 128) (128,)
5 block2_conv2 (3, 3, 128, 128) (128,)
6 block2_pool
7 block3_conv1 (3, 3, 128, 256) (256,)
8 block3_conv2 (3, 3, 256, 256) (256,)
9 block3_conv3 (3, 3, 256, 256) (256,)
10 block3_pool
11 block4_conv1 (3, 3, 256, 512) (512,)
12 block4_conv2 (3, 3, 512, 512) (512,)
13 block4_conv3 (3, 3, 512, 512) (512,)
14 block4_pool
15 block5_conv1 (3, 3, 512, 512) (512,)
16 block5_conv2 (3, 3, 512, 512) (512,)
17 block5_conv3 (3, 3, 512, 512) (512,)
18 block5_pool
19 flatten
20 fc1 (25088, 4096) (4096,)
21 fc2 (4096, 4096) (4096,)
22 predictions (4096, 1000) (1000,)
For the convolutional layers, I've read that the tuples will represent (filter_x, filter_y, filter_z, num_filters) where filter_x, filter_y, filter_z give the shape of the filter and num_filters is the number of filters. There's one bias term for each filter, so the last tuple in these rows will also equal the number of filters.
While I've read explanations of how the convolutions within a convolutional neural network behave conceptually, I seem to be having a mental block when I get to handling the shapes of the layers in the model object.
Once I know how to loop over the edges of the Keras model, with Networkx I should be able to easily code the construction of the Networkx object. The code for this might loosely resemble something like this, where keras_edges is an iterable that contains tuples formatted as (in_node, out_node, edge_weight).
import networkx as nx
g = nx.DiGraph()
g.add_weighted_edges_from(keras_edges)
nx.write_graphml(g, 'vgg16.graphml')
So to be specific, how do I loop over all the edges in a way that accounts for the shape of the layers and the pooling in the way I described above?
Since Keras doesn't have an edge element, and a Keras node seems to be something totally different (a Keras node is an entire layer when it's used, it's the layer as presented in the graph of the model)
So, assuming you are using the smallest image possible (which is equal to the kernel size), and that you're creating nodes manually (sorry, I don't know how it works in networkx):
For a convolution that:
Has i input channels (channels in the image that comes in)
Has o output channels (the selected number of filters in keras)
Has kernel_size = (x, y)
You already know the weights, which are shaped (x, y, i, o).
You would have something like:
#assuming a node here is one pixel from one channel only:
#kernel sizes x and y
kSizeX = weights.shape[0]
kSizeY = weights.shape[1]
#in and out channels
inChannels = weights.shape[2]
outChannels = weights.shape[3]
#slide steps x
stepsX = image.shape[0] - kSizeX + 1
stepsY = image.shape[1] - kSizeY + 1
#stores the final results
all_filter_results = []
for ko in range(outChannels): #for each output filter
one_image_results = np.zeros((stepsX, stepsY))
#for each position of the sliding window
#if you used the smallest size image, start here
for pos_x in range(stepsX):
for pos_y in range(stepsY):
#storing the results of a single step of a filter here:
one_slide_nodes = []
#for each weight in the filter
for kx in range(kSizeX):
for ky in range(kSizeY):
for ki in range(inChannels):
#the input node is a pixel in a single channel
in_node = image[pos_x + kx, pos_y + ky, ki]
#one multiplication, single weight x single pixel
one_slide_nodes.append(weights[kx, ky, ki, ko] * in_node)
#so, here, you have in_node and weights
#the results of each step in the slide is the sum of one_slide_nodes:
slide_result = sum(one_slide_nodes)
one_image_results[pos_x, pos_y] = slide_result
all_filter_results.append(one_image_results)
I'd like to concatenate two tensors of shape=(None, 16) in alternate fashion (so the result tensor has to be shape=(None, 32) where the first array of the first tensor is mixed in alternate fashion with the first one of the second tensor and so on.
How can I do it?
I can't loop on tensors because of unknown shape[0], zip function isn't supported for tensors (tensor object is not iterable).
I'm using Tensorflow with Python3.
Assuming the two tensors have the same shape in the outer (None) dimension and you want to alternate between rows of the two tensors, you can do this by adding a dimension with tf.expand_dims(), concatenating with tf.concat(), then reshaping with tf.reshape():
# Use these tensors as example inputs, but the shape need not be statically known.
x = tf.ones([37, 16])
y = tf.zeros([37, 16])
x_expanded = tf.expand_dims(x, 2) # shape: (37, 16, 1)
y_expanded = tf.expand_dims(y, 2) # shape: (37, 16, 1)
concatted = tf.concat([x_expanded, y_expanded], 2) # shape: (37, 16, 2)
result = tf.reshape(concatted, [-1, 32]) # shape: (37, 32)
I am finding it impossible to get the max tensor in an n-dimensional array, even by summing the tensors and using gather or gather_nd.
By max tensor I mean the set of weights with the highest sum.
I have a tensor of shape (-1, 4, 30, 256) where 256 is the weights.
I need to get the maximum set of weights for each (-1, 0, 30), (-1, 1, 30), (-1, 2, 30) and (-1, 3, 30), so under each tensor in the 2nd dimension.
This would ideally result in a (-1, 4, 256) tensor.
reduce_max and any other max function will only return the maximum element values within the last dimension, not the maximum tensor (which is the set of weights with the highest sum) in the dimension itself. I have tried:
p1 = tf.reduce_sum(tensor, axis=3) # (-1, 4, 30)
p2 = tf.argmax(p1, 2) # (-1, 4)
Which gives the appropriate index values for the 3rd dimension:
[[0, 2, 2, 0],
[0, 1, 3, 0],
...
But running tf.gather or tf.gather_nd on the above does not work, even when splitting my data beforehand and using different axes.
Further, I can get the appropriated indexes if I use gather_nd by hand, eg:
tf.gather_nd(out5, [[0,0,0], [0,1,2], [0,2,2], [0,3,0], [1,0,0], [1,1,2], [1,2,2], [1,3,1]])
But as we are using a tensorflow variable of an unknown first dimension, I cannot build these indexes.
I have searched through related workarounds and found nothing applicable.
Can anyone tell me how to accomplish this? Thanks!
edit for clarification:
The maximum tensor of weights would be the set of weights with the highest sum:
[[ 1, 2, 3], [0, 0, 2], [1, 0, 2]] would be [1, 2, 3]
I figured it out using map_fn:
I reshaped my tensor to (-1, 120, 256)
tfr = tf.reshape(sometensor, ((-1, 120, 256)))
def func(slice):
f1 = tf.reduce_sum(slice, axis=1)
f2 = tf.argmax(f1)
return(slice[f2])
bla = tf.map_fn(func, tfr)
Which returns (-1,256) with the greatested summed vector (highest set of weights).
Basically, map_fn will iterate along the 2nd to last axis, so it slices a chunk of (120,256) to func repeatedly (how ever many entries are on the first axis). It then returns the appropriate (1,256) chunk by chunk which, voila, gives the answer.