x = tf.gather(tf.ragged.constant([[0],[2,3,4],[3,4]]), tf.constant([[0,1,2],[1,2,1]]))
tf.shape(x)
has shape
<DynamicRaggedShape lengths=[2, (3, 3), (1, 3, 2, 3, 2, 3)] num_row_partitions=2>
however I expect
<DynamicRaggedShape lengths=[2, 3, (1, 3, 2, 3, 2, 3)] num_row_partitions=2>
I tried to reshape it, but is there a better solution by giving more informtion to tf.gather?
Related
How do I get the Kernel values from tf.keras.layers.Conv2D?
Here is my code:
#input image is 5 X 5 and 1 channel
input_shape = (1, 1, 5, 5)
x = tf.random.normal(input_shape)
y = tf.keras.layers.Conv2D(
2, 2, activation= tf.nn.relu, input_shape=input_shape,
data_format='channels_first')(x)
I am using tf version 2.2
I have tried y.get_weights() and this didn't work I got:
AttributeError: 'tensorflow.python.framework.ops.EagerTensor'
object has no attribute 'get_weights'
You need to actually store the layer in a variable. In your code, y is the result of the convolution. For example
input_shape = (1, 1, 5, 5)
x = tf.random.normal(input_shape)
conv_layer = tf.keras.layers.Conv2D(
2, 2, activation= tf.nn.relu, input_shape=input_shape,
data_format='channels_first')
y = conv_layer(x)
Now you should be able to use conv_layer.get_weights().
Let's assume the following function:
from tensorflow.python.keras import backend as K
def broadcast_sum(a, b):
a = K.expand_dims(a, 1)
b = K.expand_dims(b, 2)
c = a + b
cs = K.shape(c)
return K.reshape(c, (cs[0], -1, cs[-1]))
Given the two tensors of shapes (1, 3, 2) and (1, 4, 2), it correctly returns:
>>> broadcast_sum(K.placeholder((1, 3, 2)), K.placeholder((1, 4, 2)))
>>> <tf.Tensor 'Reshape_2:0' shape=(1, 12, 2) dtype=float32>
Right now, this function works only with 3D input (because of the reshape line). My question is, how can I make this work with any shape (using the same function) without knowing the shape? Of course, I'm assuming the inputs are of the same shape and at least a 3D. But how can I have a single function that works with 3D, 4D and so on?
And I'm assuming that it's always the second dimension (from left) that the function will broadcast and the rest of the dimensions are identical between the two inputs. Here are the shapes that I want to make the same function to work with:
>>> broadcast_sum(K.placeholder((1, 3, 5, 2)), K.placeholder((1, 4, 5, 2)))
>>> <tf.Tensor 'Reshape_3:0' shape=(1, 60, 2) dtype=float32>
Of course, the returned tensor is wrong right now. It should be of shape (1, 12, 5, 2).
[UPDATE]
Please also consider that the first dimension (the batch size) could be None. In fact, any of the dimensions except the rightmost one could be None.
And I'm assuming that it's always the second dimension (from left)
that the function will broadcast and the rest of the dimensions are
identical between the two inputs.
Based on this, I reuse the shape information from one of the inputs.
from tensorflow.python.keras import backend as K
def broadcast_sum(a, b):
final_shape = (a.shape[0], -1, *a.shape[2:])
a = K.expand_dims(a, 1)
b = K.expand_dims(b, 2)
c = a + b
return K.reshape(c, final_shape)
print(broadcast_sum(K.placeholder((1, 3, 2)), K.placeholder((1, 4, 2))))
print(broadcast_sum(K.placeholder((1, 3, 5, 2)), K.placeholder((1, 4, 5, 2))))
Tensor("Reshape:0", shape=(1, 4, 3, 2), dtype=float32)
Tensor("Reshape_1:0", shape=(1, 12, 5, 2), dtype=float32)
Imagine I have a tensor of shape (batch_size, a, ... , c, d, e)where are a, ... ,c,d,e are defined integers. For example (batch_size, 500, 3, 2, 2, 69) or (batch_size, 2, 2).
My question is for all tensors but let's stick to the example of tensor1.get_shape() = (?, 500, 3, 2, 2, 69)
Given that I have tensor2 with tensor2.get_shape() = (?, 500, 3, 2, 2, 14) containing indices of the last axis of tensor1, I have 2 problems:
1) I want to construct a mask for tensor1 of shape (?, 500, 3, 2, 2, 69) from tensor2. For example a possible row along the last axis for tensor2 would be [1,8,3,68,2,4,58,19,20,21,26,48,56,11] but since tensor2 is constructed from tensor1 these indices vary for new input. These are the indices of the last axis that have to be kept of tensor1. Everything else has to be masked out.
2) given that I have the mask of shape (?, 500, 3, 2, 2, 69) for tensor1, how do I mask out the undesired values while maintaining the batch size dimension? The masked out tensor should have shape (?, 500, 3, 2, 2, 14).
Answers in keras or numpy would also be neat, although knowing how to do it in numpy wouldn't solve my problem, I'd still like to know.
answer to 1:
tf.gather_nd(mask, [tf.range(tf.shape(tensor1)[0])[:,None, None, None, None, None],tf.range(tf.shape(tensor1)[1])[:,None, None, None, None],tf.range(tf.shape(tensor1)[2])[:,None, None, None],tf.range(tf.shape(tensor1)[3])[:,None, None],tf.range(tf.shape(tensor1)[4])[:,None],tensor2])
There is probably no solution to 2. I will try pytorch.
I have an input tensor of size [3, 3, 3, 64]. I would like to add an extra channel to the input at runtime, so that the tensor is of the form [3, 3, 4, 64]. The new channel weights can be initialized to 0.
My question is, how do I insert the new channel data to grow the channel dimension?
Use tf.concat
import tensorflow as tf
a = tf.ones([3, 3, 3, 64]) # your original stuff
b = tf.zeros([3, 3, 1, 64])
c = tf.concat([a, b], axis=2)
print c
Will get Tensor("concat_1:0", shape=(3, 3, 4, 64), dtype=float32)
I'm getting this error message when using conv2d_transpose:
W tensorflow/core/common_runtime/executor.cc:1102] 0x7fc81f0d6250 Compute status: Invalid argument: Conv2DBackpropInput: Number of rows of out_backprop doesn't match computed: actual = 32, computed = 4
[[Node: generator/g_h1/conv2d_transpose = Conv2DBackpropInput[T=DT_FLOAT, padding="SAME", strides=[1, 2, 2, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/cpu:0"](generator/g_h1/conv2d_transpose/output_shape, generator/g_h1/w/read, _recv_l_0)]]
However, it occurs after the graph is built while compiling the loss function (Adam). Any ideas on what would cause this? I suspect it's related to the input dimensions but I'm not sure exactly why.
Full error: https://gist.github.com/jimfleming/75d88e888044615dd6e3
Relevant code:
# l shape: [batch_size, 32, 32, 4]
output_shape = [self.batch_size, 8, 8, 128]
filter_shape = [7, 7, 128, l.get_shape()[-1]]
strides = [1, 2, 2, 1]
with tf.variable_scope("g_h1"):
w = tf.get_variable('w', filter_shape, initializer=tf.random_normal_initializer(stddev=0.02))
h1 = tf.nn.conv2d_transpose(l, w, output_shape=output_shape, strides=strides, padding='SAME')
h1 = tf.nn.relu(h1)
output_shape = [self.batch_size, 16, 16, 128]
filter_shape = [7, 7, 128, h1.get_shape()[-1]]
strides = [1, 2, 2, 1]
with tf.variable_scope("g_h2"):
w = tf.get_variable('w', filter_shape, initializer=tf.random_normal_initializer(stddev=0.02))
h2 = tf.nn.conv2d_transpose(h1, w,output_shape=output_shape, strides=strides, padding='SAME')
h2 = tf.nn.relu(h2)
output_shape = [self.batch_size, 32, 32, 3]
filter_shape = [5, 5, 3, h2.get_shape()[-1]]
strides = [1, 2, 2, 1]
with tf.variable_scope("g_h3"):
w = tf.get_variable('w', filter_shape, initializer=tf.random_normal_initializer(stddev=0.02))
h3 = tf.nn.conv2d_transpose(h2, w,output_shape=output_shape, strides=strides, padding='SAME')
h3 = tf.nn.tanh(h3)
Thanks for the question! You're exactly right---the problem is that the input and output dimensions being passed to tf.nn.conv2d_transpose don't agree. (The error may be detected when computing gradients, but the gradient computation isn't the problem.)
Let's look at just the first part of your code, and simplify it a little bit:
sess = tf.Session()
batch_size = 3
output_shape = [batch_size, 8, 8, 128]
strides = [1, 2, 2, 1]
l = tf.constant(0.1, shape=[batch_size, 32, 32, 4])
w = tf.constant(0.1, shape=[7, 7, 128, 4])
h1 = tf.nn.conv2d_transpose(l, w, output_shape=output_shape, strides=strides, padding='SAME')
print sess.run(h1)
I replaced the variables with constants --- it's easier to see what's going on.
If you try to run this code, you get a similar error:
InvalidArgumentError: Conv2DCustomBackpropInput: Size of out_backprop doesn't match computed: actual = 32, computed = 4
[[Node: conv2d_transpose_6 = Conv2DBackpropInput[T=DT_FLOAT, data_format="NHWC", padding="SAME", strides=[1, 2, 2, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/cpu:0"](conv2d_transpose_6/output_shape, Const_25, Const_24)]]
Now, the error is a little misleading --- it talks about the 'out_backprop' argument to 'Conv2DCustomBackpropInput'. The key is that tf.nn.conv2d_transpose is actually just the gradient of tf.nn.conv2d, so Tensorflow uses the same code internally (Conv2DCustomBackpropInput) to compute the gradient of tf.nn.conv2d and to compute tf.nn.conv2d_transpose.
The error means that the 'output_shape' you requested is not possible, given the shapes of 'l' and 'w'.
Since tf.nn.conv2d_transpose is the backward (gradient) counterpart of tf.nn.conv2d, one way to see what the correct shapes should be is to use the corresponding forward operation:
output = tf.constant(0.1, shape=output_shape)
expected_l = tf.nn.conv2d(output, w, strides=strides, padding='SAME')
print expected_l.get_shape()
# Prints (3, 4, 4, 4)
That is, in the forward direction, if you provided a tensor of shape 'output_shape', you would get out a tensor of shape (3, 4, 4, 4).
So one way to fix the problem is to change the shape of 'l' to (3, 4, 4, 4); if you change the code above to:
l = tf.constant(0.1, shape=[batch_size, 4, 4, 4])
everything works fine.
In general, try using tf.nn.conv2d to get a feel for what the relationship between the tensor shapes is. Since tf.nn.conv2d_transpose is its backward counterpart, it has the same relationship between input, output and filter shapes (but with the roles of the input and output reversed.)
Hope that helps!
Using padding='SAME' in tf.nn.conv2d_transpose() function may works too