I have a problem in tensorflow with tf.gradients applied to pooling:
[edit]: I was able to reproduce my expectation by changing the equation to:
gradpooltest, = tf.gradients((pooltest * pooltest)/2 , [x1])
Anyway, I am not sure why I have to do it this way and people answered below do not seem to understand my problem.
input x1:
[[ 0. 0. 0. 0. 0. 0.]
[ 0. 2. 2. 2. 0. 0.]
[ 0. -2. 0. 0. 2. 1.]
[ 0. 1. 0. 1. 2. 2.]
[ 0. 1. 1. 2. 0. 1.]
[ 0. -2. 2. 1. -1. 1.]]
pooling test forward:
[[ 2. 2. 0.]
[ 1. 1. 2.]
[ 1. 2. 1.]]
tf.gradients pool test backward:
[[ 0. 0. 0. 0. 1. 0.]
[ 0. 1. 1. 0. 0. 0.]
[ 0. 0. 0. 0. 1. 0.]
[ 0. 1. 0. 1. 0. 0.]
[ 0. 1. 0. 1. 0. 1.]
[ 0. 0. 0. 0. 0. 0.]]
but I expect actually this result by tf.gradients pool test backward:
0 0 0 0 0 0
0 2 2 0 0 0
0 0 0 0 2 0
0 1 0 1 0 0
0 1 0 0 0 1
0 0 2 0 0 0
I don't understand the tf result for tf.gradients pool test backward. (Looks like tensorflow only returns the store matrix for the locations??). Any idea why tf does not return the actual upsampling result?
Here is my code:
import numpy as np
import tensorflow as tf
sess = tf.Session()
#init input-----------------------------------------------------------
init1=np.array([ [0,0,0,0,0,0],
[0,2,2,2,0,0],
[0,-2,0,0,2,1],
[0,1,0,1,2,2],
[0,1,1,2,0,1],
[0,-2,2,1,-1,1] ],dtype="float32")
init2 = init1.reshape(1,6,6,1)
x1 = tf.Variable(init2)
#init weight-----------------------------------------------------------
init3 = np.array( [[[[3, 5], [2, -1]]]], dtype="float32")
init4 = init3.reshape(2,2,1,1)
w1 = tf.Variable(init4)
#init model-----------------------------------------------------------
model = tf.initialize_all_variables()
sess.run(model)
#print values-----------------------------------------------------------
print('x1:')
#print sess.run(x6)
x1y = tf.reshape(x1, [6, 6])
print sess.run(x1y)
###################################
#ff: pooling
###################################
#needs 4D volumes as inputs:
pooltest = tf.nn.max_pool(x1, ksize=[1, 2, 2, 1], strides=[1, 2, 2, 1], padding='SAME')
print('pooltest:')
#print sess.run(pooltest)
pooltesty = tf.reshape(pooltest, [3, 3])
print sess.run(pooltesty)
###################################
#bw: pooling
###################################
#needs 4D volumes as inputs:
gradpooltest, = tf.gradients(pooltest , [x1])
print('gradpooltest:')
#print sess.run(gradpooltest)
gradpooltesty = tf.reshape(gradpooltest, [6, 6])
print sess.run(gradpooltesty)
sess.close()
You are computing gradients of maxpool operation and this is correct — they are 1 at maximums and 0 at other locations.
Please refer to the following page: http://cs231n.github.io/optimization-2/#patterns-in-backward-flow
Imagine your max-pooling operation with a kernel size of 2x2 implemented like:
max(x1, x2, x3, x4)
Where x1, ..., x4 are the location in the input image under the kernel.
In the forward pass, you extract the maximum value, for example:
max(x1, x2, x3, x4) = x2
This means that for these 4 variables, in the forward pass only the x2 variable will be passed trough the network.
In the backward pass, thus, you have only one variable to calculate the derivative and its derivative is 1.
Therefore the output you got is correct, what do you expect is not.
Related
I tried to build my CNN layer. but I had one problem.
The CNN layer's input is image data. and I used to the backend as TensorFlow. so the CNN layer's input shape must be (number of data, height, width, channel)!
I made the data to use the list.
Let show example.
the image is 2x2 file. and the data is 'a', 'b', 'c'.
a = np.zeros((2,2))
b = np.ones((2,2))
c = np.ones((2,2)) * 2
l = [a,b,c]
l = np.array(l)
print(l)
print(l.shape)
print is
[[[0. 0.]
[0. 0.]]
[[1. 1.]
[1. 1.]]
[[2. 2.]
[2. 2.]]]
(3, 2, 2)
I want to input the 'l' data. but I was so confused.
To match the shape (number of data, height, width, channel),
what is the correct method transpose() or reshape().
the example's method output is here.
new = np.transpose(l, (1,2,0))
print("=============================")
print(new)
print(new.shape)
print("=============================")
old = np.reshape(l, (2,2,3))
print(old)
print(old.shape)
[[[0. 1. 2.]
[0. 1. 2.]]
[[0. 1. 2.]
[0. 1. 2.]]]
(2, 2, 3)
=============================
[[[0. 0. 0.]
[0. 1. 1.]]
[[1. 1. 2.]
[2. 2. 2.]]]
(2, 2, 3)
Two Method's shape is (2, 2, 3) so there's no error! but I had some suspicion.
I think the transpose method is right. but there's no evidence to use this method. what is correct?
Thank you.
I have a network that takes in a sequence of one-hot-encoded observations and feeds it into an LSTM in batches and sequences of pre-determined length.
I am trying to introduce a modification, whereby along with feeding the sequences of actual observations, I could also feed a 'supplementary' sequence, which would reflect something about each corresponding observation in the sequence. For example, if I feed a sequence of [1,2,3,4,5], I'd also like to feed another sequence of say [0,0,1,0,0] to indicate that '3' in the original sequence has a certain property, while others don't.
Following someone's suggestion, I am trying to concatenate the one-hot-encoded original sequence with this supplementary sequence (which I call the 'mode') but while this runs OK, all I seem to be achieving ultimately is adding an extra element to the one-hot vectors, which become not-so one-hot anymore)
I have reduced the experimental code to the following minimum:
BATCH = 2
SEQ = 3
VOCAB = 5
CELL_SIZE = 4
X = tf.placeholder(tf.int32, [BATCH, SEQ])
X_hot = tf.one_hot(X, VOCAB, 1.0, 0.0)
X_mode = tf.placeholder(tf.float32, [BATCH, SEQ])
X_mode_exp = tf.expand_dims(X_mode, axis=2)
X_tuple = tf.concat([X_hot, X_mode_exp], axis=2)
Hin = tf.placeholder(tf.float32, [BATCH, CELL_SIZE])
cell = rnn.GRUCell(CELL_SIZE)
Y, H = tf.nn.dynamic_rnn(cell, X_tuple, dtype=tf.float32)
with tf.Session() as sess:
gInit = tf.global_variables_initializer().run()
h = np.zeros([BATCH, CELL_SIZE])
x = np.array([[1,2,3],[3,2,1]])
x_mode = np.array([[0,1,0],[1,0,1]])
x_h_out, x_tuple_out, hout = sess.run([X_hot, X_tuple, H], feed_dict={X:x, X_mode: x_mode, Hin: h})
print('x_hot:\n{}\n'.format(x_h_out))
print('x_tuple:\n{}'.format(x_tuple_out))
Which produces the following output:
x_hot:
[[[ 0. 1. 0. 0. 0.]
[ 0. 0. 1. 0. 0.]
[ 0. 0. 0. 1. 0.]]
[[ 0. 0. 0. 1. 0.]
[ 0. 0. 1. 0. 0.]
[ 0. 1. 0. 0. 0.]]]
x_tuple:
[[[ 0. 1. 0. 0. 0. 0.]
[ 0. 0. 1. 0. 0. 1.]
[ 0. 0. 0. 1. 0. 0.]]
[[ 0. 0. 0. 1. 0. 1.]
[ 0. 0. 1. 0. 0. 0.]
[ 0. 1. 0. 0. 0. 1.]]]
What could be a better alternative approach to achieving this? Once again, what I'm after is feeding a line of 'additional information' into the LSTM which would reflect some additional information about each element in the sequence.
In Tensor, I don't understand the value of Variable. Below is my code, I think after I do
sess.run()
The value of W should be calculated, however, After print it , I find it didn't change.
The code is an MNIST example code from TensorFlow website. Anyone can explain why W doesn't change?
from tensorflow.examples.tutorials.mnist import input_data
import tensorflow as tf
mnist = input_data.read_data_sets("MNIST_dat/",one_hot=True)
x= tf.placeholder(tf.float32,[None,784])#need input x
W= tf.Variable(tf.zeros([784,10]))
b= tf.Variable(tf.zeros([10]))
y= tf.nn.softmax(tf.matmul(x,W)+b)
y_= tf.placeholder(tf.float32,[None,10])#need input y
cross_entropy = tf.reduce_mean(-tf.reduce_mean(y_*tf.log(y),reduction_indices=[1]))
train_step = tf.train.GradientDescentOptimizer(0.5).minimize(cross_entropy)
init = tf.global_variables_initializer()
sess = tf.Session()
sess.run(init)
for i in range(1000):
batch_xs,batch_ys = mnist.train.next_batch(100)
sess.run(train_step,feed_dict={x:batch_xs,y_:batch_ys})
ww=W.eval(sess)
print(ww)
OK, when I run your code, the output I get looks like this
[[ 0. 0. 0. ..., 0. 0. 0.]
[ 0. 0. 0. ..., 0. 0. 0.]
[ 0. 0. 0. ..., 0. 0. 0.]
...,
[ 0. 0. 0. ..., 0. 0. 0.]
[ 0. 0. 0. ..., 0. 0. 0.]
[ 0. 0. 0. ..., 0. 0. 0.]]
But you have to realize, that W is 784 by a 100 elements, this display only shows you a few values at the start and end of that. And for most images in MNIST, the first and the last few pixels are not going to be significant (the actual important data is somewhere in the middle, where the actual digit is drawn, right?). But if I change the print statement to
print(ww.min(), ww.max())
I get this output
-0.330358 0.429428
Which means that some weights are being trained, as expected.
So I am trying to initialize my variable with some specific weights:
W = tf.Variable(np.eye(19) , name = 'Diag')
But if i now run this code:
with tf.Session() as sess:
sess.run(tf.initialize_all_variables())
print(W.eval())
I end up with a zeros matrix:
[[ 0. 0. 0.]
[ 0. 0. 0.]
[ 0. 0. 0.]]
I do not understand what is going on since e.g.
W = tf.Variable([1,2,3] , name = 'Diag')
preserves the values. What should I do?
I guess the tf.initialize_all_variables() overwrites my values but without it I get the FailedPreconditionError that complains about unitialized variables.
I need to convert my string labels into vectors like [0, 0, ... , 1, ... 0].
As far as I could understand this is something that called one hot vector.
I have 10 classes, so 10 different string labels.
Could anyone please help with direct and inverse transformation?
I'm newbie in tensorflow so please be kind.
The forward direction is easy, since there's the tf.one_hot op:
import tensorflow as tf
original_indices = tf.constant([1, 5, 3])
depth = tf.constant(10)
one_hot_encoded = tf.one_hot(indices=original_indices, depth=depth)
with tf.Session():
print(one_hot_encoded.eval())
Outputs:
[[ 0. 1. 0. 0. 0. 0. 0. 0. 0. 0.]
[ 0. 0. 0. 0. 0. 1. 0. 0. 0. 0.]
[ 0. 0. 0. 1. 0. 0. 0. 0. 0. 0.]]
The inverse of this isn't too bad either, with tf.where to find the non-zero indices:
def decode_one_hot(batch_of_vectors):
"""Computes indices for the non-zero entries in batched one-hot vectors.
Args:
batch_of_vectors: A Tensor with length-N vectors, having shape [..., N].
Returns:
An integer Tensor with shape [...] indicating the index of the non-zero
value in each vector.
"""
nonzero_indices = tf.where(tf.not_equal(
batch_of_vectors, tf.zeros_like(batch_of_vectors)))
reshaped_nonzero_indices = tf.reshape(
nonzero_indices[:, -1], tf.shape(batch_of_vectors)[:-1])
return reshaped_nonzero_indices
with tf.Session():
print(decode_one_hot(one_hot_encoded).eval())
Prints:
[1 5 3]