Look at the code:
import tensorflow as tf
x = tf.Variable(1)
x = tf.stop_gradient(x)
y = 2 * x
g = tf.gradients(y, x)
init = tf.global_variables_initializer()
with tf.Session() as sess:
init.run()
print(sess.run(g))
I want to freeze x and the gradient of y wrt x is zero, but the output is 2, so what's wrong?
Update
import tensorflow as tf
x0 = tf.Variable(1)
x1 = tf.stop_gradient(x0)
y = 2 * x1
g = tf.gradients(y, x0)
init = tf.global_variables_initializer()
with tf.Session() as sess:
init.run()
print(sess.run(g))
I use x1 which don't overwrite x0, but when I run this code, it will raise a error. If tf.stop_gradient act as tf.identity, I think x0 will have a path to y in computation graph and the gradient is 0 rather than raising a error? Can someone tell me what tf.stop_gradient does indeed?
tf.stop_gradient() stops the gradient computation at the specified point in the computation graph. Applying it to the variable is "too late", but you can apply it to y.
import tensorflow as tf
x = tf.Variable(1)
y = tf.stop_gradient(2 * x)
g = tf.gradients(y, x)
init = tf.global_variables_initializer()
with tf.Session() as sess:
init.run()
print(sess.run(g))
Note that this will not output 0 but instead throws an error, since the gradient of y w.r.t. to x is undefined in this case and you explicitly ask for it. In a real-world scenario this would probably not be the case.
Related
I do generally use Tensorflow Keras backend but recently I'm working on a project where there is need of T.F 1.x
I'm trying a simple code, but getting an error:
x2 = tf.constant(-2.0, name="x", dtype=tf.float32)
a = tf.placeholder(name='a',dtype=tf.float32)
b = tf.constant(13.0, name="b", dtype=tf.float32)
y = tf.Variable(tf.add(tf.multiply(a, x2), b))
init = tf.global_variables_initializer()
with tf.Session() as session:
print(session.run(init,feed_dict={a:5.0}))
ValueError: initial_value must have a shape specified at y=Variable()... line.
Does anyone know the solution? Thanks in advance
The variable "y" is dependent on the variable "a" which is a placeholder. So defining the shape of "a" will make code run properly
x2 = tf.constant(-2.0, name="x", dtype=tf.float32)
a = tf.placeholder(name='A',shape=(1,),dtype=tf.float32)
b = tf.constant(13.0, name="b", dtype=tf.float32)
y = tf.Variable(tf.add(tf.multiply(a, x2), b))
init = tf.global_variables_initializer()
with tf.Session() as session:
print(session.run(init,feed_dict={a:[5.0]}))
print(session.run(y))
I have an outer loop training my model and update my x; however, each iteration in outer loop; it requires to train another model and to train the inner model; it requires the current outer iteration's value x
The general frame is as follows
with tf.Session() as sess:
# do some initial computation
x = ......
for i in (range(iters)):
loss = func(x) # compute the loss function
train_op = tf.train.AdamOptimizer(1e-4).minimize(loss)
sess.run(tf.global_variables_initializer())
for j in (range(train_steps)):
per_loss = session.run([loss])
sess.run([train_op])
# update x
x = .....
This implementation is very slow; so I decide to use placeholder
x_placeholder = tf.placeholder(tf.float64,....)
loss = func(x_placholder)
train_op = tf.train.AdamOptimizer(1e-4).minimize(loss)
with tf.Session() as sess:
# do some initial computation
x = ......
for i in (range(iters)):
sess.run(tf.global_variables_initializer())
for j in (range(train_steps)):
per_loss = session.run([train_op, loss],feed_dict={x_placeholder:x})
# update x
x = .....
However, this gives me error as follows
raise ValueError("No variables to optimize.")
ValueError: No variables to optimize.
when run the line train_op = tf.train.AdamOptimizer(1e-4).minimize(loss)
So I'm not sure how to correctly implement this in very efficient way.Any idea
Thanks
Something like this?
x_placeholder = tf.placeholder(tf.float64,....)
loss = func(x_placholder)
train_op = tf.train.AdamOptimizer(1e-4).minimize(loss)
with tf.Session() as sess:
# do some initial computation
x = ......
for i in (range(iters)):
sess.run(tf.global_variables_initializer())
for j in (range(train_steps)):
per_loss = session.run([train_op, loss],feed_dict={x_placeholder:x})
# update x
x = .....
import numpy as np
import tensorflow as tf
X_p = tf.placeholder(tf.float32,[None,3] )
y_p = tf.placeholder(tf.float32, [None,1])
print(X_p)
x = [[1,2,3],[1,2,3]]
y = [[1],[2]]
weight = tf.Variable(tf.random_normal([3,1]))
model = tf.nn.sigmoid(tf.matmul(X_p,weight)+1)
error = tf.reduce_sum(y * tf.log(model))
optimizer = tf.train.GradientDescentOptimizer(0.01).minimize(error)
init = tf.initialize_all_variables()
with tf.Session() as sess:
sess.run(init)
for x in range(100):
sess.run(optimizer, {X_p: x, y_p:y})
X_p is of shape [None,3], x is shape [2,3], y_p = [None,1], y = [2,1]
I don't really understand why placeholder is stopping numpy array to fetch the data.
The problem you are having is that you are overwriting your x variable by also using x as a loop variable. So when you try to passx into the feed dict you are passing the loop variable rather than your tensor. Try changing you loop variable to something else such as:
for i in range(100):
sess.run(optimizer, {X_p: x, y_p:y})
I was trying to assign a variable y in tensorflow which is to be dependent on x. But, even upon changing the value of x, y doesn't change.
import tensorflow as tf
sess = tf.Session()
x=tf.Variable(4,name='x')
model = tf.global_variables_initializer()
sess.run(model)
y=tf.Variable(2*x,name='y')
model = tf.global_variables_initializer()
sess.run(model)
sess.run(x)
sess.run(tf.assign(x,2))
print(sess.run(y))
I am expecting an output 4, but I'm getting 8. Any help would be appreciated.
Gramercy...
y=tf.Variable(2*x, name='y') just means y will be initialized by x*2, change this line into y = 2 * x will do as you expected.
I'm a total beginner to TensorFlow, and I'm trying to multiply two matrices together, but I keep getting an exception that says:
ValueError: Shapes TensorShape([Dimension(2)]) and TensorShape([Dimension(None), Dimension(None)]) must have the same rank
Here's minimal example code:
data = np.array([0.1, 0.2])
x = tf.placeholder("float", shape=[2])
T1 = tf.Variable(tf.ones([2,2]))
l1 = tf.matmul(T1, x)
init = tf.initialize_all_variables()
with tf.Session() as sess:
sess.run(init)
sess.run(feed_dict={x: data}
Confusingly, the following very similar code works fine:
data = np.array([0.1, 0.2])
x = tf.placeholder("float", shape=[2])
T1 = tf.Variable(tf.ones([2,2]))
init = tf.initialize_all_variables()
with tf.Session() as sess:
sess.run(init)
sess.run(T1*x, feed_dict={x: data}
Can anyone point to what the issue is? I must be missing something obvious here..
The tf.matmul() op requires that both of its inputs are matrices (i.e. 2-D tensors)*, and doesn't perform any automatic conversion. Your T1 variable is a matrix, but your x placeholder is a length-2 vector (i.e. a 1-D tensor), which is the source of the error.
By contrast, the * operator (an alias for tf.multiply()) is a broadcasting element-wise multiplication. It will convert the vector argument to a matrix by following NumPy broadcasting rules.
To make your matrix multiplication work, you can either require that x is a matrix:
data = np.array([[0.1], [0.2]])
x = tf.placeholder(tf.float32, shape=[2, 1])
T1 = tf.Variable(tf.ones([2, 2]))
l1 = tf.matmul(T1, x)
init = tf.initialize_all_variables()
with tf.Session() as sess:
sess.run(init)
sess.run(l1, feed_dict={x: data})
...or you could use the tf.expand_dims() op to convert the vector to a matrix:
data = np.array([0.1, 0.2])
x = tf.placeholder(tf.float32, shape=[2])
T1 = tf.Variable(tf.ones([2, 2]))
l1 = tf.matmul(T1, tf.expand_dims(x, 1))
init = tf.initialize_all_variables()
with tf.Session() as sess:
# ...
* This was true when I posted the answer at first, but now tf.matmul() also supports batched matrix multiplications. This requires both arguments to have at least 2 dimensions. See the documentation for more details.