Are the operations in tf.group() executed in order?
If they are not executed in order, is there a similar operation that makes them run in order? Or is there a clean way to run them in order?
My aim is to run operations A and B many times over and over again, i.e.
sess.run(A)
sess.run(B)
sess.run(A)
sess.run(B)
sess.run(A)
sess.run(B)
sess.run(A)
sess.run(B)
...
The operations do not necessarily execute in order.
Here is a test that proves this:
import tensorflow as tf
sess = tf.InteractiveSession()
a = tf.Variable(1.0)
b = tf.Variable(10.0)
c = tf.Variable(0.0)
grp = tf.group(tf.assign(c, a), tf.assign(c, b)) # this is the group op
for i in range(100):
sess.run(tf.global_variables_initializer()) # initialize c each time
sess.run(grp) # run the group op
print(sess.run(c)) # observe results
When I run this on a cpu, I get that some iterations produce 1.0 and some 10.0.
tf.group does not require the operations to be on the same device, which means that they could not be expected to follow an order.
If you want the operations to execute in order, make sure to build them with control_dependencies
import tensorflow as tf
sess = tf.InteractiveSession()
a = tf.Variable(1.0)
b = tf.Variable(10.0)
c = tf.Variable(0.0)
op1 = tf.assign(c, a)
with tf.get_default_graph().control_dependencies([op1]):
op2 = tf.assign(c, b) # op2 will execute only after op1
grp = tf.group(op1,op2)
for i in range(100):
sess.run(tf.global_variables_initializer())
sess.run(grp)
print(sess.run(c))
Related
This is my first code with tensor flow.
If I split the addition it works, but I am sure there's a way to add N tensors in one line.
import tensorflow as tf
# Create a graph.
g = tf.Graph()
# Establish the graph as the "default" graph.
with g.as_default():
# Assemble a graph consisting of the following three operations:
# * Two tf.constant operations to create the operands.
# * One tf.add operation to add the two operands.
x = tf.constant(8, name="x_const")
y = tf.constant(5, name="y_const")
z = tf.constant(4, name="z_const")
my_sum = tf.add_n(x, y, z, name="z_y_x_sum")
# Now create a session.
# The session will run the default graph.
with tf.Session() as sess:
print my_sum.eval()
Can you help me to figure out what's wrong?
You need to put the tensors into a list:
my_sum = tf.add_n([x, y, z], name="z_y_x_sum")
Look at the code:
import tensorflow as tf
x = tf.constant(1.0)
y = tf.constant(2.0)
z = tf.constant(3.0)
def f1():
return tf.Print(x, [x])
def f2():
return tf.Print(z, [z])
op = tf.cond(x>y, f1, f2)
with tf.Session() as sess:
sess.run(op)
I'm very puzzled, the output of tf.Print is 3.0
As we know, tf.Print(z, [z]) will output the value of z only when z is evaluated, but I don't think I have evaluated z.
Another question is about tf.cond, how does it add node to graph, for example how does add tf.Print to graph, I think it should relate some tensor with the return of tf.Print, otherwise tf.Print won't be executed.
I'm so puzzled.
I think you might have got the order of the arguments to tf.cond confused. The call:
tf.cond(predicate, f, g)
is equivalent to "if predicate is true then evaluate f, otherwise evaluate g"
In your example, since your predicate x > y is false, f2 is evaluated
Note
Since tensorflow 1.4, tf.cond will accept key-word arguments true_fn and false_fn, so you can avoid any confusion by writing:
tf.cond(predicate, true_fn=f, false_fn=g)
# Or equivalently...
tf.cond(predicate, false_fn=g, true_fn=f)
import tensorflow as tf
x = tf.Variable(1)
x = x + 1
init = tf.global_variables_initializer()
with tf.Session() as sess:
init.run()
print(sess.run(x))
print(sess.run(x))
The output is
2
2
But I think the output should be
2
3
The first run, x has been update to 2, and the second run, x should be 3.
Who can tell me why the second run of x is 2 either? If x can't be update by the first run, how is the parameter of neural network update?
Update
x = tf.Variable(1)
x.assign(x+1)
init = tf.global_variables_initializer()
with tf.Session() as sess:
init.run()
#print(x)
print(sess.run(x))
print(sess.run(x))
The output is
1
1
It is amazing.
Here's an analysis of your and Ishant Mrinal examples, it should help you understand what's going on here.
Example 1
x = tf.Variable(1)
Creation of a python variable x. Creation of a Tensorflow node variable_1. The python variable x holds a logical pointer to the node variable_1.
x = x + 1
Python assignment, destructive operation.
x now holds a pointer to the operation sum(variable_1, constant(1)).
init = tf.global_variables_initializer()
with tf.Session() as sess:
init.run()
usual code of initialization of variables.
print(sess.run(x))
Execution of x = execution of sum(variable_1, constant(1)) = 2
print(sess.run(x))
Execution of x = execution of sum(variable_1, constant(1)) = 2
Example 2
x = tf.Variable(1)
Creation of a python variable x. Creation of a Tensorflow node variable_1. The python variable x holds a logical pointer to the node variable_1.
init = tf.global_variables_initializer()
initialization of variable_1.
with tf.Session() as sess:
init.run()
execution of the initialization.
# here x point to the variable
print(sess.run(x))
evaluation of x = 1.
x = x + 1
Definition of a new node, exactly as in the previous example.
print(sess.run(x))
evaluation of the sum operation, thus 2.
Example 3
x = tf.Variable(1)
usual creation.
as_op = x.assign(x+1)
definition of a sum node followed by the definition of an assignment node, held by the python variable as_op.
This operation forces the order of execution between these 2 nodes. First executes the sum node, then use the result to assign it to the node variable variable_1.
with tf.Session() as sess:
init.run()
usual init ops
# here x point to the variable
print(sess.run(x))
evaluation of variable_1, thus 1.
sess.run(as_op)
execution of sum and assigment, thus temp = variable_1 + 1; variable_1 = temp;
print(sess.run(x))
extraction of the value pointed to x, thus 2.
tensorflow add op returns a updated tensor
x = x + 1
with tf.Session() as sess:
init.run()
print(x)
# here x is a addition op; hence the result same for both run
> result Tensor("add:0", shape=(), dtype=int32)
for both run x is just the same add op
print(sess.run(x)) # result: 2
print(sess.run(x)) # result: 2
if you change the location of the addition op then values will be different; since you will accessing initial value and updated value from the add op
x = tf.Variable(1)
init = tf.global_variables_initializer()
with tf.Session() as sess:
init.run()
# here x point to the variable
print(sess.run(x)) # result: 1
x = x + 1
# here x point to the add op
print(sess.run(x)) # result: 2
Using assign op to get the updated value
x = tf.Variable(1)
as_op = x.assign(x+1)
with tf.Session() as sess:
init.run()
# here x point to the variable
print(sess.run(x)) # result: 1
sess.run(as_op) # variable updated
print(sess.run(x)) # result: 2
I use the op "assign" to change the value of variables instead of "=", but I found the gradient I got is quite different. Could anyone tell me the difference and why? thanks!
Like change w = w1 to op1 = tf.assign(w, w1) sess.run(op1)
= and tf.assign are different operations.
= is a python operation, in which you assign a python value to a python variable
tf.assign is a Tensorflow operation that assigns the value to the variable ref and returns the assign operation.
= is executed in python and doesn't affect the computation graph.
tf.assign is a node in the computational graph.
To understand, let's run this simple script
import tensorflow as tf
x = tf.Variable(1)
y = tf.Variable(2)
x = y
print(x.name, y.name)
a = tf.Variable(1)
b = tf.Variable(2)
# override a, otherwise a content is 1
a = a.assign(b)
print(a.name, b.name)
init = tf.global_variables_initializer()
with tf.Session() as sess:
sess.run(init)
print(sess.run([x, y, a, b]))
print(x.name, y.name) outputs Variable_1:0 Variable_1:0
because = is executed in python and you've overwritten the variable x.
print(a.name, b.name) outputs Assign:0 Variable_3:0 because you defined an assign op in the computational graph, now a is an assign op.
When you run the defined graph, you get:
[2, 2, 2, 2]
But these values are computed differently: one is a computation in the graph, the others no.
If you forgot to assign a to the assign op created with tf.assign (thus you change the line a = a.assign(b) to a.assign(b)), then when you evaluate the graph, you'll get:
[2, 2, 1, 2]
If I want to slice after a scan operation in TensorFlow.
But I just get strange results with TensorFlow:
k = 10
x = 2
out = tf.scan(lambda previous_output, current_input: previous_output * current_input,
tf.fill([k], x), initializer=tf.constant(1))
result = out[-1] # slice with tensorflow - don't work
with tf.Session() as sess:
sess.run(tf.initialize_all_variables())
print(sess.run(out)[-1]) # works but all values are computed and stored in an np array
print(sess.run(result)) # don't work???
I get as output:
1024
3
The second value is obviously wrong and random (sometimes 0 or other values).
So my question is why? The analog code in Theano e.g. works and Theano can do some optimization when querying just the last element of the output tensor.