As per title. I'd like to make use of such operation to rename the nodes and better organize a graph. Or is there other recommended practice for renaming an existing node in the graph? Thanks!
There is tf.no_op which allows you to add an operation which does nothing.
As far as I know, you cannot rename a Tensor once created.
However, you can use additional "no-op" operations (like you said):
for a tf.Tensor: tf.identity(input_tensor, name='your_new_name')
for an operation: tf.group(input_operation, name='your_new_name')
After that, you can call the input_tensor with:
graph = tf.get_default_graph()
graph.get_tensor_by_name('your_new_name:0')
Or the input_operation with:
graph = tf.get_default_graph()
graph.get_operation_by_name('your_new_name')
Related
I create a tensorflow graph and define some tensors and run some stuff. When I'm done, I'd like to delete the graph that I made, and free up all of the resources. How can I do that thing?
temporary_graph = tf.Graph()
with temporary_graph.as_default(), tf.Session() as sess:
foo = tf.placeholder(tf.float32, (2,2))
bar = foo#foo
res = sess.run(bar, feed_dict={foo: np.ones((2,2))})
print(res)
delete_graph_and_free_up_resources(temporary_graph)
This answer claims that the context manager cleans up the graph, but this isn't the case, and the docs don't claim such a thing:
>>> temporary_graph.get_operations()
[<tf.Operation 'Placeholder' type=Placeholder>, <tf.Operation 'matmul' type=MatMul>]
What is the best way to dispose of a graph?
It is not so simple, in order to free the resources that a graph is using you need to lose every reference to that graph, so Python can request to have it deleted from memory. That means deleting direct references to the graph, but also objects referencing the graph (and transitively). That includes operations, tensors and sessions, among other things. In your example, you would need to do:
del temporary_graph, sess, foo, bar, res
And that should make it possible to have the memory freed (not sure if you might need to call the garbage collector in some cases).
As you may not, you can not do this in a function, as it depends on the live references in your program. However, if you keep all references related to the graph within a function or object you should be able to do it fine.
I'm using tensorflow keras, but my approach is to simply clear the session:
tensorflow.keras.backend.clear_session()
I'm trying to set up a DNN for classification and at one point I want to take the tensor product of a vector with itself. I'm using the Keras functional API at the moment but it isn't immediately clear that there is a layer that does this already.
I've been attempting to use a Lambda layer and numpy in order to try this, but it's not working.
Doing a bit of googling reveals
tf.linalg.LinearOperatorKronecker, which does not seem to work either.
Here's what I've tried:
I have a layer called part_layer whose output is a single vector (rank one tensor).
keras.layers.Lambda(lambda x_array: np.outer(x_array, x_array),) ( part_layer) )
Ideally I would want this to to take a vector of the form [1,2] and give me [[1,2],[2,4]].
But the error I'm getting suggests that the np.outer function is not recognizing its arguments:
AttributeError: 'numpy.ndarray' object has no attribute '_keras_history
Any ideas on what to try next, or if there is a simple function to use?
You can use two operations:
If you want to consider the batch size you can use the Dot function
Otherwise, you can use the the dot function
In both case the code should look like this:
dot_lambda = lambda x_array: tf.keras.layers.dot(x_array, x_array)
# dot_lambda = lambda x_array: tf.keras.layers.Dot(x_array, x_array)
keras.layers.Lambda(dot_lamda)( part_layer)
Hope this help.
Use tf.tensordot(x_array, x_array, axes=0) to achieve what you want. For example, the expression print(tf.tensordot([1,2], [1,2], axes=0)) gives the desired result: [[1,2],[2,4]].
Keras/Tensorflow needs to keep an history of operations applied to tensors to perform the optimization. Numpy has no notion of history, so using it in the middle of a layer is not allowed. tf.tensordot performs the same operation, but keeps the history.
I usually structure my learnable parameters in the following way in tensorflow:
learnable_weights = {
'w1': tf.get_variable(...),
...
'wn': tf.get_variable(...),
}
learnable_biases = {
'bc1': tf.get_variable(...),
...
'bd3': tf.get_variable(...)
}
The problem I started to recently encounter is the congested tensorboard graph, where I have a lot of weights in the auxilary nodes (this is a part of a big graph, and the number of these nodes is way bigger):
I tried to group them with tf.name_scope. Something like this:
with tf.name_scope('learnable_params'):
learnable_weights = {...}
learnable_biases = {...}
But this has no effect on the graph in tensorboard.
Any reason why or better any suggestions how to group the learnable parameters so that they will not clutter the tensorboard?
You could try using variable_scope instead of name_scope. AFAIK variables created via get_variable ignore name_scope, and I wouldn't be surprised if this applies to the graph organization in Tensorboard as well. I only use variable_scope to wrap anything that creates variables and I've never had these issues with "unorganized" variables.
I am trying to read a big Tensorflow project. For a project that the nodes of the computation graph are scattered around the project, I wonder if there is a way to store a Tensor node of the computation graph and add that node to the fetch list in sess.run?
For example, if I want to add probs at line 615 of
https://github.com/allenai/document-qa/blob/master/docqa/nn/span_prediction.py to a global namespace, is there a method like tf.add_node(probs, "probs"), and later I could get tf.get_node("probs"), just for the sake of conveniently passing node around the project.
A more general question would be, what will be a better idea to structure the tensorflow code and improve the efficiency of experimenting with different models.
Of course you can. To retrieve it later, you'll have to give it a name so that you can retrieve it by name. Take probs in your code as an example. It's created with tf.nn.softmax() function, the API for which is shown below.
tf.nn.softmax(
logits,
axis=None,
name=None,
dim=None
)
See the parameter name? You can add this parameter to line 615 like this:
probs = tf.nn.softmax(all_logits, name='my_tensor')
Later when you need it, you can call tf.Graph.get_tensor_by_name(name) to retrieve this tensor.
graph = tf.get_default_graph()
retrieved_probs = graph.get_tensor_by_name('my_tensor:0')
'my_tensor' is the name of the softmax operation, and ':0' should be added to the end of it meaning that you're retrieving the tensor instead of the operation. When calling Graph.get_operation_by_name(), no ':0' should be added.
You'll have to make sure that the tensor exists(it might be created in the code executed before this line, or it might be restored from a meta graph file). If it's created in a variable scope, you'll also have to add the scope name and a '/' in the front of the name param. For example, 'my_scope/my_tensor:0'.
I have two tensorflow graphs. One for training and the other for evaluation. They share a lot of variable names. When I evaluate a model I want to copy all variable values from the train graph to the test graph. Obviously, I can do it via tf.train.Saver, but this solution seems not very appropriate to me, especially the fact that we have to use the disk for this.
When you speak about multiple graphs, I assume you mean something like:
g1 = tf.Graph()
with g1.as_default():
# add your stuff
g2 = tf.Graph()
with g2.as_default():
# add other stuff
If this is correct, then are you sure you really need two graphs? Can't you have one graph consisting of two connected components?
Using multiple graphs is discouraged (p 47) because:
Multiple graphs require multiple sessions, each will try to use all available resources by default
Can't pass data between them without passing them through python/numpy, which doesn't work in distributed
It’s better to have disconnected subgraphs within one graph
This also gives you a solution how to pass variables in a non-distributed setting.