I would like to show my example below:
x = tf.placeholder(dtype=...)
a = numpy.asarray([784, 10])
z = slim.fully_connected(x, 10, weights_initializer=?)
I have tried weights_initializer = lambda x1:a, it reports the error: TypeError: () got an unexpected keyword argument 'dtype'
I also found another post here:https://github.com/tensorflow/tensorflow/issues/4016
However, I still don't know the answer. Thank you very much.
Sorry, I don't really understand what you're trying to do.
If your fully connected layer has 10 hidden neurons then your initializer must have the shape (input_shape, 10), what you're giving is a (2,) shape. Secondly, to initialize weights with a constant matrix you should use tf.constant_initializer(..) function.
Are you trying to do the following: (you can change the init function used with numpy)
import tensorflow as tf
import numpy as np
slim = tf.contrib.slim
input_size = ?
x = tf.placeholder(dtype=tf.float32, shape=[input_size])
a = np.random.normal((input_size, 10))
z = slim.fully_connected(x, 10,
weights_initializer=tf.constant_initializer(a))
init = tf.global_variables_initializer()
sess = tf.Session()
sess.run(init)
Related
My goal is to build a script to change an operation into another one using TF's graph editor. So far I tried making a script that just changes the input kernel weights of a Conv2D, but to no avail, as the interface is pretty confusing.
with tf.Session() as sess:
model_filename = sys.argv[1]
with gfile.FastGFile(model_filename, 'r') as f:
graph_def = graph_pb2.GraphDef()
text_format.Merge(f.read(), graph_def)
importer.import_graph_def(graph_def)
#my_sgv = ge.sgv("Conv2D", graph=tf.get_default_graph())
#print my_sgv
convs = find_conv2d_ops(tf.get_default_graph())
print convs
my_sgv = ge.sgv(convs)
print my_sgv
conv_tensor = tf.get_default_graph().get_tensor_by_name(convs[0].name + ':0')
conv_weights_input = tf.get_default_graph().get_tensor_by_name(convs[0].inputs[1].name)
weights_new = tf.Variable(tf.truncated_normal([1, 1, 1, 8], stddev=0.03),
name='Wnew')
ge.graph_replace(conv_tensor, {conv_weights_input: weights_new})
The error is "input needs to be a Tensor: ". Can someone please provide some insights?
Since you are dealing with a tf.Variable you don't need to use graph editor. tf.assign will be sufficient.
You can use it like the following:
assign_op = tf.assign(conv_weights_input, weights_new)
with tf.Session() as sess:
sess.run(assign_op)
If you are looking to sub out operations and not weights. Consider the following example (modified from this example):
import tensorflow as tf
import tensorflow.contrib.graph_editor as ge
def build():
a_pl = tf.placeholder(dtype=tf.float32, name="a")
b_pl = tf.placeholder(dtype=tf.float32, name="b")
c = tf.add(a_pl, b_pl, name="c")
build() #or load graph from disc
a = tf.constant(1.0, shape=[2, 3], name="a_const")
b = tf.constant(2.0, shape=[2, 3], name="b_const")
a_pl = tf.get_default_graph().get_tensor_by_name("a:0")
b_pl = tf.get_default_graph().get_tensor_by_name("b:0")
c = tf.get_default_graph().get_tensor_by_name("c:0")
c_ = ge.graph_replace(c, {a_pl: a, b_pl: b})
with tf.Session() as sess:
#no need for placeholders
print(sess.run(c_))
#will give error since a_pl and b_pl have no value
print(sess.run(c))
The issue with your code is that you're dealing with wights, and not tensors. The crux of the above example is that the first argument is the target tensor (output tensor) that have the to be replaced tensors as dependencies. The second argument are the actual tensors you want to replace.
It's also worth noting that conv_weights_input is actually a tensor, where weights_new is a tf.Variable. I believe what you want is to replace weights_new with a new conv operation with random weight initialisation.
I am able to perform addition using constant but not able to perform through tf.Variable
Below code is working fine when I am using constant for addition.
import tensorflow as tf
a = tf.constant(5)
b = tf.constant(6)
sess = tf.Session()
result = sess.run(a + b)
print(result)
But when I tried with tf.Variable it is not working, Here is mine code
import tensorflow as tf
a = tf.Variable(5)
b = tf.Variable(6)
sess = tf.Session()
result = sess.run(a + b)
print(result)
Can somebody help me ? Thanks !!!
You need to initialize the variables firstly:
import tensorflow as tf
a = tf.Variable(5)
b = tf.Variable(6)
sess = tf.Session()
Initialize variables:
sess.run(tf.global_variables_initializer())
result = sess.run(a + b)
print(result)
11
You can read more about variable initialization here, which says Unlike tf.Tensor objects, a tf.Variable exists outside the context of a single session.run call. So Before you can use a variable, it must be initialized. The initialization is Session specific, which means whenever you start a new session, and want to use these variables, you'll have to initialize them firstly.
Here is a sample:
import numpy as np
import tensorflow as tf
from tensorflow.python.ops import rnn, rnn_cell
if __name__ == '__main__':
embs = tf.Variable(np.random.random((40,5)),dtype=tf.float32)
X = np.array(np.array(range(1,25)).reshape(4, 6))
x0 = tf.placeholder(tf.int32, [None, None])
x1 = tf.nn.embedding_lookup(embs, x0)
lstm = tf.nn.rnn_cell.BasicLSTMCell(5,state_is_tuple=True)
outputs, states = tf.nn.dynamic_rnn(lstm, x1, dtype=tf.float32,time_major = True)
cost = tf.reduce_mean(outputs[:,-1,:])
optimizer = tf.train.AdagradOptimizer(learning_rate=0.12).minimize(cost)
with tf.Session() as sess:
sess.run(tf.initialize_all_variables())
result3, opt = sess.run([outputs, optimizer],{x0:X})
I use just one slice of outputs which is outputs[:,-1,:] to get a cost function. When I run the code, I got the result
F ./tensorflow/core/framework/tensor.h:581] Check failed: new_num_elements == NumElements() (0 vs. 20)
How to fix this? It's just a sample. I met this problem when I implement a hierarchical LSTM in which the representations of sentences computed by a LSTM is feed into another LSTM.
I confirmed that this is a bug in TensorFlow 0.10. Upgrading to TensorFlow 0.11 will fix the problem.
I am running the following code:
import tensorflow as tf
sess = tf.InteractiveSession()
y = tf.Variable(initial_value=[1,2])
sess.run(y, feed_dict={y: [100,2]})
Gives:
[100,2]
However, after that:
sess.run(y)
Gives the origianl value of y: [1,2].
Why doesn't the:
sess.run(y, feed_dict={y: [100,2]})
updates the value of y, and saves it?
Because feed_dict overrides the values of the keys of the dictionary.
With the statement:
sess.run(y, feed_dict={y: [100,2]})
you're telling tensorflow to replace the values of y with [100, 2] for the current computation. This is not an assignment.
Therefore, the next call
sess.run(y)
fetches the original variables and uses it.
If you want to assign a value to a variable, you have to define this operation in the computational graph, using tf.assing
If you want to use a feed dictionary, initialize a placeholder instead of a variable and define the output.
As an example (in the same style as your question code),
import tensorflow as tf
import numpy as np
sess = tf.InteractiveSession()
inputs = tf.placeholder(tf.int32, shape = (2,2))
output = tf.matmul(inputs, tf.transpose(inputs))
test_input = np.array([[10,2], [4,4]])
print test_input.shape
# (2,2)
sess.run(output, feed_dict = {inputs : test_input})
# array([[104, 48], [48, 32]], dtype=int32)
If you just want to change the value of a variable look to nessuno's answer.
I am trying to initialize a tensorflow Variable with pre-trained word2vec embeddings.
I have the following code:
import tensorflow as tf
from gensim import models
model = models.Word2Vec.load_word2vec_format('GoogleNews-vectors-negative300.bin', binary=True)
X = model.syn0
embeddings = tf.Variable(tf.random_uniform(X.shape, minval=-0.1, maxval=0.1), trainable=False)
sess.run(tf.initialize_all_variables())
sess.run(embeddings.assign(X))
And I am receiving the following error:
ValueError: Cannot create an Operation with a NodeDef larger than 2GB.
The array (X) I am trying to assign is of shape (3000000, 300) and its size is 3.6GB.
I am getting the same error if I try tf.convert_to_tensor(X) as well.
I know that it fails due to the fact that the array is larger than 2GB. However, I do not know how to assign an array larger than 2GB to a tensorflow Variable
It seems like the only option is to use a placeholder. The cleanest way I can find is to initialize to a placeholder directly:
X_init = tf.placeholder(tf.float32, shape=(3000000, 300))
X = tf.Variable(X_init)
# The rest of the setup...
sess.run(tf.initialize_all_variables(), feed_dict={X_init: model.syn0})
The easiest solution is to feed_dict'ing it into a placeholder node that you use to tf.assign to the variable.
X = tf.Variable([0.0])
place = tf.placeholder(tf.float32, shape=(3000000, 300))
set_x = X.assign(place)
# set up your session here....
sess.run(set_x, feed_dict={place: model.syn0})
As Joshua Little noted in a separate answer, you can also use it in the initializer:
X = tf.Variable(place) # place as defined above
...
init = tf.initialize_all_variables()
... create sess ...
sess.run(init, feed_dict={place: model.syn0})
try this:
import tensorflow as tf
from gensim import models
model = models.KeyedVectors.load_word2vec_format('./GoogleNews-vectors-negative300.bin', binary=True)
X = model.syn0
embeddings = tf.Variable(tf.random_uniform(X.shape, minval=-0.1, maxval=0.1), trainable=False)
sess = tf.Session()
sess.run(tf.global_variables_initializer())
embeddings.load(model.syn0, sess)