Related
I don't think I understand this feed error from tensorflow
Debug: [[ 0. 0.]]
Debug: (1, 2)
Debug: float64
2018-05-09 09:56:34.615561: W tensorflow/core/kernels/queue_base.cc:295] _0_input_producer: Skipping cancelled enqueue attempt with queue not closed
Traceback (most recent call last):
File "/home/kiran/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1323, in _do_call
return fn(*args)
File "/home/kiran/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1302, in _run_fn
status, run_metadata)
File "/home/kiran/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/errors_impl.py", line 473, in __exit__
c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.InvalidArgumentError: You must feed a value for placeholder tensor 'seqModel/a_prev' with dtype double and shape [1,2]
[[Node: seqModel/a_prev = Placeholder[dtype=DT_DOUBLE, shape=[1,2], _device="/job:localhost/replica:0/task:0/device:GPU:0"]()]]
the way I feed the placeholder is:
self.a_prev = tf.placeholder(tf.float64, shape=[1,2], name='a_prev')
batch = tf.train.batch([self.x_acc, self.y_acc,
self.prev_pos],
batch_size=1, capacity=20000, num_threads=1)
x_acc, y_acc, prev_pos = sess.run(batch)
test = np.array([[ x_acc[0,0], y_acc[0,0] ]])
print("Debug: ",test)
print("Debug:",test.shape)
print("Debug:",test.dtype)
_,X_hat_val,loss_val, X_val = sess.run([train,X_hat,loss, self.X],
feed_dict={self.a_prev : np.array([[x_acc[0,0],y_acc[0,0] ]]),
self.pos1 : np.array([[ prev_pos[0,0] ]])
})
The error does not make sense because I am feeding values to the placeholder but it says that there are no values. What does that mean?
NB: I didn't run your code, as it depends on unavailable data.
However, it's probable your error is caused by reassigning the self.a_prev attribute, line 173. With this line, self.a_prev doesn't point to the tf.placeholder(..., name='a_prev') anymore, but to a different Tensor (from self.new_evidence) - so the actual placeholder doesn't get fed when running.
Toy example for this supposition
import tensorflow as tf
import numpy as np
x_acc = np.random.rand(2, 2)
y_acc = np.random.rand(2, 2)
a_prev = tf.placeholder(tf.float64, shape=[1,2], name='a_prev')
some_results = tf.add(a_prev, 1.)
a_prev = tf.constant([[-1, -1]])
# ... now "a_prev" the python variable isn't pointing to the placeholder anymore,
# so "a_prev" the placeholder exists in the graph with no python pointer to it.
with tf.Session() as sess:
res = sess.run(some_results, feed_dict={a_prev : np.array([[x_acc[0,0],y_acc[0,0] ]])})
# "a_prev" the constant is assigned the values, not "a_prev" the placeholder,
# hence an error.
InvalidArgumentError (see above for traceback): You must feed a value for placeholder tensor 'a_prev' with dtype double and shape [1,2]
[[Node: a_prev = Placeholderdtype=DT_DOUBLE, shape=[1,2],
_device="/job:localhost/replica:0/task:0/device:GPU:0"]] [[Node: Add/_1 = _Recvclient_terminated=false,
recv_device="/job:localhost/replica:0/task:0/device:CPU:0",
send_device="/job:localhost/replica:0/task:0/device:GPU:0",
send_device_incarnation=1, tensor_name="edge_8_Add",
tensor_type=DT_DOUBLE,
_device="/job:localhost/replica:0/task:0/device:CPU:0"]]
I've successfully trained and saved a faster RCNN model for tensorflow using their object detection API. I'm now trying to run some inferences on the code, taking bits of code from this tutorial.
However, after I successfully restore the metagraph and the checkpoint, the system can't find the input and output nodes, I get the following error:
KeyError: "The name 'image_tensor:0' refers to a Tensor which does not
exist. The operation, 'image_tensor', does not exist in the graph."
The checkpoint and metagraph were created by the train.py script, on my own data, following the instructions given here.
This is my code:
OUTPUT_DIR = "my_path/models/SSD_v1/train"
CKPT_DIR = OUTPUT_DIR
LATEST_CKPT_FILENAME = "checkpoint"
LAST_CKPT_FILE = os.path.join(CKPT_DIR, LATEST_CKPT_FILENAME)
MODEL_FILENAME_PATH = os.path.join(OUTPUT_DIR, "model.ckpt.meta")
def load_image_into_numpy_array(image):
(im_width, im_height) = image.size
return np.array(image.getdata()).reshape(
(im_height, im_width, 3)).astype(np.uint8)
def test_model(images_list, path_to_ckpt=None,
meta_graph=None):
if path_to_ckpt is None:
path_to_ckpt = tf.train.latest_checkpoint(CKPT_DIR, LATEST_CKPT_FILENAME)
if meta_graph is None:
meta_graph = MODEL_FILENAME_PATH
print("test_model launched")
tf.reset_default_graph()
detection_graph = tf.Graph()
with detection_graph.as_default():
with tf.Session(graph=detection_graph) as sess:
# Restore graph
saver = tf.train.import_meta_graph(meta_graph, clear_devices=True)
print('metagraph restored')
saver.restore(sess, path_to_ckpt)
print('graph restored')
image_tensor = detection_graph.get_tensor_by_name('image_tensor:0') # This is where the error happens
# Each box represents a part of the image where a particular object was detected.
detected_boxes = detection_graph.get_tensor_by_name('detection_boxes:0')
# Each score represent how level of confidence for each of the objects.
# Score is shown on the result image, together with the class label.
detected_scores = detection_graph.get_tensor_by_name('detection_scores:0')
detected_classes = detection_graph.get_tensor_by_name('detection_classes:0')
num_detections = graph.get_tensor_by_name('num_detections:0')
print("Output tensors: ")
print(detected_boxes)
print(detected_scores)
print(detected_classes)
print('')
for i, image in enumerate(images_list):
detected_boxes, detected_scores, detected_classes, num_detect = sess.run([detected_boxes, detected_scores, detected_classes, num_detections],
feed_dict={image_tensor: image})
print(i, num_detect, detected_boxes, detected_scores, detected_classes)
def main():
directory_path = "../data/samples/"
image_files = [f for f in os.listdir(directory_path) if os.path.isfile(os.path.join(directory_path, f))]
# Expand dimensions since the model expects images to have shape: [1, None, None, 3]
image_list = [ np.expand_dims(load_image_into_numpy_array(Image.open(os.path.join(directory_path, f))), axis=0) for f in image_files]
test_model(images_list=image_list)
if __name__=="__main__":
main()
Full error stacktrace:
Traceback (most recent call last): File "/home/guillaumedelaboulaye/PR8210PANO/faster-rcnn/pano_faster_rcnn/src/run_faster_rcnn_inference.py", line 99, in <module>
main() File "/home/guillaumedelaboulaye/PR8210PANO/faster-rcnn/pano_faster_rcnn/src/run_faster_rcnn_inference.py", line 95, in main
test_model(images_list=image_list) File "/home/guillaumedelaboulaye/PR8210PANO/faster-rcnn/pano_faster_rcnn/src/run_faster_rcnn_inference.py", line 48, in test_model
image_tensor = graph.get_tensor_by_name('image_tensor:0') File "/home/guillaumedelaboulaye/PR8210PANO/faster-rcnn/venv/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 2733, in get_tensor_by_name
return self.as_graph_element(name, allow_tensor=True, allow_operation=False) File "/home/guillaumedelaboulaye/PR8210PANO/faster-rcnn/venv/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 2584, in as_graph_element
return self._as_graph_element_locked(obj, allow_tensor, allow_operation) File "/home/guillaumedelaboulaye/PR8210PANO/faster-rcnn/venv/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 2626, in _as_graph_element_locked
"graph." % (repr(name), repr(op_name))) KeyError: "The name 'image_tensor:0' refers to a Tensor which does not exist. The operation, 'image_tensor', does not exist in the graph."
In the train graph, the input/output nodes are not given those names. What you will need to do is to "export" your trained model via the export_inference_graph.py tool. I believe it currently exports it to a frozen graph or a SavedModel, but in future releases, it will export to ordinary checkpoint as well.
If you want sample code for finding the node names of the graph, referring to the object_detection_tutorial.ipynb, after the "Load a (frozen) Tensorflow model into memory." block:
for node in od_graph_def.node:
print node.name
That should list all the node names that you can then enter in the subsequent blocks.
I have followed the tensorflow serving tutorial mnist_saved_model.py
and try to train and export a text-cnn-classifier model
The pipeline is
*embedding layer -> cnn -> maxpool -> cnn -> dropout -> output layer
Tensorflow data input :
data_in = tf.placeholder(tf.int32,[None, sequence_length] , name='data_in')
transformed to
serialized_tf_example = tf.placeholder(tf.string, name='tf_example')
feature_configs = {'x': tf.FixedLenFeature(shape=[sequence_length],
dtype=tf.int64),}
tf_example = tf.parse_example(serialized_tf_example, feature_configs)
# use tf.identity() to assign name
data_in = tf.identity(tf_example['x'], name='x')
This works for training phase
but at test time
it tells
AbortionError(code=StatusCode.INVALID_ARGUMENT, details="Expects arg[0] to be int64 but string is provided")
I am confused about the above line
feature_configs = {'x': tf.FixedLenFeature(shape=[sequence_length],
dtype=tf.int64),}
I changed the line to
feature_configs = {'x': tf.FixedLenFeature(shape=[sequence_length],
dtype=tf.string),}
but it gives the following error at training time:
Traceback (most recent call last):
File "/serving/bazel-bin/tensorflow_serving/example/twitter-sentiment-cnn_saved_model.runfiles/tf_serving/tensorflow_serving/example/twitter-sentiment-cnn_saved_model.py", line 222, in <module>
embedded_chars = tf.nn.embedding_lookup(W, data_in)
File "/serving/bazel-bin/tensorflow_serving/example/twitter-sentiment-cnn_saved_model.runfiles/org_tensorflow/tensorflow/python/ops/embedding_ops.py", line 122, in embedding_lookup
return maybe_normalize(_do_gather(params[0], ids, name=name))
File "/serving/bazel-bin/tensorflow_serving/example/twitter-sentiment-cnn_saved_model.runfiles/org_tensorflow/tensorflow/python/ops/embedding_ops.py", line 42, in _do_gather
return array_ops.gather(params, ids, name=name)
File "/serving/bazel-bin/tensorflow_serving/example/twitter-sentiment-cnn_saved_model.runfiles/org_tensorflow/tensorflow/python/ops/gen_array_ops.py", line 1179, in gather
validate_indices=validate_indices, name=name)
File "/serving/bazel-bin/tensorflow_serving/example/twitter-sentiment-cnn_saved_model.runfiles/org_tensorflow/tensorflow/python/framework/op_def_library.py", line 589, in apply_op
param_name=input_name)
File "/serving/bazel-bin/tensorflow_serving/example/twitter-sentiment-cnn_saved_model.runfiles/org_tensorflow/tensorflow/python/framework/op_def_library.py", line 60, in _SatisfiesTypeConstraint
", ".join(dtypes.as_dtype(x).name for x in allowed_list)))
TypeError: Value passed to parameter 'indices' has DataType string not in list of allowed values: int32, int64
Your code is wrong:
serialized_tf_example = tf.placeholder(tf.string, name='tf_example')
that means your input is a string, such as sentence's word. Therefore:
feature_configs = {'x': tf.FixedLenFeature(shape=[sequence_length],
dtype=tf.int64),}
tf_example = tf.parse_example(serialized_tf_example, feature_configs)
That means nothing, in my opinion, because you do not though vocabulary transfer string to int. You need load your train data's vocab to get word index!
I am trying to implement a recurrent state tensor using tf.scan. The code I have at the moment is this:
import tensorflow as tf
import math
import numpy as np
INPUTS = 10
HIDDEN_1 = 20
BATCH_SIZE = 3
def iterate_state(prev_state_tuple, input):
with tf.name_scope('h1'):
weights = tf.get_variable('W', shape=[INPUTS, HIDDEN_1], initializer=tf.truncated_normal_initializer(stddev=1.0 / math.sqrt(float(INPUTS))))
biases = tf.get_variable('bias', shape=[HIDDEN_1], initializer=tf.constant_initializer(0.0))
matmuladd = tf.matmul(inputs, weights) + biases
unpacked_state, unpacked_out = tf.split(0,2,prev_state_tuple)
prev_state = unpacked_state
state = 0.9* prev_state + 0.1*matmuladd
output = tf.nn.relu(state)
return tf.concat(0,[state, output])
def data_iter():
while True:
idxs = np.random.rand(BATCH_SIZE, INPUTS)
yield idxs
with tf.Graph().as_default():
inputs = tf.placeholder(tf.float32, shape=(BATCH_SIZE, INPUTS))
with tf.variable_scope('states'):
initial_state = tf.zeros([HIDDEN_1],
name='initial_state')
initial_out = tf.zeros([HIDDEN_1],
name='initial_out')
concat_tensor = tf.concat(0,[initial_state, initial_out])
states, output = tf.scan(iterate_state, inputs,
initializer=concat_tensor, name='states')
sess = tf.Session()
# Run the Op to initialize the variables.
sess.run(tf.initialize_all_variables())
iter_ = data_iter()
for i in xrange(0, 2):
print ("iteration: ",i)
input_data = iter_.next()
out,st = sess.run([output,states], feed_dict={ inputs: input_data})
However, I get this error when running this:
Traceback (most recent call last):
File "cycles_in_graphs_with_scan.py", line 37, in <module>
initializer=concat_tensor, name='states')
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 442, in __iter__
raise TypeError("'Tensor' object is not iterable.")
TypeError: 'Tensor' object is not iterable.
(tensorflow)charlesq#Leviathan ~/projects/stuff $ python cycles_in_graphs_with_scan.py
Traceback (most recent call last):
File "cycles_in_graphs_with_scan.py", line 37, in <module>
initializer=concat_tensor, name='states')
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 442, in __iter__
raise TypeError("'Tensor' object is not iterable.")
TypeError: 'Tensor' object is not iterable.
I've already tried with pack/unpack and concat/split but I get this same error.
Any ideas how to solve this problem?
You're getting an error because tf.scan() returns a single tf.Tensor, so the line:
states, output = tf.scan(...)
...cannot destructure (unpack) the tensor returned from tf.scan() into two values (states and outputs). Effectively, the code is trying to treat the result of tf.scan() as a list of length 2, and assign the first element to states and the second element to output, but—unlike a Python list or tuple—tf.Tensor does not support this.
Instead you need to extract the values from the result of tf.scan() manually. For example, using tf.split():
scan_result = tf.scan(...)
# Assumes values are packed together along `split_dim`.
states, output = tf.split(split_dim, 2, scan_result)
Alternatively, you could use tf.slice() or tf.unpack() to extract the relevant states and output values.
I'm using the CIFAR10 example. I trained the net as it is with the code provided. The training was done successfully. As I wanted to evaluate each example only once on my data set, I have modified inputs in cifar10_input.py to the following.
def inputs(eval_data, data_dir, batch_size):
filename = os.path.join(data_dir, TEST_FILE)
filename_queue = tf.train.string_input_producer([filename],num_epochs=1)
image, label = read_and_decode(filename_queue)
float_image = tf.image.per_image_whitening(image)
min_fraction_of_examples_in_queue = 0.4
min_queue_examples = int(NUM_EXAMPLES_PER_EPOCH_FOR_EVAL *
min_fraction_of_examples_in_queue)
images, label_batch = tf.train.batch(
[image, label],
batch_size=batch_size,
num_threads=1,
capacity=min_queue_examples + 3 * batch_size)
tf.image_summary('images', images)
return images, tf.reshape(label_batch, [batch_size])
I have isolated the problem to the following:
tf.train_string_input_producer([filename], num_epochs = 1)
If I don't set num_epochs = 1, everything works fine as it is. If I do, I get the following error.
0x2cf2700 Compute status: Not found: Tensor name "input_producer/limit_epochs/epochs" not found in checkpoint files /home/jkschin/tensorflow/my_code/data/svhn/train/model.ckpt-8000
Thank you for your help!
EDIT 3 #mrry:
It still fails. Here's the trace.
Traceback (most recent call last):
File "cnn_eval.py", line 148, in <module>
tf.app.run()
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/platform/default/_app.py", line 30, in run
sys.exit(main(sys.argv))
File "cnn_eval.py", line 144, in main
evaluate()
File "cnn_eval.py", line 119, in evaluate
saver = tf.train.Saver([v for v in variables_to_restore if v.name != "input_producer/limit_epochs/epochs"])
AttributeError: 'unicode' object has no attribute 'name'
EDIT 4 #mrry:
softmax_linear/biases/ExponentialMovingAverage
conv2/biases/ExponentialMovingAverage
local4/biases/ExponentialMovingAverage
local3/biases/ExponentialMovingAverage
softmax_linear/weights/ExponentialMovingAverage
conv1/biases/ExponentialMovingAverage
local4/weights/ExponentialMovingAverage
conv2/weights/ExponentialMovingAverage
input_producer/limit_epochs/epochs
local3/weights/ExponentialMovingAverage
conv1/weights/ExponentialMovingAverage
Traceback (most recent call last):
File "cnn_eval.py", line 148, in <module>
tf.app.run()
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/platform/default/_app.py", line 30, in run
sys.exit(main(sys.argv))
File "cnn_eval.py", line 144, in main
evaluate()
File "cnn_eval.py", line 119, in evaluate
saver = tf.train.Saver([v for v in variables_to_restore if v != "input_producer/limit_epochs/epochs"])
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/saver.py", line 784, in __init__
restore_sequentially=restore_sequentially)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/saver.py", line 437, in build
vars_to_save = self._ValidateAndSliceInputs(names_to_variables)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/saver.py", line 340, in _ValidateAndSliceInputs
names_to_variables = self._VarListToDict(names_to_variables)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/saver.py", line 314, in _VarListToDict
raise TypeError("Variable to save is not a Variable: %s" % var)
TypeError: Variable to save is not a Variable: Tensor("Const:0", shape=(), dtype=string)
EDIT 5 #mrry:
saver = tf.train.Saver([tf.Variable(0.0,validate_shape=False,name=v) for v in variables_to_restore if v != "input_producer/limit_epochs/epochs"])
0x21d0cb0 Compute status: Invalid argument: Assign requires shapes of both tensors to match. lhs shape= [] rhs shape= [10]
[[Node: save/Assign_8 = Assign[T=DT_FLOAT, use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/gpu:0"](softmax_linear/biases/ExponentialMovingAverage, save/restore_slice_8/_20)]]
TL;DR: In cifar10_eval.py, change the saver constructor so that it is:
saver = tf.train.Saver([v for v in variables_to_restore
if v != "input_producer/limit_epochs/epochs"])
This problem arises because tf.train.string_input_producer() internally creates a variable (called "input_producer/limit_epochs/epochs") when its num_epochs argument is not None. When, in cifar10_eval.py a tf.train.Saver is created, it uses tf.all_variables(), which includes the implicitly-created variable from the tf.nn.string_input_producer(). This list of variables determines the set of names that TensorFlow looks up in the checkpoint file.
Currently there isn't a great way to refer to implicitly created variables, other than by their name. Therefore, the best fix is to exclude the variable from the Saver constructor by name.
Another way of eliminating the implicit variable "input_producer/limit_epochs/epochs" is to only load the trainable variables:
saver = tf.train.Saver(tf.trainable_variables())