I want to preprocess some data as the input for Tensorflow, but the file is huge and I think maybe Tensorflow can accelerate this process on GPU. And I find some operation for String base in the library, However, when I do a simple test, it seems Tensorflow not support that operation on GPU, if anyone has some idea to process String data on GPU in Tensorflow or some other advance? For now, I just think maybe I have to do it on the CUDA level.
Here is the test code:
import tensorflow as tf
sess = tf.InteractiveSession()
with tf.device('/gpu:0'):
text = tf.constant("aa a", name="LEFT")
result = tf.string_split([text], delimiter=" ")
print result.eval()
and error information:
I tensorflow/core/common_runtime/gpu/gpu_device.cc:975] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX 1070, pci bus id: 0000:01:00.0)
Traceback (most recent call last):
File "/media/freshield/BUFFER/LEARN_TENSORFLOW/11_tensorboard_graph/testing.py", line 51, in <module>
print result.eval()
File "/home/freshield/anaconda2/lib/python2.7/site-packages/tensorflow/python/framework/sparse_tensor.py", line 225, in eval
session)
File "/home/freshield/anaconda2/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 3797, in _eval_using_default_session
return session.run(tensors, feed_dict)
File "/home/freshield/anaconda2/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 767, in run
run_metadata_ptr)
File "/home/freshield/anaconda2/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 965, in _run
feed_dict_string, options, run_metadata)
File "/home/freshield/anaconda2/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1015, in _do_run
target_list, options, run_metadata)
File "/home/freshield/anaconda2/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1035, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Cannot assign a device to node 'packed': Could not satisfy explicit device specification '/device:GPU:0' because no supported kernel for GPU devices is available.
[[Node: packed = Pack[N=1, T=DT_STRING, axis=0, _device="/device:GPU:0"](LEFT)]]
Caused by op u'packed', defined at:
File "/media/freshield/BUFFER/LEARN_TENSORFLOW/11_tensorboard_graph/testing.py", line 50, in <module>
result = tf.string_split([text], delimiter=" ")
File "/home/freshield/anaconda2/lib/python2.7/site-packages/tensorflow/python/ops/string_ops.py", line 104, in string_split
source = ops.convert_to_tensor(source, dtype=dtypes.string)
File "/home/freshield/anaconda2/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 651, in convert_to_tensor
as_ref=False)
File "/home/freshield/anaconda2/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 716, in internal_convert_to_tensor
ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)
File "/home/freshield/anaconda2/lib/python2.7/site-packages/tensorflow/python/ops/array_ops.py", line 923, in _autopacking_conversion_function
return _autopacking_helper(v, inferred_dtype, name or "packed")
File "/home/freshield/anaconda2/lib/python2.7/site-packages/tensorflow/python/ops/array_ops.py", line 886, in _autopacking_helper
return gen_array_ops._pack(elems_as_tensors, name=scope)
File "/home/freshield/anaconda2/lib/python2.7/site-packages/tensorflow/python/ops/gen_array_ops.py", line 2041, in _pack
result = _op_def_lib.apply_op("Pack", values=values, axis=axis, name=name)
File "/home/freshield/anaconda2/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 763, in apply_op
op_def=op_def)
File "/home/freshield/anaconda2/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 2395, in create_op
original_op=self._default_original_op, op_def=op_def)
File "/home/freshield/anaconda2/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1264, in __init__
self._traceback = _extract_stack()
InvalidArgumentError (see above for traceback): Cannot assign a device to node 'packed': Could not satisfy explicit device specification '/device:GPU:0' because no supported kernel for GPU devices is available.
[[Node: packed = Pack[N=1, T=DT_STRING, axis=0, _device="/device:GPU:0"](LEFT)]]
Related
I'm training a small, simple neural net for a basic problem of regulating a motor's speed. I want to be able to save the model and exit the program, then load it later and resume training.
Here's the relevant code:
self.model = Sequential()
self.model.add(InputLayer(2))
self.model.add(Dense(6, activation='relu'))
self.model.add(Dense(9, activation='linear'))
self.model.compile(loss='mse', optimizer='adam', metrics=['mae'])
# ... Loop for training and Evaluation (Deep Q Learner) ...
learn(self.model)
self.model.save('motor_model', save_format='tf')
Now after it's trained I want to be able to load the model and continue training
self.model = models.load_model('motor_model', compile=False)
# ... Loop for training and Evaluation (Deep Q Learner) ...
learn(self.model)
The first time I run the model it works fine. However, after saving and loading the model it does not. Upon loading the model I am able to call the predict function:
prediction = self.model.predict(currentInput)
However, It fails when I call the predict function:
self.model.fit(self.input, target_vec.reshape(-1, 9), epochs=1, verbose=0)
The error I get is:
2019-12-07 07:22:00.762174: W tensorflow/c/c_api.cc:326] Operation
'{name:'sequential/dense/StatefulPartitionedCall'
id:33 op device:{} def:{{{node
sequential/dense/StatefulPartitionedCall}} =
StatefulPartitionedCall[Tin=[DT_FLOAT, DT
_RESOURCE, DT_RESOURCE], Tout=[DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT], _gradient_op_type="PartitionedCall-298", conf
ig="",
config_proto="\n\007\n\003CPU\020\001\n\007\n\003GPU\020\0002\002J\0008\001",
executor_type="", f=__forward_re
stored_function_body_509[]](input_1, dense/kernel, dense/bias)}}' was
changed by setting attribute after it was run b
y a session. This mutation will have no effect, and will trigger an
error in the future. Either don't modify nodes af
ter running them or create a new session. 2019-12-07 07:22:03.320478:
W tensorflow/python/util/util.cc:299] Sets are not currently
considered sequences, but th
is may change in the future, so consider avoiding using them.
Traceback (most recent call last): File
"/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/client/session.py",
line 1363, in _do_call
return fn(*args) File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/client/session.py",
line 1346, in _run_fn
self._extend_graph() File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/client/session.py",
line 1386, in _extend_graph
tf_session.ExtendSession(self._session) tensorflow.python.framework.errors_impl.InvalidArgumentError: Node
'training/Adam/gradients/gradients/sequential/dens
e_1/StatefulPartitionedCall_grad/PartitionedCall': Connecting to
invalid output 1 of source node sequential/dense_1/S
tatefulPartitionedCall which has 1 outputs.
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "ct2.py", line 47, in
leftController.to_position(target, overrideAction) File "/opt/mowzr/motor_controller.py", line 94, in to_position
self.model.fit(self.prevInput, target_vec.reshape(-1, 9), epochs=1, verbose=0) File
"/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/keras/engine/training.py",
line 766, in fit
use_multiprocessing=use_multiprocessing) File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/keras/engine/training_arrays.py",
line 680, in
fit
steps_name='steps_per_epoch') File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/keras/engine/training_arrays.py",
line 275, in
model_iteration
model.reset_metrics() File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/keras/engine/training.py",
line 953, in reset_m
etrics
m.reset_states() File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/keras/metrics.py",
line 209, in reset_states
K.batch_set_value([(v, 0) for v in self.variables]) File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/keras/backend.py",
line 3343, in batch_set_valu
e
get_session().run(assign_ops, feed_dict=feed_dict) File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/keras/backend.py",
line 490, in get_session
_initialize_variables(session) File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/keras/backend.py",
line 905, in _initialize_var
iables
[variables_module.is_variable_initialized(v) for v in candidate_vars]) File
"/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/client/session.py",
line 956, in run
run_metadata_ptr) File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/client/session.py",
line 1179, in _run
feed_dict_tensor, options, run_metadata) File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/client/session.py",
line 1357, in _do_run
run_metadata) File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/client/session.py",
line 1382, in _do_call
raise type(e)(node_def, op, message) tensorflow.python.framework.errors_impl.InvalidArgumentError: Node
'training/Adam/gradients/gradients/sequential/dens
e_1/StatefulPartitionedCall_grad/PartitionedCall': Connecting to
invalid output 1 of source node sequential/dense_1/S
tatefulPartitionedCall which has 1 outputs.
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File
"/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/client/session.py",
line 1363, in _do_call
return fn(*args) File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/client/session.py",
line 1346, in _run_fn
self._extend_graph() File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/client/session.py",
line 1386, in _extend_graph
tf_session.ExtendSession(self._session) tensorflow.python.framework.errors_impl.InvalidArgumentError: Node
'training/Adam/gradients/gradients/sequential/dens
e_1/StatefulPartitionedCall_grad/PartitionedCall': Connecting to
invalid output 1 of source node sequential/dense_1/S
tatefulPartitionedCall which has 1 outputs.
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "ct2.py", line 53, in
leftController.saveModel() File "/opt/mowzr/motor_controller.py", line 116, in saveModel
self.model.save('motor_model', save_format='tf') File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/keras/engine/network.py",
line 986, in save
signatures, options) File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/keras/saving/save.py",
line 115, in save_model
signatures, options) File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/keras/saving/saved_model/save.py",
line 74, in
save
save_lib.save(model, filepath, signatures, options) File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/saved_model/save.py",
line 924, in save
object_saver.save(utils_impl.get_variables_path(export_dir)) File
"/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/training/tracking/util.py",
line 1161, in save
session = get_session() File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/training/tracking/util.py",
line 71, in get_ses
sion
session = keras_backend.get_session() File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/keras/backend.py",
line 490, in get_session
_initialize_variables(session) File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/keras/backend.py",
line 905, in _initialize_var
iables
[variables_module.is_variable_initialized(v) for v in candidate_vars]) File
"/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/client/session.py",
line 956, in run
run_metadata_ptr) File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/client/session.py",
line 1179, in _run
feed_dict_tensor, options, run_metadata) File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/client/session.py",
line 1357, in _do_run
run_metadata) File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/client/session.py",
line 1382, in _do_call
raise type(e)(node_def, op, message) tensorflow.python.framework.errors_impl.InvalidArgumentError: Node
'training/Adam/gradients/gradients/sequential/dens
e_1/StatefulPartitionedCall_grad/PartitionedCall': Connecting to
invalid output 1 of source node sequential/dense_1/S
tatefulPartitionedCall which has 1 outputs.
I got the same error.
I don't know what exactly produces this error but there is a way to solve it (not a pretty one though). Create the model with the same architecture and just set the weights to the loaded model weights:
self.model = self.create_model()
self.model.set_weights(load_model("sample.model").get_weights())
I training maskrcnn ,use tf-1.2 can train, but I use tf-1.5 it not training
The error is as follows:
Caused by op u'pyramid_1/AssignGTBoxes/Where_6', defined at:
File "/home/zhouzd2/letrain/applications/letrain.py", line 349, in <module>
tf.app.run()
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/platform/app.py", line 124, in run
_sys.exit(main(argv))
File "/home/zhouzd2/letrain/applications/letrain.py", line 346, in main
LeTrain().model_train(user_mode)
File "/home/zhouzd2/letrain/platform/base_train.py", line 1228, in model_train
cluster=self.cluster_spec)
File "/home/zhouzd2/letrain/platform/deployment/model_deploy.py", line 226, in create_clones
outputs, feed_ops,verify_model_loss = model_fn(*args, **kwargs)
File "/home/zhouzd2/letrain/platform/base_train.py", line 1195, in clone_fn
model_loss, end_points, feed_ops = network_fn(data_direct, data_batch, int_network_fn)
File "/home/zhouzd2/letrain/applications/letrain.py", line 214, in get_loss
FLAGS.batch_size)
File "/home/zhouzd2/letrain/applications/fmrcnn/get_fmrcnn_loss.py", line 23, in model_fn
loss_weights=[0.2, 0.2, 1.0, 0.2, 1.0])
File "/home/zhouzd2/letrain/applications/fmrcnn/libs/nets/pyramid_network.py", line 580, in build
is_training=is_training, gt_boxes=gt_boxes)
File "/home/zhouzd2/letrain/applications/fmrcnn/libs/nets/pyramid_network.py", line 263, in build_heads
assign_boxes(rois, [rois, batch_inds], [2, 3, 4, 5])
File "/home/zhouzd2/letrain/applications/fmrcnn/libs/layers/wrapper.py", line 173, in assign_boxes
inds = tf.where(tf.equal(assigned_layers, l))
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/array_ops.py", line 2538, in where
return gen_array_ops.where(condition=condition, name=name)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/gen_array_ops.py", line 6087, in where
"Where", input=condition, name=name)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 3160, in create_op
op_def=op_def)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 1625, in __init__
self._traceback = self._graph._extract_stack() # pylint: disable=protected-access
InternalError (see above for traceback): WhereOp: Could not launch cub::DeviceReduce::Sum to count number of true / nonzero indices. temp_storage_bytes: 1, status: no kernel image is available for execution on the device
[[Node: pyramid_1/AssignGTBoxes/Where_6 = Where[T=DT_BOOL, _device="/job:worker/replica:0/task:0/device:GPU:0"](pyramid_1/AssignGTBoxes/Equal_6_S9493)]]
[[Node: pyramid_1/AssignGTBoxes/Reshape_8_G1028 = _Recv[client_terminated=false, recv_device="/job:worker/replica:0/task:0/device:CPU:0", send_device="/job:worker/replica:0/task:0/device:GPU:0", send_device_incarnation=5407481677180697062, tensor_name="edge_1349_pyramid_1/AssignGTBoxes/Reshape_8", tensor_type=DT_INT64, _device="/job:worker/replica:0/task:0/device:CPU:0"]()]]
No problem when loading calculation graphs, error is reported in sess.run()。
Does anyone know how to solve this problem? Or does anyone know what function can replace tf.where?
Thank you!
If you are using Visual Studio:
Right click on the project > Properies > Cuda C/C++ > Device
and add the following to Code Generation field
compute_30,sm_30;compute_35,sm_35;compute_37,sm_37;compute_50,sm_50;compute_52,sm_52;compute_60,sm_60;compute_61,sm_61;compute_70,sm_70;compute_75,sm_75;
In a task to implement the minimum risk training for a neural machine translation system I need to sample sentences and gather the respective logits for the sampled word IDs. The step of gathering looks like this:
for i in range(1,self._num_of_samples):
logits, _, _, sampled_ids = self.decoder._decoding_loop(train_mode=False,sample=True)
ind=[[[tf.constant(i),tf.constant(j),sampled_ids[i][j]] for j in range(self.batch_size)] for i in range(self.decoder.max_output_len)]
gathered_logits = tf.gather_nd(logits,ind)
sentence_sum_logit = tf.reduce_sum(gathered_logits,0)
self.sample_sen_ids = self.sample_sen_ids.write(steps[i],sampled_ids)
self.sample_logits = self.sample_logits.write(steps[i], sentence_sum_logit)
self.sample_sen_ids = tf.transpose(self.sample_sen_ids.stack())
self.sample_logits = tf.transpose(self.sample_logits.stack())
But I dont get it why after some batches I get this:
Traceback (most recent call last):
File "/home/stoyan/neurmon/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1022, in _do_call
return fn(*args)
File "/home/stoyan/neurmon/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1004, in _run_fn
status, run_metadata)
File "/usr/lib/python3.5/contextlib.py", line 66, in __exit__
next(self.gen)
File "/home/stoyan/neurmon/lib/python3.5/site-packages/tensorflow/python/framework/errors_impl.py", line 466, in raise_exception_on_not_ok_status
pywrap_tensorflow.TF_GetCode(status))
tensorflow.python.framework.errors_impl.InvalidArgumentError: slice index 49 of dimension 0 out of bounds.
[[Node: sampling/strided_slice_4900 = StridedSlice[Index=DT_INT32, T=DT_INT32, begin_mask=0, ellipsis_mask=0, end_mask=0, new_axis_mask=0, shrink_axis_mask=1, _device="/job:localhost/replica:0/task:0/cpu:0"](sampling/TensorArrayStack_3/TensorArrayGatherV3, sampling/strided_slice_4900/stack, sampling/strided_slice_4900/stack_1, sampling/strided_slice_4900/stack_2)]]
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "bin/neuralmonkey-train", line 6, in <module>
main()
File "/home/stoyan/neuralmonkey/bin/neuralmonkey/train.py", line 211, in main
initial_variables=cfg.model.initial_variables)
File "/home/stoyan/neuralmonkey/bin/neuralmonkey/learning_utils.py", line 185, in training_loop
results, meta=tf_manager.execute(batch_dataset, [trainer],train=True, summaries=False)
File "/home/stoyan/neuralmonkey/bin/neuralmonkey/tf_manager.py", line 217, in execute
for sess in self.sessions]
File "/home/stoyan/neuralmonkey/bin/neuralmonkey/tf_manager.py", line 217, in <listcomp>
for sess in self.sessions]
File "/home/stoyan/neurmon/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 767, in run
run_metadata_ptr)
File "/home/stoyan/neurmon/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 965, in _run
feed_dict_string, options, run_metadata)
File "/home/stoyan/neurmon/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1015, in _do_run
target_list, options, run_metadata)
File "/home/stoyan/neurmon/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1035, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: slice index 49 of dimension 0 out of bounds.
[[Node: sampling/strided_slice_4900 = StridedSlice[Index=DT_INT32, T=DT_INT32, begin_mask=0, ellipsis_mask=0, end_mask=0, new_axis_mask=0, shrink_axis_mask=1, _device="/job:localhost/replica:0/task:0/cpu:0"](sampling/TensorArrayStack_3/TensorArrayGatherV3, sampling/strided_slice_4900/stack, sampling/strided_slice_4900/stack_1, sampling/strided_slice_4900/stack_2)]]
Caused by op 'sampling/strided_slice_4900', defined at:
File "bin/neuralmonkey-train", line 6, in <module>
main()
File "/home/stoyan/neuralmonkey/bin/neuralmonkey/train.py", line 170, in main
cfg.build_model(warn_unused=True)
File "/home/stoyan/neuralmonkey/bin/neuralmonkey/config/configuration.py", line 86, in build_model
model = build_config(self.config_dict, self.ignored, warn_unused)
File "/home/stoyan/neuralmonkey/bin/neuralmonkey/config/builder.py", line 198, in build_config
value, config_dicts, existing_objects, 0)
File "/home/stoyan/neuralmonkey/bin/neuralmonkey/config/builder.py", line 109, in build_object
obj = instantiate_class(value[7:], all_dicts, existing_objects, depth)
File "/home/stoyan/neuralmonkey/bin/neuralmonkey/config/builder.py", line 165, in instantiate_class
obj = clazz(*bounded_params.args, **bounded_params.kwargs)
File "/home/stoyan/neuralmonkey/bin/neuralmonkey/trainers/mrt_trainer.py", line 80, in __init__
ind=[[[tf.constant(i),tf.constant(j),sampled_ids[i][j]] for j in range(self.batch_size)] for i in range(self.decoder.max_output_len)]
File "/home/stoyan/neuralmonkey/bin/neuralmonkey/trainers/mrt_trainer.py", line 80, in <listcomp>
ind=[[[tf.constant(i),tf.constant(j),sampled_ids[i][j]] for j in range(self.batch_size)] for i in range(self.decoder.max_output_len)]
File "/home/stoyan/neuralmonkey/bin/neuralmonkey/trainers/mrt_trainer.py", line 80, in <listcomp>
ind=[[[tf.constant(i),tf.constant(j),sampled_ids[i][j]] for j in range(self.batch_size)] for i in range(self.decoder.max_output_len)]
File "/home/stoyan/neurmon/lib/python3.5/site-packages/tensorflow/python/ops/array_ops.py", line 495, in _SliceHelper
name=name)
File "/home/stoyan/neurmon/lib/python3.5/site-packages/tensorflow/python/ops/array_ops.py", line 653, in strided_slice
shrink_axis_mask=shrink_axis_mask)
File "/home/stoyan/neurmon/lib/python3.5/site-packages/tensorflow/python/ops/gen_array_ops.py", line 3688, in strided_slice
shrink_axis_mask=shrink_axis_mask, name=name)
File "/home/stoyan/neurmon/lib/python3.5/site-packages/tensorflow/python/framework/op_def_library.py", line 763, in apply_op
op_def=op_def)
File "/home/stoyan/neurmon/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 2327, in create_op
original_op=self._default_original_op, op_def=op_def)
File "/home/stoyan/neurmon/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 1226, in __init__
self._traceback = _extract_stack()
InvalidArgumentError (see above for traceback): slice index 49 of dimension 0 out of bounds.
[[Node: sampling/strided_slice_4900 = StridedSlice[Index=DT_INT32, T=DT_INT32, begin_mask=0, ellipsis_mask=0, end_mask=0, new_axis_mask=0, shrink_axis_mask=1, _device="/job:localhost/replica:0/task:0/cpu:0"](sampling/TensorArrayStack_3/TensorArrayGatherV3, sampling/strided_slice_4900/stack, sampling/strided_slice_4900/stack_1, sampling/strided_slice_4900/stack_2)]]
What should this InvalidArgumentError refer to and what goes wrong?
Best,
Stoyan
According to the stack trace, the error comes from this expression in your code:
sampled_ids[i][j]
...but it's hard to tell without context whether it comes from taking the [i] slice or the [j] slice. Presumably one of the tensors in this structure has fewer than 15 (or 49 in the error message) elements in the 0th dimension. Often this can happen if your input data includes word IDs that are not present in the vocabulary used for training the model.
I have a saved checkpoint generated by graph code in a regular non-distributed setup with the constraint with tf.device('/cpu:0'): (to force model params to reside on CPU instead of GPU).
Now I converted the same code/graph to a distributed setting following the guidelines in TF-Inception.
Now when I try to restore the checkpoint in distributed setup, I get device mismatch errors. Is there a way to override the requirements saved in the checkpoint file or something?
My new distributed code has the Saver and scopes defined as:
if FLAGS.job_name == 'worker':
with tf.device(tf.train.replica_device_setter(
worker_device="/job:worker/task:%d" % FLAGS.task_id,
cluster=cluster_spec)):
# ...same network-graph code... #
restorer = tf.train.Saver()
with tf.Session() as sess:
restorer.restore(sess, 'ResNet-L50.ckpt')
My cluster has one ps and one worker, and both are on localhost. Error line:
tensorflow.python.framework.errors.InvalidArgumentError: Cannot assign a device to node 'save/restore_slice_268/shape_and_slice': Could not satisfy explicit device specification '/job:ps/task:0/device:CPU:0' because no devices matching that specification are registered in this process; available devices: /job:localhost/replica:0/task:0/cpu:0, /job:localhost/replica:0/task:0/gpu:0
[[Node: save/restore_slice_268/shape_and_slice = Const[dtype=DT_STRING, value=Tensor<type: string shape: [] values: >, _device="/job:ps/task:0/device:CPU:0"]()]]
Full error trace:
I tensorflow/core/common_runtime/gpu/gpu_device.cc:756] Creating TensorFlow device (/gpu:0) -> (device: 0, name: Quadro K2200, pci bus id: 0000:01:00.0)
Traceback (most recent call last):
File "dlaunch.py", line 85, in <module>
tf.app.run() # (tf.app.flags parsed here)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/platform/app.py", line 30, in run
sys.exit(main(sys.argv))
File "dlaunch.py", line 81, in main
dtrainer.train(server.target, cluster_spec)
File "/home/muneeb/parkingtf/dtrainer.py", line 88, in train
restorer.restore(sess, 'ResNet-L50.ckpt')
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/saver.py", line 1103, in restore
{self.saver_def.filename_tensor_name: save_path})
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 328, in run
run_metadata_ptr)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 563, in _run
feed_dict_string, options, run_metadata)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 636, in _do_run
target_list, options, run_metadata)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 658, in _do_call
e.code)
tensorflow.python.framework.errors.InvalidArgumentError: Cannot assign a device to node 'save/restore_slice_268/shape_and_slice': Could not satisfy explicit device specification '/job:ps/task:0/device:CPU:0' because no devices matching that specification are registered in this process; available devices: /job:localhost/replica:0/task:0/cpu:0, /job:localhost/replica:0/task:0/gpu:0
[[Node: save/restore_slice_268/shape_and_slice = Const[dtype=DT_STRING, value=Tensor<type: string shape: [] values: >, _device="/job:ps/task:0/device:CPU:0"]()]]
Caused by op u'save/restore_slice_268/shape_and_slice', defined at:
File "dlaunch.py", line 85, in <module>
tf.app.run() # (tf.app.flags parsed here)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/platform/app.py", line 30, in run
sys.exit(main(sys.argv))
File "dlaunch.py", line 81, in main
dtrainer.train(server.target, cluster_spec)
File "/home/muneeb/parkingtf/dtrainer.py", line 86, in train
restorer = tf.train.Saver()
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/saver.py", line 845, in __init__
restore_sequentially=restore_sequentially)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/saver.py", line 515, in build
filename_tensor, vars_to_save, restore_sequentially, reshape)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/saver.py", line 271, in _AddRestoreOps
values = self.restore_op(filename_tensor, vs, preferred_shard)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/saver.py", line 186, in restore_op
preferred_shard=preferred_shard)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/io_ops.py", line 201, in _restore_slice
preferred_shard, name=name)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/gen_io_ops.py", line 271, in _restore_slice
preferred_shard=preferred_shard, name=name)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/op_def_library.py", line 444, in apply_op
as_ref=input_arg.is_ref)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 566, in convert_to_tensor
ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/constant_op.py", line 179, in _constant_tensor_conversion_function
return constant(v, dtype=dtype, name=name)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/constant_op.py", line 166, in constant
attrs={"value": tensor_value, "dtype": dtype_value}, name=name).outputs[0]
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 2162, in create_op
original_op=self._default_original_op, op_def=op_def)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 1161, in __init__
self._traceback = _extract_stack()
The following line:
with tf.Session() as sess:
...is responsible for the error. Passing no arguments to tf.Session() creates an in-process session that can only use devices on the local machine. To work in the distributed mode, you should have something like:
# Assuming you created `server = tf.train.Server(...)` earlier.
with tf.Session(server.target) as sess:
...or, if you are connecting to a different process:
# Assuming your server is in a different process.
with tf.Session("grpc://..."):
Note that the devices are not stored in the checkpoint file, but they are being added by the tf.train.replica_device_setter(). Device configuration is a bit tricky right now, and it's something that we're working to simplify.
I am trying to classify SVHN data set by following this tutorial:
https://www.tensorflow.org/versions/0.6.0/tutorials/deep_cnn/index.html
I am using train_32x32.mat file. In order to use it with CNN code (mentioned above), I converted this .mat file to several .bin files using this simple code:
import numpy as np
import scipy.io
from array import array
read_input = scipy.io.loadmat('data/train_32x32.mat')
j=0
output_file = open('data_batch_%d.bin' % j, 'ab')
for i in range(0, 64000):
# create new bin file
if i>0 and i % 12800 == 0:
output_file.close()
j=j+1
output_file = open('data_batch_%d.bin' % j, 'ab')
# Write to bin file
if read_input['y'][i] == 10:
read_input['y'][i] = 0
read_input['y'][i].astype('uint8').tofile(output_file)
read_input['X'][:,:,:,i].astype('uint32').tofile(output_file)
output_file.close()
But when I tried to classify SVHN using these customized .bin files I'm getting stuck with error "Invalid argument: Indices are not valid (out of bounds)" listed below:
Filling queue with 20000 CIFAR images before starting to train. This will take a few minutes.
I tensorflow/core/common_runtime/local_device.cc:25] Local device intra op parallelism threads: 4
I tensorflow/core/common_runtime/local_session.cc:45] Local session inter op parallelism threads: 4
W tensorflow/core/common_runtime/executor.cc:1027] 0x1a53160 Compute status: Invalid argument: Indices are not valid (out of bounds). Shape: dim { size: 128 } dim { size: 10 }
[[Node: SparseToDense = SparseToDense[T=DT_FLOAT, Tindices=DT_INT32, _device="/job:localhost/replica:0/task:0/cpu:0"](concat, SparseToDense/output_shape, SparseToDense/sparse_values, SparseToDense/default_value)]]
Traceback (most recent call last):
File "cifar10_train.py", line 138, in <module>
tf.app.run()
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/platform/default/_app.py", line 11, in run
sys.exit(main(sys.argv))
File "cifar10_train.py", line 134, in main
train()
File "cifar10_train.py", line 104, in train
_, loss_value = sess.run([train_op, loss])
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 345, in run
results = self._do_run(target_list, unique_fetch_targets, feed_dict_string)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 419, in _do_run
e.code)
tensorflow.python.framework.errors.InvalidArgumentError: Indices are not valid (out of bounds). Shape: dim { size: 128 } dim { size: 10 }
[[Node: SparseToDense = SparseToDense[T=DT_FLOAT, Tindices=DT_INT32, _device="/job:localhost/replica:0/task:0/cpu:0"](concat, SparseToDense/output_shape, SparseToDense/sparse_values, SparseToDense/default_value)]]
Caused by op u'SparseToDense', defined at:
File "cifar10_train.py", line 138, in <module>
tf.app.run()
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/platform/default/_app.py", line 11, in run
sys.exit(main(sys.argv))
File "cifar10_train.py", line 134, in main
train()
File "cifar10_train.py", line 76, in train
loss = cifar10.loss(logits, labels)
File "/home/sarah/Documents/SVHN/cifar10.py", line 364, in loss
dense_labels = tf.sparse_to_dense(concated,[FLAGS.batch_size, NUM_CLASSES],1.0, 0.0)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/gen_sparse_ops.py", line 153, in sparse_to_dense
default_value=default_value, name=name)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/op_def_library.py", line 633, in apply_op
op_def=op_def)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 1710, in create_op
original_op=self._default_original_op, op_def=op_def)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 988, in __init__
self._traceback = _extract_stack()
I found TensorFlow CIFAR10 Example , similar issue in stackoverflow. But even if I change the label, it's still not working.
Please let me know if I did something wrong or not understanding any logic.
Thanks
Sarah
Something was wrong with my installed version of Tensorflow (might be a bug). Upgrading to new version solved the issue.
Thanks