InvalidArgumentError (see above for traceback): slice index 15 of dimension 0 out of bounds - tensorflow

Question

In a task to implement the minimum risk training for a neural machine translation system I need to sample sentences and gather the respective logits for the sampled word IDs. The step of gathering looks like this:
for i in range(1,self._num_of_samples):
logits, _, _, sampled_ids = self.decoder._decoding_loop(train_mode=False,sample=True)
ind=[[[tf.constant(i),tf.constant(j),sampled_ids[i][j]] for j in range(self.batch_size)] for i in range(self.decoder.max_output_len)]
gathered_logits = tf.gather_nd(logits,ind)
sentence_sum_logit = tf.reduce_sum(gathered_logits,0)
self.sample_sen_ids = self.sample_sen_ids.write(steps[i],sampled_ids)
self.sample_logits = self.sample_logits.write(steps[i], sentence_sum_logit)
self.sample_sen_ids = tf.transpose(self.sample_sen_ids.stack())
self.sample_logits = tf.transpose(self.sample_logits.stack())
But I dont get it why after some batches I get this:
Traceback (most recent call last):
File "/home/stoyan/neurmon/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1022, in _do_call
return fn(*args)
File "/home/stoyan/neurmon/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1004, in _run_fn
status, run_metadata)
File "/usr/lib/python3.5/contextlib.py", line 66, in __exit__
next(self.gen)
File "/home/stoyan/neurmon/lib/python3.5/site-packages/tensorflow/python/framework/errors_impl.py", line 466, in raise_exception_on_not_ok_status
pywrap_tensorflow.TF_GetCode(status))
tensorflow.python.framework.errors_impl.InvalidArgumentError: slice index 49 of dimension 0 out of bounds.
[[Node: sampling/strided_slice_4900 = StridedSlice[Index=DT_INT32, T=DT_INT32, begin_mask=0, ellipsis_mask=0, end_mask=0, new_axis_mask=0, shrink_axis_mask=1, _device="/job:localhost/replica:0/task:0/cpu:0"](sampling/TensorArrayStack_3/TensorArrayGatherV3, sampling/strided_slice_4900/stack, sampling/strided_slice_4900/stack_1, sampling/strided_slice_4900/stack_2)]]
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "bin/neuralmonkey-train", line 6, in <module>
main()
File "/home/stoyan/neuralmonkey/bin/neuralmonkey/train.py", line 211, in main
initial_variables=cfg.model.initial_variables)
File "/home/stoyan/neuralmonkey/bin/neuralmonkey/learning_utils.py", line 185, in training_loop
results, meta=tf_manager.execute(batch_dataset, [trainer],train=True, summaries=False)
File "/home/stoyan/neuralmonkey/bin/neuralmonkey/tf_manager.py", line 217, in execute
for sess in self.sessions]
File "/home/stoyan/neuralmonkey/bin/neuralmonkey/tf_manager.py", line 217, in <listcomp>
for sess in self.sessions]
File "/home/stoyan/neurmon/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 767, in run
run_metadata_ptr)
File "/home/stoyan/neurmon/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 965, in _run
feed_dict_string, options, run_metadata)
File "/home/stoyan/neurmon/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1015, in _do_run
target_list, options, run_metadata)
File "/home/stoyan/neurmon/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1035, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: slice index 49 of dimension 0 out of bounds.
[[Node: sampling/strided_slice_4900 = StridedSlice[Index=DT_INT32, T=DT_INT32, begin_mask=0, ellipsis_mask=0, end_mask=0, new_axis_mask=0, shrink_axis_mask=1, _device="/job:localhost/replica:0/task:0/cpu:0"](sampling/TensorArrayStack_3/TensorArrayGatherV3, sampling/strided_slice_4900/stack, sampling/strided_slice_4900/stack_1, sampling/strided_slice_4900/stack_2)]]
Caused by op 'sampling/strided_slice_4900', defined at:
File "bin/neuralmonkey-train", line 6, in <module>
main()
File "/home/stoyan/neuralmonkey/bin/neuralmonkey/train.py", line 170, in main
cfg.build_model(warn_unused=True)
File "/home/stoyan/neuralmonkey/bin/neuralmonkey/config/configuration.py", line 86, in build_model
model = build_config(self.config_dict, self.ignored, warn_unused)
File "/home/stoyan/neuralmonkey/bin/neuralmonkey/config/builder.py", line 198, in build_config
value, config_dicts, existing_objects, 0)
File "/home/stoyan/neuralmonkey/bin/neuralmonkey/config/builder.py", line 109, in build_object
obj = instantiate_class(value[7:], all_dicts, existing_objects, depth)
File "/home/stoyan/neuralmonkey/bin/neuralmonkey/config/builder.py", line 165, in instantiate_class
obj = clazz(*bounded_params.args, **bounded_params.kwargs)
File "/home/stoyan/neuralmonkey/bin/neuralmonkey/trainers/mrt_trainer.py", line 80, in __init__
ind=[[[tf.constant(i),tf.constant(j),sampled_ids[i][j]] for j in range(self.batch_size)] for i in range(self.decoder.max_output_len)]
File "/home/stoyan/neuralmonkey/bin/neuralmonkey/trainers/mrt_trainer.py", line 80, in <listcomp>
ind=[[[tf.constant(i),tf.constant(j),sampled_ids[i][j]] for j in range(self.batch_size)] for i in range(self.decoder.max_output_len)]
File "/home/stoyan/neuralmonkey/bin/neuralmonkey/trainers/mrt_trainer.py", line 80, in <listcomp>
ind=[[[tf.constant(i),tf.constant(j),sampled_ids[i][j]] for j in range(self.batch_size)] for i in range(self.decoder.max_output_len)]
File "/home/stoyan/neurmon/lib/python3.5/site-packages/tensorflow/python/ops/array_ops.py", line 495, in _SliceHelper
name=name)
File "/home/stoyan/neurmon/lib/python3.5/site-packages/tensorflow/python/ops/array_ops.py", line 653, in strided_slice
shrink_axis_mask=shrink_axis_mask)
File "/home/stoyan/neurmon/lib/python3.5/site-packages/tensorflow/python/ops/gen_array_ops.py", line 3688, in strided_slice
shrink_axis_mask=shrink_axis_mask, name=name)
File "/home/stoyan/neurmon/lib/python3.5/site-packages/tensorflow/python/framework/op_def_library.py", line 763, in apply_op
op_def=op_def)
File "/home/stoyan/neurmon/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 2327, in create_op
original_op=self._default_original_op, op_def=op_def)
File "/home/stoyan/neurmon/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 1226, in __init__
self._traceback = _extract_stack()
InvalidArgumentError (see above for traceback): slice index 49 of dimension 0 out of bounds.
[[Node: sampling/strided_slice_4900 = StridedSlice[Index=DT_INT32, T=DT_INT32, begin_mask=0, ellipsis_mask=0, end_mask=0, new_axis_mask=0, shrink_axis_mask=1, _device="/job:localhost/replica:0/task:0/cpu:0"](sampling/TensorArrayStack_3/TensorArrayGatherV3, sampling/strided_slice_4900/stack, sampling/strided_slice_4900/stack_1, sampling/strided_slice_4900/stack_2)]]
What should this InvalidArgumentError refer to and what goes wrong?
Best,
Stoyan

Answer 1

According to the stack trace, the error comes from this expression in your code:
sampled_ids[i][j]
...but it's hard to tell without context whether it comes from taking the [i] slice or the [j] slice. Presumably one of the tensors in this structure has fewer than 15 (or 49 in the error message) elements in the 0th dimension. Often this can happen if your input data includes word IDs that are not present in the vocabulary used for training the model.