Double Batching Tensorflow Input Data - tensorflow

Question

I am implementing an convnet for token classification of string data. I
need to take in string data from a TFRecord, batch shuffled, then perform some processing which expands the data, and batch that again. Is this possible with two batch_shuffle operations?
This is what I need to do:
enqueue filenames into a filequeue
for each serialized Example, put onto a shuffle_batch
When I pull each example off the shuffle batch, I need to PAD it, replicate it by the sequence length, concatinating position vector, this creates multiple examples for each original example from the first batch. I need to batch it again.
Of course, one solution is to just preprocess the data before loading it into TF, but that will take up way more diskspace than is necessary.
DATA
Here is some sample data. I have two "Examples". Each Example contains features of a tokenized sentences and labels for each token:
sentences = [
[ 'the', 'quick', 'brown', 'fox', 'jumped', 'over', 'the', 'lazy', 'dog' '.'],
['then', 'the', 'lazy', 'dog', 'slept', '.']
]
sent_labels = [
['O', 'O', 'O', 'ANIMAL', 'O', 'O', 'O', 'ANIMAL', 'O'],
['O', 'O', 'O', 'ANIMAL', 'O', 'O']
]
Each "Example" Now has features as below (some reducution for clarity):
features {
feature {
key: "labels"
value {
bytes_list {
value: "O"
value: "O"
value: "O"
value: "ANIMAL"
...
}
}
}
feature {
key: "sentence"
value {
bytes_list {
value: "the"
value: "quick"
value: "brown"
value: "fox"
...
}
}
}
}
Transformation
After Batching the sparse data, I receive a sentence as list of tokens:
['the', 'quick', 'brown', 'fox', ...]
I need to PAD the list first to a predetermined SEQ_LEN, and then insert
position indices into each example, rotating the positions such that the
toke I want to classify is at pos 0, and every position token is relative
to the 0 position:
[
['the', 0 , 'quick', 1 , 'brown', 2 , 'fox', 3, 'PAD', 4] # classify 'the'
['the', -1, 'quick', 0 , 'brown', 1 , 'fox', 2 'PAD', 3 ] # classify 'quick
['the', -2, 'quick', -1, 'brown', 0 , 'fox', 1 'PAD', 2 ] # classify 'brown
['the', -3, 'quick', -2, 'brown', -1, 'fox', 0 'PAD', 1 ] # classify 'fox
]
Batching and ReBatching The Data
Here is a simplified version of what I'm trying to do:
# Enqueue the Filenames and serialize
filenames =[outfilepath]
fq = tf.train.string_input_producer(filenames, num_epochs=num_epochs, shuffle=True, name='FQ')
reader = tf.TFRecordReader()
key, serialized_example = reader.read(fq)
# Dequeue Examples of batch_size == 1. Because all examples are Sparse Tensors, do 1 at a time
initial_batch = tf.train.shuffle_batch([serialized_example], batch_size=1, capacity, min_after_dequeue)
# Parse Sparse Tensors, make into single dense Tensor
# ['the', 'quick', 'brown', 'fox']
parsed = tf.parse_example(data_batch, features=feature_mapping)
dense_tensor_sentence = tf.sparse_tensor_to_dense(parsed['sentence'], default_value='<PAD>')
sent_len = tf.shape(dense_tensor_sentence)[1]
SEQ_LEN = 5
NUM_PADS = SEQ_LEN - sent_len
#['the', 'quick', 'brown', 'fox', 'PAD']
padded_sentence = pad(dense_tensor_sentence, NUM_PADS)
# make sent_len X SEQ_LEN copy of sentence, position vectors
#[
# ['the', 0 , 'quick', 1 , 'brown', 2 , 'fox', 3, 'PAD', 4 ]
# ['the', -1, 'quick', 0 , 'brown', 1 , 'fox', 2 'PAD', 3 ]
# ['the', -2, 'quick', -1, 'brown', 0 , 'fox', 1 'PAD', 2 ]
# ['the', -3, 'quick', -2, 'brown', -1, 'fox', 0 'PAD', 1 ]
# NOTE: There is no row where PAD is with a position 0, because I don't
# want to classify the PAD token
#]
examples_with_positions = replicate_and_insert_positions(padded_sentence)
# While my SEQ_LEN will be constant, the sent_len will not. Therefore,
#I don't know the number of rows, but I can guarantee the number of
# columns. shape = (?,SEQ_LEN)
dynamic_input = final_reshape(examples_with_positions) # shape = (?, SEQ_LEN)
# Try Random Shuffle Queue:
# Rebatch <-- This is where the problem is
#reshape_concat.set_shape((None, SEQ_LEN))
random_queue = tf.RandomShuffleQueue(10000, 50, [tf.int64], shapes=(SEQ_LEN,))
random_queue.enqueue_many(dynamic_input)
batch = random_queue.dequeue_many(4)
init_op = tf.group(tf.global_variables_initializer(), tf.local_variables_initializer(), tf.initialize_all_tables())
sess = create_session()
sess.run(init_op)
#tf.get_default_graph().finalize()
coord = tf.train.Coordinator()
threads = tf.train.start_queue_runners(sess=sess, coord=coord)
try:
i = 0
while True:
print sess.run(batch)
i += 1
except tf.errors.OutOfRangeError as e:
print "No more inputs."
EDIT
I'm now trying to use the RandomShuffleQueue. On each enqueue, I would like to enqueue a batch with shape(None, SEQ_LEN). I've modified the code above to reflect this.
I no longer get complaints about the input shapes, but the queuing does hang at sess.run(batch)

Answer 1

I was approaching the entire problem incorrectly. I mistakenly was thinking that I had to define the complete shape of the batch while inserting into tf.batch_shuffle, but I actually needed to define only the shape of each element that I was inputing, and set enqueue_many=True.
Here is the the correct code:
single_batch=1
input_batch_size = 64
min_after_dequeue = 10
capacity = min_after_dequeue + 3 * input_batch_size
num_epochs=2
SEQ_LEN = 10
filenames =[outfilepath]
fq = tf.train.string_input_producer(filenames, num_epochs=num_epochs, shuffle=True)
reader = tf.TFRecordReader()
key, serialized_example = reader.read(fq)
# Dequeue examples of batch_size == 1. Because all examples are Sparse Tensors, do 1 at a time
first_batch = tf.train.shuffle_batch([serialized_example], ONE, capacity, min_after_dequeue)
# Get a single sentence and preprocess it shape=(sent_len)
single_sentence = tf.parse_example(first_batch, features=feature_mapping)
# Preprocess Sentence. shape=(sent_len, SEQ_LEN * 2). Each row is example
processed_inputs = preprocess(single_sentence)
# Re batch
input_batch = tf.train.shuffle_batch([processed_inputs],
batch_size=input_batch_size,
capacity=capacity, min_after_dequeue=min_after_dequeue,
shapes=[SEQ_LEN * 2], enqueue_many=True) #<- This is the fix
init_op = tf.group(tf.global_variables_initializer(), tf.local_variables_initializer(), tf.initialize_all_tables())
sess = create_session()
sess.run(init_op)
#tf.get_default_graph().finalize()
coord = tf.train.Coordinator()
threads = tf.train.start_queue_runners(sess=sess, coord=coord)
try:
i = 0
while True:
print i
print sess.run(input_batch)
i += 1
except tf.errors.OutOfRangeError as e:
print "No more inputs."