I'm trying to evaluate the NeuralGAMS LTR model using a test set and I'm doing this to create it:
# Build Test Dataset for Model Ingestion
features = preprocess_features(features_file)
feature_cols = np.array(features['cols'])
# Create specs for pipeline
context_spec_ = {}
example_spec_ = {feat: tf.io.FixedLenFeature(shape=(1,), dtype=tf.float32, default_value=0.0) for feat in feature_cols}
label_spec_ = ('relevance_label', tf.io.FixedLenFeature(shape=(1,), dtype=tf.int64, default_value=-1))
dataset_hparams = tfr.keras.pipeline.DatasetHparams(
train_input_pattern='tfrecord_HJ/HJ/all_data/train.tfrecords',
valid_input_pattern='tfrecord_GD/Golden_Data/LibSVM_format/test.tfrecords',
train_batch_size=128,
valid_batch_size=1,
dataset_reader=tfr.keras.pipeline.DatasetHparams.dataset_reader)
# Define Dataset Builder
dataset_builder = tfr.keras.pipeline.SimpleDatasetBuilder(
{},
example_spec_,
mask_feature_name="example_list_mask",
label_spec=label_spec_,
hparams=dataset_hparams,
sample_weight_spec=None)
ds = dataset_builder.build_valid_dataset()
When I use this ds for predictions this way:
predictions = model.predict(ds)
I get "When providing an infinite dataset, you must specify the number of steps to run (if you did not intend to create an infinite dataset, make sure to not call repeat() on the dataset)." error.
My questions are:
Why is an infinite dataset created when I'm just reading from an existing tfrecord to create it.
When I add 'steps', how do I get the predictions for all the data
How do I map these back to the inputs?
Better way to get predictions?
Related
I am following the Google Machine Learning Intensive Course. But it uses version 1.x of TensorFlow, so I was planning to change the exercises to be able to run them in TensorFlow 2.0. But I am stuck in that exercise:
https://colab.research.google.com/notebooks/mlcc/first_steps_with_tensor_flow.ipynb?utm_source=mlcc&utm_campaign=colab-external&utm_medium=referral&utm_content=firststeps-colab&hl=es#scrollTo=7UwqGbbxP53O
Specifically the code:
def my_input_fn(features, targets, batch_size=1, shuffle=True, num_epochs=None):
"""Trains a linear regression model of one feature.
Args:
features: pandas DataFrame of features
targets: pandas DataFrame of targets
batch_size: Size of batches to be passed to the model
shuffle: True or False. Whether to shuffle the data.
num_epochs: Number of epochs for which data should be repeated. None = repeat indefinitely
Returns:
Tuple of (features, labels) for next data batch
"""
# Convert pandas data into a dict of np arrays.
features = {key:np.array(value) for key,value in dict(features).items()}
# Construct a dataset, and configure batching/repeating.
ds = Dataset.from_tensor_slices((features,targets)) # warning: 2GB limit
ds = ds.batch(batch_size).repeat(num_epochs)
# Shuffle the data, if specified.
if shuffle:
ds = ds.shuffle(buffer_size=10000)
# Return the next batch of data.
features, labels = ds.make_one_shot_iterator().get_next()
return features, labels
I have replaced features, labels = ds.make_one_shot_iterator().get_next() with features, labels = tf.compat.v1.data.make_one_shot_iterator(ds).get_next()
and it seems to work but make_one_shot_iterator() is depreceated, so, how can i replace it?
Also according to https://github.com/tensorflow/tensorflow/issues/29252 , I have tried
features, labels = ds.__iter__()
next(ds.__iter__())
return features, labels
but it returns the error __iter __ () is only supported inside of tf.function or when eager execution is enabled.
I am quite inexperienced in python and follow the course as a hobbyist. Any ideas on how to solve it? Thank you.
After several tests, the python hang was a local problem.
To replace features, labels = ds.make_one_shot_iterator (). Get_next () I have tried several things:
features, labels = ds.__iter__().get_next()
iterator = ds.__iter__()
features, labels = iterator.get_next()
it = iter(ds)
features, labels = next(it)
All three cases return __iter__() is only supported inside of tf.function or when eager execution is enabled. so I tried:
features, labels = ds
return ds
And also just:
return features, labels
And both returns the same error, finally I tried:
return ds
And mysteriously it worked, I have no idea why, but it did.
1). I doubt, that you've really got what you wanted. Because if your input really needed to be multi-input - then your ds unlikely suits, you just need the list... something like this:
features = tf.compat.v1.data.make_one_shot_iterator(train_dataset).get_next()
image, label = features['image'], features['label']
2). Concerning Iterator - now it is belonging to 'tf.data' - with 'tf.data.Iterator.get_next()' method as opposed to previous tf.data.Datasetds.make_one_shot_iterator() -- 'Dependency Invertion' (D from SOLID principles of dev.) perhaps was done, perhaps to refactor....
New Iterator-entity now could be used for tf.data.Dataset.from_generator() objects feeding from fn_generator in async-mode each chunk of data yielded -- here is example of Custom-tfds.core.GeneratorBasedBuilder overwritting...
I think, the overall architecture of tf-lib was refactored a little-bit, because the input started to eat batch-by-batch itself (due to dev.'s implementations) -- & make_one_shot_iterator applied for Dataset no more needed... Even for debugging there is .as_numpy_iterator(), & make_one_shot_iterator no more considered to be needed by developers
though sometimes people use:
iterator = iter(batched_dataset)
next_element = iterator.get_next()
cannot assume where this could be needed yet
P.S. BTW, as I remember smth from Debugger, if your container is hashable or not iterable (or correct me) - you can try:
iterator = iter(dataset)
# batch_features, batch_labels = iterator.get_next()
el = iterator.get_next()
batch_features= el[:]
print(batch_features)
batch_labels= el[:-1]
print(batch_labels)
works OK
I created a dataset in TFRecord format for testing. Every entry contains 200 columns, named C1 - C199, each being a strings list, and a label column to denote the labels. The code to create the data can be found here: https://github.com/codescv/tf-dist/blob/8bb3c44f55939fc66b3727a730c57887113e899c/src/gen_data.py#L25
Then I used a linear model to train the data. The first approach looks like this:
dataset = tf.data.TFRecordDataset(data_file)
dataset = dataset.prefetch(buffer_size=batch_size*10)
dataset = dataset.map(parse_tfrecord, num_parallel_calls=5)
dataset = dataset.repeat(num_epochs)
dataset = dataset.batch(batch_size)
features, labels = dataset.make_one_shot_iterator().get_next()
logits = tf.feature_column.linear_model(features=features, feature_columns=columns, cols_to_vars=cols_to_vars)
train_op = ...
with tf.Session() as sess:
sess.run(train_op)
The full code can be found here: https://github.com/codescv/tf-dist/blob/master/src/lr_single.py
When I run the code above, I get 0.85 steps/sec (batch size being 1024).
In the second approach, I manually get batches from Dataset into python, then feed them to a placeholder, like this:
example = tf.placeholder(dtype=tf.string, shape=[None])
features = tf.parse_example(example, features=tf.feature_column.make_parse_example_spec(columns+[tf.feature_column.numeric_column('label', dtype=tf.float32, default_value=0)]))
labels = features.pop('label')
train_op = ...
dataset = tf.data.TFRecordDataset(data_file).repeat().batch(batch_size)
next_batch = dataset.make_one_shot_iterator().get_next()
with tf.Session() as sess:
data_batch = sess.run(next_batch)
sess.run(train_op, feed_dict={example: data_batch})
The full code can be found here: https://github.com/codescv/tf-dist/blob/master/src/lr_single_feed.py
When I run the code above, I get 5 steps/sec. That is 5x faster than the first approach. This is what I do not understand, because theoretically the second should be slower due to the extra serialization/deserialization of data batches.
Thanks!
There is currently (as of TensorFlow 1.9) a performance issue when using tf.data to map and batch tensors that have a large number of features with a small amount of data in each. The issue has two causes:
The dataset.map(parse_tfrecord, ...) transformation will execute O(batch_size * num_columns) small operations to create a batch. By contrast, feeding a tf.placeholder() to tf.parse_example() will execute O(1) operations to create the same batch.
Batching many tf.SparseTensor objects using dataset.batch() is much slower than directly creating the same tf.SparseTensor as the output of tf.parse_example().
Improvements to both these issues are underway, and should be available in a future version of TensorFlow. In the meantime, you can improve the performance of the tf.data-based pipeline by switching the order of the dataset.map() and dataset.batch() and rewriting the dataset.map() to work on a vector of strings, like the feeding based version:
dataset = tf.data.TFRecordDataset(data_file)
dataset = dataset.prefetch(buffer_size=batch_size*10)
dataset = dataset.repeat(num_epochs)
# Batch first to create a vector of strings as input to the map().
dataset = dataset.batch(batch_size)
def parse_tfrecord_batch(record_batch):
features = tf.parse_example(
record_batch,
features=tf.feature_column.make_parse_example_spec(
columns + [
tf.feature_column.numeric_column(
'label', dtype=tf.float32, default_value=0)]))
labels = features.pop('label')
return features, labels
# NOTE: Parallelism might not be as useful, because the individual map function now does
# more work per invocation, but you might want to experiment with this.
dataset = dataset.map(parse_tfrecord_batch)
# Add a prefetch at the end to pipeline execution.
dataset = dataset.prefetch(1)
features, labels = dataset.make_one_shot_iterator().get_next()
# ...
EDIT (2018/6/18): To answer your questions from the comments:
Why is dataset.map(parse_tfrecord, ...) O(batch_size * num_columns), not O(batch_size)? If parsing requires enumeration of the columns, why doesn't parse_example take O(num_columns)?
When you wrap TensorFlow code in a Dataset.map() (or other functional transformation) a constant number of extra operations per output are added to "return" values from the function and (in the case of tf.SparseTensor values) "convert" them to a standard format. When you directly pass the outputs of tf.parse_example() to the input of your model, these operations aren't added. While they are very small operations, executing so many of them can become a bottleneck. (Technically the parsing does take O(batch_size * num_columns) time, but the constants involved in parsing are much smaller than executing an operation.)
Why do you add a prefetch at the end of the pipeline?
When you're interested in performance, this is almost always the best thing to do, and it should improve the overall performance of your pipeline. For more information about best practices, see the performance guide for tf.data.
I have a GCMLE experiment and I am trying to upgrade my input_fn to use the new tf.data functionality. I have created the following input_fn based off of this sample
def input_fn(...):
dataset = tf.data.Dataset.list_files(filenames).shuffle(num_shards) # shuffle up the list of input files
dataset = dataset.interleave(lambda filename: # mix together records from cycle_length number of shards
tf.data.TextLineDataset(filename).skip(1).map(lambda row: parse_csv(row, hparams)), cycle_length=5)
if shuffle:
dataset = dataset.shuffle(buffer_size = 10000)
dataset = dataset.repeat(num_epochs)
dataset = dataset.batch(batch_size)
iterator = dataset.make_one_shot_iterator()
features = iterator.get_next()
labels = features.pop(LABEL_COLUMN)
return features, labels
my parse_csv is the same as what I used previously, but it is not currently working. I can fix some of the issues, but I don't fully understand why I am having these issues. Here is the start of my parse_csv() function
def parse_csv(..):
columns = tf.decode_csv(rows, record_defaults=CSV_COLUMN_DEFAULTS)
raw_features = dict(zip(FIELDNAMES, columns))
words = tf.string_split(raw_features['sentences']) # splitting words
vocab_table = tf.contrib.lookup.index_table_from_file(vocabulary_file = hparams.vocab_file,
default_value = 0)
....
Right away this tf.string_split() stops working and the error is ValueError: Shape must be rank 1 but is rank 0 for 'csv_preprocessing/input_sequence_generation/StringSplit' (op: 'StringSplit') with input shapes: [], []. -- this is easily solved by packing raw_features['sentences'] into a tensor via [raw_features['sentences']] but I do not understand why this is needed with the this dataset approach? How come in the old version this worked fine? For the shapes to match up with the rest of my model, I end up needing to remove this extra dimension at the end via words = tf.squeeze(words, 0) because I add this "unecessary" dimension to the tensor.
For whatever reason, I am also getting an error that the table is not initialized tensorflow.python.framework.errors_impl.FailedPreconditionError: Table not initialized. however, this code works completely fine with my old input_fn() (see below) so I don't know why I would now need to initialize the tables? I have not figured out a solution to this part. Is there anything that I am missing to be able to use tf.contrib.lookup.index_table_from_file within my parse_csv function?
For reference, this is my old input_fn() that still does work:
def input_fn(...):
filename_queue = tf.train.string_input_producer(tf.train.match_filenames_once(filenames),
num_epochs=num_epochs, shuffle=shuffle, capacity=32)
reader = tf.TextLineReader(skip_header_lines=skip_header_lines)
_, rows = reader.read_up_to(filename_queue, num_records=batch_size)
features = parse_csv(rows, hparams)
if shuffle:
features = tf.train.shuffle_batch(
features,
batch_size,
min_after_dequeue=2 * batch_size + 1,
capacity=batch_size * 10,
num_threads=multiprocessing.cpu_count(),
enqueue_many=True,
allow_smaller_final_batch=True
)
else:
features = tf.train.batch(
features,
batch_size,
capacity=batch_size * 10,
num_threads=multiprocessing.cpu_count(),
enqueue_many=True,
allow_smaller_final_batch=True
)
labels = features.pop(LABEL_COLUMN)
return features, labels
UPDATE TF 1.7
I am revisiting this with TF 1.7 (which should have all of the TF 1.6 features mentioned in #mrry answer) but I'm still unable to replicate the behavior. For my old input_fn() I am able to gete around 13 steps/sec. The new function that I am using is as follows:
def input_fn(...):
files = tf.data.Dataset.list_files(filenames).shuffle(num_shards)
dataset = files.apply(tf.contrib.data.parallel_interleave(lambda filename: tf.data.TextLineDataset(filename).skip(1), cycle_length=num_shards))
dataset = dataset.apply(tf.contrib.data.map_and_batch(lambda row:
parse_csv_dataset(row, hparams = hparams),
batch_size = batch_size,
num_parallel_batches = multiprocessing.cpu_count()))
dataset = dataset.prefetch(1)
if shuffle:
dataset = dataset.shuffle(buffer_size = 10000)
dataset = dataset.repeat(num_epochs)
iterator = dataset.make_initializable_iterator()
features = iterator.get_next()
tf.add_to_collection(tf.GraphKeys.TABLE_INITIALIZERS, iterator.initializer)
labels = {key: features.pop(key) for key in LABEL_COLUMNS}
return features, labels
I believe that I am following all of the performance guildines such as 1) use prefetch 2) use map_and_batch with num_parallel_batches = cores 3) use parallel_interleave 4) applying shuffle before the repeat. The only steps I am not using is the cache suggestion, but would expect that to really only help for epochs beyond the first one as well as "applying interleave, prefetch and shuffle first." -- however I found that having prefetch and shuffle after the map_and_batch was ~10% speedup.
BUFFER ISSUE
The first performance issue that I am noticing is with my old input_fn() it took me about 13 wall clock minutes to get through 20k steps, and yet even with the buffer_size of 10,000 (which I take to mean we are waiting until we have 10,000 batches processed) I am still waiting more than 40 minutes for the buffer to get full . Does it make sense to take this long? If I know that my sharded .csv's on GCS are already randomized, is it acceptable to have this shuffle/buffer size smaller? I am trying to replicate the behavior from tf.train.shuffle_batch() -- however, it seems that at worst it should take the same 13 mins that it took to reach 10k steps in order to fill up the buffer?
STEPS/SEC
Even once the buffer has filled up, the global steps/sec tops out around 3 steps/sec (often as low as 2 steps/sec) on the same model with the previous input_fn() that is getting ~13 steps/sec.
SLOPPY INTERLEAVE
I finall tried to replace parallel_interleave() with sloppy_interleave() as this is another suggestion from #mrry. When I switched to sloppy_interleave I got 14 steps/sec! I know this means that it is not deterministic, but that should really just mean it is not deterministic from one run (or epoch) to the next? Or are there larger implications for this? Should I be concerned about any real difference between the old shuffle_batch() method and sloppy_interleave? Does the fact that this results in a 4-5x improvement suggest what the previous blocking factor was?
In TF 1.4 (which is currently the latest version of TF that works with GCMLE) you will not be able to use make_one_shot_iterator() with the lookup tables (see relevant post) you will need to use Dataset.make_initializable_iterator() and then initialize iterator.initalizer with your default TABLES_INITIALIZER (from this post). Here is what the input_fn() should look like:
def input_fn(...):
dataset = tf.data.Dataset.list_files(filenames).shuffle(num_shards)
# Define `vocab_table` outside the map function and use it in `parse_csv()`.
vocab_table = tf.contrib.lookup.index_table_from_file(
vocabulary_file=hparams.vocab_file, default_value=0)
dataset = dataset.interleave(
lambda filename: (tf.data.TextLineDataset(filename)
.skip(1)
.map(lambda row: parse_csv(row, hparams),
num_parallel_calls=multiprocessing.cpu_count())),
cycle_length=5)
if shuffle:
dataset = dataset.shuffle(buffer_size=10000)
dataset = dataset.repeat(num_epochs)
dataset = dataset.batch(batch_size)
iterator = dataset.make_initializable_iterator()
features = iterator.get_next()
# add iterator.intializer to be handled by default table initializers
tf.add_to_collection(tf.GraphKeys.TABLE_INITIALIZERS, iterator.initializer)
labels = features.pop(LABEL_COLUMN)
return features, labels
When you use tf.data.TextLineDataset, each element is a scalar string. In this respect, it is more similar to using tf.TextLineReader.read(), rather than the batch version tf.TextLineReader.read_up_to(), which returns a vector of strings. Unfortunately the tf.string_split() op demands a vector input (although this could potentially be changed in future), so the shape manipulation is currently necessary.
Lookup tables interact a little differently with the functions in tf.data. The intuition is that you should declare the lookup table once outside the Dataset.map() call (so that it will be initialized once) and then capture it inside the parse_csv() function to call vocab_table.lookup(). Something like the following should work:
def input_fn(...):
dataset = tf.data.Dataset.list_files(filenames).shuffle(num_shards)
# Define `vocab_table` outside the map function and use it in `parse_csv()`.
vocab_table = tf.contrib.lookup.index_table_from_file(
vocabulary_file=hparams.vocab_file, default_value=0)
def parse_csv(...):
columns = tf.decode_csv(rows, record_defaults=CSV_COLUMN_DEFAULTS)
raw_features = dict(zip(FIELDNAMES, columns))
words = tf.string_split([raw_features['sentences']]) # splitting words
# Use the captured `vocab_table` here.
word_indices = vocab_table.lookup(words)
# ...
features = ...
# NOTE: Structure the output here so that you can simply return
# the dataset from `input_fn()`.
labels = features.pop(LABEL_COLUMN)
return features, labels
# NOTE: Consider using `tf.contrib.data.parallel_interleave()` to perform
# the reads in parallel.
dataset = dataset.interleave(
lambda filename: (tf.data.TextLineDataset(filename)
.skip(1)
.map(lambda row: parse_csv(row, hparams),
num_parallel_calls=multiprocessing.cpu_count())),
cycle_length=5)
if shuffle:
dataset = dataset.shuffle(buffer_size=10000)
dataset = dataset.repeat(num_epochs)
dataset = dataset.batch(batch_size)
# NOTE: Add prefetching here to run the input pipeline in the background.
dataset = dataset.prefetch(1)
# NOTE: This requires TensorFlow 1.5 or later, but this change simplifies the
# initialization of the lookup table.
return dataset
The neural network I am currently working on is accepting a sparse tensor as input. I am reading my data from a TFRecord as follows:
_, examples = tf.TFRecordReader(options=options).read_up_to(
filename_queue, num_records=batch_size)
features = tf.parse_example(examples, features={
'input_feat': tf.SparseFeature(index_key='input_feat_idx',
value_key='input_feat_values',
dtype=tf.int64,
size=SIZE_FEATURE)})
It works like a charm but I was looking at the tf.data API which looks more convenient for a lot of tasks and I am not sure how to read tf.SparseTensor objects like I do with the tf.RecordReader and tf.parse_example(). Any idea?
TensorFlow 1.5 will add native support for tf.SparseTensor in the core transformations. (This is currently available if you pip install tf-nightly, or build from source on the master branch of TensorFlow.) This means that you can write your pipeline as the following:
# Create a dataset of string records from the input files.
dataset = tf.data.TFRecordReader(filenames)
# Convert each string record into a `tf.SparseTensor` representing a single example.
dataset = dataset.map(lambda record: tf.parse_single_example(
record, features={'input_feat': tf.SparseFeature(index_key='input_feat_idx',
value_key='input_feat_values',
dtype=tf.int64,
size=SIZE_FEATURE)})
# Stack together up to `batch_size` consecutive elements into a `tf.SparseTensor`
# representing a batch of examples.
dataset = dataset.batch(batch_size)
# Create an iterator to access the elements of `dataset` sequentially.
iterator = dataset.make_one_shot_iterator()
# `next_element` is a `tf.SparseTensor`.
next_element = iterator.get_next()
I am facing to an issue that can't solve with what I found on the internet.
I have build my neural network and connect it to inpute pipeline.
Reading data from tfrecord, with tf.train.batch and queueRunners, Coords, etc..
I have build my NN into a python class named "Model" that I use like :
model = Model(...all hyperparameter here...)
...
model.predict()
or
model.step()
All the training phase works very well.
But now I would like to add a test phase every X epoch/step of training.
I really don't know how to do this.
I have several idea but I don't find the best one:
Duplicate the code into my class to get : loss_train and loss_test, and so on for each node of my graph ? (using sharing variable between train and test)
create 2 instance of my model :
model_train = Model(reuse=false)
model_test = Model(reuse=true)
use tf.make_template ? I really don't found any good exemple of this fonction ...
any other solution ?
I would appreciate any suggestion,
I came across the same Problem when experimenting with TFRecords Datasets. There are several possibilities. Since I wanted to do this on a Computer with only one GPU anyways I implemented it as follows:
# Training Dataset
train_dataset = tf.contrib.data.TFRecordDataset(train_files)
train_dataset = train_dataset.map(parse_function)
train_dataset = train_dataset.shuffle(buffer_size=10000)
train_dataset = train_dataset.batch(200)
# Validation Dataset
validation_dataset = tf.contrib.data.TFRecordDataset(val_files)
validation_dataset = validation_dataset.map(parse_function)
validation_dataset = validation_dataset.batch(200)
# A feedable iterator is defined by a handle placeholder and its structure. We
# could use the `output_types` and `output_shapes` properties of either
# `training_dataset` or `validation_dataset` here, because they have
# identical structure.
handle = tf.placeholder(tf.string, shape=[])
iterator = tf.contrib.data.Iterator.from_string_handle(handle,
train_dataset.output_types, train_dataset.output_shapes)
next_element = iterator.get_next()
# Generate the Iterators
training_iterator = train_dataset.make_initializable_iterator()
validation_iterator = validation_dataset.make_one_shot_iterator()
# The `Iterator.string_handle()` method returns a tensor that can be evaluated
# and used to feed the `handle` placeholder.
training_handle = sess.run(training_iterator.string_handle())
validation_handle = sess.run(validation_iterator.string_handle())
Then for accessing the elements, you can just go like:
img, lbl = sess.run(next_element, feed_dict={handle: training_handle})
And exchange the handle dependant on what you are willing to do ATM.
Keep in mind that this is not parallelizable, however. Following this link, you can get insight into the different methods of creating multiple input pipelines Tensorflow | Reading Data.