Let us suppose I am importing an image dataset using "image_dataset_from_directory" with no labels, with the shuffling argument set to True.
train_ds, val_ds = image_dataset_from_directory(
my_data_folder,
label_mode=None,
seed=123,
image_size=(224,224),
batch_size=32,
validation_split=0.4,
subset='both',
shuffle=True) # when True, only the training set is shuffled
Afterwards, I want to keep track of the file paths I imported, so I used "file_paths" property. Because the file paths are not in batches, I had to do the following:
file_paths_dataset = tf.data.Dataset.from_tensor_slices(train_ds.file_paths).batch(32)
train_ds = tf.data.Dataset.zip((train_ds, file_paths_dataset))
Seems everything is OK. But now the shuffling starts. Everytime I access train_ds, it is shuffled. When I say "access", I am referring to one of the following options:
next(iter(train_ds))
for this_batch in train_ds:
print(list(train_ds))[batch]
All these options reshuffle the dataset.
The problem is the following: only the first column (with the images) is shuffled. The paths are not shuffled, and now they are unsynced with the respective images.
Honestly, I cannot see any appication where it is usefull to shuffle independently columns of a dataset.
Now, imagine I was adding not the list of paths, but a list of labels? When I trained the data, everything would be messed up.
Anyway, I believe shuffling should be applied to all columns of a dataset. This is done with I import a labeled dataset, but not when I merge datasets using the zip method.
I also don't know how to print my dataset in a reproductive way other than setting shuffle=False when importing the data. I don't want to do it, because of the following statement I found in Keras documentation when using model.fit:
shuffle: Boolean (whether to shuffle the training data before each epoch) or str (for 'batch'). This argument is ignored when x is a generator or an object of tf.data.Dataset.
link
So if I set shuffle to False, and because I used a generator, the argument is ignored and this means I cannot shuffle the data when training my model. Thats not what I want. I want to shuffle ONLY when training my model, and not otherwise.
Maybe I am doing something wrong and there is a better way to do this. I don't know. Anyone knows a workaround? I want my file path list to be synced with the respective images all the time.
Best regards
Related
I have a really simple code that takes the training data from MNIST and then chooses the last 10,000 examples as validation set, then deletes the last 10,000 examples from the training set.
import tensorflow as tf
import tensorflow_datasets as tfds
(X_train, Y_train)= tf.keras.datasets.mnist.load_data()
X_valid = X_train[-10000:]
Y_valid = Y_train[-10000:]
X_train = X_train[0:40000]
Y_train = Y_train[0:40000]
However, this is very dumb in my opinion and I would like to make the data splitting procedure more sophisticated in the following ways:
I should specify which percentage of the data I want as validation set, instead of just taking the last whatever samples
I need a way to make sure that the data is balanced after I partition it into training and validation. Grabbing a portion could cause a the training examples associated with some digits to be very few.
Surprisingly I went through almost every Tensorflow tutorial and none of them does any validation (except for https://www.tensorflow.org/guide/keras/writing_a_training_loop_from_scratch, which uses the same dumb data splitting methodology as above). Most examples just directly splits the data into train and test which we almost never do in real life.
Could someone please advise?
The keras.datasets.mnist dataset loads the dataset by Yann LeCun (Refer Doc)
The dataset is setup in such a way that it contains 60,000 training data and 10,000 testing data.
Since the load_data() just returns Numpy arrays, you can easily concatenate the train and test arrays into a single array, after which you can play with the new array as you like.
If you want a validation set out of the training set, then you can shuffle the training set first and then extract the validation set.
All these operations will be simple Numpy Array operations and wouldn't even require any Tensorflow functionality.
I've recently switched my modeling framework to use custom Tensorflow Estimators and Datasets, and am quite happy overall with this workflow.
However, I've just noticed an issue with how my dataset_input_fn loads data form tfrecords. My input function is modeled after the example in the Tensorflow documentation. The issue arises when I have more examples than I can fit into RAM. If I have 1e6 examples, and set my shuffle buffer_size to 1e5, a subset of 1e5 examples is selected once, shuffled, and then iterated on. Meaning my model is only trained on 10% of my overall dataset. My code that sets up this behavior is borrowed exactly from the Tensorflow documentation example code:
dataset = dataset.map(parser)
dataset = dataset.shuffle(buffer_size=10000)
dataset = dataset.batch(32)
dataset = dataset.repeat(num_epochs)
iterator = dataset.make_one_shot_iterator()
My question: is it possible to fill the shuffle buffer with new examples outside of the initial 1e5 as I train? Is this type of functionality supported with a one_shot_iterator? Do I need to use an initializable iterator?
Thanks!
I have found what appears to be a tenable workaround for now. Through some experimentation, I learned that when instantiating a TFRecordDataset,
filenames = ["file1.tfrecord", ..., "filen.tfrecord"]
dataset = tf.data.TFRecordDataset(filenames)
and setting up a shuffle buffer:
dataset = dataset.shuffle(buffer_size=10000)
the buffer is only populated with the first 10000 examples from however many tf records that requires. For example, in my case, I have ~300 tfrecord files containing 4096 examples each. On examination, my shuffle buffer appears to consists only of examples from the first 3 tf records in my filenames list. Since my filenames list is static, this means that my model is only trained of my first 3 tfrecords!
My fix for now is pretty simple. In my training loop I already alternate between Estimator.train and Estimator.evaluate, and I noticed that each time I call Estimator.train, the shuffle buffer is repopulated. My solution then is to shuffle my filenames each time my input_fn is called. This is not a very elegant solution, but does achieve the desired effect of allowing my to iterate across all tfrecords.
#My Crappy Fix: shuffle file names in input_fn
np.random.shuffle(filenames)
dataset = tf.data.TFRecordDataset(filenames)
What's annoying about this solution (aside from its kludginess) is that my minibatches are not "globally random". Rather, they are selected form a small subset of tf records, and only that subset is used for each training/evaluation cycle. One way to mitigate this is to increase my shuffle buffer size or decrease my tfrecord size, I'll probably do both of these. Finally, I think it's worth noting that if
shuffle_buffer_size < (tf_record_size + minibatch_size)
then, as far as I can tell, my TFRecordDataset will pull from a single tfrecord file!
Finally, I don't think the relevant tensorflow documentation conveys these complexities well. The documentation alludes to the ability to train on large datasets that don't fit into memory, but doesn't provide much detail. It seems unlikely that the tf authors had in mind my hacky strategy when writing this, so I remain curious to see if there's a better approach.
I have a network with weights filled by manual tf.assign, and now I want to save the network with the weight values but without the placeholder inputs. It seems tf.train.Saver works only when I have the feed_dict available, and tf.train.export_meta_graph only saves the network structure. I tried pickle and dill but they both have errors. Are there any better solutions for this kind of saving?
Placeholders convert the input data into Tensors so I guess they are an important part of the Graph and I don't understand why you don't want to include them.
Even if you use tf.assign, you can freeze the graph, which means combining the structure with the weights. What freezing does is to convert Tensorflow variables into constants.
You have to save the structure of your graph:
gdef = g.as_graph_def()
tf.train.write_graph(gdef,".","graph.pb",False)
Then save the weights (after training)
saver.save(sess, 'tmp/my-weights')
And freeze the graph according to the tutorial in https://github.com/tensorflow/tensorflow/tree/master/tensorflow/contrib/lite
After that, you can use the Graph.
I have downloaded many face images from web. In order to learn Tensorflow I want to feed those images to a simple fully-connected neural network with a single hidden layer. I have found an example code in here.
Since I am a beginner, I don't know how to train, evaluate, and test the network with the downloaded images. The code owner used a '.mat' file and a .pkl file. I don't understand how he organized training and test set.
In order to run the code with my images;
Do I need to divide my images into training, test, and validation folders and turn each folder into a mat file? How am I going to provide labels for the training?
Besides, I don't understand why he used a '.pkl' file?
All in all, I would like to change this code so that I can find test, training , and validation set classification performance with my image dataset.
It might be an easy question, but it is important for me as it is a starting step. Thanks for your understanding.
First, you don't have to use .mat files nor pickles. Tensorflow expects numpy array.
For instance, let's say you have 70000 images of size 28x28 (=784 dimensions) belonging to 10 classes. Let's also assume that you'd like to train a simple feedforward neural network to classify the images.
The first step would be to split the images between train and test (and validation, but let's put this aside for the sake of simplicity). For the sake of the example, let's imagine that you chose randomly 60000 images for your training set and 10000 for your test set.
The second step would be to ensure that your data has the right format. Here, you'd like your training set to consist in one numpy array of shape (60000, 784) for the images and another one of shape (60000, 10) for the labels (if you use one-hot encoding to represent your classes). As for your test set, you should have an array of shape (10000, 784) for the images and one of shape (10000, 10) for the labels.
Once you have these big numpy arrays, you should define placeholders that will allow you to feed data to you network during training and evaluation.
images = tf.placeholder(tf.float32, shape=[None, 784])
labels = tf.placeholder(tf.int64, shape=[None, 10])
The None here means that you can feed a batch of any size, i.e. as many images as you want, as long as you numpy array is of shape (anything, 784).
The third step consists in defining your model as well as the loss function and the optimizer.
The fourth step consists in training your network by feeding it with random batches of data using the placeholders created above. As your network is training, you can periodically print its performance like the training loss/accuracy as well as the test loss/accuracy.
You can find a complete and very simple example here.
I'm new to TensorFlow and am getting a bit tripped up on the mechanics of reading data. I set up a TensorFlow graph on the mnist data, but I'd like to modify it so that I can run one program to train it + save the model out, and run another to load said graph, make predictions, and compute test accuracy.
Where I'm getting confused is how to bypass the original I/O system in the training graph and "inject" an image to predict or an (image, label) tuple of test data for accuracy testing. To read the training data, I'm using this code:
_, input_data = util.read_examples(
paths_to_files,
batch_size,
shuffle=shuffle,
num_epochs=None)
feature_map = {
'label': tf.FixedLenFeature(
shape=[], dtype=tf.int64, default_value=[-1]),
'image': tf.FixedLenFeature(
shape=[NUM_PIXELS * NUM_PIXELS], dtype=tf.int64),
}
example = tf.parse_example(input_data, features=feature_map)
I then feed example to a convolution layer, etc. and generate the output.
Now imagine that I train my graph with that code specifying the input, save out the graph and weights, and then restore the graph and weights in another script for prediction -- I'd like to take (say) 10 images and feed them to the graph to generate predictions. How do I "inject" those 10 images so that the predictions come out the other end?
I played around with feed dictionaries and placeholders, but I'm not sure if they're the right things for me to use... it seems like they rely on having data in memory, as opposed to reading from a queue of test data, for example.
Thanks!
A feed dictionary with placeholders would make sense if you wanted to perform a small number of inferences/evaluations (i.e. enough to fit in memory) - e.g. if you were serving a simple model or running small eval loops.
If you specifically want to infer or evaluate large batches then you should use the same approach you've used for training, but with a different path to your test/eval/live data. e.g.
_, eval_data = util.read_examples(
paths_to_files, # CHANGE THIS BIT
batch_size,
shuffle=shuffle,
num_epochs=None)
You can use this as a normal python variable and set up successive, dependent steps to use this as a provided variable. e.g.
def get_example(data):
return tf.parse_example(data, features=feature_map)
sess.run([get_example(path_to_your_data)])