I've skimmed over all tensorflow tutorials in which all data sets were loaded in RAM due to their small size. However, my own data (~30 Gb of images) can not be loaded in memory, therefore I'm looking for effective ways of reading images for further processing. Could anyone provide me examples of how can I do that?
P.S. I have two files train_images and validation_images that contain:
<path/to/img> <label>
This is what you're looking for: Tensorflow read images with labels
The exact code snippet is like this:
def read_labeled_image_list(image_list_file):
"""Reads a .txt file containing pathes and labeles
Args:
image_list_file: a .txt file with one /path/to/image per line
label: optionally, if set label will be pasted after each line
Returns:
List with all filenames in file image_list_file
"""
f = open(image_list_file, 'r')
filenames = []
labels = []
for line in f:
filename, label = line[:-1].split(' ')
filenames.append(filename)
labels.append(int(label))
return filenames, labels
def read_images_from_disk(input_queue):
"""Consumes a single filename and label as a ' '-delimited string.
Args:
filename_and_label_tensor: A scalar string tensor.
Returns:
Two tensors: the decoded image, and the string label.
"""
label = input_queue[1]
file_contents = tf.read_file(input_queue[0])
example = tf.image.decode_png(file_contents, channels=3)
return example, label
# Reads pfathes of images together with their labels
image_list, label_list = read_labeled_image_list(filename)
images = ops.convert_to_tensor(image_list, dtype=dtypes.string)
labels = ops.convert_to_tensor(label_list, dtype=dtypes.int32)
# Makes an input queue
input_queue = tf.train.slice_input_producer([images, labels],
num_epochs=num_epochs,
shuffle=True)
image, label = read_images_from_disk(input_queue, num_labels=num_labels)
# Optional Preprocessing or Data Augmentation
# tf.image implements most of the standard image augmentation
image = preprocess_image(image)
label = preprocess_label(label)
# Optional Image and Label Batching
image_batch, label_batch = tf.train.batch([image, label],
batch_size=batch_size)
The recommended way is to put it into sharded protobuf files, where encoded jpeg and label(s) are features of a tf.Example. build_image_data.py in the tensorflow/models repository shows how to create such a database of image/label pairs from a directory structure, you'll need to adapt it a bit to your case (it's straightforward). Then for training time you can look at image_processing.py where it shows how to go from the tf.Example proto to image/label tensors (extract decoded jpg and label from the Example record, decode jpg, resize, apply augmentations as needed, then enqueue).
Tutorial on udacity has stochastic method explained in https://github.com/tensorflow/tensorflow/blob/master/tensorflow/examples/udacity/4_convolutions.ipynb, you can use the same with one change, instead of saving all images in single pickle file, save them in chunks of batch_size that you are using. That way at a time, you can load only as much data as used in the one batch.
Related
It might be a very simple question, but as a newbie in keras and machine learning, I am unable to solve this issue. This is a two-class classification problem. My code is written (Source:Kaggle) in keras with a Tensorflow backend.
I have a directory that contains two folders named "cat" and "dog". Each folder has multiple images of size 224 x 224 pixels. The total image size is more than 32 GB. The label will be based on the folder names, i.e., if folder name contains "cat", the label will be "0" else "1".
Code Snippet (Source:Kaggle):
def get_images(directory):
Images = []
Labels = []
label = 0
for labels in os.listdir(directory): #Main Directory where each class label is present as folder name.
if labels == 'cat': #Folder contain 'cat' Images get the '0' class label.
label = 0
elif labels == 'dog':
label = 1
for image_file in os.listdir(directory+labels): #Extracting the file name of the image from Class Label folder
image = cv2.imread(directory+labels+r'/'+image_file) #Reading the image (OpenCV)
image = cv2.resize(image,(224,224)) #Resize the image, Some images are different sizes. (Resizing is very Important)
Images.append(image)
Labels.append(label)
return shuffle(Images,Labels,random_state=817328462) #Shuffle the dataset you just prepared. 817328462
def get_classlabel(class_code):
labels = {0:'cat', 1:'dog'}
return labels[class_code]
Images, Labels = get_images('./path_of_data_set') #Extract the training images from the folders.
Images = np.array(Images)
Labels = np.array(Labels)
def sequence():
model = Models.Sequential()
...
model=sequence();
model.summary()
# Train the model with the new callback
model.fit(Images, Labels, batch_size=32, epochs=100, validation_split=0.10, verbose=1)
If the number of .png images is small, then my code is running perfectly. The problem arises when I am using 32GB image data. Then I am getting a memory issue. I check lots of post in this regard and found lots of solutions, but I am unable to implement them in this code.
Can you please tell me how can I feed the data into the model, so that it should not show memory issues?
Check here. Details are available. You may need to add few more lines. https://www.tensorflow.org/api_docs/python/tf/keras/preprocessing/image/ImageDataGenerator
I want to train a model in tf.keras of Tensorflow 2.0 with data that is bigger than my ram, but the tutorials only show examples with predefined datasets.
I followed this tutorial:
Load Images with tf.data, I could not make this work for data on numpy arrays or tfrecords.
This is an example with array being transformed into tensorflow datasets. What I want is to make this work for multiple numpy array files or multiple tfrecords files.
train_dataset = tf.data.Dataset.from_tensor_slices((x_train, y_train))
# Shuffle and slice the dataset.
train_dataset = train_dataset.shuffle(buffer_size=1024).batch(64)
# Since the dataset already takes care of batching,
# we don't pass a `batch_size` argument.
model.fit(train_dataset, epochs=3)
If you have tfrecords files:
path = ['file1.tfrecords', 'file2.tfrecords', ..., 'fileN.tfrecords']
dataset = tf.data.Dataset.list_files(path, shuffle=True).repeat()
dataset = dataset.interleave(lambda filename: tf.data.TFRecordDataset(filename), cycle_length=len(path))
dataset = dataset.map(parse_function).batch()
parse_function handles decoding and any kind of augmentation.
In case with numpy arrays, you can construct dataset either from a list of filenames or from list of arrays. Labels are just a list. Or they could be taken from file while parsing single example.
path = #list of numpy arrays
or
path = os.listdir(path_to files)
dataset = tf.data.Dataset.from_tensor_slices((path, labels))
dataset = dataset.map(parse_function).batch()
parse_function handles decoding:
def parse_function(filename, label): #Both filename and label will be passed if you provided both to from_tensor_slices
f = tf.read_file(filename)
image = tf.image.decode_image(f))
image = tf.reshape(image, [H, W, C])
label = label #or it could be extracted from, for example, filename, or from file itself
#do any augmentations here
return image, label
To decode .npy files, the best way is to use reshape without read_file or decode_raw, but first load numpys with np.load:
paths = [np.load(i) for i in ["x1.npy", "x2.npy"]]
image = tf.reshape(filename, [2])
or try using decode_raw
f = tf.io.read_file(filename)
image = tf.io.decode_raw(f, tf.float32)
Then just pass batched dataset to model.fit(dataset). TensorFlow 2.0 allows simple iteration over dataset. No need to use iterator. Even in later versions of 1.x API you could just pass dataset to .fit method
for example in dataset:
func(example)
Almost all examples on github or other blogs uses mnist dataset for demo. When I am trying to use same deep NN for my images data I encounter following problem.
They use:
batch_x, batch_y = mnist.train.next_batch(batch_size)
# Run optimization op (backprop)
sess.run(train_op, feed_dict={X: trainimg, Y: trainlabel, keep_prob: 0.8})
next_batch method to feed data in batches.
My question is:
Do we have any similar method to feed data in batches?
You should have a look at tf.contrib.data.Dataset. You can create an input pipeline: define the source, apply a transforation, and batch it. See the programmer's guide for importing data.
From the documentation:
The Dataset API enables you to build complex input pipelines from simple, reusable pieces. For example, the pipeline for an image model might aggregate data from files in a distributed file system, apply random perturbations to each image, and merge randomly selected images into a batch for training
EDIT:
I guess what you have is an array of pictures (filenames). Here is an example from the programmer's guide.
Depending on your input files, the transformation part will change. Here is the extract for consuming an array of picture files.
# Reads an image from a file, decodes it into a dense tensor, and resizes it
# to a fixed shape.
def _parse_function(filename, label):
image_string = tf.read_file(filename)
image_decoded = tf.image.decode_image(image_string)
image_resized = tf.image.resize_images(image_decoded, [28, 28])
return image_resized, label
# A vector of filenames.
filenames = tf.constant(["/var/data/image1.jpg", "/var/data/image2.jpg", ...])
# labels[i] is the label for the image in filenames[i].
labels = tf.constant([0, 37, ...])
dataset = tf.contrib.data.Dataset.from_tensor_slices((filenames, labels))
dataset = dataset.map(_parse_function)
# Now you have a dataset of (image, label). Basically kind of a list with
# all your pictures encoded along with a label.
# Batch it.
dataset = dataset.batch(32)
# Create an iterator.
iterator = dataset.make_one_shot_iterator()
# Retrieve the next element.
image_batch, label_batch = iterator.get_next()
You could also shuffle your images.
Now you can use your image_batch and label_batch as placeholders in your model definition.
I'm new to tensorflow, but i already followed and executed the tutorials they promote and many others all over the web.
I made a little convolutional neural network over the MNIST images. Nothing special, but i would like to test on my own images.
Now my problem comes: I created several folders; the name of each folder is the class (label) the images inside belong.
The images have different shapes; i mean they have no fixed size.
How can i load them for using with Tensorflow?
I followed many tutorials and answers both here on StackOverflow and on others Q/A sites. But still, i did not figure out how to do this.
The tf.data API (tensorflow 1.4 onwards) is great for things like this. The pipeline will looks something like the following:
Create an initial tf.data.Dataset object that iterates over all examples
(if training) shuffle/repeat the dataset;
map it through some function that makes all images the same size;
batch;
(optionall) prefetch to tell your program to collect the preprocess subsequent batches of data while the network is processing the current batch; and
and get inputs.
There are a number of ways of creating your initial dataset (see here for a more in depth answer)
TFRecords with Tensorflow Datasets
Supporting tensorflow version 1.12 onwards, Tensorflow datasets provides a relatively straight-forward API for creating tfrecord datasets, and also handles data downloading, sharding, statistics generation and other functionality automatically.
See e.g. this image classification dataset implementation. There's a lot of bookeeping stuff in there (download urls, citations etc), but the technical part boils down to specifying features and writing a _generate_examples function
features = tfds.features.FeaturesDict({
"image": tfds.features.Image(shape=(_TILES_SIZE,) * 2 + (3,)),
"label": tfds.features.ClassLabel(
names=_CLASS_NAMES),
"filename": tfds.features.Text(),
})
...
def _generate_examples(self, root_dir):
root_dir = os.path.join(root_dir, _TILES_SUBDIR)
for i, class_name in enumerate(_CLASS_NAMES):
class_dir = os.path.join(root_dir, _class_subdir(i, class_name))
fns = tf.io.gfile.listdir(class_dir)
for fn in sorted(fns):
image = _load_tif(os.path.join(class_dir, fn))
yield {
"image": image,
"label": class_name,
"filename": fn,
}
You can also generate the tfrecords using lower level operations.
Load images via tf.data.Dataset.map and tf.py_func(tion)
Alternatively you can load the image files from filenames inside tf.data.Dataset.map as below.
image_paths, labels = load_base_data(...)
epoch_size = len(image_paths)
image_paths = tf.convert_to_tensor(image_paths, dtype=tf.string)
labels = tf.convert_to_tensor(labels)
dataset = tf.data.Dataset.from_tensor_slices((image_paths, labels))
if mode == 'train':
dataset = dataset.repeat().shuffle(epoch_size)
def map_fn(path, label):
# path/label represent values for a single example
image = tf.image.decode_jpeg(tf.read_file(path))
# some mapping to constant size - be careful with distorting aspec ratios
image = tf.image.resize_images(out_shape)
# color normalization - just an example
image = tf.to_float(image) * (2. / 255) - 1
return image, label
# num_parallel_calls > 1 induces intra-batch shuffling
dataset = dataset.map(map_fn, num_parallel_calls=8)
dataset = dataset.batch(batch_size)
# try one of the following
dataset = dataset.prefetch(1)
# dataset = dataset.apply(
# tf.contrib.data.prefetch_to_device('/gpu:0'))
images, labels = dataset.make_one_shot_iterator().get_next()
I've never worked in a distributed environment, but I've never noticed a performance hit from using this approach over tfrecords. If you need more custom loading functions, also check out tf.py_func.
More general information here, and notes on performance here
Sample input pipeline script to load images and labels from directory. You could do preprocessing(resizing images etc.,) after this.
import tensorflow as tf
filename_queue = tf.train.string_input_producer(
tf.train.match_filenames_once("/home/xxx/Desktop/stackoverflow/images/*/*.png"))
image_reader = tf.WholeFileReader()
key, image_file = image_reader.read(filename_queue)
S = tf.string_split([key],'/')
length = tf.cast(S.dense_shape[1],tf.int32)
# adjust constant value corresponding to your paths if you face issues. It should work for above format.
label = S.values[length-tf.constant(2,dtype=tf.int32)]
label = tf.string_to_number(label,out_type=tf.int32)
image = tf.image.decode_png(image_file)
# Start a new session to show example output.
with tf.Session() as sess:
# Required to get the filename matching to run.
tf.initialize_all_variables().run()
# Coordinate the loading of image files.
coord = tf.train.Coordinator()
threads = tf.train.start_queue_runners(coord=coord)
for i in xrange(6):
# Get an image tensor and print its value.
key_val,label_val,image_tensor = sess.run([key,label,image])
print(image_tensor.shape)
print(key_val)
print(label_val)
# Finish off the filename queue coordinator.
coord.request_stop()
coord.join(threads)
File Directory
./images/1/1.png
./images/1/2.png
./images/3/1.png
./images/3/2.png
./images/2/1.png
./images/2/2.png
Output:
(881, 2079, 3)
/home/xxxx/Desktop/stackoverflow/images/3/1.png
3
(155, 2552, 3)
/home/xxxx/Desktop/stackoverflow/images/2/1.png
2
(562, 1978, 3)
/home/xxxx/Desktop/stackoverflow/images/3/2.png
3
(291, 2558, 3)
/home/xxxx/Desktop/stackoverflow/images/1/1.png
1
(157, 2554, 3)
/home/xxxx/Desktop/stackoverflow/images/1/2.png
1
(866, 936, 3)
/home/xxxx/Desktop/stackoverflow/images/2/2.png
2
For loading images of equal size just use this:
tf.keras.preprocessing.image_dataset_from_directory(dir)
docs: https://www.tensorflow.org/api_docs/python/tf/keras/preprocessing/image_dataset_from_directory
To load images with different shapes , tf provides a pipeline implementation (ImageGenerator):
from tensorflow.keras.preprocessing.image import ImageDataGenerator
TARGET_SHAPE = (500,500)
BATCH_SIZE = 32
train_dir = "train_images_directory" #ex: images/train/
test_dir = "train_images_directory" #ex: images/test/
train_images_generator = ImageDataGenerator(rescale=1.0/255,)
train_data_gen =
image_train_gen.flow_from_directory(batch_size=BATCH_SIZE,
directory=train_dir,
target_size=TARGET_SHAPE,
shuffle=True,
class_mode='sparse')
# do the same for validation and test dataset
# 1- image_generator 2- load images from directory with target shape
I'm trying to use slim interface to create and train a convolutional neural network, but I couldn't figure out how to specify the batch size for training.
During the training my net crashes because of "Out of Memory" on my graphic card.
So I think that should be a way to handle this condition...
Do I have to split the data and the labels in batches and then explicitly loop or the slim.learning.train is taking care of it?
In the code I paste train_data are all the data in my training set (numpy array)..and the model definition is not included here
I had a quick loop to the sources but no luck so far...
g = tf.Graph()
with g.as_default():
# Set up the data loading:
images = train_data
labels = tf.contrib.layers.one_hot_encoding(labels=train_labels, num_classes=num_classes)
# Define the model:
predictions = model7_2(images, num_classes, is_training=True)
# Specify the loss function:
slim.losses.softmax_cross_entropy(predictions, labels)
total_loss = slim.losses.get_total_loss()
tf.scalar_summary('losses/total loss', total_loss)
# Specify the optimization scheme:
optimizer = tf.train.GradientDescentOptimizer(learning_rate=.001)
train_tensor = slim.learning.create_train_op(total_loss, optimizer)
slim.learning.train(train_tensor,
train_log_dir,
number_of_steps=1000,
save_summaries_secs=300,
save_interval_secs=600)
Any hints suggestions?
Edit:
I re-read the documentation...and I found this example
image, label = MyPascalVocDataLoader(...)
images, labels = tf.train.batch([image, label], batch_size=32)
But It's not clear at all how to feed image and label to be passed to tf.train.batch... as MyPascalVocDataLoader function is not specified...
In my case my data set are loaded from a sqlite database and I have training data and labels as numpy array....still confused.
Of course I tried to pass my numpy arrays (converted to constant tensor) to the tf.train.batch like this
image = tf.constant(train_data)
label = tf.contrib.layers.one_hot_encoding(labels=train_labels, num_classes=num_classes)
images, labels = tf.train.batch([image, label], batch_size=32)
But seems not the right path to follow... it seems that the train.batch wants only one element from my data set...(how to pass this? it does not make sense to me to pass only train_data[0] and train_labels[0])
Here you can create the tfrecords which is the special type of binary file format used by the tensorflow. As you mentioned you have the training images and the labels, you can easily create the TFrecords for training and validation.
After creating the TFrecords, all you need to right is decode the images from the encoded TFrecords and give it to your model input. There you can select the batch size and all.