I am new in using pytorch and in ML in general.
I am building encoder-decoder architecture, similar to autoencoder, but input and output are different numpy arrays.
I have two folders “input” and “output”, where input and output pairs are store in .npy format.
At the moment I called each pair of input and output with the same name.
So, for example, input/output files of pair 1 has same name, but stored in different folders.
I am not sure how to use how to use DatasetFolder, so my dataset will become X,Y which I will be able to split in train/test and feed to the model.
Thanks
In case someone will face same issue.
class En_De_coder_dataset(Dataset):
def __init__(self, in_dir, out_dir, transform=None):
self.in_dir = in_dir
self.out_dir = out_dir
self.transform = transform
def __len__(self):
return len(os.listdir(self.in_dir))
def __getitem__(self, index):
input = torch.from_numpy(np.load(self.in_dir + '/grid_aug_' + str(index) + '.npy')).float()
output = torch.from_numpy(np.load(self.out_dir + '/grid_aug_' + str(index) + '.npy')).float()
if self.transform:
input = self.transform(input)
output = self.transform(output)
return (input, output)
Related
I'm working on a model where I have two losses and 2 different outputs. One output takes y as an Image just like Autoencoder / U-Net architecture. The other output is simple Binary classification which takes as 0/1.
So what I'm trying to pull off is Siamese Based Unet. Basically Reconstruct the image based on mae loss and create a branch from Bottleneck layer so that it can predict whether 2 images are similar or not based on the Eunclidean distance.
Keras has a ImageDataGenerator where you can use class_mode='input' to generate a corresponding image as y label and also class_mode=binary to generate a 0/1 value given in the column. But how can I generate both things in the same generator. Problem is that the Siamese Branch will accept 2 inputs at the same time.
Have you checked the Sequence object
. It allows one to create custom data generators. The idea is to subclass Sequence and then override methods len, getitem. len should return number of batches in the sequence. The logic that returns the source and target pair is written inside getitem. It should return a batch of data. Incase of multi input models, you can write getitem such that its output data include a dictionary mapping to the input layer of your model(key=layername). Similarly for an output tensor. More information can be found in the link to the official doc I have added above. Best
Edit
This is the gist as per my understanding of your problem.:
class Dataset(Sequence):
def __init__(self, filenames, batchsize, shape):
self.filenames = filenames # List of filenames
self.batchsize = batchsize
self.shape = shape # Shape to which image should be
# resized
def __len__(self):
return len(self.filenames) // batchsize
def __getitem__(self, idx):
i = idx * self.batchsize
X_1 = np.zeros((self.batchsize, self.shape[0], self.shape[1], 3)
y = np.zeros((self.batchsize, --, --, ..., --)) # Depends on
# your target choice
filenames = self.filenames[i:i+self.batchsize]
for index, filename in enumerate(filenames):
image = cv2.imread(filename)
# Preprocess
image = your_preprocess(image)
X[index] = image
# You can include your pipeline for other
# Input also.
# Similarly obtain target values and load to y.
return {"layername": X_1, "layername": X_2}, {"layername": y}
I am new to pytorch .
I have big dataset consist of two txt files one for data and other for target data .
In training file each line is list of length 340, In target each line is list of 136.
I would like to ask how i can define my dataset so I can use Dataloader to load my data to train pytorch model?
I apricate you answers
Dataset from torch.utils.data is an abstract class representing a dataset. Your custom dataset should inherit Dataset and override the following methods:
__len__() so that len(dataset) returns the size of the dataset.
__getitem__() to support the indexing such that dataset[i] can be used to get ith sample
Eg of writing custom Dataset
i have written a general custom dataloader for you as your problem statement.
here data.txt has data and label.txt has labels.
import torch
from torch.utils.data import Dataset
class CustomDataset(Dataset):
def __init__(self):
with open('data.txt', 'r') as f:
self.data_info = f.readlines()
with open('label.txt', 'r') as f:
self.label_info = f.readlines()
def __getitem__(self, index):
single_data = self.data_info[index].rstrip('\n')
single_label = self.label_info[index].rstrip('\n')
return ( single_data , single_label)
def __len__(self):
return len(self.data_info)
# Testing
d = CustomDataset()
print(d[1]) # should output data along with label
This will be a basic for your case but have to do some changes that matches your case.
Note : you have to make required changes as per your dataset
I am trying to train some layers of a network whose inputs are an image and a scalar. Please see the figure below for a better understanding. .
As you can see only the dark yellow layers will be trained. So I need to freeze the rest, that is for later.
Purpose of this architecture is to map images (chest x-rays) to 14 kinds of diseases.
The images are stored in the following directory: /home/akde/chexnet/CheXNet-Keras/data/images
Names of the images are the image IDs.
A dataframe maps images (Images are named as the Image ID) to classes (diseases)
As you can see an image can be mapped to more than one class (disease).
Another dataframe maps the images (Image IDs) to the patient age. You can see it below.
Image is the first input and patient age is the second.
So in short, for each image id, I have an image and age value which are in 2 separate dataframes.
I can already test (gives absurd results since the network is not trained, but still proves that the network accepts the input and gives some result) it using the following code.
res3 = model3.predict( [test_image, a] )
where a is the scalar input while the test_image is the image input.
My training data is stored in multiple dataframes, having read that post, I deduce that flow_from_dataframe should be used.
The first thing I have done was to see this post which explains how to use mixed inputs. That gave me some background but since it does not use fit_generator (instead uses fit) it did not solve my problem.
Then I have read this post which does not use multiple inputs. Again no clue.
Afterwards, I have seen this post, which takes 2 images as input ( not one image one scalar). So again no help.
Even though I haven't found a solution to my problem I have written the following piece of code which will be the skeleton the solution.
datagen=ImageDataGenerator(rescale=1./255., validation_split=0.25)
train_generator = datagen.flow_from_dataframe(traindf,
directory="/home/akde/chexnet/CheXNet-Keras/data/images",
class_mode="other",
x_col="Image Index",
y_col=["Atelectasis", "Cardiomegaly", "Effusion", "Infiltration", "Mass",
"Nodule", "Pneumonia", "Pneumothorax", "Consolidation", "Edema",
"Emphysema", "Fibrosis", "Pleural_Thickening", "Hernia"],
color_mode="rgb",
batch_size=32,
target_size=(224, 224)
)
STEP_SIZE_TRAIN=train_generator.n//train_generator.batch_size
model3.compile(optimizers.rmsprop(lr=0.0001, decay=1e-6),loss="categorical_crossentropy",metrics=["accuracy"])
model3.fit_generator(generator=train_generator,
steps_per_epoch=STEP_SIZE_TRAIN,
epochs=10
)
I know this piece of code is far from the solution.
So how can I create a generator that uses 2 dataframes which are explained earlier (the one that maps images to the diseases and the other one which maps image IDs to age).
In other words, what is the way of writing a generator that takes an image and a scalar value as an input, considering the fact that both are represented in dataframes. How can I write the generator that written in bold below.
model3.fit_generator(**generator=train_generator**,
steps_per_epoch=STEP_SIZE_TRAIN,
epochs=10
)
For your purpose you need to create a custom generator.
I will recommand you to take a deep look at this link :
https://blog.ml6.eu/training-and-serving-ml-models-with-tf-keras-3d29b41e066c
And especially this code :
import ast
import numpy as np
import math
import os
import random
from tensorflow.keras.preprocessing.image import img_to_array as img_to_array
from tensorflow.keras.preprocessing.image import load_img as load_img
def load_image(image_path, size):
# data augmentation logic such as random rotations can be added here
return img_to_array(load_img(image_path, target_size=(size, size))) / 255.
class KagglePlanetSequence(tf.keras.utils.Sequence):
"""
Custom Sequence object to train a model on out-of-memory datasets.
"""
def __init__(self, df_path, data_path, im_size, batch_size, mode='train'):
"""
df_path: path to a .csv file that contains columns with image names and labels
data_path: path that contains the training images
im_size: image size
mode: when in training mode, data will be shuffled between epochs
"""
self.df = pd.read_csv(df_path)
self.im_size = im_size
self.batch_size = batch_size
self.mode = mode
# Take labels and a list of image locations in memory
self.wlabels = self.df['weather_labels'].apply(lambda x: ast.literal_eval(x)).tolist()
self.glabels = self.df['ground_labels'].apply(lambda x: ast.literal_eval(x)).tolist()
self.image_list = self.df['image_name'].apply(lambda x: os.path.join(data_path, x + '.jpg')).tolist()
def __len__(self):
return int(math.ceil(len(self.df) / float(self.batch_size)))
def on_epoch_end(self):
# Shuffles indexes after each epoch
self.indexes = range(len(self.image_list))
if self.mode == 'train':
self.indexes = random.sample(self.indexes, k=len(self.indexes))
def get_batch_labels(self, idx):
# Fetch a batch of labels
return [self.wlabels[idx * self.batch_size: (idx + 1) * self.batch_size],
self.glabels[idx * self.batch_size: (idx + 1) * self.batch_size]]
def get_batch_features(self, idx):
# Fetch a batch of images
batch_images = self.image_list[idx * self.batch_size: (1 + idx) * self.batch_size]
return np.array([load_image(im, self.im_size) for im in batch_images])
def __getitem__(self, idx):
batch_x = self.get_batch_features(idx)
batch_y = self.get_batch_labels(idx)
return batch_x, batch_y
Hope this will help to find your solution !
I want to train a model in tf.keras of Tensorflow 2.0 with data that is bigger than my ram, but the tutorials only show examples with predefined datasets.
I followed this tutorial:
Load Images with tf.data, I could not make this work for data on numpy arrays or tfrecords.
This is an example with array being transformed into tensorflow datasets. What I want is to make this work for multiple numpy array files or multiple tfrecords files.
train_dataset = tf.data.Dataset.from_tensor_slices((x_train, y_train))
# Shuffle and slice the dataset.
train_dataset = train_dataset.shuffle(buffer_size=1024).batch(64)
# Since the dataset already takes care of batching,
# we don't pass a `batch_size` argument.
model.fit(train_dataset, epochs=3)
If you have tfrecords files:
path = ['file1.tfrecords', 'file2.tfrecords', ..., 'fileN.tfrecords']
dataset = tf.data.Dataset.list_files(path, shuffle=True).repeat()
dataset = dataset.interleave(lambda filename: tf.data.TFRecordDataset(filename), cycle_length=len(path))
dataset = dataset.map(parse_function).batch()
parse_function handles decoding and any kind of augmentation.
In case with numpy arrays, you can construct dataset either from a list of filenames or from list of arrays. Labels are just a list. Or they could be taken from file while parsing single example.
path = #list of numpy arrays
or
path = os.listdir(path_to files)
dataset = tf.data.Dataset.from_tensor_slices((path, labels))
dataset = dataset.map(parse_function).batch()
parse_function handles decoding:
def parse_function(filename, label): #Both filename and label will be passed if you provided both to from_tensor_slices
f = tf.read_file(filename)
image = tf.image.decode_image(f))
image = tf.reshape(image, [H, W, C])
label = label #or it could be extracted from, for example, filename, or from file itself
#do any augmentations here
return image, label
To decode .npy files, the best way is to use reshape without read_file or decode_raw, but first load numpys with np.load:
paths = [np.load(i) for i in ["x1.npy", "x2.npy"]]
image = tf.reshape(filename, [2])
or try using decode_raw
f = tf.io.read_file(filename)
image = tf.io.decode_raw(f, tf.float32)
Then just pass batched dataset to model.fit(dataset). TensorFlow 2.0 allows simple iteration over dataset. No need to use iterator. Even in later versions of 1.x API you could just pass dataset to .fit method
for example in dataset:
func(example)
I'm currently working on a classification problem on text input basis and my main question is the following:
Am I correct in assuming that I can parse my complete sentence as one string to the vocabulary column or do I need to split the sentence in its words - like a list of strings?
My data looks something like this:
A B text
1 .. .. My first example text
2 .. .. My second example text
(Beside my text input feature there are also some other categorical information - but they are not relevant in this context)
And my code looks basically like this:
// data import and data preparation
categorical_voc = tf.feature_column.categorical_column_with_vocabulary_list(key="text", vocabulary_list=vocabulary_list)
embedding_initializer = tf.random_uniform_initializer(-1.0, 1.0)
embed_column_dim = math.ceil(len(vocabulary_list) ** 0.25)
embed_column = tf.feature_column.embedding_column(
categorical_column=categorical_voc,
dimension=embed_column_dim,
initializer=embedding_initializer,
trainable=True)
estimator = tf.estimator.DNNClassifier(
optimizer=optimizer,
feature_columns=feature_columns,
hidden_units=hidden_units,
activation_fn=activation_fn,
dropout=dropout,
n_classes=target_size,
label_vocabulary=target_list,
config=config)
train_input_fn = tf.estimator.inputs.pandas_input_fn(
x=train_data,
y=train_target,
batch_size=batch_size,
num_epochs=1,
shuffle=True)
estimator.train(input_fn=train_input_fn)
Thanks for your help :)
Edit 1:
For the ones who need the custom input function.
def input_fn(features, labels, batch_size):
if labels is None:
dataset = tf.data.Dataset.from_tensor_slices(features)
else:
dataset = tf.data.Dataset.from_tensor_slices((features, labels))
# Shuffle, repeat, and batch the examples.
dataset = dataset.shuffle(100).repeat().batch(batch_size)
return dataset
def train_input_fn():
return input_fn(features=_train_data,
labels=_train_target,
batch_size=train_batch_size)
estimator.train(input_fn=lambda: train_input_fn(), steps=total_training_steps, hooks=train_hooks)
For those who had the same problem figuring out how to handle a sentence within a vocabulary column ..
My conclusion so far is that I have to feed the vocabulary column with an array of strings. The only issue here is that the pandas_input_fn() does not support a series of lists. Thats why I went back to my custom input function!