I randomly browsed some images in cifar100, and found many images like this:
.
Anything went wrong? Or cifar100 indeed consist of such images?
I think you did not load in the images correctly. Take a look at loading an image from cifar-10 dataset to see that others also have those problems. The correct way to reshape one of those cifar images is as follows:
single_img_reshaped = np.transpose(np.reshape(single_img,(3, 32,32)), (1,2,0))
Related
I am quite new to TensorFlow, and have never worked with TFRecords before.
I have downloaded a dataset of images from online and the download format was TFRecord.
This is the file structure in the downloaded dataset:
1.
2.
E.g. inside "test"
What I want to do is load in the training, validation and testing data into TensorFlow in a similar way to what happens when you load a built-in dataset, e.g. you might load in the MNIST dataset like this, and get arrays containing pixel data and arrays containing the corresponding image labels.
(train_images, train_labels), (test_images, test_labels) = tf.keras.datasets.mnist.load_data()
However, I have no idea how to do so.
I know that I can use dataset = tf.data.TFRecordDataset(filename) somehow to open the dataset, but would this act on the entire dataset folder, one of the subfolders, or the actual files? If it is the actual files, would it be on the .TFRecord file? How do I use/what do I do with the .PBTXT file which contains a label map?
And even after opening the dataset, how can I extract the data and create the necessary arrays which I can then feed into a TensorFlow model?
It's mostly archaeology, and plus a few tricks.
First, I'd read the README.dataset and README.roboflow files. Can you show us what's in them?
Second, pbtxt are text formatted so we may be able to understand what that file is if you just open it with a text editor. Can you show us what's in that.
The think to remember about a TFRecord file is that it's nothing but a sequence of binary records. tf.data.TFRecordDataset('balls.tfrecord') will give you a dataset that yields those records in order.
Number 3. is the hard part, because here you'll have binary blobs of data, but we don't have any clues yet about how they're encoded.
It's common for TFRecord filed to contian serialized tf.train.Example.
So it would be worth a shot to try and decode it as a tf.train.Example to see if that tells us what's inside.
ref
for record in tf.data.TFRecordDataset('balls.tfrecord'):
break
example = tf.train.Example()
example.ParseFromString(record.numpy())
print(example)
The Example object is just a representation of a dict. If you get something other than en error there look for the dict keys and see if you can make sense out of them.
Then to make a dataset that decodes them you'll want something like:
def decode(record):
return tf.train.parse_example(record, {key:tf.io.RaggedFeature(dtype) for key, dtype in key_dtypes.items()})
ds = ds.map(decode)
I have some medical images of nii.gz format which are of different shapes. I want to resize all to the same shape inorder to feed to a deep learnig model, I tried using resample_img() of nibabel, but it destroys my images. I want to do some other function just to resize it to a particular shape, say (512,512,129).
Someone please help me in this regard. I am stuck in this step for quite a good number of days.
Maybe you can use this:
https://scikit-image.org/docs/dev/api/skimage.transform.html
I saw it in one of the papers. Here is the example in function ScaleToFixed:
https://github.com/sacmehta/3D-ESPNet/blob/master/Transforms.py
Here is how I did it. I have the volume of shape 320x320x130 (black and white so no rgb dimension). I want to make it twice as small. This worked for me:
import skimage.transform as skTrans
im = nib.load(file_path).get_fdata()
result1 = skTrans.resize(im, (160,160,130), order=1, preserve_range=True)
You can use TorchIO:
import torchio as tio
image = tio.ScalarImage('path/to/image.nii.gz')
transform = tio.CropOrPad((512,512,129))
output = transform(image)
If you would like to keep the original field of view, you could use the Resample transform instead.
Disclaimer: I'm the main developer of TorchIO.
I am trying to train a neural network to detect steganographic images using Tensorflow and Nvidia Digits. I loaded a data set which has two sub directories - Cover Images and Steg Images. I think the network has to process the cover/stegano image pairs together to learn which are the covers and which are steganographic images. Am I correct?
How does batch size work? If I give 1 does it take one image from both sub directories and process them? or do I have to input batch number as 2 for that?
How does shuffling data on each epoch work? does it shuffle both sub directories equally? as an example will 1.jpg be the third photo on both folders or will it be different on them both?
I think the network has to process the cover/stegano image pairs
together to learn which are the covers and which are steganographic
images. Am I correct?
I am not familiar with object detection (right?) in Nvidia Digits, so please check out their tutorials for more information.
You need to think about the kind of labeling the training data first. Usually in the examples I see only use one training folder and one validation folder (each: images and labels) - Digits divides your dataset, e.g. into 90 % training and 10 % validation images.
How does batch size work? If I give 1 does it take one image from both
sub directories and process them? or do I have to input batch number
as 2 for that?
With batch number you tell Digits how many images you use per iteration. It's used for dataset division (memory for calculations is limited; you can't fit the whole dataset into one iteration). In one epoch the whole dataset is processed.
As written above, one image at a time, as far as I know.
How does shuffling data on each epoch work? does it shuffle both sub
directories equally? as an example will 1.jpg be the third photo on
both folders or will it be different on them both?
The data should be shuffled automatically.
I am trying to do a Deep Learning project by using Tensorflow.
Each of my data sets contains 2 files( PNGimage file + TXTvectors file ), where are put in different folders as follow:
./data/image/ #Folders contains different size of images
./data/vector/ #Folders contains vectors of corresponding image
#For example: apple.png + apple.txt
The example content of vector shows as follow:
10.0,2.5,5,13
And since image size are different, the resize and some transformation apply on vectors are required. It is important to make sure that I can do these processing during Tensorflow is running. Is there any good way to manage this kind of datasets?
I referred to a lot of basic tutorial however most of them are not so many details about arrange customized data input and output. Please give me some advice!
I recommend you to take a look at TFRecords and queues. Basically the idea is the following: you resize all your images to the same format and store them together with your txt vectors in one TFRecord file. This is done separately before you run your model.
When you create your model you create a queue which reads data from the TFRecord file and feeds it to your model.
I am reading tutorials about TensorFlow visualization and found out Tensorboard. I would like to know how can I visualize for example, Iris dataset taken from UCI Machine Learning repository. I have been able to run a specified port on localhost which shows TensorBoard, but do not know how to visualize a locally taken dataset there. I searched on google but really could not find how to do. Could you help me, please ?
If i understand you correctly then you wish to use tf.summary.image. The documentation is here: https://www.tensorflow.org/api_docs/python/tf/summary/image
Some example usage from my code is:
x_pl=tf.placeholder(tf.float32, [None,height,width,channels], name="ImageIn")
tf.summary.image('input', x_pl, 10)
x_pl is where I feed my image data in.
In my cummary declaration I say that I want to create a summary called 'input' and to take 10 images from x_pl.
Read the summary-writer example/tutorial here: https://www.tensorflow.org/get_started/summaries_and_tensorboard
You will need to merge your summaries:
merged = tf.summary.merge_all()
You will need to declare a summary-writer a bit like this:
train_writer = tf.summary.FileWriter(FLAGS.summaries_dir + '/train',sess.graph)
See the above tutorial/example to understand how Tensorboard works. Not that you will want to replace the summaries with image summaries for your purposes.