I am trying to convert images to numpy tensor that is 4 dimensional and I want to save this into a file, preferrably a csv file. I already have the image array in the dimension I want them but now the problem is saving to a file. Please how do I go about doing this.
Numpy has a lot of options for IO of array data:
If binary format is Ok, you can use np.save to save the 4D tensor in a binary (".npy") format. The file can be read again with np.load. This is a very convenient way to save numpy data, and it works for numeric arrays of any number of dimensions.
np.savetxt can write a 1D or 2D array in CSV-like text format. You could use np.reshape to flatten your tensor down to 1D or 2D and then use np.savetxt. The downside is the file doesn't keep track of the full 4D shape, so you'll need to track that separately in some way.
If storing in text representation is important, a better option may be to convert the tensor to string with np.array2string, then write the string to file. This works even for arrays with more than 2 dimensions.
The .tofile method simply dumps the element data as a raw binary file. No shape or another other metadata is preserved, but the binary file is easy to read into other programs.
Related
I have the 3d array np.zeros((24, 24, 100)), fill it with some '1's,
and can visualize it (in 3d) using pyplot:
open3d is more suitable to view this in 3d, and update it realtime, so I want to visualize this with open3d. But I don't understand how to visualize my binary voxelgrid using open3d.
This example is what I want (the voxel bunny); How can I convert my binary 3d array to the open3d voxelgrid format?
Converting to pointcloud is also an option, but there applies the same question.
I am quite new to TensorFlow, and have never worked with TFRecords before.
I have downloaded a dataset of images from online and the download format was TFRecord.
This is the file structure in the downloaded dataset:
1.
2.
E.g. inside "test"
What I want to do is load in the training, validation and testing data into TensorFlow in a similar way to what happens when you load a built-in dataset, e.g. you might load in the MNIST dataset like this, and get arrays containing pixel data and arrays containing the corresponding image labels.
(train_images, train_labels), (test_images, test_labels) = tf.keras.datasets.mnist.load_data()
However, I have no idea how to do so.
I know that I can use dataset = tf.data.TFRecordDataset(filename) somehow to open the dataset, but would this act on the entire dataset folder, one of the subfolders, or the actual files? If it is the actual files, would it be on the .TFRecord file? How do I use/what do I do with the .PBTXT file which contains a label map?
And even after opening the dataset, how can I extract the data and create the necessary arrays which I can then feed into a TensorFlow model?
It's mostly archaeology, and plus a few tricks.
First, I'd read the README.dataset and README.roboflow files. Can you show us what's in them?
Second, pbtxt are text formatted so we may be able to understand what that file is if you just open it with a text editor. Can you show us what's in that.
The think to remember about a TFRecord file is that it's nothing but a sequence of binary records. tf.data.TFRecordDataset('balls.tfrecord') will give you a dataset that yields those records in order.
Number 3. is the hard part, because here you'll have binary blobs of data, but we don't have any clues yet about how they're encoded.
It's common for TFRecord filed to contian serialized tf.train.Example.
So it would be worth a shot to try and decode it as a tf.train.Example to see if that tells us what's inside.
ref
for record in tf.data.TFRecordDataset('balls.tfrecord'):
break
example = tf.train.Example()
example.ParseFromString(record.numpy())
print(example)
The Example object is just a representation of a dict. If you get something other than en error there look for the dict keys and see if you can make sense out of them.
Then to make a dataset that decodes them you'll want something like:
def decode(record):
return tf.train.parse_example(record, {key:tf.io.RaggedFeature(dtype) for key, dtype in key_dtypes.items()})
ds = ds.map(decode)
I am evaluating a couple of object detection models on a data set and was planning on performing pre-processing on the data using standardization to zero mean and unit variance. But I don't know how to store the images when they have been pre-processed. Currently they are in jpg format, but what format can be used after I have pre-processed them? Some of the models I evaluate are yolov4, yolov5, and SSD.
If i instead scaled the pixel values from 0-255 to 0-1, what image format could I then use?
Also, if I train the object detector on pre-processed images and then want to apply it to a video, I assume I need to somehow pre-process the video to get decent results. How would I go about doing that?
I have calculated mean and std on my data set using the python module cv2. I read the images using imread which returns a numpy array. Then I subtract mean and divide with std. This gives me a numpy array with both negative and positive floating point values. But when I try to save this numpy array as an image using the function imwrite(filename, array), it doesn't work. I assume because the numpy array isn't allowed to contain negative values.
I receive an image from network which I convert to a numpy array using fromfile function. Now I want to pass this unit8 type array to decode_image of tensorflow v1. How can I achieve that? I do not want to have a file on disk and do it all in-memory.
It turned out that the answer is pretty simple. Supposed nd is the 1-D array returned from fromfile then you can call function tostring and pass that to decode_image of tensorflow.
I write this code:
matrix = C.softmax(model).eval(data).
But matrix.shape, matrix.size give me errors. So I'm wondering, how can I output the shape of CNTK variable?
First note that eval() will not give you a CNTK variable, it will give you a numpy array (or a list of numpy arrays, see the next point).
Second, depending on the nature of the model it is possible that what comes out of eval() is not a numpy array but a list. The reason for this is that if the output is a sequence then CNTK cannot guarrantee that all sequences will be of the same length and it therefore returns a list of arrays, each array being one sequence.
Finally, if you truly have a CNTK variable, you can get the dimensions with .shape