I am currently using tensorboardX to visualize input images while training a ResNet image classifier. Is there a way to add the image title along with the added image? I would like to have the image name (as stored in the dataset) displayed below the image in the tensorboard display.
So far I have tried passing a comment parameter into my tensorboard writer, which does not seem to do the job.
Currently, the relevant lines of my code are:
pretrain_train_writer = SummaryWriter('log/pretrain_train')
img_grid = vutils.make_grid(inputs[tp_idx_0], normalize=True, scale_each=True, nrow=8)
pretrain_val_writer.add_image('true_positive_class_0', img_grid, global_step=epoch, comment = img_path)
there is no way of doing it directly with tensorboard, instead you have to create images with titles using matplotlib and then supply them to tensorboard. Here is a sample code from the tensorboard documentation:
def plot_to_image(figure):
"""Converts the matplotlib plot specified by 'figure' to a PNG image and
returns it. The supplied figure is closed and inaccessible after this call."""
# Save the plot to a PNG in memory.
buf = io.BytesIO()
plt.savefig(buf, format='png')
# Closing the figure prevents it from being displayed directly inside
# the notebook.
plt.close(figure)
buf.seek(0)
# Convert PNG buffer to TF image
image = tf.image.decode_png(buf.getvalue(), channels=4)
# Add the batch dimension
image = tf.expand_dims(image, 0)
return image
def image_grid():
"""Return a 5x5 grid of the MNIST images as a matplotlib figure."""
# Create a figure to contain the plot.
figure = plt.figure(figsize=(10,10))
for i in range(25):
# Start next subplot.
plt.subplot(5, 5, i + 1, title=class_names[train_labels[i]])
plt.xticks([])
plt.yticks([])
plt.grid(False)
plt.imshow(train_images[i], cmap=plt.cm.binary)
return figure
# Prepare the plot
figure = image_grid()
# Convert to image and log
with file_writer.as_default():
tf.summary.image("Training data", plot_to_image(figure), step=0)
here is the link to the doc: https://www.tensorflow.org/tensorboard/image_summaries
Related
I'm currenly trying working with tensorflow dataset 'tf_flowers', and noticed that a lot of images consist mostly of black canvas, like this:
flower1
flower2
Is there any easy way to remove/or filter it out? Preferably it should work on batches, and compile into a graph with #tf.function, as I plan to use it also for bigger datasets with dataset.map(...)
The black pixels are just because of padding. This is a simple operation that allows you to have network inputs having the same size (i.e. you have batches containing images with the of size: 223x221 because smaller images are padded with black pixels).
An alternative to padding that removes the need of adding black pixels to the image, is that of preprocessing the images by:
removing padding via cropping operation
resizing the cropped images to the same size (e.g. 223x221)
You can do all of these operations in simple python, thanks to tensorflow map function. First, define your python function
def py_preprocess_image(numpy_image):
input_size = numpy_image.shape # this is (223, 221)
image_proc = crop_by_removing_padding(numpy_image)
image_proc = resize(image_proc, size=input_size)
return image_proc
Then, given your tensorflow dataset train_data, map the above python function on each input:
# train_data is your tensorflow dataset
train_data = train_data.map(
lambda x: tf.py_func(preprocess_image,
inp = [x], Tout=[tf.float32]),
num_parallel_calls=num_threads
)
Now, you only need to define crop_by_removing_padding and resize, which operate on ordinary numpy arrays and can thus be written in pure python code. For example:
def crop_by_removing_padding(img):
xmax, ymax = np.max(np.argwhere(img), axis=0)
img_crop = img[:xmax + 1, :ymax + 1]
return img_crop
def resize(img, new_size):
img_rs = cv2.resize(img, (new_size[1], new_size[0]), interpolation=cv2.INTER_CUBIC)
return img_rs
I have 100 large-ish (1000x1000) images which I want to use as a training data set for a texture analysis system. I want to randomly generate texture swatches of about 200x200. What is the best way to do this? I would prefer to not preprocess all of the swatches so that each epoch is trained with slightly different swatches.
My initial (naive?) implementation included preprocessing layers in the model that do random crops on the image and just do a ton of epochs to accommodate the small number of large pictures, however after about ~400 epochs TF would crash without exception (it would just exit).
I now find myself coding a data generator (tf.keras.utils.Sequence) that will return a batch of swatches on request, but I feel like I'm reinventing the wheel and it is getting clunky - making me think this can't be the best way.
What is the best way to handle such a situation where you have a somewhat small dataset that you dynamically create more samples from?
I have written a function that will segment an image. Code is below
import cv2
def image_segment( image_path, img_resize, crop_size):
image_list=[]
img=cv2.imread(image_path)
img=cv2.resize(img, img_resize)
shape=img.shape
xsteps =int( shape[0]/crop_size[0])
ysteps = int( shape[1]/crop_size[1])
print (xsteps, ysteps)
for i in range (xsteps):
for j in range (ysteps):
x= i * crop_size[0]
xend=x + crop_size[0]
y= j * crop_size[1]
yend = y + crop_size[1]
cropped_image = cropped_image=img[x: xend, y: yend]
image_list.append(cropped_image)
return image_list
below is an example of use
# This code provides input to the image_segment function
image_path=r'c:\temp\landscape.jpg' # location of image
width=1000 # width to resize input image
height= 800 # height to resize input image
image_resize=( width, height) # specify original image (width, height)
crop_width=200 # width of desired cropped images
crop_height=400 # height of desired cropped images
# Note to get full set of cropped images width/cropped_width and height/cropped_height should be integer values
crop_size=(crop_height, crop_width)
images=image_segment(image_path, image_resize, crop_size) # call the function
The code below will display the resized input image and the resultant cropped images
# this code will display the resized input image and the resultant cropped images
import matplotlib.pyplot as plt
from matplotlib.pyplot import imshow
img=cv2.imread(image_path) # read in the image
input_resized_image=cv2.resize(img, image_resize) # resize the image
imshow(input_resized_image) # show the resized input image
r=len(images)
plt.figure(figsize=(20, 20))
for i in range(r):
plt.subplot(5, 5, i + 1)
image=images # scale images between 0 and 1 becaue pre-processor set them between -1 and +1
plt.imshow(image[i])
class_name=str(i)
plt.title(class_name, color='green', fontsize=16)
plt.axis('off')
plt.show()
I'm working on a project do detect faces and I'm using the following code:
# demonstrate face detection on 5 Celebrity Faces Dataset
from os import listdir
from PIL import Image
from numpy import asarray
from matplotlib import pyplot
from mtcnn.mtcnn import MTCNN
# extract a single face from a given photograph
def extract_face(filename, required_size=(160, 160)):
# load image from file
image = Image.open(filename)
# convert to RGB, if needed
image = image.convert('RGB')
# convert to array
pixels = asarray(image)
# create the detector, using default weights
detector = MTCNN()
# detect faces in the image
results = detector.detect_faces(pixels)
# extract the bounding box from the first face
x1, y1, width, height = results[0]['box']
# bug fix
x1, y1 = abs(x1), abs(y1)
x2, y2 = x1 + width, y1 + height
# extract the face
face = pixels[y1:y2, x1:x2]
# resize pixels to the model size
image = Image.fromarray(face)
image = image.resize(required_size)
face_array = asarray(image)
return face_array
# specify folder to plot
#folder = '5-celebrity-faces-dataset/train/ben_afflek/'
folder = '/content/drive/My Drive/ufc-project/5-celebrity-faces-dataset/train/ben_afflek'
i = 1
# enumerate files
for filename in listdir(folder):
# path
path = folder + '/' + filename
# get face
face = extract_face(path)
print(i, face.shape)
# plot
pyplot.subplot(2, 7, i)
pyplot.axis('off')
pyplot.imshow(face)
i += 1
pyplot.show()
I got this code from this tutorial: https://machinelearningmastery.com/how-to-develop-a-face-recognition-system-using-facenet-in-keras-and-an-svm-classifier/?unapproved=549711&moderation-hash=6b355b586de2f2ff191df14529849990#comment-549711
It's working, but I'm getting the following warning:
WARNING:tensorflow:6 out of the last 11 calls to <function Model.make_predict_function..predict_function at 0x7f0016e2eae8> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating #tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your #tf.function outside of the loop. For (2), #tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.
Does anyone know how to fix it?
You are creating the detector each time you call extract face. Move the face detector creation out from the loop and pass it as an argument to the function.
Move the line detector = MTCNN() outside of extract_face specifically before the loop.
Starting from the Tensorflow CNN example, I'm trying to modify the model to have multiple images as an input (so that the input has not just 3 input channels, but multiples of 3 by stacking images).
To augment the input, I try to use random image operations, such as flipping, contrast and brightness provided in TensorFlow.
My current solution to apply the same random distortion to all input images is to use a fixed seed value for these operations:
def distort_image(image):
flipped_image = tf.image.random_flip_left_right(image, seed=42)
contrast_image = tf.image.random_contrast(flipped_image, lower=0.2, upper=1.8, seed=43)
brightness_image = tf.image.random_brightness(contrast_image, max_delta=0.2, seed=44)
return brightness_image
This method is called multiple times for each image at graph construction time, so I thought for each image it will use the same random number sequence and consequently, it will result in have the same applied image operations for my image input sequence.
# ...
# distort images
distorted_prediction = distort_image(seq_record.prediction)
distorted_input = []
for i in xrange(INPUT_SEQ_LENGTH):
distorted_input.append(distort_image(seq_record.input[i,:,:,:]))
stacked_distorted_input = tf.concat(2, distorted_input)
# Ensure that the random shuffling has good mixing properties.
min_queue_examples = int(num_examples_per_epoch *
MIN_FRACTION_EXAMPLES_IN_QUEUE)
# Generate a batch of sequences and prediction by building up a queue of examples.
return generate_sequence_batch(stacked_distorted_input, distorted_prediction, min_queue_examples,
batch_size, shuffle=True)
In theory, this works fine. And after doing some test runs, this really seemed to solve my problem. But after a while, I found out that I'm having a race-condition, because I use the input pipeline of the CNN-example code with multiple threads (which is the suggested method in TensorFlow to improve performance and reduce memory consumption at runtime):
def generate_sequence_batch(sequence_in, prediction, min_queue_examples,
batch_size):
num_preprocess_threads = 8 # <-- !!!
sequence_batch, prediction_batch = tf.train.shuffle_batch(
[sequence_in, prediction],
batch_size=batch_size,
num_threads=num_preprocess_threads,
capacity=min_queue_examples + 3 * batch_size,
min_after_dequeue=min_queue_examples)
return sequence_batch, prediction_batch
Because multiple threads create my examples, it is not guaranteed anymore that all image operations are performed in the right order (in sense of the right order of random operations).
Here I came to a point where I got completely stuck. Does anyone know how to solve this problem to apply the same image distortion to multiple images?
Some thoughts of mine:
I thought about to do some synchronizations arround these image distortion methods, but I could find anything provided by TensorFlow
I tried to generate to generate a random number for e.g. the random brightness delta using tf.random_uniform() by myself and use this value for tf.image.adjust_contrast(). But the result of the TensorFlow random generator is always a tensor, and I have not found a way to use this tensor as a parameter for tf.image.adjust_contrast() which expects a simple float32 for its contrast_factor parameter.
A solution that would (partly) work would be to combine all images to a huge image using tf.concat(), apply random operations to change contrast and brightness, and split the image afterwards. But this would not work for random flipping, because this would (at least in my case) change the order of the images, and there is no way to detect whether tf.image.random_flip_left_right() has performed a flip or not, which would be required to fix the wrong order of images if necessary.
Here is what I came up with by looking at the code of random_flip_up_down and random_flip_left_right within tensorflow :
def image_distortions(image, distortions):
distort_left_right_random = distortions[0]
mirror = tf.less(tf.pack([1.0, distort_left_right_random, 1.0]), 0.5)
image = tf.reverse(image, mirror)
distort_up_down_random = distortions[1]
mirror = tf.less(tf.pack([distort_up_down_random, 1.0, 1.0]), 0.5)
image = tf.reverse(image, mirror)
return image
distortions = tf.random_uniform([2], 0, 1.0, dtype=tf.float32)
image = image_distortions(image, distortions)
label = image_distortions(label, distortions)
I would do something like this using tf.case. It allows you to specify what to return if certain condition holds https://www.tensorflow.org/api_docs/python/tf/case
import tensorflow as tf
def distort(image, x):
# flip vertically, horizontally, both, or do nothing
image = tf.case({
tf.equal(x,0): lambda: tf.reverse(image,[0]),
tf.equal(x,1): lambda: tf.reverse(image,[1]),
tf.equal(x,2): lambda: tf.reverse(image,[0,1]),
}, default=lambda: image, exclusive=True)
return image
def random_distortion(image):
x = tf.random_uniform([1], 0, 4, dtype=tf.int32)
return distort(image, x[0])
To check if it works.
import numpy as np
import matplotlib.pyplot as plt
# create image
image = np.zeros((25,25))
image[:10,5:10] = 1.
# create subplots
fig, axes = plt.subplots(2,2)
for i in axes.flatten(): i.axis('off')
with tf.Session() as sess:
for i in range(4):
distorted_img = sess.run(distort(image, i))
axes[i % 2][i // 2].imshow(distorted_img, cmap='gray')
plt.show()
How can I draw regular grid lines over a tiff image?
I want to draw regular square grids for each interval (say 100 by 100 pixels) over the image and save that with the drawings. I also need to overlay each grid id as '1','2',...at the middle of each grid box.
You will need the python imaging library (PIL) installed. (See here https://pypi.python.org/pypi/PIL). See these answers for examples of ways to install PIL: answer 1, answer 2
Right, with that installed, the following code should do what you ask for:
import matplotlib.pyplot as plt
import matplotlib.ticker as plticker
try:
from PIL import Image
except ImportError:
import Image
# Open image file
image = Image.open('myImage.tiff')
my_dpi=300.
# Set up figure
fig=plt.figure(figsize=(float(image.size[0])/my_dpi,float(image.size[1])/my_dpi),dpi=my_dpi)
ax=fig.add_subplot(111)
# Remove whitespace from around the image
fig.subplots_adjust(left=0,right=1,bottom=0,top=1)
# Set the gridding interval: here we use the major tick interval
myInterval=100.
loc = plticker.MultipleLocator(base=myInterval)
ax.xaxis.set_major_locator(loc)
ax.yaxis.set_major_locator(loc)
# Add the grid
ax.grid(which='major', axis='both', linestyle='-')
# Add the image
ax.imshow(image)
# Find number of gridsquares in x and y direction
nx=abs(int(float(ax.get_xlim()[1]-ax.get_xlim()[0])/float(myInterval)))
ny=abs(int(float(ax.get_ylim()[1]-ax.get_ylim()[0])/float(myInterval)))
# Add some labels to the gridsquares
for j in range(ny):
y=myInterval/2+j*myInterval
for i in range(nx):
x=myInterval/2.+float(i)*myInterval
ax.text(x,y,'{:d}'.format(i+j*nx),color='w',ha='center',va='center')
# Save the figure
fig.savefig('myImageGrid.tiff',dpi=my_dpi)
Which, if used on the grace_hopper.png example file, produces the following output:
This can be done effectively in two lines by looping over the image data at your grid intervals. Using the canonical image from the SIPI database as an example
import pylab as plt
# Load the image
img = plt.imread("lena512color.tiff")
# Grid lines at these intervals (in pixels)
# dx and dy can be different
dx, dy = 100,100
# Custom (rgb) grid color
grid_color = [0,0,0]
# Modify the image to include the grid
img[:,::dy,:] = grid_color
img[::dx,:,:] = grid_color
# Show the result
plt.imshow(img)
plt.show()
The answer by #tom may be more robust as it works with the matplotlib library. I'll leave this example up for its simplicity.
Let me just leave it here
def draw_grid(image, line_space=20):
H, W = image.shape
image[0:H:line_space] = 1
image[:, 0:W:line_space] = 1