I have a trained auto-encoder model which I want to visualize the latent layer in tensor-board.
How can I do it ?
el1 = Conv2D(8, (3, 3), activation='relu', padding='same', input_shape=(224, 224, 3))
el2 = MaxPooling2D((2, 2), padding='same')
el3 = Conv2D(8, (3, 3), activation='relu', padding='same')
el4 = MaxPooling2D((2, 2), padding='same')
dl1 = Conv2DTranspose(8, (3, 3), strides=2, activation='relu', padding='same')
dl2 = Conv2DTranspose(8, (3, 3), strides=2, activation='relu', padding='same')
output_layer = Conv2D(3, (3, 3), activation='sigmoid', padding='same')
autoencoder = Sequential()
autoencoder.add(el1)
autoencoder.add(el2)
autoencoder.add(el3)
autoencoder.add(el4)
autoencoder.add(dl1)
autoencoder.add(dl2)
autoencoder.add(output_layer)
autoencoder.compile(optimizer='adam', loss="binary_crossentropy")
logdir = os.path.join("logs/fit/", datetime.datetime.now().strftime("%Y%m%d-%H%M%S"))
tensorboard_callback = tf.keras.callbacks.TensorBoard(logdir, histogram_freq=1)
autoencoder.fit(X_train, X_train, epochs=100, batch_size=64, validation_data=(X_test, X_test), verbose=1,
callbacks=[tensorboard_callback])
After the model was fitted, how can I add the latent layer into tensor-board and view it after running tsne or pca ?
You can follow the guide: Visualizing Data using the Embedding Projector in TensorBoard.
I assumed that by "latent layer" you mean "latent space", i.e the representation of the encoded input.
In your case, if you want to represent your latent space, it's first needed to extract the encoder part from your autoencoder. This can be achieved with the functional API of keras:
# After fitting the autoencoder, we create a model that represents the encoder
encoder = tf.keras.Model(autoencoder.input, autoencoder.get_layer(el4.name).output)
Then, it's possible to calculate the latent representation of your test set using the encoder:
latent_test = encoder(X_test)
Then, by following the guide linked above, the latent representation can be saved in a Checkpoint to be visualized with the Tensorboard projector:
# Save the weights we want to analyze as a variable.
# The weights need to have the shape (Number of sample, Total Dimensions)
# Hence why we flatten the Tensor
weights = tf.Variable(tf.reshape(latent_test,(X_test.shape[0],-1)), name="latent_test")
# Create a checkpoint from embedding, the filename and key are the
# name of the tensor.
checkpoint = tf.train.Checkpoint(latent_test=weights)
checkpoint.save(os.path.join(logdir, "embedding.ckpt"))
from tensorboard.plugins import projector
# Set up config.
config = projector.ProjectorConfig()
embedding = config.embeddings.add()
# The name of the tensor will be suffixed by `/.ATTRIBUTES/VARIABLE_VALUE`.
embedding.tensor_name = "latent_test/.ATTRIBUTES/VARIABLE_VALUE"
projector.visualize_embeddings(logdir, config)
Finally, the projector can be accessed by running the Tensorboard:
$ tensorboard --logdir /path/to/logdir
Finally an image of the projector with PCA (here with some random data):
I have built the following object recognition model for my school assignment to predict classes from the CIFAR-10 dataset. The assignment requires that I use the VALID padding for all convolution and pooling layers.
def _build_cifar10_model(num_C1_channels=50, num_C2_channels=60, use_dropout=False):
model = Sequential()
# reshape 1D array of length 3072
# to a matrix of shape 32x32x3
model.add(Input(shape=(3072,)))
model.add(Reshape(target_shape=(32, 32, 3), input_shape=(3072,)))
# 24x24x3
model.add(Conv2D(filters=num_C1_channels, kernel_size=(9, 9), padding='valid', activation='relu'))
# 12x12x3
model.add(MaxPooling2D(pool_size=(2, 2), strides=(2, 2), padding='valid'))
# 8x8x3
model.add(Conv2D(filters=num_C2_channels, kernel_size=(5, 5), padding='valid', activation='relu'))
# 4x4x3
model.add(MaxPooling2D(pool_size=(2, 2), strides=(2, 2), padding='valid'))
model.add(Flatten())
model.add(Dense(units=300))
if use_dropout:
model.add(Dropout(rate=0.5))
model.add(Dense(units=10, activation='softmax'))
if use_dropout:
model.add(Dropout(rate=0.5))
return model
However, building this model is throwing the following error:
InvalidArgumentError: Negative dimension size caused by subtracting 1 from 0 for '{{node max_pooling2d_15/MaxPool}} = MaxPool[T=DT_FLOAT, data_format="NHWC", ksize=[1, 2, 2, 1], padding="VALID", strides=[1, 2, 2, 1]](conv2d_23/Relu)' with input shapes: [?,24,24,0].
I figured that it is because when padding is VALID, a (32, 32, 3) image becomes size (24, 24, 0) after applying that 1st Conv2D layer with kernel_size=9 on it. Not sure why the RGB channels are completely lost.
Is there a way to go around this issue while maintaining the VALID padding?
Sorry in advance as this is the first time I am building such a model.
Use InputLayer instead of Input at the start of your model. It will give you the first layer of the model instead of a symbolic tensor.
Replace the line with this:
model.add(InputLayer(input_shape=(3072,)))
The rest of the code seems to execute for me and gives me a 4x4 layer after the convolution which you feed into your dropout/dense layers.
I want to visualize the layers in capsule networks. For that I need shape of intermediate layers. The code is as follows:
from keras.models import Model
model = Sequential()
conv1 = model.add(Conv2D(256, (9, 9), padding='valid', strides = 1, input_shape = (28, 28, 1), activation = 'relu', name = 'conv1'))
model.add(Conv2D(256, (9,9), padding='valid', strides = 2, name = 'primarycaps_conv2d'))
model.add(keras.layers.core.Reshape([-1,8]))
model.add(keras.layers.core.Lambda(squash, name = 'hello'))
layer_name = 'conv1'
intermediate_layer_model = Model(inputs=model.input,
outputs=model.get_layer(layer_name).output)
intermediate_output = intermediate_layer_model.predict(data)
Throws the following error:
NameError: name 'data' is not defined
How to overcome this?
Referring this, I got the solution as follows:
model = Sequential()
conv1 = model.add(Conv2D(256, (9, 9), padding='valid', strides = 1, input_shape = (28, 28, 1), activation = 'relu', name = 'conv1'))
model.add(Conv2D(256, (9,9), padding='valid', strides = 2, name = 'primarycaps_conv2d'))
model.add(keras.layers.core.Reshape([-1,8]))
model.add(keras.layers.core.Lambda(squash, name = 'hello'))
idx = 3 # index of desired layer
input_shape = model.layers[idx].get_output_shape_at(0)
print(input_shape)
get the output shape as follows:
(None, 1152, 8).
The problem of getting specific layer's output is solved.
My immediate problem is that all of the various CNN regression models I've tried always return the same (or very similar) values and I'm trying to figure out why. But I would be open to a wide range of suggestions.
My dataset looks like this:
x: 64x64 greyscale images arranged into a 64 x 64 x n ndarray
y: Values between 0 and 1, each corresponding to an image (think of this as some sort of proportion)
weather: 4 weather readings from the time each image was taken (ambient temperature, humidity, dewpoint, air pressure)
The goal is to use the images and weather data to predict y. Since I'm working with images, I thought a CNN would be appropriate (please let me know if there are other strategies here).
From what I understand, CNNs are most often used for classification tasks--it's rather unusual to use them for regression. But in theory, it shouldn't be too different--I just need to change the loss function to MSE/RMSE and the last activation function to linear (although maybe a sigmoid is more appropriate here since y is between 0 and 1).
The first hurdle I ran into was trying to figure out how to incorporate the weather data, and the natural choice was to incorporate them into the first fully connected layer. I found an example here: How to train mix of image and data in CNN using ImageAugmentation in TFlearn
The second hurdle I ran into was determining an architecture. Normally I would just pick a paper and copy its architecture, but I couldn't find anything on CNN image regression. So I tried a (fairly simple) network with 3 convolutional layers and 2 fully connected layers, then I tried VGGNet and AlexNet architectures from https://github.com/tflearn/tflearn/tree/master/examples
Now the problem I'm having is that all of the models I'm trying output the same value, namely the mean y of the training set. Looking at tensorboard, the loss function flattens out fairly quickly (after around 25 epochs). Do you know what's going on here? While I do understand the basics of what each layer is doing, I have no intuition on what makes a good architecture for a particular dataset or task.
Here is an example. I am using VGGNet from the tflearn examples page:
tf.reset_default_graph()
img_aug = ImageAugmentation()
img_aug.add_random_flip_leftright()
img_aug.add_random_flip_updown()
img_aug.add_random_90degrees_rotation(rotations=[0, 1, 2, 3])
convnet = input_data(shape=[None, size, size, 1],
data_augmentation=img_aug,
name='hive')
weathernet = input_data(shape=[None, 4], name='weather')
convnet = conv_2d(convnet, 64, 3, activation='relu', scope='conv1_1')
convnet = conv_2d(convnet, 64, 3, activation='relu', scope='conv1_2')
convnet = max_pool_2d(convnet, 2, strides=2, name='maxpool1')
convnet = conv_2d(convnet, 128, 3, activation='relu', scope='conv2_1')
convnet = conv_2d(convnet, 128, 3, activation='relu', scope='conv2_2')
convnet = max_pool_2d(convnet, 2, strides=2, name='maxpool2')
convnet = conv_2d(convnet, 256, 3, activation='relu', scope='conv3_1')
convnet = conv_2d(convnet, 256, 3, activation='relu', scope='conv3_2')
convnet = conv_2d(convnet, 256, 3, activation='relu', scope='conv3_3')
convnet = max_pool_2d(convnet, 2, strides=2, name='maxpool3')
convnet = conv_2d(convnet, 512, 3, activation='relu', scope='conv4_1')
convnet = conv_2d(convnet, 512, 3, activation='relu', scope='conv4_2')
convnet = conv_2d(convnet, 512, 3, activation='relu', scope='conv4_3')
convnet = max_pool_2d(convnet, 2, strides=2, name='maxpool4')
convnet = conv_2d(convnet, 512, 3, activation='relu', scope='conv5_1')
convnet = conv_2d(convnet, 512, 3, activation='relu', scope='conv5_2')
convnet = conv_2d(convnet, 512, 3, activation='relu', scope='conv5_3')
convnet = max_pool_2d(convnet, 2, strides=2, name='maxpool5')
convnet = fully_connected(convnet, 4096, activation='relu', scope='fc6')
convnet = merge([convnet, weathernet], 'concat')
convnet = dropout(convnet, .75, name='dropout1')
convnet = fully_connected(convnet, 4096, activation='relu', scope='fc7')
convnet = dropout(convnet, .75, name='dropout2')
convnet = fully_connected(convnet, 1, activation='sigmoid', scope='fc8')
convnet = regression(convnet,
optimizer='adam',
learning_rate=learning_rate,
loss='mean_square',
name='targets')
model = tflearn.DNN(convnet,
tensorboard_dir='log',
tensorboard_verbose=0)
model.fit({
'hive': x_train,
'weather': weather_train
},
{'targets': y_train},
n_epoch=1000,
batch_size=batch_size,
validation_set=({
'hive': x_val,
'weather': weather_val
},
{'targets': y_val}),
show_metric=False,
shuffle=True,
run_id='poop')
To get at what my objects are:
x_train is an ndarray of shape (n, 64, 64, 1)
weather_train is an ndarray of shape (n, 4)
y_train is an ndarray of shape (n, 1)
Overfitting is another concern, but given that the models perform poorly on the training set, I think I can worry about that later.
To address your concern regarding the same predicted value for all instances in your test set. You have a couple options here that don't involve changing the structure of your conv net:
You can rescale your target variable using sklearn StandardScaler() (which standardizes features by removing the mean and scaling to unit variance)
Scale pixel data; generally performance increases with scaled pixel data, as a rule of thumb always divide pixel data by 255.0 (shown at end of post)
You can play around with learning rate and the error function (the reason that the CNN is outputting the same value for all predictions is because that is what it has determined is the point of minimum error)
Next. If you are trying to perform regression ensure that your final fully connected layer uses a linear activation function instead of sigmoid. A linear activation function takes inputs to the neuron multiplied by the neuron weight and creates an output proportional to the input.
convnet = fully_connected(convnet, 1, activation='linear', scope='fc8')
Lastly. I have recently implemented ResNet50 for regression tasks in Keras. Here is the construction of that network, this version does not permit loading of pretrained weights and it must receive images of shape (224, 224, 3).
from keras.layers.normalization import BatchNormalization
from keras.layers.convolutional import Conv2D, MaxPooling2D, DepthwiseConv2D
from keras.layers.core import Activation, Dropout, Dense
from keras.layers import Flatten, Input, Add, ZeroPadding2D, GlobalAveragePooling2D, GlobalMaxPooling2D
from keras.models import Model
from keras import backend
def block1(x, filters, kernel_size=3, stride=1, conv_shortcut=True, name=None):
"""
A residual block
:param x: input tensor
:param filters: integer, filters of the bottleneck layer
:param kernel_size: kernel size of bottleneck
:param stride: stride of first layer
:param conv_shortcut: use convolution shortcut if true, otherwise identity shortcut
:param name: string, block label
:return: Output tensor of the residual block
"""
# bn_axis = 3 if backend.image_data_format() == 'channels_last' else 1
bn_axis = -1
if conv_shortcut is True:
shortcut = Conv2D(4 * filters, 1, strides=stride, name=name+'_0_conv')(x)
shortcut = BatchNormalization(axis=bn_axis, epsilon=1.001e-5, name=name+'_0_bn')(shortcut)
else:
shortcut = x
x = Conv2D(filters, 1, strides=stride, name=name+'_1_conv')(x)
x = BatchNormalization(axis=bn_axis, epsilon=1.001e-5, name=name+'_1_bn')(x)
x = Activation('relu', name=name+'_1_relu')(x)
x = Conv2D(filters, kernel_size, padding='SAME', name=name+'_2_conv')(x)
x = BatchNormalization(axis=bn_axis, epsilon=1.001e-5, name=name+'_2_bn')(x)
x = Activation('relu', name=name+'_2_relu')(x)
x = Conv2D(4 * filters, 1, name=name+'_3_conv')(x)
x = BatchNormalization(axis=bn_axis, epsilon=1.001e-5, name=name+'_3_bn')(x)
x = Add(name=name+'_add')([shortcut, x])
x = Activation('relu', name=name+'_out')(x)
return x
def stack1(x, filters, blocks, stride1=2, name=None):
"""
a set of stacked residual blocks
:param x: input tensor
:param filters: int, filters fof the bottleneck layer in the block
:param blocks: int, blocks in the stacked blocks,
:param stride1: stride of the first layer in the first block
:param name: stack label
:return: output tensor for the stacked blocks
"""
x = block1(x, filters, stride=stride1, name=name+'_block1')
for i in range(2, blocks+1):
x = block1(x, filters, conv_shortcut=False, name=name+'_block'+str(i))
return x
def resnet(height, width, depth, stack_fn, use_bias=False, nodes=256):
"""
:param height: height of image, int
:param width: image width, int
:param depth: bn_axis or depth, int
:param stack_fn: function that stacks residual blocks
:param nodes: width of nodes included in top layer of CNN, int
:return: a Keras model instance
"""
input_shape = (height, width, depth)
img_input = Input(shape=input_shape)
x = ZeroPadding2D(padding=((3, 3), (3, 3)), name='conv1_pad')(img_input)
x = Conv2D(64, 7, strides=2, use_bias=use_bias, name='conv1_conv')(x)
x = ZeroPadding2D(padding=((1, 1), (1, 1)), name='pool1_pad')(x)
x = MaxPooling2D(3, strides=2, name='pool1_pool')(x)
x = stack_fn(x)
# top layer
x = GlobalAveragePooling2D(name='avg_pool')(x)
x = Dense(nodes, activation='relu')(x)
# perform regression
x = Dense(1, activation='linear')(x)
model = Model(img_input, x)
return model
def resnet50(height, width, depth, nodes):
def stack_fn(x):
x = stack1(x, 64, 3, stride1=1, name='conv2')
x = stack1(x, 128, 4, name='conv3')
x = stack1(x, 256, 6, name='conv4')
x = stack1(x, 512, 3, name='conv5')
return x
return resnet(height, width, depth, stack_fn, nodes=nodes)
Which can be implemented using some x_train, x_test, y_train, y_test data (where x_train/test is image data and y_train,y_test data are numeric values on the interval [0, 1].
scaler = MinMaxScaler()
images = load_images(df=target, path=PATH_features, resize_shape=(224, 224), quadruple=True)
images = images / 255.0 # scale pixel data to [0, 1]
images = images.astype(np.float32)
imshape = images.shape
target = target[Target]
target = quadruple_target(target, target=Target)
x_train, x_test, y_train, y_test = train_test_split(images, target, test_size=0.3, random_state=101)
y_train = scaler.fit_transform(y_train)
y_test = scaler.transform(y_test)
model = resnet50(imshape[1], imshape[2], imshape[3], nodes=256)
opt = Adam(lr=1e-5, decay=1e-5 / 200)
model.compile(loss=lossFN, optimizer=opt)
history = model.fit(x_train, y_train, validation_data=(x_test, y_test), verbose=1, epochs=200)
pred = model.predict(x_test)
I am a newbie in machine learning.
Actually, I used my unet code for image segmentation using one input image slice (192x912) and one output mask image (192x192)
My Unet code is contained several CNN layer and I usually used one input image (192x912) and one its corresponding mask binary image for training.
Code related with above explanation is as below.
def get_unet():
inputs = Input((img_rows, img_cols, 1))
conv1 = Conv2D(32, (3, 3), activation='relu', padding='same')(inputs)
conv1 = Conv2D(32, (3, 3), activation='relu', padding='same')(conv1)
pool1 = MaxPooling2D(pool_size=(2, 2))(conv1)
drop1 = Dropout(0.2)(pool1)
conv2 = Conv2D(64, (3, 3), activation='relu', padding='same')(drop1)
conv2 = Conv2D(64, (3, 3), activation='relu', padding='same')(conv2)
pool2 = MaxPooling2D(pool_size=(2, 2))(conv2)
drop2 = Dropout(0.2)(pool2)
'''''''
return model
model.fit(imgs_train, imgs_mask_train, batch_size=32, epochs=100, verbose=2, shuffle=True, validation_split=0.1, callbacks=[model_checkpoint])
it works well. But, now, I would like to use multi input image when I train my network. So, I add another train data and edit my code like below.
def get_unet():
inputs = Input((img_rows, img_cols, 1))
conv1 = Conv2D(32, (3, 3), activation='relu', padding='same')(inputs)
conv1 = Conv2D(32, (3, 3), activation='relu', padding='same')(conv1)
pool1 = MaxPooling2D(pool_size=(2, 2))(conv1)
drop1 = Dropout(0.2)(pool1)
conv2 = Conv2D(64, (3, 3), activation='relu', padding='same')(drop1)
conv2 = Conv2D(64, (3, 3), activation='relu', padding='same')(conv2)
pool2 = MaxPooling2D(pool_size=(2, 2))(conv2)
drop2 = Dropout(0.2)(pool2)
'''''''
return model
model.fit([imgs_train, imgs_other_train], imgs_mask_train, batch_size=32, epochs=100, verbose=2, shuffle=True, validation_split=0.1, callbacks=[model_checkpoint])
but when I run the train code, I got error message as below.
"ValueError: Error when checking model input: the list of Numpy arrays that you are passing to your model is not the size the model expected. Expected to see 1 array(s), but instead got the following list of 2 arrays: "
I think my U net needs to be changed for multi input, but I don't know where I have to change.
Please help and give me any comments.
Thanks.
This is actually rather easy. All you need to do would adjust your input size I believe.
inputs = Input((img_rows, img_cols, size)) would have worked. Or you could have used concat like:
inputs = []
for _ in range(num_inputs):
inputs.append(Input((self.input_height, self.input_width, self.input_features)))
x = concatenate(inputs)
x = Conv2D(32, (3, 3), activation='relu', padding='same')(x)
You can check something similar I implemented here