TF / Keras error: InputLayer not a Checkpointable - tensorflow

I am trying out the newly added TPU support on Google Colab with the simple cats vs dogs dataset.
After creating a simple CNN, I tried to export the model to TPU. But it failed with error
TypeError: Checkpointable._track_checkpointable() passed type <class 'keras.engine.topology.InputLayer'>, not a Checkpointable.
Here's the code that I wrote on Colab.
model = models.Sequential()
model.add(layers.Conv2D(32, (3,3), activation='relu', input_shape=(150, 150, 3)))
model.add(layers.MaxPooling2D((2,2)))
model.add(layers.Conv2D(64, (3,3), activation='relu'))
model.add(layers.MaxPooling2D((2,2)))
model.add(layers.Flatten())
model.add(layers.Dropout(0.5))
model.add(layers.Dense(512, activation='relu'))
model.add(layers.Dense(1, activation='sigmoid'))
model.summary()
train_datagen = ImageDataGenerator(rescale=1./255)
train_generator = train_datagen.flow_from_directory(train_dir, target_size=(150,150), batch_size=20, class_mode='binary')
tpu_model = tf.contrib.tpu.keras_to_tpu_model(model, strategy=tf.contrib.tpu.TPUDistributionStrategy(tf.contrib.cluster_resolver.TPUClusterResolver(tpu="grpc://" + os.environ['COLAB_TPU_ADDR'])))
My guess is I am doing something wrong in train_generator. But I am not sure what it is. Any help would be highly appreciated.

If you're using or import layers from Keras instead of TensorFlow like this:
from keras import layers,models
from keras.preprocessing.image import ImageDataGenerator
import tensorflow as tf
You will get error like you mention above :
TypeError: Checkpointable._track_checkpointable() passed type <class 'keras.engine.topology.InputLayer'>, not a Checkpointable.
So, you can import layers directly from TensorFlow like my code below:
from tensorflow.keras import layers,models
from keras.preprocessing.image import ImageDataGenerator
import tensorflow as tf
or you can see my full code here:
https://gist.github.com/ilmimris/8218e397dd35ab693404e95db32dc574

Related

How many nodes should I have in the last layer of neural network for binary classification?

I believed that, if I have a binary-classification problem then I should always have only 1 node in the last layer, since the last layer has to decide about the classification. However, in the following code it is not true.
Let's download the pizza/steak datasets (image dataset) and prepare the data using the ImageDataGenerator:
import zipfile
import tensorflow as tf
from tensorflow.keras.layers import Input, Dense, Dropout, Conv2D, MaxPooling2D, Flatten
from tensorflow.keras.preprocessing.image import ImageDataGenerator
from tensorflow.keras.preprocessing import image_dataset_from_directory
from tensorflow.keras.applications import EfficientNetB0, resnet50
from tensorflow.keras.models import Sequential
import numpy as np
import pandas as pd
!wget https://storage.googleapis.com/ztm_tf_course/food_vision/pizza_steak.zip
zip_ref = zipfile.ZipFile("pizza_steak.zip", "r")
zip_ref.extractall()
zip_ref.close()
train_directory = './pizza_steak/train/'
test_directory = './pizza_steak/test/'
IMAGE_SIZE = (224, 224)
image_data_generator = ImageDataGenerator(rescale=1. / 255,
zoom_range=0.2,
shear_range=0.2,
rotation_range=0.2)
train_dt = image_data_generator.flow_from_directory(directory=train_directory,
class_mode='categorical',
batch_size=32,
target_size=IMAGE_SIZE)
test_dt = image_data_generator.flow_from_directory(directory=test_directory,
class_mode='categorical',
batch_size=32,
target_size=IMAGE_SIZE)
and then build, compile a neural-network and fit the data on it:
model = Sequential()
model.add(Conv2D(filters=16, kernel_size=3, activation='relu'))
model.add(Conv2D(filters=16, kernel_size=3, activation='relu'))
model.add(MaxPooling2D())
model.add(Conv2D(filters=16, kernel_size=3, activation='relu'))
model.add(Conv2D(filters=16, kernel_size=3, activation='relu'))
model.add(MaxPooling2D())
model.add(Flatten())
model.add(Dense(1, activation='sigmoid'))
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
model.fit(train_dt,
epochs=5,
validation_data=test_dt,
validation_steps=len(test_dt)
As you can see the val_accuracy is not better than 0.5000, which is very bad!
And now if you just change the last layer to model.add(Dense(2, activation='sigmoid')) and run the same model with 2 nodes in the last layer, you will end up with a far better result, such as val_accuracy: 0.8680.
How should know, how many nodes should I have in the last layer when I have a binary-classification model?
Thanks to #Dr.Snoopy, i add an answer here jut to complete the question.
The point is how do we label our data using the image_data_generator.flow_from_directory().
If we set the class_mode='categorical' then the target is ONE_HOT and the number of nodes in the last layer is equal to "number of classes of target feature". In my case, it is a binary feature, so i need to have 2 nodes in the last layer.
However, if we use class_mode='binary' then the target is indexed and we can have only one node in the last layer.

InvalidArgumentError: Incompatible shapes: [29] vs. [29,7,7,2]

so I'm new right here and in Python also. I'm trying to make my own network. I found some pictures of docs and cats 15x15 and unfortunatly couldn't make this basic network...
So, these are libraries which I'm using
from tensorflow.keras.models import Sequential
from tensorflow.keras import utils
from tensorflow.keras.datasets import mnist
from tensorflow.keras.layers import Dense
import numpy as np
import matplotlib.pyplot as plt
import tensorflow as tf
import keras
from tensorflow.keras.layers import Conv2D
from tensorflow.keras.layers import MaxPooling2D
from tensorflow.keras.layers import GlobalMaxPooling2D
Body
train_dataset = tf.keras.preprocessing.image_dataset_from_directory(
'drive/MyDrive/cats vs dogs/cats vs dogs/training',
color_mode="rgb",
batch_size=32,
image_size=(150, 150),
shuffle=True,
seed=42,
validation_split=0.1,
subset='training',
interpolation="bilinear",
follow_links=False,
)
validation_dataset = tf.keras.preprocessing.image_dataset_from_directory(
'drive/MyDrive/cats vs dogs/cats vs dogs/training',
color_mode="rgb",
batch_size=32,
image_size=(150, 150),
shuffle=True,
seed=42,
validation_split=0.1,
subset='validation',
interpolation="bilinear",
follow_links=False,
)
test_dataset = tf.keras.preprocessing.image_dataset_from_directory(
'drive/MyDrive/cats vs dogs/cats vs dogs/test',
batch_size = 32,
image_size = (150, 150),
interpolation="bilinear"
)
model = Sequential()
model.add(keras.Input(shape=(150, 150, 3)))
model.add(Conv2D(32, 5, strides=2, activation="relu"))
model.add(Conv2D(32, 3, activation="relu"))
model.add(MaxPooling2D(3))
model.add(Dense(250, activation='sigmoid'))
model.add(Dense(100))
model.add(MaxPooling2D(3))
model.add(Dense(2))
model.summary()
model.compile(loss='binary_crossentropy',
optimizer='adam',
metrics=['accuracy'])
history = model.fit(train_dataset, validation_data=validation_dataset, epochs=5, verbose=2)
And I get this error
Incompatible shapes: [29] vs. [29,7,7,2]
[[node gradient_tape/binary_crossentropy/mul_1/BroadcastGradientArgs
(defined at /usr/local/lib/python3.7/dist-packages/keras/optimizer_v2/optimizer_v2.py:464)
]] [Op:__inference_train_function_4364]
Errors may have originated from an input operation.
Input Source operations connected to node
gradient_tape/binary_crossentropy/mul_1/BroadcastGradientArgs:
In[0] gradient_tape/binary_crossentropy/mul_1/Shape:
In[1] gradient_tape/binary_crossentropy/mul_1/Shape_1
I was trying to change from binary_crossentropy to categorical_crossentrapy but it didn't help, I suppose my mistake is in datasets or inputs but I don't know how to solve it :(
Really hope to find help here!
[my architecture][1]
[1]: https://i.stack.imgur.com/w4Y9N.png
You need to flatten your prediction somewhere, otherwise you are outputing an image (29 samples of size 7x7 with 2 channels), while you simply want a flat 2 dimensional logits (so shape 29x2). The architecture you are using is somewhat odd, did you mean to have flattening operation before first Dense layer, and then no "maxpooling2d" (as it makes no sense for flattened signal)? Mixing relu and sigmoid activations is also quite non standard, I would encourage you to start with established architectures rather than try to compose your own to get some intuitions.
model = Sequential()
model.add(keras.Input(shape=(150, 150, 3)))
model.add(Conv2D(32, 5, strides=2, activation="relu"))
model.add(Conv2D(32, 3, activation="relu"))
model.add(MaxPooling2D(3))
model.add(Flatten())
model.add(Dense(250, activation="relu"))
model.add(Dense(100, activation="relu"))
model.add(Dense(2))
model.summary()

AttributeError: 'Sequential' object has no attribute 'run_eagerly'

I'm trying to try using this model to train on rock, paper, scissor pictures. However, it was trained on 1800 pictures and only has an accuracy of 30-40%. I was then trying to use TensorBoard to see whats going on, but the error in the title appears.
from keras.models import Sequential
from keras.layers import Dense, Dropout, Flatten
from keras.layers import Conv2D, MaxPooling2D
from tensorflow.python.keras.callbacks import TensorBoard
model = Sequential()
model.add(Conv2D(256, kernel_size=(4, 4),
activation='relu',
input_shape=(64,64,3)))
model.add(Conv2D(196, (4, 4), activation='relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.25))
model.add(Conv2D(196, (4, 4), activation='relu'))
model.add(Conv2D(196, (4, 4), activation='relu'))
model.add(Dropout(0.25))
model.add(Conv2D(128, (4, 4), activation='relu'))
model.add(Conv2D(128, (4, 4), activation='relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.25))
model.add(Conv2D(96, (4, 4), activation='relu'))
model.add(Conv2D(96, (4, 4), activation='relu'))
model.add(Dropout(0.25))
model.add(Flatten())
model.add(Dense(128, activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(3, activation='softmax'))
''' here it instantiates the tensorboard '''
tensorboard = TensorBoard(log_dir="C:/Users/bamla/Desktop/RPS project/Logs")
model.compile(loss="sparse_categorical_crossentropy",
optimizer="SGD",
metrics=['accuracy'])
model.summary()
''' Here its fitting the model '''
model.fit(x_train, y_train, batch_size=50, epochs = 3, callbacks=
[tensorboard])
This outputs:
Traceback (most recent call last):
File "c:/Users/bamla/Desktop/RPS project/Testing.py", line 82, in <module>
model.fit(x_train, y_train, batch_size=50, epochs = 3, callbacks=
[tensorboard])
File "C:\Users\bamla\AppData\Local\Programs\Python\Python37\lib\site-
packages\keras\engine\training.py", line 1178, in fit
validation_freq=validation_freq)
File "C:\Users\bamla\AppData\Local\Programs\Python\Python37\lib\site-
packages\keras\engine\training_arrays.py", line 125, in fit_loop
callbacks.set_model(callback_model)
File "C:\Users\bamla\AppData\Local\Programs\Python\Python37\lib\site-
packages\keras\callbacks.py", line 68, in set_model
callback.set_model(model)
File "C:\Users\bamla\AppData\Local\Programs\Python\Python37\lib\site-
packages\tensorflow\python\keras\callbacks.py", line 1509, in set_model
if not model.run_eagerly:
AttributeError: 'Sequential' object has no attribute 'run_eagerly'
Also, if you have any tips on how to improve the accuracy it would be appreciated!
The problem is here:
from keras.models import Sequential
from keras.layers import Dense, Dropout, Flatten
from keras.layers import Conv2D, MaxPooling2D
from tensorflow.python.keras.callbacks import TensorBoard
Do not mix keras and tf.keras imports, these are not compatible with each other, and produce weird errors as the ones you are seeing.
I changed from tensorflow.python.keras.callbacks import TensorBoard
to from keras.callbacks import TensorBoard and it worked for me.
for me, this did the job:
from tensorflow.keras import datasets, layers, models
from tensorflow import keras
It seems that you are mixing imports from keras and tensorflow.keras (last one is preferred).
https://www.pyimagesearch.com/2019/10/21/keras-vs-tf-keras-whats-the-difference-in-tensorflow-2-0/
And most importantly, going forward all deep learning practitioners
should switch their code to TensorFlow 2.0 and the tf.keras package.
The original keras package will still receive bug fixes, but moving
forward, you should be using tf.keras.
Try with:
import tensorflow
Conv2D = tensorflow.keras.layers.Conv2D
MaxPooling2D = tensorflow.keras.layers.MaxPooling2D
Dense = tensorflow.keras.layers.Dense
Flatten = tensorflow.keras.layers.Flatten
Dropout = tensorflow.keras.layers.Dropout
TensorBoard = tensorflow.keras.callbacks.TensorBoard
model = tensorflow.keras.Sequential()

Sampled_Softmax input parametrers

I am working on a Speaker recognition problem, I have very big number of classes so I need to use, tf.nn.sampled_softmax_loss to speed up training time. The problem is I am using Keras with Tensorflow as Backend, but Keras doesnt implement Sampled_softmax so I need to use Tensorflow function, but its unclear which should be the inputs of the tf.nn.sampled_softmax_loss() function.
My model and input are as below:
from preprocess import *
import keras
from sklearn.model_selection import train_test_split
from keras.models import Sequential
from keras.layers import Dense, Dropout, Flatten, Conv2D, MaxPooling2D,AveragePooling2D
from keras.utils import to_categorical
from keras.utils import plot_model
from keras.models import load_model
from keras import regularizers
import numpy as np
from keras.callbacks import EarlyStopping
from keras import metrics
import os
import tensorflow as tf
from tensorflow.python.framework import dtypes
from importance_sampling.training import ImportanceTraining
epochs = 50
batch_size = 100
verbose = 1
labels= get_labels(big_numpy_files_path)
num_classes = len(labels)
#save_data_as_numpy_array(max_len = feature_dim_1, max_len2 = feature_dim_2,origin_path=data_set_path,destination_path=numpy_files_path)
#Get X & Y
X_train, X_valid, X_test ,y_train,y_valid,y_test = get_x_y_data(split_ratio=0.8, random_state=42,maxsamples=20,path=big_numpy_files_path)
#reshape X for input into CNN
X_train, X_valid, X_test = reshape_prepare_for_input(X1=X_train, X2=X_valid, X3=X_test,channel=1)
#Dimensions
dim_1 = X_train.shape[1]
dim_2 = X_train.shape[2]
dim_3 = X_train.shape[3]
#one hot encoding of Y
y_train_hot = to_categorical(y_train)
y_valid_hot = to_categorical(y_valid)
y_test_hot = to_categorical(y_test)
#Model
model = Sequential()
model.add(Conv2D(128, kernel_size=(6, 6),strides=2, activation='relu', input_shape=(X_train.shape[1], X_train.shape[2], X_train.shape[3])))
model.add(Conv2D(64, kernel_size=(2, 2),strides=1, activation='relu'))
model.add(Conv2D(32, kernel_size=(2, 2), activation='relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.3))
model.add(Flatten())
model.add(Dense(256, activation='relu', use_bias=True,kernel_regularizer=regularizers.l2(0.01)))
model.add(Dropout(0.2))
model.add(Dense(256, activation='relu',use_bias=True,kernel_regularizer=regularizers.l2(0.02)))
model.add(Dropout(0.3))
model.add(Dense(256, activation='relu',use_bias=True,kernel_regularizer=regularizers.l2(0.02)))
model.add(Dropout(0.3))
model.add(Dense(num_classes, activation='softmax'))
loss=keras.losses.categorical_crossentropy
model.compile(loss=loss,optimizer='adamax',metrics=['accuracy'])
earlystopping = EarlyStopping(monitor='val_loss', min_delta=0.001, patience=15, verbose=verbose, mode='auto')
model.fit(X_train, y_train_hot,batch_size=batch_size,callbacks=[earlystopping],epochs=epochs, verbose=verbose, validation_data=(X_valid, y_valid_hot))
If I want to replace the above loss function with Samled softmax, If I'm trying to do something like below, what should be the inputs give my above architecture and should I add tf.reduce_mean?
for more details on the code, is https://github.com/selimelawwa/Speaker_Verification

super(type, obj): obj must be an instance or subtype of type in Keras

I implement the following to build tiny yolo v2 from scratch using Keras with Tensorflow backend
My code was working fine in Keras 2.1.5
But when i updated to Keras 2.1.6 i ran in to an error
""kernel_constraint=None,
TypeError: super(type, obj): obj must be an instance or subtype of type ""
Please help me out
Thank you so much
import tensorflow as tf
import keras
from keras.preprocessing.image import ImageDataGenerator
from keras.models import Sequential
from keras.layers import Dense, Conv2D, MaxPooling2D, Dropout, Flatten,
Reshape, LeakyReLU, BatchNormalization
def yolo():
model = Sequential()
model.add(Conv2D(16,(3,3), padding='same',input_shape=(416,416,3),data_format='channels_last'))
model.add(LeakyReLU(alpha=0.1))
model.add(MaxPooling2D(pool_size=(2,2)))
model.add(Conv2D(32,(3,3), padding='same'))
model.add(BatchNormalization(axis=-1))
model.add(LeakyReLU(alpha=0.1))
model.add(MaxPooling2D(pool_size=(2,2)))
model.add(Conv2D(64,(3,3), padding='same'))
model.add(BatchNormalization(axis=-1))
model.add(LeakyReLU(alpha=0.1))
model.add(MaxPooling2D(pool_size=(2,2)))
model.add(Conv2D(128,(3,3), padding='same'))
model.add(BatchNormalization(axis=-1))
model.add(LeakyReLU(alpha=0.1))
model.add(MaxPooling2D(pool_size=(2,2)))
model.add(Conv2D(128,(3,3), padding='same'))
model.add(BatchNormalization(axis=-1))
model.add(LeakyReLU(alpha=0.1))
model.add(MaxPooling2D(pool_size=(2,2)))
model.add(Conv2D(12,(1,1), padding='same'))
model.add(BatchNormalization(axis=-1))
model.add(LeakyReLU(alpha=0.1))
model.add(Reshape((13,13,2,6)))
return model
model = yolo()
model.summary()
It can be caused by working without restarting the python kernel after the update.