I've been trying to train this CNN Model, It's a Tensorflow tutorial and I just changed the dataset ( I used fruit 360 dataset) without altering the core of the code. When it finishes training the accuracy stays constant at 0.8565 it doesn't change and when I try and test some images it almost always wrong.
What am I doing wrong?
Code output after executing
Here's the code I used
[enter image description here][1]import matplotlib.pyplot as plt
import numpy as np
import os
import PIL
import tensorflow as tf
import tarfile
from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras.models import Sequential
from tensorflow.keras import datasets, layers, models
from tensorflow import keras
import pathlib
dataset_url = "https://file.io/z5JM3sYAWXv4"
data_dir = tf.keras.utils.get_file(origin=dataset_url,
fname='tomatos',
untar=True,
extract=True)
data_dir = pathlib.Path(data_dir)
print(data_dir)
file_count = sum(len(files) for _, _, files in os.walk(r'tomatos'))
print(file_count)
batch_size = 32
img_height = 180
img_width = 180
train_ds = tf.keras.utils.image_dataset_from_directory(
data_dir,
validation_split=0.2,
subset="training",
seed=123,
image_size=(img_height, img_width),
batch_size=batch_size)
val_ds = tf.keras.utils.image_dataset_from_directory(
data_dir,
validation_split=0.2,
subset="validation",
seed=123,
image_size=(img_height, img_width),
batch_size=batch_size)
class_names = train_ds.class_names
print(class_names)
AUTOTUNE = tf.data.AUTOTUNE
train_ds = train_ds.cache().shuffle(1000).prefetch(buffer_size=AUTOTUNE)
val_ds = val_ds.cache().prefetch(buffer_size=AUTOTUNE)
num_classes = len(class_names)
model = Sequential([
layers.Rescaling(1./255, input_shape=(img_height, img_width, 3)),
layers.Conv2D(16, 3, padding='same', activation='relu'),
layers.MaxPooling2D(),
layers.Conv2D(32, 3, padding='same', activation='relu'),
layers.MaxPooling2D(),
layers.Conv2D(64, 3, padding='same', activation='relu'),
layers.MaxPooling2D(),
layers.Flatten(),
layers.Dense(128, activation='relu'),
layers.Dense(num_classes)
])
model.compile(optimizer='adam',
loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
metrics=['accuracy'])
model.summary()
epochs=2
history = model.fit(
train_ds,
validation_data=val_ds,
epochs=epochs
)
data_augmentation = keras.Sequential(
[
layers.RandomFlip("horizontal",
input_shape=(img_height,
img_width,
3)),
layers.RandomRotation(0.1),
layers.RandomZoom(0.1),
]
)
model = Sequential([
data_augmentation,
layers.Rescaling(1./255),
layers.Conv2D(16, 3, padding='same', activation='relu'),
layers.MaxPooling2D(),
layers.Conv2D(32, 3, padding='same', activation='relu'),
layers.MaxPooling2D(),
layers.Conv2D(64, 3, padding='same', activation='relu'),
layers.MaxPooling2D(),
layers.Dropout(0.2),
layers.Flatten(),
layers.Dense(128, activation='relu'),
layers.Dense(num_classes)
])
model.compile(optimizer='adam',
loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
metrics=['accuracy'])
model.summary()
epochs = 4
history = model.fit(
train_ds,
validation_data=val_ds,
epochs=epochs
)
sunflower_url = "https://puffycarrot.com/wp-content/uploads/2017/04/Green-tomatoes.jpg"
sunflower_path = tf.keras.utils.get_file('tomato2', origin=sunflower_url)
img = tf.keras.utils.load_img(
sunflower_path, target_size=(img_height, img_width)
)
img_array = tf.keras.utils.img_to_array(img)
img_array = tf.expand_dims(img_array, 0) # Create a batch
predictions = model.predict(img_array)
score = tf.nn.softmax(predictions[0])
print(
"This image most likely belongs to {} with a {:.2f} percent confidence."
.format(class_names[np.argmax(score)], 100 * np.max(score))
)
#Yaman Tarawneh, I tried replicating your above mentioned code in Google colab (using TF 2.8) and in Pycharm (using TF 2.7) and did not find the error.
Please check the output image for Pycharm :
and got the same output in Google colab :
Total params: 3,988,898
Trainable params: 3,988,898
Non-trainable params: 0
_________________________________________________________________
Epoch 1/4
78/78 [==============================] - 8s 41ms/step - loss: 0.0309 - accuracy: 0.9835 - val_loss: 5.6374e-07 - val_accuracy: 1.0000
Epoch 2/4
78/78 [==============================] - 2s 25ms/step - loss: 5.7533e-07 - accuracy: 1.0000 - val_loss: 2.7360e-07 - val_accuracy: 1.0000
Epoch 3/4
78/78 [==============================] - 2s 25ms/step - loss: 3.0400e-07 - accuracy: 1.0000 - val_loss: 1.3978e-07 - val_accuracy: 1.0000
Epoch 4/4
78/78 [==============================] - 2s 25ms/step - loss: 1.7403e-07 - accuracy: 1.0000 - val_loss: 7.2102e-08 - val_accuracy: 1.0000
This image most likely belongs to Tomato not Ripened with a 100.00 percent confidence.
For further analysis if the issue still persists, Please let us know which Python and Tensorflow version are you using.
I am trying to do a small assignment project where i am trying to use Siamese network to do speaker recognition. I am trying to use talos for finding best possible hyperparameters. When trying a single set of hyperparamter the code went through OK but when i am giving sets of hyperparameters i am getting MemoryError.
Background (not sure if required but for the sake of it kindly allow me to explain the same)
Splitted audio into 5 seconds chunk each
Calculated MFCC and MFCC_del and MFCC_del_del of each audio sample and conctenated them all so each audio sample is converted into(24x939) data point/matrix
Created siamese network by having two sets of matrixes where we have one set of all "similar" paired speaker audio samples and another being that of "disimilar speakers".
Total of 6000 test points and 14886 training points (not using validation now will use later)
def yield_arrays_train(array_x_train_feat1,array_x_train_feat2,array_y_train,batch_size):
while 1:
for i in range(14886):
X_feat1_train = (array_x_train_feat1[i:i+batch_size,:,:].astype(np.float32))
X_feat2_train = (array_x_train_feat2[i:i+batch_size,:,:].astype(np.float32))
Y_train = (array_y_train[i:i+batch_size].astype(np.float32))
yield ([(np.array(X_feat1_train)),(np.array(X_feat2_train))],(np.array(Y_train)))
def yield_arrays_val(array_x_test_feat1,array_x_test_feat2,array_y_test,batch_size):
while 1:
for i in range(60):
X_feat1_test = (array_x_test_feat1[i:i+batch_size,:,:].astype(np.float32))
X_feat2_test = (array_x_test_feat2[i:i+batch_size,:,:].astype(np.float32))
Y_test = (array_y_test[i:i+batch_size].astype(np.float32))
yield ([(np.array(X_feat1_test)),(np.array(X_feat2_test))],(np.array(Y_test)))
train_generator=yield_arrays_train(xtrain_np_img1,xtrain_np_img2,y_train_numpy,6)
val_generator=yield_arrays_val(xtest_np_img1,xtest_np_img2,y_test_numpy,6)
def siamese(generator,validation_data):
W_init = tf.keras.initializers.he_normal(seed=100)
b_init = tf.keras.initializers.he_normal(seed=50)
input_shape = (24,939)
left_input = Input(input_shape)
right_input = Input(input_shape)
encoder = Sequential()
encoder.add(Conv1D(filters=8,kernel_size=6, padding='same', activation='relu',input_shape=input_shape,kernel_initializer=W_init, bias_initializer=b_init))
encoder.add(BatchNormalization())
encoder.add(Dropout(.1))
encoder.add(MaxPool1D())
encoder.add(Conv1D(filters=6,kernel_size=4, padding='same', activation='relu'))
encoder.add(BatchNormalization())
encoder.add(Dropout(.1))
encoder.add(MaxPool1D())
encoder.add(Conv1D(filters=4,kernel_size=4, padding='same', activation='relu'))
encoder.add(BatchNormalization())
encoder.add(Dropout(.1))
encoder.add(MaxPool1D())
encoder.add(Flatten())
encoder.add(Dense(10,activation='relu'))
encoder.add(Dropout(.1))
encoded_l = encoder(left_input)
encoded_r = encoder(right_input)
distance = Lambda(euclidean_distance, output_shape=eucl_dist_output_shape)([encoded_l, encoded_r])
adam = optimizers.Adam(lr=.1, beta_1=0.1, beta_2=0.999,decay=.1, amsgrad=False)
earlyStopping = EarlyStopping(monitor='loss',min_delta=0,patience=3,verbose=1,restore_best_weights=False)
callback_early_stop_reduceLROnPlateau=[earlyStopping]
model = Model([left_input, right_input], distance)
model.compile(loss=contrastive_loss, optimizer=adam,metrics=[accuracy])
model.summary()
#history = model.fit([(x_train[:,:,:,0]).astype(np.float32),(x_train[:,:,:,1]).astype(np.float32)],y_train, validation_data=([(x_val[:,:,:,0]).astype(np.float32),(x_val[:,:,:,1]).astype(np.float32)], y_val) ,batch_size=params['batch_size'],epochs=params['epochs'],callbacks=callback_early_stop_reduceLROnPlateau)
history =model.fit_generator(generator=train_generator,validation_data=val_generator,steps_per_epoch=2481,epochs=5, validation_steps=1000,verbose=1,callbacks=callback_early_stop_reduceLROnPlateau,use_multiprocessing=False,workers=0)
return history,model
siamese(train_generator,val_generator)
OUTPUT:
Model: "model_6"
__________________________________________________________________________________________________
Layer (type) Output Shape Param # Connected to
==================================================================================================
input_11 (InputLayer) (None, 24, 939) 0
__________________________________________________________________________________________________
input_12 (InputLayer) (None, 24, 939) 0
__________________________________________________________________________________________________
sequential_6 (Sequential) (None, 10) 45580 input_11[0][0]
input_12[0][0]
__________________________________________________________________________________________________
lambda_6 (Lambda) (None, 1) 0 sequential_6[1][0]
sequential_6[2][0]
==================================================================================================
Total params: 45,580
Trainable params: 45,544
Non-trainable params: 36
__________________________________________________________________________________________________
Epoch 1/5
2481/2481 [==============================] - 29s 12ms/step - loss: 0.0056 - accuracy: 0.9986 - val_loss: 0.8333 - val_accuracy: 0.1667
Epoch 2/5
2481/2481 [==============================] - 28s 11ms/step - loss: nan - accuracy: 0.9993 - val_loss: nan - val_accuracy: 0.8333
Epoch 3/5
2481/2481 [==============================] - 28s 11ms/step - loss: nan - accuracy: 1.0000 - val_loss: nan - val_accuracy: 0.8333
Epoch 4/5
2481/2481 [==============================] - 28s 11ms/step - loss: nan - accuracy: 1.0000 - val_loss: nan - val_accuracy: 0.8333
Epoch 00004: early stopping
(<keras.callbacks.callbacks.History at 0x2969fa3fd88>,
<keras.engine.training.Model at 0x2969f2a17c8>)
So without any Talos or it is working with some one set of random hyperparameters. Now i use Talos for first time
p = {
'filter1':[1,2,4,6,8,12,16,24],
'kernel_size1':[2,4,6,8,12],
'filter3' : [1,2,4,6,8],
'kernel_size3' : [1,2,4,6,8,12],
'decay' :[.1,0.01,.001,.0001,.00001],
'droprate1' :[.1,.2,.3],
'filter2':[1,2,4,6,8],
'kernel_size2':[2,4,6,8,12],
'droprate4' : [.1,.2,.3],
'droprate2' :[.1,.2,.3],
'unit1': [10,24,36,64,128,256],
'droprate3': [.1,.2,.3],
'lr' :[(.1,0.01,.001,.0001,.00001)],
'batch_size' : [1,2],
'epochs': [4,8,10] }
def siamese(generator,validation_data,params):
W_init = tf.keras.initializers.he_normal(seed=100)
b_init = tf.keras.initializers.he_normal(seed=50)
input_shape = (24,939)
left_input = Input(input_shape)
right_input = Input(input_shape)
encoder = Sequential()
encoder.add(Conv1D(filters=(params['filter1']),kernel_size=(params['kernel_size1']), padding='same', activation='relu',input_shape=input_shape,kernel_initializer=W_init, bias_initializer=b_init))
encoder.add(BatchNormalization())
encoder.add(Dropout((params["droprate1"])))
encoder.add(MaxPool1D())
encoder.add(Conv1D(filters=(params["filter2"]),kernel_size=(params['kernel_size2']), padding='same', activation='relu'))
encoder.add(BatchNormalization())
encoder.add(Dropout((params["droprate2"])))
encoder.add(MaxPool1D())
encoder.add(Conv1D(filters=(params["filter3"]),kernel_size=(params['kernel_size3']), padding='same', activation='relu'))
encoder.add(BatchNormalization())
encoder.add(Dropout((params['droprate3'])))
encoder.add(MaxPool1D())
encoder.add(Flatten())
encoder.add(Dense((params['unit1']),activation='relu'))
encoder.add(Dropout((params['droprate4'])))
encoded_l = encoder(left_input)
encoded_r = encoder(right_input)
distance = Lambda(euclidean_distance, output_shape=eucl_dist_output_shape)([encoded_l, encoded_r])
adam = optimizers.Adam(lr=params['lr'], beta_1=0.1, beta_2=0.999,decay=params['decay'], amsgrad=False)
earlyStopping = EarlyStopping(monitor='loss',min_delta=0,patience=3,verbose=1,restore_best_weights=False)
callback_early_stop_reduceLROnPlateau=[earlyStopping]
model = Model([left_input, right_input], distance)
model.compile(loss=contrastive_loss, optimizer=adam,metrics=[accuracy])
model.summary()
#history = model.fit([(x_train[:,:,:,0]).astype(np.float32),(x_train[:,:,:,1]).astype(np.float32)],y_train, validation_data=([(x_val[:,:,:,0]).astype(np.float32),(x_val[:,:,:,1]).astype(np.float32)], y_val) ,batch_size=params['batch_size'],epochs=params['epochs'],callbacks=callback_early_stop_reduceLROnPlateau)
history=model.fit_generator(generator=train_generator,validation_data=val_generator,steps_per_epoch=2481,epochs=5,validation_steps=1000,verbose=1,callbacks=callback_early_stop_reduceLROnPlateau,use_multiprocessing=False,workers=0)
return history,model
t=ta.Scan(x=[xtrain_np_img1,xtrain_np_img2],y=y_train_numpy,x_val=[xtest_np_img1,xtest_np_img2],y_val=y_test_numpy,model=siamese,params=p,experiment_name='exp_1')
The Error i get is:
---------------------------------------------------------------------------
MemoryError Traceback (most recent call last)
<ipython-input-9-df856388a4bb> in <module>
1 #t=ta.Scan(x=xtrain_np_img1_img2,y=y_train_numpy,x_val=xtest_np_img1_img2,y_val=y_test_numpy,model=siamese,params=p,experiment_name='exp_1')
2
----> 3 t=ta.Scan(x=[xtrain_np_img1,xtrain_np_img2],y=y_train_numpy,x_val=[xtest_np_img1,xtest_np_img2],y_val=y_test_numpy,model=siamese,params=p,experiment_name='exp_1')
~\anaconda3\envs\MyEnv\lib\site-packages\talos\scan\Scan.py in __init__(self, x, y, params, model, experiment_name, x_val, y_val, val_split, random_method, seed, performance_target, fraction_limit, round_limit, time_limit, boolean_limit, reduction_method, reduction_interval, reduction_window, reduction_threshold, reduction_metric, minimize_loss, disable_progress_bar, print_params, clear_session, save_weights)
194 # start runtime
195 from .scan_run import scan_run
--> 196 scan_run(self)
~\anaconda3\envs\MyEnv\lib\site-packages\talos\scan\scan_run.py in scan_run(self)
7
8 from .scan_prepare import scan_prepare
----> 9 self = scan_prepare(self)
10
11 # initiate the progress bar
~\anaconda3\envs\MyEnv\lib\site-packages\talos\scan\scan_prepare.py in scan_prepare(self)
28 round_limit=self.round_limit,
29 time_limit=self.time_limit,
---> 30 boolean_limit=self.boolean_limit
31 )
32
~\anaconda3\envs\MyEnv\lib\site-packages\talos\parameters\ParamSpace.py in __init__(self, params, param_keys, random_method, fraction_limit, round_limit, time_limit, boolean_limit)
42
43 # create the parameter space
---> 44 self.param_space = self._param_space_creation()
45
46 # handle the boolean limits separately
~\anaconda3\envs\MyEnv\lib\site-packages\talos\parameters\ParamSpace.py in _param_space_creation(self)
133 if len(self.param_index) > 100000:
134
--> 135 final_grid = list(it.product(*self._params_temp))
136 out = np.array(final_grid, dtype='object')
137
MemoryError:
My question is since i am new to python/ machine learning etc
How to properly use fit_generator (earlier i tried using just fit function and my windows would give screen of death because of memory error i switched to trying fit_generator but still same error) . I think i am not writing code properly?
Is my laptop specifications so poor to run this code?
My specifications
RAM : 16 GB
windows 10 64 bit
CPU : intel core i5 (quad core)
GPU: it has inbuilt GTX1050 but i havent configured it as i tried earlier but didnt had much luck
I am tring to use fit_generator and Talos (for hyperparameter tuning) . Earlier when i was using fit method i got MemoryError so when i searched here people said i should try using fit_generator. Earlier i was giving tooo many parameters so even with fit_generator i was getting MemoryError now i have reduced the number of parameters and now i am getting different error. Please find code and error below.
CODE::::::
def yield_arrays_train(array_x_train_feat1=xtrain_np_img1,array_x_train_feat2=xtrain_np_img2,array_y_train=y_train_numpy,batch_size=6):
while 1:
for i in range(14886):
X_feat1_train = (array_x_train_feat1[i:i+batch_size,:,:].astype(np.float16))
X_feat2_train = (array_x_train_feat2[i:i+batch_size,:,:].astype(np.float16))
Y_train = (array_y_train[i:i+batch_size].astype(np.float16))
yield ([(np.array(X_feat1_train)),(np.array(X_feat2_train))],(np.array(Y_train)))
def yield_arrays_val(array_x_test_feat1,array_x_test_feat2,array_y_test,batch_size):
while 1:
for i in range(60):
X_feat1_test = (array_x_test_feat1[i:i+batch_size,:,:].astype(np.float16))
X_feat2_test = (array_x_test_feat2[i:i+batch_size,:,:].astype(np.float16))
Y_test = (array_y_test[i:i+batch_size].astype(np.float16))
yield ([(np.array(X_feat1_test)),(np.array(X_feat2_test))],(np.array(Y_test)))
def siamese (array_x_train_feat1=xtrain_np_img1,array_x_train_feat2=xtrain_np_img2,array_y_train=y_train_numpy,array_x_test_feat1=xtest_np_img1,array_x_test_feat2=xtest_np_img2,array_y_test=y_test_numpy):
W_init = tf.keras.initializers.he_normal(seed=100)
b_init = tf.keras.initializers.he_normal(seed=50)
input_shape = (24,939)
left_input = Input(input_shape)
right_input = Input(input_shape)
encoder = Sequential()
encoder.add(Conv1D(filters=8,kernel_size=6, padding='same', activation='relu',input_shape=input_shape,kernel_initializer=W_init, bias_initializer=b_init))
encoder.add(BatchNormalization())
encoder.add(Dropout(.1))
encoder.add(MaxPool1D())
encoder.add(Conv1D(filters=6,kernel_size=4, padding='same', activation='relu'))
encoder.add(BatchNormalization())
encoder.add(Dropout(.1))
encoder.add(MaxPool1D())
encoder.add(Conv1D(filters=4,kernel_size=4, padding='same', activation='relu'))
encoder.add(BatchNormalization())
encoder.add(Dropout(.1))
encoder.add(MaxPool1D())
encoder.add(Flatten())
encoder.add(Dense(10,activation='relu'))
encoder.add(Dropout(.1))
encoded_l = encoder(left_input)
encoded_r = encoder(right_input)
distance = Lambda(euclidean_distance, output_shape=eucl_dist_output_shape)([encoded_l, encoded_r])
adam = optimizers.Adam(lr=.1, beta_1=0.1, beta_2=0.999,decay=.1, amsgrad=False)
earlyStopping = EarlyStopping(monitor='loss',min_delta=0,patience=3,verbose=1,restore_best_weights=False)
callback_early_stop_reduceLROnPlateau=[earlyStopping]
model = Model([left_input, right_input], distance)
model.compile(loss=contrastive_loss, optimizer=adam,metrics=[accuracy])
model.summary()
#history = model.fit([(x_train[:,:,:,0]).astype(np.float32),(x_train[:,:,:,1]).astype(np.float32)],y_train, validation_data=([(x_val[:,:,:,0]).astype(np.float32),(x_val[:,:,:,1]).astype(np.float32)], y_val) ,batch_size=params['batch_size'],epochs=params['epochs'],callbacks=callback_early_stop_reduceLROnPlateau)
history=model.fit_generator(generator=yield_arrays_train(array_x_train_feat1,array_x_train_feat2,array_y_train,6),validation_data=yield_arrays_val(array_x_test_feat1,array_x_test_feat2,array_y_test,6),steps_per_epoch=2481,epochs=5, validation_steps=1000,verbose=1,callbacks=callback_early_stop_reduceLROnPlateau,use_multiprocessing=False,workers=0)
return history,model
siamese (xtrain_np_img1,xtrain_np_img2,y_train_numpy,xtest_np_img1,xtest_np_img2,y_test_numpy)
OUTPUT::::::::
WARNING:tensorflow:From C:\Users\DELL\AppData\Roaming\Python\Python37\site-packages\keras\backend\tensorflow_backend.py:4070: The name tf.nn.max_pool is deprecated. Please use tf.nn.max_pool2d instead.
Model: "model_1"
__________________________________________________________________________________________________
Layer (type) Output Shape Param # Connected to
==================================================================================================
input_1 (InputLayer) (None, 24, 939) 0
__________________________________________________________________________________________________
input_2 (InputLayer) (None, 24, 939) 0
__________________________________________________________________________________________________
sequential_1 (Sequential) (None, 10) 45580 input_1[0][0]
input_2[0][0]
__________________________________________________________________________________________________
lambda_1 (Lambda) (None, 1) 0 sequential_1[1][0]
sequential_1[2][0]
==================================================================================================
Total params: 45,580
Trainable params: 45,544
Non-trainable params: 36
__________________________________________________________________________________________________
WARNING:tensorflow:From C:\Users\DELL\anaconda3\envs\MyEnv\lib\site-packages\tensorflow\python\ops\math_grad.py:1250: add_dispatch_support.<locals>.wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
WARNING:tensorflow:From C:\Users\DELL\AppData\Roaming\Python\Python37\site-packages\keras\backend\tensorflow_backend.py:422: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead.
Epoch 1/5
2481/2481 [==============================] - 30s 12ms/step - loss: 0.0024 - accuracy: 0.9992 - val_loss: 0.8333 - val_accuracy: 0.1667
Epoch 2/5
2481/2481 [==============================] - 28s 11ms/step - loss: 6.9194e-05 - accuracy: 0.9999 - val_loss: 0.8333 - val_accuracy: 0.1667
Epoch 3/5
2481/2481 [==============================] - 28s 11ms/step - loss: nan - accuracy: 0.9993 - val_loss: nan - val_accuracy: 0.8333
Epoch 4/5
2481/2481 [==============================] - 28s 11ms/step - loss: nan - accuracy: 1.0000 - val_loss: nan - val_accuracy: 0.8333
Epoch 5/5
2481/2481 [==============================] - 28s 11ms/step - loss: nan - accuracy: 1.0000 - val_loss: nan - val_accuracy: 0.8333oss: nan - acc
Epoch 00005: early stopping
(<keras.callbacks.callbacks.History at 0x26cf45c6ec8>,
<keras.engine.training.Model at 0x26cf3e364c8>)
So above code is working without Talos when i am using single set of parameters hard coded
if i use
p = {
'filter1':[4,6,8],
'kernel_size1':[2,4,6],
'filter3' : [2,4,6,],
'kernel_size3' : [4,6,8],
'decay' :[.1,0.01,.001],
'droprate1' :[.1,.2,.3],
'filter2':[4,6,8],
'kernel_size2':[8,12],
'droprate4' : [.1,.2],
'droprate2' :[.1,.2],
'unit1': [10,36,64],
'droprate3': [.1,.2],
'lr' :[.1,0.01,.001]
}
def siamese (array_x_train_feat1=xtrain_np_img1,array_x_train_feat2=xtrain_np_img2,array_y_train=y_train_numpy,array_x_test_feat1=xtest_np_img1,array_x_test_feat2=xtest_np_img2,array_y_test=y_test_numpy,params=p):
W_init = tf.keras.initializers.he_normal(seed=100)
b_init = tf.keras.initializers.he_normal(seed=50)
input_shape = (24,939)
left_input = Input(input_shape)
right_input = Input(input_shape)
encoder = Sequential()
encoder.add(Conv1D(filters=(params['filter1']),kernel_size=(params['kernel_size1']), padding='same', activation='relu',input_shape=input_shape,kernel_initializer=W_init, bias_initializer=b_init))
encoder.add(BatchNormalization())
encoder.add(Dropout((params['droprate1'])))
encoder.add(MaxPool1D())
encoder.add(Conv1D(filters=(params['filter2']),kernel_size=(params['kernel_size2']), padding='same', activation='relu'))
encoder.add(BatchNormalization())
encoder.add(Dropout((params['droprate2'])))
encoder.add(MaxPool1D())
encoder.add(Conv1D(filters=(params['filter3']),kernel_size=(params['kernel_size3']), padding='same', activation='relu'))
encoder.add(BatchNormalization())
encoder.add(Dropout((params['droprate3'])))
encoder.add(MaxPool1D())
encoder.add(Flatten())
encoder.add(Dense((params['unit1']),activation='relu'))
encoder.add(Dropout((params['droprate4'])))
encoded_l = encoder(left_input)
encoded_r = encoder(right_input)
distance = Lambda(euclidean_distance, output_shape=eucl_dist_output_shape)([encoded_l, encoded_r])
adam = optimizers.Adam(lr=params['lr'], beta_1=0.1, beta_2=0.999,decay=.1, amsgrad=False)
earlyStopping = EarlyStopping(monitor='loss',min_delta=0,patience=3,verbose=1,restore_best_weights=False)
callback_early_stop_reduceLROnPlateau=[earlyStopping]
model = Model([left_input, right_input], distance)
model.compile(loss=contrastive_loss, optimizer=adam,metrics=[accuracy])
model.summary()
history=model.fit_generator(generator=yield_arrays_train(array_x_train_feat1,array_x_train_feat2,array_y_train,6),validation_data=yield_arrays_val(array_x_test_feat1,array_x_test_feat2,array_y_test,6),steps_per_epoch=2481,epochs=5, validation_steps=1000,verbose=1,callbacks=callback_early_stop_reduceLROnPlateau,use_multiprocessing=False,workers=0)
return history,model
t=ta.Scan(x=[xtrain_np_img1.astype(np.float16),xtrain_np_img2.astype(np.float16)],y=y_train_numpy,x_val=[xtest_np_img1,xtest_np_img2],y_val=y_test_numpy,model=siamese,params=p,experiment_name='exp_1')
ERROR
----> 3 t=ta.Scan(x=[xtrain_np_img1.astype(np.float16),xtrain_np_img2.astype(np.float16)],y=y_train_numpy,x_val=[xtest_np_img1,xtest_np_img2],y_val=y_test_numpy,model=siamese,params=p,experiment_name='exp_1')
~\anaconda3\envs\MyEnv\lib\site-packages\talos\scan\Scan.py in __init__(self, x, y, params, model, experiment_name, x_val, y_val, val_split, random_method, seed, performance_target, fraction_limit, round_limit, time_limit, boolean_limit, reduction_method, reduction_interval, reduction_window, reduction_threshold, reduction_metric, minimize_loss, disable_progress_bar, print_params, clear_session, save_weights)
194 # start runtime
195 from .scan_run import scan_run
--> 196 scan_run(self)
~\anaconda3\envs\MyEnv\lib\site-packages\talos\scan\scan_run.py in scan_run(self)
24 # otherwise proceed with next permutation
25 from .scan_round import scan_round
---> 26 self = scan_round(self)
27 self.pbar.update(1)
28
~\anaconda3\envs\MyEnv\lib\site-packages\talos\scan\scan_round.py in scan_round(self)
17 # fit the model
18 from ..model.ingest_model import ingest_model
---> 19 self.model_history, self.round_model = ingest_model(self)
20 self.round_history.append(self.model_history.history)
21
~\anaconda3\envs\MyEnv\lib\site-packages\talos\model\ingest_model.py in ingest_model(self)
8 self.x_val,
9 self.y_val,
---> 10 self.round_params)
<ipython-input-27-fe409e1ff506> in siamese(array_x_train_feat1, array_x_train_feat2, array_y_train, array_x_test_feat1, array_x_test_feat2, array_y_test, params)
11
12 encoder = Sequential()
---> 13 encoder.add(Conv1D(filters=(params['filter1']),kernel_size=(params['kernel_size1']), padding='same', activation='relu',input_shape=input_shape,kernel_initializer=W_init, bias_initializer=b_init))
14 encoder.add(BatchNormalization())
15 encoder.add(Dropout((params['droprate1'])))
~\AppData\Roaming\Python\Python37\site-packages\keras\legacy\interfaces.py in wrapper(*args, **kwargs)
89 warnings.warn('Update your `' + object_name + '` call to the ' +
90 'Keras 2 API: ' + signature, stacklevel=2)
---> 91 return func(*args, **kwargs)
92 wrapper._original_function = func
93 return wrapper
~\AppData\Roaming\Python\Python37\site-packages\keras\layers\convolutional.py in __init__(self, filters, kernel_size, strides, padding, data_format, dilation_rate, activation, use_bias, kernel_initializer, bias_initializer, kernel_regularizer, bias_regularizer, activity_regularizer, kernel_constraint, bias_constraint, **kwargs)
351 kernel_constraint=kernel_constraint,
352 bias_constraint=bias_constraint,
--> 353 **kwargs)
354
355 def get_config(self):
~\AppData\Roaming\Python\Python37\site-packages\keras\layers\convolutional.py in __init__(self, rank, filters, kernel_size, strides, padding, data_format, dilation_rate, activation, use_bias, kernel_initializer, bias_initializer, kernel_regularizer, bias_regularizer, activity_regularizer, kernel_constraint, bias_constraint, **kwargs)
107 self.filters = filters
108 self.kernel_size = conv_utils.normalize_tuple(kernel_size, rank,
--> 109 'kernel_size')
110 self.strides = conv_utils.normalize_tuple(strides, rank, 'strides')
111 self.padding = conv_utils.normalize_padding(padding)
~\AppData\Roaming\Python\Python37\site-packages\keras\utils\conv_utils.py in normalize_tuple(value, n, name)
37 if len(value_tuple) != n:
38 raise ValueError('The `' + name + '` argument must be a tuple of ' +
---> 39 str(n) + ' integers. Received: ' + str(value))
40 for single_value in value_tuple:
41 try:
ValueError: The `kernel_size` argument must be a tuple of 1 integers. Received: [2, 4, 6]
If i give just one set of hyperparameter just to check do i get different error or same error
p = {
'filter1':[6],
'kernel_size1':[4],
'filter3' : [4],
'kernel_size3' : [6],
'decay' :[.1],
'droprate1' :[.1],
'filter2':[4],
'kernel_size2':[8],
'droprate4' : [.1],
'droprate2' :[.1],
'unit1': [10],
'droprate3': [.1],
'lr' :[.1]
}
def siamese (array_x_train_feat1=xtrain_np_img1,array_x_train_feat2=xtrain_np_img2,array_y_train=y_train_numpy,array_x_test_feat1=xtest_np_img1,array_x_test_feat2=xtest_np_img2,array_y_test=y_test_numpy,params=p):
W_init = tf.keras.initializers.he_normal(seed=100)
b_init = tf.keras.initializers.he_normal(seed=50)
input_shape = (24,939)
left_input = Input(input_shape)
right_input = Input(input_shape)
encoder = Sequential()
encoder.add(Conv1D(filters=(params['filter1']),kernel_size=(params['kernel_size1']), padding='same', activation='relu',input_shape=input_shape,kernel_initializer=W_init, bias_initializer=b_init))
encoder.add(BatchNormalization())
encoder.add(Dropout((params['droprate1'])))
encoder.add(MaxPool1D())
encoder.add(Conv1D(filters=(params['filter2']),kernel_size=(params['kernel_size2']), padding='same', activation='relu'))
encoder.add(BatchNormalization())
encoder.add(Dropout((params['droprate2'])))
encoder.add(MaxPool1D())
encoder.add(Conv1D(filters=(params['filter3']),kernel_size=(params['kernel_size3']), padding='same', activation='relu'))
encoder.add(BatchNormalization())
encoder.add(Dropout((params['droprate3'])))
encoder.add(MaxPool1D())
encoder.add(Flatten())
encoder.add(Dense((params['unit1']),activation='relu'))
encoder.add(Dropout((params['droprate4'])))
encoded_l = encoder(left_input)
encoded_r = encoder(right_input)
distance = Lambda(euclidean_distance, output_shape=eucl_dist_output_shape)([encoded_l, encoded_r])
adam = optimizers.Adam(lr=params['lr'], beta_1=0.1, beta_2=0.999,decay=.1, amsgrad=False)
earlyStopping = EarlyStopping(monitor='loss',min_delta=0,patience=3,verbose=1,restore_best_weights=False)
callback_early_stop_reduceLROnPlateau=[earlyStopping]
model = Model([left_input, right_input], distance)
model.compile(loss=contrastive_loss, optimizer=adam,metrics=[accuracy])
model.summary()
history=model.fit_generator(generator=yield_arrays_train(array_x_train_feat1,array_x_train_feat2,array_y_train,6),validation_data=yield_arrays_val(array_x_test_feat1,array_x_test_feat2,array_y_test,6),steps_per_epoch=2481,epochs=5, validation_steps=1000,verbose=1,callbacks=callback_early_stop_reduceLROnPlateau,use_multiprocessing=False,workers=0)
return history,model
t=ta.Scan(x=[xtrain_np_img1.astype(np.float16),xtrain_np_img2.astype(np.float16)],y=y_train_numpy,x_val=[xtest_np_img1,xtest_np_img2],y_val=y_test_numpy,model=siamese,params=p,experiment_name='exp_1')
ERROR:::::::
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-31-4c923301ede6> in <module>
1 #t=ta.Scan(x=xtrain_np_img1_img2,y=y_train_numpy,x_val=xtest_np_img1_img2,y_val=y_test_numpy,model=siamese,params=p,experiment_name='exp_1')
2
----> 3 t=ta.Scan(x=[xtrain_np_img1.astype(np.float16),xtrain_np_img2.astype(np.float16)],y=y_train_numpy,x_val=[xtest_np_img1,xtest_np_img2],y_val=y_test_numpy,model=siamese,params=p,experiment_name='exp_1')
~\anaconda3\envs\MyEnv\lib\site-packages\talos\scan\Scan.py in __init__(self, x, y, params, model, experiment_name, x_val, y_val, val_split, random_method, seed, performance_target, fraction_limit, round_limit, time_limit, boolean_limit, reduction_method, reduction_interval, reduction_window, reduction_threshold, reduction_metric, minimize_loss, disable_progress_bar, print_params, clear_session, save_weights)
194 # start runtime
195 from .scan_run import scan_run
--> 196 scan_run(self)
~\anaconda3\envs\MyEnv\lib\site-packages\talos\scan\scan_run.py in scan_run(self)
24 # otherwise proceed with next permutation
25 from .scan_round import scan_round
---> 26 self = scan_round(self)
27 self.pbar.update(1)
28
~\anaconda3\envs\MyEnv\lib\site-packages\talos\scan\scan_round.py in scan_round(self)
17 # fit the model
18 from ..model.ingest_model import ingest_model
---> 19 self.model_history, self.round_model = ingest_model(self)
20 self.round_history.append(self.model_history.history)
21
~\anaconda3\envs\MyEnv\lib\site-packages\talos\model\ingest_model.py in ingest_model(self)
8 self.x_val,
9 self.y_val,
---> 10 self.round_params)
<ipython-input-30-fe409e1ff506> in siamese(array_x_train_feat1, array_x_train_feat2, array_y_train, array_x_test_feat1, array_x_test_feat2, array_y_test, params)
11
12 encoder = Sequential()
---> 13 encoder.add(Conv1D(filters=(params['filter1']),kernel_size=(params['kernel_size1']), padding='same', activation='relu',input_shape=input_shape,kernel_initializer=W_init, bias_initializer=b_init))
14 encoder.add(BatchNormalization())
15 encoder.add(Dropout((params['droprate1'])))
~\AppData\Roaming\Python\Python37\site-packages\keras\engine\sequential.py in add(self, layer)
164 # and create the node connecting the current layer
165 # to the input layer we just created.
--> 166 layer(x)
167 set_inputs = True
168 else:
~\AppData\Roaming\Python\Python37\site-packages\keras\engine\base_layer.py in __call__(self, inputs, **kwargs)
461 'You can build it manually via: '
462 '`layer.build(batch_input_shape)`')
--> 463 self.build(unpack_singleton(input_shapes))
464 self.built = True
465
~\AppData\Roaming\Python\Python37\site-packages\keras\layers\convolutional.py in build(self, input_shape)
139 name='kernel',
140 regularizer=self.kernel_regularizer,
--> 141 constraint=self.kernel_constraint)
142 if self.use_bias:
143 self.bias = self.add_weight(shape=(self.filters,),
~\AppData\Roaming\Python\Python37\site-packages\keras\engine\base_layer.py in add_weight(self, name, shape, dtype, initializer, regularizer, trainable, constraint)
277 if dtype is None:
278 dtype = self.dtype
--> 279 weight = K.variable(initializer(shape, dtype=dtype),
280 dtype=dtype,
281 name=name,
~\anaconda3\envs\MyEnv\lib\site-packages\tensorflow\python\ops\init_ops.py in __call__(self, shape, dtype, partition_info)
513 if partition_info is not None:
514 scale_shape = partition_info.full_shape
--> 515 fan_in, fan_out = _compute_fans(scale_shape)
516 if self.mode == "fan_in":
517 scale /= max(1., fan_in)
~\anaconda3\envs\MyEnv\lib\site-packages\tensorflow\python\ops\init_ops.py in _compute_fans(shape)
1445 fan_in = shape[-2] * receptive_field_size
1446 fan_out = shape[-1] * receptive_field_size
-> 1447 return int(fan_in), int(fan_out)
1448
1449
TypeError: int() argument must be a string, a bytes-like object or a number, not 'list'
To me all these seems to be python error and nothing to do with talos (hyperparameter optimization tool)
My questions are
what mistakes am i making in code?
Is there any mistake in my logic of writing code?
I have another query but for that i would start another thread to understand the problem that i am solving of identfying speakers thru MFCC+MFCC_del+MFCC_del_del and using siamese network will it work?
Full code can be accessed from this link but i havent uploaded the datafiles to google drive so it wont work but one may see major part of the full code
https://colab.research.google.com/drive/1Twssq_MiQ4RxMZ69A595SFIgs4JhG18u?usp=sharing
##############################
EDIT: I was just checking one link https://www.kaggle.com/sohaibanwaar1203/talos-hyper-parameter-optimization
In this the code is
params = {'lr': (0.1, 0.01,1 ),
'epochs': [10,5,15],
'dropout': (0, 0.40, 0.8),
'optimizer': ["Adam","Adagrad","sgd"],
'loss': ["binary_crossentropy","mean_squared_error","mean_absolute_error"],
'last_activation': ["softmax","sigmoid"],
'activation' :["relu","selu","linear"],
'clipnorm':(0.0,0.5,1),
'decay':(1e-6,1e-4,1e-2),
'momentum':(0.9,0.5,0.2),
'l1': (0.01,0.001,0.0001),
'l2': (0.01,0.001,0.0001),
'No_of_CONV_and_Maxpool_layers':[1,2],
'No_of_Dense_Layers': [2,3,4],
'No_of_Units_in_dense_layers':[64,32],
'Kernal_Size':[(3,3),(5,5)],
'Conv2d_filters':[60,40,80,120],
'pool_size':[(3,3),(5,5)],
'padding':["valid","same"]
}
lr = params['lr']
epochs=params['epochs']
dropout_rate=params['dropout']
optimizer=params['optimizer']
loss=params['loss']
last_activation=params['last_activation']
activation=params['activation']
clipnorm=params['clipnorm']
decay=params['decay']
momentum=params['momentum']
l1=params['l1']
l2=params['l2']
No_of_CONV_and_Maxpool_layers=params['No_of_CONV_and_Maxpool_layers']
No_of_Dense_Layers =params['No_of_Dense_Layers']
No_of_Units_in_dense_layers=params['No_of_Units_in_dense_layers']
Kernal_Size=params['Kernal_Size']
Conv2d_filters=params['Conv2d_filters']
pool_size_p=params['pool_size']
padding_p=params['padding']
OUTPUT
print (type((params['epochs'])))
<class 'list'>
So i see even in this blog the output is list so i dont understand why it is giving error.
I have the same error and I solved it by forcing the value into int.
encoder.add(Conv1D(filters=(params['filter1']),kernel_size=**int(**(params['kernel_size1'])**)**, padding='same', activation='relu',input_shape=input_shape,kernel_initializer=W_init, bias_initializer=b_init))
This solved my problem.